We gave an AI agent a licence to act - and built the leash

The pager fires. checkout_latency_seconds is climbing, http_5xx_total crosses the threshold. Before I've finished reading the alert, SENTRY - an on-call agent running on TrueForge - has already queried Prometheus, pulled the last GitHub deploy, and correlated the two. It has a theory. It does not act on it. A card appears: Allow or Deny. I click Allow. That pause is the whole point of this post.

Why an agent harness, not just a chatbot#

Chatbots answer questions. Agents act on infrastructure, and acting on infrastructure requires three things a chat window doesn't give you: tools it can actually call, a place to run arbitrary code without touching production, and a control layer that stops it before something destructive happens. That's the job of an agent harness. I built SENTRY on TrueForge, TrueFoundry's open-source harness, specifically to see how much of that control layer I'd get for free and how much I'd have to write myself.

The job#

SENTRY is an on-call incident responder for a payment-failures alert against a self-hosted shop service. When the error rate spikes, it triages over MCP - Grafana and Prometheus for metrics, GitHub for deploy history - correlates a cause inside a Daytona sandbox, and stops cold in front of a human before any rollback executes. Once the fix lands, it verifies recovery against Prometheus and files an RCA issue on GitHub. To make this testable on demand rather than waiting for a real outage, I built a small chaos lab: a Fastify shop service instrumented with prom-client, a synthetic traffic generator running at roughly 4 requests per second, and a custom MCP server, sentry-lab, whose lab_inject_bad_deploy and lab_restore tools are the same ones both the agent and my operator console call. Flip CHAOS_ERROR_RATE to 0.5 and the container breaks itself on cue.

The wiring#

Alert (Grafana) -> TrueForge session -> sentry-oncall agent
  ├─ MCP: grafana connector  -> Prometheus queries (read-only, autonomous)
  ├─ MCP: github connector   -> deploy history correlation (read-only)
  ├─ Subagents               -> parallel triage (metrics scout / log analyst)
  ├─ Daytona sandbox         -> bisect + aggregation code (Code Mode)
  ├─ APPROVAL GATE           -> sentry-lab MCP destructive tools pause here
  └─ Post-recovery           -> verification query + GitHub RCA issue

The agent manifest is the contract. The whole leash is three lines of configuration:

"mcp_servers": [
  { "name": "grafana",    "preload": true },
  { "name": "github",     "preload": true },
  { "name": "sentry-lab",
    "require_approval_for_tools": ["lab_inject_bad_deploy", "lab_restore"] }
]

Grafana and GitHub queries are read-only and run autonomously. Anything touching sentry-lab's injection or restore tools pauses the turn and emits a tool.approval_required event - the harness does this because I declared those tools destructive, not because I asked nicely somewhere in a prompt.

TrueForge runs in hosted mode on a self-hosted server - Docker Compose with Postgres and Redis behind it - driven through its HTTP API and chat UI. Nothing executes until a human resumes it. Ground truth after an Allow: CHAOS_ERROR_RATE=0.5 inside the container, verified by exec-ing into it. After a Deny: the value held at 0.5, the agent reported the denial back and asked for guidance rather than retrying silently.

Skills are just git-backed SKILL.md playbooks living in the repo, registered via the API and loaded progressively - I measured 218 tokens for the two of them in the input breakdown, so this isn't a hand-wave, it's a real budget line. The playbook itself carries the licence language:

7. STOP. Do not call any write/destructive tool. Recommend the action
   and wait for human approval.

For triage speed, two named subagents, metrics_scout and log_analyst, fan out in parallel and their merged verdict matched the numbers I could verify by hand.

The line I'd want a reader to walk away with: the harness did the runtime engineering - execution loop, streaming, tool routing, the gate itself. I did the incident-response engineering - the manifest, two SKILL.md files, three MCP connector registrations, the ~150-line chaos-control server, the victim service, the compose stack, and the tests.

What TrueForge handled#

The part that actually impressed me wasn't the happy path, it was what survived interruption. I killed the server mid-investigation and the session picked back up where it left off - persistence across a hard restart, not a soft reconnect. Subagent orchestration ran with real context isolation rather than dumping everything into one shared window. Oversized tool responses were offloaded to sandbox files with only a preview kept in context, and once history itself crossed roughly 50k tokens the harness compacted it into a summary. After recovery, the RCA didn't come back as a wall of markdown - it streamed in as an inline chart card (Generative UI): baseline value, peak during the incident, post-recovery verification. Sandbox provisioning was on-demand per session - I could watch the sandbox.created event fire and then a Python process print its own status line inside it:

None of that is agent logic I had to write; it's harness plumbing I got to assume.

What broke#

This is the section the organizers explicitly ask for, and it's the one worth trusting most, because none of it is invented.

Qodo, reviewing all 14 merged PRs, caught a real bug before it shipped: the deterministic chaos generator used a linear congruential generator whose multiply step exceeded Number.MAX_SAFE_INTEGER, silently corrupting the sequence. The fix was switching to Math.imul for uint32 arithmetic. Review bots earn their keep on exactly this kind of boring, easy-to-miss precision bug.

Then the tool-name wall: with deferred tool loading, the model called query_prometheus by name, correctly, and TrueForge came back with "not found in tool mapping." Servers that aren't preloaded don't expose callable functions yet. The fix was one line - preload the grafana server - but it cost a debugging hour before I found it.

GRAFANA_API_TOKEN doesn't exist, not really. mcp-grafana v1.1.0 renamed it to GRAFANA_SERVICE_ACCOUNT_TOKEN, and the daemon just logged apiKey=false while my environment file looked perfectly correct. Related: mcp-grafana ships no Docker image at all - the GHCR pull gets denied, and the release assets are plain binaries - so it runs natively on the host behind a bearer token.

Denial semantics surprised me too: a Deny is terminal for that specific tool-call instance. Re-approving the old call id does nothing; the agent has to issue a fresh call. Worth knowing before you demo it live in front of anyone.

And rate() lies right after you fix things - post-restore dashboards stayed red for minutes because a [5m] window keeps averaging in the spike. I ended up asserting recovery on [30s] windows for anything that needs to reflect live state.

Numbers#

Everything ran on free-tier endpoints (Ollama Cloud and other official provider free tiers), with TrueForge treating them as provider entries.

I didn't run a GLM comparison myself, so I won't claim one - but it's worth citing TrueFoundry's own August 2026 benchmark: they report the same accuracy as Claude Managed Agents at 27% lower cost on Opus-class tasks, and roughly 75% cheaper when routing to GLM-5.2 ($2.9 vs $11.8 per run across 14 enterprise tasks, blind LLM judge). Zero dollars went to model APIs on my end; the only real spend was a handful of provisioned Daytona sessions.

Close#

The interesting result here isn't that an agent can investigate an outage - plenty of tools can do that. It's that SENTRY cannot act without a human in the loop, and that constraint made the demo more convincing, not less. A licence to act is only trustworthy if there's a leash attached to it.

Screenshots

Healthy baseline vs incident spike - same Grafana panel, 5xx flat at 0 then shooting to ~0.47/s at 16:20:

Links

• Repo: instax-dutta/trueforge-sentry
• Demo video: (add link after recording)
• Hackathon: The Agent Harness Hackathon
• TrueForge: trueforge.dev | github.com/truefoundry/trueforge

Tags: #ai-agents #mcp #opensource #sre #hackathon

Disclosure: most of the code in this project was written by AI coding agents under my direction, reviewed by Qodo on every one of the 14 merged pull requests, and verified against live infrastructure before it counted. I can walk through the reasoning behind any decision on request.

The Agent Harness Hackathon is organised by WeMakeDevs with TrueFoundry and Qodo. SENTRY is not affiliated with any spy franchise, whatever the codename suggests.