Barndoor.ai
CASE STUDY
How Moda’s SRE Agent Helped Barndoor AI Reduce On-Call Burdens
Published · By Moda Labs
Highlights
450 alert groups triaged a month, zero touched first by a human. Every alert gets acknowledged, investigated across Grafana, code, Kubernetes, and the database, and closed with a root cause and a Jira ticket on the incident record.
Ack time down 90%, to 40 seconds. On-call engineers had a median of 6 minutes to respond to alerts before, but the proactive SRE agent brought it down to 40 seconds.
Incident-response capacity 2x’d. Alert-group volume doubled from 200 → 450, but the amount of allocated incident response hours from engineers remained flat.
6 weeks from a 2,500-line Go loop to a 620-line declarative agent. A deployed agent built on Bobi, Moda’s open-source framework for agent teams, with flexibility to update and refine the agent in YAML.
Greatly improved visibility. An alert setup that can err on the side of false positives without generating a level of noise that’s unmanageable.
The Challenge
Barndoor AI offers AI governance software that helps companies deploy AI and agents safely with their AI gateway. Maintaining the platform meant 75+ alert rules, 700+ actionable firings in a typical month, translating to ~60 dev hours a month on incident response, which is almost half a developer.
Right before we started, the platform team coded a v1 Go agent: about 2,500 lines of hand-rolled LLM loop, queue consumer, and retry logic. While their home-rolled agent was able to ack alerts and filter out transient ones, it wasn’t able to investigate errors or file and update tickets, and most importantly, every behavior change to the agent meant editing a bespoke runtime.
The Solution
Moda Labs deployed a persistent site reliability engineering agent that handled the first response for alerts, diagnosed the root cause, checked for existing issues in Jira, and posted updates or escalated accordingly.
As a result, we 2x’d the incident response capacity of the team and dropped alert acknowledge time by 90%.
The agent was deployed via our open-source framework, Bobi. It has two roles (director, engineer), one YAML workflow, a tool guide, and a skills index.
How the SRE agent works
At the top level, the director agent handles all human communication via Slack and response to any incoming events from Grafana. It understands what needs to be done, then dispatches a fresh SRE engineer agent to handle investigation and analysis.
The Bobi CLI enables all agents within the SRE team to talk to each other and collaborate.
The alert investigation and triage workflow
When an alert happens, Grafana OnCall fires into the Bobi event bus queue. Then, the SRE agent team begins its workflow:
- 01
Check for notes. The director checks for existing agent notes on this alert group and for sibling groups in the last six hours. It marks duplicates accordingly and escalates for investigation where needed.
- 02
Acknowledge the alert. The director acks the OnCall group and posts “triage started.”
- 03
Grafana investigation. An SRE agent runs PromQL and Cloud Logging queries and is told to push for root cause here. It hands back a finding and a confidence flag.
- 04
Code trace. An SRE agent greps a read-only snapshot of the monorepo baked into the image for the log strings it saw.
- 05
Kubernetes and database, only if needed. If the above wasn’t confident, the SRE agent will look at pod status, events, logs for the alert’s namespace, and cluster health.
- 06
Synthesize findings. The SRE agent generates a root cause analysis with explicit gaps, evidence chain, severity, recommendation, and an escalation call.
- 07
Jira ticketing. The SRE agent runs a workflow, which finds an open ticket for this issue or creates one. This ensures one ticket per alert issue and minimizes noise.
- 08
Post the conclusion. The director agent posts the verdict: alert and root cause, impact, single next action, escalate yes or no, Jira key. Every note lands on the Grafana OnCall alert group and mirrors into the alert’s Slack thread for full auditability.
What live alerts taught us
After a quick initial deployment, we made several improvements to the deployed SRE agent using live alert traffic.
This step of the project is what separates a “set-and-forget” agent that collects dust on a shelf from one that the team relies on every day. In just a few weeks, we noticed and corrected a few things.
Limiting concurrent triage workflows. On July 27, four concurrent triages killed the container at both 1Gi and 2Gi. Barndoor’s alert volume was the driver. The agent now runs one triage at a time.
Expanding alert deduplication. On July 31, an alert group got a full second triage exactly 15 minutes after the first. As a result, we upped the window to 30 minutes to check for duplicates.
The Impact
We noticed meaningful changes as Barndoor moved from no agent to the Bobi SRE agent.
- Who takes the first pass
No agent
On-call engineer
Bobi SRE agent
Agent acks, investigates, tickets, concludes
- Investigation on the incident record
No agent
None
Bobi SRE agent
Grafana, code, k8s, DB evidence chain plus verdict
- Time to first ack, median
No agent
6 min
Bobi SRE agent
40 seconds
- Human-hours on incident response, per month
No agent
50–60h per 200 alert groups
Bobi SRE agent
50–60h per 450 alert groups
Furthermore, the SRE agent enabled Barndoor to set up more alerts so that they have comprehensive coverage of failure modes across the platform. In the past, they had to carefully set alerts to minimize noise of false positives.
With an SRE agent handling triage and initial investigation, they can increase coverage without worrying about the noise.
“You want to err on the side of false positives with alerts, so you have visibility into your product. And with agents, you can have that without worrying about the noise.”
ARCHITECTURE
Looking Under the Hood
Project Setup
- Team
- From Moda Labs, one embedded engineer and one reviewing engineer and light PM support, partnering with Barndoor’s platform engineer and agent owner
- Framework
- Bobi, Moda’s open-source framework for agent teams. The SRE team is a declarative package of about 620 lines of YAML and Markdown with no imperative agent code
- Models
- Claude for both roles, routed through Barndoor’s LLM gateway; Claude Code as the agent runtime
- Tool access
- Barndoor’s own MCP aggregator (Grafana, Kubernetes, CNPG, ArgoCD, Atlassian, skills) with a default-deny policy. The agent is read-only except for one sanctioned write, the Jira ticket
- Runtime
- GKE with a queue-bridge sidecar, a Cloudflare Queue and OpenTelemetry for metrics
The system has four layers:
- 01
Alert fires and Bobi responds, within the AI gateway: Grafana OnCall fires and the event goes into Bobi’s event queue. All actions go through the AI gateway, ensuring that tool calls and agent actions are approved and monitored.
- 02
Initial response, including deduplication: The sidecar pulls each alert on a 15-minute lease, drops duplicates inside a 30-minute window, posts it to Bobi’s event server, and acks the queue only after ingestion succeeds.
- 03
Dispatch of the triage workflow: Bobi’s auto-dispatch launches the triage workflow for every firing alert. One triage runs at a time to avoid concurrency issues.
- 04
Alert investigation and response: The SRE role investigates using MCP and a read-only snapshot of the codebase, and hands back structured fields (finding, confidence, needs database) that the Director turns into notes. The Director handles all communications with users and upstream systems like Grafana.
Why This Worked
Deploy-then-iterate strategy. The Bobi framework allowed Moda to get a functional agent up and running within the first week that consumed real alerts and performed investigations, allowing us ample time to tune the deployment with Barndoor’s engineers against live use cases, resulting in a robust solution along with the knowledge-share necessary for them to take long-term ownership.
A workflow, not a prompt. Barndoor wanted to dictate the investigation order, Grafana first and everything else only when needed, rather than ask a model nicely. The YAML DAG makes that order a fact, and the two-role split means the model that investigates can’t resolve an alert and the model that resolves alerts can’t be talked into it by a log line.
Moda maintains the framework. The bugs in the framework were quickly and easily resolved because we maintained it, and we didn’t have to wait for support from a SaaS product. Each was fixed upstream within days and shipped in a pinned release.
What’s Next
Barndoor will continue expanding the use of its SRE agent, fully replacing the engineer for first response. They also plan on building a downstream engineering agent to pick up Jira tickets from the SRE agent to code and test fixes for issues found from alerts.
