How Instacart's Blueberry AI Helps On-Call Engineers Tame Production Bugs

When production goes sideways, the first twenty minutes are chaos. You're juggling Slack alerts, digging through dashboards, trying to remember which service owns what, and praying the logs make sense. Instacart knows this pain all too well. Their answer? An AI assistant named Blueberry that sits right inside your incident channel and does the grunt work before you even finish your coffee.

The Problem: Context Overload Before You Even Start

In large-scale operations, the bottleneck isn't usually finding the fix—it's gathering the context to even know where to look. A typical incident kicks off with a flood of questions: Which service is this? What changed in the last deploy? Are the error rates spiking? Did we see this before? Engineers burn precious minutes (sometimes hours) just assembling the puzzle pieces before they can start diagnosing.

That's the gap Blueberry was built to close. It's not a magic wand that auto-fixes everything; it's more like a hyper-efficient junior partner that hands you a neatly organized incident brief the moment an alert fires.

Meet Blueberry: An AI That Does the Homework

Blueberry isn't a single monolithic bot. When an alert triggers, it spawns about ten sub-agents in parallel. These agents fan out across Instacart's internal systems—pulling service ownership data, scanning recent deployments, analyzing logs and metrics, and searching through years of past incident reports. Within roughly three minutes, they generate a root-cause hypothesis backed by actual evidence, and drop it right into the Slack thread where the on-call engineer is already working.

That's the key: the engineer never leaves Slack. No tab-hopping, no frantic Googling, no mental gymnastics to remember where the runbook lives. The context comes to them.

Numbers That Actually Mean Something

Instacart shared some impressive stats from April: Blueberry completed around 25,000 diagnoses across more than 270 Slack channels. But the number that really jumps out is accuracy. When they first rolled it out, the system's diagnostic accuracy sat at 60%. After hooking it up to over 14 years of historical incident data as a factual backbone, that number jumped to above 90%.

That's not just a cool demo—that's a system that's genuinely useful. And it's not just about accuracy; it's about speed. Getting a plausible root cause in three minutes versus thirty means you can start triaging immediately, even if you end up going in a different direction.

It's Not Magic: How Blueberry Stays Grounded

The biggest fear with AI in production ops is hallucination. You don't want a bot confidently telling you the database is the problem when it's actually a bad deploy. Blueberry tackles this by grounding its reasoning in real data. It's not just a generic language model riffing on probabilities; it's connected to Instacart's internal resources—incident history, service ownership, logs, deployment records, and other debugging signals.

The architecture uses a tool-aware framework. Agents can call out to connected systems to retrieve information, and they maintain persistent state throughout the investigation. That means they remember what they've already checked and what they still need to look at, which is crucial for complex incidents that span multiple services.

How It Fits Into the Incident Workflow

Blueberry integrates with Instacart's Slack-based incident response process. When an alert fires, the system kicks in automatically. It doesn't just spit out a guess—it provides a structured summary: here's what's happening, here's what we've found so far, here's our leading hypothesis, and here's the evidence backing it up.

Importantly, Blueberry doesn't take any action on its own. It doesn't deploy hotfixes or roll back services. The human engineer remains in charge of all decisions and actions. The AI is there to inform, not to act. That's a deliberate design choice, and it's the right one for now—nobody wants an autonomous agent making changes to production without explicit sign-off.

Learning From Every Incident

One of the coolest aspects is how Blueberry gets smarter over time. It doesn't just rely on static data; it learns from each incident it helps resolve. The system captures the knowledge gained during the debugging process—what was tried, what worked, what didn't—and folds that back into its knowledge base. This means future incidents benefit from the collective experience of every team that's used it.

Instacart's software engineering director, Alan Wong, puts it well: Blueberry changes the starting point for on-call engineers. Instead of starting from zero, they begin with a rich set of context—logs, relevant system info, and historical comparisons—so they can dive straight into deeper analysis.

What This Means for Bug Tracking and Incident Management

Blueberry is a prime example of how AI is reshaping incident management. It's not replacing the bug tracking system; it's augmenting the human workflow that surrounds it. By automating the tedious parts—context gathering, hypothesis generation, pattern matching against past incidents—it lets engineers focus on the actual problem-solving.

For anyone who's ever been paged at 3 AM for a mysterious error, this is a huge relief. The AI doesn't solve the bug for you, but it hands you a well-lit map of the dark cave. That's a game-changer in terms of mean time to resolution.

Beyond the Hype: What Makes It Work

Instacart's experience suggests that successful AI ops tools aren't just about the model. They're about the engineering around it: the context you feed it, the workflows you build, the tools you integrate, and the feedback loops you establish. Blueberry's monthly stats back this up—99.9% workflow success rate, over 58,000 MCP tool calls, and adaptability to about 60 different team workflows.

So, what can other teams learn from this? Start with your incident history. That's your goldmine. Feed it to your AI, connect it to your observability stack, and let it do the heavy lifting of context assembly. The future of incident response isn't about replacing humans—it's about giving them superpowers.