Building AI Agents That Actually Work in Production

The Demo-to-Production Gap

A ChatGPT wrapper takes 2 hours to build. A production AI agent that handles 10,000 tasks per day, gracefully degrades when the LLM is down, retries failed operations, logs every decision for audit, and alerts humans when confidence drops below threshold — that takes engineering.

Here's what most demos ignore:

• Rate limiting — LLM APIs throttle you. Your agent needs a queue, backoff, and priority routing.
• Observability — When an agent makes a bad decision at 3am, you need the full trace: input, prompt, model response, action taken.
• Fallback chains — Model down? Switch providers. Provider down? Degrade gracefully. Never let the system halt.
• Human-in-the-loop — Agents should escalate when uncertain, not guess. Low confidence = human review.
• Idempotency — If the agent crashes mid-task and restarts, it should pick up where it left off, not re-process everything.

Our Agent Architecture: MCP + Rust

Every agent we build follows the Model Context Protocol (MCP) — an open standard for connecting AI models to tools. Each agent is a Rust binary that exposes tools via MCP and connects to any LLM:

// Agent architecture — each is a standalone Rust service
┌─────────────────────────────────────────────┐
│  AI Agent (Rust binary, ~3MB)               │
│                                              │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  │
│  │  MCP API  │  │  Task    │  │  Health   │  │
│  │  (tools)  │  │  Queue   │  │  Monitor  │  │
│  └──────────┘  └──────────┘  └──────────┘  │
│       │              │              │        │
│  ┌──────────────────────────────────────┐   │
│  │        State Machine (sqlx)          │   │
│  │  pending → processing → done/failed  │   │
│  └──────────────────────────────────────┘   │
│       │              │              │        │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  │
│  │ LLM API  │  │ Database │  │  Alerts   │  │
│  │ (Claude) │  │ (Neon PG)│  │ (Resend)  │  │
│  └──────────┘  └──────────┘  └──────────┘  │
└─────────────────────────────────────────────┘

Pattern: State Machine for Every Task

The most important pattern: every agent task is a state machine persisted in PostgreSQL. If the agent crashes, it resumes from the last saved state.

CREATE TABLE agent_tasks (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  agent_name TEXT NOT NULL,
  state TEXT NOT NULL DEFAULT 'pending',
  -- 'pending' | 'processing' | 'awaiting_human'
  -- | 'completed' | 'failed'
  input JSONB NOT NULL,
  output JSONB,
  confidence FLOAT,
  error TEXT,
  attempts INT DEFAULT 0,
  max_attempts INT DEFAULT 3,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- Agent picks up tasks atomically
-- (no double-processing even with multiple replicas)
UPDATE agent_tasks
SET state = 'processing', updated_at = NOW()
WHERE id = (
  SELECT id FROM agent_tasks
  WHERE state = 'pending'
    AND attempts < max_attempts
  ORDER BY created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING *;

The FOR UPDATE SKIP LOCKED is critical — it allows multiple agent replicas to pull tasks without conflicts. This is how you scale agents horizontally.

Pattern: Confidence-Based Routing

Not every task needs the same treatment. Our agents score their own confidence on every decision:

• High confidence (>0.9) — Auto-execute. No human review.
• Medium confidence (0.6-0.9) — Execute but flag for spot-check audit.
• Low confidence (<0.6) — Route to human. Agent explains its reasoning and waits.

This means 80%+ of tasks are fully automated, while humans only see the edge cases — the exact inverse of a typical manual process.

Real Results

One client's document processing pipeline:

• Before: 12 people, 3 weeks per batch, 94% accuracy
• After: 1 agent, 4 hours per batch, 99.97% accuracy, 24/7 operation
• Cost: $1.2M/year → $14K/year (agent infra + LLM API costs)

Open Source

We're progressively open-sourcing our agent framework. Our MCP server base library, task queue, and health monitoring are available on GitHub. The goal: make production AI agents as easy to deploy as a web server.

Follow our work at github.com/pdaxt