Why We Build AI Systems in Rust (And You Should Too)

The Problem With Python in Production

Python is perfect for prototyping. You can go from idea to working demo in hours. But when that demo needs to handle 10,000 requests per second, run 24/7 with sub-millisecond latency, and not cost you $2,400/month in cloud bills — Python falls apart.

Here's what we measured across our client deployments:

These aren't benchmarks on a test machine. These are production numbers from real client deployments running on Google Cloud Run.

Our Stack: Rust + Axum + SQLx

Every AI system we build uses the same battle-tested stack:

• Axum 0.8 — Tokio-based web framework. Type-safe routing, middleware, extractors.
• SQLx — Compile-time checked SQL queries. No ORM overhead. Catches SQL bugs before deployment.
• Askama — Type-safe HTML templates. Server-rendered pages at sub-millisecond speed.
• tokio — Async runtime that handles 100K+ concurrent connections per process.

Open Source: Our MCP Server Framework

We open-source the infrastructure we build on. Our MCP (Model Context Protocol) server framework is used across all 230+ tools we've deployed:

// Rust MCP server — 50 lines to production
use axum::{Router, Json, extract::State};
use serde::{Deserialize, Serialize};
use sqlx::PgPool;

#[derive(Deserialize)]
struct ToolRequest {
    name: String,
    arguments: serde_json::Value,
}

#[derive(Serialize)]
struct ToolResponse {
    content: Vec<Content>,
}

async fn handle_tool(
    State(pool): State<PgPool>,
    Json(req): Json<ToolRequest>,
) -> Json<ToolResponse> {
    match req.name.as_str() {
        "search_contacts" => {
            let query = req.arguments["query"]
                .as_str().unwrap_or("");
            let results = sqlx::query_as!(
                Contact,
                "SELECT * FROM contacts
                 WHERE name ILIKE $1 LIMIT 20",
                format!("%{query}%")
            )
            .fetch_all(&pool)
            .await
            .unwrap_or_default();

            Json(ToolResponse {
                content: vec![Content::text(
                    serde_json::to_string(&results)
                        .unwrap()
                )],
            })
        }
        _ => Json(ToolResponse {
            content: vec![Content::text(
                "Unknown tool"
            )],
        }),
    }
}

This pattern — Axum router + SQLx compile-time queries + JSON tool interface — is the foundation of every AI tool we build. It compiles to a single 3MB binary that starts in 5ms.

When to Use Rust (And When Not To)

We're not Rust zealots. Here's our honest take:

Use Rust when:

• Your AI system needs to run in production 24/7
• Latency matters (APIs, real-time processing, user-facing)
• Cloud costs are a concern at scale
• You need a single binary deploy (no dependency hell)
• Memory safety is non-negotiable (finance, healthcare, government)

Keep Python for:

• Model training and experimentation (PyTorch, notebooks)
• Quick prototypes and one-off scripts
• When your team only knows Python and speed-to-market trumps everything

Our approach: prototype in Python if needed, then build the production system in Rust. The prototype proves the concept. The Rust system runs the business.