The 0→1 Craft

Evaluating LLMs in Production: The Metrics That Actually Matter

MR
Mrinal Rai
5 min read

Stop Trusting Public Benchmarks

Every week, I see another foundation model claim to have beaten the state-of-the-art on a standard academic test. It looks great in a whitepaper, but it means absolutely nothing when you are trying to parse unstructured transport invoices from a mid-sized logistics firm in Haryana. Public benchmarks measure how well a model performs on static data, not how it behaves when the prompt deviates by one percent or the latency spikes during a peak shipping season.

When we started integrating generative AI into our visibility platform, we spent too much time looking at MMLU scores. We quickly learned that a model's 'intelligence' is irrelevant if it cannot handle the messy, noisy reality of Indian supply chain data. If your LLM-based agent hallucinates a delivery status for a truck stuck at a toll booth, your client loses money. In production, precision is not a feature; it is a business requirement.

Building Your Golden Dataset

The only metric that matters is how your model performs on your proprietary data. I call this the 'Golden Set.' It is not a generic test suite; it is a collection of 500 to 1,000 highly specific examples that represent the exact edge cases your users face daily. If you are building a tool to summarize dispatch reports, your golden set should include scanned PDFs with poor lighting, regional dialects in driver logs, and incomplete trip sheets. If the model fails on those, it is not production-ready, no matter what the benchmark says.

Defining Ground Truth

We treat our golden set as an evolving living document. Every time we encounter a failure, we don't just patch the prompt. We turn that failure into a new entry in the golden set. This prevents regression. When you have a solid golden set, evaluation becomes mathematical rather than anecdotal. You run your pipeline, compare outputs against your ground truth, and calculate a strict success rate.

In the world of B2B SaaS, the difference between a demo and a product is a robust evaluation framework that catches the edge cases you haven't even thought of yet.

The Three Pillars of Production Evals

To move past the prototype stage, I rely on three specific measurement layers:

  • Deterministic Logic Checks: We force the output into a specific schema. If the model returns a date format that doesn't match our database or misses a required field, the test fails immediately. No semantic analysis needed.
  • LLM-as-a-Judge: We use a higher-order, more capable model to grade the outputs of our production model based on a rubrics-based prompt. We evaluate for tone, factual accuracy, and brevity. It’s not perfect, but it scales far better than manual review.
  • The 'Human-in-the-Loop' Threshold: We have a strict policy: any edge case where the model’s confidence score is below 0.85 must trigger a human review. We have built an internal dashboard where our support operations team can correct the model's output in real-time, feeding that data back into our fine-tuning loop.

When to Ship

You will never achieve 100 percent accuracy. Stop chasing it. The goal is to reach a performance baseline that is demonstrably better than the manual, error-prone processes your customers are currently using. If your LLM reduces the time to process a shipment document by 70 percent, even with a small error rate that humans can catch, that is a massive win. Ship it, observe the behavior in the wild, and keep refining your golden set. Production is where the real learning starts.

aisaasengineeringproduct-managementscheduled30
Join the Chronicle
Get essays, reflections, and builder wisdom — delivered rarely, crafted carefully.
Continue Reading