The 0→1 Craft

Building for AI: Three Architectures I Wish I'd Known Earlier

MR
Mrinal Rai
5 min read

Building for AI: Three Architectures I Wish I'd Known Earlier

When we started integrating LLMs into our supply chain visibility stack, the initial euphoria lasted about two weeks. Then came the hallucinations, the latency spikes, and the realization that throwing a prompt at a model is not the same as building a product. In the Indian B2B landscape, where data is messy and enterprise stakeholders have zero tolerance for error, the standard demo-ware approach falls apart. After thousands of trips tracked and countless debugging sessions, here are three architectural patterns that actually work.

1. The Retrieval-Augmented Sandbox

Most engineers treat Retrieval-Augmented Generation (RAG) as a simple vector search fetch. That is a mistake. In supply chain logistics, context is everything. You cannot just pass the latest status of a truck; you need the historical behavior of that transporter, the typical dwell time at that specific warehouse, and the client’s specific communication preferences. We moved to a sandbox architecture where we decouple the retrieval from the reasoning layer. Instead of pulling raw data into the prompt, we pull it into a local state machine that validates the data against our business rules before the model even sees it. If the context retrieval doesn't match the schema, the loop restarts. It prevents the model from being gaslit by garbage data.

2. Orchestration as a Deterministic Layer

We used to let the model decide the entire workflow—if a delay is detected, the agent writes the email, calculates the ETA, and notifies the client. It was elegant until it wasn't. The moment an LLM decided that a truck stuck in heavy rain for six hours was 'not significant enough to report,' I realized we were over-indexing on intelligence. Now, we use a deterministic orchestration layer. The LLM acts as the summarizer and sentiment parser, but the decision to escalate, draft a message, or call an API is managed by a traditional, hard-coded state engine. Think of the LLM as the consultant and the deterministic code as the project manager. The manager always has the final veto.

The most expensive lesson in AI development is believing that your model is the brains of the operation rather than just a very fast, very expensive intern.

3. Continuous Evaluation Loops

You cannot deploy an AI feature and forget it. We built an internal evaluation suite that runs against our production logs every night. Every time a new version of a model is released, or we tweak our prompt templates, the system runs 500 'golden cases' through the pipeline. These are scenarios where we know exactly what the output should be. If the new logic fails even three of these, it doesn't hit production. It is a dry, unglamorous, and absolutely necessary layer of infrastructure. Building for AI is 20 percent model tuning and 80 percent building the testing rig that ensures the model doesn't drift into mediocrity.

Building these architectures requires a shift in mindset. It is not about how cool your model is; it is about how robust your pipes are. When you are tracking a million trips a month, 'probably correct' is not a strategy. It is a liability. Focus on the plumbing, keep your logic deterministic where it counts, and treat your evaluations as your most valuable asset.

aisaasengineeringproduct-managementzero-to-onescheduled30
Join the Chronicle
Get essays, reflections, and builder wisdom — delivered rarely, crafted carefully.
Continue Reading