Shipping LLMs in Production: An Azure AI Foundry Field Guide

Every enterprise wants an LLM feature; almost none are ready for one. The gap is rarely the model — it's everything around the model: evaluation, guardrails, latency budgets, cost controls, and the boring operational plumbing that separates a demo from a feature people depend on. At XAI.ma we run full-stack AI deployment cycles on Azure AI Foundry, and this is the field guide that emerged.

Start from the model catalog, not the frontier model

The first decision is which model, and the model catalog changes the conversation. Frontier models are not automatically the right answer. Enterprise features are graded on behavioral criteria — groundedness, refusal precision, format compliance, cost, latency — and a smaller model often beats a frontier one on the metrics that actually matter once you've engineered the context.

# model selection is an evaluation problem, not a taste problem
candidates = ["gpt-4.1-mini", "gpt-4o", "llama-3.3-70b"]
scores = evaluate(candidates, eval_set, metrics=["groundedness", "latency_p95", "cost_per_1k"])
winner = max(candidates, key=lambda m: weighted_score(scores[m]))

The discipline: decide before you tune. If you haven't defined the metric that makes one model better than another, you'll argue about it on a call for months.

Prompt flow: prompts become assets

Foundry's Prompt Flow turns prompts, tools, and evaluation into versioned assets instead of strings buried in code. That matters more than it sounds. When a prompt changes, you get a diff. When a version regresses, you can roll back. When an auditor asks why the model started refusing something, there is an answer beyond "a string changed somewhere."

Treat prompts like code: version them, review them, and test them against a golden set before they ship. The prompt is production code, full stop.

Evals are the product's immune system

Azure AI Foundry's evaluation workflows are where the platform pays for itself. You define the criteria that define a good answer for your domain — not generic quality, but groundedness against your sources, adherence to your format, refusal behavior on off-topic inputs — then run every candidate version through the same set.

The evaluation set is the highest-leverage artifact in the whole project. It is built from real production questions, real edge cases, and real failures. Every time something regresses in production, it goes into the eval set, which means the eval set compounds: the system can only get more rigorous, never less.

Guardrails before users, not after

Content safety and guardrail configuration are not a checkbox; they are the difference between an enterprise feature and a liability. The configuration is behavioral: what the system refuses, how it refuses, and — critically — how it behaves when it cannot answer. A good guardrail config produces graceful refusals that keep the user in the flow, not robotic apologies.

The pattern that works: guardrails at the boundary, not the prompt. Model inputs and outputs pass through policy enforcement as data, so the policy is auditable and the prompt stays clean.

Deployment is a throttled, measured process

Real users do not need the latest model on day one. The deployment pattern that survives enterprise scrutiny:

1. Batch validation against the eval set — no score, no deploy.
2. Canary rollout with a percentage of traffic, monitored against guardrails and error rates.
3. Automatic rollback on a defined threshold — the system restores the previous version faster than a human can intervene.

None of this is glamorous. It is the difference between "we deployed a model" and "we ship model updates without incidents."

Cost and latency are features

For an enterprise customer, a response that takes eight seconds is broken regardless of its quality. Foundry gives you the observability to budget both: token usage per request, latency percentiles, cost per operation. Set budgets before launch — cost per query, max latency — and watch them like uptime.

The cheapest latency wins are almost always contextual: shorter prompts, tighter retrieval, cached system prompts. Optimize those before you blame the model.

Security, governance, and data sovereignty

Enterprises will not let you near their data on vibes. The conversation that unlocks real deployments is the one about where the data goes and who controls it. Azure AI Foundry's enterprise backbone — managed identities, private endpoints, network isolation, and audit logs — is often the reason a conservative client says yes when a raw API integration would have been a no.

Three governance rules that always earn their keep:

• Data stays in the tenant. Model calls run against the customer's own endpoints and data boundaries; nothing trains on their prompts by default. That single property closes more security reviews than any feature list.
• Every call is auditable. Inputs, outputs, model version, latency, and decision are logged. When the audit team asks "what did the model see and say," the answer is a query, not a story.
• Retention and deletion are configured, not assumed. Define how long prompts and logs live, and make the cleanup automatic. Sovereign AI — a term that comes up constantly in my work — is not a marketing phrase; it is a set of enforceable guarantees about who holds the keys.

The pattern here is identical to the delivery discipline: the client's trust comes from provable boundaries, not promises. If you can demonstrate where the data lives and what happens to it, the governance conversation goes from blocker to checkbox.

The pattern that transfers

Every LLM product follows the same spine: catalog selection → prompt-flow engineering → guardrails → evals → throttled deployment → monitoring. Get the spine right and the specific model, framework, or platform is almost a detail. Get the spine wrong and the best model in the world will not save you.

This is the shape of the work we do at XAI.ma — and it's the discipline that lets enterprises bet on AI without betting the company. Get in touch if you're deploying LLMs in production and want a second opinion.