MLOps vs LLMOps: Infrastructure for Foundation Models

Viacheslav Dubrov

2025-05-06 · Updated 2026-10-08 · 16 min · AI Infrastructure

This article is for ML and platform engineers who already run MLOps and now operate a system built on a foundation model. In such a system, a regression can come from a changed prompt, index, tool permission, or route while the model weights stay the same. The article covers which MLOps practices still apply, what else you need to version when prompts, retrieval, tools, and routing can change behavior, how to evaluate and trace that system, and which tools do each job.

What MLOps already solved

Each row of the diagram is one step of the MLOps loop: what MLOps already does there, and what a foundation-model system adds. The dashed edge carries reviewed production failures back to the offline check as tests. MLOps established these practices, and they still apply when the model generates text:

• lineage from data, code, configuration, and model to a release
• repeatable training or build pipelines
• offline validation before promotion
• registries and immutable artifact identity
• staged rollout, service-level objectives, rollback, and incident response
• infrastructure telemetry and capacity planning
• access control, retention, and audit policy for data

Most of the orchestration tools you already run still fit, but not on every path:

Foundation models do not remove these needs. They make the old phrase “model version” too small.

The release unit became a system manifest

The diagram shows the manifest of a support assistant in two releases. The top rows are what MLOps already versions, the bottom rows are what a foundation-model system adds, and the highlighted rows are the only entries that changed.

For a foundation-model system, I recommend a manifest that records at least:

application code revision
model provider + model/revision + serving configuration
system and task prompts
response schema + decoding parameters
retrieval corpus snapshot + parser + chunker + embedding model + index
tool definitions + implementation revisions + permission policy
routing, fallback, caching, and budget policy
safety and business-rule revisions
evaluation dataset + scorer revisions

The exact list depends on the system. The rule is to identify every independently changing component that can alter user-visible behavior.

Treat a provider model name as changeable unless the provider documents an immutable revision. A checkpoint hash helps identify self-hosted weights, but I would still record the runtime, quantization, chat template, and parallelism configuration. The model entry comes from a registry: use MLflow Model Registry for weights and adapters you train, and the Hugging Face Hub for open weights, pinned to a commit with the revision argument rather than a branch name.

Store the manifest as a file next to each release. When a release regresses, diff it against the previous one. In this example for a support assistant, only the answer prompt and the retrieval snapshot changed:

-    answer: prompts/answer.md@v9
+    answer: prompts/answer.md@v10
-    corpus: kb-snapshot-2026-09-28
-    index: kb-idx-2026-09-28-01
+    corpus: kb-snapshot-2026-10-01
+    index: kb-idx-2026-10-01-01

The diff names the components to check first: the answer prompt and the new index. The weights, tools, and routing did not change.

Which component to inspect first

The manifest diff tells you what changed. The kind of failure tells you where to look when the diff alone does not explain it:

For the choice between changing the prompt, adding retrieval, and fine-tuning, see fine-tuning vs RAG vs prompting.

Five components that can change behavior

The sections below take five of the manifest diagram’s components in order: the model and its serving, then prompts and schema, the retrieval corpus and index, tools and permissions, and safety and business rules. Each section says what to record, what to test, and which tools do the job. Routing and fallback have their own section with the gateway.

1. Model and serving

The first decision is whether a provider hosts the model or you serve it yourself. A hosted API removes GPU work. Each request crosses the network, you pay per token, the data you send leaves your network, and the provider controls the weights, revisions, quotas, and price. Self-hosting gives you control of the revision, quantization, batching, region, and cache. You also run the GPU capacity, and you pay for it whether it is busy or idle.

The top of the diagram shows where the data goes and who controls what in each mode. The chart below it shows how each mode’s cost grows with traffic. The lines cross at the break-even traffic, where the monthly GPU bill equals the API bill for the same tokens. In both modes, the manifest records the route and the same task suite runs.

To find the break-even point, divide the GPU cost per month by the API price per token. An example with list prices on 7 October 2026 for Llama 3.3 70B:

• Together AI charges $1.04 per million tokens, input or output.
• In FP8 the 70 billion weights take about 70 GB, so the model fits on two 80 GB H100 GPUs with room for the KV cache. Lambda rents a two-GPU H100 SXM instance at $4.19 per GPU per hour.
• Two GPUs for 730 hours (one month) cost 2 × 730 × $4.19 = $6,117.
• $6,117 ÷ $1.04 per million tokens = about 5.9 billion tokens per month, or about 2,240 tokens per second, every second of the month.

Below that traffic, the API costs less. Above it, self-hosting costs less only if the two GPUs can serve that rate, so measure the throughput with your own prompts first. The example counts only GPU rent. Engineering time, a second replica for availability, and storage move the break-even point higher.

If you self-host, choose the serving engine by deployment shape:

Models too large for one GPU need tensor or pipeline parallelism at serving time; Scaling LLMs with multi-GPU and multi-node parallelism explains each method. To choose a weight format for the runtime and hardware, see LLM quantization formats.

In either mode, include latency, errors, throughput, saturation, cost, and task-quality checks in the release decision.

2. Prompts and schema

For a system that assembles requests at runtime, version the assembled request rather than prompt text alone: message ordering, tool descriptions, response schema, decoding, retries, truncation, and surrounding code can each change behavior.

A small prompt change can change the output format. Here are three versions of a sentiment prompt:

v1: "Classify this text as positive or negative: {text}"
v2: "You are a sentiment analyzer. Classify: {text}"
v3: "Analyze sentiment. Return only 'positive' or 'negative': {text}"

v1 and v2 can return a sentence such as “The sentiment is positive.” v3 asks for one word. The code that parses the answer has to change with the prompt, so the prompt, the schema, and the parser go into the manifest together.

Treat parsable JSON as a transport check, then test the task’s business invariants separately.

3. Retrieval corpus and index

Retrieval adds an independent data product between the source of truth and the model:

The top row is built ahead of requests, and each step records a version. The bottom row follows one request, and the numbered marks show the three separate tests: authorization, the retrieved evidence, and the answer.

For a retrieval-augmented system, retain the source revision, parser and chunker versions, embedding identity, index build, access-control metadata, and deletion state. The original RAG design separates retrieval from generation; use that separation to test evidence retrieval, authorization filtering, and evidence use independently. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks describes the retrieve-then-generate architecture. RAG evaluation metrics covers the metrics for each check, and RAG evaluation tools compares the libraries that compute them.

Choose the index store by who uses it. An in-process library such as FAISS is enough when one service owns the index and it fits in that service’s memory. Use a vector database server such as Pinecone, Weaviate, or Milvus when several services share the index, when you need metadata filters at query time, or when the index outgrows one machine. For the retrieval design itself (one pass, iterative search, or GraphRAG), see RAG architecture.

Choose the update method by how stale an answer may be. A scheduled rebuild works when a day-old answer is acceptable. When a change must appear within minutes, stream updates, for example Kafka events into an embedding job that writes to the index. For an immutable snapshot, record which index build served each answer. For a mutable index, also trace the query-visible ingestion revision or watermark, where available, and the retrieved document revisions for each request. Pinecone’s query-response log sequence numbers illustrate why a write receipt alone does not identify what a query saw. Keep immutable snapshots or replayable source history if exact data replay is required; a watermark alone cannot recover overwritten content.

Do not use a vector index as a substitute for a feature store or warehouse: similarity retrieval, point-in-time features, and analytical facts have different query and consistency needs.

The three cards compare a historical feature lookup, similarity retrieval, and aggregation. The historical lookup returns values valid at the past event time, a vector database ranks documents by query similarity, and a warehouse returns totals over many rows. Keep the warehouse for analytics.

In Feast, historical joins build training and evaluation data for past event times. Its online store serves the latest materialized feature values per entity at low latency.

4. Tools and permissions

When a model can call APIs, put these checks around every tool call: tool authentication, least privilege, argument validation, timeouts, idempotency, approval policy, and postcondition checks. NIST’s Generative AI Profile identifies risks from harmful content, privacy, and security.

The diagram shows these checks in order, where a call stops (a rejected permission or argument, a denied approval, or a result that does not match the expected postcondition), and the trace event each step records.

A timeout can leave a write’s outcome unknown even after the remote service commits it. Reconcile the outcome before another write: check the operation or resource, or use a documented idempotent replay. Retry only under a supported idempotency contract or another verified safe recovery method; AWS explains the required server support. A repeated key can return the original failure, as Stripe’s idempotency contract documents. Use a supported compensation action when available, and send unresolved outcomes for review.

Trace which tool was offered, selected, called, rejected, retried, reconciled, and confirmed. A fluent final answer cannot prove that the action path was correct.

5. Safety and business rules

Content filters are one control, not a complete guardrail layer. Threats also include prompt injection, cross-tenant retrieval, secret disclosure, excessive agency (an entry in the OWASP Top 10 for LLM Applications), untrusted model files or custom code loaded at runtime, and unsafe tool arguments.

Express deterministic business rules outside the model where possible. Define residual risks, test adversarial cases, and assign an owner for policy changes. NIST’s Generative AI Profile is a useful risk inventory, not a turnkey acceptance test.

Evaluation decides the release

For release decisions on free-form output, do not rely on one aggregate accuracy number. Use several evidence types and define which one is authoritative for each failure mode.

An evaluation stack for a foundation-model system has six layers:

1. Deterministic checks: schema validity, citations present, allowed tools, argument constraints, policy rules, latency, and budget.
2. Component metrics: retrieval recall and ranking, tool-selection accuracy, tool-argument correctness, and route selection.
3. End-to-end tasks: representative inputs with explicit success criteria and slice labels.
4. Model-based scoring: rubric-driven judgments calibrated against expert labels and monitored for judge drift.
5. Human review: ambiguous, consequential, novel, or sampled cases where automation is not authoritative.
6. Adversarial tests: injection, data-boundary, abuse, refusal, and side-effect cases tied to the threat model.

Store per-example results, not only averages, so the release review can inspect regressions by language, tenant, document type, or action class.

For a deployment decision, compare a candidate manifest with the current release on the same versioned suite. Set thresholds for quality, safety, latency, and cost together, and reject a cheaper route that fails the task suite.

If every result is within its threshold, start a staged rollout. If one fails, reject the candidate. If a slice result is unclear, send it to human review.

To choose an evaluation tool for traces, datasets, and CI checks, see AI agent evaluation tools.

Traces connect production back to evaluation

The top of the diagram is the trace of one request. Metrics see the slow model call; only the trace shows the unauthorized document and the wrong account. The bottom row is what happens next: the reviewed failure becomes a labeled regression case, evaluation compares a candidate change against the current release, and the dashed edge carries the staged release back to production.

Infrastructure metrics can show a slow model call. They cannot show that retrieval returned an unauthorized document or that a tool was called with the wrong account. MLflow’s tracing documentation describes traces that capture intermediate steps and metadata needed to inspect this path.

Capture a trace across the decision path:

• release-manifest identity and request correlation
• model and provider calls, latency, usage, and finish state
• retrieval query, query-visible index revision or watermark, document identifiers and revisions, scores, and filtering decisions
• prompt/template revision without indiscriminately storing sensitive content
• tool offers, arguments, approvals, results, and side-effect identifiers
• policy decisions, retries, fallbacks, and final outcome

Traces are a new store of sensitive data. Apply minimization, redaction, tenant isolation, encryption, sampling, retention, and access review before collecting full prompts or documents. OpenTelemetry’s GenAI semantic conventions can help with interoperability, but they remain in development. Pin the convention or schema version, instrumentation, and collector versions.

User feedback can prioritize investigation, but a thumbs-up is not ground truth. Preserve the surrounding trace and obtain expert labels for consequential cases. Label Studio collects labels for text, images, and other data types; Argilla collects feedback on text and LLM outputs and can push the datasets to the Hugging Face Hub. AI agent evaluation in production shows how to turn labeled traces into a versioned regression suite.

The serving gateway owns routing and policy

In the diagram each request takes one route through the gateway: by plan, by a small traffic share for a new model, or to a fallback when the provider fails. Every route returns the same response contract, and the task suite evaluates every route offline before promotion.

A gateway can decouple application clients from providers or self-hosted engines. Responsibilities worth putting there include:

• authentication, tenant budgets, quotas, and rate limits
• stable request and response contracts
• route selection by capability, region, latency, or evaluated quality
• routing by subscription tier, for example a larger model for paid plans
• sending a small share of traffic to a new model before a full rollout
• bounded retries, circuit breakers, and explicit fallback semantics
• cache partitioning and sensitive-data policy
• usage attribution and release-manifest propagation

LiteLLM Proxy is one open-source gateway: it exposes many providers through one OpenAI-compatible API and sets budgets and rate limits per key or team.

Fallback is a behavior change as much as a reliability mechanism. If a smaller model, alternate provider, or reduced context changes task quality, evaluate and trace that branch as its own route.

Do not put every orchestration decision into the gateway. Keep domain rules close to the application and make ownership visible.

Fine-tuning and LoRA

The last row of the failure table in “Which component to inspect first” is the case for fine-tuning: the behavior the task needs cannot be recovered from the prompt or the retrieved context. LoRA freezes pretrained weights and adds trainable low-rank matrices, reducing the number of trainable parameters for the downstream task. That optimization does not remove dataset governance, base-model licensing, evaluation, serving compatibility, or rollback requirements.

Record the adapter settings in the manifest next to the base model revision: the rank (r), the scaling factor (lora_alpha), and which layers it changes (target_modules). Each one changes what the adapter learns. The LLM fine-tuning guide covers datasets, QLoRA, and training tools, and the LoRAX serving guide covers serving many adapters on one base model.

A practical adoption sequence

1. Define the user task, harm boundaries, service objectives, and cost envelope.
2. Create the release manifest before introducing a prompt registry, vector database, or gateway product.
3. Build a small, sliced evaluation set and deterministic component tests.
4. Instrument one end-to-end trace with privacy controls and stable release identities.
5. Promote through shadow, canary, or limited traffic with an explicit rollback trigger.
6. Turn reviewed production failures into regression cases and repeat.

Add infrastructure only when it does a job the manifest names or removes a measured bottleneck.

References

• MLflow: evaluating production traces - Reuse production traces for evaluation and score intermediate trace information.
• MLflow: LLM and agent tracing - Capture intermediate steps and trace metadata for investigation.
• OpenTelemetry GenAI semantic conventions - Current development-status conventions for GenAI spans, metrics, events, and provider-specific data.
• NIST AI 600-1: Generative AI Profile - Risk categories for prompt injection, privacy, security, and related generative-AI concerns.
• OWASP Top 10 for LLM Applications - Includes excessive agency (LLM06:2025) and prompt injection (LLM01:2025).
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks - Original retrieve-then-generate architecture.
• LoRA: Low-Rank Adaptation of Large Language Models - Frozen pretrained weights plus trainable low-rank matrices for downstream adaptation.
• PEFT LoRA reference - LoraConfig parameters and get_peft_model.
• vLLM documentation - Serving engine with continuous batching, quantization, multi-LoRA, and an OpenAI-compatible server.
• NVIDIA Dynamo-Triton - Inference server formerly named Triton Inference Server, with a TensorRT-LLM backend.
• LiteLLM Proxy - Open-source gateway with one API across providers, budgets, rate limits, and fallbacks.