MLOps vs LLMOps: Infrastructure for Foundation Models

Viacheslav Dubrov

2025-05-06 · Updated 2026-07-15 · 7 min · AI Infrastructure

When a foundation-model release regresses, the weight file may be unchanged: a prompt, retrieval index, tool permission, route, or policy can be the moving part. For ML, platform, and AI application engineers who already operate conventional MLOps, the practical artifact is therefore a release manifest that names those dependencies and points to the first control to inspect when a failure appears.

This is an editorial operating model, not a claim that every application needs the same platform. Keep MLOps lineage, staged delivery, monitoring, and rollback; expand the unit you version when generated answers or tool actions make behavior depend on more than the weights.

TL;DR. Keep MLOps lineage, automation, staged delivery, monitoring, and rollback. Extend the release manifest to include provider or weights, prompts, schemas, retrieval state, tools, routing, and policy. Promote that manifest through layered evaluations, then use privacy-aware traces to turn production failures into new tests.

What MLOps already solved

MLOps established controls that do not become obsolete when the model generates text:

• lineage from data, code, configuration, and model to a release
• repeatable training or build pipelines
• offline validation before promotion
• registries and immutable artifact identity
• staged rollout, service-level objectives, rollback, and incident response
• infrastructure telemetry and capacity planning
• access control, retention, and audit policy for data

Foundation models do not remove these needs. They make the old phrase “model version” too small.

The release unit became a system manifest

For a foundation-model system, I recommend a manifest that records at least:

application code revision
model provider + model/revision + serving configuration
system and task prompts
response schema + decoding parameters
retrieval corpus snapshot + parser + chunker + embedding model + index
tool definitions + implementation revisions + permission policy
routing, fallback, caching, and budget policy
safety and business-rule revisions
evaluation dataset + scorer revisions

The exact list depends on the system. The editorial rule is to identify every independently changing component that can alter user-visible behavior.

Treat a provider model name as changeable unless the provider documents an immutable revision. A checkpoint hash helps identify self-hosted weights, but I would still record runtime, quantization, template, and parallelism configuration.

Five failure surfaces expanded

The difference between MLOps and LLMOps is clearer in failure analysis than in tool lists.

1. Model and serving

For planning, separate hosted-model concerns (provider availability, quotas, regional processing, and usage cost) from self-hosted concerns (weight supply, GPU capacity, batching, cache policy, quantization, and serving). The relevant controls depend on the chosen deployment.

In either mode, include latency, errors, throughput, saturation, cost, and task-quality checks in the release decision.

2. Prompt, schema, and orchestration

For a system that assembles requests at runtime, version the assembled request rather than prompt text alone: message ordering, tool descriptions, response schema, decoding, retries, truncation, and surrounding code can each change behavior.

Treat parsable JSON as a transport check, then test the task’s business invariants separately.

3. Retrieval and context

Retrieval adds an independent data product between the source of truth and the model:

For a retrieval-augmented system, retain the source revision, parser and chunker versions, embedding identity, index build, access-control metadata, and deletion state. The original RAG design separates retrieval from generation; use that separation to test evidence retrieval, authorization filtering, and evidence use independently. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks describes the retrieve-then-generate architecture.

Do not use a vector index as a substitute for a feature store or warehouse: similarity retrieval, point-in-time features, and analytical facts have different query and consistency needs.

4. Tools and actions

When a model can call APIs, include tool authentication, least privilege, argument validation, timeouts, idempotency, approval policy, and postcondition checks in the operational boundary. NIST’s Generative AI Profile identifies risks from harmful content, privacy, and security; the listed controls are a scoped engineering recommendation, not a complete control catalog.

Trace which tool was offered, selected, called, rejected, retried, and committed. A fluent final answer cannot prove that the action path was correct.

5. Safety, security, and policy

Content filters are one control, not a complete guardrail layer. Threats also include prompt injection, cross-tenant retrieval, secret disclosure, excessive agency, untrusted model or node code, and unsafe tool arguments.

Express deterministic business rules outside the model where possible. Define residual risks, test adversarial cases, and assign an owner for policy changes. NIST’s Generative AI Profile is a useful risk inventory, not a turnkey acceptance test.

Evaluation becomes a release gate

For release decisions on free-form output, do not rely on one aggregate accuracy number. Use several evidence types and define which one is authoritative for each failure mode.

Build an evaluation stack with several evidence types:

1. Deterministic checks: schema validity, citations present, allowed tools, argument constraints, policy rules, latency, and budget.
2. Component metrics: retrieval recall and ranking, tool-selection accuracy, tool-argument correctness, and route selection.
3. End-to-end tasks: representative inputs with explicit success criteria and slice labels.
4. Model-based scoring: rubric-driven judgments calibrated against expert labels and monitored for judge drift.
5. Human review: ambiguous, consequential, novel, or sampled cases where automation is not authoritative.
6. Adversarial tests: injection, data-boundary, abuse, refusal, and side-effect cases tied to the threat model.

Store per-example results, not only averages, so the release review can inspect regressions by language, tenant, document type, or action class.

For a deployment gate, compare a candidate manifest with the current release on the same versioned suite. Set thresholds for quality, safety, latency, and cost together; a cheaper route that fails the task is not an optimization.

Traces connect production back to evaluation

Infrastructure metrics can show a slow model call. They cannot show that retrieval returned an unauthorized document or that a tool was called with the wrong account. MLflow’s tracing documentation describes traces that capture intermediate steps and metadata needed to inspect this path.

Capture a trace across the decision path:

• release-manifest identity and request correlation
• model and provider calls, latency, usage, and finish state
• retrieval query, document identifiers, scores, and filtering decisions
• prompt/template revision without indiscriminately storing sensitive content
• tool offers, arguments, approvals, results, and side-effect identifiers
• policy decisions, retries, fallbacks, and final outcome

Tracing creates a new data-governance surface. Apply minimization, redaction, tenant isolation, encryption, sampling, retention, and access review before collecting full prompts or documents. OpenTelemetry’s GenAI semantic conventions can help with interoperability, but they remain in development. Pin the convention or schema version, instrumentation, and collector versions.

The improvement loop is:

production trace → triaged failure → labeled regression case
                 → candidate change → offline comparison
                 → staged release → monitored outcome

User feedback can prioritize investigation, but a thumbs-up is not ground truth. Preserve the surrounding trace and obtain expert labels for consequential cases.

The serving gateway is a policy boundary

A gateway can decouple application clients from providers or self-hosted engines. Responsibilities worth putting there include:

• authentication, tenant budgets, quotas, and rate limits
• stable request and response contracts
• route selection by capability, region, latency, or evaluated quality
• bounded retries, circuit breakers, and explicit fallback semantics
• cache partitioning and sensitive-data policy
• usage attribution and release-manifest propagation

Fallback is a behavior change as much as a reliability mechanism. If a smaller model, alternate provider, or reduced context changes task quality, evaluate and trace that branch as its own route.

Do not put every orchestration decision into the gateway. Keep domain rules close to the application and make ownership visible.

Fine-tuning is one intervention, not the maturity ladder

Choose the intervention from observed failure:

LoRA freezes pretrained weights and adds trainable low-rank matrices, reducing the number of trainable parameters for the downstream task. That optimization does not remove dataset governance, base-model licensing, evaluation, serving compatibility, or rollback requirements.

A practical adoption sequence

1. Define the user task, harm boundaries, service objectives, and cost envelope.
2. Create the release manifest before introducing a prompt registry, vector database, or gateway product.
3. Build a small, sliced evaluation set and deterministic component tests.
4. Instrument one end-to-end trace with privacy controls and stable release identities.
5. Promote through shadow, canary, or limited traffic with an explicit rollback trigger.
6. Turn reviewed production failures into regression cases and repeat.

Add infrastructure only when it owns a named control or removes a measured bottleneck. “LLMOps platform” is not an architecture requirement.

Conclusion

LLMOps is MLOps applied to a larger behavioral unit. The model remains important, but prompts, retrieved evidence, tool permissions, routing, and policy can change the outcome without changing the weights.

Version that whole unit, evaluate it before release, trace it with privacy boundaries, and roll it back as one system.

References

• MLflow: evaluating production traces - Reuse production traces for evaluation and score intermediate trace information.
• MLflow: LLM and agent tracing - Capture intermediate steps and trace metadata for investigation.
• OpenTelemetry GenAI semantic conventions - Current development-status conventions for GenAI spans, metrics, events, and provider-specific data.
• NIST AI 600-1: Generative AI Profile - Risk categories for prompt injection, privacy, security, and related generative-AI concerns.
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks - Original retrieve-then-generate architecture.
• LoRA: Low-Rank Adaptation of Large Language Models - Frozen pretrained weights plus trainable low-rank matrices for downstream adaptation.