LoRAX Serving Guide: LoRA Adapters on Kubernetes at Scale

Viacheslav Dubrov

2025-10-22 · Updated 2026-08-16 · 10 min · Model Engineering · AI Infrastructure

One base model and many LoRA adapters create an unusual serving problem. The base weights are shared, but every request may need a different set of adapter weights. A conventional one-deployment-per-variant design wastes GPU memory when most variants are idle.

LoRAX addresses that long tail. It loads adapters on demand, batches requests for different adapters, and moves adapter weights between GPU and CPU memory. The attractive headline is “thousands of fine-tuned models on one GPU.” The engineering question is narrower: given your active adapter set, arrival pattern, and latency target, does exchange scheduling help enough to justify another serving runtime?

This guide is for inference and platform engineers who need to serve many LoRA variants from one base model. It shows how to test the documented APIs, size the working set, and turn the starter Helm chart into an explicit production plan.

Summary. Choose LoRAX when many compatible LoRA adapters share one base model and traffic is sparse or long-tailed. Capacity depends on the active working set, not the catalog size. Pin the runtime, authenticate and allowlist adapter IDs, add durable artifact caching and probes, route for cache locality, and benchmark cold and warm paths separately.

The serving problem is the working set

LoRA freezes a base model and learns low-rank updates for selected weight matrices. The resulting adapter is usually much smaller than a full checkpoint, but its size still depends on rank, target modules, layer count, and dtype. Fixed claims such as “every adapter is 100 MB” are poor capacity inputs.

For serving, distinguish three quantities:

• Catalog size: every adapter a platform can resolve from storage
• Active working set: adapters receiving requests within the cache-retention window
• Concurrent set: adapters represented in batches at the same moment

A catalog can contain thousands of adapters without fitting thousands in VRAM. What matters is how frequently the active set changes, how large those adapters are, and whether requests for different adapters can share useful batches.

LoRAX combines four mechanisms:

1. The base model remains resident for all compatible adapters.
2. A request names an adapter, which can be resolved from Hugging Face, Predibase, or a filesystem.
3. Adapter exchange scheduling prefetches and offloads weights between GPU and CPU memory.
4. Heterogeneous continuous batching groups requests targeting different adapters.

The LoRAX project reports that heterogeneous batching keeps throughput and latency nearly constant as concurrent adapter count grows in its benchmarks. Treat that vendor result as a hypothesis for your workload. Prompt length, output length, rank, target modules, batch occupancy, cache churn, and GPU generation can change the result.

The restored chart below comes from Predibase’s November 2023 LoRAX launch report. Predibase benchmarked Llama 2 7B with queries spread across 1 to 128 adapters on one NVIDIA A10G. This chart shows the 1-to-32-adapter cost comparison for processing one million tokens, divided evenly among the adapters.

Archived project comparison, reproduced from Predibase’s 2023 LoRAX report. LoRAX and dedicated costs use GPU-hours from that experiment. At the source’s level of detail, the GPT-3.5 Turbo line uses the fine-tuned model’s per-token cost.

Read the flat orange bars as a result of that benchmark design, not as a current price promise. The test sent a fixed total of one million tokens and let different adapters share a batch. The dedicated baseline assumed separate hosted compute for each model, so its cost rose with model count.

Predibase did not disclose these benchmark inputs:

• prompt-to-output token mix
• request lengths
• batch size
• adapter rank
• GPU-hour price
• exact dedicated-deployment utilization model
• exact historical GPT-3.5 input and output prices

Therefore, the report does not contain enough information to reproduce the plotted dollar values independently.

The chart also does not cover sparse traffic, cold downloads, current cloud prices, newer GPUs, other base models, or adapters with different ranks. It supports the case for workload replay. It cannot replace one.

When LoRAX is a plausible fit

LoRAX deserves a benchmark when all of these are true:

• adapters were trained against the same supported base model and tokenizer contract
• traffic spans many adapters, with a meaningful long tail
• loading a cold adapter on demand is preferable to reserving a deployment for it
• tenant or task routing already exists at the application boundary
• the team can operate a specialized inference runtime and its cache behavior

Typical cases include tenant-specific assistants, many domain variants, and online experiments that share a base checkpoint.

LoRAX is a weaker fit when a few adapters take most of the traffic, or when the models do not share a base. It is also weaker when hard latency targets cannot tolerate cold loads, or when the platform cannot safely control which artifacts the server loads. A standard vLLM or TGI deployment with a fixed adapter set may be simpler in those cases.

Do not use Kubernetes merely because the catalog is large. First prove the runtime and adapter compatibility on one GPU.

Try one base and one adapter locally

The LoRAX README recommends its prebuilt container. Use an immutable image digest in a real environment. main is shown here only because it is the repository’s documented quick-start tag. These examples are illustrative, not locally verified in this checkout. No LoRAX container run, API response, or client execution is included here. Confirm them against the image and dependency versions that you deploy.

mkdir -p data

docker run --rm --gpus all --shm-size 1g \
    -p 8080:80 \
    -v "$PWD/data:/data" \
    ghcr.io/predibase/lorax:main \
    --model-id mistralai/Mistral-7B-Instruct-v0.1

The documented minimum is Linux, Docker, an Ampere-or-newer NVIDIA GPU, and CUDA 11.8-compatible drivers. Model licenses and gated repositories may also require a Hugging Face token.

Start with the base model:

curl http://127.0.0.1:8080/generate \
    -H 'Content-Type: application/json' \
    -d '{
      "inputs": "[INST] Give one reason to measure cold-adapter latency. [/INST]",
      "parameters": {"max_new_tokens": 64}
    }'

Then send a compatible adapter:

curl http://127.0.0.1:8080/generate \
    -H 'Content-Type: application/json' \
    -d '{
      "inputs": "[INST] Solve: Natalia sold 48 clips in April and half as many in May. What is the total? [/INST]",
      "parameters": {
        "max_new_tokens": 64,
        "adapter_id": "vineetsharma/qlora-adapter-Mistral-7B-Instruct-v0.1-gsm8k"
      }
    }'

The first request may download and load the adapter. Later requests can use cached artifacts and resident weights. Record both paths. A single warm request says little about long-tail behavior.

Use an OpenAI-compatible client

LoRAX exposes an OpenAI-compatible chat endpoint. The model field identifies the adapter:

from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://127.0.0.1:8080/v1",
)

response = client.chat.completions.create(
    model="alignment-handbook/zephyr-7b-dpo-lora",
    messages=[
        {"role": "user", "content": "Explain cache locality in two sentences."},
    ],
    max_tokens=100,
)

print(response.choices[0].message.content)

The server does not require an API key by default. That is convenient for localhost and unsafe as an Internet-facing configuration. Put authentication, tenant authorization, quotas, and adapter allowlisting in front of it.

Define a compatibility gate

Before an adapter enters the catalog, verify at least:

• declared base model and revision
• tokenizer and chat-template compatibility
• LoRA rank and target modules supported by the runtime
• artifact format and tensor shapes
• license, provenance, and integrity digest
• a small behavioral and regression test suite

Reject incompatible artifacts during registration rather than on a user’s first request.

Understand residency before deploying

The base model consumes the largest fixed share of GPU memory. Adapter weights, KV cache, batch workspace, and runtime kernels compete for the remainder. CPU RAM can hold offloaded adapters, while /data stores downloaded artifacts.

These are not interchangeable tiers. A disk or Hub artifact must be read and materialized before it becomes a CPU- or GPU-resident adapter. Measure the transitions separately:

Capacity planning should replay the real adapter popularity distribution. Uniform random adapter IDs create a different cache problem from a Zipf-like tenant workload.

Deploy the repository chart carefully

The repository contains charts/lorax, so the repeatable starting point is a pinned repository revision and a local chart:

git clone https://github.com/predibase/lorax.git
cd lorax
git checkout <reviewed-commit-or-release>

helm dependency update charts/lorax
helm template lorax charts/lorax -f values.production.yaml
helm upgrade --install lorax charts/lorax \
    --namespace inference \
    --create-namespace \
    -f values.production.yaml

As reviewed on July 15, 2026, the chart values and Deployment template show defaults that deserve attention:

• the image tag is latest
• /data is an emptyDir, so pod replacement discards downloaded artifacts
• liveness and readiness probes are empty
• the Hugging Face token is modeled as a literal environment value
• one GPU is requested by default

The chart is useful scaffolding. It is not a production policy.

Start from the chart’s actual value structure

The chart nests runtime configuration under deployment, and launcher arguments are a list of name/value pairs. A minimal overlay looks like this:

deployment:
    replicas: 1

    image:
        repository: ghcr.io/predibase/lorax
        tag: "<tested-release-tag>"

    args:
        - name: "--model-id"
          value: "mistralai/Mistral-7B-Instruct-v0.1"
        - name: "--max-input-length"
          value: "2048"
        - name: "--max-total-tokens"
          value: "3072"
        - name: "--max-batch-total-tokens"
          value: "8192"
        - name: "--max-batch-prefill-tokens"
          value: "4096"

    resources:
        requests:
            nvidia.com/gpu: "1"
        limits:
            nvidia.com/gpu: "1"

    env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
              secretKeyRef:
                  name: lorax-hub
                  key: token

    readinessProbe:
        httpGet:
            path: /health
            port: http
        periodSeconds: 5
        failureThreshold: 600

service:
    serviceType: ClusterIP
    port: 80

The token limits above are example starting values, not sizing recommendations. Derive them from representative prompts, expected concurrency, GPU memory, and load tests.

Patch the gaps explicitly

The current chart template passes deployment.env through toYaml, so a normal values overlay can use valueFrom.secretKeyRef for Hub credentials, as above. It hard-codes the emptyDir volumes, has no startupProbe hook, and always formats the image as repository:tag. A values file alone therefore cannot provide durable /data, a startup probe, or image digest pinning; use a reviewed chart fork, post-render patch, or higher-level manifest layer for those changes. Put in that production layer:

• a PersistentVolumeClaim or node-local artifact cache mounted at /data
• a startup probe before an aggressive liveness probe
• pod disruption policy and topology spread for multiple replicas
• NetworkPolicy, service account restrictions, and an authenticated gateway
• image digest pinning and artifact integrity checks

Do not place a Hub token directly in a committed values file. Two replicas provide useful high availability only if both can load the base model. The router must also avoid sending every cold adapter to both pods.

Route for cache locality

Round-robin balancing can turn every replica into a cold cache. A useful router maps an allowed adapter ID to one replica by hash or another stable rule. It keeps a failover path for times when that replica is unavailable.

The routing key must come from authenticated application state, not an arbitrary public URL parameter. Otherwise a caller can force remote downloads, churn caches, or probe private adapter names.

LoRAX or vLLM?

Current vLLM documentation describes LoRA modules declared at startup and dynamic loading through endpoints or resolver plugins. It warns that runtime adapter updating has security risks and should not be used in production outside an isolated, trusted environment.

The useful comparison is operational rather than numerical:

Avoid rules such as “LoRAX for 1,000 adapters, vLLM for ten.” Catalog size alone does not determine performance. Benchmark both with the same base, adapters, prompts, ranks, arrival trace, and hardware.

A production acceptance test

Before expanding the catalog, run a replay that includes:

1. A fixed hot set to establish warm throughput and latency.
2. A long-tail distribution to measure CPU and artifact-cache hits.
3. A burst of previously unseen but allowlisted adapters.
4. Pod replacement to measure base and adapter recovery.
5. One unavailable or corrupt adapter to verify isolation and fallback.
6. Concurrent tenants to verify authentication, quotas, and metric labels.

Track these metrics:

• request rate
• queue time
• time to first token
• inter-token latency
• total latency
• adapter load time
• cache-hit class
• GPU memory
• CPU memory
• download bytes
• failures by reason

Keep adapter IDs out of unbounded metric labels. Map them to controlled dimensions or sampled traces.

Define acceptance thresholds before the test. Set a maximum cold-path error rate and a warm P99 target. Also set a cache-hit target for the observed popularity distribution and a recovery-time objective after pod loss.

Conclusion

LoRAX turns many compatible fine-tunes from a fleet of base-model copies into an adapter placement problem. That can be a strong design for long-tail workloads, but it does not make cost or latency flat by definition. The active working set, exchange path, batch mix, and storage layer still set the result.

Prove those mechanics on one GPU first. Then take Kubernetes seriously: pin artifacts, preserve the cache, protect credentials, authorize adapters, route for locality, and measure every residency path. If LoRAX beats a current vLLM configuration under the same replay, the deployment decision has evidence behind it.

References

• LoRAX repository and README — supported features, requirements, APIs, and Helm chart
• LoRAX launch report and 2023 benchmark — source and conditions for the restored cost chart
• LoRAX chart values and Deployment template — current defaults and volume behavior
• vLLM LoRA adapters — static and dynamic adapter serving
• LoRA paper — low-rank adaptation method
• Hugging Face PEFT documentation — adapter formats and training integration