Schema-Guided Reasoning: vLLM, XGrammar, and Pydantic

Viacheslav Dubrov

2025-12-28 · Updated 2026-06-07 · 10 min · AI Agents · Model Engineering

This article is for Python engineers who need model output that downstream code can validate. You will learn how to define a Pydantic schema, request structured output from vLLM, and add application checks for meaning and policy.

Retrying an LLM call does not guarantee valid JSON. The next sample may fail in the same way, and repeated calls add latency and cost.

Schema-Guided Reasoning (SGR) enforces a schema while the model generates each token. You define the required fields with Pydantic, and the inference engine blocks tokens that would violate that structure. The result is syntactically valid by construction rather than by retry.

TL;DR. SGR uses constrained decoding to keep an LLM’s output within a Pydantic schema. vLLM can use XGrammar to constrain the generated structure. Validate semantic rules in application code.

What is Schema-Guided Reasoning?

Schema-Guided Reasoning is a technique that Rinat Abdullin described in July 2025. Instead of letting the model freely complete text (which can be inconsistent or ambiguous), you give it a strict template that defines:

• what steps the response should represent
• the intended order of those steps, so a reviewer can inspect the path from data to decision
• where it should focus attention

Think of it as a cognitive checklist the model has to follow.

What the schema controls

Fields such as churn_analysis, margin_math, and max_discount_percent make the intended intermediate outputs explicit. A schema constrains the returned shape. It does not, by itself, make one field depend on another or prove that the decision is correct.

That gives you:

• reproducible reasoning across repeated runs
• auditable outputs where every step is inspectable
• intermediate fields you can grade against a test dataset
• smaller models that become workable, since the schema supplies the structure the model would otherwise have to learn
• Abdullin writes that a 5–10% accuracy boost is “not uncommon” in cases he has observed; this is a practitioner observation, not a benchmark result, so measure it on your own workload

SGR vs Chain of Thought vs prompt engineering

The three approaches differ mostly in how strongly they constrain the model.

Prompt engineering: semantic persuasion

Please analyze the customer data and output your response as valid JSON
with the following structure: {"discount": <number>, "reason": <string>}
Be careful with the formatting!

You are hoping the model’s understanding of “output JSON” outweighs its tendency to be conversational. A model update, a temperature change, or a different few-shot example can break your parser.

Chain of Thought: useful reasoning trace, same structure problem

Let's think step by step:
1. First, I'll analyze the customer's churn risk...
2. Then I'll calculate the margin...
3. Therefore, I recommend a 15% discount.

CoT can improve task accuracy when the prompt, model, task, and evaluation support it, but it leaves the result as prose that is hard to parse reliably. You may end up making a second LLM call just to extract structured data.

SGR: structured chain of thought

SGR can make intermediate fields available for inspection and evaluation. Whether that improves task accuracy depends on the model, prompt, task, and how those fields are used; the schema itself only formalizes their shape:

class PricingLogic(BaseModel):
    # 1. Data Analysis (must complete before decision)
    churn_analysis: str = Field(..., description="Analyze churn_probability")
    financial_analysis: str = Field(..., description="Analyze cart_value and margin")

    # 2. Math Enforcement (explicit calculation)
    margin_math: str = Field(..., description="Calculate: 'Cart $X * Y% = $Z'")

    # 3. Decision Constraint (bounded by prior analysis)
    max_discount_percent: float = Field(..., description="Max allowed discount")

    # 4. Final Output
    offer_code: str
    customer_message: str

The schema describes these fields in that order. A single object schema does not create a separate validation step between them. Use separate calls or application checks when later decisions must depend on earlier results.

SGR patterns

SGR has three core patterns that compose into bigger workflows.

1. Cascade: sequential reasoning steps

Cascade represents a reasoning order in one structured response. It does not enforce a state transition between fields.

from pydantic import BaseModel
from typing import Literal, Annotated
from annotated_types import Ge, Le

class CandidateEvaluation(BaseModel):
    """Evaluate a job candidate with enforced reasoning order."""

    # Step 1: Summarize (forces context awareness)
    brief_candidate_summary: str

    # Step 2: Rate (bounded integer)
    rate_skill_match: Annotated[int, Ge(1), Le(10)]

    # Step 3: Decide (constrained choices)
    final_recommendation: Literal["hire", "reject", "hold"]

Good fits: candidate evaluation, document classification, compliance analysis, medical diagnosis.

The model is asked to return brief_candidate_summary, rate_skill_match, and final_recommendation in that order. If the order is a policy requirement, enforce it with separate calls or deterministic application logic.

2. Routing: a semantic switch statement

Routing makes the model commit to one path from a set of options, implemented with Union types.

from pydantic import BaseModel
from typing import Literal, Union

class FeatureLookup(BaseModel):
    """Route to database lookup."""
    rationale: str
    tool_name: Literal["fetch_user_features"] = "fetch_user_features"
    user_id: str

class GeneralResponse(BaseModel):
    """Standard response for non-pricing queries."""
    tool_name: Literal["respond"] = "respond"
    content: str

class RouterSchema(BaseModel):
    """The model must pick exactly ONE branch."""
    action: Union[FeatureLookup, GeneralResponse]

Good fits: intent classification, tool selection, support triage, multi-agent dispatch.

The branch-specific Literal values help validation distinguish the union members. They do not make routing correct. Validate the result and dispatch it in application code. For an explicit Pydantic discriminator, configure and test a discriminated union.

3. Cycle: repeated reasoning with lists

Cycle forces the model to produce multiple items, with bounds on how many.

from pydantic import BaseModel
from typing import List, Literal, Annotated
from annotated_types import MinLen, MaxLen

class RiskFactor(BaseModel):
    explanation: str
    severity: Literal["low", "medium", "high"]

class RiskAssessment(BaseModel):
    """Generate 2-4 risk factors."""
    factors: Annotated[List[RiskFactor], MinLen(2), MaxLen(4)]

Good fits: risk assessment, issue extraction, parallel tool calls, multi-step planning.

The MinLen and MaxLen bounds force at least 2 and at most 4 items. Combined with Routing, this is how you dispatch a fixed-width batch of tool calls.

Making SGR work: constrained decoding

The patterns above are just Pydantic schemas. The thing that makes them binding is constrained decoding (also called Structured Output).

Constrained decoding modifies the token generation step. Instead of letting the model sample freely from its vocabulary, the engine applies a grammar mask that blocks tokens that would violate the schema. It happens at the inference engine, not in your application code.

[!TIP] SGR does not require “reasoning models” like o1 or DeepSeek-R1. It works fine with instruction-tuned models, and especially well with models distilled from reasoning ones.

Cloud providers that support it

The following provider documentation advertised structured-output support when this article was updated on 2026-06-07. Support, schema subsets, strictness, and failure behavior vary by model and endpoint:

Inference engines that support it

For self-hosted models, the major engines all have a constrained decoding backend:

Why this article focuses on vLLM and XGrammar

A few reasons:

• vLLM is one of the most widely deployed open-source LLM inference engines, so what you build here ports easily.
• XGrammar is implemented in C++ and the cited benchmarks measure its overhead under specific conditions. Measure it with your model, schema, hardware, and serving configuration.
• vLLM’s API is OpenAI-compatible, which keeps migration from cloud providers cheap.
• XGrammar handles complex nested schemas, unions, and recursive structures.

How XGrammar enforces schemas

Where the masking happens

XGrammar modifies the output logits after the model’s forward pass and before sampling. It does not change the model itself. It filters which tokens can be selected.

A standard inference loop looks like this:

1. Input tokens → GPU Forward Pass → Logits (probability scores for all ~128K tokens)
2. Logits → Sampling (temperature, top-p, etc.) → Next Token
3. Repeat until done

XGrammar slips between steps 1 and 2:

1. Input tokens → GPU Forward Pass → Raw Logits
2. Raw Logits → XGrammar Logits Processor → Masked Logits
3. Masked Logits → Sampling → Next Token (guaranteed valid)
4. Repeat until done

The model still computes its full probability distribution on the GPU. XGrammar’s GrammarMatcher generates the bitmask on the CPU, then the serving engine moves that bitmask to the logits’ device and applies it in place before sampling. For GPU logits, XGrammar uses a GPU kernel for that application. Invalid tokens get their logits set to -∞, which makes their probability exactly 0 after softmax.

Two phases

XGrammar splits the work into compile-time and runtime. This design reduces repeated grammar work.

Phase 1: grammar compilation, once per schema

# This happens once per schema
tokenizer_info = xgr.TokenizerInfo.from_huggingface(tokenizer)
grammar_compiler = xgr.GrammarCompiler(tokenizer_info)
compiled_grammar = grammar_compiler.compile_json_schema(schema_json)

During compilation, XGrammar:

1. Converts the JSON Schema to a Context-Free Grammar.
2. Builds a Pushdown Automaton (PDA), which is a state machine with a stack so it can handle nested structures like {"a": {"b": {"c": ...}}}.
3. Pre-computes which tokens are valid at each grammar position. The result is the “adaptive token mask cache.”
4. Categorizes tokens as “context-independent” (cacheable) or “context-dependent” (must be checked at runtime against the stack state).

[!NOTE] The XGrammar paper reports that about 99% of tokens were context-independent in its measurements (paper). Treat that number as benchmark-specific, not as a universal ratio.

Phase 2: runtime mask generation, every token

At each generation step:

1. The GrammarMatcher tracks the current position in the grammar.
2. It looks up the pre-computed mask for context-independent tokens.
3. It runs the PDA to check the remaining context-dependent tokens.
4. It combines them into a final bitmask, moves it to the logits’ device, and applies it there.

Why pushdown automata and not regex?

Because of nesting. A regular expression (a finite state machine) cannot reliably match structures like:

{ "user": { "profile": { "settings": { "theme": "dark" } } } }

The hard part is the closing braces }}}: you need to remember how many you opened. A Pushdown Automaton has a stack that tracks this, so it can handle arbitrary nesting depth. That is also why XGrammar can enforce Union types, nested objects, and recursive schemas, where regex-based approaches fall short.

A concrete example: generating a float field

When the model is generating "max_discount_percent":, XGrammar knows from the schema that a float comes next. The valid token set depends on the parser state and tokenizer. At the start of the number, the mask can admit a digit or a minus sign; after a digit, it can admit continuations such as another digit, a decimal point, or an exponent marker.

• A quote, {, [, true, false, or null cannot begin this number, so the mask blocks their tokens at that state.
• The forward pass might have assigned high probability to the token for "fifteen". Because that token cannot continue this numeric field, the mask removes it and the model must choose a valid numeric continuation.

What affects overhead

Three reasons:

1. Mask generation and transfer. XGrammar generates masks on the CPU, and the serving engine transfers them to the logits’ device for in-place application. The amount of overlap depends on the serving implementation and workload.
2. Caching. Most of the validity work is done at compile time. Runtime is mostly cache lookups.
3. C++ implementation. The hot path is C++, not Python, and the mask is applied to logits in place.

The cited benchmarks report low overhead for their tested grammars, tokenizers, hardware, and workloads. Those results do not establish a general latency or throughput guarantee.

Practical implementation with vLLM

The sgr-discount-manager project is an illustrative external demo. This article does not pin its commit or claim that the snippets below were run in this repository.

Project structure

sgr/
├── agent.py            # Main orchestration
├── models/
│   └── schemas.py      # Pydantic SGR schemas
├── prompts/
│   ├── routing.py      # Phase 1 prompts
│   └── pricing.py      # Phase 3 prompts
├── store/
│   └── hybrid_store.py # Hot/Cold data retrieval
└── utils/
    └── llm_client.py   # LLM client wrapper with xgrammar

Step 1: define the schemas

# sgr/models/schemas.py
from pydantic import BaseModel, Field
from typing import Literal, Union

# --- Phase 1: Routing (Union for branching) ---
class FeatureLookup(BaseModel):
    """Route to DB lookup if pricing context is needed."""
    rationale: str
    tool_name: Literal["fetch_user_features"] = "fetch_user_features"
    user_id: str

class GeneralResponse(BaseModel):
    """Standard response for non-pricing queries."""
    tool_name: Literal["respond"] = "respond"
    content: str

class RouterSchema(BaseModel):
    action: Union[FeatureLookup, GeneralResponse]

# --- Phase 2: Pricing Logic (Cascade for sequential reasoning) ---
class PricingLogic(BaseModel):
    """
    Structured response for dynamic pricing. The fields record an intended analysis→decision flow.
    """
    # 1. Data Analysis (Reflection)
    churn_analysis: str = Field(...,
        description="Analyze churn_probability (High > 0.7).")
    financial_analysis: str = Field(...,
        description="Analyze cart_value and profit_margin.")

    # 2. Hard Math Enforcement
    margin_math: str = Field(...,
        description="Calculate absolute profit: 'Cart $200 * 0.20 Margin = $40'.")

    # 3. Model-proposed decision; application code approves it.
    max_discount_percent: float = Field(...,
        description="Proposed discount percentage. Application code enforces policy.")

Step 2: an LLM client that turns on XGrammar

# sgr/utils/llm_client.py
import json
from typing import TypeVar

from openai import OpenAI
from pydantic import BaseModel

T = TypeVar("T", bound=BaseModel)

class LLMClient:
    """Wrapper for vLLM with XGrammar-enforced structured generation."""

    def __init__(self, base_url: str = "http://localhost:8000/v1"):
        # Local vLLM commonly has no authentication. EMPTY is not authentication.
        self.client = OpenAI(base_url=base_url, api_key="EMPTY")
        self.model = self._get_available_model()

    def _get_available_model(self) -> str:
        """Auto-detect the model running on vLLM server."""
        try:
            models = self.client.models.list()
            if models.data:
                return models.data[0].id
        except Exception:
            pass
        return "Qwen/Qwen2.5-7B-Instruct"

    def run_sgr(self, messages: list[dict], schema_class: type[T]) -> T:
        """Run inference with Schema-Guided Response constraints.

        Uses vLLM structured outputs to constrain the JSON shape at generation time.
        """
        schema_dict = schema_class.model_json_schema()

        # Enhance system message with schema for model guidance
        enhanced_messages = messages.copy()
        if enhanced_messages and enhanced_messages[0]["role"] == "system":
            schema_json = json.dumps(schema_dict, indent=2)
            enhanced_messages[0] = {
                "role": "system",
                "content": (
                    enhanced_messages[0]["content"]
                    + f"\n\nRespond with JSON matching this schema:\n{schema_json}"
                ),
            }

        # vLLM v0.12+ structured outputs. Configure the backend on the server.
        completion = self.client.chat.completions.create(
            model=self.model,
            messages=enhanced_messages,
            temperature=0.1,  # Reduces sampling variation; it is not deterministic.
            extra_body={"structured_outputs": {"json": schema_dict}},
        )

        raw_response = completion.choices[0].message.content
        return schema_class.model_validate_json(raw_response)

[!NOTE] In current vLLM, structured_outputs: {"json": schema_dict} requests JSON that matches the schema. Configure the structured-output backend with the server’s --structured-outputs-config.backend option when needed. This is software enforcement in the inference server, not hardware enforcement.

Step 3: orchestrate the agent

# sgr/agent.py
from decimal import Decimal

from .models.schemas import PricingLogic, RouterSchema
from .prompts.routing import build_routing_prompt
from .prompts.pricing import build_pricing_context_prompt, ASSISTANT_FETCH_MESSAGE
from .store.hybrid_store import HybridFeatureStore
from .utils.llm_client import LLMClient

def approve_discount(offer: PricingLogic, context: dict) -> Decimal:
    """Enforce the pricing policy independently of the model's explanation."""
    cart_value = Decimal(str(context["current_cart_value"]))
    margin = Decimal(str(context["cart_profit_margin"]))
    proposed = Decimal(str(offer.max_discount_percent))

    if cart_value <= 0 or not Decimal("0") <= margin <= Decimal("1"):
        raise ValueError("Invalid pricing context")

    gross_profit = cart_value * margin
    discount_cost = cart_value * proposed / Decimal("100")
    policy_cap = min(margin * Decimal("100"), Decimal("20"))
    if not Decimal("0") <= proposed <= policy_cap or discount_cost > gross_profit:
        raise ValueError("Proposed discount violates pricing policy")

    return proposed.quantize(Decimal("0.01"))

def pricing_agent(user_query: str, user_id: str) -> str:
    """Process a pricing query with three-phase SGR workflow."""

    llm = LLMClient()
    feature_store = HybridFeatureStore()

    # Build conversation history
    history = [
        {"role": "system", "content": build_routing_prompt(user_id)},
        {"role": "user", "content": user_query},
    ]

    # --- Phase 1: Routing (Uses RouterSchema) ---
    print(f"🤖 Processing: '{user_query}' for {user_id}")
    decision = llm.run_sgr(history, RouterSchema)
    print(f"📍 Routing decision: {decision.action.tool_name}")

    if decision.action.tool_name == "respond":
        return decision.action.content

    # --- Phase 2: Context Retrieval ---
    if decision.action.tool_name == "fetch_user_features":
        print(f"🔍 Fetching features for {user_id}...")
        context = feature_store.get_user_context(user_id)

        if not context:
            return "Error: User profile not found."

        print(f"   [Data] LTV: ${context.get('user_ltv')} | "
              f"Margin: {context.get('cart_profit_margin', 0) * 100}%")

        # Inject context into conversation
        history.append({"role": "assistant", "content": ASSISTANT_FETCH_MESSAGE})
        history.append({
            "role": "user",
            "content": build_pricing_context_prompt(
                churn_prob=context.get("churn_probability", 0.5),
                cart_val=context.get("current_cart_value", 100),
                margin=context.get("cart_profit_margin", 0.2),
                user_ltv=context.get("user_ltv", 0),
            ),
        })

        # --- Phase 3: model proposal, then deterministic policy enforcement ---
        print("🧠 Proposing Offer (Schema Enforced)...")
        offer = llm.run_sgr(history, PricingLogic)
        approved_discount = approve_discount(offer, context)

        # Audit log: reasoning is inspectable; pricing is application-enforced.
        print(f"   [Audit] Math: {offer.margin_math}")
        print(f"   [Audit] Approved Discount: {approved_discount}%")

        return (
            "We value your loyalty! Here's a special "
            f"{approved_discount}% discount with code SAVE{approved_discount:.0f}."
        )

    return "I'm sorry, I couldn't process your request."

if __name__ == "__main__":
    response = pricing_agent("I want a discount or I'm leaving!", "user_102")
    print(f"\n💬 Final Reply: {response}")

Step 4: run vLLM with XGrammar

# Start vLLM server with XGrammar backend
vllm serve \
    --model Qwen/Qwen2.5-7B-Instruct \
    --port 8000 \
    --structured-outputs-config.backend xgrammar

# Run the agent
uv run python -m sgr.agent

Illustrative output

🤖 Processing: 'I want a discount or I'm leaving!' for user_102
📍 Routing decision: fetch_user_features
🔍 Fetching features for user_102...
   [Data] LTV: $1,500 | Margin: 20%
🧠 Proposing Offer (Schema Enforced)...
   [Audit] Math: Cart $200 * 0.20 Margin = $40
   [Audit] Approved Discount: 15.00%

💬 Final Reply: We value your loyalty! Here's a special 15.00% discount
   with code SAVE15.

This output is illustrative. The schema constrains the shape, but the application must verify that the arithmetic and discount policy are correct before using the offer.

Schema, vLLM, and production checklist

Schema design

1. Order fields by intended reasoning flow. Treat that order as documentation unless separate calls or application checks enforce it.
2. Write descriptive Field descriptions. They guide the model’s attention as much as the field name does.
3. Constrain with Literal and Annotated. Use Literal["a", "b"] for enums and Annotated[int, Ge(1), Le(10)] for bounds.
4. Keep schemas focused. One schema per reasoning phase, then compose with multiple calls.

vLLM configuration

1. Use a low temperature (0.1-0.3) to reduce sampling variation. For repeatable tests, fix the model and serving configuration, use the server’s deterministic mode if it provides one, and verify the result.
2. Let XGrammar handle the structure. Do not fight it with formatting instructions in the prompt.
3. Measure token usage against the same prompt, model, schema, and task. SGR can emit more or fewer tokens than a free-form CoT response.

Production considerations

1. Version your schemas the same way you version APIs.
2. Even with SGR, network and server errors still need graceful handling.
3. Log raw SGR outputs for compliance and debugging, then log the deterministic policy decision separately.
4. Recompute prices, limits, permissions, and other semantic rules in application code before use; test boundary values that the schema alone cannot reject.

Conclusion

You describe a reasoning topology in Pydantic and let constrained decoding enforce the output shape. The result is:

• syntactically valid by construction, which removes the retry-and-reparse loop
• auditable at the field level
• usable with smaller models, because they no longer have to nail the format on their own
• potentially cheaper to run, if validation reduces retries or the chosen schema and model use fewer output tokens

The sgr-discount-manager demo is an external starting point. Check its pinned dependencies and current vLLM compatibility before treating it as a runnable reference.

Key Takeaways

1. Schema-Guided Reasoning makes an intended reasoning topology explicit instead of relying only on prose instructions.
2. Constrained decoding prevents invalid JSON at generation time, which is cleaner than validating and retrying afterward.
3. Put analysis fields before decision fields when the response should document that reasoning path. Use separate calls or application checks when order must be enforced.
4. Use SGR when downstream code depends on structure, not when free-form prose is the product.

References

SGR Framework

• Schema-Guided Reasoning (SGR) — Rinat Abdullin’s original framework
• SGR Patterns — Cascade, Routing, Cycle patterns

xgrammar

• XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models — Yixin Dong et al., arXiv:2411.15100 (technical paper with benchmarks)
• xgrammar GitHub — Fast, flexible structured generation library
• xgrammar Documentation — Official docs with quick start guide
• xgrammar Quick Start — Getting started with xgrammar
• Achieving Efficient Structured Generation with XGrammar — MLC blog post on xgrammar internals

vLLM

• vLLM Structured Outputs — Official documentation

Demo Project

• sgr-discount-manager — Older illustrative demo; it does not contain every code example in this post