Production Prompt Engineering: How Do I Test Changes?
Define successful outputs, collect representative cases, and compare one prompt change against a fixed baseline. Improve the instruction, examples, or supplied context according to the observed failure. Keep a held-out set to check whether the change works beyond the examples used to design it.
A production prompt needs to handle missing, ambiguous, and invalid inputs as well as ordinary requests.
Turn failures into test cases
Suppose a model extracts invoice fields. Include ordinary invoices, missing values, unfamiliar layouts, conflicting dates, and documents that are not invoices. Define how the system should handle each case before editing the prompt.
| Failure | Change to compare | Check |
|---|---|---|
| Required field missing | Clarify the field and add a relevant example | Field is present when supported |
| Value invented | Specify absent-value behavior and provide evidence | Unsupported field stays empty |
| Response cannot be parsed | Use a supported output schema | Parser accepts the complete response |
| Correct value from the wrong source | Preserve source identity in context | Value belongs to the requested document |
Schema-constrained output can address syntax failures. vLLM’s structured-output documentation describes supported output constraints. A valid JSON object can still contain the wrong date or an unsupported claim, so validate values separately.
Keep the comparison interpretable
Hold the model version, decoding settings, source data, tool definitions, and output limits constant while testing the prompt change. Record quality, failure categories, tokens, and latency. Repeat nondeterministic cases enough to see whether the improvement persists.
Few-shot examples should cover the real variation, including missing or ambiguous inputs. Add them for a measured problem rather than assuming more examples always help. Context position can also matter: Lost in the Middle found uneven evidence use across positions in its tested models and tasks.
Use explicit stages when they make a failure easier to isolate, but include the extra calls and interfaces in the comparison. Enforce permissions and other hard application rules in code; a system message alone cannot guarantee them.
The production-prompting section of the LLM Engineering Guide explains the main techniques and their limitations.