Fast Answers for AI Engineering Decisions

Choose the question you need to answer. Each guide gives a starting recommendation, explains when it changes, and links to a detailed article. These are choices to test on your own task, not a ranking of universal winners.

AI agents

• Do I need an agent or a fixed workflow?

  Use a fixed workflow when you can define the steps and branches. Consider model-chosen steps when fixed rules cannot handle the task reliably.

• Which agent framework should I start with?

  Choose by the control you need over steps, saved progress, and recovery. Compare a small SDK with a graph-based workflow before adding more components.

• Which agent evaluation tool should I use?

  Start with the job: inspecting runs, testing changes, or checking tool actions. A tool can help run your tests; you still need to define a successful task.

• How should I secure an AI agent?

  Limit what its tools can do. Check permissions before actions, isolate code execution, and ask for approval when the consequences require it.

• Should I use function calling, MCP, a CLI, or skills?

  These serve different purposes and can work together. Choose how to expose a tool separately from the instructions that explain when to use it.

• When does schema-guided reasoning help?

  Use it when you need to check intermediate fields before taking an action. A valid structure still needs evidence checks and application rules.

Retrieval and documents

• Which NER model fits my entities?

  Start with your labels and examples. Stable labels, changing labels, overlapping spans, and facts inferred across sentences need different tests.

• Which OCR tool fits my documents?

  Test the output you need: text, reading order, tables, or specific fields. Clean scans and complex page layouts may need different approaches.

• Do I need basic RAG, agentic RAG, or GraphRAG?

  Start with one retrieval pass. Consider repeated search when later questions depend on earlier results, or GraphRAG when you need summaries across a collection.

• Which RAG evaluation tools and metrics should I use?

  Check retrieved evidence separately from the generated answer. Choose metrics by the failure you need to find and the reference data you have.

• When should I add embeddings or a reranker?

  Measure a keyword-search baseline first. Add another stage only when it fixes a measured retrieval or ranking problem within your time budget.

Models and local tools

• Which local LLM tool should I use on macOS?

  Choose by how you want to work: a desktop interface, a local API, or Python code. Check that the selected model actually runs on your Mac.

• Which LLM quantization format should I use?

  Check what your runtime and hardware support first. Then compare memory use, speed, and quality on your task.

• Should I prompt, use RAG, or fine-tune?

  Try prompting for instructions, retrieval for missing knowledge, and fine-tuning for behavior you can demonstrate with examples. You may need more than one.

LLM engineering questions

These answers cover the 45 topics in the LLM Engineering Guide. Start with one question, then follow its guide link for the equations, diagrams, and wider context. The existing quantization answer appears under Models and local tools above.

GPU hardware

• Why is LLM decode memory-bound and prefill compute-bound?
• How GPU memory hierarchy affects LLM inference speed
• Which GPU hardware specifications matter for LLM serving?

Inference latency and memory

• How to measure LLM latency using TTFT, TPOT, and P99
• LLM throughput vs goodput: which metric should you use?
• How much KV cache memory does an LLM request require?
• Prefill vs decode: why LLM inference has two phases
• MHA vs MQA vs GQA: how attention changes KV memory use

Inference optimizations

• How CUDA kernel fusion improves LLM inference speed
• What FlashAttention changes in transformer inference
• What is Flash-Decoding, and when does it speed up LLMs?
• Continuous vs static batching: how LLM scheduling works
• What does PagedAttention change in LLM KV allocation?
• When does speculative decoding make LLM serving faster?
• How prefix caching reuses LLM prompts and cached tokens
• How to stream LLM responses using SSE and usage chunks

Model architecture and tokenization

• What operations run inside a transformer decoder layer?
• Encoder vs decoder models: which architecture fits?
• MoE total vs active parameters: what memory do you need?
• BPE vs SentencePiece vs tiktoken: how tokenizers differ
• What LLM context windows limit and how RoPE extends them

Training and alignment

• Pretraining vs Fine-Tuning: How Do LLM Stages Differ?
• LoRA vs QLoRA: Which Fine-Tuning Method Should I Use?
• BF16 vs FP16 vs FP8: Which Format Should I Train In?
• Gradient Checkpointing: How Does It Save GPU Memory?
• DeepSpeed ZeRO Stages: Which Training State Is Sharded?
• FSDP vs DeepSpeed ZeRO: Which Fits My Training Setup?
• Chinchilla Scaling: Does More Training Reduce LLM Cost?
• RLHF vs DPO vs GRPO: How Do Alignment Methods Differ?
• LLM Distillation: How Do Small Models Learn From Teachers?
• Synthetic Training Data: How Do I Generate Useful Examples?

Deployment and model selection

• LLM Parallelism: How Do TP, PP, DP, and EP Differ?
• LLM Serving Frameworks: How Should I Choose a Runtime?
• GPU Selection for LLMs: How Much GPU Memory Do I Need?
• LLM Routing vs Cascading: Which Model Handles a Query?

Retrieval, agents, and prompts

• Embedding vs Generative Models: What Does Each Output?
• Production RAG Pipeline: Which Stages Should I Evaluate?
• ReAct vs ReWOO: Which Agent Execution Pattern Fits?
• Production Prompt Engineering: How Do I Test Changes?

Production operations

• How should LLM APIs limit requests, tokens, and concurrency?
• What causes LLM serving failures, and how do you fix them?
• Which metrics should you monitor in an LLM serving system?
• How do you reduce LLM costs without losing answer quality?
• How do you plan GPU capacity and autoscaling for LLMs?

For the full explanations, explore the long-form articles.