BLUF: Start with prompt engineering, add retrieval when the answer depends on changing or traceable knowledge, and fine-tune only when repeated evaluation shows a stable behavioural defect that examples can teach. These are not three competing products. They modify different parts of the system: prompts change the request context, RAG changes the evidence available at inference time, and fine-tuning changes model behaviour encoded in weights or adapters. Most enterprise systems need a sequence rather than a winner: prompt baseline → eval set → RAG for knowledge → PEFT for stable behaviour. Choosing by fashion usually bakes volatile facts into weights, adds retrieval latency to a formatting task, or funds a training pipeline before the team can measure improvement.
The three levers act at different layers
Prompt engineering changes instructions and demonstrations supplied with each request. It is the fastest reversible lever: version a system prompt, add a few representative examples, and evaluate the candidate against the same dataset. It is appropriate when the task is still moving, query volume is modest, and the required behaviour fits within the context budget. Its cost is paid repeatedly in input tokens and prompt maintenance.
Retrieval-augmented generation changes the evidence path. Documents are parsed, chunked, embedded and indexed; a query is authenticated, filtered, retrieved through dense and often sparse search, re-ranked, and only then supplied to the generator with source identifiers. RAG is the default for facts that change independently of the model—policies, product catalogues, contracts, tickets and operating procedures—because freshness can be updated at the index boundary without retraining weights. OpenAI’s retrieval documentation describes vector stores, semantic search, query rewriting and attribute filtering; production systems commonly add keyword retrieval and a re-ranker because semantic similarity alone can miss identifiers and exact clauses.
Fine-tuning changes model behaviour from a training dataset. Full fine-tuning updates many or all parameters; parameter-efficient fine-tuning (PEFT) updates a small adapter parameter set. Hugging Face documents PEFT as adapting large pretrained models without tuning every parameter, reducing training and checkpoint storage requirements while retaining task performance in many settings. LoRA and QLoRA are implementation families, not guarantees of quality. They still require representative data, held-out evaluation, model-version management and a rollback path.
Knowledge distillation is adjacent but distinct. Hugging Face TRL describes Generalized Knowledge Distillation as training a smaller student with teacher feedback, including feedback on student-generated sequences, to reduce inference cost and memory footprint. Distillation answers “can a smaller model reproduce this capability?”; it does not answer whether volatile enterprise knowledge belongs in weights.
Architecture and data flows
Prompt-only path — Client → policy/authentication → prompt template + few-shot examples → model → schema/guardrail validation → response. Store the prompt version, model version, request class, token count, latency and evaluator outcome. The principal operational asset is not the prompt text; it is the regression dataset that tells you whether a shorter or different prompt is better.
RAG path — Client → identity and tenant context → query normalisation → sparse + dense retrieval under mandatory metadata filters → re-ranker → context budgeter → prompt + cited evidence → model → citation and groundedness checks → response. The index build is a separate flow: source connector → parsing/OCR → classification and ACL propagation → chunking → embedding → index version. The retrieval log must bind query, identity-policy decision, index version, retrieved source IDs, scores and final citations. For the evidence model behind that requirement, see the RAG security operating model.
PEFT path — Production failures and approved examples → de-identification/licence review → train/validation/test split → adapter training → offline eval by request segment → load and latency test → controlled rollout → drift monitoring. Keep the base-model digest, adapter digest, tokenizer, training-data snapshot, hyperparameters and evaluation report together. An adapter that cannot be reconstructed and rolled back is not a production artefact.
Combined path — Retrieve current facts, then apply a tuned behavioural layer. RAG+PEFT is useful when answers must use fresh evidence and follow a stable domain-specific action: for example, retrieve the latest service policy while a PEFT adapter enforces a reliably structured disposition code. Keep the responsibilities separate. The adapter should learn how to decide or format; the index should supply what is currently true.
Decision tree: route the use case before selecting tooling
1 — Does correctness depend on facts that change weekly, daily or per transaction? If yes, use retrieval or a transactional tool call. Do not encode those facts in a fine-tune. If no, continue.
2 — Must the answer cite a governed source, enforce document-level permissions, or support deletion without retraining? If yes, use RAG even when the corpus changes slowly. Keep evidence outside weights and make retrieval reconstructible. If no, continue.
3 — Is the remaining defect behavioural and stable—format, classification boundary, tone, tool selection, domain shorthand—and represented by enough approved examples? If yes, benchmark PEFT or supervised fine-tuning against the prompt baseline. If no, continue.
4 — Is traffic low or the task still changing? If yes, keep prompt engineering and few-shot examples. Training and adapter operations will cost more than the repeated tokens. If no, calculate whether shorter prompts or a smaller tuned model repay training and serving complexity at the measured volume.
5 — Is the latency SLA tight enough that retrieval and re-ranking consume the budget? First test prompt-only or PEFT for stable knowledge; otherwise precompute retrieval, cache by permission-safe key, reduce top-k, or route only evidence-dependent requests through RAG. Never remove access filters or citation checks to win latency.
6 — Do you need both fresh knowledge and repeatable behaviour? Combine RAG+PEFT, but evaluate four candidates—prompt baseline, RAG, PEFT, RAG+PEFT—so that the combination has to earn its added failure surface.
Decision matrix: what each method optimises
Criterion: knowledge volatility. — Prompting: acceptable for a few manually maintained facts, poor for broad changing corpora. — RAG: strongest; update the source and index without retraining. — PEFT: weak for changing facts; retraining cadence becomes a data-freshness mechanism, which is expensive and hard to audit.
Criterion: behavioural consistency. — Prompting: good until instructions become long or edge cases conflict. — RAG: does not reliably change behaviour; better evidence cannot repair a systematic formatting or routing defect. — PEFT: strongest when the behaviour is stable and examples cover the production distribution.
Criterion: latency. — Prompting: one model call, but long instructions and examples increase prefill. — RAG: retrieval, filtering and re-ranking add network and compute stages before generation. — PEFT: can shorten prompts and may permit a smaller model, but adapter loading, model variants and cold starts can add operational latency. Measure p50 and p95 end to end.
Criterion: inference cost. — Prompting: repeated examples consume tokens on every call. — RAG: retrieval infrastructure plus retrieved tokens; caching is constrained by identity and freshness. — PEFT: upfront training and evaluation; savings appear only if prompt reduction or a smaller model outweighs training, hosting and model-variant costs at real volume.
Criterion: governance exposure. — Prompting: sensitive examples leave the application boundary on every request unless hosted locally or contractually controlled. — RAG: sensitive data remains in a governed retrieval boundary but retrieved snippets still reach the model; enforce tenant filters before search and minimise context. — PEFT: training data influences weights/adapters and deletion is not equivalent to deleting a document from an index. Legal, privacy and IP questions about training rights, processor terms and data-subject obligations require counsel; engineering can provide lineage, access controls and deletion mechanics, not the legal conclusion.
Criterion: reversibility. — Prompting: immediate version rollback. — RAG: roll back index, chunker, embedder or re-ranker independently if versions are retained. — PEFT: adapter rollback is straightforward only when the base model and serving runtime remain compatible; merged weights and provider-managed fine-tunes create a broader dependency.
A measurement model instead of generic cost claims
Do not use a universal “RAG costs more” or “fine-tuning is cheaper” claim. Calculate each candidate on your own traffic. Monthly prompt-only cost = requests × (input tokens × input price + output tokens × output price). Add prompt-maintenance and regression-test labour.
Monthly RAG cost = prompt-only generation cost after context insertion + embedding for changed documents + vector/search service + re-ranking + parsing/OCR + index operations. Measure retrieval p50/p95, context tokens, cache hit rate under permission-safe keys and the proportion of queries that actually require retrieval.
Monthly PEFT cost = amortised training and evaluation + hosted endpoint or GPU allocation + adapter storage/loading + serving tokens + retraining operations. The break-even question is: how many calls are required before reduced prompt tokens or a smaller model repay the training and model-variant burden? Use invoices and load tests, not vendor list prices alone.
Quality needs the same discipline. Track task success, schema validity, grounded citation precision for RAG, hallucination rate under knowledge-domain shift, refusal correctness, and results by request segment. The release should compare candidate with baseline and block harmful segment regressions—the operating pattern in enterprise LLM release gates.
Failure modes that decide the architecture
1. Fine-tuning used as a database. A training set contains current products, prices or policies; the model answers them confidently after the source has changed. Detection: time-sliced evaluation using facts introduced after the training snapshot. Mitigation: retrieve facts, tune behaviour.
2. RAG used to fix instructions. The retriever returns excellent evidence, but the model still violates a JSON contract or chooses the wrong tool. Better chunking will not repair a behavioural defect. Mitigation: strengthen schema validation and prompts, then evaluate PEFT if the failure repeats.
3. Prompt accretion. Every incident adds another paragraph until instructions conflict, token cost grows and no one knows which clause changed behaviour. Mitigation: prompts as versioned code, a removal test for every clause, and a regression gate.
4. Evaluation leakage. Fine-tune examples, prompt few-shots and evaluation cases overlap or are paraphrases, producing a convincing offline result that disappears under new inputs. Mitigation: split by customer, document, incident or time—not random rows—and keep a final untouched test set.
5. RAG permission leakage. Application-layer filtering is applied after vector search or omitted on a fallback path, so an unauthorised document influences ranking or context. Mitigation: mandatory pre-retrieval tenant/classification filters, negative tests and retrieval audit logs.
6. Adapter and quantisation interaction. A LoRA adapter passes evaluation on the training runtime and regresses after merging or quantising the base. Hugging Face documents QLoRA as adding trainable LoRA parameters over a 4-bit quantised model, but runtime combinations still require direct validation. Use the promotion checks in quantisation and adapters in production.
7. Domain shift hidden by one average. The tuned model improves familiar request forms and fails on German, long-context, rare classes or new policy language. Mitigation: segment-specific gates and a challenge set collected after the training window.
8. Combination without ablation. RAG+PEFT looks best, but no one knows whether retrieval, tuning or simply the new prompt caused the gain. Mitigation: evaluate all four variants under identical data, model and grader versions.
Practical evaluation and rollout checklist
— Freeze 100–300 representative requests, stratified by request class, language, sensitivity and failure severity; add a later time slice for domain shift.
— Establish the shortest prompt-only baseline that meets the output contract. Record token use, task success, p50/p95 latency and failure categories.
— Add RAG only for cases requiring external evidence. Measure retrieval recall on labelled sources, citation precision, grounded answer success, index freshness lag and permission-negative tests.
— Train PEFT only against a named recurring behavioural defect. Keep train/validation/test groups disjoint by real business entity or time.
— Run prompt, RAG, PEFT and combined candidates through the same evaluator version. OpenAI’s eval documentation supports datasets, testing criteria and run-level pass/fail results; whichever framework you use, preserve item-level outcomes and grader versions.
— Load-test the complete path, not the model in isolation. Include retrieval, re-ranking, adapter selection, schema retries and cold starts.
— Set promotion thresholds before reading results: quality floor, maximum harmful-regression count, p95 latency ceiling, cost per successful task and rollback conditions.
— Canary by request class, retain the baseline route, and reconcile observed token/search/GPU use with invoices after rollout.
— Define triggers: corpus freshness breach → index rebuild; behavioural drift → new evaluation and possibly adapter retraining; base-model change → full regression, never automatic adapter carry-over.
Limitations and trade-offs
Prompt engineering remains the correct production choice more often than architecture diagrams imply. It is transparent and reversible, but recurring examples tax every request and long instruction sets become fragile. RAG improves freshness and provenance, but introduces an index lifecycle, permission enforcement, retrieval misses and extra latency. PEFT can compress instructions and stabilise behaviour, but creates training-data, serving-compatibility and retraining obligations.
No method guarantees factuality. RAG can retrieve the wrong evidence; a fine-tuned model can become more confidently wrong; a stronger prompt can still be ignored under adversarial or unusual input. The control is evaluation plus constrained system design, not the adaptation label.
Provider capabilities and economics change. OpenAI’s model-optimisation guide explicitly frames evals, prompting and fine-tuning as an iterative optimisation cycle, while Hugging Face exposes many PEFT methods and quantised-training combinations. That abundance is a reason to keep the decision criteria stable—volatility, behaviour, latency, volume and governance—even when the specific model changes.
What to do next
Take one production use case and classify every failing example into three buckets: missing instruction, missing evidence, or stable behavioural defect. Build the prompt baseline first. Add retrieval only to the missing-evidence bucket. Consider PEFT only after the stable-defect bucket is large enough to split into training and untouched tests. That sequence usually gives you the cheapest useful answer and, more importantly, makes each added component accountable for a measured improvement.
If you need an independent architecture and evaluation design before committing to a retrieval platform or training pipeline, I work with engineering teams on exactly this boundary: representative datasets, ablation matrix, security controls, load tests and promotion gates. Bring real failure cases and traffic/latency constraints; those artefacts are more useful than a preferred framework.
Primary and authoritative sources
Hugging Face PEFT documentation — PEFT purpose, supported adaptation methods and integration model: https://huggingface.co/docs/peft/en/index
Hugging Face PEFT quantisation guide — QLoRA, 4-bit quantised bases, LoRA adapters and implementation caveats: https://huggingface.co/docs/peft/en/developer_guides/quantization
Hugging Face TRL Generalized Knowledge Distillation Trainer — teacher/student distillation, on-policy student outputs and trainer configuration: https://huggingface.co/docs/trl/en/gkd_trainer
OpenAI Developer Docs, Model optimisation — iterative eval, prompt and fine-tuning workflow and supported fine-tuning purposes: https://developers.openai.com/api/docs/guides/model-optimization
OpenAI Developer Docs, Working with evals — datasets, testing criteria, run results and regression workflows: https://developers.openai.com/api/docs/guides/evals
OpenAI Developer Docs, Retrieval — vector stores, semantic search, query rewriting and attribute filtering: https://developers.openai.com/api/docs/guides/retrieval
