Prompt Engineering Is Not Engineering: The Case for Structured Evals

Prompt engineering is the practice of crafting and refining the text inputs to an LLM to produce better outputs. At its best, it is systematic experimentation: form a hypothesis ("adding step-by-step instructions will improve reasoning accuracy"), make a targeted change, measure the impact against a held-out evaluation set, and keep or discard the change based on the evidence. At its worst — which is how most teams practice it — it is intuition-driven trial and error with no systematic measurement.

The term "engineering" in prompt engineering is aspirational for most practitioners. Real engineering involves measurement, reproducibility, and the ability to explain why a change works. Most prompt iteration does not meet this bar: developers try variants, pick the one that "seems better" on the examples they happen to test, and ship it without knowing whether it improved or degraded performance on the full distribution of inputs.

This matters because LLM behavior is non-linear. A prompt change that improves performance on the 10 examples you tested may degrade performance on the 100 examples you did not test. Without a structured evaluation set, you have no way to know. The history of LLM application development is full of "improvements" that were later discovered to have introduced regressions on edge cases.

As a system scales — more users, more query types, more edge cases — the fraction of the input space covered by the developer's intuitive test cases shrinks. A prompt optimized on 20 examples covers an increasingly small fraction of real production inputs. The mismatch between dev-time intuition and production reality grows, and the system degrades silently.

Intuition-based prompting also creates a ratchet effect: each developer who touches the prompt improves it for their mental model of what the system should do, which may conflict with what the system was originally designed to do. Without a canonical evaluation set that encodes the intended behavior, prompt changes accumulate contradictions and the prompt grows longer and more fragile.

The organizational failure mode is prompt ownership ambiguity. If every developer on a team is empowered to edit the system prompt "to fix a bug," and no one owns a regression test suite, the prompt becomes a palimpsest of conflicting instructions. The system behavior becomes unpredictable and no one can explain why. This is the production equivalent of a codebase with no tests and no code review.

A structured eval is a test suite: a dataset of inputs, expected outputs or evaluation criteria, and code that runs each input through the system and scores the result. For prompt evaluation specifically, the test suite should include: happy path examples (the inputs the system is designed to handle well), edge cases (boundary conditions and unusual inputs), adversarial examples (inputs designed to elicit incorrect behavior), and regression cases (inputs that broke in the past and must not break again).

Scoring can be exact match (for structured output tasks), rule-based (does the output contain keyword X? does the JSON have the required fields?), embedding similarity (is the output semantically similar to the expected output?), or LLM-as-judge (rate the output on a rubric). Each has different cost/reliability tradeoffs. For prompt evaluation, LLM-as-judge with a well-constructed rubric is often the most practical — it handles open-ended text tasks that are hard to score with rules.

The eval suite lives in version control alongside the prompts. When a prompt changes, the eval suite runs automatically in CI. The test results are reported as a diff: which cases improved, which degraded, and by how much. This is what makes prompt development engineering: you have a reproducible measurement of impact.

The eval-driven development loop for prompts: (1) identify a failure mode from user feedback, production logs, or adversarial testing; (2) write an eval case that reproduces the failure — an input and the expected correct output; (3) run the eval suite, confirm the new case fails; (4) make a targeted prompt change designed to address the failure; (5) run the eval suite again — the new case should pass, and previously passing cases should still pass; (6) merge the prompt change and the new eval case together.

This loop has three properties that make it engineering: it is reproducible (anyone can run the eval suite and get the same result), it is cumulative (the eval suite grows with every bug fix), and it is preemptive (regressions are caught in CI before merging). The contrast with intuition-based prompting is stark — you can demonstrate exactly what you changed and exactly what impact it had.

The loop also surfaces prompt conflicts: when a change that fixes case A breaks case B, you have discovered a tension in the prompt design. The resolution is either to specialize the prompt (different prompts for different input types), to add more explicit instructions, or to accept a tradeoff. Either way, you make the decision explicitly with evidence rather than discovering it later from user complaints.

A prompt regression test suite starts with cases from production failures. Every time a user reports a bad output, extract the input, document the expected output, and add it to the suite. This builds a test suite that is specifically calibrated to your system's actual failure modes — not theoretical ones.

Complementing production failure cases with systematic coverage: add at least 10 examples for each distinct capability the prompt is supposed to handle (answering from context, refusing out-of-scope questions, formatting output as JSON, handling multi-part questions). For each capability, include at least one adversarial case: an input designed to make the system fail.

Test suite maintenance is ongoing. Prompts change, models change (provider model updates are silent), and user behavior evolves. Review your test suite quarterly: prune cases that are redundant, add cases for new features, and re-run all cases against the current production system to establish a baseline. A test suite that has not been updated in six months is probably testing the wrong things.

Eval failure one: testing only the happy path. If your eval suite contains only inputs that the system is designed to handle well, it will always pass. Add adversarial examples, out-of-scope queries, malformed inputs, and inputs that exploit known model biases.

Eval failure two: using exact string match for non-deterministic output. LLMs produce different outputs on different runs at non-zero temperature. Exact match evals will fail intermittently, producing false positives in CI. Use property-based assertions (does the output contain the required claim? does it validate against the JSON schema?) or set temperature to 0 for eval runs.

Eval failure three: prompt-eval co-contamination. If the person writing the prompt is the same person writing the eval cases, the eval cases will reflect the prompt author's mental model — not the user's needs. Involve domain experts or real users in writing eval cases. At minimum, write eval cases before writing the prompt, so the cases reflect requirements rather than implementation.

Prompt engineering alone is sufficient when: the task is well-defined and the base model handles it with appropriate instructions, the input distribution is narrow and well-understood, and the quality bar is moderate. Customer-facing chatbots, structured data extraction tasks, and classification tasks with clear categories are often well-served by careful prompt engineering with a structured eval suite.

Prompt engineering is not sufficient when: the task requires knowledge the model was not trained on (use RAG), the task requires consistent specialized behavior the model resists (use fine-tuning), the latency budget is tight and a large context window is needed to handle all cases (use a smaller fine-tuned model), or the model's base behavior conflicts with what you need regardless of instruction (use fine-tuning on demonstration data).

The most common waste of engineering time in AI systems is attempting to prompt-engineer a capability that requires fine-tuning or RAG. If you have written a 2,000-token system prompt and the system still fails on 30% of inputs, you are likely hitting a capability ceiling that instruction following cannot overcome. Recognize the signal early and switch approaches.