Raw view: Prompt Iteration With a Small Eval Set

No invisible, zero-width, or bidi-control characters detected.

# Prompt Iteration With a Small Eval Set

Improve the prompt at `{{PROMPT_PATH}}` for the task: {{TASK_DESCRIPTION}}. Success criteria as stated by the owner: {{SUCCESS_CRITERIA}}. Target model: {{MODEL}}. Iteration budget: {{ITERATION_BUDGET}} rounds (default 6).

The rule of the loop: no prompt change ships without a measured improvement on the eval set and no regression on the held-out cases. Intuition picks what to try; the score decides what stays.

## Phase 1: Define success

1. Convert {{SUCCESS_CRITERIA}} into a rubric of 3-6 criteria, each scorable per case as pass/fail or 0-2. Each criterion must be checkable by reading one output ("includes a citation for every factual sentence", "returns valid JSON matching the schema", "under 150 words"). Vague criteria ("helpful") are split or dropped.
2. Decide the scoring method per criterion: `exact` (string or schema match, scripted), `rubric-llm` (a grader model applies a written rule), or `human`. Prefer `exact` wherever possible; use `rubric-llm` with a precise rule and a different model than {{MODEL}} where judgment is needed.
3. Write the grader prompt for any `rubric-llm` criterion: the rule, two short examples of pass and fail, and an instruction to output only `PASS` or `FAIL` plus one sentence. Test the grader on three hand-labeled outputs; if it disagrees with you, fix the rule before trusting it.

**Artifact:** `eval/RUBRIC.md` with criteria, scoring method, and grader prompts.

## Phase 2: Build the eval set

4. Collect 15-30 input cases. Sources, in order of value: real inputs that failed in production or review; real inputs that worked; synthetic cases that target each rubric criterion; adversarial cases (empty input, very long input, input containing instructions to the model, off-topic input). Cover each criterion with at least three cases.
5. For `exact` criteria, write the expected output or the check per case. For others, write a one-line note on what a passing answer must contain.
6. Split: roughly 70% development set, 30% held-out. Never read the held-out outputs while iterating; they are scored only at the end of each round and used solely to detect overfitting.
7. **STOP POINT.** Present the rubric and the case list (inputs and expected properties). Wait for the owner to confirm the cases represent the real distribution and the rubric matches what they care about.

**Artifact:** `eval/cases.jsonl` (id, input, expected or notes, split) and a runner script that calls {{MODEL}} with the prompt, saves outputs, and applies the rubric.

## Phase 3: Baseline

8. Run the current prompt on the development set at a fixed temperature (0 where available) and the production settings. Score every case on every criterion. Record the per-criterion pass rate and the total.
9. Run it twice more and record variance. If the same case flips between runs, note it as unstable; stability is itself a target.
10. Read every failing output. Group failures by cause (missing instruction, ambiguous instruction, contradictory instructions, missing example, format drift, model limitation). The groups are your change candidates.

**Artifact:** `eval/results/round-0.md` with scores, variance, and the failure groups.

## Phase 4: Iterate

11. For each round up to {{ITERATION_BUDGET}}:
    a. Pick the single largest failure group. Form a hypothesis about the prompt change that fixes it. Prefer, in order: removing a contradiction or ambiguity; adding a specific instruction with the reason behind it; adding or fixing an example; restructuring with tags so inputs are distinguishable from instructions; adding a self-check step. Avoid adding emphasis words or repeating instructions; those rarely move scores and often cause over-compliance.
    b. Make exactly that one change. Save the prompt as a new version (`prompt.v<N>.md`). Record the hypothesis.
    c. Run the development set (same settings, same number of repeats). Score.
    d. **Check:** total improved and no criterion dropped more than noise. If yes, run the held-out set once; if held-out did not regress, keep the version. If no, revert to the previous version and record what you learned about the failure group.
    e. Re-read the failures. Regroup. Continue.
12. **STOP POINT** when: the budget is spent; the development set scores at the target with held-out within noise; or two consecutive rounds produce no kept change (the remaining failures are likely model limitations or need new cases).

**Artifact:** `eval/results/round-<N>.md` per round with hypothesis, diff summary, scores, decision.

## Phase 5: Close

13. Compare the final prompt to the baseline on both sets. Read the held-out failures once, now, and classify them; do not iterate on them.
14. Audit the final prompt for contradictions and dead instructions (rules that no case exercises). Remove dead instructions one at a time, re-running the development set after each; a shorter prompt that scores the same is better.
15. Write `eval/REPORT.md`: baseline vs final per criterion, the changes kept with their measured effect, the changes rejected with why, remaining failure classes, and the cases to add next time.

**Final artifact:** the new prompt version, the eval set, the runner, and `eval/REPORT.md`.

## Rules throughout

- One change per round. Two changes in one round make the result uninterpretable.
- Never edit a case to make the prompt pass it. If a case is wrong, fix it and re-run the baseline too.
- Keep inputs that contain instructions to the model in the set; the prompt must treat them as data.
- Record every run, including the ones you revert.