If you opened this article with the question “where do I find 50 eval queries,” I suggest starting with a different question.

The hard part has never been scraping together 50 queries. The hard part is turning every single query into a case that can be scored consistently: what is it testing, what counts as success, which point in time and data state does it depend on, which intermediate steps cannot be skipped, and how should it feed into the next round of regression testing after a failure.

A query is not yet a case. It only becomes usable for evaluation when the success criteria, contextual state, and necessary constraints are all explicit.

A case that can actually be scored looks roughly like this:

case = {
  query,               # Input: the question, task instruction
  judgment,            # Success criteria: statements / rubric, not a "golden answer"
  trajectory_anchors,  # Intermediate trajectory anchors: expect-tools, key intermediate states, forbidden items
  world_state,         # World state: reference_time, data snapshot identifier
  metadata             # Source, capability dimension, difficulty, version
}

The schema above is the outline for this article:

  • Where queries come from — sources and reverse synthesis

  • How to write judgments — how to write the criteria

  • Why trajectory_anchors and world_state cannot be omitted — what else a case needs to carry

  • How metadata supports coverage and iteration — how many cases are enough, and the data flywheel

Where Queries Come From

Listed in order of availability by project stage.

Dogfooding. During development, the failures the team sees firsthand while using the system — the success criteria are often obvious right then and there. Each one is a ready-made case. There is a trap, of course: people familiar with the system unconsciously steer clear of question patterns it handles poorly, so this batch of cases skews toward the happy path. Do not count on diversity here.

Requirements and acceptance criteria. Every functional description in the PRD, every check item in the acceptance checklist — each is naturally the embryo of a case. The flaw is equally obvious: the language in requirements documents is the PM’s language, not the user’s language.

Production logs and systems already live. Whether an existing chatbot application or an off-the-shelf SaaS system — the point here is not to randomly sample logs, but to mine high-frequency queries (demand) from a business perspective.

Synthetic data. Use an LLM to generate, supplement, and expand. This comes last not because it is unimportant, but because it should never stand alone. Good synthesis is always seeded with one of the three real-data sources above.

Reverse Synthesis

Trying to think up questions from scratch usually yields nothing useful. But answers and raw material are often already sitting in existing resources — reverse-engineer questions from them.

RAG scenarios. A test set entry is (query, judgment, references). Coming up with queries out of thin air is hard, but references are already there. Pick a document passage, ask an LLM to generate “questions this passage can answer,” and you have a QA pair — with answers that are grounded and verifiable. But this has a problem too: directly synthesized queries rarely sound like something a real person would ask.

Text2SQL scenarios. A case is (query, golden-sql, expect-tools, expect-data, …). Missing natural-language queries? But the high-frequency SQL executed in your SaaS system is already there, and so are the annotated reporting scripts. Feed these to an LLM and reverse-engineer “how a business user would phrase this SQL in a single sentence” — the golden-sql practically writes itself.

Synthesis pipeline. Quite a few evaluation frameworks support synthesizing test sets from existing documents (e.g., DeepEval’s synthesizer), but the results tend to be stiff. Building your own pipeline is not complicated. Two principles: first, mutate from real seeds rather than generating from nothing; second, introduce personas. Synthetic data sounds unnatural largely because there is no “person asking the question” — give the LLM a set of user profiles (role, intent, expression habits, which you can build from your business ontology).

Filter. Run synthesized results through an automated check first: deduplication, low-quality removal (length, format, etc.), and out-of-scope filtering. Machine screening first, then human review.

HITL. The human review step cannot be skipped. But human effort is expensive, so here is a principle that will recur throughout this article: favor true/false questions over multiple choice, and multiple choice over open-ended questions — minimize annotation friction wherever possible.

How to Write Judgments

How do you define “got it right”?

The most intuitive approach: an eval set shaped as (query, golden-answer), then have an LLM-as-a-Judge compare the system output against the golden-answer for “consistency.” The judge immediately faces a dilemma — to what degree of “consistency” counts as consistent? Scoring variance goes through the roof, and the scores are unexplainable.

Wherever possible, break the “golden answer” into verifiable decision points: statements (factual claims the response must / must not include), or a rubric (a checklist of scoring and penalty points).

This also solves a cost problem along the way. Asking domain experts to write golden answers is both expensive and useless: writing a single “perfect answer” takes half an hour, and it is still hard to use for evaluation.

  • Experts do not write answers — they only mark scoring points and penalty points. Their judgment of “what matters most, what must never be wrong” is the truly scarce resource.

  • Use the rubric to derive a reference answer in reverse, and leave it to the LLM to write. The cost approaches zero.

(Once again: turn open-ended questions into true/false questions. Lower annotation friction — that principle.)

What Else a Case Needs to Carry

Having the query and judgment is not enough.

world_state: the world state the case depends on

This is a real pitfall we stepped into. When evaluating Text2SQL, a batch of cases used time-relative phrasing — “how were East China sales last month.” The golden-sql was generated based on “last month” relative to the day the case was written. A few weeks later, running regression on the same case, “last month” as understood by the system was now a different month. Ever since, two fields have been added to the schema: reference_time and a data snapshot identifier. Whether mocking the system clock or injecting context, both can be used.

trajectory_anchors: anchors along the intermediate trajectory

For a chatbot, testing the final answer may be enough. For an agent, testing only the end-to-end result misses a huge amount of information.

Anchors include but are not limited to: expect-tools (which tools should be called), key intermediate states, and forbidden items (calls or states that must never occur). As for “which trajectory is better,” coarse metrics like fewer turns or lower token consumption can only serve as a floor — anchor hit rate carries far more information.

Annotating intermediate processes has another benefit: a large number of judgments can be downgraded from judge-LLM to code-level comparison — faster, cheaper, and free of randomness.

How Many Cases Are Enough

What matters is not the total count. It is the distribution.

A coverage matrix: one axis is capability dimensions (the task types the system is expected to cover), the other axis is difficulty or scenario variants (straightforward single-turn / multi-constraint / multi-hop / …).

This is where the metadata field earns its keep: tag each case with its capability dimension, difficulty, and source, so the coverage matrix can be computed automatically rather than tallied by hand.

The Data Flywheel

The first version of an eval set is static. But the system will keep surfacing new problems, and the eval set keeps updating.

The source is, naturally, production. Once live (how to sample, how to trace — covered in detail in part five), every online failure is a candidate case. Once the pipeline of “failure recirculation” is established, the eval set transforms from a maintenance burden into an asset that appreciates alongside your system.

Then, version-control the eval set.

Series Navigation

LLMOps + Evaluation Frameworks

Data Synthesis Frameworks / Data Synthesis Pipelines

Data Annotation