One Year After “The Second Half”
Back in the first half of 2025, Yao Shunyu — then still at OAI — wrote in The Second Half:
“I think we should fundamentally re-think evaluation. It means not just to create new and harder benchmarks, but to fundamentally question existing evaluation setups and create new ones, so that we are forced to invent new methods beyond the working recipe.”
AI’s “second half” has already arrived, and one of its central themes is precisely this rethinking of evaluation.
Standing in mid-2026 and looking back, everything has changed fast — but some things that don’t change are still right there.
The buzzword has shifted from context engineering to loop engineering, yet when you break a Loop apart, the core piece remains the Evaluator — it’s what decides when the loop stops and what signal drives improvement. The packaging changed, but the kernel didn’t. All that really happened is that a lot of what used to be manual iteration got automated.
Why Traditional Development Intuitions Break Down
Randomness
Engineers are no strangers to “build the verification first, then optimize.” In software development, test cases are supposed to be ready during the requirements phase — nobody wants to discover that the system doesn’t meet requirements only after it ships. In machine learning it’s even more true: you carve out a validation set before training begins, and during experimentation you use metrics to iterate fast, rather than shipping the model and crossing your fingers. Evaluation first — it’s an old engineering rule.
If agent development were just about copy-pasting this rule, things would be simple. But the analogy falls apart: software testing is deterministic, while evaluating LLM systems is statistical.
In the old world of unit tests, there were only two states — green or red — and running a test once was no different from running it a hundred times. But in a probabilistic model, the same case run ten times might succeed eight times and fail twice (pass@K or Pass^K). This means we’re no longer in a binary pass/no-pass world; we have to adopt the language of statistics (pass-ratio, variance, confidence interval). Every measurement is merely a sample.
Complexity Accumulation
In traditional systems, complexity comes from the number of components, but each component is deterministic. An agent system, by contrast, is non-determinism multiplied by component count — the model, prompt, tool definitions, retrieval, memory, multi-turn orchestration — every layer introduces its own randomness, and they are chained in series.
Put differently, earlier on we evaluated models; now we evaluate an entire system (model + harness). As noted in Why benchmarking is hard, “many moving parts and degrees of freedom affect the final result.”
When ten non-deterministic components are strung together, the global impact of any local change exceeds what the human brain can reason through. This kind of impact cannot be deduced in advance; it can only be measured after the fact. The systems we face are increasingly resembling complex systems in nature: hard to predict, only possible to observe — formulate a hypothesis, run an experiment, look at the data.
That makes induction, deduction, and controlled-variable qualitative and quantitative analysis especially important. Naturally, this is also one of the only two useful things a former physics student gets trained on in school (the other being how to learn an entire semester’s course and pass the exam in two weeks [doge]).
Why Evaluation Must Precede Optimization
“Evaluation first” is most easily misunderstood as “write tests early.”
A Yardstick Before Optimization
Optimization requires at least three preconditions: knowing where the system currently stands; knowing which direction you want it to move; and being able to compare results before and after a change. All three depend on the same thing — a yardstick.
Without evaluation, “it got better” is just a feeling. You must first answer: “what does ‘good’ mean?” Take a customer service agent as an example. “Better performance” can mean at least six different things: more complete answers, faster responses, fewer escalations to human agents, no unauthorized promises, higher first-contact resolution rate, improved user satisfaction. These are not only different — they can actively conflict with each other.
Once evaluation criteria are set, they will in turn determine how the system is designed.
Eval Is Not a Report Card — It Is the Objective Function
Agent optimization is almost always multi-objective. Improvement in one metric often comes at the expense of another — getting better locally does not mean the system got better overall. So before you start optimizing, the evaluation framework must answer four questions: what is the core objective; which metrics are negotiable; which metrics must not regress; and which failures are unacceptable red lines. Otherwise, you are doing Brownian motion.
Evaluation is not here to prove yourself.
Build the Loop First
This is about iteration efficiency. Without evaluation, every change requires a human to re-experience, discuss, and argue — feedback is measured in days. With a minimal eval set, iteration becomes a closed loop:
Propose a hypothesis → modify the system → run evaluation → analyze changes → decide to keep or roll back.
Feedback goes from “measured in days” to “measured in minutes.” The earlier you build this loop, the faster every subsequent optimization becomes.
Someone will say: we are still in the demo phase, building this whole thing is too expensive. That judgment comes from an over-weighted imagination of what an “evaluation system” looks like — a platform, a labeling team, thousands of data points. The first version needs none of that: 20 to 50 cases, a set of metrics, a simple evaluation script, half a day of work. It will be crude, but it can already answer the most expensive question of all: this change — did it make things better, or worse?
Humans Are Moving Up the Stack
We used to write code line by line. Then came the coding agent — autonomously coding, running tests, fixing bugs, committing — and humans simply could not keep up with its output velocity anymore. Reviewing every change one by one stopped making sense. Look back at every major shift from the Industrial Revolution to today: humans were never straightforwardly “replaced.” Instead, they kept moving up the stack — from executor, to reviewer, to system designer and observer.
This ascent forces a question you cannot dodge: if you no longer do the work yourself, and you no longer inspect every output, where does your judgment enter the system?
There is only one channel left: the evaluation criteria. Your definition of “good,” your taste — all of it must be encoded into executable Tasks, verifiable success criteria, and reproducible eval sets. The evaluation system is not an accessory bolted onto the system. It is the form that human judgment takes inside an automated system.
And “optimizing a system you cannot directly reason about by measuring it and feeding back the results” — that idea is not new at all. It is what experimental science has been doing for centuries.
Conclusion
- Agent behavior requires statistical measurement
- Complex systems require experimentation, not intuition
- Evaluation is not just acceptance — it’s development infrastructure
About This Series
This article is the opening piece of the Agent Eval Practical Guide series. This series attempts to answer one question: When a system’s behavior can no longer be reasoned about, how do we know it has improved?
Why evaluation must come first (this article) → How to build your first eval set → How to choose metrics → How to evaluate chatbots, RAG, and agents respectively → How to test continuously after going live → How evaluation becomes the way your team works.
