[{"content":"When Was the Last Time You Did Old-School Programming Old-school programming is, by now, something of a derogatory term. Not using an agent, typing everything out line by line — it sounds like you\u0026rsquo;ve fallen behind. And with the help of coding agents, people like us have gradually become the thing we used to look down on — a kind of technical manager who doesn\u0026rsquo;t really understand anything.\nBut what actually gives me the chills isn\u0026rsquo;t this.\nExperience, intuition, the judgment of whether an approach is any good — they all come from things you\u0026rsquo;ve done with your own hands. With Fable 5 in hand, you often slip into the illusion that \u0026ldquo;I can beat 99% of my peers.\u0026rdquo; But over this past year and a half, how much have I actually grown? Have I grown at all? Or has it just been the model getting stronger?\nMicrosoft\u0026rsquo;s 2026 Work Trend Index surveyed 20,000 employees across ten countries and found that only 16% can move fluidly between \u0026ldquo;directing AI\u0026rdquo; and \u0026ldquo;doing it themselves.\u0026rdquo; These people are classified as Frontier Professionals. They share one common habit: deliberately leaving some work to be done without AI, to keep their skills sharp.\nThis doesn\u0026rsquo;t prove that old-school programming prevents skill atrophy. But it reminds me that doing things by hand may carry a value that doesn\u0026rsquo;t show up in any single delivery.\nAs for those even worse off than me — the recent graduates who haven\u0026rsquo;t yet built up much of a foundation — where are they supposed to start accumulating?\nThe Task Is Done, but the Learning May Never Have Happened In the past, onboarding a newcomer followed a fairly clear path. You started them on a small, well-scoped task: fix a bug, add an endpoint. They\u0026rsquo;d gradually get comfortable with the code, understand the business, and only then move up to systems and architecture.\nThe value of these tasks came in two parts. One was getting the work done; the other was letting the newcomer go through a complete feedback loop: understand the problem, propose an approach, implement it yourself, discover what you missed, and revise your own understanding.\nThese days it\u0026rsquo;s more like: a senior colleague hands you a glider with no engine and says, go fly.\nThis isn\u0026rsquo;t entirely a matter of responsibility. A lot of the time, even people who genuinely want to mentor others face an awkward choice — do I spend a few sentences having an agent write this module, or do I spend the better part of a day explaining the background to the newcomer, spelling out the requirements, waiting for them to finish, reviewing the work, and then pointing out the problems?\nIt gets worse. You hand the task to the newcomer, and the newcomer hands the whole thing to AI. Ask them why it\u0026rsquo;s implemented this way, and odds are they can\u0026rsquo;t answer.\nThe task is done, but the learning may never have happened.\nThis is also where last month\u0026rsquo;s coffee chat started: AI is taking over a slice of the entry-level tasks, and those tasks used to double as the newcomer\u0026rsquo;s training ground.\nOutput and Cultivation Used to Share the Same Work A disclaimer up front: not all menial work deserves to be missed. Some repetitive labor teaches you something the first time through and is just burning time by the fiftieth. AI taking that over is something to welcome, plain and simple.\nWhat actually needs to be recognized are the tasks that look simple but carry real feedback. They\u0026rsquo;re usually small enough for a newcomer to own end to end, real enough that getting it wrong exposes a problem, and they still have someone checking, questioning, and backstopping.\nThat\u0026rsquo;s also where the mentor comes in: pick the right problem, help when the newcomer gets stuck, point out what they missed just as they\u0026rsquo;re sure they got it right, and then decide when to hand them something bigger.\nThis training has always had a cost. The mentor\u0026rsquo;s time, the newcomer\u0026rsquo;s trial and error, the team\u0026rsquo;s tolerance for inefficiency — someone has to bear all of it.\nIt\u0026rsquo;s just that, in the past, that cost was folded into the production process.\nThe CRUD a newcomer writes is part of what ships; the bug they fix is a real problem. The project moves forward, and the person grows along with it. The company gets both today\u0026rsquo;s output and a more skilled employee for tomorrow, without having to book every bit of coaching as a separate training expense. Day-to-day work really did absorb a huge amount of training that was never priced on its own.\nWhat AI changes is the accounting: the same delivery can be completed with less junior labor. So the organization gets a choice — keep the output, skip the cultivation that used to happen alongside it.\nOutput and cultivation have been decoupled.\nThe cultivation half didn\u0026rsquo;t disappear. It just lost its host, becoming something that must be funded separately, scheduled separately, and produces no direct return.\nThat\u0026rsquo;s what I mean by decoupling. It hasn\u0026rsquo;t happened across every task, and it may never happen completely — but it\u0026rsquo;s already enough to change how organizations allocate work.\nThe \u0026ldquo;cultivation system\u0026rdquo; we thought existed is, to a large degree, a byproduct.\nTalent Debt: Who Will Pay for Tomorrow\u0026rsquo;s Judgment? In a Fortune article, Jeff Raikes gives this long-term risk a name: \u0026ldquo;talent debt.\u0026rdquo;\nThe employment data offers some warning signs as well. In its August 2026 update, the Stanford Digital Economy Lab finds that in AI-exposed occupations, employment among 22–25-year-old workers is about 19% lower than what the trend among their low-exposure peers would predict — with no such gap among senior workers. The researchers also stress that this is a descriptive difference, not something to be read directly as a causal estimate of AI-driven job loss, nor evidence of economy-wide job displacement.\nIf the entry points into an industry shrink first, where will the senior people it later needs come from?\nAnswering \u0026ldquo;companies should value cultivation\u0026rdquo; isn\u0026rsquo;t enough. The harder question is: under what conditions will companies keep paying for cultivation?\nIn the past, the output a newcomer produced while learning covered part of the cost; once they grew, the company benefited again from their higher capability. Now, if the task at hand can be handed to AI more cheaply, cultivation has to justify itself even more on future returns.\nAnd future returns are uncertain. How much will they grow? How long will it take? And will they leave for another company once they do?\nThe benefits of cultivation accrue to the individual, their current employer, and their future employers; the company that bears the cost of mentoring gets to keep only part of it. This mismatch existed before — AI just makes it harder to hide behind near-term output.\nSo I\u0026rsquo;m reluctant to chalk all of this up to \u0026ldquo;companies getting less patient.\u0026rdquo; For some companies, hiring fewer newcomers and leaning on AI more may genuinely be the more cost-effective choice under current constraints.\nBut if a large number of companies make that choice at the same time, learning opportunities across the industry could keep shrinking.\nIn the short run, every company raises its per-person productivity. In the long run, everyone hopes to hire people who already have judgment from the outside. As for where those people actually made the leap from newcomer to senior — the job description won\u0026rsquo;t say.\nWhat Cultivation Can Actually Change Before discussing how to patch this path, there\u0026rsquo;s a prerequisite question: how much can we actually cultivate in a person?\nCraft, domain knowledge, and analytical methods are relatively easy to break down into concrete training tasks. Curiosity, initiative, a sense of responsibility, and the willingness to keep exploring amid uncertainty — not so much.\nI\u0026rsquo;ve genuinely seen a few young people with high agency who, amplified by AI, accomplish things their peers can barely manage. But more often, what I see is struggle and confusion.\nAre we cultivating these tendencies, or simply discovering people who already have them and then giving them the right environment? I don\u0026rsquo;t have an answer. And I\u0026rsquo;m not willing to sort people into \u0026ldquo;has agency\u0026rdquo; and \u0026ldquo;doesn\u0026rsquo;t have agency\u0026rdquo; buckets on that basis. Someone who appears passive in an environment with no context, no support, and no feedback proves very little.\nMaybe reality is still cruel.\nAt the very least, we can narrow the question: Can we design an experience that lets someone make a judgment, see the result, and then revise that judgment?\nThe four directions below all start from here. They require different conditions, and none of them amounts to a complete answer.\n1. Finding tasks where output and training are still bound together \u0026ldquo;Deliberately preserving some inefficiency to train newcomers\u0026rdquo; is hard to sustain as a long-term arrangement. More worth looking for are tasks where simply doing the work requires understanding context, making trade-offs, and touching the results.\nFor example, I\u0026rsquo;ve observed that some people who write code are being sent to customer sites to do Forward Deployed Engineer (FDE)-type work.\nOn the surface, the boundaries between engineering, product, and delivery blur together. But from another angle, real customer needs rarely arrive in the form of a complete spec. Facing real customers, being accountable for outcomes, and getting yelled at on the spot when you get it wrong — these are precisely the few roles where the bundling still holds. This may not be chaos; it\u0026rsquo;s a category of role being rediscovered.\nBut \u0026ldquo;going to the field\u0026rdquo; by itself isn\u0026rsquo;t cultivation. If nobody helps explain the context, you have no authority to adjust the approach, and all you get when something goes wrong is blame, then you\u0026rsquo;re just exposing newcomers to pressure earlier.\n2. Shifting from \u0026ldquo;doing\u0026rdquo; to \u0026ldquo;judging\u0026rdquo; requires closing the feedback gap One natural answer: newcomers no longer need to write so many leaf tasks — they can go straight to reviewing AI\u0026rsquo;s output and learning to make judgments.\nBut what exactly is someone who\u0026rsquo;s never fallen into the pit themselves looking at when they review?\nYou can\u0026rsquo;t simply say \u0026ldquo;you have to have made mistakes yourself to have judgment.\u0026rdquo; Human experience also comes from observation, cases, and other people\u0026rsquo;s feedback. What matters more is whether, after making a judgment, you get a chance to learn why you were right or wrong.\nJudge → predict consequences → verify → explain the deviation.\nThe aviation industry\u0026rsquo;s flight simulation training offers a reference point: people can practice operations and decision-making without bearing the full risk of real flight. In software, some localized problems are equally well suited to becoming training environments — fault localization, data consistency, and missed permissions, for example.\nBut \u0026ldquo;will this architecture turn into technical debt five years from now?\u0026rdquo; is hard to answer reliably with a single simulation.\n3. Letting the payer capture the returns on cultivation If companies are expected to pay for cultivation on their own, historically it has only happened under three conditions:\nExtremely high employee-turnover costs — people can\u0026rsquo;t be poached, or poaching them costs too much;\nExtremely long cycles — semiconductors, for instance: if you can\u0026rsquo;t afford to wait, there is no next generation;\nSafety-critical, externally mandated — aviation and medicine: no license without training.\nSoftware hits none of the three. Turnover is high, product cycles are measured in weeks, and when a bug ships you just roll back. Expecting software companies to pay for this on their own is unrealistic.\nSo only two paths remain: either wait for some kind of \u0026ldquo;crash\u0026rdquo; to forcibly internalize the cost, or have someone outside the company pay for it.\n4. Sharing what no single company can bear What remains is probably collective funding.\nThis slot has always existed throughout history — guilds, apprenticeship systems, professional associations. The very reason they exist is that no single employer has an incentive to cultivate general-purpose talent, but the industry as a whole needs it.\nThe open-source community has actually been quietly playing this role all along, providing both output and feedback, with the cost shared across all participants.\nGrowth No Longer Happens Just as a Byproduct Back to decoupling itself.\nAll of our past default assumptions about growth — schools teach the fundamentals, companies provide a training ground, time naturally produces seniority — rested on a single assumption: that cultivation would happen incidentally.\nAI makes that relationship far less dependable: delivery can keep growing even as people experience independent judgment and self-correction less and less.\nWhat AI is dismantling is precisely that default — the one where cultivation happens as a byproduct.\nGoing forward, growth may demand more deliberately designed tasks, time set aside, and clarity about who bears the cost.\nA line I\u0026rsquo;m fond of:\nEvery flower blooms in its own time, and only you know what kind of flower you are.\nIn the era when \u0026ldquo;incidentally\u0026rdquo; still held, this line was a comfort. Now it reads more like a reminder: as ready-made growth paths grow fewer, you need to know, more actively, what you want to grow into.\nBut that half alone isn\u0026rsquo;t enough. Knowing what you want to grow into still requires a place that lets you grow.\nPlaces willing to give a little time to those who haven\u0026rsquo;t bloomed yet will no longer appear incidentally.\nRecommended Reading How AI assistance impacts the formation of coding skills\nThe Impact of Generative AI on Critical Thinking\nCanaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence\nJeff Raikes: The talent debt I warned about is now showing up in the data\nJeff Raikes: AI is capturing cognition — and most companies are building a talent debt they don\u0026rsquo;t see yet\nWhen Work No Longer Lets Us Learn\nThe Young People No Longer Needed, and the Middle-Aged Workers Who Do Their Work for Them\n","permalink":"https://niraya666.github.io/en/posts/ai-decouples-production-and-cultivation/","summary":"AI is decoupling output from cultivation: as entry-level tasks are absorbed, how talent debt forms, where judgment comes from, and why growth no longer just happens on its own.","title":"AI Decouples Production from Cultivation"},{"content":"Who judges? And how?\nThere are three mainstream approaches to scoring today:\n- Code-based Evaluators\n- LLM-as-a-Judge\n- Human Evaluation\nCode-based Evaluators Code-based Evaluators are suited for tasks with hard rules, explicit states, or computable answers.\nThis type of evaluation is fast, low-cost, and produces stable results — ideal for CI/CD pipelines as regression tests after every change to the model, prompt, tools, or workflow.\nFor task execution specifically, evaluation should care about the final result or system state, not just what the model said. For tasks with clearly defined answers, you can directly check whether the model output matches the expected result.\nDuring R\u0026amp;D, we often need to quickly identify an optimization direction — choosing a model, modifying a prompt, or tuning hyperparameters. This process requires running a large number of experiments. If every round depends on human annotation and scoring, the evaluation cycle, labor cost, and subjective variance all become confounding factors, and experiments become hard to reproduce reliably.\nThis is where Code-based Evaluators come in: scoring through automated, standardized, reproducible, and quantifiable methods.\nAt its core, this involves two parts: datasets and evaluation metrics. Datasets were covered in the previous article, so this one focuses on metrics.\nHow to Choose Metrics Metric selection is fundamentally a modeling exercise — translating business objectives into quantifiable questions. Different task types call for different evaluation metrics.\nFor classification tasks, Accuracy is a common baseline metric. But in scenarios with class imbalance — medical diagnosis, for instance — relying on Accuracy alone can be highly misleading. You may need to additionally track Recall, and depending on the actual problem, choose metrics like F1-Score, MCC (Matthews Correlation Coefficient), or ROC curves.\nFor tasks with definitive answers — a final numerical value in a math problem, a classification label, or a fixed identifier — evaluation is relatively straightforward. Exact Match can directly check whether outputs are consistent; pass@k measures the probability that the model succeeds at least once across k samples.\nEvaluating LLM-generated responses, by contrast, is more challenging.\nExact Match works for scenarios where answer expression is tightly constrained, but has clear limitations for open-domain QA. N-gram metrics like BLEU and ROUGE focus on lexical overlap and struggle to capture full semantic relationships. Methods like BERTScore compute semantic similarity via embeddings, but introduce a new model variable — and similarity scores are usually relative values that still require calibration and benchmarking.\nThese open-text tasks typically call for LLM-as-a-Judge.\nAs model capabilities grow, evaluation targets are also expanding beyond accuracy to dimensions like safety, trustworthiness, stability, and cost.\nDon\u0026rsquo;t Let Metrics Replace the Task Objective BLEU, ROUGE, and Embedding Similarity can all be computed automatically by code — but \u0026ldquo;can be computed automatically\u0026rdquo; does not mean they effectively reflect business quality.\nA response with high semantic similarity can still miss a critical constraint; a response with low lexical overlap can still be entirely correct. Metrics are merely measurement proxies for business objectives. Once a proxy decouples from the real objective, optimizing the metric may actually push the system away from the problem you genuinely need to solve.\nLLM-as-a-Judge Many tasks have no single correct answer.\nFor example: determining whether a response is complete, whether it faithfully adheres to reference materials, whether it overlooks risks, or whether an analysis genuinely addresses the user\u0026rsquo;s question. These criteria can be described in words, but are exceedingly difficult to encode as an exhaustive set of rules.\nThis is where LLM-as-a-Judge comes in. Common approaches include:\nClassification: pass/fail judgment, or categorizing the type of error.\nScoring: assigning a score of 1–5 against an explicit rubric.\nPairwise comparison: comparing version A and version B, judging which better meets the criteria.\n(and more…)\nAmong these, pairwise comparison is generally more stable than assigning absolute scores. When presented with two concrete answers, humans also tend to find it easier to judge which is better, rather than precisely articulating \u0026ldquo;why this is a 3 and not a 4.\u0026rdquo;\nThe value of an LLM Judge lies in scale. It can perform initial screening and repeat evaluation across large volumes of open text, and can output error categories and scoring rationales, helping teams pinpoint issues.\nBut a Judge\u0026rsquo;s score is still a model output — a scalable approximation of judgment, one that itself needs to be evaluated and calibrated.\nWhat to Watch Out for with LLM Judges Judge one clear dimension at a time\n\u0026ldquo;Please comprehensively evaluate the quality of this response\u0026rdquo; rarely yields stable results. Break it down into factual correctness, completeness, relevance, and safety — and write observable scoring descriptors for each. The results become far more interpretable.\nMake the rubric executable\n\u0026ldquo;Professional content\u0026rdquo; and \u0026ldquo;good user experience\u0026rdquo; are both too vague. A more effective rubric spells out which facts must be included, which errors count as critical failures, and what performance each score level corresponds to.\nThis is also why the previous article recommended breaking Golden Answers into Statements and Rubrics: having a Judge verify a clear set of decision points is far more stable than comparing whether two long passages are \u0026ldquo;similar.\u0026rdquo;\nRequire evidence and reasoning\nKeeping only a score makes it hard to detect where the Judge went wrong. Have the Judge cite evidence from the evaluated response and output structured reasoning — this makes human spot-checking and downstream analysis feasible.\nAddress known biases\nJudges may favor longer answers, more familiar formats, options appearing earlier in the list, and content that matches their own stylistic patterns.\nWhen doing pairwise comparisons, randomly swap answer order to reduce position bias. For critical cases, repeat evaluation and check for consistency via self-consistency. When necessary, aggregate results from multiple Judges.\nFor content that is \u0026ldquo;partially correct but severely wrong in specific areas,\u0026rdquo; Judges can still miss failures. Knowledge boundary issues and hallucinations in specialized domains also require human review.\nPin and record versions\nThe Judge model, prompt, temperature, and rubric are all part of the evaluator. Change any one of them, and the evaluation scale may drift. Experiment logs should preserve the full version and final scores.\nCalibrate with human-annotated samples\nBefore large-scale execution, prepare a small set of samples judged by domain experts and check the Judge\u0026rsquo;s alignment with human conclusions. For high-risk or specialized domains, ongoing spot-checking is also necessary.\nHuman Evaluation Human evaluation is the most expensive and slowest approach, but it shoulders work that the other two methods cannot accomplish independently.\nHuman capacity for complex semantic understanding and value judgment gives Human Evaluation an irreplaceable position — and also accounts for its higher cost.\nSystematic Human Evaluation, depending on the completeness of available evaluation resources, can take several forms:\nWhen no ready-made dataset exists: provide complete task descriptions and structured scoring guidelines. For example, when evaluating a dialogue system, clearly define dimensions like \u0026ldquo;coherence,\u0026rdquo; \u0026ldquo;relevance,\u0026rdquo; and \u0026ldquo;safety,\u0026rdquo; and require annotators to provide scoring rationales.\nWhen a dataset already exists: provide annotators with model inputs, outputs, and scoring guidelines together.\nWhen both a dataset and automated scoring results exist: have human annotators review the results through Error Annotation.\nHuman evaluation cannot be reduced to \u0026ldquo;grab a few people and have them try it out based on gut feeling.\u0026rdquo; Systematic human evaluation requires, at minimum: task descriptions, model inputs and outputs, scoring dimensions, score definitions, and examples. When necessary, annotators should also record error categories, severity, and scoring rationales.\nCompared to vague scoring, Error Annotation is often far more useful. Having domain experts point out where the error occurred, what type it belongs to, and how severe it is — the resulting information feeds directly into error analysis and eval set iteration.\nWhat to Watch Out for with Human Evaluation Choose people who genuinely possess judgment ability\nFluency can be left to general annotators; whether a professional conclusion is reliable requires domain experts. Expert time is scarce — concentrate it on the parts that truly demand professional judgment.\nReduce the cognitive burden of annotation\nThe previous article introduced a principle: favor true/false questions over multiple choice, and multiple choice over open-ended questions. Breaking complex tasks into explicit decision points lowers costs and reduces interpretation variance across annotators.\nRun a small-scale pilot first\nBefore scaling up annotation, test a small batch of samples to check whether the instructions are clear and whether edge cases produce disagreement. If different annotators consistently fail to reach agreement, the problem likely lies in the rubric itself.\nDocument disagreements\nDisagreement is not just noise. It may signal ambiguity in the task definition, or that the business side has yet to converge on a unified standard. Further judgment is needed on whether the disagreement stems from insufficient knowledge, unclear instructions, or genuinely different value judgments.\nEvaluators Need Controlled Variables Too Regardless of which scoring method you use, evaluation results are affected by experimental setup.\nPrompts affect results. A sound prompt should include a clear task description, the context needed to complete the task, and specific, unambiguous questions.\nRepeat experiments to reduce random error. Design multiple prompt variants for the same task, vary temperature, shuffle order, and so on.\nStructured Output is also an experimental variable. Structured output helps improve evaluation controllability and reduces extraneous content, but can also alter the model\u0026rsquo;s output distribution. When your actual deployment uses JSON, Schema, or Tool Calling, evaluation should ideally run under the same interface conditions.\nSummary Code-based Evaluators are suited for tasks that can be directly verified through hard rules, definitive answers, or system state.\nLLM-as-a-Judge is suited for tasks that require open-ended semantic judgment at scale — but it carries its own randomness, biases, and scale drift.\nHuman Evaluation is responsible for defining standards, calibrating automated evaluation results, and handling samples that are high-stakes, highly specialized, or contentious.\n","permalink":"https://niraya666.github.io/en/posts/agent-eval-guide-03/","summary":"Part three of the Agent Eval Practical Guide: a systematic overview of when to use code-based evaluation, LLM judging, and human evaluation, how to translate business goals into quantifiable metrics, and how evaluators themselves should be evaluated and calibrated.","title":"Agent Eval Practical Guide 03: How to Choose Evaluation Methods"},{"content":"If you opened this article with the question \u0026ldquo;where do I find 50 eval queries,\u0026rdquo; I suggest starting with a different question.\nThe hard part has never been scraping together 50 queries. The hard part is turning every single query into a case that can be scored consistently: what is it testing, what counts as success, which point in time and data state does it depend on, which intermediate steps cannot be skipped, and how should it feed into the next round of regression testing after a failure.\nA query is not yet a case. It only becomes usable for evaluation when the success criteria, contextual state, and necessary constraints are all explicit.\nA case that can actually be scored looks roughly like this:\ncase = { query, # Input: the question, task instruction judgment, # Success criteria: statements / rubric, not a \u0026#34;golden answer\u0026#34; trajectory_anchors, # Intermediate trajectory anchors: expect-tools, key intermediate states, forbidden items world_state, # World state: reference_time, data snapshot identifier metadata # Source, capability dimension, difficulty, version } The schema above is the outline for this article:\nWhere queries come from — sources and reverse synthesis\nHow to write judgments — how to write the criteria\nWhy trajectory_anchors and world_state cannot be omitted — what else a case needs to carry\nHow metadata supports coverage and iteration — how many cases are enough, and the data flywheel\nWhere Queries Come From Listed in order of availability by project stage.\nDogfooding. During development, the failures the team sees firsthand while using the system — the success criteria are often obvious right then and there. Each one is a ready-made case. There is a trap, of course: people familiar with the system unconsciously steer clear of question patterns it handles poorly, so this batch of cases skews toward the happy path. Do not count on diversity here.\nRequirements and acceptance criteria. Every functional description in the PRD, every check item in the acceptance checklist — each is naturally the embryo of a case. The flaw is equally obvious: the language in requirements documents is the PM\u0026rsquo;s language, not the user\u0026rsquo;s language.\nProduction logs and systems already live. Whether an existing chatbot application or an off-the-shelf SaaS system — the point here is not to randomly sample logs, but to mine high-frequency queries (demand) from a business perspective.\nSynthetic data. Use an LLM to generate, supplement, and expand. This comes last not because it is unimportant, but because it should never stand alone. Good synthesis is always seeded with one of the three real-data sources above.\nReverse Synthesis Trying to think up questions from scratch usually yields nothing useful. But answers and raw material are often already sitting in existing resources — reverse-engineer questions from them.\nRAG scenarios. A test set entry is (query, judgment, references). Coming up with queries out of thin air is hard, but references are already there. Pick a document passage, ask an LLM to generate \u0026ldquo;questions this passage can answer,\u0026rdquo; and you have a QA pair — with answers that are grounded and verifiable. But this has a problem too: directly synthesized queries rarely sound like something a real person would ask.\nText2SQL scenarios. A case is (query, golden-sql, expect-tools, expect-data, …). Missing natural-language queries? But the high-frequency SQL executed in your SaaS system is already there, and so are the annotated reporting scripts. Feed these to an LLM and reverse-engineer \u0026ldquo;how a business user would phrase this SQL in a single sentence\u0026rdquo; — the golden-sql practically writes itself.\nSynthesis pipeline. Quite a few evaluation frameworks support synthesizing test sets from existing documents (e.g., DeepEval\u0026rsquo;s synthesizer), but the results tend to be stiff. Building your own pipeline is not complicated. Two principles: first, mutate from real seeds rather than generating from nothing; second, introduce personas. Synthetic data sounds unnatural largely because there is no \u0026ldquo;person asking the question\u0026rdquo; — give the LLM a set of user profiles (role, intent, expression habits, which you can build from your business ontology).\nFilter. Run synthesized results through an automated check first: deduplication, low-quality removal (length, format, etc.), and out-of-scope filtering. Machine screening first, then human review.\nHITL. The human review step cannot be skipped. But human effort is expensive, so here is a principle that will recur throughout this article: favor true/false questions over multiple choice, and multiple choice over open-ended questions — minimize annotation friction wherever possible.\nHow to Write Judgments How do you define \u0026ldquo;got it right\u0026rdquo;?\nThe most intuitive approach: an eval set shaped as (query, golden-answer), then have an LLM-as-a-Judge compare the system output against the golden-answer for \u0026ldquo;consistency.\u0026rdquo; The judge immediately faces a dilemma — to what degree of \u0026ldquo;consistency\u0026rdquo; counts as consistent? Scoring variance goes through the roof, and the scores are unexplainable.\nWherever possible, break the \u0026ldquo;golden answer\u0026rdquo; into verifiable decision points: statements (factual claims the response must / must not include), or a rubric (a checklist of scoring and penalty points).\nThis also solves a cost problem along the way. Asking domain experts to write golden answers is both expensive and useless: writing a single \u0026ldquo;perfect answer\u0026rdquo; takes half an hour, and it is still hard to use for evaluation.\nExperts do not write answers — they only mark scoring points and penalty points. Their judgment of \u0026ldquo;what matters most, what must never be wrong\u0026rdquo; is the truly scarce resource.\nUse the rubric to derive a reference answer in reverse, and leave it to the LLM to write. The cost approaches zero.\n(Once again: turn open-ended questions into true/false questions. Lower annotation friction — that principle.)\nWhat Else a Case Needs to Carry Having the query and judgment is not enough.\nworld_state: the world state the case depends on\nThis is a real pitfall we stepped into. When evaluating Text2SQL, a batch of cases used time-relative phrasing — \u0026ldquo;how were East China sales last month.\u0026rdquo; The golden-sql was generated based on \u0026ldquo;last month\u0026rdquo; relative to the day the case was written. A few weeks later, running regression on the same case, \u0026ldquo;last month\u0026rdquo; as understood by the system was now a different month. Ever since, two fields have been added to the schema: reference_time and a data snapshot identifier. Whether mocking the system clock or injecting context, both can be used.\ntrajectory_anchors: anchors along the intermediate trajectory\nFor a chatbot, testing the final answer may be enough. For an agent, testing only the end-to-end result misses a huge amount of information.\nAnchors include but are not limited to: expect-tools (which tools should be called), key intermediate states, and forbidden items (calls or states that must never occur). As for \u0026ldquo;which trajectory is better,\u0026rdquo; coarse metrics like fewer turns or lower token consumption can only serve as a floor — anchor hit rate carries far more information.\nAnnotating intermediate processes has another benefit: a large number of judgments can be downgraded from judge-LLM to code-level comparison — faster, cheaper, and free of randomness.\nHow Many Cases Are Enough What matters is not the total count. It is the distribution.\nA coverage matrix: one axis is capability dimensions (the task types the system is expected to cover), the other axis is difficulty or scenario variants (straightforward single-turn / multi-constraint / multi-hop / …).\nThis is where the metadata field earns its keep: tag each case with its capability dimension, difficulty, and source, so the coverage matrix can be computed automatically rather than tallied by hand.\nThe Data Flywheel The first version of an eval set is static. But the system will keep surfacing new problems, and the eval set keeps updating.\nThe source is, naturally, production. Once live (how to sample, how to trace — covered in detail in part five), every online failure is a candidate case. Once the pipeline of \u0026ldquo;failure recirculation\u0026rdquo; is established, the eval set transforms from a maintenance burden into an asset that appreciates alongside your system.\nThen, version-control the eval set.\nSeries Navigation Previous: Agent Eval Practical Guide 01: The More Complex the System, the More Evaluation Must Come First Next: Agent Eval Practical Guide 03: How to Choose Evaluation Methods Appendix: Recommended Tools LLMOps + Evaluation Frameworks\nlangfuse\nharbor\nDeepEval\nTrulens\nRAGAS\nlangsmith\nData Synthesis Frameworks / Data Synthesis Pipelines\nDistilabel\ndataJuicer\ncocoindex\nData Annotation\nargilla ","permalink":"https://niraya666.github.io/en/posts/agent-eval-guide-02/","summary":"Part two of the Agent Eval Practical Guide: from query sources, reverse synthesis, and judgment design, to the necessity of world_state and trajectory_anchors — a complete breakdown of what makes an eval case production-grade.","title":"Agent Eval Practical Guide 02: How to Build an Eval Set"},{"content":"One Year After \u0026ldquo;The Second Half\u0026rdquo; Back in the first half of 2025, Yao Shunyu — then still at OAI — wrote in The Second Half:\n\u0026ldquo;I think we should fundamentally re-think evaluation. It means not just to create new and harder benchmarks, but to fundamentally question existing evaluation setups and create new ones, so that we are forced to invent new methods beyond the working recipe.\u0026rdquo;\nAI\u0026rsquo;s \u0026ldquo;second half\u0026rdquo; has already arrived, and one of its central themes is precisely this rethinking of evaluation.\nStanding in mid-2026 and looking back, everything has changed fast — but some things that don\u0026rsquo;t change are still right there.\nThe buzzword has shifted from context engineering to loop engineering, yet when you break a Loop apart, the core piece remains the Evaluator — it\u0026rsquo;s what decides when the loop stops and what signal drives improvement. The packaging changed, but the kernel didn\u0026rsquo;t. All that really happened is that a lot of what used to be manual iteration got automated.\nWhy Traditional Development Intuitions Break Down Randomness Engineers are no strangers to \u0026ldquo;build the verification first, then optimize.\u0026rdquo; In software development, test cases are supposed to be ready during the requirements phase — nobody wants to discover that the system doesn\u0026rsquo;t meet requirements only after it ships. In machine learning it\u0026rsquo;s even more true: you carve out a validation set before training begins, and during experimentation you use metrics to iterate fast, rather than shipping the model and crossing your fingers. Evaluation first — it\u0026rsquo;s an old engineering rule.\nIf agent development were just about copy-pasting this rule, things would be simple. But the analogy falls apart: software testing is deterministic, while evaluating LLM systems is statistical.\nIn the old world of unit tests, there were only two states — green or red — and running a test once was no different from running it a hundred times. But in a probabilistic model, the same case run ten times might succeed eight times and fail twice (pass@K or Pass^K). This means we\u0026rsquo;re no longer in a binary pass/no-pass world; we have to adopt the language of statistics (pass-ratio, variance, confidence interval). Every measurement is merely a sample.\nComplexity Accumulation In traditional systems, complexity comes from the number of components, but each component is deterministic. An agent system, by contrast, is non-determinism multiplied by component count — the model, prompt, tool definitions, retrieval, memory, multi-turn orchestration — every layer introduces its own randomness, and they are chained in series.\nPut differently, earlier on we evaluated models; now we evaluate an entire system (model + harness). As noted in Why benchmarking is hard, \u0026ldquo;many moving parts and degrees of freedom affect the final result.\u0026rdquo;\nWhen ten non-deterministic components are strung together, the global impact of any local change exceeds what the human brain can reason through. This kind of impact cannot be deduced in advance; it can only be measured after the fact. The systems we face are increasingly resembling complex systems in nature: hard to predict, only possible to observe — formulate a hypothesis, run an experiment, look at the data.\nThat makes induction, deduction, and controlled-variable qualitative and quantitative analysis especially important. Naturally, this is also one of the only two useful things a former physics student gets trained on in school (the other being how to learn an entire semester\u0026rsquo;s course and pass the exam in two weeks [doge]).\nWhy Evaluation Must Precede Optimization \u0026ldquo;Evaluation first\u0026rdquo; is most easily misunderstood as \u0026ldquo;write tests early.\u0026rdquo;\nA Yardstick Before Optimization Optimization requires at least three preconditions: knowing where the system currently stands; knowing which direction you want it to move; and being able to compare results before and after a change. All three depend on the same thing — a yardstick.\nWithout evaluation, \u0026ldquo;it got better\u0026rdquo; is just a feeling. You must first answer: \u0026ldquo;what does \u0026lsquo;good\u0026rsquo; mean?\u0026rdquo; Take a customer service agent as an example. \u0026ldquo;Better performance\u0026rdquo; can mean at least six different things: more complete answers, faster responses, fewer escalations to human agents, no unauthorized promises, higher first-contact resolution rate, improved user satisfaction. These are not only different — they can actively conflict with each other.\nOnce evaluation criteria are set, they will in turn determine how the system is designed.\nEval Is Not a Report Card — It Is the Objective Function Agent optimization is almost always multi-objective. Improvement in one metric often comes at the expense of another — getting better locally does not mean the system got better overall. So before you start optimizing, the evaluation framework must answer four questions: what is the core objective; which metrics are negotiable; which metrics must not regress; and which failures are unacceptable red lines. Otherwise, you are doing Brownian motion.\nEvaluation is not here to prove yourself.\nBuild the Loop First This is about iteration efficiency. Without evaluation, every change requires a human to re-experience, discuss, and argue — feedback is measured in days. With a minimal eval set, iteration becomes a closed loop:\nPropose a hypothesis → modify the system → run evaluation → analyze changes → decide to keep or roll back.\nFeedback goes from \u0026ldquo;measured in days\u0026rdquo; to \u0026ldquo;measured in minutes.\u0026rdquo; The earlier you build this loop, the faster every subsequent optimization becomes.\nSomeone will say: we are still in the demo phase, building this whole thing is too expensive. That judgment comes from an over-weighted imagination of what an \u0026ldquo;evaluation system\u0026rdquo; looks like — a platform, a labeling team, thousands of data points. The first version needs none of that: 20 to 50 cases, a set of metrics, a simple evaluation script, half a day of work. It will be crude, but it can already answer the most expensive question of all: this change — did it make things better, or worse?\nHumans Are Moving Up the Stack We used to write code line by line. Then came the coding agent — autonomously coding, running tests, fixing bugs, committing — and humans simply could not keep up with its output velocity anymore. Reviewing every change one by one stopped making sense. Look back at every major shift from the Industrial Revolution to today: humans were never straightforwardly \u0026ldquo;replaced.\u0026rdquo; Instead, they kept moving up the stack — from executor, to reviewer, to system designer and observer.\nThis ascent forces a question you cannot dodge: if you no longer do the work yourself, and you no longer inspect every output, where does your judgment enter the system?\nThere is only one channel left: the evaluation criteria. Your definition of \u0026ldquo;good,\u0026rdquo; your taste — all of it must be encoded into executable Tasks, verifiable success criteria, and reproducible eval sets. The evaluation system is not an accessory bolted onto the system. It is the form that human judgment takes inside an automated system.\nAnd \u0026ldquo;optimizing a system you cannot directly reason about by measuring it and feeding back the results\u0026rdquo; — that idea is not new at all. It is what experimental science has been doing for centuries.\nConclusion Agent behavior requires statistical measurement Complex systems require experimentation, not intuition Evaluation is not just acceptance — it\u0026rsquo;s development infrastructure About This Series This article is the opening piece of the Agent Eval Practical Guide series. This series attempts to answer one question: When a system\u0026rsquo;s behavior can no longer be reasoned about, how do we know it has improved?\nWhy evaluation must come first (this article) → How to build your first eval set → How to choose metrics → How to evaluate chatbots, RAG, and agents respectively → How to test continuously after going live → How evaluation becomes the way your team works.\n","permalink":"https://niraya666.github.io/en/posts/agent-eval-guide-01/","summary":"The opening piece of the Agent Eval Practical Guide series: unpacking the non-determinism and complexity accumulation of Agent systems, arguing why evaluation is not a report card but an objective function, and how to compress feedback from days to minutes with a minimal evaluation loop.","title":"Agent Eval Practical Guide 01: The More Complex the System, the More Evaluation Must Come First"},{"content":"In agent-memory research, two benchmarks are virtually inescapable — everyone runs them: LoCoMo and LongMemEval. But does a high score on either truly prove anything? Are there more targeted benchmarks better suited to specific use cases? Should memory evaluation be reduced to accuracy alone? What other metrics matter, and how are the data synthesis pipelines behind these benchmarks actually constructed?\nThese are the questions this article sets out to answer.\nOverall Agent-memory benchmarks broadly fall into two categories: conversational memory and agent task improvement. The former concerns whether the system can remember important user information; the latter focuses on whether the agent can reduce similar types of errors over time.\nNaturally, the evaluation approaches differ — for task improvement, standard coding benchmarks can often serve the purpose. This article focuses on the former: conversational memory.\nMemory benchmarks can be understood along two orthogonal axes.\nAxis 1: What content / scenario is being tested (horizontal) — factual conversational memory (LoCoMo / LongMemEval), personalization \u0026amp; user profiling (PersonaMem / PersonaLens), experiential / environmental memory (LongMemEval-V2), and multimodal memory.\nAxis 2: What capability dimension is being tested (vertical) — lifecycle abilities (update, conflict, forgetting, abstention, temporal reasoning). These cut across all scenarios.\nLong-term Conversational Memory The earliest and most frequently evaluated category of personal memory is long-term conversational memory: given dialogues spanning multiple turns and multiple sessions, can the agent remember user-relevant information and correctly recall it in subsequent questions or responses?\nThis type of benchmark closely resembles traditional long-context QA. The emphasis is on whether the agent can locate, understand, and apply user-related memories across very long conversation histories — both within long sessions and across sessions.\nLoCoMo 📄 LoCoMo Paper · GitHub Repository\nLoCoMo (Long Conversation Memory) is one of the most frequently cited benchmarks in agent-memory research.\nLoCoMo\u0026rsquo;s data is generated through a hybrid human-machine pipeline: starting from predefined personas and temporal event graphs, two LLM agents generate long conversation skeletons, which human annotators then verify and edit to ensure long-range consistency, event grounding, and multimodal content plausibility. The public release contains 10 ultra-long conversations, each averaging roughly 300 turns and 9K tokens, spanning up to 35 sessions.\nLoCoMo defines three evaluation tasks.\nThe first task is Question Answering. This is the part most commonly used by subsequent memory system papers. QA questions fall into five categories:\nCategory What It Tests Challenge Count Single-hop Factual recall within a single session Finding answers directly from one part of the dialogue 841 Multi-hop Synthesizing information across multiple sessions Piecing together evidence from multiple sources 282 Temporal reasoning Temporal reasoning Understanding \u0026ldquo;before,\u0026rdquo; \u0026ldquo;after,\u0026rdquo; \u0026ldquo;months later,\u0026rdquo; and other temporal relations 321 Open-domain / commonsense Dialogue information + external commonsense knowledge Not pure retrieval — requires commonsense inference 96 Adversarial Unanswerable or trick questions Preventing the model from hallucinating in long contexts 446 The second task is Event Summarization. The model must summarize the conversation history into a temporal event graph, testing whether it understands events, causal relationships, and temporal ordering in long-term dialogues.\nThe third task is Multimodal Dialogue Generation. LoCoMo\u0026rsquo;s dialogues include image sharing and image reactions, so this task evaluates whether the model can combine past dialogue and multimodal cues to generate coherent responses consistent with character state and long-term narrative.\nFrom a data structure perspective (locomo10.json), each element in the JSON array represents one complete long conversation with its annotations.\nEvaluation methodology and metrics. Evaluation is not LLM-Judge-dominated.\nQA: normalized / partial-match F1; RAG systems additionally report retrieval recall / recall@k. Event Summarization: ROUGE + FactScore, where FactScore decomposes into atomic facts and computes precision / recall / F1. Multimodal Dialogue Generation: BLEU-1/2, ROUGE-L, MM-Relevance. Its limitations, however, are clear. At roughly 9K tokens on average, the public version is no longer sufficient in 2026 to meaningfully differentiate long-context models. Furthermore, most follow-up work focuses primarily on LoCoMo\u0026rsquo;s QA component, reducing it to a text retrieval / QA benchmark; the event summarization and multimodal dialogue generation tasks remain underutilized in practical system comparisons.\nThis has led to targeted corrections such as LoCoMo-Plus and LoCoMo-Refined. LoCoMo-Plus\u0026rsquo;s direction is particularly noteworthy: it attempts to move from surface-level factual recall to cognitive memory — testing whether a model can apply implicit user states, goals, values, and constraints from earlier dialogues in subsequent scenarios.\n📄 LoCoMo-Plus Paper · LoCoMo-Refined GitHub Repository\nLongMemEval Compared to LoCoMo, LongMemEval is more directly oriented toward long-term interaction memory for chat assistants. Its goal is to evaluate whether an assistant can record, retrieve, update, and reason about user information across sustained, multi-session user interactions.\n📄 LongMemEval Paper · GitHub Repository\nLongMemEval contains 500 questions covering five categories of long-term memory capability:\nCapability Meaning Information Extraction Recalling specific details from long interaction histories, including details once mentioned by the user or assistant Multi-Session Reasoning Synthesizing information across multiple sessions for aggregation, comparison, or inference Knowledge Update Recognizing changes in user information and updating old memories with new information Temporal Reasoning Understanding temporal attributes in user information, including explicit timestamps and session timestamps Abstention Knowing when to refuse to answer rather than hallucinate, when the relevant information is absent from history LongMemEval offers different difficulty tiers. LongMemEval-S provides roughly 115K tokens of history per question, typically spanning 30–40 sessions; LongMemEval-M scales to roughly 500 sessions and 1.5M tokens. This makes it far more suitable than LoCoMo for testing the memory scalability of real personal assistants over extended use.\nFor evaluation, LongMemEval employs LLM-as-a-Judge, with GPT-4o judging whether a model\u0026rsquo;s answer is correct. It also proposes a useful memory system analysis framework: decomposing a long-term memory system into three stages — indexing, retrieval, and reading.\nFrom a personal memory perspective, LongMemEval is closer to real user-assistant scenarios than LoCoMo. It tests not only \u0026ldquo;does it remember,\u0026rdquo; but also whether user information updates over time, whether information can be synthesized across sessions, whether temporal reasoning is handled correctly, and whether the system refuses to answer when information is absent.\nThat said, LongMemEval remains fundamentally a long-context QA / retrieval-oriented benchmark. Its core evaluation format still revolves around finding evidence in chat histories and answering questions. But real-world agents face a harder problem: state is dynamic, making the benchmark susceptible to retrieval engineering.\nFrom LongMemEval to LongMemEval-V2 📄 LongMemEval-V2 GitHub Repository\nNo longer merely testing \u0026ldquo;can a chat assistant remember long-term conversations,\u0026rdquo; but rather \u0026ldquo;can an agent form long-term experiential memory in complex environments.\u0026rdquo;\nV2\u0026rsquo;s object of study is no longer ordinary user-assistant chat history, but web-agent trajectories, corresponding to five categories of memory ability:\nV2 Capability Meaning Static state recall Remembering static environmental information, such as page layouts, module functions, important locations Dynamic state tracking Tracking how states and actions change the environment over time Workflow knowledge Remembering the procedures for completing recurring tasks in specific environments Environment gotchas Recognizing recurring local failure patterns and environmental traps Premise awareness Knowing which assumptions valid in other environments do not hold in the current deployment V2\u0026rsquo;s context scale explodes:\nsmall tier: 25M tokens medium tier: 115M tokens This far exceeds mainstream context windows.\nThis represents arguably one of the biggest directional shifts in AI memory research over the past year.\n\u0026ldquo;How can LLMs remember long-term conversations\u0026rdquo; → \u0026ldquo;How can agents accumulate long-term work experience\u0026rdquo;\nOr put differently: chatbot memory → agentic / experiential memory.\nFrom a data synthesis pipeline perspective:\nLongMemEval is constructed by first designing the user memories to be tested, then embedding those memories into long chat histories. It roughly follows three steps:\nquestion curation: An ontology of 164 user attributes (e.g., lifestyle, belongings) is defined. An LLM generates user background paragraphs for each attribute, from which candidate QA pairs are generated (with human filtering, rewriting, and difficulty calibration). Annotators decompose answers into one or more evidence statements (some carrying temporal information). evidence session construction: Each evidence statement is embedded into a task-oriented user-assistant session, using self-chat to generate dialogue. The user does not explicitly state the evidence; instead, information is revealed indirectly through requests for help, advice, or task completion. history compilation: Relevant evidence sessions are inserted into a longer user-assistant chat history. Irrelevant sessions come from two sources: self-chat sessions synthesized from other non-conflicting user attributes, and ShareGPT + UltraChat. All sessions are assigned reasonable timestamps, producing a coherent, length-configurable long-term chat history. At its core, this remains needle-in-a-haystack for personal memory.\nBy contrast, LongMemEval-V2 is constructed by first collecting agent trajectories from task execution in environments, then extracting from those trajectories the knowledge an experienced agent ought to remember.\nIt roughly follows four steps:\ntrajectory collection: Web-agent trajectories are collected from WebArena, WorkArena, and WorkArena++ via the AgentLab unified interface. Each trajectory includes task goal, state observations, screenshots, accessibility tree, actions, and state transitions. Both successful and failed trajectories are collected through rejection sampling. question annotation: Questions are constructed around five memory ability categories: static state recall, dynamic state tracking, workflow knowledge, environment gotchas, and premise awareness. The model is tested on whether it can answer candidate questions without trajectory evidence — if it can, the questions are filtered or rewritten (preventing the model from answering via public knowledge alone). answer trajectory labeling: Each question requires labeling which trajectories genuinely contain answer evidence, with the requirement that evidence be verifiable in screenshots or trajectory states (a batch of seed answer-bearing trajectories is used with Codex to expand the candidate annotation pool, followed by human verification of ambiguous cases). haystack creation: The Small tier constructs one shared 100-trajectory haystack each for WebArena and ServiceNow; the Medium tier constructs roughly 500-trajectory question-specific haystacks for each question. EverMemBench 📄 EverMemBench Paper\nFrom dyadic conversations to multi-person collaboration. Can an agent embedded in long-term team collaboration maintain memory across multiple people, groups, and timelines?\nEverMemBench simulates enterprise collaboration scenarios. The data includes multiple projects, sub-projects, employees, group conversations, and evidence-grounded QA. The overall structure resembles:\nproject ├── sub-project / group chat │ ├── day 1 messages │ ├── day 2 messages │ └── ... ├── employees / personas ├── task timeline └── QA items with evidence references Evaluation pipeline:\nAdd → Search → Answer → Evaluate EverMemBench\u0026rsquo;s QA is organized across three dimensions and nine subtasks:\nDimension Subtasks Count What It Evaluates Fine-grained Recall Single-hop Retrieval, Multi-hop Trajectory, Temporal Duration 762 Whether precise facts, links, dates, people, task statuses can be found Memory Awareness Constraint, Proactivity, Update 1,097 Whether past rules, constraints, and updates can be applied to new scenarios Profile Understanding Style, Skill, Role 541 Whether long-term interaction enables understanding of personal style, skills, and roles EverMemBench\u0026rsquo;s evaluation approach can be divided into two layers: task protocol and scoring method.\nFor task protocol, it uses a streaming multi-group protocol: during the ingestion phase, the system receives daily batches in chronological order and must maintain its own memory state; during the evaluation phase, questions are posed. This protocol tests how the system writes, updates, retrieves, and uses memory while continuously receiving information, rather than dumping all history into context at once.\nFor scoring:\nQuestion Type Evaluation Method Multiple-choice Directly determine whether A/B/C/D is correct Open-ended LLM-as-a-Judge determines whether the generated answer and gold answer are semantically equivalent Memory system Add → Search → Answer → Evaluate Full-context LLM Feed the complete dialogue as context directly, without retrieval Data construction can be summarized as:\nblueprint and profile generation → chunk-wise dialogue synthesis → evidence-grounded QA construction User Profiling and Personalization PersonaMem 📄 PersonaMem Paper · GitHub Repository\nThe core question is not \u0026ldquo;can it recite a particular fact,\u0026rdquo; but: can the model form and update user profiles across long-range, multi-turn, multi-session user interaction histories, and use the most current user state to generate personalized responses?\nThree capabilities: whether the model can internalize users\u0026rsquo; intrinsic characteristics and preferences; whether it can track the temporal evolution of user profiles / preferences; and whether it can generate personalized responses in new scenarios accordingly.\nThe setup: given the user-LLM interaction history up to a certain point in time, plus a first-person in-situ user query, the model must select the response that best matches the current user state.\nIt tests four things:\nRemembering facts / preferences users have shared: e.g., a user once expressed a preference for a certain type of music, diet, or learning style.\nUpdating memories: Preferences change — the model must not cling to old preferences, e.g., shifting from liking pizza to going gluten-free due to a gluten allergy.\nRetrieving relevant fragments from history: Information may appear far back in long histories, with \u0026ldquo;lost in the middle\u0026rdquo; problems.\nApplying memories to responses, not merely parroting: The hardest part is giving new recommendations based on current preferences, or transferring to new scenarios. V1 found models perform reasonably on simple factual recall, but are significantly weaker at \u0026ldquo;giving new suggestions\u0026rdquo; and \u0026ldquo;applying user preferences across scenarios.\u0026rdquo;\nV1 data scale: roughly 20 personas, 180+ interaction histories, each history up to 60 multi-turn sessions, covering 15 personalized task scenarios; the HF dataset provides 32K, 128K, and 1M context-length versions.\nIt defines 7 types of in-situ user queries to cover different memory / personalization skills:\nType What It Tests Recall user-shared facts Remembering facts, activities, and interests the user has shared Suggest new ideas Recommending new things not present in history, while respecting user constraints Acknowledge latest user preferences Recognizing the user\u0026rsquo;s most recent preferences, not outdated ones Track full preference evolution Tracking the complete change in preferences from past to present Revisit reasons behind preference updates Remembering the reasons behind preference changes Provide preference-aligned recommendations Proactively offering suggestions consistent with current preferences Generalize to new scenarios Transferring user characteristics learned from other scenarios to new tasks V1\u0026rsquo;s primary evaluation is 4-way multiple-choice accuracy: the model sees the full conversation history, the current user query, and four candidate responses, and selects the response that best matches the current user profile.\nV2 improvements:\nV1 Issue / Limitation V2 Corresponding Change Small scale: V1 has only ~20 personas, 180+ histories, 15 task scenarios V2 scales to 1,000 personas, 26,100 preferences / Q\u0026amp;A pairs, 325 conversation topics, 335 user query topics. Preferences relatively explicit: V2 paper notes V1 still focused mainly on explicit preferences V2 focuses on implicit personas / implicit preferences, embedding preferences within realistic interactions like email rewriting, translation, knowledge queries, and photo sharing. Evaluation biased toward multiple-choice: V1\u0026rsquo;s primary evaluation is MCQ; generative evaluation was constrained by logit availability V2 provides both MCQ and open-ended evaluation; open-ended uses LLM-as-a-judge; the paper reports aggregating judgments from three GPT-5-Chat judges. No train/validation split for model improvement V2 is not just a benchmark — it provides train/val/test splits: text benchmark 5,000, train 18,500, val 2,600; multimodal benchmark 5,000, train 19,200, val 2,110. Insufficient privacy/control scenarios for real-world use V2 adds scenarios where the user leaks sensitive information or explicitly asks \u0026ldquo;don\u0026rsquo;t remember this preference,\u0026rdquo; and requires target answers to avoid using sensitive/private information. More like a long-context test, high cost V2 proposes/evaluates agentic memory: compressing long histories into roughly 2K-token, readable, updatable memory; the paper claims roughly 16× token savings relative to 32K context, and surpasses direct long-context models on both MCQ and open-ended. User preferences easily confused with others\u0026rsquo; information V2 data columns include an explicit who label, distinguishing whether the current preference belongs to the user or someone else, and adds unrelated math/code questions as distractors. Single modality / limited scenario coverage V2 has both text and multimodal benchmark suites, and includes multimodal photo sharing, multilingual translation, health/medical, therapy, and other scenarios. V2\u0026rsquo;s positioning reflects a clear pivot: from \u0026ldquo;explicit/dynamic preference memory\u0026rdquo; to \u0026ldquo;implicit preference inference + trainable agentic memory.\u0026rdquo;\nV2 particularly focuses on cases where user preferences are not stated directly but are embedded in long dialogues — for instance, when a user asks the model to polish an email, the email content may inadvertently reveal seasonal allergies.\nEach benchmark row in V2 includes: persona_id, path to the full persona file, links to 32K/128K chat history, user_query, correct_answer, three incorrect_answers, current preference, preference category, relevant dialogue snippets, whether the preference belongs to the user, whether it has been updated, old preference, sensitive information, and token distance from relevant snippet to query.\n🤗 PersonaMem-v2 Dataset\nV2\u0026rsquo;s primary evaluation metrics:\nMCQ accuracy: The model selects the most personalized answer from four plausible options. Open-ended personalization accuracy / judge score: The model generates a free-form response, then LLM-as-a-judge determines whether it reflects the ground-truth preference; the paper reports aggregation from multiple GPT-5-Chat judges. Grouped analysis: Performance is broken down by preference type, scenario, whether the preference was dynamically updated, who, distance to relevant snippet, and other dimensions. The official evaluation scripts also generate these category breakdowns. PersonaLens Where PersonaMem treats personalization as a memory QA problem, PersonaLens is more concerned with whether personalization gets squeezed out by task objectives during task completion.\n📄 PersonaLens Paper\nIn terms of composition, it has 1,500 user profiles (demographic attributes drawn from the PRISM Alignment dataset, with preferences generated by an LLM based on demographics to ensure internal consistency), covering roughly 20 domains from Books, Finance, Flights to Restaurants.\nEvaluation methodology: dual-agent automatic evaluation — a user agent (implemented with Claude 3 Sonnet) simulates a user with a specific profile and context conversing with the assistant under test; a judge agent uses LLM-as-a-Judge to rate personalization, response quality, and task completion, distinguishing between single-domain (TSD) and multi-domain (TMD) tasks.\nA key finding: there exists a trade-off between task completion rate (TCR) and personalization — the more the assistant buries its head in getting the task done, the more likely it is to ignore user preferences.\nBy contrast, static QA has no \u0026ldquo;task objective\u0026rdquo; competing with personalization for attention.\nMultimodal Personal Memory Memory originates as text. But real personal memory is clearly more than that.\nCan an agent build long-term memory across text, images, and video, and then invoke those memories in downstream tasks?\nIn much of what is called multimodal memory, images are merely captioned into text and fed back into the text pipeline. LoCoMo itself includes a multimodal component (image sharing), but the vast majority of subsequent work has reduced it to pure text QA.\nA useful way to think about this is to ask: at which stage does multimodality actually play a role?\nInput / storage side: preserving visual information across sessions.\nA user shares an image in an earlier session; the system must remember its content later.\n📄 Mem-Gallery Paper\nMem-Gallery explicitly points out that existing benchmarks either evaluate memory in purely textual multi-session settings, or evaluate multimodal understanding in localized single-session contexts — none genuinely assess how multimodal memory is preserved, organized, and evolved over long dialogue trajectories. It evaluates along three dimensions: memory extraction \u0026amp; test-time adaptation, memory reasoning, and memory knowledge management.\nRetrieval side: cross-modal retrieval within visual history. Storage is only the first step; whether the right image / frame can be found based on the current need is another matter entirely.\nLayered with personalization: multimodal × user modeling.\nPersona-MME targets long-term personalized multimodal LLMs by constructing persistent multimodal persona contexts (user profile, frequently occurring social relationships, objects the user owns) and then generating three types of multimodal dialogue — explicit entities, implicit facts, and distractors — to control the memory signals introduced.\nLayered with real / embodied scenarios. This falls within the embodied / world model direction: egocentric wearable assistants, robot memory, video world model memory capabilities, etc. (Beyond the scope of this article.)\nOn the positive side, Mem-Gallery, MemLens, and similar work from early 2026 represent a correction to the \u0026ldquo;images get captioned away\u0026rdquo; problem.\nBut this category as a whole is still very young. It has not yet converged on a universally-adopted benchmark like LoCoMo; modality coverage, task formats, and evaluation standards remain inconsistent across efforts.\nMemory Lifecycle Capabilities The preceding sections were organized by \u0026ldquo;what content is being tested\u0026rdquo;: conversational facts, user profiles, multimodal.\nBut can the agent update old memories? Handle conflicting information? Know when to forget or abstain? Track how preferences change over time? These are the lifecycle capabilities of memory.\nThe lifecycle landscape is fragmented — a collection of specialized benchmarks each targeting a single capability, most emerging only from late 2025 through 2026. This is itself a signal: the field is only just realizing that accuracy alone is not enough. Specialized evaluations are beginning to appear but have not yet converged.\nBelow, organized by capability dimension, with representative benchmarks and the metrics — distinct from accuracy — that each introduces.\nUpdate\nA memory system must remember the most recent version, not the first thing that was said.\nLongMemEval already includes a dedicated knowledge-update subset (described earlier). Going further, some work has turned this into a metric rather than just a data split: Forgetting-Aware Memory Accuracy specifically penalizes reliance on stale or invalidated memories.\nConflict Resolution\nWhen two contradictory pieces of information coexist in memory, can the system ensure only the latest, correct one takes effect?\nMemoryAgentBench addresses this: decomposing memory capability into accurate retrieval, test-time learning, long-range comprehension, and conflict resolution, with a dedicated Conflict Resolution Accuracy metric ensuring that when contradictions exist, only the most recent valid fact is used.\nForgetting / Selective Retention\nNot all information should be kept forever: expired information and anything the user explicitly asks \u0026ldquo;don\u0026rsquo;t remember this\u0026rdquo; should be discarded.\nMemoryAgentBench uses its newly constructed FactConsolidation dataset to specifically test selective forgetting.\nUser-initiated forgetting is also tied to privacy, as in PersonaMem-v2\u0026rsquo;s privacy scenarios (user requests not to remember a certain preference).\nAbstention\nIf you don\u0026rsquo;t know, say you don\u0026rsquo;t know. A memory system must recognize \u0026ldquo;insufficient evidence\u0026rdquo; and abstain, rather than hallucinate.\nLongMemEval specifically includes abstention-type questions to test this, with corresponding metrics of abstention rate / abstention accuracy — again, not captured by standard accuracy.\nFrom Passive Recall to Memory-Driven Action\nMemoryArena shows that models scoring near-perfect on recall benchmarks plummet in performance the moment memory must drive action. Many so-called memory benchmarks are essentially retrieval benchmarks, testing \u0026ldquo;can the needle be found\u0026rdquo; rather than \u0026ldquo;can memory actually be put to use.\u0026rdquo;\nEvaluation Metrics: Beyond Accuracy If we only look at accuracy, what gets missed?\nBorrowing an existing layered framework, a recent memory survey proposed a four-layer evaluation stack that serves as a useful skeleton for this section.\nLayer 1: Task Effectiveness\nDid the task actually get done?\nCommon metrics include success rate, factual correctness, plan completion rate. In the QA format typical of conversational memory benchmarks, these are the familiar: token-level F1, BLEU, binary/rating judgments from LLM-as-a-Judge, and MCQ accuracy.\nThese metrics are not equivalent to one another — which one you choose changes the conclusion. Even though LLM-as-a-Judge is closer to semantic correctness, it introduces the judge\u0026rsquo;s own biases.\nAt this point, \u0026ldquo;who is SOTA\u0026rdquo; depends on which metric you trust.\nLayer 2: Memory Quality\nKey metrics include: precision / recall of retrieved records, contradiction rate, staleness distribution, and coverage of task-relevant facts.\nSome work on LongMemEval-S has achieved nearly 99% retrieval recall for certain question types using retrieval alone — but that\u0026rsquo;s retrieval recall, not end-to-end accuracy.\nStaleness distribution and contradiction rate focus on \u0026ldquo;how much of the stored memory has expired or is internally contradictory.\u0026rdquo;\nCorrespondingly, specialized metrics have emerged, such as Forgetting-Aware Memory Accuracy (penalizing reliance on stale memories) and conflict resolution accuracy (ensuring only the latest version of contradictory information takes effect).\nLayer 3: Efficiency\nCore metrics: latency per memory operation, prompt tokens consumed by memory content, retrieval call count per step, and storage growth over time.\nA methodological detail worth noting: rigorous work (e.g., HiMem) reports latency by measuring only the time spent on memory retrieval itself, excluding LLM inference, to isolate memory system efficiency and avoid contamination by the speed of the underlying model. When reading efficiency numbers, pay attention to which segment is actually being measured.\nLayer 4: Governance\nDoes the memory system comply with privacy and control requirements?\nMetrics include: privacy leakage rate, deletion compliance (when the user says \u0026ldquo;don\u0026rsquo;t remember / delete this,\u0026rdquo; does the system actually comply), and access-scope violations (unauthorized access, especially in multi-user shared memory scenarios).\nPersonaMem-v2 already incorporates such scenarios: users leaking sensitive information, or explicitly requesting \u0026ldquo;don\u0026rsquo;t remember this preference,\u0026rdquo; with target answers required to avoid using such sensitive / private information.\nThese four layers are often in tension with one another.\nA thorough system evaluation should therefore present an operating curve across the four layers (a tunable trade-off among accuracy / store-size / latency), rather than a single isolated best score.\nData Synthesis Methods Before discussing any memory benchmark, a more fundamental question should be asked: how were these multi-turn dialogues constructed? Because the synthesis method almost entirely determines what a benchmark actually tests — and what it is inherently incapable of testing.\nCurrent mainstream benchmark dialogue sources fall roughly into three tiers:\nPurely human-authored real data (virtually nonexistent) LLM-generated skeleton + human verification (exemplified by LoCoMo: based on predefined personas and temporal event graphs, dialogue skeletons are generated using an LLM agent architecture, then human annotators verify and edit for long-range consistency) Needle-in-a-haystack assembly (exemplified by LongMemEval: questions written by human experts, evidence-bearing sessions simulated by LLMs and manually edited, while the vast majority of irrelevant sessions are sampled from ShareGPT, UltraChat, and simulated dialogues) Personalization-oriented benchmarks rely more heavily on persona-driven fully-automated synthesis. PersonaMem uses a GPT-4o-driven modular pipeline to generate multi-session, temporally-structured interaction histories from personas, embedding preferences that evolve over time within the dialogues; its in-situ query appends the user\u0026rsquo;s current first-person question to the end of the history, testing whether \u0026ldquo;the model can answer using the most recent preferences.\u0026rdquo;\nSince needle-in-a-haystack is fundamentally a needle-finding test, what it predominantly measures is retrieval capability within long contexts, rather than the organizational and evolutionary capabilities of a memory system.\nWhen LongMemEval selects haystack sessions, it deliberately chooses sessions similar in topic and format to the evidence sessions, but without introducing conflicting information, so as not to invalidate the questions. This engineering-sound choice means that this benchmark, from its construction starting point, deliberately avoids knowledge conflicts — and is therefore inherently incapable of testing conflict resolution.\nAs for the common concern about \u0026ldquo;whether LLM-generated data is trustworthy\u0026rdquo;: the answer lies not in LLM hallucination per se, but in the filtering and verification after generation. PersonaMem-v2, for example, applies four hard filters to every Q\u0026amp;A pair — the question must be unanswerable without the history (preventing data leakage and generation artifacts), the correct option must genuinely reflect user preferences, incorrect options must never reflect them, and the format must not retain generation traces. Ultimately, only roughly 30% of generated Q\u0026amp;A pairs pass all filters.\nThe pass rate itself reveals that, without rigorous filtering, a significant proportion of synthetic dialogues contain shortcuts exploitable by models.\nThus, the synthesis pipeline itself defines the boundaries of what a benchmark can test.\nClosing No single benchmark can cover everything that agent-memory entails. Every facet — factual recall, personalized inference, multimodality, update, conflict, forgetting, abstention, memory-driven action — requires purpose-built data and purpose-designed metrics.\nSo when an agent-memory project claims yet another SOTA, the set of questions worth asking remains the same: SOTA on which benchmark? What core capability does that benchmark measure, and what properties does it miss? What does its data construction allow it to test, and what is it structurally unable to test? Which layers of metrics does it report, and are the missing layers precisely the ones it sacrificed?\n","permalink":"https://niraya666.github.io/en/posts/agent-memory-benchmark-summary/","summary":"A comprehensive survey of agent-memory benchmarks: from LoCoMo and LongMemEval to PersonaMem, covering evaluation methodology, metrics, and data synthesis pipelines across conversational memory, user profiling, and multimodal memory.","title":"Beyond SOTA: What Agent-Memory Benchmarks Actually Measure — and What They Miss"},{"content":"Prologue: From an Old Question to a New One I remember when I first started out in this industry, I followed a question on Zhihu:\n\u0026ldquo;The internet has been fought to a bloody red ocean, yet smart manufacturing remains a wasteland — why won\u0026rsquo;t BAT (Baidu, Alibaba, Tencent) step into it?\u0026rdquo;\nOver five years later, this question still has no definitive answer — but the times have changed.\nBack then, the subtext was: technology is scarce, industries are waiting to be transformed, and as long as the top-tier players are willing to enter the fray, things will happen. Reality has since falsified that assumption in nearly every way imaginable. And now, LLMs have pushed the \u0026ldquo;technology is scarce\u0026rdquo; premise even further. But ask anyone pushing AI adoption in a traditional industry, and odds are they\u0026rsquo;ll tell you: it\u0026rsquo;s not that simple.\nAnd so this old question takes on a new form in the AI era:\n\u0026ldquo;AI is already this powerful — why can\u0026rsquo;t these industries get moving?\u0026rdquo;\nRecently, the term FDE (Forward Deployed Engineer) has suddenly caught fire among major AI companies — and at its core, it\u0026rsquo;s a response to this very question: if the models are already powerful enough, then the \u0026ldquo;last mile\u0026rdquo; bottleneck must lie in the customer site, in the data, in the process, in the organization.\nThis article is my attempt to answer this old question, drawing on years of observation and hands-on practice.\nLet me put the conclusion up front:\nWhat most industries are missing right now isn\u0026rsquo;t AI — it\u0026rsquo;s X-Ops.\nIn other words, the question has never been \u0026ldquo;should we adopt AI?\u0026rdquo; — it\u0026rsquo;s whether this industry, this company, this production line, this workflow has been transformed into a system that AI can plug into, understand, participate in, and feed back into.\nWhy Has the Software Industry Been Hit Hardest by AI? It\u0026rsquo;s not AI. It\u0026rsquo;s DevOps.\nIf you look at the fields where AI has truly \u0026ldquo;broken in\u0026rdquo; over the past two years, the software industry itself appears to have taken the biggest hit. The reason the software industry has been hit hardest by AI isn\u0026rsquo;t that its practitioners are the smartest, nor that models happen to favor code — it\u0026rsquo;s that over the past two decades, it inadvertently transformed itself into the industry that\u0026rsquo;s easiest for AI to plug into.\nAI-Friendliness Wasn\u0026rsquo;t Born This Way If I had to list the traits that put AI in its element within the software industry, here\u0026rsquo;s what I\u0026rsquo;d pick:\nOpen corpus, easy to access: GitHub\nShort feedback cycles: Write code, compile, run tests, see results — feedback arrives in seconds or minutes\nAI-friendly context: Code has comments, modules have interfaces, changes have diffs — nearly everything is textual and structured\nVerifiable correctness: Unit tests, integration tests, CI pipelines provide an automatic grading mechanism for whatever the AI writes\nThese four pieces together set the iteration flywheel in motion: generate → verify → get feedback → fix → regenerate.\nBut go back to around the year 2000, and the software industry actually looked a lot more like today\u0026rsquo;s traditional industries. Release cycles were measured in years. A single version would be polished for months and shipped on a CD. What truly changed everything was the transformation of the delivery model brought by the internet. Software no longer went through \u0026ldquo;manufacture → copy → ship → install\u0026rdquo; — instead it became \u0026ldquo;develop → release → get feedback → update → re-release.\u0026rdquo; Once this closed loop took shape, the entire way the industry organized itself was forced to restructure:\nBecause you needed to release frequently, you needed version control (Git)\nBecause you needed to integrate frequently, you needed CI/CD\nBecause you needed to collaborate across teams, you needed code review, issue tracking, API documentation\nBecause things could go wrong in production at any moment, you needed logging, monitoring, alerting, observability\nBecause releasing carried high risk, you needed gradual rollout, rollback, A/B testing\nBecause collaboration kept scaling up, you needed Agile, DevOps, SRE (Site Reliability Engineering)\nThis entire apparatus is what we today collectively call DevOps. But the essence of DevOps isn\u0026rsquo;t a particular combination of tools — it\u0026rsquo;s an engineering culture that makes the entire process of \u0026ldquo;how software is produced, delivered, operated, and improved\u0026rdquo; explicit, versioned, automated, and observable.\nThe \u0026ldquo;Byproduct\u0026rdquo; DevOps Left Behind Became AI\u0026rsquo;s Food Here\u0026rsquo;s the most interesting part: DevOps was never designed for AI. It solved problems of human-to-human collaboration, the balance between quality and speed, the engineering challenge of delivering at scale.\nBut its byproducts almost perfectly hit every piece of infrastructure the AI era demands:\nWhat DevOps left behind What it means for AI Git repos + commit history Traceable evolution history and training data Issues / PRs / Code Review Labeled human decision data Unit tests / CI An automatic grader — AI changes get verified instantly README / API docs / Design Docs Structured context Logs / Traces / Metrics Runtime feedback signals Canary deployment / Rollback A safe trial-and-error mechanism In other words, the software industry spent twenty years turning its production process into a system that is extraordinarily friendly to any external agent — whether that agent is a new hire, an outsourced team, or an AI.\nWhen the coding agent arrived, it didn\u0026rsquo;t need the software industry to do any extra adapting. It simply plugged into a collaboration infrastructure that had been ready and waiting for it all along.\nKnowledge Accumulation Got \u0026ldquo;Embedded\u0026rdquo; Into the Production Process This is the single thing the software industry has done that other industries should learn from most.\nIn many industries, \u0026ldquo;knowledge management\u0026rdquo; is something that requires its own project approval, its own budget, its own performance review — someone has to write documentation, maintain the wiki, run training sessions. But because it has no direct link to business output, it\u0026rsquo;s always the lowest priority, and it never gets done well.\nIn the software industry, by contrast, knowledge accumulation isn\u0026rsquo;t an \u0026ldquo;extra thing\u0026rdquo; — it\u0026rsquo;s a byproduct of the production process.\nAn engineer doesn\u0026rsquo;t need to make any \u0026ldquo;extra effort\u0026rdquo; to do knowledge management. As long as he follows the established practices, knowledge gets automatically captured in a structured form.\nFlip the lens to most traditional industries: knowledge lives in people\u0026rsquo;s heads, in some sheet of some Excel file, in WeChat chat histories. It\u0026rsquo;s not that knowledge isn\u0026rsquo;t being produced — it\u0026rsquo;s that it\u0026rsquo;s scattered everywhere, exists in all manner of forms, hard to find and hard to read.\nThe other side of this coin: the reason AI still can\u0026rsquo;t break into other industries isn\u0026rsquo;t that the models aren\u0026rsquo;t strong enough — it\u0026rsquo;s that those industries haven\u0026rsquo;t yet prepared themselves to onboard any \u0026ldquo;external agent\u0026rdquo; — whether that agent is an AI, or a new employee.\nWhy Can\u0026rsquo;t Most Industries Absorb AI? A world without a source of truth.\nDirty, Dirty, Dirty, Dirty Anyone who\u0026rsquo;s done one or two AI deployment projects in a traditional industry will know these scenes all too well:\nDirty data: numbers don\u0026rsquo;t match across systems, systems don\u0026rsquo;t match reality, and nobody knows what a given data field actually means.\nDirty processes: the SOP (Standard Operating Procedure) and manuals are written in meticulous detail, but nothing actually runs the way the flowchart says it does.\nDirty systems: a dozen-plus systems spanning different eras and different vendors — interfaces either don\u0026rsquo;t exist, or they\u0026rsquo;re SOAP, or the workflow is \u0026ldquo;export to Excel and import it again.\u0026rdquo;\nDirty knowledge: the core know-how lives inside a veteran\u0026rsquo;s head, in WeChat chat logs, in some unmaintained Excel file nobody owns.\nStack all four kinds of dirty on top of each other, and for AI to actually deliver value becomes nearly impossible.\nBut the deeper problem isn\u0026rsquo;t that \u0026ldquo;things are dirty right now\u0026rdquo; — it\u0026rsquo;s that over the past decade-plus, digital transformation didn\u0026rsquo;t actually solve this. In many places, it made it worse.\nMost Turned Into Dashboards Over the past decade, nearly every halfway-respectable industrial enterprise has done \u0026ldquo;digital transformation,\u0026rdquo; \u0026ldquo;Industrial Internet,\u0026rdquo; \u0026ldquo;smart factory,\u0026rdquo; \u0026ldquo;centralized data platform.\u0026rdquo; But if you look at what these projects actually delivered, the vast majority produced the same thing: a big screen, a few dashboards, a handful of reports.\nThe various systems, each abbreviated to an English acronym, all sounded right. The project proposals wrote up smoothly. The acceptance demos all ran fine. But they share one common trait: they don\u0026rsquo;t change how the enterprise operates — they merely \u0026ldquo;display\u0026rdquo; the results of how the enterprise already operates.\nAnd so they all ended up as big-screen dashboards for the boss. Display is not the same as change. It became a kind of digital bureaucracy.\nTools, Processes, Organization Let me try to break down the things in an enterprise that \u0026ldquo;can be digitized / made intelligent\u0026rdquo; into three layers.\nTool Layer\nWhat this layer does, fundamentally, is: take things where \u0026ldquo;people already know what to do\u0026rdquo; and do them faster, more reliably, cheaper.\nThe vast majority of \u0026ldquo;AI projects\u0026rdquo; and \u0026ldquo;digital transformation projects\u0026rdquo; operate at this layer. The reason is simple — they\u0026rsquo;re easy to get approved, easy to sign off, easy to tell a story about. (I also call this peripheral intelligence.)\nProcess Layer\nTake the links that used to depend on human coordination, meeting-driven decisions, and experience-based judgment, and turn them into system-driven chains with feedback and room for optimization.\nThe value here is far higher than at the Tool Layer, but it demands data that flows, systems that connect, processes that are explicit, and accountability that is clear. This is where the vast majority of enterprises get stuck — because it requires you to clean up the \u0026ldquo;four kinds of dirty\u0026rdquo; first.\nOrganization Layer\nThe enterprise itself, as a complex system: how it is described, how it is run, how it is improved.\nPeople can handle local problems by experience, but they struggle to manage a high-dimensional, dynamic, multi-objective, tightly-constrained system in real time. What the third layer aims to do is turn this complex system itself into something that can be modeled, monitored, and tuned.\nThis is the layer where AI-Native should truly come into play. But it has an awkward characteristic: it\u0026rsquo;s a hard sell.\nIf you accept this three-layer framework, one thing becomes clear: what makes AI transformation genuinely hard isn\u0026rsquo;t the models — it\u0026rsquo;s the dirty work.\nAnd here\u0026rsquo;s the problem: nobody wanted to do this dirty work over the past few decades, and it\u0026rsquo;s hard to imagine anyone wanting to do it in the future. Because it doesn\u0026rsquo;t win glory, doesn\u0026rsquo;t earn credit, doesn\u0026rsquo;t make slides, doesn\u0026rsquo;t advance careers.\nGenuine and Pseudo Tacit Knowledge There\u0026rsquo;s a widely-circulated saying: \u0026ldquo;A lot of what we do in this industry is tacit knowledge — AI can\u0026rsquo;t learn it.\u0026rdquo;\nThat\u0026rsquo;s half right, and half wrong.\nGenuine tacit knowledge absolutely exists. But pseudo tacit knowledge is far more common:\nThings that could have been written down, but nobody wrote them. Things that could have been structured, but the organization doesn\u0026rsquo;t reward it. Things that could have been shared, but sharing them means losing personal leverage. A huge amount of what gets called \u0026ldquo;experience\u0026rdquo; in vertical industries actually falls into the latter category. Treating pseudo tacit knowledge as genuine tacit knowledge and enshrining it has two results: first, the cost of bringing new people into the field is enormous — the only path is \u0026ldquo;follow people, watch people, put in the years\u0026rdquo;; second, AI deployment hits a wall, because what AI feeds on most is accessible, structured, verifiable context.\nAnd the reason the software industry lets newcomers ramp up fast and AI plug in quickly is precisely that it has institutionalized the practice of making explicit what can be made explicit — if you don\u0026rsquo;t make it explicit, your PR won\u0026rsquo;t pass, CI won\u0026rsquo;t run, code review won\u0026rsquo;t approve. The process forces you to spit your knowledge out.\nBehind this are two fundamentally different organizational forms.\nThe internet and open-source communities are organized around products, systems, collaboration, and reusability — they form a product organization.\nMany traditional industries, by contrast, are expert organizations.\nDimension Product Organization Expert Organization Code / work output Code must be readable by others Whoever understands it calls the shots Interfaces / usage Interfaces must be usable by others Whoever has seniority makes the call Role of documentation Documentation must be readable by newcomers Documentation serves audits, not collaboration Problem handling Bugs must be reproducible by others Whoever stepped on the landmine first has authority Module design Modules must be replaceable Whoever controls the equipment, data, and process has the leverage Knowledge flow Knowledge must flow across teams Newcomers learn by following people, not systems Newcomer growth Depends on documentation, processes, systems, and community accumulation Depends on masters, mentors, and experience transmission Source of authority Comes from product quality, system capability, and collaboration efficiency Comes from experience, seniority, resource control, and historical accumulation An expert organization feels \u0026ldquo;safer\u0026rdquo; on the surface — because critical capability is locked inside a few people and won\u0026rsquo;t spiral out of control. But it has one fatal side effect: it is a system closed to outsiders (whether those outsiders are newcomers or agents).\nWhich leads to an awkward situation: AI can\u0026rsquo;t get in, because newcomers can\u0026rsquo;t get in either.\nSo, Where Is the Bottleneck? Back to the opening question: why can\u0026rsquo;t most industries absorb AI?\nBecause over the past few decades, they never built an internal system that is \u0026ldquo;friendly to outsiders.\u0026rdquo; This isn\u0026rsquo;t a problem of \u0026ldquo;AI can\u0026rsquo;t get in\u0026rdquo; — it\u0026rsquo;s a problem of \u0026ldquo;no systematic force can get in.\u0026rdquo; AI has simply shone a fresh light on it.\nAnd the real irony is — the way many industries are investing in AI right now is still piling more \u0026ldquo;agent\u0026rdquo;-branded dashboards onto the first layer (the Tool Layer), rather than paying down the debt they\u0026rsquo;ve accumulated on the second and third layers.\nThe ROI Trap Most companies, when crunching the numbers on AI, are using an algorithm from a previous era. That algorithm determines that they can see at most a small sliver of AI\u0026rsquo;s value — and it also dooms their AI transformation to remain stuck at the level of \u0026ldquo;a cheaper person.\u0026rdquo;\nThree Layers of ROI Here I draw on the framework from The Three Layers of ROI for AI Agents.\nLayer 1 ROI: how many people AI replaced, how much time it saved, how much cost it cut.\nThis is the AI value that the vast majority of companies grasp most easily. Its strength is that it\u0026rsquo;s intuitive; its weakness is that it has a low ceiling.\nBecause at its core, it\u0026rsquo;s about making old processes cheaper and faster — it doesn\u0026rsquo;t change how the company operates. The role AI plays here is simply that of a \u0026ldquo;cheaper person.\u0026rdquo;\nLayer 2 ROI: AI enables people to do things they previously couldn\u0026rsquo;t do, or couldn\u0026rsquo;t do well.\nThe value is far higher than Layer 1, but the barrier to entry is also far higher. It requires companies to be willing to spend on \u0026ldquo;invisible infrastructure.\u0026rdquo; The vast majority of companies get stuck at this layer. Not because they can\u0026rsquo;t do the math, but because this investment demands that the CEO have 3-5 years of patience, while most managers operate with a 6-12 month horizon.\nLayer 3 ROI: the agent is part of the organization.\nIts core question isn\u0026rsquo;t \u0026ldquo;what can AI do for me?\u0026rdquo;, but: what permissions does the agent hold within the organization? At this point, it is no longer just a \u0026ldquo;tool\u0026rdquo; — it becomes a new kind of actor in how the organization runs, and the company\u0026rsquo;s entire way of working must be redesigned around it.\nThis is what AI-Native really means. Not \u0026ldquo;we use AI,\u0026rdquo; but \u0026ldquo;the organization has redesigned how it works around AI.\u0026rdquo; The returns are non-linear, compounding, and long-term.\nWhy Does the Money Always Flow to Layer 1? It\u0026rsquo;s not that companies can\u0026rsquo;t see the value of Layer 2 and Layer 3 — it\u0026rsquo;s that their decision-making mechanisms can only write checks for Layer 1.\nThis is the most awkward part of AI transformation. The reason Layer 2 and Layer 3 ROI is so high is precisely because they are built on \u0026ldquo;paying down debt\u0026rdquo; — cleaning up dirty data, untangling broken processes, capturing scattered knowledge, and connecting isolated systems.\nAnd this \u0026ldquo;debt\u0026rdquo; is what every manager over the past several decades racked up in pursuit of their own short-term KPIs.\nSo in the end, everyone settles on a kind of low-level consensus:\nBuild an AI system that doesn\u0026rsquo;t change power structures, doesn\u0026rsquo;t change business processes, doesn\u0026rsquo;t challenge existing KPIs — one that can be signed off on, can be demoed, can be written up in reports.\nIt looks like the company \u0026ldquo;uses AI,\u0026rdquo; but nothing about how the company actually operates has changed in any meaningful way.\nAnd so, the vast majority of companies only know how to calculate Layer 1 ROI. They will naturally use AI as a cheaper person. But there\u0026rsquo;s a fatal problem here — when everyone uses AI as a cheaper person, the market will rapidly erase that value. Advantages from cost reduction are short-term, replicable, and something competitors can catch up to quickly.\nThe Essence of AI Transformation Isn\u0026rsquo;t Technical A somewhat uncomfortable conclusion:\nThe deeper AI transformation goes, the less it becomes a technical problem and the more it becomes a problem of authority, process, and organization.\nThe Philosophy of X-Ops The essence of turning something into Ops is transforming a complex system into a control system that can receive feedback, self-regulate, and learn.\nBy this point, the first three sections have laid out the problem with considerable clarity:\nThe software industry was hit hardest by AI, not because it understands AI best, but because it spent the last two decades building DevOps\nMost traditional industries cannot absorb AI, not because AI isn\u0026rsquo;t powerful enough, but because they have no source of truth\nMost enterprises\u0026rsquo; AI transformations stall at the first layer, not because the ROI math doesn\u0026rsquo;t add up, but because their decision-making apparatus can only fund what produces visible ROI\nIf an industry genuinely wants to absorb AI, what it needs isn\u0026rsquo;t \u0026ldquo;AI\u0026rdquo; — it\u0026rsquo;s its own flavor of Ops. I call this X-Ops, where X can be any industry that wants to be genuinely transformed by AI.\nBut X-Ops is not simply \u0026ldquo;copy DevOps into another industry.\u0026rdquo; It\u0026rsquo;s a deeper way of thinking — about how an organization turns itself into a system that can be continuously observed, continuously fed back into, and continuously improved.\nA Control-Theoretic Lens: The Organization as a Feedback System The core idea of control theory is remarkably simple: for any system to operate continuously in an uncertain environment and improve itself, it must have a closed loop — sense, decide, act, receive feedback, then return to sensing.\nDevOps, in fact, closely resembles a control system.\nWhen you break DevOps apart, it maps almost directly onto a control-theoretic structure:\nControl Theory Concept DevOps Counterpart Controlled system Production software systems, infrastructure, user experience Target state SLO (Service Level Objective), business metrics, reliability goals, delivery targets Sensor Logs, metrics, tracing, monitoring, alerting, user feedback Controller CI/CD pipelines, release strategies, platform rules, team decisions Actuator Deployment, rollback, scaling, rate limiting, circuit breaking, feature flags Disturbance Traffic spikes, code defects, dependency failures, user behavior changes Feedback loop Monitor → assess → adjust → observe again Learning mechanism Incident postmortems, continuous improvement, automated test hardening So DevOps is not merely \u0026ldquo;automated deployment.\u0026rdquo;\nThe problem with the traditional development model is that feedback is too slow: requirements go live months later, incidents are only discovered after they happen, and only after ops takes over does development realize the code was never operable.\nDevOps fixes this by shortening the feedback chain:\nCode commit → automated testing → automated deployment → production observability → user feedback → rapid correction.\nThis is remarkably similar to negative feedback in control theory: detect the deviation, correct it, and bring the system back within the target range.\nAt the same time, \u0026ldquo;stability\u0026rdquo; in DevOps is not stillness — it is dynamic stability. This point is crucial. Traditional management likes to pursue \u0026ldquo;no change,\u0026rdquo; believing that fewer changes bring stability. The DevOps logic is closer to control theory: stability isn\u0026rsquo;t the absence of change, but remaining controllable amidst continuous change. Or, put another way:\nSmall-step changes, rapid detection, rapid rollback, rapid recovery, continuous learning.\nNow look at the organizational level. The vast majority of companies are not a closed-loop system. They are more like an open-loop pipeline: leadership makes a call → orders are issued → execution happens → … and then nothing.\nAnd AI\u0026rsquo;s real value is built precisely on top of closed loops. The working style that suits models best is small-step iteration: propose a change, run it, see the results, correct, and go again. If an organization itself lacks a mechanism for \u0026ldquo;runnable, verifiable, rollback-able,\u0026rdquo; then AI can\u0026rsquo;t find its place — it can\u0026rsquo;t know whether it got things right, and it can\u0026rsquo;t know where to go next.\nSo the first meaning of X-Ops is this: transform the organization from an open-loop system into a closed-loop system.\nThe Three Foundations Control theory provides an abstract lens. To make it concrete, I believe there are three foundations.\nThe Fact Layer\nDoes the system have visibility? A source of truth — this doesn\u0026rsquo;t mean all data must be piped into a data warehouse, nor that all knowledge must become a knowledge graph. But certain things must be established as \u0026ldquo;the source of fact.\u0026rdquo; You can describe it in terms of \u0026ldquo;data lineage\u0026rdquo; or \u0026ldquo;ontology,\u0026rdquo; but the core purpose is the same: solving the problem of AI-ready organizational context. This layer addresses what AI ought to trust.\nThe Change Record\nDoes the system have adjustability? AI has no hands. The physical world and the digital world are inherently separated. To prevent the gap between the two from growing ever wider, critical changes must be traceable. This can be loosely compared to Git, but the point isn\u0026rsquo;t to slap on \u0026ldquo;Git\u0026rdquo; — it\u0026rsquo;s that the act of change itself must be institutionalized. Who changed what? Why? What is the blast radius? This layer addresses whether AI can understand the organization\u0026rsquo;s evolution.\nThe Evaluation \u0026amp; Feedback System\nDoes the system have learnability? For the vast majority of deep learning or AI deployment projects, I\u0026rsquo;ve consistently insisted on evaluation first: build a test set → define evaluation metrics → design the evaluation methodology → evaluate → optimize → evaluate again. This approach is a universal methodology from data science, originally applied to complex system optimization problems. It generalizes far beyond its original domain.\nBuilding Short Feedback Loops Some will say: \u0026ldquo;In our industry, safety requirements are too strict, quality demands too high, customer expectations too unforgiving — we can\u0026rsquo;t iterate fast the way the internet does.\u0026rdquo;\nBut the higher-risk the system, the stronger its version management should be, the more rigorous its testing, the more comprehensive its automated verification, the more meticulous its change records. In other words, the harder it is to \u0026ldquo;iterate fast,\u0026rdquo; the more an industry needs Ops — not for speed, but for control.\nThe essence of DevOps has never been \u0026ldquo;fast.\u0026rdquo; It\u0026rsquo;s \u0026ldquo;still controllable while being fast.\u0026rdquo; Build your own feedback cadence — it might be monthly, quarterly, annually, but it must be a closed loop.\nThe Human Use of Human Beings The deepest meaning of X-Ops is redefining the role of human beings within the organization.\nIn the past, people in organizations primarily played two roles: the executor (the one who does the work) and the decider (the one who calls the shots). The former is responsible for getting things done; the latter is responsible for deciding what to do and how. Before the age of AI, both roles were scarce — execution required skill, judgment required experience.\nAI has changed this structure. Execution grows cheaper by the day, and judgment is beginning to be partially taken over by AI.\nLooking back at past waves of technological change, technology rarely eliminates human value outright — it more often shifts where value resides. From the Industrial Revolution to AI, the human role has been steadily moving from \u0026ldquo;directly operating on objects\u0026rdquo; to \u0026ldquo;designing, monitoring, and regulating a system that operates on objects.\u0026rdquo;\nAt this point, human value hasn\u0026rsquo;t disappeared — it has moved up a level.\nP vs NP offers a metaphor: finding an answer may be extraordinarily hard, but verifying one may be relatively easy. Just as a year ago we were still doing classical programming by hand, today the conversation has shifted to harness engineering — from \u0026ldquo;I build the thing myself, with my own hands\u0026rdquo; to \u0026ldquo;I define the goals, constraints, tests, acceptance criteria, and feedback loops, and let the machine generate results within a controllable range.\u0026rdquo; The critical position for humans is no longer being the producer of every specific output, but rather the higher-level evaluator, goal-setter, constraint-designer, feedback-loop builder.\nOf course, people\u0026rsquo;s fear of \u0026ldquo;automation\u0026rdquo; is more a matter of control and distribution — and in the age of AI, what deepens the fear even further is that it now encroaches on the position of \u0026ldquo;evaluator\u0026rdquo; itself. But all of that is a discussion for another day.\nThe Three Dangers of X-Ops X-Ops sounds like a perfectly correct methodology: make things explicit, make things versioned, build feedback loops, continuously improve. But the more correct something is, the more easily it mutates inside an organization. Especially in large companies and traditional industries, any methodology eventually risks becoming new processes, new systems, new forms — even new bureaucracy.\nThe first danger is tool worship.\nX-Ops is not \u0026ldquo;buy a set of Ops tools.\u0026rdquo; If you think bringing in Git and CI/CD means you\u0026rsquo;ve done X-Ops, you\u0026rsquo;ve most likely just wrapped a new layer of bureaucratic shell around the same old thing.\nThe second danger is metric alienation.\nThe moment you build a measurement system, someone in the organization will start optimizing the metric itself rather than the goal behind it — reward hacking. Goodhart\u0026rsquo;s Law: when a measure becomes a target, it ceases to be a good measure. The goal of X-Ops is to use measurement to drive improvement, not to use measurement to punish people. The moment metrics are used to assign blame, the organization instinctively begins to fabricate data and hide problems, and you\u0026rsquo;re right back on the well-worn path to digital bureaucracy.\nThe third danger is the illusion of control.\nThe most dangerous deviation of all is treating engineering as everything — everything must be measurable, everything must be automatable, everything must be regression-testable. A truly mature X-Ops is the discipline of knowing what can be engineered, and what should not be.\nConclusion Back to the opening question.\nBecause the question isn\u0026rsquo;t just whether AI is there — it\u0026rsquo;s whether the industry itself has become a system that AI can understand, participate in, and optimize.\nOtherwise, AI is just a more advanced shell wrapped around the same old way of operating.\nReal AI transformation isn\u0026rsquo;t about putting AI into the enterprise — it\u0026rsquo;s about turning the enterprise into a system that can run together with AI.\nOtherwise, so-called intelligent transformation, in the end, will merely become another form of digital bureaucracy.\n","permalink":"https://niraya666.github.io/en/posts/ai-transformation-hardest-part-isnt-ai-but-dirty-work/","summary":"Why AI transformation stalls after model selection: dirty data, undocumented workflows, integration debt, evaluation gaps, and organizational resistance are the real bottlenecks.","title":"The Hardest Part of AI Transformation Isn't AI — It's the Dirty Work Nobody Wants to Do"},{"content":"If you pull apart agent memory, there are really two distinct threads running through it.\nThe first is personal memory: how an agent comes to understand you — your preferences, long-term context, implicit constraints, and past decisions. This kind of problem lives closer to long-context QA, and the evaluation tends to focus on \u0026ldquo;does it remember, does it understand, and can it retrieve the right user information at the right time.\u0026rdquo;\nThe second is procedural memory: how an agent avoids repeating mistakes, distills reusable lessons from both successes and failures, and gets better the more it runs — compounding improvement over time. This thread is concerned with things like task success rates on agentic benchmarks, tool-calling efficiency, error-rate reduction, token-cost reduction, and whether real-world usage builds genuine engineering compound interest.\nThis article follows the second thread.\nThese days, agent harnesses like Claude Code, Hermes-agent, OpenClaw, and others accumulate a wealth of trajectories over extended use. Those trajectories capture some of the most valuable signals from an agent\u0026rsquo;s interaction with its environment.\nThe problem, though, is that even we humans don\u0026rsquo;t make great use of these trajectories. Most of the time, they\u0026rsquo;re treated as logs, debug records, or at best stuffed back into context as retrieval fodder. An agent harness that\u0026rsquo;s been running for six months might have accumulated thousands of task trajectories — and yet the next time a similar problem comes up, the agent starts from scratch, exploring the same dead ends and stepping into the same traps.\nThis raises a core question:\nHow do you actually turn trajectories into skills?\nTake a system like Hermes-agent. After enough mileage, the list of skills keeps growing. But how many of those skills are genuinely effective?\nThis article draws on a few recent pieces of work to discuss a question that\u0026rsquo;s becoming increasingly important: how to automatically mine, summarize, validate, and optimize agent skills from large collections of agent trajectories — skills that future tasks can actually reuse.\nHermes-agent as the Entry Point Hermes-agent has an interesting pitch: an agent that gets better the more you use it. In practice, this means distilling past experience into external memory that can be reused in the future. That external memory falls roughly into two categories.\nThe first is declarative memory — factual knowledge about the user, the project, the environment, and preferences. Concretely, this takes the form of USER.md and MEMORY.md. This kind of memory answers the question \u0026ldquo;how can the agent know you better.\u0026rdquo;\nThe second is procedural memory — skills. Concretely, this means sessions and messages written into state.db, plus the skills that get distilled into markdown files.\nMore specifically, the automatic skill learning in Hermes-agent is closer to a prompt-driven self-sedimentation mechanism. Its prompt gives the agent a triggering rule: when you finish a complex task, go through multiple tool calls, fix a tricky error, or discover a non-trivial workflow, you should save that approach as a skill. And if you find a skill you are using is outdated, incomplete, or wrong, patch it immediately — don\u0026rsquo;t wait for the user to point it out.\nSKILLS_GUIDANCE = ( \u0026#34;After completing a complex task (5+ tool calls), fixing a tricky error, \u0026#34; \u0026#34;or discovering a non-trivial workflow, save the approach as a \u0026#34; \u0026#34;skill with skill_manage so you can reuse it next time.\\n\u0026#34; \u0026#34;When using a skill and finding it outdated, incomplete, or wrong, \u0026#34; \u0026#34;patch it immediately with skill_manage(action=\u0026#39;patch\u0026#39;) — don\u0026#39;t wait to be asked. \u0026#34; \u0026#34;Skills that aren\u0026#39;t maintained become liabilities.\u0026#34; ) The actual persistence is handled by the skill_manage tool, which provides a few basic actions: create creates a new SKILL.md, patch does partial replacement of an existing skill, edit does a full rewrite, write_file adds supporting files, and delete removes skills that are no longer needed.\nThe upside is that it is simple, direct, and interpretable. But it is also rough around the edges.\nIt mostly relies on the model\u0026rsquo;s own judgment to decide what experience is worth turning into a skill. It feels more like post-hoc sedimentation after a single task, rather than distilling common patterns across multiple successful and failed trajectories. There is no evaluation. Does it actually improve task success rates? Are the skills genuinely effective? Over time, the skill bank keeps growing, but which skills are still useful and which have gone stale is unclear. It demonstrates the simplest possible loop — the agent discovers a workflow while working, writes it up as a skill, and closes the circle. It also exposes the real problems that follow-up research needs to solve: how to automatically induce skills from large numbers of trajectories, how to validate skills, how to merge them, how to optimize them, and how to keep the skill bank from turning into a junkyard.\nOutside the core Hermes-agent, Nous has also spun out a separate project called hermes-agent-self-evolution, which pushes this skill-sedimentation idea toward an external optimizer style of self-evolution. The idea is to read existing skills, prompts, tool descriptions, or code, combine them with execution traces, use DSPy + GEPA to generate candidate variants, and then produce a better version through evaluation, constraint checking, and human review.\n# https://github.com/NousResearch/hermes-agent-self-evolution/README.md Read current skill/prompt/tool ──► Generate eval dataset │ ▼ GEPA Optimizer ◄── Execution traces │ ▲ ▼ │ Candidate variants ──► Evaluate │ Constraint gates (tests, size limits, benchmarks) │ ▼ Best variant ──► PR against hermes-agent Think of it as an external optimizer: it treats skills / prompts / tool descriptions as optimizable objects, uses execution traces to explain why something failed, and searches for better text variants. That said, this project is still in its early stages — most features are still listed as planned.\nTrace2Skill: Batch Trajectory Induction https://arxiv.org/abs/2603.25158\nWhen we\u0026rsquo;re dealing not with a single skill or a single task, but with a large collection of successful and failed trajectories, how do we automatically induce transferable skills from them?\nA single trajectory from a single task is easily colored by randomness. The lesson learned from one run isn\u0026rsquo;t necessarily a stable pattern. If every local lesson gets written straight into the skill bank, you end up with fragmentation, duplication, conflicting rules, and overfitting.\nTrace2Skill starts from a different premise: collect a batch of trajectories first, then induce common patterns from the overall distribution.\nThe full Trace2Skill pipeline breaks down into three stages:\nStage 1: Trajectory Generation frozen agent + initial skill S0 → roll out on evolution tasks → collect successful trajectories T+ and failed trajectories T- Stage 2: Parallel Multi-Agent Patch Proposal dispatch one analyst sub-agent per trajectory → success analyzer A+ summarizes winning patterns → failure analyzer A- performs agentic root-cause analysis → each trajectory produces a trajectory-local skill patch Stage 3: Conflict-Free Patch Consolidation merge all patches hierarchically → deduplicate, resolve conflicts, filter out spurious patterns → distill into a unified skill directory S* In essence, this decomposes \u0026ldquo;from trajectories to skill\u0026rdquo; into three sub-problems:\nWhere does experience come from? From the agent\u0026rsquo;s real rollouts in the task environment. How is local experience expressed? As trajectory-local patches, not written directly into the final skill. How do multiple local experiences become a stable skill? Through hierarchical merging, conflict detection, and format validation — culminating in a unified skill directory. That said, Trace2Skill isn\u0026rsquo;t a full-blown self-learning agent operating in a real environment. In the paper and codebase, it\u0026rsquo;s evaluated on SpreadsheetBench, where the error analyst can compare against ground truth and verify a failure cause with minimal repair. This works well for benchmarking.\nBut in real-world settings, most tasks don\u0026rsquo;t have a clear ground truth. Whether an analysis report is good enough, or whether a code change matches the user\u0026rsquo;s intent — these are hard to judge with a single evaluator.\nSkillOpt: Treating Skills as Trainable External Parameters https://arxiv.org/abs/2605.23904\nCan a skill be further optimized? In other words, is a skill just a one-shot summary of experience, or can it be continuously trained through feedback — like model parameters? This is precisely where SkillOpt enters the picture.\nSkillOpt asks: given a current skill, how well does it perform across a batch of tasks? Which rules should be added, deleted, or replaced? Did the change actually make the agent stronger? And if it didn\u0026rsquo;t, should the change be rejected?\nThe implication is that skills aren\u0026rsquo;t just experience-summary documents — they\u0026rsquo;re external state that can be optimized. SkillOpt borrows ideas from deep learning and applies them to the optimization of natural-language skills. The basic setup: the target model stays frozen; the execution harness stays frozen; the task environment stays frozen. The only thing being optimized is the external skill document — through text-space optimization, analogous to deep learning training.\nThe SkillOpt pipeline breaks down into several steps:\ncurrent skill S_t ↓ target agent runs a batch of tasks with S_t → scored trajectories ↓ optimizer model analyzes success/failure mini-batches ↓ proposes add / delete / replace edits ↓ caps each round\u0026#39;s edits via a textual learning rate ↓ validates on a held-out selection split ↓ accept only if score improves; otherwise reject and log as negative feedback ↓ after multiple epochs, export best_skill.md The optimizer model can propose changes, but whether they\u0026rsquo;re accepted isn\u0026rsquo;t determined by the model\u0026rsquo;s confidence — it\u0026rsquo;s determined by the held-out validation score.\nHow It Works Forward pass: the target agent runs a rollout batch\nThe frozen target agent takes the current skill and executes a batch of tasks. The harness records task metadata, messages, tool calls, observations, command outputs, final answers, verifier feedback, and benchmark-specific context.\nThis mirrors the forward pass in deep learning — except the output isn\u0026rsquo;t a loss tensor. It\u0026rsquo;s:\nscored trajectories: task current skill agent messages tool calls observations final answer verifier feedback score This is critical: SkillOpt depends on scored rollouts. Without a score, the downstream validation gate has nothing to work with.\nBackward pass: minibatch reflection\nThese trajectories tell the optimizer two kinds of things:\nSuccessful trajectories: which rules in the current skill are likely working, and which behaviors should be preserved. Failed trajectories: what guidance the current skill is missing, which rules aren\u0026rsquo;t precise enough, and which actions need to be prohibited or replaced. SkillOpt doesn\u0026rsquo;t cram all trajectories into the optimizer model at once. Instead, it splits successes and failures into reflection mini-batches.\nFailure mini-batches produce corrective edits. They answer:\nWhat common problems do these failed trajectories expose? What checking steps is the current skill missing? Which existing rules are too vague? Which behaviors should be explicitly banned? Success mini-batches produce preservation edits. They answer:\nWhich behaviors led to success? Which rules should be kept or emphasized? Which effective workflows shouldn\u0026#39;t be broken by future edits? Add / Delete / Replace\nThe optimizer model\u0026rsquo;s output isn\u0026rsquo;t a full rewrite of the skill — it\u0026rsquo;s a set of structured edits. Local edits make each update more controllable and auditable, as opposed to rewriting the entire SKILL.md.\nTextual Learning Rate\nSkill updates are constrained as controlled textual operations. In model training there\u0026rsquo;s a learning rate; SkillOpt brings this idea to text-space: each round limits how many edits can be applied. This addresses the common problem where agent skills easily \u0026ldquo;learn into chaos.\u0026rdquo; The core insight: skill evolution should be continuous, not a tear-down-and-rebuild every time.\nValidation Gate\nAfter each round produces a candidate skill, SkillOpt has the same frozen target agent re-run tasks on the held-out selection split using that candidate. The modification is only accepted if the candidate skill\u0026rsquo;s score strictly exceeds the current skill\u0026rsquo;s score.\nRejected-Edit Buffer\nIf a candidate skill doesn\u0026rsquo;t pass the validation gate, SkillOpt doesn\u0026rsquo;t simply discard the attempt. It records the rejected edits, telling future optimizer runs:\nWhich modification directions were tried but failed; Which rules seemed reasonable but hurt performance; Which failure patterns still haven\u0026#39;t been resolved; Which edits caused regressions. It\u0026rsquo;s essentially an optimizer memory, but more granular: which skill edits were attempted but failed validation.\nFinal output: best_skill.md\nLimitations SkillOpt has clear limitations.\nFirst, it\u0026rsquo;s heavily dependent on scored trajectories. Tasks need a reasonably reliable scoring mechanism — whether exact match, unit tests, a benchmark evaluator, or LLM-as-judge. You need some way to produce comparable scores. This is relatively straightforward in benchmarks, but much harder in real-world environments.\nAutoSkill: Online Lifelong Skill Lifecycle https://arxiv.org/abs/2603.01145\nHow should skills be continuously generated, maintained, and reused?\nAs users interact with an agent over the long term, they keep expressing stable preferences, work habits, formatting requirements, task constraints, and operational patterns. AutoSkill attempts to turn these transient interaction experiences into explicit, reusable, maintainable, and retrievable skills.\nReal user-agent interactions are very different from benchmark trajectories.\nIn a benchmark, task boundaries are clear, inputs and outputs are well-defined, and there\u0026rsquo;s often an evaluator. Whether a spreadsheet task is solved correctly, whether code passes tests, whether a web task is completed — these can all be scored fairly directly.\nReal user interactions are rarely like that. Users repeatedly express needs that aren\u0026rsquo;t really \u0026ldquo;task-shaped.\u0026rdquo; These don\u0026rsquo;t have a single right answer like an exam question, but they matter a lot.\nTraditional memory can preserve these preferences, but memory tends to be just facts or fragments. For example:\nThe user prefers a formal style. The user dislikes excessive technical jargon. The project uses pnpm. This kind of memory handles \u0026ldquo;knowing what,\u0026rdquo; but doesn\u0026rsquo;t necessarily translate into \u0026ldquo;how to do it later.\u0026rdquo; AutoSkill\u0026rsquo;s goal is to further abstract these recurring interaction experiences into skills — procedural artifacts with trigger conditions, behavioral goals, constraints, and execution methods.\nSo AutoSkill\u0026rsquo;s core insight is this: user interaction experience shouldn\u0026rsquo;t just be saved as memory — it should be abstracted into executable, long-lived skills.\nThe overall closed loop:\nUser conversation / agent trajectory / conversation history ↓ Extract reusable skill candidate ↓ Retrieve similar existing skills ↓ Decision: discard / add / merge ↓ Write or update SKILL.md in SkillBank ↓ On future requests: query rewrite + hybrid retrieval ↓ Inject relevant skills into LLM prompt ↓ New feedback triggers another round of skill evolution At the heart of it are two loops.\nThe first is the response generation loop. When a new request arrives, AutoSkill rewrites the retrieval query based on the current query and historical context, retrieves relevant skills from the SkillBank, injects them into the prompt, and helps the model generate a response.\nThe second is the skill evolution loop. After each interaction, the system tries to extract reusable skill candidates from user input and interaction traces, then decides whether the candidate should be discarded, added as a new skill, or merged into an existing skill.\nThe key design point in AutoSkill: new candidates don\u0026rsquo;t go straight into the SkillBank. They first retrieve the most similar existing skills, and only the candidate plus a small set of nearest-neighbor skills are sent to the management judge — not the entire SkillBank jammed into the prompt. This serves two purposes: first, it avoids skill duplication; second, it keeps the management decision scalable.\nThe management judge makes three types of decisions:\ndiscard: The candidate is too vague, low-signal, non-transferable, one-off, or already covered by the existing bank. add: The candidate represents a new durable capability. merge: The candidate and an existing skill belong to the same capability; the existing skill should be updated. The decision is based on four dimensions:\njob-to-be-done deliverable type hard constraints / success criteria required tools / workflow Only merge when stripping away instance-level details still leaves the same capability; otherwise add; if it\u0026rsquo;s not worth keeping long-term, discard.\nIf the decision is merge, AutoSkill invokes the skill merging module. This isn\u0026rsquo;t simple text concatenation — it performs a semantic union:\nPreserve the original skill\u0026#39;s capability identity. Incorporate the candidate\u0026#39;s new reusable constraints, examples, and execution details. Remove duplicate content. Drop outdated or irrelevant topic constraints. Avoid regression — keep the important checks from the original skill. Don\u0026#39;t introduce new standards the user never explicitly set. After merging, the skill version number increments.\nAutoSkill persists skills as regular files. The default layout:\nSkillBank/ Users/\u0026lt;user_id\u0026gt;/\u0026lt;skill-slug\u0026gt;/SKILL.md Common/\u0026lt;skill-slug\u0026gt;/SKILL.md vectors/ \u0026lt;embedding-signature\u0026gt;.meta.json \u0026lt;embedding-signature\u0026gt;.ids.txt \u0026lt;embedding-signature\u0026gt;.vecs.f32 index/ skills-bm25.* skill_usage_stats.json Users/\u0026lt;user_id\u0026gt; holds user-personalized skills, Common holds shared skills, vectors caches embeddings, and index stores the BM25 index and usage statistics.\nThat said, AutoSkill has limitations.\nThe main pipeline relies heavily on an LLM to judge whether a candidate is worth saving, whether it should be merged, and how to merge — rather than being based on scored validation. It\u0026rsquo;s better suited for learning preferences, styles, workflows, and personalized constraints. For complex tool-using tasks, extracting skills purely from user interaction may not be enough. SkillX: Hierarchical Skill Knowledge Base https://arxiv.org/abs/2604.04804\nThe core idea behind SkillX is to organize agent experience into a hierarchical Skill Knowledge Base rather than a flat list of skills. Or put differently: experience should be decomposed into skills at different granularities that can be composed together.\nTake a strong agent, have it roll out on training tasks, collect successful trajectories; distill those trajectories into three layers of skills — planning, functional, and atomic; then apply clustering, quality filtering, schema validation, and iterative refinement across multiple rounds. The end result is a Skill Knowledge Base that is retrievable, injectable, and transferable to weaker agents.\nThe overall pipeline looks like this:\nTraining tasks ↓ Strong backbone agent rollout ↓ Filter successful trajectories ↓ Extract three-layer skills: Planning Skills Functional Skills Atomic Skills ↓ Clustering / merging / splitting ↓ Two-stage filtering: general quality filter tool schema filter ↓ Skill Library update: add / modify / keep ↓ Explore under-used / failure-prone tools ↓ Synthesize new tasks, continue rollout, continue extracting skills ↓ Form a plug-and-play SkillKB The framework breaks agent experience into three layers.\nPlanning Skill captures \u0026ldquo;how to organize a long-horizon task\u0026rdquo;\nFunctional Skill: reusable tool subroutines\nAtomic Skill: tool-level usage constraints and failure patterns\nAnd SkillX doesn\u0026rsquo;t stop after a single pass of extracting skills from trajectories. It introduces iterative skills refinement, letting the SkillKB keep improving across multiple rounds.\nStarting from the current skill library, it rolls out on training tasks, extracts multi-level skills, then applies refinement operators — Skills Merge and Skills Filter — and finally updates the Skill Library via add / modify / keep.\nThe value of SkillX is that it turns agent experience from a single agent\u0026rsquo;s private experience into transferable public skills. Think of it as a form of non-parametric distillation: not distilling a strong model\u0026rsquo;s capabilities into a weaker model\u0026rsquo;s parameters, but distilling a strong model\u0026rsquo;s execution experience into an external SkillKB — and the weaker model gains capability by retrieving and injecting from that SkillKB.\nOf course, SkillX has clear boundaries too. It suits scenarios where the tool environment is relatively stable, and it\u0026rsquo;s designed primarily for tool-using agents.\nRethinking Through the Lens of RL If we re-examine Traj2Skill (as an umbrella for this family of methods) from an RL perspective, the essence may not be \u0026ldquo;distilling knowledge from trajectories\u0026rdquo; — it\u0026rsquo;s:\nExtracting reusable policy improvement units from the distribution of historical behavior.\nAn agent\u0026rsquo;s behavioral process in an environment can be abstracted as a policy generating trajectories and maximizing expected return:\n$$ J(\\pi)=\\mathbb{E}{\\tau\\sim\\pi}\\left[\\sum{t=0}^T\\gamma^t r_t\\right] $$\nHere, a trajectory records the complete interaction — state, observation, tool calls, and environment feedback.\nSo the goal of Traj2Skill isn\u0026rsquo;t just to extract human-readable lessons from a trajectory. It\u0026rsquo;s to find structured behavioral patterns in existing trajectories such that the new policy $\\pi\u0026rsquo;$ achieves a higher expected return on future tasks than the original policy $\\pi$:\n$$ J(\\pi\u0026rsquo;) \u0026gt; J(\\pi) $$\nThe key difference is that Traj2Skill typically doesn\u0026rsquo;t update model parameters directly. Instead, it updates an external skill library — separate from the model itself.\ntrajectory → experience signal → update external skill library → guide future policy Viewed this way, a skill is not ordinary memory, nor a compressed version of a task log. It\u0026rsquo;s more like an externalized policy prior.\nIf we bring Agentic-RL into the comparison, Agentic-RL is closer to end-to-end policy optimization — it optimizes the model itself. Traj2Skill, by contrast, is closer to modular policy improvement — it optimizes reusable behavioral modules external to the model.\nFrom an engineering standpoint, this is also what makes the skill library appealing. An LLM-based policy is expensive to train, slow to update, weak in interpretability, and ill-suited for frequent fine-tuning. But a skill library is externalized, readable, editable, searchable, and versionable. It offers a lighter-weight path for policy improvement than directly updating model parameters. This is likely what motivates work like SkillOpt: bringing skills into a closed loop of \u0026ldquo;generate — evaluate — optimize.\u0026rdquo;\nLooked at differently, the two approaches aren\u0026rsquo;t substitutes — they can complement each other. That\u0026rsquo;s exactly what SAGE / SkillRL are exploring: incorporating skills into reinforcement learning or policy optimization frameworks.\nZooming out further: trajectories are too long, too specific, too contingent. What\u0026rsquo;s truly valuable is extracting behavioral structures that transfer across tasks. So where does Traj2Skill sit — closer to offline RL, or behavior cloning?\nA lot of current Traj2Skill work is actually closer to behavior cloning. The basic pipeline of trajectory mining is: summarize frequent patterns, compress, reuse. But there\u0026rsquo;s a problem here: a successful trajectory is not the same as an optimal trajectory.\nThis is the credit assignment problem. The final return of a long-horizon agent trajectory may come from only a handful of key decisions — not from every step along the way. What\u0026rsquo;s truly worth distilling into a skill is not necessarily the most frequently occurring behavioral snippets, but the behavioral structures that can consistently improve expected return on future tasks.\nThis perspective also exposes the hardest problem for Traj2Skill: where does the reward come from? Benchmarks have standard answers, but real-world environments are different — there\u0026rsquo;s often no simple ground truth. This means that the ceiling of a real-world Traj2Skill system depends heavily on the quality of its evaluator.\nIf we lay these pieces of work side by side, they each tackle different parts of the trajectory-to-skill pipeline:\nHermes-agent: self-sedimentation after a single trajectory Trace2Skill: experience induction across multiple trajectories SkillOpt: evaluation-driven skill optimization AutoSkill: long-term skill lifecycle management SkillX: hierarchical skill organization and strong-to-weak transfer So the core of Traj2Skill-style work breaks down into two questions.\nThe first is SkillOps: how to create, merge, retrieve, inject, version, evaluate, and retire skills — without letting the skill bank degrade into a junkyard.\nThe second is the deeper question: how to perform credit assignment over trajectories to identify behavioral structures that genuinely improve expected return.\nThe former solves for long-term operation. The latter answers whether this is really about \u0026ldquo;summarizing experience\u0026rdquo; — or about policy improvement.\n","permalink":"https://niraya666.github.io/en/posts/from-agent-trajectory-to-agent-skill-compounding-returns/","summary":"How to convert agent trajectories into reusable skills through extraction, validation, deduplication, evaluation, and continuous optimization.","title":"From Agent Trajectory to Agent Skill: How to Turn Experience into Compounding Returns"},{"content":"Prologue \u0026ldquo;Have you Harnessed today?\u0026rdquo;\nIn 2026, \u0026ldquo;Harness\u0026rdquo; is probably the most-used word in the AI agent space — and it comes with enormous misunderstandings. Look back at the past 2–3 years of AI development and you\u0026rsquo;ll see a mushrooming of agent frameworks, scaffolding, and orchestration tools, most of which sank deep into the mire of technical debt before vanishing without a trace.\nIn this article, I\u0026rsquo;m not going to dwell on the grand vision of harness engineering (a term that, in my view, carries the connotation of agents self-iterating and optimizing). Let\u0026rsquo;s just talk about the harness itself.\nThis article answers exactly one question: what are the irreplaceable core components of a future-facing, minimalist agent harness?\nAnd, within today\u0026rsquo;s technical context, how to sidestep the technical debt accumulated over the past two years and choose — or build — the right agent harness.\nBuild the harness well, and the agent will take care of the rest.\nSetting the Record Straight: What Exactly Is a Harness? To understand what a harness really is, we need to do a bit of cyber-archaeology.\nIn traditional software engineering, a test harness is an automated environment built to execute tests, with a core goal of \u0026ldquo;de-manualization.\u0026rdquo;\nIn the LLM and agent space, harness was initially used more in the context of LLM evaluation. On complex benchmarks like SWE-bench, a bare LLM cannot score highly — it needs an external loop and tools to assist. At this stage, harness was confined to the eval domain.\nIt wasn\u0026rsquo;t until applications like Claude Code appeared that agent harness gradually took on a new meaning: the foundation that supports AI work.\nThe earliest source we can find: on September 23, 2025, developer Vivek Trivedy was the first to systematically articulate \u0026ldquo;agent harness\u0026rdquo; in a blog post.\n\u0026ldquo;Agent Harness: External set of functionality to enhance a model\u0026rsquo;s runtime execution. …\nBut the concept truly exploded after two articles: Anthropic: Effective harnesses for long-running agents (Nov 26, 2025) and OpenAI: Harness engineering: leveraging Codex in an agent-first world (Feb 11, 2026).\nSimply put, in one sentence: a harness is the software shell wrapped around an LLM — it turns a model that can only talk into an agent that can get things done, and even into a long-running agent that operates for hours at a stretch.\nVarious ecosystems, LangChain included, have been staking claims by inventing a dizzying array of concepts. So what exactly is the difference between agent frameworks, runtimes, scaffolding, and harnesses?\nDrawing on LangChain\u0026rsquo;s article: Agent Frameworks, Runtimes, and Harnesses - oh my!\nRuntime: the low-level infrastructure that solves \u0026ldquo;how an agent runs stably and persistently in production.\u0026rdquo;\nFramework: provides a set of abstractions and programming models to help you \u0026ldquo;write agent applications more easily.\u0026rdquo;\nHarness: layers default configurations, tools, and workflows on top of a framework, creating a \u0026ldquo;high-level kit you can grab and run tasks with out of the box.\u0026rdquo;\n(Concepts are all man-made — the thing comes first, the concept follows.)\nDesign Philosophy: Less is More The lessons distilled from How To Be A World-Class Agentic Engineer and Learning the Bitter Lesson converge on a single idea: Less is More. The various structures and inductive biases humans inject may buy short-term wins, but over the long run they harden into constraints — they should be progressively stripped away as models and compute improve.\nHere is the corollary of The Bitter Lesson in the agent space, stated bluntly:\nEvery line of \u0026ldquo;smart\u0026rdquo; control logic you write in your harness is a bet against future model capabilities. Consider:\nIn 2024 you wrote an elaborate ReAct state machine → In 2025, models natively support multi-turn tool calling, and your state machine becomes dead weight.\nIn 2025 you wrote a complex planning module → In 2026, models\u0026rsquo; reasoning capabilities are strong enough to plan autonomously, and your Planner becomes a ceiling rather than a scaffold.\nThis is not an argument against having a harness. It is an argument for asking, of every component you put in: \u0026ldquo;Could the model learn to do this on its own?\u0026rdquo;\nIf the answer is \u0026ldquo;sooner or later,\u0026rdquo; it is an auxiliary component — write it with a demolition plan from day one. If the answer is \u0026ldquo;never\u0026rdquo; (because it touches the physical world outside the LLM), then it qualifies as a core component.\nFlipped the other way: minimal, least-necessary components are precisely what make a harness resilient enough to survive the next 3–6 months of model iteration.\nConcretely, taking \u0026ldquo;long-running\u0026rdquo; as the target, harness components fall into two categories:\nCore components: agent loop, context management, atomic core tools, file system, hooks and subagent isolation (or task management)\nAuxiliary components: (short-term patches — destined for replacement in the long run)\nCore Components Agent Loop Every story about agents begins with the agent loop. A good agent loop is necessarily lean — the LLM drives the decision-making, not a set of pre-scripted paths imposed by humans.\nAt the lowest level, an LLM is still an autoregressive model that predicts the next token. This fundamental premise holds for the foreseeable future. The Agent Loop does only one thing: execute tools → capture results → append to context → call the LLM again.\ndef agent_loop(query): messages = [{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: query}] while True: response = client.messages.create( model=MODEL, system=SYSTEM, messages=messages, tools=TOOLS, max_tokens=8000, ) messages.append({\u0026#34;role\u0026#34;: \u0026#34;assistant\u0026#34;, \u0026#34;content\u0026#34;: response.content}) if response.stop_reason != \u0026#34;tool_use\u0026#34;: return results = [] for block in response.content: if block.type == \u0026#34;tool_use\u0026#34;: output = run_bash(block.input[\u0026#34;command\u0026#34;]) results.append({ \u0026#34;type\u0026#34;: \u0026#34;tool_result\u0026#34;, \u0026#34;tool_use_id\u0026#34;: block.id, \u0026#34;content\u0026#34;: output, }) messages.append({\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: results}) (from learn-claude-code)\nThis is the core loop of an agent. All the \u0026ldquo;intelligence\u0026rdquo; comes from the same place: the LLM\u0026rsquo;s internal reasoning as it predicts the next token. The harness simply feeds the tool response back in, so that this reasoning process can continue.\nOf course, from an engineering and production standpoint, the agent loop is still this while True loop at heart — but it needs a substantial defensive layer to build resilience.\nAccording to unofficial analysis of Claude Code\u0026rsquo;s source, queryLoop (src/query.ts:241) wraps the core loop with over 1,700 lines of defensive machinery, including:\nStreaming handling\nPermission checks\nTool format processing and validation\nError recovery\n…\nContext Management Over the past six-plus months, the term \u0026ldquo;Context Engineering\u0026rdquo; has started appearing with increasing frequency. It isn\u0026rsquo;t about simple prompt tuning — it addresses a more fundamental tension: the contradiction between ever-accumulating messages and the LLM\u0026rsquo;s finite context window and attention budget. In a long-running scenario, continuously appending messages eventually leads to:\nContext overflow: hitting the physical upper bound of the LLM\u0026rsquo;s context window\nContext Rot: staying within the window doesn\u0026rsquo;t mean the model can process everything effectively — too much irrelevant information can severely degrade the model\u0026rsquo;s reasoning\nHow does a well-designed harness handle context?\nContext management is not just about splicing and trimming prompts. The central principle is token cost — or more precisely, KV-cache hit rate. In other words: optimize context while preserving cache hits.\n1. Microcompact\nCleans up intermediate tool results generated during earlier rounds of tool use. Based on source-code analysis, Claude Code distinguishes two approaches:\nTime-based Microcompact: When the user returns after being away for a while, the server-side prompt cache has already expired. There is no cache to protect, so the harness directly constructs new message objects to replace old tool-result content.\nCached Microcompact (Context editing): While the cache is still alive, message content must not be replaced. Instead, the harness uses the Anthropic API\u0026rsquo;s Context editing mechanism to delete specific tool_result entries at the API level.\n2. Summary, Compact, Auto-Compact\nWhen a threshold is triggered — manually or automatically — the harness calls an LLM to summarize and condense the existing context, replacing the previous context with the result. In Claude Code, for example, the summary must include:\n// The summary must contain the following 9 sections 1. Primary Request and Intent // The user\u0026#39;s main request and intent 2. Key Technical Concepts // Key technical concepts involved 3. Files and Code Sections // Files and code sections touched 4. Errors and fixes // Errors encountered and how they were fixed 5. Problem Solving // Problem-solving steps taken 6. All user messages // All messages from the user 7. Pending Tasks // Tasks still pending 8. Current Work // Work currently in progress 9. Optional Next Step // Suggested next step (optional) In the Anthropic Claude API, the system prompt is independent from Messages, which enables a \u0026ldquo;full summary\u0026rdquo; model — the entire message history is replaced with a summary.\nBy contrast, OpenAI\u0026rsquo;s Codex uses a layered approach of \u0026ldquo;initial context + most recent user messages (up to 20k tokens) + summary\u0026rdquo;:\n3. Offload\nUnlike compact, which is lossy compression, offload uses the file system to persist context into files for permanent storage, freeing the corresponding context-window space. The offloaded content can be loaded back on demand — this is lossless compression.\n4. Isolation\nFor processes that involve extensive tool calls and exploration, large tasks are decomposed into smaller sub-tasks handled by subagents. This keeps each sub-task\u0026rsquo;s context independent and isolated, preventing context contamination.\nThe goal of context management: find the optimal balance among information completeness × KV-cache hit rate × token cost.\nAtomic Tools Strictly speaking, what LLMs call tool use isn\u0026rsquo;t really \u0026ldquo;using tools.\u0026rdquo; The model itself is merely outputting tool-call text in a special format; the actual tool execution is carried out by the harness, which parses that text and translates it into real calls against the environment.\nBut more tools is not better. An excess of tools not only reduces call accuracy but also eats up precious context, causing noticeable Context Rot. For an agent, what really matters isn\u0026rsquo;t the number of tools — it\u0026rsquo;s whether the tools possess:\nAtomicity\nOrthogonality\nCompleteness\nComposability\nPut another way, the tool layer should be \u0026ldquo;primitivized\u0026rdquo; as much as possible — each tool does exactly one well-defined thing, but these tools can be freely composed to build complex behaviors.\nWhether it\u0026rsquo;s Manus\u0026rsquo;s three-tier tool design or Anthropic\u0026rsquo;s proposed tool-search-tool and programmatic tool calling, the core question behind both is the same:\nFor an agent / LLM, what counts as a \u0026ldquo;native tool\u0026rdquo;?\nA native tool isn\u0026rsquo;t a highly specialized interface that engineers wrap for business convenience. It\u0026rsquo;s a tool form that the model is more familiar with, finds easier to understand, and can invoke more reliably. You could also call them agent-oriented tools.\nbash is all you need\nLow-level capabilities like Bash, file reads and writes, text editing, and regex processing are the most common operational interfaces in the Linux and software-engineering world. They appear with extremely high frequency in training data, which also makes them far more likely to fall within the training distribution of agentic RL. By contrast, those hand-crafted custom tools — with ambiguous semantics and wildly inconsistent interface styles — often impose a significantly heavier burden on the agent\u0026rsquo;s understanding and tool-selection process.\nSo, rather than giving an agent hundreds of highly specialized third-party plugins, it\u0026rsquo;s better to prioritize a small set of stable, low-level, composable foundational capabilities.\n(CRUD for agents?)\nFile System In an agent system, the file system means far more than just \u0026ldquo;saving files.\u0026rdquo;\nOn one hand, the file system can persist information that would otherwise live only inside the context window — intermediate results, plans, drafts, observation notes, and working memory (i.e., the context offload discussed earlier).\nOn the other hand, the file system itself is one of the most natural execution environments. Especially in coding-agent scenarios, code, configuration, logs, test output, scripts, and documentation are all organized around files to begin with. Rather than inventing yet another set of memory interfaces or tool abstractions, providing basic operations like read, write, and edit aligns better with how agents actually work — and is closer to real OS-level workflows.\nFor these reasons, the file system in a harness typically plays at least three critical roles:\nWorking Memory: stores intermediate state, task plans, incremental conclusions, and temporary artifacts\nExecution Environment: hosts code, scripts, configuration, inputs/outputs, and run results\nIsolation Boundary / Sandbox: constrains the agent\u0026rsquo;s operational scope and permissions through controlled directories, virtual file systems, or sandbox environments\nThe paper Everything is Context: Agentic File System Abstraction for Context Engineering borrows the Unix philosophy of \u0026ldquo;Everything is a file\u0026rdquo; and applies it to the agent\u0026rsquo;s file system.\nSubagent The subagent began as a context isolation mechanism. It has since evolved into a fundamental unit of task scheduling and multi-agent coordination.\nIn many tasks, the main agent\u0026rsquo;s context is an extremely precious resource. Operations like searching a large codebase, running tests, debugging errors, or performing long-running analysis tend to generate enormous amounts of procedural noise. If all those intermediate steps are stuffed into the main conversation, they not only pollute the primary context but also degrade the quality of subsequent reasoning and decision-making.\nThe natural solution, then, is: spawn a subagent and let it handle the \u0026ldquo;dirty work\u0026rdquo; in an isolated context.\nThe main conversation doesn\u0026rsquo;t need to consume the full process — it only needs to receive a summarized result. From the main agent\u0026rsquo;s perspective, it simply called a higher-order tool — an AgentTool — and then waits for (or later receives) a block of summary text, which it writes back into its own context as a tool_result. In other words, introducing subagents does not alter the structure of the core queryLoop.\nZooming out further, the subagent is also a task-scheduling mechanism.\nThis scheduling generally takes two forms:\nSynchronous subagent: called like a regular tool, blocking until the result arrives\nAsynchronous subagent: runs as a background task; the main conversation continues forward, and the result is collected later\nOnce you enter asynchronous mode, the system must maintain a global task registry (e.g., AppState.tasks in Claude Code), running in the background, and inject completed results back into the main agent\u0026rsquo;s agent loop as a message.\nWhen multiple background subagents run concurrently, the subagent\u0026rsquo;s value is no longer just context isolation — it also becomes the starting point for multi-agent orchestration.\nHooks To keep the agent loop lean, new capabilities should not keep piling into the loop itself.\nOtherwise, a cycle that was originally just four steps — \u0026ldquo;execute tools → capture results → append to context → call the LLM again\u0026rdquo; — quickly gets dragged into a giant if-else state machine by all manner of business logic.\nThe point of hooks is to move that complexity outside the loop.\nThrough events like PreToolUse, PostToolUse, PreCompact, SubagentStop, and others, the harness can capture state at key inflection points, inject context, intercept risks, and validate results — without requiring the loop itself to understand every rule.\nSo the essence of hooks isn\u0026rsquo;t \u0026ldquo;a few more extension points.\u0026rdquo; It\u0026rsquo;s separating mechanism from policy:\nThe loop is responsible for advancing execution\nHooks are responsible for exerting control\nOn the surface, hooks just give you a list of event names. But more importantly, these events define the critical boundaries within an agent system.\nOnce the boundaries are clear, many capabilities no longer require direct modification of the loop itself — they can be injected as external policies:\nSecurity policies can hook around tool calls\nMemory governance can hook around compact operations\nTask orchestration can hook onto the subagent lifecycle\n…\nBoundary Components Beyond the six core components discussed above, there are a handful of concepts that come up often in conversation. Not every one of them is essential, but understanding where they sit helps you avoid over-engineering.\nCheckpointing / Recovery\nFor a long-running agent, error recovery and state checkpointing genuinely matter. But from the harness\u0026rsquo;s perspective, checkpointing is more of a cross-cutting capability that spans multiple components: the file system handles persistence, hooks trigger saves at key milestones, and the agent loop resumes from the breakpoint. It\u0026rsquo;s hard to isolate checkpointing as a standalone \u0026ldquo;component\u0026rdquo; — it\u0026rsquo;s really the emergent result of several core components working together.\nPlanner / Todo-list\nRemember in early 2026, Claude Code dropped its built-in todo-list tool. The reason was straightforward: the reasoning capability of the new model generation had become strong enough that an explicit, manually maintained todo list was no longer necessary. This is a perfect echo of The Bitter Lesson — the planning module you write today is quite likely the first auxiliary component to be ripped out tomorrow. If you absolutely must build one, make sure it can be cleanly detached rather than deeply coupled to the loop.\nSkills\nSkills are essentially a composite built from core components: a command registry, on-demand message injection, and a pile of markdown files. They don\u0026rsquo;t belong to the harness\u0026rsquo;s lower layer — they sit in the application layer above it.\nMCP / Plugins\nMCP is a protocol layer, not part of the harness itself. Supporting MCP is certainly a plus, but what it solves is \u0026ldquo;how to expose external resources,\u0026rdquo; not \u0026ldquo;what core capabilities a harness should have.\u0026rdquo;\nMemory\nMemory is undeniably important, but as of today the industry hasn\u0026rsquo;t converged on a canonical form. Whether you look at OpenClaw or Claude Code, the prevailing approach is a lightweight memory scheme based on markdown files. This \u0026ldquo;files-as-memory\u0026rdquo; approach may seem crude, but it aligns perfectly with the harness\u0026rsquo;s minimalist philosophy — instead of building an elaborate memory database, just let the agent manage its own long-term state through the file interface it already knows best.\nAgent Team and Multi-agent\nWhen you scale out to multiple agents, the harness components inside each agent stay exactly the same — the same agent loop, the same context management, the same atomic tools. The only new addition is the connection mechanism between agents.\nIf your harness is well-designed for a single agent, 90% of the infrastructure can be reused directly when you extend to a team. The difference between a subagent and an agent team member can be summarized in the table below:\nDimension Subagent Agent Team Member Lifecycle Task complete → destroyed Survives across tasks, may run for hours Identity Anonymous (only a task description) Has a role identity (\u0026ldquo;reviewer\u0026rdquo;, \u0026ldquo;coder\u0026rdquo;, \u0026ldquo;tester\u0026rdquo;) State Stateless, vanishes after returning results Has persistent state (its own context history, memory files) Spawn method Spawned by the main agent Pre-defined or dynamically spawned From the harness\u0026rsquo;s point of view, a multi-agent system requires a few additional things:\nAgent registry: who is running, what is their role, current status\nLifecycle hooks: on_spawn, on_idle, on_terminate\nGraceful shutdown and resource cleanup: this is the most overlooked — and most critical — piece\nA subagent is unidirectional (main agent dispatches a task → subagent returns a result), whereas an agent team needs bidirectional or even multi-directional communication. But the design of that communication should still follow the principle of \u0026ldquo;fit the LLM\u0026rsquo;s native habits\u0026rdquo;: a shared file system is, itself, the most natural communication channel.\nHarness Engineering = Bigger Loop Due to space constraints, I won\u0026rsquo;t go deep into harness engineering here. Borrowing the perspective from What Harness Engineering Discusses: A Unified Framework of Three Scaling Dimensions, the core keyword of harness engineering is scalability — extending across time, expanding across space, and evolving interaction paradigms.\nFrom my perspective, I personally consider harness engineering and compound engineering to be synonyms. The real value lies in: how do you make an agent run better and better over time?\nThe prerequisite for all of this is a usable harness plus a bigger loop.\nA complete agent system has two loops:\nInner loop (agent loop): the \u0026ldquo;tool call → result capture → context append → next decision\u0026rdquo; cycle within a single task. This is the primary focus of this article.\nOuter loop (evolution loop): the \u0026ldquo;execute → evaluate → feedback → improve\u0026rdquo; cycle across tasks. This is the primary focus of harness engineering.\nYou can see these ideas at play in Andrej Karpathy\u0026rsquo;s autoresearch and in Ralph Loop.\nThe outer loop doesn\u0026rsquo;t need a separate architecture — it\u0026rsquo;s the inner loop\u0026rsquo;s core components, reused on a larger time scale. Hooks provide observation points, the file system provides persistent memory, subagents provide parallel experimentation capability, and context management ensures runs don\u0026rsquo;t spiral out of control.\nDesign Principles for a Good Harness 1. Minimality\nIf a component isn\u0026rsquo;t strictly necessary, don\u0026rsquo;t add it. For every component, ask yourself:\n\u0026ldquo;If I remove this, does the agent stop working entirely — or does it just run a little less well?\u0026rdquo;\nThe former is core. The latter is auxiliary.\n2. Removability\nAuxiliary components must be cleanly removable, not entangled with core components.\nThis is The Bitter Lesson applied to engineering: whatever you write today,\nmust be easy for your future self, six months later, to delete cleanly.\n3. Model Agnosticism\nThe harness should not bake in special-case adaptations for any particular model\u0026rsquo;s quirks.\nA good harness: swap the model, change the API call, and it runs.\n(Prompt optimization for a specific model is a different matter, of course.)\n4. Evolution-Ready\nThe ultimate goal of a harness is not \u0026ldquo;run a task,\u0026rdquo;\nbut \u0026ldquo;provide an environment where the agent gets better over time.\u0026rdquo;\nEvery component\u0026rsquo;s design should answer:\nHow does it contribute data, feedback, or tunable parameters to the outer loop?\nConclusion An agent harness does not get better as it gets more complex. Quite the opposite — its value lies in restraint.\nThe history of agents from 2023 to 2026 is, at its core, a history of human-designed control logic being continuously outcompeted by model capabilities. The real mission of a harness is to provide the model with a stable, safe, evolvable runtime environment — and then get out of the way, handing the decisions back to the model itself.\nFurther Reading Anthropic: Effective harnesses for long-running agents\nlearn-claude-code\nOpenAI: Harness engineering: leveraging Codex in an agent-first world\nLearning the Bitter Lesson\nAgent Frameworks, Runtimes, and Harnesses - oh my!\nHarness design for long-running application development\nDifference Between Agent Harnesses \u0026amp; Agent Frameworks\nHarness Engineering Is Cybernetics\nThe importance of Agent Harness in 2026\nThe Claude Code SDK and the Birth of HaaS (Harness as a Service)\nAgent design lessons from Claude Code\nAgent Harness is the Real Product\nHarness capabilities\nAgents 2.0: From Shallow Loops to Deep Agents\n","permalink":"https://niraya666.github.io/en/posts/agent-harness-minimalist-design-guide-2026/","summary":"A practical 2026 guide to the minimum architecture of an AI agent harness: loops, tools, filesystems, context management, memory, subagents, and observability.","title":"On Agent Harness: A Minimalist Design Guide for AI Agent Harnesses in 2026"},{"content":"Why We Need a Thinking Budget Have you ever watched an LLM, mid-stream in deep thought, spiral into an endless loop of self-doubt?\nChain of thought (CoT) does improve model accuracy to some degree, and reasoning mode has steadily become a must-have feature across major open-source models. But a new problem has followed close behind: runaway thinking. LLMs frequently exhibit overthinking — excessive redundant speculation, constant self-negation — all of it driving up the token budget. These issues are especially pronounced in smaller reasoning models obtained through distillation.\nSo a natural idea presents itself: give the model a thinking budget. Use a small budget for simple problems, and a larger one for hard problems.\nReasoning Control from Mainstream Model Providers Across mainstream model providers, you\u0026rsquo;ll find that APIs and application services generally include mechanisms for controlling reasoning length.\nOpenAI: provides parameters like reasoning.effort to hint at the model\u0026rsquo;s thinking depth (for o1 and o3 series models)\nAnthropic: allows triggering different levels of thinking through budget_tokens or specific prompts (e.g., \u0026ldquo;think\u0026rdquo; \u0026lt; \u0026ldquo;think hard\u0026rdquo; \u0026lt; \u0026ldquo;think harder\u0026rdquo; \u0026lt; \u0026ldquo;ultrathink.\u0026rdquo;)\nGemini \u0026amp; Qwen: offer an explicit thinking_budget parameter directly in their APIs\nOpen-Source Implementation: Budget Forcing So how should developers — especially those deploying LLMs locally — implement thinking-budget?\nThe most intuitive approach is to use a prompt, but the results are underwhelming. Unless the model was specifically trained for it during training, explicitly telling it to \u0026ldquo;think with fewer tokens\u0026rdquo; generally does little. In contrast, the Qwen3 series of hybrid thinking models incorporated /think and /no_think markers during training, so adding /no_think to the system prompt or user prompt at inference time can switch the model out of thinking mode.\n(Section 4.3) \u0026ldquo;To better integrate the two modes and enable users to dynamically switch the model\u0026rsquo;s thinking process, we design chat templates for Qwen3, as shown in Table 9. Specifically, for samples in thinking mode and non-thinking mode, we introduce /think and /no think flags in the user query or system message, respectively. This allows the model to follow the user\u0026rsquo;s input and select the appropriate thinking mode accordingly.\u0026rdquo; …\n(Section 4.4): \u0026ldquo;Format Following: In addition to explicit instructions, we expect the model to adhere to specific formatting conventions. For instance, it should respond appropriately to the /think and /no think flags by switching between thinking and non-thinking modes\u0026hellip;\u0026rdquo; … (Section 4.7): \u0026ldquo;ThinkFollow: Involves multi-turn dialogues with randomly inserted /think and /no think flags to test whether the model can correctly switch thinking modes based on user queries.\u0026rdquo;\nWhile reading through Qwen3\u0026rsquo;s technical report and related documentation, I came across the approach Qwen officially provides the open-source community for thinking-budget.\nIt proposes a two-step implementation:\nStep 1: Generate. This is no different from a standard generation — you simply set max_new_tokens to thinking_budget.\n# first generation until thinking budget generated_ids = model.generate( **model_inputs, max_new_tokens=thinking_budget ) output_ids = generated_ids[0][input_length:].tolist() Step 2: Check for \u0026lt;/think\u0026gt;.\nThat is, check whether the token ID for \u0026lt;/think\u0026gt; (151668) appears in the output.\nIf it does: The model completed its thinking within budget — no intervention needed. If it doesn\u0026rsquo;t: The thinking process was cut off by the budget. Inject the early_stopping_text, concatenate it with the previous output, and feed it as a new round of input for a second model.generate call. # check if the thinking process has finished (151668 is \u0026lt;/think\u0026gt;) if 151668 not in output_ids: print(\u0026#34;thinking budget is reached\u0026#34;) # Prepare the early-stopping prompt early_stopping_text = \u0026#34;\\n\\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\\n\u0026lt;/think\u0026gt;\\n\\n\u0026#34; early_stopping_ids = tokenizer([early_stopping_text], ...).input_ids.to(model.device) # Concatenate the prompt to the end of the first generation input_ids = torch.cat([generated_ids, early_stopping_ids], dim=-1) # Run the second generation generated_ids = model.generate( input_ids=input_ids, ... ) But this approach requires two rounds of generation. Is there a more elegant way to do it?\nThe answer is yes — by using a LogitsProcessor.\nThe idea comes from this blog post and a GitHub issue discussion.\nThe core principle: create a custom LogitsProcessor that inspects the state at every token generation step:\nDetermine whether the model is currently between \u0026lt;think\u0026gt; and \u0026lt;/think\u0026gt; Count the number of tokens generated since \u0026lt;think\u0026gt; began When the count reaches thinking_budget, forcibly modify the probability distribution (logits) for the next token: set the probability of \u0026lt;/think\u0026gt; to 1.0 and all other tokens to negative infinity This implements thinking-budget through constrained decoding.\nfrom typing import List, Optional import torch class ThinkLogitsProcessor: \u0026#34;\u0026#34;\u0026#34;A logits processor that limits the number of thinking tokens.\u0026#34;\u0026#34;\u0026#34; def __init__(self, think_start_token, think_end_token, num_think_tokens: int = 100): \u0026#34;\u0026#34;\u0026#34; Initialize the think logits processor. Args: tokenizer: The tokenizer used for the model num_think_tokens: Maximum number of tokens allowed in thinking section \u0026#34;\u0026#34;\u0026#34; self.num_think_tokens = num_think_tokens self.think_start_token = think_start_token self.think_end_token = think_end_token def __call__( self, input_ids: List[int], logits: torch.Tensor, ) -\u0026gt; torch.Tensor: \u0026#34;\u0026#34;\u0026#34; Process the logits to enforce \u0026lt;/think\u0026gt; token when needed. Args: input_ids: List of input token IDs. logits: Tensor of logits for the next token. Returns: Processed logits tensor. \u0026#34;\u0026#34;\u0026#34; # Check if we\u0026#39;re in a thinking section if self.think_start_token in input_ids and self.think_end_token not in input_ids: # Find the position of the last \u0026lt;think\u0026gt; token think_start_pos = len(input_ids) - 1 - input_ids[::-1].index(self.think_start_token) # Calculate number of tokens since \u0026lt;think\u0026gt; tokens_since_think = len(input_ids) - think_start_pos - 1 # If we\u0026#39;ve reached the maximum thinking length, force \u0026lt;/think\u0026gt; if tokens_since_think \u0026gt;= self.num_think_tokens: # Set all other logits to -inf except for \u0026lt;/think\u0026gt; logits = torch.full_like(logits, float(\u0026#39;-inf\u0026#39;)) logits[self.think_end_token] = 1.0 return logits Here\u0026rsquo;s how to use it with vLLM:\n(Note: after vLLM upgraded to the V1 engine, Python-based LogitsProcessors are no longer supported. The following code requires a pre-V1 version of vLLM, such as 0.7.3.)\nfrom transformers import AutoTokenizer from vllm import LLM, SamplingParams tokenizer = AutoTokenizer.from_pretrained(\u0026#34;Qwen/Qwen3-4B\u0026#34;) think_start_token = tokenizer.convert_tokens_to_ids(\u0026#34;\u0026lt;think\u0026gt;\u0026#34;) think_end_token = tokenizer.convert_tokens_to_ids(\u0026#34;\u0026lt;/think\u0026gt;\u0026#34;) sampling_params = SamplingParams( temperature=temperature, max_tokens=max_tokens, logits_processors=[ThinkLogitsProcessor(think_start_token, think_end_token, num_think_tokens)] ) llm = LLM(model=\u0026#34;Qwen/Qwen3-4B\u0026#34;) conversations = [\u0026#34;your-conversations\u0026#34;] outputs = llm.chat(messages=conversations, sampling_params=sampling_params, use_tqdm=False) All of the above approaches use a hard-cutoff strategy — forcibly inserting \u0026lt;/think\u0026gt;. The result can feel rather abrupt, and the model often continues a stretch of thinking in its response anyway, following its own momentum.\nRefinements to this approach, such as a soft cutoff (linearly or polynomially increasing the output probability of \u0026lt;/think\u0026gt; as the budget limit approaches), can improve things to some degree.\nBeyond that, inserting special tokens or directly injecting natural-language prompts are also viable options.\nDifficulty-Based Truncation But a new problem arises: a one-size-fits-all budget is far from perfect. For simple questions, the budget is too long; for hard questions, it\u0026rsquo;s too short. Manually setting a budget isn\u0026rsquo;t practical during synthetic data generation and rollout. The most intuitive approach is to use a classification model to predict an optimal budget.\nIn TALE, the authors use a zero-shot prompt that asks the model to analyze the question and estimate the minimum number of tokens required to generate a complete and accurate response, then embed the budget value into the prompt, like:\nTask: Analyze the given question and estimate the minimum number of tokens required to generate a complete and accurate response. Please Give the response by strictly following this format: [[budget]], for example, Budget: [[12]]. In practice, when examining model outputs, you\u0026rsquo;ll notice that the model easily falls into patterns of guessing, uncertainty, and self-doubt, such as:\nOr, maybe XXX?\n\u0026hellip;\nMaybe the answer is XX?\n\u0026hellip;\nMaybe I made a calculation error?\n\u0026hellip;\nMaybe I got something wrong?\n\u0026hellip;\nIn particularly long CoT traces — especially toward the tail end — the model essentially gets trapped in a doom loop of self-doubt and circular reasoning. One could perhaps capture these uncertainty markers and, when they appear frequently, boost the probability of \u0026lt;/think\u0026gt; to trigger early stopping. However, this approach is really only suitable for small models — especially distilled reasoning models. The formulaic patterns of \u0026ldquo;self-doubt\u0026rdquo; and \u0026ldquo;uncertainty\u0026rdquo; seen in these small models likely originate from the teacher model, but the small model itself lacks (or struggles to develop) genuine self-correction capability. As a result, all those \u0026ldquo;wait…\u0026rdquo;, \u0026ldquo;But wait…\u0026rdquo;, \u0026ldquo;Alternatively…\u0026rdquo; — these may not actually contribute much to reasoning (this needs experimental corroboration).\nCode import torch from typing import List, Dict, Tuple, Set, Optional from collections import defaultdict class UncertaintyThinkLogitsProcessor: \u0026#34;\u0026#34;\u0026#34; A logits processor that limits thinking based on detected uncertainty patterns. Optimized for efficiency and memory usage. \u0026#34;\u0026#34;\u0026#34; def __init__( self, tokenizer, think_start_token: int, think_end_token: int, uncertainty_phrases: Dict[str, float], uncertainty_threshold: float = 5.0, growth_strategy: str = \u0026#34;linear\u0026#34;, growth_factor: float = 1.0, max_think_tokens: int = 1000, debug: bool = False ): \u0026#34;\u0026#34;\u0026#34;Initialize the uncertainty-based think logits processor.\u0026#34;\u0026#34;\u0026#34; self.tokenizer = tokenizer self.think_start_token = think_start_token self.think_end_token = think_end_token self.uncertainty_threshold = uncertainty_threshold self.growth_strategy = growth_strategy self.growth_factor = growth_factor self.max_think_tokens = max_think_tokens self.debug = debug # Process uncertainty phrases and build efficient lookup structures self.uncertainty_phrases = {} self.phrase_by_length = defaultdict(list) # Group phrases by length for efficient lookup for phrase, score in uncertainty_phrases.items(): token_ids = tuple(tokenizer.encode(phrase, add_special_tokens=False)) self.uncertainty_phrases[token_ids] = score self.phrase_by_length[len(token_ids)].append(token_ids) # Sort by length for more efficient searching (shorter phrases first) self.sorted_lengths = sorted(self.phrase_by_length.keys()) self.max_phrase_length = max(self.sorted_lengths) if self.sorted_lengths else 0 # State tracking - optimized data structures self.accumulated_uncertainty = 0.0 self.last_processed_position = 0 # Track last processed position instead of full input length self.detected_phrase_positions = set() # Use set for O(1) lookup, store positions only # Pre-allocate tensor for logits modification to avoid repeated allocation self._force_think_end_logits = None if self.debug: print(f\u0026#34;Initialized with {len(uncertainty_phrases)} uncertainty phrases\u0026#34;) print(f\u0026#34;Max phrase length: {self.max_phrase_length}\u0026#34;) print(f\u0026#34;Uncertainty threshold: {uncertainty_threshold}\u0026#34;) def _find_uncertainty_phrases(self, tokens: List[int], start_scan_pos: int = 0) -\u0026gt; List[Tuple[int, int, float]]: \u0026#34;\u0026#34;\u0026#34; Efficiently find uncertainty phrases in tokens using optimized algorithm. Returns list of (start_pos, end_pos, uncertainty_score) tuples. \u0026#34;\u0026#34;\u0026#34; found_phrases = [] tokens_len = len(tokens) # Only scan from the specified start position for pos in range(start_scan_pos, tokens_len): # Check phrases of each length, starting from shortest for phrase_len in self.sorted_lengths: if pos + phrase_len \u0026gt; tokens_len: break # Can\u0026#39;t fit this phrase length # Extract candidate phrase candidate = tuple(tokens[pos:pos + phrase_len]) # Check if this phrase exists in our lookup for this length if candidate in self.phrase_by_length[phrase_len]: # Verify it\u0026#39;s actually in our main dictionary (should always be true) if candidate in self.uncertainty_phrases: found_phrases.append((pos, pos + phrase_len - 1, self.uncertainty_phrases[candidate])) return found_phrases def __call__( self, input_ids: List[int], logits: torch.Tensor ) -\u0026gt; torch.Tensor: \u0026#34;\u0026#34;\u0026#34;Process logits based on uncertainty in thinking.\u0026#34;\u0026#34;\u0026#34; # Convert input_ids to list if it\u0026#39;s a tensor (avoid repeated conversion) if isinstance(input_ids, torch.Tensor): input_ids = input_ids.tolist() # Early exit if no uncertainty phrases configured if not self.uncertainty_phrases: return logits # Check if we\u0026#39;re in a thinking section try: think_start_pos = len(input_ids) - 1 - input_ids[::-1].index(self.think_start_token) in_thinking = self.think_end_token not in input_ids[think_start_pos:] except ValueError: # No think_start_token found return logits if in_thinking: thinking_tokens = input_ids[think_start_pos + 1:] tokens_since_think = len(thinking_tokens) # Only process new tokens since last call new_tokens_start = max(0, self.last_processed_position) if tokens_since_think \u0026gt; new_tokens_start: # Calculate scan start position to catch phrases spanning the boundary scan_start = max(0, new_tokens_start - self.max_phrase_length + 1) # Find new uncertainty phrases new_phrases = self._find_uncertainty_phrases(thinking_tokens, scan_start) # Process only truly new phrases (not already detected) for start_pos, end_pos, base_score in new_phrases: phrase_id = (start_pos, end_pos) if phrase_id not in self.detected_phrase_positions: # Apply growth strategy if self.growth_strategy == \u0026#34;poly\u0026#34;: uncertainty_increase = base_score * (tokens_since_think ** self.growth_factor) else: # linear or default uncertainty_increase = base_score self.accumulated_uncertainty += uncertainty_increase self.detected_phrase_positions.add(phrase_id) if self.debug: phrase_tokens = thinking_tokens[start_pos:end_pos + 1] phrase_text = self.tokenizer.decode(phrase_tokens) print(f\u0026#34;Detected: \u0026#39;{phrase_text}\u0026#39; pos:{start_pos}-{end_pos} \u0026#34; f\u0026#34;score:+{uncertainty_increase:.2f} total:{self.accumulated_uncertainty:.2f}\u0026#34;) # Update last processed position self.last_processed_position = tokens_since_think # Check if we should force \u0026lt;/think\u0026gt; should_force_end = ( self.accumulated_uncertainty \u0026gt;= self.uncertainty_threshold or tokens_since_think \u0026gt;= self.max_think_tokens ) if should_force_end: if self.debug: reason = (\u0026#34;uncertainty threshold\u0026#34; if self.accumulated_uncertainty \u0026gt;= self.uncertainty_threshold else \u0026#34;max tokens\u0026#34;) print(f\u0026#34;Forcing \u0026lt;/think\u0026gt; due to {reason}\u0026#34;) # Efficiently create force-end logits (reuse tensor if possible) if self._force_think_end_logits is None or self._force_think_end_logits.shape != logits.shape: self._force_think_end_logits = torch.full_like(logits, float(\u0026#39;-inf\u0026#39;)) else: self._force_think_end_logits.fill_(float(\u0026#39;-inf\u0026#39;)) self._force_think_end_logits[self.think_end_token] = 1.0 return self._force_think_end_logits else: # Reset state when not in thinking mode or when thinking has ended self._reset_state() return logits def _reset_state(self): \u0026#34;\u0026#34;\u0026#34;Reset the processor state efficiently.\u0026#34;\u0026#34;\u0026#34; self.accumulated_uncertainty = 0.0 self.last_processed_position = 0 self.detected_phrase_positions.clear() # More efficient than creating new set def reset(self): \u0026#34;\u0026#34;\u0026#34;Reset the processor state.\u0026#34;\u0026#34;\u0026#34; self._reset_state() # Also clear the cached tensor self._force_think_end_logits = None Usage\nfrom vllm import LLM, SamplingParams from transformers import AutoTokenizer # Define uncertainty phrases uncertainty_phrases = { # Chinese uncertainty phrases \u0026#34;可能我哪里算错了\u0026#34;: 2.0, \u0026#34;我不确定\u0026#34;: 1.5, \u0026#34;让我想想\u0026#34;: 1.0, \u0026#34;可能\u0026#34;: 0.5, \u0026#34;也许\u0026#34;: 0.5, \u0026#34;或许\u0026#34;: 0.5, \u0026#34;等等\u0026#34;: 0.8, \u0026#34;我需要重新思考\u0026#34;: 1.2, \u0026#34;有点困惑\u0026#34;: 1.0, \u0026#34;我搞错了\u0026#34;: 1.5, \u0026#34;我可能理解错了\u0026#34;: 1.8, \u0026#34;这个有点棘手\u0026#34;: 1.0, \u0026#34;需要再想想\u0026#34;: 1.0, \u0026#34;可能我的分析有误？\u0026#34;: 2.0, \u0026#34;可能我的理解有误？\u0026#34;:2.0, \u0026#34;？\u0026#34;:1.0, \u0026#34;正确？\u0026#34;:1.0, \u0026#34;或者\u0026#34;: 1.0, \u0026#34;这似乎不符\u0026#34;:0.8, # English uncertainty phrases \u0026#34;I\u0026#39;m not sure\u0026#34;: 1.5, \u0026#34;I need to reconsider\u0026#34;: 1.2, \u0026#34;Let me think\u0026#34;: 1.0, \u0026#34;maybe\u0026#34;: 0.5, \u0026#34;perhaps\u0026#34;: 0.5, \u0026#34;wait\u0026#34;: 0.8, \u0026#34;hmm\u0026#34;: 0.7, \u0026#34;let me check again\u0026#34;: 1.2, \u0026#34;I might be wrong\u0026#34;: 1.5, \u0026#34;I\u0026#39;m confused\u0026#34;: 1.2, \u0026#34;I\u0026#39;m not certain\u0026#34;: 1.5, \u0026#34;?\u0026#34;:1.0, \u0026#34;That can\u0026#39;t be right\u0026#34;:1.0, \u0026#34;Wait\u0026#34;:0.7, \u0026#34;right?\u0026#34;:0.5 } # Setup llm = LLM(model=\u0026#34;Qwen/Qwen3-4B\u0026#34;) tokenizer = AutoTokenizer.from_pretrained(\u0026#34;Qwen/Qwen3-4B\u0026#34;) # Get token IDs think_start_token = tokenizer.convert_tokens_to_ids(\u0026#34;\u0026lt;think\u0026gt;\u0026#34;) think_end_token = tokenizer.convert_tokens_to_ids(\u0026#34;\u0026lt;/think\u0026gt;\u0026#34;) uncertainty_processor = UncertaintyThinkLogitsProcessor( tokenizer=tokenizer, think_start_token=think_start_token, think_end_token=think_end_token, uncertainty_phrases=uncertainty_phrases, uncertainty_threshold=5.0, growth_strategy=\u0026#34;linear\u0026#34;, max_think_tokens=1000, debug=False ) sampling_params = SamplingParams( n=5, temperature=0.7, max_tokens=5000, # top_p = 0.95, # top_k = 20, logits_processors=[uncertainty_processor] ) Budget Guidance The paper Steering LLM Thinking with Budget Guidance introduces the concept of Budget Guidance.\nDrawing on the idea of Classifier Guidance from Diffusion Models, this approach does not force the model to output specific content or a particular format. Instead, it \u0026ldquo;softly\u0026rdquo; guides the generation process so that it naturally converges toward a target length. It does this through a lightweight predictor module (Auxiliary Predictor) that, at each token the LLM generates, predicts the length distribution of the remaining thinking process.\nBased on Bayes\u0026rsquo; theorem, the budget-constrained generation problem p(next token | context, remaining length \u0026lt;= budget) is transformed into the form p(next token | context) * P(remaining length \u0026lt;= budget | context, next token):\n$$ p(Y_t|X,Y_{\u0026lt;t}, L_t\\leq \\bar{l}-t) \\to p(Y_t|X,Y_{\u0026lt;t})\\cdot Pr(L_t \\leq \\bar{l}-t|X,Y_{\u0026lt;t},Y_t) $$\nHere p(Y_t|X,Y_{\u0026lt;t}) is the original LLM\u0026rsquo;s output probability, and Pr(L_t \\leq \\bar{l}-t|X,Y_{\u0026lt;t},Y_t) is the guidance score provided by the Auxiliary Predictor.\nIn this paper, the Auxiliary Predictor uses a BERT-base model. Rather than outputting a single length prediction, it models the logarithm of the remaining length as a Gamma distribution. BERT takes as input all hidden states from the LLM\u0026rsquo;s final layer at each generation step, and outputs two parameters — the Gamma distribution\u0026rsquo;s shape parameter lambda and rate parameter alpha. Using the Gamma distribution\u0026rsquo;s CDF, it computes a probability value based on the current remaining budget (target total budget - length generated so far) — this is the guidance score.\nPredictor training: On OpenR1-Math-220k, a complete thinking process is truncated at different positions, and the predictor learns to estimate the length required from the truncation point to the end. The training objective is to maximize the log-likelihood of the predicted Gamma distribution, making the predicted distribution as close as possible to the true remaining length.\nIs Longer Thinking Really Better? In Lost at the Beginning of Reasoning, the authors discovered that the model\u0026rsquo;s reasoning process has an extremely strong dependency on the initial reasoning step. Once the first step goes wrong, the model struggles to correct the error in subsequent steps, which means the final answer is highly likely to be wrong too — hence \u0026ldquo;Lost at the Beginning.\u0026rdquo;\nBuilding on this finding, the authors proposed a sampling strategy called Early Pruning. The core idea: first generate multiple short candidates for the first reasoning step, then use a reward model to quickly evaluate their quality, keep only the top few, and continue generating the full chain of thought exclusively along those promising paths.\nThey also created a new benchmark called LaBoR (Lost at the Beginning of Reasoning) — a benchmark specifically designed to measure how well long chain-of-thought models can self-correct when the opening move is flawed. Each sample consists of a question paired with a deliberately flawed first reasoning step.\nClosing Thoughts This article has focused on how to effectively control an LLM\u0026rsquo;s thinking budget, though it\u0026rsquo;s worth noting that the discussion has primarily centered on inference-side solutions. The core advantage of these methods lies in their flexibility and \u0026ldquo;plug-and-play\u0026rdquo; nature — they require no modifications to the model itself, making them ideal for local deployment and rapid validation.\nThis, however, is also the limitation of this article. Inference-side control is more like an \u0026ldquo;external constraint,\u0026rdquo; which inevitably feels somewhat crude or lacks robustness. A more comprehensive, more fundamental thinking-budget solution necessarily requires the interplay between inference and training. In particular, during post-training, markers for different reasoning intensities can be introduced alongside specialized training, so that on the inference side, more efficient thinking can be activated through instructions.\nReferences Claude Code: Best practices for agentic coding\nGemini API: Set a thinking budget\nQwen3: Limit thinking length\nQwen3: Thinking budget\nLimiting Qwen 3\u0026rsquo;s Thinking\nLimit thinking tokens #15418\nToken-Budget-Aware LLM Reasoning\nSteering LLM Thinking with Budget Guidance\nLost at the Beginning of Reasoning\n","permalink":"https://niraya666.github.io/en/posts/understanding-llm-thinking-budget-control/","summary":"A practical analysis of LLM overthinking and thinking budget control solutions, comparing reasoning length and compute budget configurations across mainstream model APIs.","title":"Stop Overthinking: A Practical Guide to LLM Thinking Budget Control"},{"content":"About Me Post-modern human AI practitioner and lifelong learner Work enthusiast World explorer INFJ Reading, listening, creating Welcome to my personal blog!\n","permalink":"https://niraya666.github.io/en/about/","summary":"\u003ch3 id=\"about-me\"\u003eAbout Me\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003ePost-modern human\u003c/li\u003e\n\u003cli\u003eAI practitioner and lifelong learner\u003c/li\u003e\n\u003cli\u003eWork enthusiast\u003c/li\u003e\n\u003cli\u003eWorld explorer\u003c/li\u003e\n\u003cli\u003eINFJ\u003c/li\u003e\n\u003cli\u003eReading, listening, creating\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eWelcome to my personal blog!\u003c/p\u003e","title":"About"},{"content":"","permalink":"https://niraya666.github.io/en/travel-map/","summary":"","title":"Travel Map"}]