What actually cuts a coding agent's token use: 19 measured studies

Anyone who leans on a coding agent hits the same wall sooner or later: the five-hour window closes, or the weekly limit is gone by Thursday. The usual advice is to tell the agent to be brief, to read less, or to give it a search tool. We wanted to know which of these actually lowers what a task spends, so between 7 and 9 October 2026 we tested them one at a time against a fixed baseline.

This is the report, written for people who use these agents rather than for a journal. It keeps the failures, because they taught us more than the successes did, and it says plainly where a number stops meaning what it seems to mean.

A word on who is writing. We make AigoPilot, a Windows app that runs Claude Code, Codex and Grok sessions side by side. The research was done by our lab, AigoPilot Labs, to decide what to build next. None of what follows is in the app yet; the box further down says what is.

How we counted tokens

A coding task spends tokens in several places. The model reads instructions and source code, looks at what its tools return, writes code, and often goes round again after a test fails. Shrinking one of those does not make the task cheaper if another grows to fill the gap. So every study used one measure:

Task tokens   = input tokens + output tokens,
                summed over every first and corrective attempt

Reduction (%) = 100 x (1 - treatment task tokens / baseline task tokens)

A total across tasks is always the sum of one group against the sum of the other, never an average of percentages. Cached input is already part of input, and reasoning tokens, which are reported as part of output, were not counted twice. One Codex CLI call can hold several rounds between the model and its tools, so a count of calls is not a count of requests to the service.

Three distinctions kept us honest:

• Bytes are not tokens. A smaller excerpt or test log is a result about text size until the model's real usage has been measured.
• Tokens are not a bill. Cached input and output are priced and counted differently. We did not measure money or subscription quota.
• The task is not the research. Building tools, preparing experiments and our own conversations about them were outside the counters. We make no claim that this work has paid for itself.

Quality was part of the goal. Depending on the study, an output had to pass the existing tests unchanged, add permanent new tests, pass separate acceptance tests, show that its new tests fail on the original code, leave protected files alone, and survive a review of the actual change. Saving tokens while quietly changing something else did not count as a saving.

How the experiments were set up

The first study used a small made-up backend. The later ones used fixed snapshots of two of our own codebases: TokenLens, a small tool that reads agent transcripts, and FairRev, a much larger production monorepo whose chosen snapshot held 434 source and test files and 91,848 lines. Each study worked on particular modules, not the whole repository.

Every run started from a fresh copy. Within a study the requested model and reasoning setting stayed the same: gpt-6.1-sol at medium effort, through Codex CLI 0.160.1 in the main studies. The logs do not show which model the service actually resolved, so we report what we asked for, not what we can prove ran.

Our recording setup kept the prompts, answers, usage events, test output and code diffs of every run. Each study's protocol fixed the number of runs, their order, the attempt limit and the pass rule before the first model call. Where the design allowed it, the order of the two arms alternated. Pilots were kept apart from the confirmation studies that followed them, and failed attempts stayed in the totals.

Experiments ran in containers, away from the live repositories. Graders ran separately, with no network and no account credentials, and the coding model never saw them. When the infrastructure broke after the model had already answered, we evaluated the saved answer locally instead of paying for a new one.

What this is not: a large randomised trial. The tasks were chosen on purpose, service load and caching were outside our control, and the code review was done by our research assistant, who knew which arm each output came from. "Review" in this report means someone read the code. It does not mean a blind review by independent people.

What did not work: instructions, retrieval and shorter logs

The first idea was the obvious one: better instructions would stop the agent wandering through the code. We added a fixed block of working rules, called Benjamin-Plus, to the project's AGENTS.md. On the made-up backend, total tokens fell by 8.58% over 36 runs and every output passed. But only ten of eighteen pairs improved, and the interval across tasks ran from -3.5% to +18.2%. Promising, and uncertain.

On three real TokenLens tasks the same rules made no difference at all: 319,653 tokens against 319,640, a gap of thirteen tokens.

So we pulled the idea apart and tested each piece on its own:

Each row has its own baseline. The rows are not a ranking of products, and their savings do not add up.

Two of them deserve a closer look.

Serena, a code navigation tool that answers through the TypeScript language server, was set up with four read-only tools. It was not an embeddings search. In the first pilot the connection worked and the model simply never called it. Later studies made the model use it, and then use it before anything else. Neither saved tokens on the migration we tested. That tells you something about these workflows; it does not show that retrieval tools are useless in general.

Filtering the test output did exactly what it was built to do: in the treatment runs, raw test output shrank by 70.47%. Total tokens fell by 3.66%. Making one piece of text smaller is a different thing from making the task cheaper.

By this point a pattern was showing. Giving the agent another tool or another instruction did not stop it from doing what it did before. What might matter was how the work itself was divided. So the next experiments changed who runs the steps that need no judgement.

The change that cut total tokens by 87.56%

In the new arrangement, a local coordinator applies the files and runs the tests, and the model only returns an answer in a fixed format.

For mechanical migrations, a TypeScript tool first found every reference and prepared a patch. The model got the relevant code and the patch, checked them, and returned the complete test files as JSON. The coordinator applied that answer to an isolated copy, ran the visible checks and, when something failed, sent back the real failure. Attempts were capped and every one of them was counted.

The baseline was Codex the way you would normally use it: a writable project and its own tool loop. The new arm changed three things at once: how the context arrives, who runs things, and what shape the answer takes. The result belongs to that whole arrangement. We did not measure each part separately.

A six-run pilot cut total tokens by 75.62%. The confirmation used twelve fresh runs: two migrations, three repeats, two methods. One migration renamed a Shopify domain API across seven product files and kept the old name as an alias of the same function. The other moved a shared pricing API across four main files and left a separate public adapter as it was.

Total tokens fell by 87.56%. All six pairs improved and all twelve outputs passed. The rule fixed in advance asked for at least 20% overall and for each migration, a saving in every pair, and a full quality pass. It passed.

The checks were 139 selected existing tests, five separate acceptance tests per task, permanent regression tests and a review of the product and test changes. The full monorepo suite and the database tests were outside this evaluation.

Now the part that matters most. Input without the cached part fell by only 8.87%, and output by 25.67%. Nearly all of the big drop came from cached input, most likely because the normal agent sends its growing conversation again with each of its 60 tool calls and most of that is served from cache. So 87.56% is a true result for total reported tokens on these two migrations. It is not 87.56% off your bill, your quota or coding in general. The two arms were also timed differently enough that we make no claim about speed.

Beyond prepared patches

A separate pilot asked whether the same arrangement works when nobody has prepared the solution. The model got a real TokenLens bug, the whole main module, the relevant existing tests, the instructions and the package metadata, and wrote the fix itself. The coordinator applied it and ran the tests.

Over six fresh runs on three tasks, total tokens fell from 318,198 to 60,290, or 81.05%. Every pair improved and every output passed. But there was only one pair per task, output tokens went up by 92.03% and the time from start to finish went up by 38.65%. Writing out complete files costs output and time, even when the total is far lower.

A small, early result, then. It raised the next question: would exact edits, instead of whole files, make the structured arrangement cheaper still?

Shorter answers, almost the same total

We kept the structured arrangement and changed only the answer format. One arm returned whole changed files. The other returned exact pairs of old text and new text for existing files, and full content only for new ones.

A local applier checked paths, protected files and exact matches before writing anything. A missing or ambiguous match was refused. A compact answer was a proposal checked by a machine, never an order to replace a file with a fragment.

The six-run pilot looked good: 10.22% fewer total tokens, 53.62% less output, 44.03% less time. The eighteen-run confirmation told another story: 51.88% less output and 36.15% less time, but only 0.56% fewer total tokens. The arithmetic shows why:

Output saved:       41,906 - 20,167 = 21,739 tokens
Extra input:       181,109 - 160,509 = 20,600 tokens
Net saving:                             1,139 tokens

Total:             202,415 to 201,276 = 0.56% fewer

The compact arm needed two correction calls on top of its nine first calls; the whole-file arm needed one. Each correction sent the context again, and that ate almost all of the output saved. The 95% interval for the total ran from -23.15% to +34.12%, and the SQL task got more expensive overall. The confirmation failed its rule.

Review also found a performance bug in one compact answer: it copied a growing buffer over and over, which made a very long line take quadratic time. We documented and repaired it after the study, without rewriting the measured answer or counting the repaired copy as a success.

The lesson is a narrow one. Compact edits can cut generation and waiting time a great deal. Whether they cut the whole task depends on how much input goes in and how often a correction is needed. Note too that this comparison had a much tougher baseline than the 81.05% pilot: it put two structured formats against each other, not a structured workflow against a normal agent.

Corrections that send the current state once

Corrections came next. The old correction prompt could carry the first request, the model's previous answer and the current changed files all together. When that answer had already been applied, the same state went out two or three times.

The new builder keeps the task and the protection rules, includes each needed current file once, and drops the old answer only when its effect is already in those files. An answer that was refused or not applied stays, with the reason. Test records known to be passing can be shortened; failures, syntax errors, diffs, stack traces and anything unrecognised stay in full. The complete raw logs stay on disk.

A controlled test started both arms from the same four failure states: two SQL test failures we had seen earlier and two defects we planted on purpose. Across eight fresh correction runs, input fell from 97,368 to 69,455 tokens (28.67%) and total correction tokens from 98,411 to 70,440 (28.42%), with every pair better. The rule for the correction stage passed.

That test left out the first attempt on purpose. To see the effect on whole tasks, the next study ran twenty-four fresh runs from the original code across six tasks. Total tokens fell from 266,945 to 256,252, or 4.01%, short of the 10% required. Only five of twelve pairs improved and the interval included zero. Only five of the twenty-four runs needed a correction at all.

The two results agree with each other. If corrections are a small share of a task, cutting them by 28% cannot cut the task by 28%. With everything else unchanged, the overall saving would be the correction share times its reduction. Real runs are messier, since answer length and the number of corrections move as well.

We kept the new correction builder as a sound improvement. We do not present it as proof of a 28.42% saving on coding tasks in general.

A fairer evaluator, with the old results left alone

The whole-task study showed one more source of waste: the evaluator was too strict in one place. It accepted an explicit assertion failure as proof that a new test exposes the original bug. Some perfectly good tests made the old code throw an error before the assertion was reached, and passed on the fixed code. In those cases the evaluator asked for more work for no reason.

The revised evaluator accepts such a failure only when it can tie the new test and its stack trace to the changed product module. Import errors, syntax errors at load time, infrastructure faults, timeouts and anything unknown still need someone to look. Identical test bytes and untouched protected files are still required.

Replaying thirty saved calls offline found four earlier calls, 76,914 tokens in all, that came after the point where the new rule would have accepted the work. That is usage already recorded beyond a changed decision point, not a fresh saving we measured. The old study's scores and totals stay as they were. Passing the visible checks still does not replace separate acceptance tests or a review.

An earlier TypeScript confirmation had needed a documented correction to its grader, which had treated an internal import name as a public requirement. We kept both versions of that evidence and applied the corrected rule to every output, so a grader's bug was never mistaken for a bug in the product.

Less source code up front, tried on new tasks

With the duplication gone from corrections, we went back to the first prompt. An offline audit found no file sent twice in the first requests, so the opening was to leave out source code the task did not need.

The candidate prompts kept the task, the instructions, the package metadata and every included existing test. Only unrelated parts of the source were cut, each marked as an excerpt with the original file's hash and the line ranges kept. Three other tasks kept their whole source because cutting would not have helped. By text size alone the cut was 18.9%, which we did not report as a token saving.

A twelve-run pilot then compared full and hand-picked source on three tasks we had studied before. Both arms used the revised evaluator, the same answer format and the same correction rules, and corrections went back to complete current files.

Total tokens fell from 142,137 to 117,070 (17.64%), all twelve outputs passed and five of six pairs improved. But the first calls alone saved only 9.77%. Of the 25,067 tokens saved, 14,097 came from corrections, including one fewer correction in the trimmed arm. The pilot does not show that a shorter context reliably means fewer corrections.

So the confirmation used three tasks the model had never seen:

1. Unicode-aware LIKE matching with explicit escape rules.
2. Rolling time windows built from events that arrive in any order, with timestamp conversion and totals kept as they were.
3. The first and last time in a transcript, found in true time order when timestamps carry different time-zone offsets.

Eighteen fresh runs, three per method per task. Every run finished in one model call. Total tokens fell from 174,666 to 159,300, or 8.80%. Every pair improved, input fell by 9.99% and output barely moved. The 95% interval was 8.01% to 9.61%. With no corrections in either arm, this is the effect of the first prompt by itself.

And it still failed. The saving was under the 10% required, and review found a compatibility bug in one trimmed-context output, even though it passed the fixed tests.

That output skipped the old Date conversion for numeric timestamps. As a result it let through values outside the valid Date range and treated fractions of a millisecond differently. Timestamps of 0.9 and 1.1 milliseconds with a window of 1 gave two windows under the old behaviour and one under the new code.

We ran the same new compatibility tests against all six outputs for that task, the original code and a reference version. Only that one output failed. It was not repaired or swapped in the results. The finding is the reason the package below has a compatibility guard; it is not a reason to call the finished experiment a success.

What we built from it: Context Kit 1.0.0

The research ends in a package meant to be built into AigoPilot. It does not contain the whole research setup, and it does not run a coding agent by itself. The host program stays in charge of finding the files it is allowed to read, calling the model, applying patches, running tests and writing the final change.

The intended flow:

1. The host approves the task and the files.
2. The kit picks the first context carefully, and sends whole files when it is unsure.
3. The host calls the model, then checks and applies the patch.
4. Visible tests run, with proof that the new tests fail on the original code.
5. When a correction is needed, each current file goes in once.
6. Separate acceptance tests, compatibility checks and a review follow.
7. The result is a ready or not ready decision, backed by evidence.

Choosing source code without guessing

The selector parses supported JavaScript with the bundled Acorn parser; it never runs project code. It may leave out named top-level functions from supported files. Imports, classes, variables, other top-level statements and the local functions they appear to depend on stay in.

The starting points come from targets the host names or from symbols the task mentions. Tests, configuration and instructions are always sent whole. Unsupported syntax, unclear imports, dynamic code, a parser error or too small a saving all fall back to the whole file. Imported modules that are supplied stay whole too. The selector does not crawl the repository and does not fetch missing files.

Every result carries the source hashes and the line or byte ranges, and each fragment is checked against the original UTF-8 text. That makes a selection easy to inspect. It does not prove that the selection is complete for any JavaScript someone could write, which is why the fallback and the separate checks stay.

Six checks before "ready"

Corrections send each current file once. Answers already applied are not sent again; refused answers and useful failure output stay.

A change is ready only after six checks: visible tests, proof of the regression, separate acceptance, compatibility, untouched protected files and code review. One failure rejects it. Missing or unclear evidence means someone has to look. And "ready" publishes, deploys and writes nothing.

The checks are statements made by a trusted host and tied to the hashes of what was produced. The kit cannot prove by itself that the tests really ran; the host has to produce and check that evidence. The timestamp guard covers the TokenLens behaviour we know about. It is not a general compatibility suite.

What the final checks show

The package passes 43 Python tests and four tests of its Node adapter. Its offline fixtures rebuilt all eighteen saved answers exactly, confirmed that the code each historical edit touched was inside the new automatically chosen context, accepted the seventeen outputs that review had passed and rejected the one known bad output. We unpacked the ZIP into a separate folder and the acceptance suite passed again. Archive integrity, content hashes and the hashes of the bundled dependencies were all checked.

For the three saved task contexts, the selector produced these prompt sizes:

These are byte counts, not tokens. Replaying saved answers proves they can be rebuilt and checks the evidence supplied. It does not show that the model would write the same answers from the new prompts. The old candidates' test suites were not run again in this replay, and the final check made no model calls and no network calls.

What this means if you use a coding agent

• Measure the whole task. Add up input and output over every attempt, corrections included. A shorter answer or a smaller log is not a saving until the total moves.
• Look at cached and uncached input separately. A big drop in total tokens can be mostly cached input. Check uncached input and output before deciding that something saves money or quota.
• Let scripts do the mechanical part. Renames and other changes a tool can find and apply are where the large saving appeared. Have the agent review a prepared change instead of finding every reference itself through dozens of tool calls.
• Installed is not the same as used. Read the transcript to see whether the agent ever called the tool you added.
• Watch the correction loop. Do not resend the old answer next to the files it already changed. Send the current files once, and keep the real failure output.
• Test what should stay the same. A trimmed context can produce code that passes the new tests and quietly breaks old behaviour. Keep tests for the old behaviour as well as for the new feature.

What we can and cannot claim

The research gives a working split of jobs: local code prepares and applies the well-defined steps, the model reasons and writes code within a fixed contract, the coordinator supplies the current state and the real failures, and separate checks decide when the work is done.

Four findings held up across the studies:

1. Changing the workflow can save a lot on the right task. The 87.56% is confirmed for two specific mechanical migrations, with a wide gap between the total and uncached input.
2. Compressing one piece does not decide the total. Shorter logs, smaller answers and lighter corrections each touch a different part of the budget.
3. The correction loop and its evaluator matter. Duplicated state wastes input, and an evaluator that is too narrow asks for work a better rule would not.
4. Compatibility needs its own checks. An output can do what was asked, pass the existing tests and still change behaviour that should have stayed the same.

No single percentage sums up these studies. They have different baselines, tasks and denominators. The intervals describe variation inside small, fixed sets of tasks, not what you will get on another repository. Much of the work was exploration that changed course as it went, followed by fresh confirmation studies. It was never one experiment registered in advance.

Today Context Kit 1.0.0 is ready to be built in, and it is not an AigoPilot feature yet. The next evidence that counts will come from tests inside the app and from a separately approved live trial of automatic selection on new tasks and repositories, with every attempt counted, compatibility checked, fallbacks recorded, and tokens, cost and time kept apart.

Questions people ask

The studies behind this report

This report is also published on our Medium page.

This report summarises our own experiment records. It has not been peer reviewed. Each study has its own protocol, usage totals and raw files. In the order they ran:

• 7 October: Benjamin-Plus rules on a made-up backend (36 main runs)
• 7 October: Benjamin-Plus rules on three TokenLens tasks
• 8 October: Serena available but not required
• 8 October: Summarised test output, with an offline audit of command output
• 8 October: Instructions to read only what is needed
• 8 October: Serena required
• 8 October: Serena required before any other search
• 8 October: TypeScript rename tool, pilot
• 8 October: TypeScript rename tool, confirmation
• 8 October: Ready-made patch in the normal agent loop
• 8 October: Structured workflow on migrations, pilot
• 8 October: Structured workflow on migrations, confirmation
• 9 October: Structured workflow on bug fixes, pilot
• 9 October: Compact edits, pilot
• 9 October: Compact edits, confirmation
• 9 October: Correction context, controlled test
• 9 October: Correction context, whole tasks
• 9 October: Evaluator replay and audit of first prompts (offline, no model calls)
• 9 October: Less source code up front, pilot
• 9 October: Less source code up front, confirmation on new tasks
• 9 October: Context Kit 1.0.0, its integration contract, offline verification and acceptance report
• Codex CLI: github.com/openai/codex
• Serena: github.com/oraios/serena
• Acorn, the JavaScript parser in the kit: github.com/acornjs/acorn