This page explains how Niro measures tokens, speed and accuracy, and how you can measure them on your own work. It does not give typical savings: they change from one task to the next and from one person to the next. Where a sentence is reasoning rather than measurement, it says so.

Words used on this page:

How tokens are counted

Characters, divided by 2.09. Niro counts the characters of every answer it returns, and turns them into tokens at 2.09 characters a token. That rate was measured against the model's own token counts, in Keycloak A/B transcripts, on results of more than 2,000 characters each. Over 69 Niro answers the median was 2.09 (middle half 1.83 to 2.25). Over 367 shell-tool results it was 2.10. Code and file paths pack more tokens into a character than English prose, for which 4 characters a token is the usual rule of thumb.

The end-of-task line uses this rate on both of its sides, so the comparison between them does not depend on it.

How the by-hand figure is estimated

The end-of-task line and niro impact compare what Niro's answers cost with an estimate of finding the same things with text searches and file reads. The estimate follows how agents with only shell tools were seen to work: one search per name asked about, then about 40 lines read around each place an answer showed you, and one more search for what a name search cannot reach, such as the classes that run behind an interface. Only what you were shown is priced. The full method, with a worked example of the arithmetic, is in Check what Niro did.

How the estimate is checked. It is compared with real A/B runs: what the shell-only session of a task really read, less what the Niro session of the same task still read with shell tools. In our A/B runs so far, added up over the tasks that printed a token figure, the two came close. For one task they can differ by several times either way, because the two sessions did not make the same searches. So read a single line's "instead of" as a rough order of size, not as a measurement.

How look-ups and round trips are counted

niro impact also counts, from the answers themselves, how many searches and file reads they stood for, and how many waits one after another an agent working by hand would have needed (a walk three levels deep needs at least three searches in a row). Both are counted from what each answer covered; neither is typed by the agent. How each is counted is in section 3 of Check what Niro did.

How the answer input was measured

The answer input asks for a short form of an answer, such as a count, the names, or the places only (Get shorter answers). To decide which forms to build, recorded Niro calls from real sessions were read together with what the agent did next. For each call, the smallest short form that still held what the agent then used was applied to the real answer text, and the tokens before and after were compared. The shapes were then run again on the release engine, on the calls whose code still existed, to check that estimate against real answers.

Why a short answer pays more than once. This is reasoning, not a measurement. An answer stays in the conversation for every later turn, so a shorter answer is cheaper on every turn that follows it, not only on the turn that asked for it.

How we run an A/B, and how you can

What an A/B is. The same task is run twice: once with Niro and once without, with everything else the same. Ours are run like this:

Why Niro over MCP can save less than Niro from the shell. An MCP answer enters the conversation whole. A shell command's output can be cut with grep or head before it does. The answer input is how an agent asks for a shorter answer inside MCP.

A run can show that the answer key is wrong. In one of our tasks, both arms, in both runs, left out tests that the key named, and said why: at run time those tests never reach the method. Reading the code confirmed it, so the key was corrected and both arms were scored against the corrected key. When the two arms agree against your key, check the key against the code before you score.

Limits we keep in mind. With one or two runs per task, one fact or a few cents on one task is within run-to-run noise, so compare totals over many tasks. Results from one agent version should not be compared with another version's. Our runs so far ask questions about code; none measured a full edit with its tests.

To run your own, with the switch that turns Niro off for one Claude Code session and the numbers worth comparing, see section 4 of Check what Niro did.

How accuracy pays off over a long session

What is measured. In our Keycloak A/B runs, cache reads were most of the tokens in both arms. So most of a session's tokens are the conversation being re-read on each turn, and anything that enters the conversation is paid for again on every later turn.

What follows, as reasoning (not measured). A missed caller costs more than the search that missed it:

  1. The agent lists callers with a text search and misses one that goes through an interface. In one of our Keycloak A/B sets, every fact the shell-only agent missed was of this kind.
  2. It edits the method and updates the callers it found. The missed caller now passes the wrong value.
  3. A test fails later in the session, or the bug ships. The agent goes back: it searches again, reads the files again, and reads the failing test's output. Each of those results enters the conversation.
  4. Every later turn re-reads all of that. A longer conversation also reaches the agent's context limit sooner, and the agent then compacts it (replaces the history with a summary), which drops detail it may need.

A list that says "complete" when it is, and "PARTIAL" with what was cut when it is not, avoids step 3. Niro's lists also have known gaps, and they are listed in FAQ: wrong or missing answers; the playbooks show where to check with a text search.

Speed

What Measured Where
A Niro call through MCP, warm 0.04 to 0.07 s the code-mode timing run, 10 calls each
The same call through niro ask (a new process each time) about 0.45 s same run
Full build of Keycloak (9,103 files) 182 s same run, machine load about 2
Full build of Keycloak, for these docs 3 min 17 s the build log, 4 parse workers, load not recorded

A build's time depends on the size of your code, the number of cores and how busy the machine is.

What is not measured yet

Check what Niro did (the end-of-task line, niro impact, and how to run your own A/B), Get shorter answers, How Niro fits your workflow.