# Tokens, speed and accuracy

**This page explains how Niro measures tokens, speed and accuracy, and how you can measure them on your own work.** It does not give typical savings: they change from one task to the next and from one person to the next. Where a sentence is reasoning rather than measurement, it says so.

Words used on this page:

- a **token** is the unit a language model reads and is billed by, roughly half a word of code;

- a **session** is one conversation with a coding agent;

- **cache reads** are the tokens of the conversation so far, which the model re-reads, at a lower price, on every turn;

- **cache writes** are new tokens added to that stored conversation, billed once at a higher price;

- **MCP** (the Model Context Protocol) is the standard way a coding agent calls outside tools such as Niro's.

## How tokens are counted

**Characters, divided by 2.09.** Niro counts the characters of every answer it returns, and turns them into tokens at 2.09 characters a token. That rate was measured against the model's own token counts, in Keycloak A/B transcripts, on results of more than 2,000 characters each. Over 69 Niro answers the median was 2.09 (middle half 1.83 to 2.25). Over 367 shell-tool results it was 2.10. Code and file paths pack more tokens into a character than English prose, for which 4 characters a token is the usual rule of thumb.

The end-of-task line uses this rate on both of its sides, so the comparison between them does not depend on it.

## How the by-hand figure is estimated

The end-of-task line and `niro impact` compare what Niro's answers cost with an estimate of finding the same things with text searches and file reads. The estimate follows how agents with only shell tools were seen to work: one search per name asked about, then about 40 lines read around each place an answer showed you, and one more search for what a name search cannot reach, such as the classes that run behind an interface. Only what you were shown is priced. The full method, with a worked example of the arithmetic, is in [Check what Niro did](https://help.niroai.dev/cli/evaluate-what-niro-did/).

**How the estimate is checked.** It is compared with real A/B runs: what the shell-only session of a task really read, less what the Niro session of the same task still read with shell tools. In our A/B runs so far, added up over the tasks that printed a token figure, the two came close. For one task they can differ by several times either way, because the two sessions did not make the same searches. So read a single line's "instead of" as a rough order of size, not as a measurement.

## How look-ups and round trips are counted

`niro impact` also counts, from the answers themselves, how many searches and file reads they stood for, and how many waits one after another an agent working by hand would have needed (a walk three levels deep needs at least three searches in a row). Both are counted from what each answer covered; neither is typed by the agent. How each is counted is in section 3 of [Check what Niro did](https://help.niroai.dev/cli/evaluate-what-niro-did/).

## How the `answer` input was measured

The `answer` input asks for a short form of an answer, such as a count, the names, or the places only ([Get shorter answers](https://help.niroai.dev/cli/answer-shapes/)). To decide which forms to build, recorded Niro calls from real sessions were read together with what the agent did next. For each call, the smallest short form that still held what the agent then used was applied to the real answer text, and the tokens before and after were compared. The shapes were then run again on the release engine, on the calls whose code still existed, to check that estimate against real answers.

**Why a short answer pays more than once.** This is reasoning, not a measurement. An answer stays in the conversation for every later turn, so a shorter answer is cheaper on every turn that follows it, not only on the turn that asked for it.

## How we run an A/B, and how you can

**What an A/B is.** The same task is run twice: once with Niro and once without, with everything else the same. Ours are run like this:

- each task has an answer key, a list of facts read from the source by hand before any run;

- each session runs in headless Claude Code, on one pinned version with auto-update off, on the same model;

- each session starts with no project instructions and no memory, and with web tools off;

- the "without" arm has shell and file tools only; the "with" arm has the same tools plus Niro, over MCP or from the shell;

- in the "with" arm, `mark_task_complete` is blocked, so the end-of-task call a real session makes is not in its cost;

- in our latest runs, the "with" arm also had six lines of guidance asking it to use the `answer` input where it fits. niro-cli does not install those lines, and no run measured how much they helped;

- a blind reviewer marks each fact found or missed, with tool names removed from the answers, and counts anything invented;

- cost, wall time, tool calls, cache writes, cache reads and output tokens are read from each session's own result event.

**Why Niro over MCP can save less than Niro from the shell.** An MCP answer enters the conversation whole. A shell command's output can be cut with `grep` or `head` before it does. The `answer` input is how an agent asks for a shorter answer inside MCP.

**A run can show that the answer key is wrong.** In one of our tasks, both arms, in both runs, left out tests that the key named, and said why: at run time those tests never reach the method. Reading the code confirmed it, so the key was corrected and both arms were scored against the corrected key. When the two arms agree against your key, check the key against the code before you score.

**Limits we keep in mind.** With one or two runs per task, one fact or a few cents on one task is within run-to-run noise, so compare totals over many tasks. Results from one agent version should not be compared with another version's. Our runs so far ask questions about code; none measured a full edit with its tests.

To run your own, with the switch that turns Niro off for one Claude Code session and the numbers worth comparing, see section 4 of [Check what Niro did](https://help.niroai.dev/cli/evaluate-what-niro-did/).

## How accuracy pays off over a long session

**What is measured.** In our Keycloak A/B runs, cache reads were most of the tokens in both arms. So most of a session's tokens are the conversation being re-read on each turn, and anything that enters the conversation is paid for again on every later turn.

**What follows, as reasoning (not measured).** A missed caller costs more than the search that missed it:

- The agent lists callers with a text search and misses one that goes through an interface. In one of our Keycloak A/B sets, every fact the shell-only agent missed was of this kind.

- It edits the method and updates the callers it found. The missed caller now passes the wrong value.

- A test fails later in the session, or the bug ships. The agent goes back: it searches again, reads the files again, and reads the failing test's output. Each of those results enters the conversation.

- Every later turn re-reads all of that. A longer conversation also reaches the agent's context limit sooner, and the agent then compacts it (replaces the history with a summary), which drops detail it may need.

A list that says "complete" when it is, and "PARTIAL" with what was cut when it is not, avoids step 3. Niro's lists also have known gaps, and they are listed in [FAQ: wrong or missing answers](https://help.niroai.dev/cli/faq-wrong-or-missing-answers/); the playbooks show where to check with a text search.

## Speed

| What | Measured | Where |
| --- | --- | --- |
| A Niro call through MCP, warm | 0.04 to 0.07 s | the code-mode timing run, 10 calls each |
| The same call through `niro ask` (a new process each time) | about 0.45 s | same run |
| Full build of Keycloak (9,103 files) | 182 s | same run, machine load about 2 |
| Full build of Keycloak, for these docs | 3 min 17 s | the build log, 4 parse workers, load not recorded |

A build's time depends on the size of your code, the number of cores and how busy the machine is.

## What is not measured yet

- **Edits.** Every A/B so far asked questions about code; none measured a full change with its tests.

- **Many runs per question.** Our A/B runs ran each question once or twice per arm.

- **Helper agents.** No run in any Keycloak set started one, so the pattern where a helper does the reading and returns a summary is not measured either way.

## Related

[Check what Niro did](https://help.niroai.dev/cli/evaluate-what-niro-did/) (the end-of-task line, `niro impact`, and how to run your own A/B), [Get shorter answers](https://help.niroai.dev/cli/answer-shapes/), [How Niro fits your workflow](https://help.niroai.dev/cli/lifecycle-overview/).