How to reduce token spend in an AI agent pipeline

To reduce token spend in an AI agent pipeline, first log input, output and cached tokens for every model call with a run ID and a step name, so you can see which loop or tool step burns them. Then cut the waste you find: put the stable prompt prefix first so provider caching can hit, cap steps, tool calls, tokens and time per run, and drop old tool output from the context. Move easy steps to a smaller model or a batch tier only when a check shows quality holds, and report accuracy next to cost on repeated runs. The figures on this page come from public papers, not Slancha results.

Agent spend tends to pile up in a few places: a loop that re-reads a long tool result, a retry that never stops, or a prompt that breaks the cache. Find that step before you change models, and keep a check in place so a cheaper pipeline is still a working one.

By Slancha. Published 3 November 2026.

1. Find the step that burns the tokens

A monthly invoice tells you the total. It does not tell you which workflow, loop or tool call produced it. For that you need a record per model call: input tokens, output tokens, cached input tokens, the model, a run ID that follows the task through every step, and a step name such as "plan", "search" or "summarize".

The OpenTelemetry semantic conventions for generative AI give you vendor-neutral names for these fields. A model-call span carries gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.usage.cache_read.input_tokens, and the spec says the input count should include the cached tokens. Agent and tool spans add gen_ai.agent.name, gen_ai.tool.name and gen_ai.conversation.id, so you can sum tokens by run, by agent and by tool. The conventions are still marked Development, so pin the version you emit.

With that in place, group spend by step name and sort. The answer to "which loop burned the tokens?" is a query, not a guess. Keep the same run IDs in your traces, and you can turn the costly runs into test cases later, as the post on agent evals from production traces describes.

2. Put the stable prefix first

Each turn of an agent loop resends the system prompt, the tool definitions and any fixed examples. Prompt caching reuses the work a model already did on that repeated text. The Prompt Cache paper (Gim and co-authors, MLSys 2024) points out that many prompts share segments such as system messages, prompt templates and context documents, and that cached work for a segment can only be reused when it appears at the same position. The authors of The Complexity Trap (Lindenbauer and co-authors, 2025) note that several hosted model APIs charge up to ten times less for a cache hit than for a miss.

So order the prompt from most stable to least stable: system prompt, tool definitions, fixed examples, then the conversation, then this turn's input. Anything that changes per call, such as a timestamp, a user name or a shuffled tool list, belongs at the end. A date stamp at the top of the system prompt can turn every call into a cache miss. Your per-step logs will show it: cached input tokens near zero on a step that sends the same prefix every turn.

3. Give every run a budget

An agent with no limit can loop until something else stops it. Set caps per run on steps, tool calls, total tokens and wall time. When a run hits a cap, stop it with a clear status, return what it has, or hand it to a person. Do not let it retry from the top.

Log the cap that fired. If one workflow hits its step cap often, that points to a prompt or tool problem worth fixing, not a cap worth raising. Budgets also bound the cost of a bad deploy: a broken tool can only waste as many calls as the cap allows.

4. Stop paying for the same tokens twice

Every tool result you leave in the context is billed again on each later turn. A long file read or search result from early in the run is still input many turns later. In the preliminary runs of the study below, tool observations made up about 84 percent of the tokens in an average turn of a software engineering agent.

You do not always need a model to fix this. The Complexity Trap (Lindenbauer and co-authors, DL4C workshop at NeurIPS 2025) compared ways to manage an agent's context on a software engineering benchmark across five model configurations. The authors report that masking older environment observations halved cost relative to the raw agent while matching, and sometimes slightly exceeding, the solve rate of LLM summarization. Start with the simple version: keep the last few tool results in full and replace older ones with a short placeholder.

Then look for repeated work. Retry loops that resend the same failing call, tools that return the same data twice in one run, and steps that re-plan from scratch all show up in per-step logs as repeated input.

5. Send easy and slow work somewhere cheaper

Not every step needs your largest model. A cascade sends a request to a cheaper model first and moves it up only when the answer does not pass a check. FrugalGPT (Chen, Zaharia and Zou, 2023) reports that its cascade could match the performance of the best individual LLM it tested with up to 98 percent cost reduction, or improve accuracy by 4 percent at the same cost, in the authors' experiments. Those are their tasks, not yours. A cascade is only as good as the check that decides when to move up, so test the routing on your own cases before you trust it.

Work nobody waits for, such as nightly reprocessing or a regression run, can go through a provider's batch tier instead of live calls. The guide to cutting eval cost and the guide to faster evals apply the same caching and batching ideas to judge calls.

6. Measure accuracy next to cost

A cheaper pipeline that fails more tasks is not cheaper per task done. AI Agents That Matter (Kapoor and co-authors, 2024) found that a narrow focus on accuracy, without attention to other metrics, has left state-of-the-art agents needlessly complex and costly. The authors argue that cost and accuracy belong in the same report, and that reports should list input and output token counts next to dollar costs, so anyone can recompute the cost at current prices. Hold your own reports to the same rule: do not report a lower bill without the pass rate next to it.

Run the old and new pipeline on the same cases, more than once each, and report cost per run and pass rate side by side. Agents vary from run to run, so one pass proves little. The sample-size guide shows how many cases a given margin needs.

That check has a price. Every change you test means more agent runs and more judge calls. The guide to what LLM evals cost breaks that bill down, and the eval cost calculator lets you price it for your own volume.

Where Slancha fits

Slancha aims to make the expensive part of evals faster and cheaper. Checking that a cheaper agent pipeline still works is an eval, and its judge calls are part of the bill. You keep your current eval or observability platform for traces and review. Pilots by arrangement.

Request a pilot

Questions people ask

How do I track what an agent costs per workflow?

Log input, output and cached tokens for every model call, tagged with a run ID, a workflow name and a step name. The OpenTelemetry conventions for generative AI define standard attribute names for these counts. Sum by workflow for cost per run, and by step to find the loop or tool call that spends the most.

Will prompt caching help my agent?

It helps when each call resends a long, unchanged prefix, which agent loops do. Put the system prompt, tool definitions and fixed examples first, and anything that changes per call last. Cached work is reused only when the same text sits at the same position, so one changed character near the top can cause a miss.

Should I summarize old tool output or drop it?

Try dropping it first. One study found that masking older observations matched the solve rate of model summarization at lower cost on its benchmark. Keep the latest tool results in full, replace older ones with a short placeholder, and check the pass rate before and after.

Can I switch easy steps to a smaller model?

Yes, if a check confirms the smaller model still passes on those steps. Route by step, send hard cases to the larger model, and compare pass rates on the same cases over repeated runs before you switch.

How do I know a cheaper pipeline did not break something?

Run the old and new versions on the same test cases, more than once each, and report cost per run next to pass rate. A lower bill with a lower pass rate can cost more per task done.

Sources

Semantic conventions for generative AI (attribute registry docs/registry/attributes/gen-ai.md and "Semantic Conventions for GenAI agent and framework spans"), OpenTelemetry, status Development, 2026. Claims used: attribute names gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens, gen_ai.agent.name, gen_ai.tool.name, gen_ai.conversation.id; input tokens "SHOULD include cached tokens"; status Development. (The old opentelemetry.io gen-ai page now redirects here.)
AI Agents That Matter, Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir and Arvind Narayanan, arXiv:2407.01502, 2024. Claims used: narrow focus on accuracy without attention to other metrics; state-of-the-art agents needlessly complex and costly; cost measured alongside accuracy; reports should include input/output token counts in addition to dollar costs.
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, Lingjiao Chen, Matei Zaharia and James Zou, arXiv:2305.05176, 2023. Claims used: LLM cascade; matches the best individual LLM with up to 98 percent cost reduction, or improves accuracy by 4 percent at the same cost.
The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management, Tobias Lindenbauer, Igor Slinko, Ludwig Felder, Egor Bogomolov and Yaroslav Zharov, arXiv:2508.21433, 4th DL4C workshop at NeurIPS 2025. Claims used: five model configurations; observation masking halves cost relative to the raw agent while matching, and sometimes slightly exceeding, the solve rate of LLM summarization (abstract); observation tokens make up around 84 percent of an average agent turn in preliminary runs (section 1); several hosted APIs charge up to 10x less for cache hits than misses (section 5.2).
Prompt Cache: Modular Attention Reuse for Low-Latency Inference, In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal and Lin Zhong, MLSys 2024, arXiv:2311.04934. Claims used: prompts share segments such as system messages, templates and documents; cached attention states for a segment are reused only at the same position.

Checked against arXiv (2407.01502, 2305.05176, 2508.21433, 2311.04934) and the OpenTelemetry semantic-conventions-genai repository on 3 November 2026.