That sounds like bad business. It isn't. Every prompt token your agent sends is a token the model has to read, attend to, and reason over. Most of those tokens are not your carefully crafted instructions nor user interactions. They are tool outputs: raw JSON from API calls, verbose log dumps, unfiltered search results, and whatever your RAG pipeline decided was relevant. Most of those tokens are unnecessary, and the LLM’s job is to figure out which ones matter.
By Alan Blount (@zeroasterisk) Google Cloud, Agent Platform
Do you really need all those tokens? Can we reduce by half and not lose quality? → Kinda.
We tested a context compression proxy in front of Gemini 3.8 Flash on Agent Platform (formerly Vertex AI) and measured four production shaped agent workloads. Prompt tokens dropped 76% with no loss of task accuracy, and time to first token improved. The solution we used for this article is headroom, a popular open source project, and we wanted to make sure it worked well with Gemini.
Then we tried to break it. Compression is not totally free, and the interesting cost is not latency.
Tool calls are the root cause, not your prompts
You probably know this already, but the context window is precious and there are a lot of “junk tokens” in there. Managing that context window for every turn of the LLM is most of the job for agent harnesses and agent frameworks, and more tokens is usually worse. So as a dev, you get a tight system prompt, great skills or few shot examples, curated to just what you need. Then your agent calls one tool and drops 50,000 tokens of possibly relevant raw JSON into the context window.
That output contains the information the model needs, buried inside field names the model already knows, array wrappers it does not care about, and structural repetition that adds zero signal. An SRE incident log is mostly repeating timestamp formats and severity tags. A 500 row analytics response repeats the same seven column names 500 times.
The standard response is to write a better tool. That is a great idea but the lift is high and sometimes you’re not in control of every possible use case your agent might handle. The more it “figures out how to get stuff done on it’s own” the less you are crafting perfect tool outputs. A generic context compression proxy like headroom handles whatever comes into the context window, prior to LLM inference. Here are some of the features:
- SmartCrusher handles JSON and structured data. It identifies statistical patterns in arrays and objects to map out repeated keys, enum distributions, and numeric ranges. Instead of sending 500 rows of raw JSON representation, it sends a compressed equivalent that preserves the values and their relationships while stripping the structural scaffolding. This is where the 70% to 90% savings come from on tool outputs.
- CodeCompressor uses AST-aware compression for code, preserving imports, function signatures, and type annotations while summarizing implementation bodies. It knows the difference between the structural information a model needs to reason about code and the line-by-line detail it can just reconstruct.
- Kompress-v2-base handles natural language text and unstructured content, applying a trained compression model in order to reduce prose while preserving meaning.
The whole pipeline is reversible. Headroom caches the originals locally so your data never leaves your machine, and the model can call headroom_retrieve to get the uncompressed version of any block if it needs the full detail. In practice, it rarely does.
What we measured
Headroom (0.37.0, Apache 2.0) is a context compression layer that sits between your agent and the model. It classifies each content block by type and routes it to a specialized compressor, then forwards the reduced payload upstream. You start it on a port and point your agent at it. Let’s start with benchmarks.
We ran four scenarios, each built from realistic tool outputs at realistic sizes:
SRE incident root cause. 350 log entries of Kubernetes and microservice output, plus Prometheus metrics, with a connection pool cascade failure buried inside.
Codebase security audit. Code search results across 40 files plus a git diff, containing a JWT signature verification bypass.
Enterprise BigQuery analytics. 500 tabular billing records with two fraud outliers hidden in the distribution.
Multi-turn RAG synthesis. 25 dense specification chunks describing an internal platform, queried for exact infrastructure values.
Both legs send byte identical payloads. Five repetitions each, on gemini-3.8-flash via the Vertex AI global endpoint, temperature 0.1, thinking disabled.
Results
Ground truth accuracy was 100% on both legs, in all four scenarios. Every scenario's token count was identical across all five repetitions, on both legs, so compression here is fully deterministic.
Median time to first token improved from 3,182 ms to 2,580 ms. End to end wall time was a wash: 6,692 ms against 6,738 ms. Prefill gets cheaper, and the local Python proxy spends that savings back on its own CPU. Three of the four scenarios saw faster TTFT. The exception was the smallest payload, the security audit, where 6,811 tokens are not enough bloat to pay for the proxy hop.
Modeled cost fell 69.7%. Treat that as an estimate, not an invoice: it is computed from stated rates of $0.75 per 1M input and $3.75 per 1M output, not read from billing.
Reproduce it
Compression is lossy, and that is a design choice
Every article about prompt compression tells you accuracy was preserved and moves on. Ours said 100% too. But it’s a good idea to explore possible failure mode.
The benchmark asks the model to find outliers. The compressor preserves outliers, error records, and statistical extremes by design. That is a benchmark shaped exactly like the tool being tested, and a 100% score on it proves less than it appears to.
So we asked a different question. We took the same 500 row table and asked for one ordinary record, usr_10250, sitting unremarkably in the middle of the distribution, and told the model to say NOT PRESENT rather than guess.
The baseline returned all five fields exactly: country DE, tier free, 5,258 monthly calls, $643.73 spend, last active 2026-08-02.
The compressed leg said:
That is the honest shape of an 87% reduction on tabular data. The compressor did not losslessly re-encode 500 rows into 6,152 tokens, because that is not possible. It kept the schema, the distribution, and the anomalies, and dropped the unremarkable middle. This could be a deal breaker.
Two things make this workable rather than disqualifying.
First, the model refused to fabricate. Faced with missing data and an explicit instruction to admit absence, it admitted the absence and did not hallucinate. This is a big win.
Second, this is what the retrieval half of compress-cache-retrieve exists for. The proxy keeps the original bytes in a local content addressed store and exposes a retrieval tool. If your agent uses this tool, it can recover from a missing row and pay the cost to retrieve the whole payload or a smaller raw chunk back on demand.
You have to decide if this is a trade you are willing to make. Everything is faster and cheaper, but because of the omitted content you require a model which doesn’t hallucinate and you will incur the cost of extra turns and tool usage to get the omitted rows, if you need them.
Wiring it up
The simplest deployment is a local proxy. You could run this on your laptop, with your deployed agent, or as a standalone service on Cloud Run or GKE.
Agent Platform (formerly Vertex AI) serves standard REST endpoints and the Google GenAI SDK lets you override the base URL, so the agent code does not change.
Here is what this looks like when building an agent using Google ADK with Gemini 3.8 Flash:
Your agent framework remains completely standard. The compression layer operates out-of-band as an infrastructure optimization, exactly where network and token governance belongs.
You can also try this with opencode or many popular agent harnesses today
The takeaway
Token reduction is the least interesting benefit. The real one is attention budget. Every token of boilerplate competes with every other token during inference, and 50,000 tokens of repeated JSON keys is 50,000 opportunities for the model to drift away from the thing you asked it.
Compression buys that budget back, at a price you should know before you ship it: your agent gets excellent answers about the shape of the data and can no longer look up an arbitrary row. Pick the workloads where that trade is obviously correct, start with the one tool in your stack that produces the messiest output, and measure both code paths.
Source: github.com/headroomlabs-ai/headroom
If you found this breakdown useful, check out our previous piece on agent loop resilience and long-horizon harness design. Follow @GoogleCloudTech for more deep dives from the builder trenches.