Compression and Token-Reduction Tools

26.1 Four Different Things Called Compression #

The category is muddled because four unrelated techniques share a single name. Each compresses something different, saves in a different lane, and fails in a different way.

Sorting a tool into the right row is the first analytical step because the economics differ by an order of magnitude. Output brevity acts on the most expensive lane, and generation itself is never discounted—caching reduces input only. Input compression acts on the cheapest lane and may be fighting a cached prefix that already costs a tenth of the base rate (§22), essentially reducing its actual benefit if not eliminating it.

That last point deserves emphasis because it is the interaction nobody advertises. Compressing a prefix that is already a cache hit is close to pointless: the cache has taken it from $2.00 per million tokens to $0.20, and halving what remains saves $0.10. The same cut is worth $1.00 on uncached input and $5.00 on output. Worse, if the compressor’s output varies between requests, it rewrites the prefix and destroys the cache, converting cheap reads into full-price misses. A compressor must be deterministic over stable content or it costs more than it saves.

26.2 The Research Base #

Prompt compression has a genuine academic literature behind it, and it proves more encouraging than most practitioners assume.

LLMLingua, from Microsoft Research, uses a small language model to identify and drop low-information tokens, reporting up to 20× compression with limited quality loss—on GSM8K, exact-match scores fell by 1.44 and 1.52 points at 14× and 20× respectively.⁵⁶ LLMLingua-2 reformulates compression as token classification, training a BERT-scale encoder by distillation, and reports better quality than its predecessor at equal ratios with lower latency.⁵⁶ LongLLMLingua adds question-aware compression that scores segment relevance against the query, reporting performance improvements of up to 21.4 percent at roughly 4× fewer tokens on NaturalQuestions.⁵⁶

That last result is the interesting one, and the mechanism is simply §12.2 restated: compression that removes distractors raises the signal-to-noise ratio, and the model performs better on the smaller prompt. Compression and curation are the same operation performed by different agents.

The counterweight comes from inside that same literature rather than from outside it. LLMLingua-2’s own reported single-document QA score falls from 35.5 at 3× to 29.8 at 5×—5.7 score points, a 16.1 percent relative decline—over a change in ratio that vendors describe as conservative. Those two figures are easiest to read where a later paper proposing a competing method re-tabulates them as its baseline, but that paper’s own table marks them as taken from the LLMLingua-2 authors rather than re-measured, so they are the method’s published behavior and not independent confirmation of it.^56,57 Practitioner writing on the tool landscape agrees qualitatively—useful reduction at moderate ratios, a quality cost at aggressive ones—without converging on numbers anyone can cite.⁵⁸ My own working band, offered as a heuristic and not as measurement, is to expect a third to two-thirds reduction to be roughly free on prose-heavy context, and to treat anything past four-fifths as a change requiring its own evaluation. [unmeasured]

The working rule is this: compression trades quality for ratio, and on the one extractive method measured above, raising the ratio from 3× to 5× cost 5.7 score points. Below some ratio you are mostly removing noise; above it you are removing signal. Be exact about what two scores can carry because it is less than the shape of that sentence suggests. They establish a drop between two settings and nothing more: two points fit a straight line as well as a curve—35.5 and 29.8 sit exactly on q(x) = 44.05 − 2.85x—so they do not establish curvature, a slope change, a cliff, or the location of a knee. The shape in figure 10 is drawn to illustrate the trade-off and is not traced from data. So treat “quality fell by this much between these two ratios, for this method on this task” as the claim, locate your own usable ceiling by measurement, and do not carry either the magnitude or any assumed shape onto a different task, model or compressor.

26.3 A Worked Example: Caveman #

One tool repays close examination, since it is widely adopted, spans three of the four categories above, and, unusually, publishes its own negative results.

Caveman began as a Claude Code skill instructing the agent to answer in terse, telegraphic prose, and its early releases carried a fixed 65 percent output ratio into their savings reports. Its own documentation then does something rare: it withdrew the number. A correction committed on September 8, 2026 states that no reviewed aggregate output-reduction result is published, that the 65 percent ratio had never had a committed reviewed result behind it, that current reports ignore those historical estimates while preserving the original history rows, and that the input cost the skill adds is not measured there either.⁵⁹ The standing caveats survive the withdrawal: the skill shrinks output tokens only, it adds input tokens of its own, and on already-terse workloads the result can be net negative.⁵⁹

The withdrawal is the instructive event. A figure that had been running in released reports for months turned out to rest on no committed comparison, and the same correction removed the input-side number anyone would need to turn an output reduction into a net one. What the project now tells its users to do instead is to compare provider-billed totals on the same task with the tool on and off, and to switch it off for any workload where the billed total goes up.⁵⁹

A second methodological point rides along with that one, and it generalizes well beyond one tool. A compression claim measured against an unprompted baseline tells you what the instruction bought, not what the tool did. The correct control is your current prompt plus “be concise,” which costs nothing and takes ten seconds.

The later releases move into structural compression and modality shift. A local proxy sits between the agent and the provider, typing each payload and routing it to a per-content-type compressor, including re-encoding JSON tool results as TOON (§3.1) when that measures smaller. Targets, which its documentation labels as inferred: roughly 70–90 percent on JSON, 85–95 percent on logs, 40–70 percent on code, 60–80 percent on diffs, 80–95 percent on search results, and 50–80 percent on text.⁵⁹ A pinned 54-run benchmark reports 33.2 percent fewer provider-reported input tokens than direct Claude Code while passing all eighteen exact-answer checks, labeled as a benchmark counterfactual rather than a measurement of real traffic.⁵⁹

The modality shift is the most striking mechanism, and the one most in need of scrutiny. Text is priced per token; a dense text slab rendered to a PNG is priced as image input. The project reports 55,413 estimated text tokens becoming 11,402 estimated image tokens on a dense payload, a 79 percent reduction, and applies the same trick to skill bodies—one skill going from 1,069 to 415 estimated tokens.⁵⁹ It also states the limit plainly: the technique only pays on dense, long-line content, sparse code with short lines is not profitable because the page carries more overhead than the text it replaces, and it runs only for models with measured render legibility.⁵⁹

To make the per-type targets concrete, consider what each compressor is actually doing. A log at 85–95 percent:

before (18 lines, ~240 tokens)
  2026-09-08T14:22:01Z INFO  starting worker pool size=8
  2026-09-08T14:22:01Z INFO  connected to postgres host=db-1
  2026-09-08T14:22:02Z INFO  processed batch id=1 rows=500
  ... 12 more INFO lines ...
  2026-09-08T14:22:09Z ERROR failed to commit batch id=14
    psycopg2.errors.SerializationFailure: could not serialize access
    at worker.py:212 in commit_batch

after (~30 tokens)
  [15 INFO lines elided]
  ERROR failed to commit batch id=14
    psycopg2.errors.SerializationFailure: could not serialize access
    at worker.py:212 in commit_batch

Nothing required to answer “why did this fail” was removed. That is the easy case, and it is why the log target is the most aggressive on the list.

Code at 40–70 percent is a different proposition:

before
  def commit_batch(conn, rows):
      cur = conn.cursor()
      try:
          cur.executemany(INSERT_SQL, rows)
          conn.commit()
      except SerializationFailure:
          conn.rollback()
          raise
      finally:
          cur.close()

after
  def commit_batch(conn, rows):   # body elided, 9 lines
      ...

The signature survives, while the behavior does not. If the agent’s task is “call this function,” the compression is free. If the task is “explain why batches fail intermittently,” the compressor just deleted the answer. This is why the code ratio is the conservative one, and why the recovery handle in §26.4 matters more here than anywhere else.

26.4 What These Tools Cost You #

Price is the least of it. Six further costs, ordered roughly by how often each is overlooked:

26.5 Evaluating a Savings Claim #

Six questions settle most claims, and asked in this order, most resolve after the second.

26.6 The Order of Operations #

Compression is the last thing to reach for rather than the first, since the free levers are considerably larger.

1. Delete what should not be there.        §14, §15  — free, improves quality
2. Move always-on content on-demand.       §14.1-2   — free, improves quality
3. Cut tool surface area.                  §17.1     — free, improves quality
4. Make the cache work.                    §22       — ~10× on input, no quality cost
5. Route by task class.                    §27       — 5-20× on the routed traffic
6. Add a brevity instruction.              §8.1      — free, immediate
7. Only now: compression tooling.          this §    — real, but bounded and lossy

Steps one through three reduce dilution and therefore improve output quality (§12.1). Step four is a pure win. Compression sits below all of them because it is the only step on the list that can make the model’s input worse.

A team that has not done steps one through six and is evaluating a compression proxy is optimizing the wrong term.

26.7 Failure Modes #

• Compressing a cached prefix. Trading a 0.1× read for a 1.25× write.
• Believing an output-token headline is a bill. Output is one lane, and on cache-heavy agentic workloads it is not the largest one by volume.
• No terse control arm. You measured the instruction, not the tool.
• Aggressive ratios on agentic code work. Compression research is dominated by question answering and summarization. It is not silent on code—LLMLingua-2 and EFPC both report a LongBench Code column, and LongLLMLingua carries code completion in its task inventory—but code completion over a fixed context is not editing and debugging a repository over a long horizon, and nothing cited here measures the second. Treat aggressive ratios on agentic code work as unmeasured rather than as the QA knee transferred. [unmeasured]
• Adding a proxy without reviewing it. It sees every request and every credential.
• Compressing before curating. The cheapest tokens to remove are the ones that should never have been in the window.