Does prompt compression actually cut your Copilot bill? Partly, and sometimes it raises it

GitHub featured Caveman, an open-source toolkit that compresses prompts to cut token usage. Fewer tokens is real. Fewer credits is conditional — and on one surface compression can cost you more.

GitHub's Open Source Friday this week featured Caveman, Julius Brussee's open-source toolkit that "compresses prompts to reduce token usage while preserving output quality." It's a good idea, cleanly scoped, and the framing is honest — it claims a token reduction, not a bill reduction.

Worth keeping those two claims separate, because on GitHub Copilot specifically the path from "fewer tokens" to "fewer credits" has three conditions on it, and one of them can run backwards.

Where it saves nothing at all

Start with the surface most developers spend most of their day in.

Code completions and Next Edit suggestions consume no AI credits. They are covered by the seat. Compressing the context sent to an inline completion makes it faster and possibly better, but it cannot make it cheaper, because its price was already zero.

This isn't a knock on compression — latency is a real thing to want. It just means that if your organisation's Copilot usage is mostly inline suggestions, prompt compression is a developer-experience project and should be justified as one. The credit line won't move.

Everything below applies only to the metered surfaces: agents, chat, and code review.

Where input isn't the dominant term

On metered surfaces, compression is attacking one component of a larger bill, and often not the biggest one.

An agent run is a loop: read, plan, edit, re-read, verify. Across that loop the same context gets resent on nearly every iteration, and the run also generates output tokens, which are typically priced higher than input. So the total for a task looks roughly like:

(input per call × number of calls) + (output per call × number of calls)

Compression shrinks the first factor of the first term. It does nothing to output, and nothing to the iteration count. If an agent takes eleven passes to land a change, halving the prompt is worth much less than getting it to land in four.

That ordering is the useful takeaway: iteration count is the bigger lever, and it's the one people don't pull, because it looks like a quality problem rather than a cost problem. Tighter instructions, better repository structure and clearer acceptance criteria reduce passes, and reducing passes reduces every term at once.

Token count and credit count are related, not identical. The cache effect is the reason a change that provably reduces tokens can still increase spend.

The condition that runs backwards

This is the one worth knowing before you deploy a compressor in front of an agent.

Cached input is dramatically cheaper than fresh input, and cache hits depend on a stable prefix — the beginning of your context being byte-identical to the previous request. That's why long agent loops are affordable at all: the bulk of the context is resent, and most of it is served from cache at a fraction of the fresh rate.

A compressor that rewrites the front of the context — summarising differently each pass, reordering sections, varying its own phrasing — breaks that prefix. Every iteration then re-pays the fresh rate on tokens that were previously nearly free. The token count on your side genuinely went down. The bill went up.

The fix isn't to avoid compression, it's to be careful where you apply it: keep the stable, reusable part of the context byte-stable and compress the volatile tail. System instructions, repository conventions and architectural context should be identical every time. Compress the specific task, the diff, the retrieved snippets.

Where compression genuinely wins

It's worth naming the cases where this pays clearly, because they're real and some of them are large:

Single-pass, large-context tasks. One-shot analysis over a big input, where there is no loop to cache into and input dominates the total. This is compression's best case by a distance.
CI and batch invocations. Automated calls that fire per commit or per pull request, at volume, with no human in the loop to notice quality drift. Multiply a modest per-call saving by every push.
Retrieval-heavy prompts. Anything that stuffs many retrieved chunks into context, where a meaningful share is near-duplicate or irrelevant. Compression here often improves output as well, because it raises signal density.

In all three, the shape is the same: input is the dominant term and the prefix isn't being reused.

How you'd actually know

You cannot verify any of this from GitHub's own data, and it's better to say so plainly.

The AI credit ledger is one number per user per day. It carries no input/output/cached split, no per-model breakdown, and no per-surface breakdown, and it lags roughly two to three days. So "we deployed compression and saved X" is not a claim GitHub will confirm or deny for you.

What that leaves is a before-and-after on cost per accepted change, which needs three things: a baseline captured before you deploy anything, a window long enough to clear the two-to-three-day lag, and no other change landing in the same window. GitHub retains most metric detail for around 100 days, so if the baseline wasn't snapshotted while it was happening, the comparison doesn't exist and can't be reconstructed afterwards.

Credit where it's due

Caveman's framing is the right one: it claims reduced token usage, not reduced spend, and it's open source so you can read what it does to your context rather than trusting a description. That's a considerably more honest posture than most things marketed at AI cost right now, including some things marketed at the same problem we work on.

Fewer tokens is a good goal. Just measure it in credits per landed change before you tell your CFO it's a saving — and check your cache hit rate before you tell anyone it worked.