← FinOpsAid blog

Are we normal? Reading the Benchmarking page and its percentile bands

Percentiles, cohort size, k-anonymity suppression and your marker on the band — what the FinOpsAid Benchmarking page compares, and what a high percentile does and doesn't mean.

Benchmarking answers the question every executive asks after seeing a cost number for the first time: is that a lot? Internal trends can't answer it. A 12% rise means one thing if peers rose 20% and something else entirely if they fell.

The page is deliberately small, and two design decisions shape everything on it: it's opt-in, and it's k-anonymity gated.

The banner is the important part

At the top: "Cross-tenant benchmarking is opt-in and k-anonymity gated — cohorts below the threshold are suppressed."

Both halves matter.

Opt-in means you only appear in other organisations' cohorts if you chose to, and the page is empty until you do. An empty page reading "No benchmarking data (opt-in required)" is a settings state, not a failure.

K-anonymity gated means a cohort is only shown when enough organisations are in it that no individual one can be inferred. When the cohort is too small the card shows a grey cohort suppressed (k-anon) chip instead of the percentiles.

Reading one benchmark card

Each metric gets its own card:

1. Metric name at the top.
2. Your value, large, in compact notation.
3. p25 / p50 / p75 across the width — the quartile boundaries of the cohort.
4. The band, with a marker showing where you sit.
5. Your percentile and the cohort size underneath.

Read p50, not the mean. The median is robust to outliers, which matters enormously in cost data where one enormous organisation can drag an average somewhere useless.

Read the cohort size before the percentile. A 68th percentile against 34 organisations is a real position. The same figure against 6 is noise wearing a number's clothes.

What a high percentile does and doesn't mean

It means you're spending more per unit than most of the cohort on that metric. It does not mean you're wasting money.

Higher spend per developer can reflect a deliberate strategy — heavier AI tooling, faster CI, more parallelism — bought on purpose to buy velocity. The benchmark tells you that you're an outlier; whether that's the outlier you meant to be is a judgement only you can make.

The genuinely useful reading is a change in percentile over time. Drifting from the 50th to the 80th while your own absolute numbers looked stable means the market moved and you didn't. That's the signal an internal trend line structurally cannot give you.

The other useful reading is inconsistency across metrics. High percentile on cost per developer and low on adoption is a bad combination: expensive tooling that people aren't using. High on both is at least coherent.

Why the metric set is small

Cross-organisation comparison only works where the denominator means the same thing everywhere. "Cost per active developer" is comparable because both terms are defined identically for every tenant. "Cost per repository" is much weaker, because one organisation's monorepo is another's two hundred services.

The page shows the metrics that survive that test. A short list of comparable numbers is worth more than a long list of numbers that only look comparable. There's a fuller treatment of the method in benchmarking GitHub Copilot spend.

What people get wrong on this page

• Reading an empty page as broken. It's opt-in.
• Treating suppression as a data problem. It's a privacy guarantee, and it protects you too.
• Quoting a percentile without the cohort size. They're one number, not two.
• Assuming high percentile means waste. It means different. Whether it's wrong is your call.
• Comparing a single snapshot instead of tracking drift. Percentile movement over quarters is the real signal.

The 60-second read

• Check the cohort size on every card before reading its percentile.
• Compare your value to p50, not to the top of the band.
• Look for inconsistency across metrics — expensive plus low adoption is the finding.
• Note today's percentiles so you can compare them next quarter.
• Treat any card at the extremes as a question to answer, not a verdict to accept.

Frequently asked questions

Why does a benchmark card show no percentiles?

The cohort was too small to compare without letting one organization infer another's figures, so the comparison is suppressed and a k-anonymity chip appears instead. Your own value still shows. The same rule that hides a thin cohort from you is what stops anyone reconstructing your numbers.

Does a high percentile mean we are wasting money?

No, it means you are spending more per unit than most of the cohort. That can be a deliberate strategy — heavier AI tooling or faster CI bought to buy velocity. The benchmark tells you that you are an outlier; whether it is the outlier you intended is your judgement.

Why is the benchmarking page empty?

Cross-tenant benchmarking is opt-in. Until your organization opts in there is no cohort to compare against and the page says so. Opting in is also what places you in other organizations' cohorts, which is why it is a deliberate choice rather than a default.

Want to see where you sit? Connect your GitHub org — read-only and free during beta — or explore the demo first. Next: how to read the Reports page.