← FinOpsAid blog

What breaks the glass? Designing GitHub cost alerts people don't mute

Most cost alerting fails in one of two ways — nothing fires, or everything does and the channel gets muted. Here's how to decide what actually deserves to interrupt a human, who it should reach, and why you should never page anyone on a modeled number.

Ask what your organization gets alerted on for GitHub spend and you'll usually get one of two answers. The first is a slightly embarrassed nothing. The second is a confident list of six alerts that all route to the same channel, which somebody muted in March.

Those are the same answer. An alert that nobody reads is functionally identical to an alert that doesn't exist, and it's worse in one respect: it lets everyone believe the problem is covered. Getting this right is less about thresholds than about deciding, deliberately, what is allowed to interrupt a person — and then defending that list.

Every alert is a withdrawal from a shared account

Attention is the scarce resource here, and it's pooled. The Actions cost alert competes with the on-call page, the failing deploy, and the security finding. Each new alert you add makes every existing alert slightly less likely to be acted on, which means the honest question for any proposed alert isn't "would this be useful to know?" — almost everything is — but "is this worth degrading the response to everything else?"

Most cost signals fail that test, and that's fine. Cost is rarely an emergency. The bill is monthly, the meter is continuous, and finding out about a spike tomorrow morning is nearly always sufficient. That single observation eliminates the majority of the alerts teams are tempted to build.

Three tiers, and only one of them interrupts

The structure that survives contact with reality has three levels, and moving something up a level should require an argument.

Interrupt. Reaches a person now, out of band. Reserved for things that are both expensive and reversible right now — a commitment you'll breach, a runaway you can stop. If the response is "we'll look at it in the morning", it isn't this tier.

Notify. Lands in a team channel or inbox same-day, no interruption. This is where nearly all cost signal belongs: a high-severity anomaly, a budget crossing 80%, a team's spend materially out of pattern. Someone sees it within the working day and that is fast enough.

Digest. A weekly or monthly roll-up nobody has to react to individually. Trends, drift, low-severity deviations, the long tail. Digests are where you put the things that matter in aggregate but never individually, and they're the release valve that keeps the other two tiers credible.

What belongs where, on GitHub

Mapping it concretely, for a typical org:

Interrupt is a short list. A hard budget you've committed to externally, at the point it will actually be breached rather than after. A runaway that's still running — a workflow retry loop, a self-hosted fleet scaled up and forgotten, Codespaces that stopped honouring their idle timeout. The common thread is that stopping it in the next hour saves real money.

Notify covers most of the value. High-severity spend anomalies with their attribution attached, budget utilisation crossing a warning band, a new SKU appearing on the bill for the first time, a bulk seat assignment. How you detect those deviations is its own problem, and it's the harder half — but once detected, same-day is the right speed.

Digest takes everything else: cost drivers period over period, seat utilisation drift, idle Codespaces inventory, storage creep, per-team trend. This is also the right home for the numbers you want reviewed but never chased.

An alert needs an owner before it needs a threshold

The most common reason a well-designed alert fails is routing. It fires correctly, into a channel owned by nobody, and dies there.

Cost alerts are unusually prone to this because spend is billed at the org level while the cause is local. "Org Actions spend is anomalous" sent to a finance distribution list produces a forwarded email and no fix. The same signal, attributed to the repository and the workflow and delivered to the team that owns it, produces a revert. That's the entire difference, and it's the practical reason attribution matters more than totals — you can't route what you can't attribute.

So write the owner down before the threshold. If no name can be attached, don't build the alert; put the signal in a digest instead and stop pretending it's actionable.

Design the recovery, not just the trigger

Alerts that fire cleanly and never resolve become background noise within a week.

Give each one an explicit clear condition, so a resolved problem visibly resolves. Add hysteresis — if a budget alert fires at 80%, don't re-fire it every time the figure oscillates across the line. Deduplicate by cause rather than by occurrence, so one runaway workflow generates one thread rather than forty. And decide what happens to an alert nobody acknowledged, because "it fired and everyone assumed someone else had it" is the most common post-mortem line in cost incidents.

Two alerts you should not build

First, don't alert on every repository. The instinct to add a per-repo cost threshold across three hundred repos produces a channel that's statistically guaranteed to fire daily and mean nothing. Rank instead: the top few deviations, once a day.

Second — and this one is a principle, not a preference — don't page anyone on a modeled number. A large share of per-repo, per-user and self-hosted-runner costs are allocations, not billed facts. They're perfectly good for spotting a trend and perfectly bad as the basis for waking someone at midnight, because the first time an interrupt turns out to have been an artefact of your allocation rule, the whole alerting system loses its licence. Interrupt on billed and metered quantities. Notify and digest on the modeled ones, clearly labelled as modeled.

A setup worth keeping

• Start with two alerts, not twelve. A budget breach and a top-severity anomaly. Add more only when a real miss justifies it.
• Route by cause, not by function. The team that made the change, not the team that owns the budget.
• Put the comparison in the message. The current value, the baseline it broke, and the link that proves it.
• Review fired alerts monthly. Anything that fired and was never acted on should be demoted a tier or deleted.
• Escalate on age, not on size. An unacknowledged notify at 48 hours is more concerning than a big number at hour one.
• Keep interrupts rare enough to still work. If the interrupt tier fires more than a few times a quarter, it isn't an interrupt tier any more.

Alerting maturity tracks the rest of your practice closely, and it's usually the dimension organizations overrate about themselves. Six configured alerts and a muted channel is not coverage.

FinOpsAid ships this shape by default: Alerts carries severity-ranked anomalies and budget breaches with the repo, workflow, or team attached to each one, and every notification states whether the figure behind it is billed or modeled — plus the sync it came from, so nobody has to guess how current it is.

Frequently asked questions

Why does everyone ignore our cost alerts?

Because every alert is a withdrawal from a shared account of attention, and most setups overdraw it. When a channel produces alerts that require no action, people mute the channel — and then miss the one that mattered. Fewer, better-placed alerts beat comprehensive coverage every time.

What deserves to interrupt someone about cost?

Very little. Reserve interruption for a hard limit being crossed on a figure that is billed, not modeled, where a named person can act immediately. Everything else belongs in a daily or weekly digest, which is where most cost signals genuinely belong.

Should I alert on modeled cost figures?

Never page anyone on a modeled number. An estimate carries error bars, and waking someone for a figure that might be wrong is how an alerting system loses its credibility permanently. Modeled figures are fine in a digest where they can be read alongside their basis.

Want cost alerts that land on the team that can fix them? Connect your GitHub org — read-only and free during beta — or explore the demo dashboard first.