Training grants flatten, inference bills climb, and the people who get paged at 3 a.m. when a model starts emitting garbage are rarely in the room where the budget is set. This piece looks at what LLM maintenance work costs the people doing it, why the bus factor on critical paths sits so low, and which practical moves actually help before the next outage.

The Quiet Budget Squeeze

Money for a new training run is a line item someone can point at. A grant proposal names the cluster hours, the dataset, the expected benchmark movement. Inference spend is harder to defend because it scales with users, and usage growth reads as success until the invoice arrives. When the two compete for the same envelope, training usually wins the argument and inference usually wins the bill.

Maintainers absorb the gap. Headcount freezes first. The on-call rotation that had four people has three, then two and a half, because one engineer is splitting time with a research team. Nobody announces this as a decision. It shows up as a calendar that gets harder to fill and a pager that gets handed to whoever did not object loudly enough.

Budget reviews tend to skip the people who page. A review deck shows cost per token, latency percentiles, and GPU utilisation. It rarely shows hours spent on call, or the number of rollbacks, or how long a single maintainer has been the only person who knows how the quantisation pipeline is wired. That absence is not malice. It is a measurement gap, and it has the same effect as a cut.

What On-Call Actually Means

The alert that wakes an LLM maintainer is rarely a clean crash. It is drift. A prompt template changed upstream, a tokenizer version shifted, and now a summarisation endpoint returns output that is subtly wrong in a way no health check catches. The service is up. The answers are worse. Deciding whether that is a page or a morning task is itself a judgement call made at an hour when judgement is poor.

Rollback decisions get made without full context. The person holding the pager may not know which eval run last touched the retrieval index, or whether the regression predates the deploy. They reconstruct the timeline from commit messages and a shared channel, then pick the least bad option. This is normal operational work in any system. It is sharper here because the failure mode is statistical and the tooling for it is younger.

One maintainer usually holds the runbook. Not a document, necessarily, but the working knowledge of which knob to turn. That person becomes a single point of failure on the critical path, and everyone quietly knows it. The bus factor concept, the risk that capabilities live in one head, applies to ML deployments as sharply as to any codebase.

The Bus Factor Problem

Ask a small ML team who understands the quantisation pipeline and the honest answer is often one name. Two people might understand the eval harness well enough to change it safely. The rest of the team can run things, not repair them. This is a familiar shape in infrastructure work, and this site has covered similar dependencies in build minutes inside CI contracts, where a single billing detail quietly governs the whole pipeline.

Handoffs get documented in scattered threads. A decision about temperature settings lives in a chat message from eight months ago. The reason a particular checkpoint was pinned is in a code comment that references a ticket nobody can open. None of this matters until someone leaves for two weeks, and then it matters a great deal.

Vacations become risk events. A maintainer booking ten days off triggers a scramble to find cover, and the cover is usually someone who can restart the service but not diagnose it. The team learns to schedule around absences rather than build capacity to survive them. That is a symptom worth naming in a planning meeting, because it is cheaper to fix than the outage it predicts.

Funding Models That Miss Maintenance

Grants fund new training runs. They fund the exciting part, the benchmark result, the demo. Nobody writes a grant for the person answering pages at 3 a.m., because that work does not produce a figure anyone can cite. The funding follows novelty, and maintenance is by definition the opposite of novelty.

Sponsorship tiers reward features. A backer wants their name on a release note, not on an on-call rota. Open-source business models often assume paid support exists as a product, but that assumption holds for projects large enough to sell contracts. A small LLM deployment maintained by two engineers has no support tier to sell and no customers who would buy one.

There is a real objection here, and it is fair: sponsors cannot easily audit maintenance spend, so funding it feels like writing an open cheque. That is true. The alternative, though, is that maintenance gets funded by borrowing against the maintainers' goodwill until it runs out, and the bill arrives as an outage rather than an invoice. The trade-off belongs in the open, not buried in a calendar.

Wear Inside the Maintainer Role

The role swings between research and operations, often within an hour. A maintainer reads a paper in the morning and debugs a memory leak in the afternoon. Context switching at that granularity is expensive, and it is rarely counted as work because neither half looks like the other. The research half is visible. The operational half is the part that keeps the service alive.

Alert fatigue dulls the real signal. When a channel produces forty notifications a day and three of them matter, people stop reading carefully. The dangerous alert gets the same glance as the noisy one. This is a well-worn operational problem, and the log pipeline piece on incident response costs makes a related point about where the real expense hides.

Career progression on a maintenance track is unclear. Promotions reward launches. A maintainer who kept a service stable for a year has a thin portfolio by that measure, even though the stability was the point. Quiet attrition precedes visible outages, and by the time the departure is on a calendar the knowledge has already started to leave.

What Maintainers Can Do Now

Write the runbook before the next outage, not after. A single page listing the pinned checkpoint, the tokenizer version, the rollback command, and the person to call is worth more than a wiki full of stale links. Keep it where the pager holder will actually look.

Pair a second person on the eval harness this quarter. Sit them down with the maintainer who owns it and have them change one threshold together. The goal is that two names, not one, can answer a question about it under pressure.

Track on-call hours in the budget request. Log the pages, the after-hours work, and the rollbacks, then put the total next to the training spend. A number in a deck gets argued with. An absence gets ignored.

Negotiate maintenance as a named line item, even a small one. It changes the conversation from whether the work happens to who pays for it, and it gives the next budget review something concrete to defend.

Rotate pager duty across at least three people, and accept that the first rotation will be slower than the expert. The slowdown is the cost of not having a bus factor of one. Pay it deliberately, on a quiet week, rather than discovering it during an incident.

What a Realistic Maintenance Line Item Looks Like

A maintenance line item does not need to be large to change behaviour. Start with the hours. If the rotation covers nights and weekends with three people, that is a quantifiable slice of each person's week. Multiply by a loaded hourly rate and you have a number that can sit beside cluster hours without looking out of place. The exact rate will vary by organisation, but the structure of the argument does not.

Then add the tooling that keeps the bus factor above one. A staging environment that mirrors production tokenizer versions, a small regression suite that runs on every prompt template change, and a shared dashboard that shows drift before users report it. None of these are expensive relative to a training run, and each one shortens the path from alert to diagnosis. The trade-off is that they consume engineering time that could go to the next experiment. That is the trade-off worth naming in the review, because the alternative is paying for it during an outage when the options are worse.

Finally, write down what the maintenance buys. Uptime is abstract until it is gone. A short note that lists the incidents avoided, the rollbacks that took minutes instead of hours, and the questions answered without escalation gives the next budget conversation a concrete record. Sponsors and managers argue with abstractions. They rarely argue with a log.