BYTEHERDER"When did you last upgrade this cluster?"
The answer came back as a date fourteen months earlier, and then a pause, and then: "we keep meaning to."
Four people in the room, all of them competent, all of them slightly embarrassed, none of them able to say what would actually break. That last part is the whole story.
How a cluster gets here
Let me back up, because nobody decides to fall two minor versions behind.
The cluster was built properly. Declarative, version-pinned, delivered through Flux, with a staging cluster built from the same manifests. Whoever set it up knew what they were doing.
Then the person who set it up left. Then a release went badly on a different system and everyone got cautious for a quarter. Then the deprecation notes for the next version mentioned an API the team used and nobody had time to work out how much of it they used. Then it was four versions later and the gap was big enough to be frightening on its own.
Every one of those steps is reasonable. The result is not.
The thing that kept it stuck was not technical. It was that "what happens if we upgrade" had never been answered, so everyone in the room was holding a private worst case, and the private worst cases did not have to agree with each other or with reality. Risk that nobody has measured gets treated as infinite, and you cannot put infinite risk in a maintenance window.
Measuring it instead of estimating it
So we stopped talking about it and spent a day finding out. Three things, in this order.
What are we actually using that goes away? Not what the release notes deprecate. What this cluster calls.
# removed and deprecated APIs still present in live objects and in the manifests
kubectl-pluto detect-all-in-cluster -o wide
kubectl-pluto detect-files -d ./clusters/prod
Across nine namespaces this came back with four hits. Two were in a Helm chart we had not updated in a year, and upgrading the chart resolved both. One was a PodDisruptionBudget on an old API version, a one-line change. One was in a vendor CRD, which is the only genuinely awkward category, and it turned out the vendor had shipped a compatible version five months earlier.
Four findings. Half a day of work. This was the item everybody had been most afraid of.
Does the control plane come back? Build the cluster again from the same manifests, on throwaway infrastructure, at the target version.
That is the whole test, and it is only meaningful if the manifests really are the source of truth. This is where a lot of teams discover that they are not, and that discovery is worth the day on its own. Ours were, mostly. Two secrets had been created by hand in 2024 and existed nowhere in git. We found them because the rebuilt cluster came up with two pods in CreateContainerConfigError, which is a much nicer way to find out than during a real upgrade.
Does the data survive? Restore last night's backup into the rebuilt cluster and see if the application starts.
Not "does the backup job report success". Restore it. The backup job had been reporting success for fourteen months. The restore failed on the first attempt, because the backup captured the PVCs but not the CSI volume snapshot class they referenced, so nothing could bind. That is a fifteen-minute fix once you know, and it had been sitting there silently the entire time the cluster was too risky to upgrade.
The upgrade
Forty minutes, on a Tuesday afternoon, not a weekend.
One node group at a time, kubectl drain with a real PodDisruptionBudget doing its job, watch the workload, move on. No incident. Nothing in the error budget. The most eventful moment was a monitoring alert firing because a node went NotReady during its own planned drain, which is a tuning problem for another day.
Forty minutes, against fourteen months of not doing it.
What that day was actually worth
The upgrade was the least valuable thing that happened.
The valuable outputs were a restore that works, two secrets that now exist in git, and a written procedure with the four commands in it. The next upgrade is now a scheduled task rather than a standing conversation, and it will be, because the thing that made it frightening was never the upgrade.
There is a version of this post that ends with "test your backups", and that is true and you have read it before. The more useful framing is this one: an unmeasured risk does not sit at its true value in people's heads. It sits at the worst thing anyone in the room can imagine, and it keeps climbing, because a version gap that grows makes the imagined worst case grow with it.
Measuring costs a day. It almost always comes back smaller than the room expected. And on the occasions it comes back larger, you have found out on a Tuesday with a throwaway cluster instead of at two in the morning with the real one.
Ask when the cluster was last upgraded. If the answer has a pause in it, the problem is not the version.