DevOps

A Kubernetes cost audit that cut a client's cluster bill by a third

OPS

Resource requests in a Kubernetes deployment tend to get set once, conservatively, by whoever first deployed the service, and never revisited unless something breaks. A client's cluster had grown that way over roughly two years, service by service, and actual usage had long since diverged from what was originally requested, mostly in the direction of over-provisioning, since the safe instinct when unsure is always to ask for more than you think you need.

Right-sizing CPU and memory requests based on real historical usage data, rather than initial guesses, let us shrink the node pool meaningfully and cut the client's monthly cluster bill by close to a third with no observable performance impact. We pulled several weeks of actual resource utilization metrics per pod rather than relying on point-in-time snapshots, since a single measurement can easily land during an unusually quiet or unusually busy period and produce a misleading right-sizing recommendation either direction.

A handful of services turned out to be significantly under-provisioned rather than over, which was the more urgent finding even though it wasn't the headline cost story; a couple of pods were hitting their memory limits and being throttled or restarted often enough to be a quiet reliability problem nobody had connected to resource limits specifically, since the symptom looked like intermittent application slowness rather than an infrastructure ceiling.

We also found real savings in cleaning up requests that had simply never been updated after a service's workload characteristics changed, a batch processing job that used to run large overnight loads but had since been redesigned to process smaller, more frequent batches, still requesting resources sized for the old workload pattern it no longer ran.

The process we'd recommend to any team facing a similarly aged cluster: pull real historical usage data before touching any request values, right-size in both directions rather than assuming the fix is always "reduce," and treat this as a recurring quarterly exercise rather than a one-time cleanup, since the same conservative-guess drift that created the problem the first time will happen again as new services get deployed by developers erring on the side of caution, exactly as their predecessors did.

Rolling out the changes safely mattered as much as identifying them. We staged the request adjustments service by service rather than applying the full set of changes cluster-wide at once, watching for pod evictions, throttling events, or latency regressions after each change before moving to the next service, since a right-sizing recommendation based on historical data can still be wrong if a service's usage pattern is about to change for reasons the historical window didn't capture, a planned feature launch, for instance, that would meaningfully increase load right after the audit concluded.

We also set up ongoing alerting on the gap between requested and actual resource usage per service, rather than relying on the next scheduled audit to catch drift, so a service trending meaningfully away from its right-sized request gets flagged automatically well before the next quarterly review. That alerting has already caught one service ramping up usage faster than expected, letting the team adjust its requests proactively instead of waiting for a performance complaint to surface the problem the hard way.

← Back to the journal

Have a project in mind?
Let’s talk.

Tell us where you are and where you want to go. We'll map the fastest route between the two.

Currently accepting new clients