With more of our client work split into independent services, deploying and scaling each one by hand on individual EC2 instances started to feel unsustainable, and Kubernetes was the obvious next step to evaluate. We ran a client's full backend on a managed Kubernetes cluster for the first time this year, and the honest answer to whether it was worth it turned out to depend heavily on which client we were asking about.
The client and workload that made the case
The client we chose for this first real deployment had a dozen or so independently deployable services, genuine elasticity in traffic tied to daily and seasonal patterns, and an existing team already comfortable with Docker from our earlier work moving them onto containerized local development. That combination mattered — Kubernetes' value shows up most clearly when you already have several independent services and traffic that actually benefits from elastic scaling, and this client had both.
What it solved
- Rolling deploys with automatic rollback if health checks fail, replacing what used to be a manual, nerve-wracking process of watching a deploy and hoping nothing broke
- Horizontal scaling of individual services based on load, without touching the others, which previously meant manually provisioning new EC2 instances ahead of an expected traffic spike and hoping the estimate was right
- A consistent way to configure and deploy every service regardless of which language it is written in, since Kubernetes does not care whether a service is Go, Node, or PHP as long as it speaks the same deployment contract
What it cost us
- A real learning curve for the team — YAML manifests, ingress controllers, and cluster networking are not trivial to reason about, and we underestimated how long it would take engineers comfortable with Docker Compose to become comfortable with the equivalent Kubernetes concepts
- More moving pieces to monitor than a handful of EC2 instances behind a load balancer, since a cluster introduces its own layer of things that can fail independently of the applications running on it
- A managed cluster's baseline cost is not nothing, even before any workloads are running on it, which matters for smaller clients in a way it does not for larger ones
The specific mistakes we made along the way
Our first attempt at resource requests and limits for each service's pods was based on rough guesses rather than actual measured usage, and it showed — a couple of services were under-provisioned and got evicted under load during the first real traffic spike after launch, while others were over-provisioned and wasted cluster capacity that could have gone to something else. We now run every service under realistic load in a staging cluster first and set resource requests based on actual observed usage with a reasonable safety margin, rather than guessing and correcting after a production incident, which is a more disciplined process than we originally budgeted time for but has paid for itself since.
Where we drew the line for smaller clients
For this particular client, with a dozen or so independent services and real elasticity in traffic, the tradeoff was worth it. We would not recommend it yet for a client with two or three services and steady, predictable load — the operational overhead outweighs the benefit at that scale, and a couple of well-managed EC2 instances behind a load balancer will serve them better. We have started using a rough internal rule of thumb in scoping conversations: fewer than about five independently deployable services with mostly steady traffic, stick with plain instances; more than that, or genuinely spiky traffic, Kubernetes starts earning its complexity.
Staffing implications we did not fully anticipate
Running Kubernetes well is not a one-time setup cost — it is an ongoing operational commitment that needs at least one person on the team who actually understands cluster internals well enough to debug a genuinely confusing failure, not just someone who can write a deployment manifest by copying an existing one. We ended up sending two engineers through a proper Kubernetes course rather than relying entirely on learning-by-doing, which was a real investment of time but one we think was necessary given how much of this client's infrastructure now depends on the cluster running correctly.
What is next
We expect to bring Kubernetes to one or two more clients next year, chosen specifically for having the same shape of problem — several independent services and genuinely variable traffic — rather than adopting it as a default recommendation across every client regardless of fit. The lesson from this first deployment was less about the technology itself and more about matching the tool's actual strengths to a client's actual situation, rather than reaching for the technology everyone is currently excited about.
Namespace and access control decisions
One area we spent more time on than expected was structuring the cluster's namespaces and access controls before the first real workload went live. Rather than a single flat namespace for everything, we split staging and production into separate namespaces from day one, with role-based access control limiting which team members could deploy directly to production versus staging. This felt like unnecessary ceremony during setup, when the cluster had exactly one client's workloads on it, but it meant the second client we eventually migrated onto shared cluster infrastructure could be onboarded without reworking any of the access model, since the pattern was already established rather than needing to be retrofitted under time pressure.
Cost monitoring specifically for Kubernetes
A managed cluster's billing model — a mix of node costs, load balancer costs, and a per-cluster management fee — is genuinely harder to reason about at a glance than a handful of individually priced EC2 instances, and we had at least one month where actual spend meaningfully exceeded what we had quoted the client, traced back to a couple of services with resource requests set generously "to be safe" that were reserving far more capacity than they ever actually used. We now review resource utilization against requested resources monthly for any Kubernetes-hosted client, specifically to catch this kind of quiet cost creep before it becomes a pattern rather than a one-time surprise.
Disaster recovery testing
We also made a point of actually testing what happens if the cluster itself has a bad day, rather than assuming Kubernetes' self-healing reputation meant we did not need to plan for it. A deliberate exercise simulating a node failure showed pods rescheduling onto healthy nodes largely as expected, but also surfaced a gap in how one service handled its own in-memory cache being lost on a forced restart, which had never been tested under the old EC2-based setup because a single long-running instance rarely restarted unexpectedly in the first place. That kind of gap is exactly the sort of thing that only reveals itself once you deliberately go looking for it, rather than waiting for a real incident to find it for you.
Documentation as part of the rollout, not an afterthought
Given how much of this project was genuinely new territory for the team, we kept a running internal document of every non-obvious decision and its reasoning as we made it, rather than relying on the eventual cluster configuration to speak for itself. This turned out to matter more than expected a few months in, when a question came up about why a particular service's resource limits were set the way they were, and the answer — a specific incident during an early load test — was still findable in that document rather than lost to memory or requiring someone to reverse-engineer the reasoning from the YAML alone. We also learned to budget real calendar time for the inevitable cluster upgrade cadence itself, since a managed Kubernetes offering's underlying version does not stay still, and skipping a minor version upgrade for too long turns a routine maintenance task into a much riskier multi-version jump later, something we now schedule deliberately rather than deferring indefinitely.
What we tell a client asking "should we do this too"
When a prospective client asks whether they should be running Kubernetes because a competitor is, we walk them through the same checklist we used internally rather than giving a straight yes or no. How many independently deployable services do they realistically have today, and how many do they expect in the next year. How variable is their actual traffic, measured rather than assumed. Does anyone on their team, or ours, have the bandwidth to own cluster operations as an ongoing responsibility rather than a one-time setup task. Most conversations that start with genuine curiosity about Kubernetes end with a client choosing to wait, which we consider a good outcome rather than a missed sale, since recommending infrastructure a client cannot yet operationally support well is not a service to them.