Splitting a system into separate services is the easy part. The harder, less discussed problem is designing the APIs between those services so you have not simply rebuilt the monolith's tight coupling with extra network hops in between — a failure mode sometimes called a distributed monolith, and one we came close to on a recent project before catching it.
How we ended up close to a distributed monolith
The project in question started with good intentions: a handful of services, each with a clear-sounding name and responsibility. What actually happened over the following months was less clean. One service calling into three others synchronously to answer a single request, meaning any one of the four being slow or down took the whole chain down with it, and no one had designed that dependency chain deliberately — it accumulated one reasonable-seeming addition at a time, the same way the original monolith this project was meant to improve on had accumulated its own tangled dependencies.
The database-sharing problem
Several services shared a database schema "temporarily," which of course became permanent and meant a schema change in one service could silently break another. This is, in our experience, the single most common way a microservices project quietly turns back into a monolith without anyone deciding that on purpose — sharing a database feels like a small, pragmatic shortcut in the moment, and by the time anyone notices how load-bearing that shortcut has become, untangling it is a much bigger project than avoiding it would have been.
What we changed course on
- We had one service calling into three others synchronously to answer a single request, meaning any one of the four being slow or down took the whole chain down with it
- Several services shared a database schema "temporarily," which of course became permanent and meant a schema change in one service could silently break another
- Nobody owned the overall shape of the service graph, meaning any individual engineer's locally reasonable decision could accumulate into a globally tangled system with no one positioned to notice until it was already a problem
What we do now
- Each service owns its own data, full stop — no service reads another's database directly, and any data another service needs is either requested through that owning service's own API or replicated deliberately via an explicit, documented event
- Anything that does not need an immediate answer goes through a message queue instead of a synchronous call, following the same pattern we had already adopted for order processing on a different client project
- We version internal APIs the same way we would a public one, since "internal" does not mean the consumers are any less real, and a breaking change to an "internal-only" API has caused just as much production pain for us historically as a breaking change to something public-facing
Making ownership explicit
Beyond the technical patterns, we started assigning explicit ownership to each service, with one team or individual responsible for reviewing any change to that service's public contract, even changes proposed by another team. This sounds like pure process overhead, and it does add a small amount of friction to changes that cross a service boundary, but that friction is exactly the point — the previous, friction-free state is what let three services quietly grow to depend on internal database details of a fourth without anyone deciding that dependency was acceptable.
A concrete example of the difference this made
A recent feature request needed a change to how one service represented a customer's loyalty tier. Under the old, looser conventions, this would likely have meant directly altering a shared database column that at least two other services also read from, discovered only when something downstream broke. Under the new conventions, the change went through the owning service's API, with a new field added alongside the old one and a migration period during which both were populated, giving every consuming service time to move over deliberately rather than breaking without warning. It took longer than the shortcut would have, and nothing broke.
The actual lesson
The boundary between services matters more than the services themselves. Get that wrong and microservices give you all the operational complexity of a distributed system with none of the actual independence you were hoping for. We now treat "does this change respect service boundaries" as a real code review question on every project with more than a couple of services, not an afterthought raised only once something has already gone wrong.
How we introduce this thinking to a new team
Explaining "don't let services share a database" in the abstract tends to land as a reasonable-sounding rule that is easy to nod along to and then quietly violate under deadline pressure, the same way the original problem on this project actually happened. What has worked better is walking new team members through the specific incident described above — the loyalty tier change that would have broken two other services under the old approach — as a concrete story rather than an abstract principle, since a specific example of what actually goes wrong sticks with people in a way a general rule rarely does on its own.
Contract testing between services
Beyond code review and API versioning, we added contract tests between services that have real interdependencies, where each consuming service defines its expectations of the API it depends on, and those expectations get checked automatically whenever the providing service changes. This catches a category of problem that manual code review alone tends to miss — a change that looks locally reasonable to the team making it, but silently violates an assumption a different team's service was relying on, discovered by an automated test rather than a production incident days or weeks later.
Where we still allow exceptions to the rules
None of these rules are absolute, and we have made deliberate, documented exceptions where a synchronous call genuinely is the right choice — a payment authorization step, for instance, where the caller needs an immediate answer before proceeding and asynchronous processing would meaningfully change the user experience for the worse. The distinction we now apply is whether the caller can proceed usefully without an immediate answer; if yes, it goes through a queue, and if genuinely not, a synchronous call is fine, as long as the service being called has its own health and timeout handling in place so a slow dependency degrades gracefully rather than cascading into a full outage.
Revisiting this a year later
Looking back on this project roughly a year after the initial redesign, the service boundaries we drew have held up better than we expected, with only one genuine boundary dispute since — a reporting service that needed data spanning two other services' domains, resolved by having it consume both services' public APIs and aggregate the result itself, rather than either service growing to own data that was not really its responsibility. We have since turned this project's lessons into a short internal architecture guide that every new multi-service project reviews at kickoff, treating it as a living document updated whenever a new project surfaces a boundary question the existing guide did not already cover clearly.
How this changed our project kickoff conversations generally
Any new project expected to involve more than a couple of services now gets a dedicated architecture session before any code is written, walking through exactly which service will own which piece of data and how services will communicate, with the resulting diagram treated as a living reference rather than a one-time planning exercise. This single session has caught at least two potential shared-database shortcuts before they were ever built, simply by making someone explicitly draw the boundary and defend it out loud in front of the rest of the team, rather than letting it emerge implicitly through a series of individually reasonable-seeming pull requests. That same architecture session has also become the place where we decide, deliberately and in writing, which of the exceptions described above actually apply to a given project, rather than leaving that judgment call to whichever engineer happens to hit the relevant decision point first, weeks into the build with a deadline already bearing down on them. It is a small procedural addition, but it has paid for itself several times over already.