A client's checkout endpoint had grown to do payment processing, inventory decrement, confirmation email, and a shipping label request to a third-party carrier, all synchronously in one request. Under normal traffic it worked, but any slowness in the carrier's API turned into a slow, sometimes-timing-out checkout for the shopper, and the whole flow would fail as a unit if any one of its four steps had a bad moment.
How the original checkout was built, and why it grew this way
None of this happened by design. The checkout endpoint started small — charge the card, decrement inventory — and each new requirement got bolted on in the most direct way available at the time: a confirmation email here, a shipping label call there, an analytics event a few months later. Every individual addition made sense in isolation. The cumulative effect was a single request handler that could fail for reasons that had nothing to do with whether the shopper's card was actually valid, and a checkout that was only as fast as its slowest dependency on any given day.
The redesign
- Checkout now does only what must happen before responding: charge the card and reserve inventory
- Everything else — confirmation email, shipping label generation, analytics events — gets published as a message to RabbitMQ
- Separate worker processes consume each queue, retrying independently on failure without blocking the others
- A dead-letter queue catches anything that fails repeatedly, so we get alerted instead of silently losing a shipping label request
Why RabbitMQ over the alternatives
We considered a simpler approach using a database-backed job table, which we already use on smaller projects, but this client's volume and the mix of different consumers made a proper message broker worth the added operational piece. RabbitMQ's routing flexibility, particularly topic exchanges, let us fan a single "order placed" event out to several unrelated consumers without the checkout code knowing anything about them, which meant adding a new consumer later required zero changes to the code that publishes the original event. A database-backed queue can be made to do something similar, but it tends to require the publisher to know about every consumer's table, which reintroduces exactly the coupling we were trying to remove.
What the worker processes actually look like
Each consumer is a small, standalone Node process, deliberately kept separate from the main API application rather than run as a background thread inside it, so a crash in the shipping-label worker cannot take down the checkout API and vice versa. Each worker acknowledges a message only after it has fully completed its work, so a worker that crashes mid-task leaves the message unacknowledged and RabbitMQ redelivers it to another worker instance rather than losing it. This does mean every consumer has to be written to tolerate being run more than once on the same message — generating a duplicate shipping label if a worker crashes right after the label prints but before it acknowledges the message, for instance — so we made shipping label generation idempotent by checking for an existing label tied to the order before requesting a new one.
Monitoring the queues, not just the servers
Queue depth turned out to be one of the most useful metrics we added anywhere in this system. A slow but functioning carrier API shows up first as a growing backlog in the shipping-label queue, well before it would show up as an actual customer complaint, which gives the team a genuine early warning rather than finding out about a problem after a shopper does. We built a small dashboard specifically for this, with alerting thresholds tuned per queue, since a growing backlog in the analytics-events queue is a low-urgency annoyance while the same growth in the confirmation-email queue is something we want to know about immediately.
The dead-letter queue in practice
A message that fails processing repeatedly — say, a shipping carrier's API rejecting a malformed address permanently rather than intermittently — gets routed to a dead-letter queue after a configured number of retry attempts, rather than being retried forever or silently dropped. We review the dead-letter queue daily, and it has surfaced a handful of genuine data problems, like a shopper's address containing characters the carrier's API could not handle, that would previously have failed silently somewhere in the old synchronous flow with no clear record of what happened or why.
Results
Checkout response time dropped from a peak of several seconds to consistently under 400ms, and carrier API slowness no longer affects the shopper's experience at all, since a slow shipping label request now happens entirely out of band from the checkout response the shopper is actually waiting on. Adding a new consumer, like a recent loyalty-points worker, needed zero changes to the checkout endpoint, which was the whole point of decoupling publishers from consumers in the first place.
It is more infrastructure to run and monitor than we had before — RabbitMQ itself needs its own uptime monitoring, and we added dashboards specifically tracking queue depth so a struggling consumer shows up as a growing backlog before it becomes an outage. But for a system with several independent downstream effects from one event, it is the right shape, and we would make the same call again on the next project that has this same one-event-many-consumers pattern.
Message schema versioning
Once several independent consumers depend on the shape of a published message, changing that shape becomes a much bigger deal than it would be inside a single monolithic codebase, where a compiler or at least a shared type definition would catch a mismatch immediately. We now version our message schemas explicitly, including a version field in every published message, and we have a rule that a consumer must tolerate receiving an older version gracefully during any rollout window where publishers and consumers are briefly running different versions of the code. This came from a real incident: an early change that added a required field to the "order placed" message broke the shipping-label worker for about twenty minutes during a deploy, because the worker had been updated to expect the new field before the publisher had actually started sending it.
Idempotency lessons learned the hard way
Making the shipping label worker idempotent sounds simple in the abstract — check if a label already exists before requesting a new one — but the actual carrier API we integrate with does not offer a clean way to query "does a label already exist for this order," so we had to build our own tracking table recording every label request and its outcome before we call the carrier at all. Without this, a redelivered message following a worker crash could genuinely double-charge the client's carrier account for a duplicate label on the same order, which is exactly the kind of bug that is invisible until an invoice reconciliation catches it weeks later.
Rolling out without downtime
Deploying a new version of a consumer used to mean briefly stopping message processing for that queue, which was an acceptable tradeoff early on but became noticeably worse as order volume grew and the associated backlog took longer to drain afterward. We now run multiple instances of each consumer behind RabbitMQ's normal competing-consumers pattern, and deploy them one at a time, so at least one instance of every consumer is always processing the queue during a rollout. This was a small operational change but removed what had become a recurring, if minor, source of delayed shipping labels immediately after every deploy.
Extending the pattern to returns processing
A few months after the original rollout, we applied the same publish-and-consume pattern to the client's returns flow, which had grown its own tangle of synchronous steps — refund processing, inventory restock, a customer notification email — mirroring the exact problem the original checkout redesign solved. Having already built the RabbitMQ infrastructure, monitoring, and dead-letter handling for checkout meant the returns migration took a few days rather than the couple of weeks the original project needed, which is the kind of compounding return on infrastructure investment that made the original, more involved project worth the up-front cost.
Documentation as a first-class artifact, not an afterthought
With four independent consumers now reading from the same set of published events, we started keeping a single internal document listing every message type, its current schema version, and every known consumer subscribed to it. It sounds like a small bit of process overhead for what is, technically, "just documentation," but it has already prevented at least one near-miss where a developer considered removing a field from a message without realizing a consumer nobody on the current team had touched in months still depended on it.