Each service in our stack had accumulated its own ad hoc approach to logging and metrics over the years, which made cross-service debugging genuinely painful. One service logged structured JSON, another wrote plain text lines with a format nobody could quite remember the conventions for, and a third emitted metrics to a system two services no longer used. Standardizing on OpenTelemetry this year finally gave us one consistent instrumentation layer across all of it, and the project ended up touching nearly every service we run in some form.
The problem with a patchwork of tools
Distributed tracing across service boundaries, previously nearly impossible with our patchwork of logging approaches, became straightforward once every service emitted spans in a common format. Before OpenTelemetry, tracking a single user request across four or five services meant manually correlating timestamps across differently formatted log files, a process that was slow even when it worked and simply impossible when clocks between services had drifted by more than a few hundred milliseconds. We'd estimate the average cross-service incident took two to three times longer to diagnose under the old approach than it does now, purely due to how much manual correlation work was involved.
The inconsistency wasn't anyone's fault exactly; each service had adopted whatever logging approach made sense at the time it was built, often years apart, by different engineers with different backgrounds and no shared standard to follow. That's a completely normal way for a stack to end up fragmented, and it's also exactly the kind of problem that only gets more expensive to fix the longer it's left alone.
Rolling out the migration
The migration was gradual by necessity; instrumenting a dozen services properly took the better part of two quarters, done incrementally alongside regular feature work rather than as a dedicated project that would have pulled the team off feature delivery for months. We prioritized the services most frequently involved in cross-service incidents first, reasoning correctly that those would deliver the clearest value soonest and give us the strongest internal case for continuing the rollout to lower-priority services.
- Distributed tracing across service boundaries, previously nearly impossible with our patchwork of logging approaches, became straightforward once every service emitted spans in a common format.
- Instrumenting a dozen services properly took the better part of two quarters, done incrementally alongside regular feature work rather than as a dedicated blocking project.
- Auto-instrumentation libraries covered the majority of common cases, HTTP calls, database queries, but every service still needed some hand-written spans around business-specific logic to get genuinely useful traces.
Auto-instrumentation libraries covered the majority of common cases, HTTP calls, database queries, message queue operations, out of the box with minimal code changes, which made the first pass of each service's migration faster than we'd budgeted for. But every service still needed some hand-written spans around business-specific logic to get genuinely useful traces; a trace that shows an HTTP call and a database query but nothing about which business rule triggered them is only marginally more useful than no trace at all. Writing those custom spans thoughtfully, rather than instrumenting everything indiscriminately and drowning the useful signal in noise, took real judgment and a few rounds of trial and error to get the granularity right.
Cost and storage considerations
Trace data volume grew faster than we initially estimated, and we had to introduce sampling, capturing a smaller percentage of traces for high-volume, low-value endpoints like health checks while keeping full capture on anything customer-facing, to keep storage and query costs manageable. Getting the sampling configuration right took a couple of iterations; our first pass sampled too aggressively and we lost visibility into some of the intermittent, hard-to-reproduce issues that distributed tracing exists to help catch in the first place.
What we'd do differently
The upfront instrumentation cost was real, but the first production incident debugged with full distributed tracing, instead of grepping through a dozen inconsistent log formats, justified the investment on its own. That incident, a slow database query in one service silently degrading response times for three others downstream, took about twenty minutes to diagnose with tracing in place, compared to what we estimate would have been several hours under the old approach based on similar incidents from before the migration. If we were starting this again, we'd introduce sampling policy earlier rather than reactively after a cost spike, and we'd write a short internal guide on span granularity before the second team started their migration, since a fair amount of the inconsistency we're still cleaning up between services traces back to each team independently deciding what "a useful span" looks like.
Choosing a backend for the collected data
OpenTelemetry itself is vendor-neutral by design, which was a deliberate factor in choosing it over a vendor-specific instrumentation SDK; we wanted the freedom to change where the data actually lands without re-instrumenting every service if we ever needed to switch backends. We settled on routing traces and metrics through an OpenTelemetry collector to a managed observability backend rather than self-hosting the storage and query layer ourselves, judging that operating a high-volume tracing backend reliably was a distraction from the actual goal of better cross-service visibility. That decision has already paid off once, when we needed to add a second backend temporarily during a vendor evaluation; because every service was already emitting standard OpenTelemetry data, pointing a second collector pipeline at the new backend required no changes to any service's code at all.
Training the team on the new tooling
Instrumentation alone doesn't help if engineers don't know how to use the resulting traces during an actual incident, so we ran a couple of internal workshops walking through real traces from past incidents, reconstructed after the fact, to build intuition for reading a distributed trace under pressure. That turned out to matter more than we expected; the first few engineers to use the new tracing during a live incident were noticeably slower to extract value from it than the same engineers were a month later, once they'd built some familiarity with how to navigate a trace waterfall and spot the span that's actually responsible for a slowdown rather than one that's just downstream of it.
Standardizing dashboards across services
Once every service emitted data in a common format, we built a small set of standard dashboards, error rate, latency percentiles, and a service dependency map, that apply uniformly across every instrumented service rather than each team building its own bespoke dashboard from scratch. That consistency has made on-call handoffs considerably smoother, since an engineer covering an unfamiliar service during an on-call rotation now finds the same dashboard layout they're used to from their own team's services, rather than needing to learn a new dashboard's conventions in the middle of responding to an active incident.