DevOps

Speeding up CI in a growing monorepo with GitHub Actions

OPS

As more projects moved into a shared monorepo this year, our GitHub Actions pipeline crept from a few minutes to over fifteen, with most jobs running full test suites regardless of which package actually changed in a given pull request. Nobody had designed the pipeline to behave this way; it was the natural, gradual result of adding each new package's tests to the same workflow file without ever revisiting whether the workflow's structure still matched the size of the repo it was running against.

How the slowdown actually accumulated

The monorepo started with two packages sharing a single CI workflow, which was reasonable when the whole thing built and tested in under three minutes. Every new package added since then got its own test job appended to the same workflow, running on every single pull request regardless of whether that package's files had actually changed. By the time we audited it, a one-line documentation fix in a package with no dependents was still triggering full test runs across a dozen unrelated packages, each spinning up its own dependency install and test run in sequence, which is where the bulk of the fifteen minutes was actually going.

Path-based filtering as the first fix

Path-based filtering so jobs only run for packages actually affected by a given change cut our average pipeline time by more than half almost immediately. GitHub Actions supports path filters at the workflow level, but we needed something more precise than top-level path matching, since a shared internal package being changed should trigger tests for every package that depends on it, not just the package that changed directly. We built a small script that reads the monorepo's dependency graph and, given a set of changed files, outputs the full list of packages that need testing, including anything downstream of a shared package, which then drives a matrix of jobs generated dynamically for that specific pull request rather than a fixed, always-run job list.

  • Path-based filtering so jobs only run for packages actually affected by a given change cut our average pipeline time by more than half.
  • Caching node_modules and build artifacts between runs, rather than reinstalling from scratch every time, accounted for most of the remaining improvement.

Caching accounted for most of the rest

Caching node_modules and build artifacts between runs, rather than reinstalling from scratch every time, accounted for most of the remaining improvement. GitHub Actions' cache action, keyed off a hash of the lockfile, meant a pull request that didn't touch dependencies could skip the install step entirely and restore a cached node_modules directory in a few seconds instead of running a fresh install that, across a monorepo this size, was taking well over a minute on its own. We applied the same approach to build output for packages that compile TypeScript or bundle assets, caching build artifacts keyed off a hash of the relevant source files, so an unaffected package's build step became a cache restore rather than a full rebuild.

Smaller wins that added up

A few additional changes contributed smaller but real gains: splitting one long-running integration test suite that had been blocking the whole pipeline into a separate job that runs in parallel with everything else rather than sequentially at the end, and moving linting to run against only changed files rather than the entire repository on every single run, since a full-repo lint pass was taking almost as long as the actual test suite for no benefit on a pull request that touched three files. We also set a hard timeout on every job, since we discovered more than one instance of a hung job silently consuming its full default timeout before failing, wasting minutes that nobody had noticed because the eventual failure looked the same either way.

What this bought us day to day

Building the dependency graph script safely

Getting the affected-packages script right took a couple of false starts. Our first version relied purely on each package's declared dependencies in its own package.json, which missed a category of implicit coupling where one package imported directly from another's source files through a relative path rather than through the declared package boundary, a pattern left over from before the monorepo had consistent internal package boundaries enforced. We had to extend the script to also parse actual import statements as a fallback signal, not just declared dependencies, before it reliably caught every case that should have triggered a downstream test run, and we added a small test suite of its own specifically to guard against the script silently under-triggering tests again in the future as new packages and import patterns get added.

Rollout and the trust-building period

We rolled the new dynamic matrix out gradually, running it in parallel alongside the old always-run-everything workflow for about two weeks and comparing which packages each approach selected for a sample of real pull requests, before fully cutting over and deleting the old workflow. That overlap period mattered for building trust across the team as much as for catching bugs; a CI change that silently skips tests it shouldn't have is a much scarier failure mode than a slow pipeline, and having two weeks of side-by-side evidence that the new approach wasn't quietly missing anything made the team comfortable relying on it going forward rather than remaining suspicious of a faster pipeline that might be cutting corners.

Handling flaky tests that the faster pipeline made more visible

An unexpected side effect of running fewer, more targeted test jobs was that flaky tests, ones that failed intermittently for reasons unrelated to the actual change being tested, became more disruptive rather than less, since a pull request now typically triggers a much smaller number of jobs and a single flaky failure among three jobs is a much larger fraction of that pull request's CI signal than the same flaky failure was as one job among thirty. We had to invest in identifying and either fixing or quarantining our worst offenders, a small number of integration tests with timing-dependent assertions, since the faster, leaner pipeline had less redundancy to absorb an occasional bad result without someone needing to manually investigate.

Ongoing maintenance the new setup requires

The dynamic matrix approach isn't maintenance-free the way a simpler, always-run-everything workflow was, even with its slowness. Anytime a new package is added to the monorepo, someone needs to confirm it's properly represented in the dependency graph the affected-packages script builds from, and we added a lightweight CI check of its own that fails if a new package directory appears without a corresponding entry in the graph configuration, specifically so this doesn't become a manual step someone has to remember indefinitely as the monorepo keeps growing.

Documenting the setup so it survives team turnover

A dynamic, dependency-graph-driven CI setup is meaningfully more clever than the always-run-everything workflow it replaced, which cuts both ways: it's faster, but it's also less immediately legible to a new engineer who's never seen anything but a simple, static workflow file before. We wrote an internal doc walking through exactly how the affected-packages script determines what to run, with a couple of worked examples, specifically so the setup doesn't become something only its original author fully understands, which is a trap clever infrastructure can fall into quietly if documentation isn't treated as part of the deliverable rather than an afterthought.

We also revisited how failures get reported to the team once the pipeline structure changed. With the old setup, a failure notification named a single monolithic workflow run, leaving whoever picked it up to dig through logs to find which package actually broke. The new per-package job structure means a failure notification names the specific package and test file involved directly, which has shaved real time off the average time-to-first-response on a broken build, simply because the person picking it up no longer has to do that initial triage step by hand.

Neither of the two main changes required exotic tooling, just paying attention to what CI was actually doing on every single run rather than assuming the existing setup was still appropriate for the repo's current size. Fifteen minutes down to under five has made the whole team noticeably more willing to push small, frequent commits instead of batching changes together to avoid triggering a slow pipeline repeatedly, which was an unplanned but genuinely valuable side effect: faster CI didn't just save time waiting, it changed how people worked, resulting in smaller pull requests that are easier to review and revert individually rather than large batched changes that took a slow pipeline's cost into account when deciding how to size a change.

← Back to the journal

Have a project in mind?
Let’s talk.

Tell us where you are and where you want to go. We'll map the fastest route between the two.

Currently accepting new clients