We had a scare, a plugin update corrupted several tables on a client's site and the host's own backups were a week stale. It got fixed, restored from an older backup plus a manual reconstruction of what changed in between from the client's own memory of recent edits, but it shouldn't have been that close, and the reconstruction step in particular was the kind of stressful, error-prone work that shouldn't be part of anyone's recovery plan.
What actually happened
The plugin in question ran a database migration on update that, due to a bug in that specific version, partially completed before failing, leaving several tables in an inconsistent state, some updated to the new schema, others not, with a handful of rows caught in between and effectively corrupted. It wasn't malicious, it wasn't even particularly unusual as plugin bugs go, but the practical effect for the client was the same as if it had been: broken pages, missing content, and a site that needed real recovery work before it could go back to normal.
Our first instinct was to check the host's backup system, which we'd always assumed was handling this for us. It turned out their backup snapshot was a week old, apparently their retention and frequency were both looser than we'd assumed, something we'd never actually verified because we'd never needed to use it before.
What we built
Every project now gets a simple cron job that runs mysqldump nightly, compresses the output, and keeps a rolling seven days of backups on the server plus a copy pushed to a separate storage account, off the same host entirely, so a problem with the server itself, not just the database, doesn't take out the backups along with it.
- A nightly cron entry running mysqldump with credentials read from a file outside the web root, not hardcoded in the script itself.
- gzip compression on the output, since an uncompressed SQL dump of even a modest site adds up in storage over a rolling week of retention.
- A rotation step that deletes anything older than seven days, so storage doesn't grow unbounded.
- A separate, off-host copy pushed after each successful dump, using a simple file transfer to a different storage account, so a host-level failure doesn't also wipe out the backup.
It's not sophisticated, but it means the worst case is losing a day of content instead of a week, and it means we're not depending on a host's own backup policy that we've now learned not to take for granted without checking it directly.
Why we didn't trust the host's own backups going forward
We did ask the host directly what their actual backup frequency and retention policy is, and got a vague answer that amounted to "we do back things up, generally." That's not something we're comfortable building a client recovery plan around, not because we think the host is being dishonest, but because "generally" isn't a number we can plan a recovery time objective against. Owning the backup process ourselves means we know exactly what our worst-case data loss window is, one day, rather than trusting an opaque third-party process we have no visibility into.
Testing that the backups actually work
A backup you've never restored from is a backup you don't actually know works. We picked one project and did a full test restore onto a separate, disposable server, walking through the entire process end to end rather than just confirming a dump file exists and looks roughly the right size. That test caught a permissions issue in our restore script that would have cost real time during an actual emergency, when the last thing anyone wants is to be debugging a restore script instead of restoring the actual site.
We're now doing that test restore, at minimum, whenever we set up the backup system on a new client project, rather than assuming a working dump automatically means a working restore.
What we learned about credential handling along the way
Building this also forced us to clean up something we'd been sloppy about, database credentials for the backup script living in plain text in a location that, while not publicly accessible, wasn't as locked down as it should have been. We moved credentials into a file outside the web root entirely with restrictive file permissions, readable only by the user the cron job runs as, which is a small change but closes off an entire category of risk if a server were ever compromised through some unrelated vulnerability.
Where this sits in our process now
Cheap insurance, and it's now on our project launch checklist right next to setting up analytics and confirming SSL is configured correctly, one of a handful of unglamorous items that don't demo well to a client but that we're not willing to launch a project without anymore. We've priced the setup time into every project's estimate rather than treating it as optional extra scope, since the alternative, explaining to a client after the fact why we didn't have this in place, is a conversation we're not interested in having again.
Retrofitting older projects
The harder question was what to do about client sites we'd already launched before this became standard practice. We went back through every active maintenance contract and added the backup cron job to each one over the following couple of weeks, treating it as unbilled cleanup work rather than something to invoice separately, since the gap was our own process failure to catch earlier, not a new feature the client asked for. It felt like the right call even though it cost us some unbilled hours, given that the alternative was knowingly leaving older clients exposed to the exact scenario that had just scared us on a newer one.
A note on how much this actually costs to run
Storage for a rolling week of compressed SQL dumps, even for our larger client databases, has turned out to be a genuinely small cost, small enough that we've folded it into our existing hosting overhead rather than passing it through as a line item on client invoices. The cron job itself runs in well under a minute on every site we've set it up on so far, quiet enough that we mostly only notice it exists when checking the off-host storage account periodically to confirm new backups are actually arriving on schedule, which we've started doing as a monthly spot check rather than assuming a silent system is a working one.
The broader habit this reinforced
More than the specific backup mechanics, this whole episode reinforced a habit we're trying to apply more broadly this year: verify the safety net actually exists rather than assuming it does because someone, somewhere, probably set one up. We'd made the same unverified assumption about our own local development backups too, it turned out, and a quick audit there found a similarly stale situation on one of our own machines. Fixing that took twenty minutes once we actually looked, which is the whole point, most of these gaps aren't hard to close, they're just easy to never notice until something forces the question.
We're now trying to build a short, informal habit of asking "have we actually verified this, or are we just assuming it," specifically for anything described as a safety net, backups, error logging, uptime monitoring, rather than waiting for a scare on each one individually before checking. It's a cheap question to ask and, so far, an uncomfortably useful one every time we've actually asked it out loud instead of assuming the answer, usually turning up at least one gap that's been sitting there quietly for longer than either of us would like to admit. Small, boring, unglamorous work, and exactly the kind we keep learning pays for itself the one time it matters. We would rather spend an afternoon now writing a script neither of us enjoys writing than spend a weekend later reconstructing a client database from memory and a stale host snapshot, which is exactly the position this whole habit exists to keep us out of going forward. Cheap, boring, and now simply part of how every project starts, which is exactly where we want a safeguard like this to live. A short list, easy to forget, easy to keep once it is written down.