Backups and monitoring: what your system needs after launch

On launch day, everyone feels the project is finished. But from that moment on, orders, invoices, and stock movements start passing through the system every day, and when the system stops, the business itself stops.

Unless it’s clear who protects the data and who monitors the system, the first major failure will be the first real test of those questions. In this article, we explain what your system needs once it’s in day-to-day operation: backups you can rely on, monitoring that catches problems early, alerts that reach a specific person, and a written plan for what happens when something fails.

A backup whose restore has not been tested is no guarantee

Taking a backup doesn’t mean it can be restored. The problem shows up when you need it: an incomplete file, a backup that won’t open, or a restore that takes far longer than the business can tolerate.

That’s why the US Cybersecurity and Infrastructure Security Agency (CISA), in its guide to countering ransomware, recommends regularly testing the availability and integrity of backups in a disaster recovery scenario. In practice, that means restoring a full backup to a separate environment and checking that the system runs on it and that the latest transactions are in it.

Then there are decisions for the business owner to make, not the technical team alone:

• How often is a backup taken? The real question behind it: how much work are you willing to lose and re-enter if the system fails right now?
• How long are backups kept? That’s tied to your accounting and legal obligations, and to how long an error in the data might go unnoticed.
• What is an acceptable time for the system to be back? An online store in its sales season is different from a monthly reporting system.

We do not set generic numbers for these decisions, because they vary from one company to another. But we ask that they be written down and approved, and that backups be designed around them.

Suppose an employee deleted a set of items by mistake, and no one noticed until two weeks later. If backups are kept for only a few days, you won’t find one from before the mistake. That’s why the retention period is tied to how quickly errors are caught in your business, not to storage space alone.

More than one location, and separate access

A backup kept on the same server is lost along with it if the disk fails or the server is breached. So backups are kept in more than one place, including a copy off-site, in another data center or with an independent storage provider.

The CISA guide notes that many ransomware variants search for accessible backups in order to delete or encrypt them, and it recommends offline, encrypted backups and enabling deletion protection on storage resources. The practical result is that the account that administers the system must not be able to delete the backups.

This is where backups meet permissions. We explained in “User permissions and audit trails: the basis of trust in any business system” why no single account should have enough permissions to misuse the system on its own. The same rule applies here: the person who runs the system day to day is not the person who has permission to delete the backups.

Suppose the password to the server administrator account is stolen. If the backups sit under that same account, they could be deleted along with the system within the same hour. If they’re stored with separate access and protected from deletion, your restore point stays intact.

What to monitor

Monitoring means your team learns about a problem before a customer or an employee calls to report it. You don’t need to monitor everything, only the signals that usually come before an outage:

• Availability: do the system and its APIs respond from outside the internal network?
• Errors: a sudden rise in requests that end in an error.
• Slow operations: a report or screen that has started taking an unusually long time to open.
• Disk space: the database, files, and logs grow every day until the disk is full.
• Background jobs: posting journal entries, syncing branches, or preparing reports, when they fail silently.
• Certificate expiry: the security certificate for the website or the APIs, as its expiry date approaches.

Suppose the nightly sync job between branches stops. Nothing shows on the screens, but the warehouse stock levels drift further from reality day after day. Monitoring that alerts on the first failure catches the problem the same day, instead of it being discovered at stocktake.

Logs

When a failure happens, the logs are the main source for understanding what went on. OWASP guidance recommends that, for each event, the application keep a record of when it happened, where, who did it, and what happened, and that logs be protected against tampering. We add that logs should be copied off the server that produces them, so they remain available if it fails.

Logs also help with everyday questions, such as why a payment failed or why a report was late, without asking the employee to repeat what they did step by step.

Alerts that reach a named person, and a runbook for failures

An alert sent to a shared email address that nobody reads isn’t much different from no alert at all. Each type of alert gets a named person responsible, a stand-in for when they’re away, and a channel they actually see, such as WhatsApp or an alerting app on their phone.

Alerts also lose value when there are too many of them. If the team gets lots of alerts every day that call for nothing, it will ignore the one that matters. So alerts are reviewed after launch, and any that don’t call for real action are removed.

The alert itself is tested, just as the backup is: a simple failure is staged in the test environment, and you check that the message reached the right person and that they knew what to do.

The CISA guide also recommends creating an incident response plan, maintaining it, and practicing it regularly. For business systems, a simple written runbook that answers specific questions is enough to start with:

1. How do we know there’s a failure, and who confirms it?
2. Who is informed inside the company, and what do we tell the branches and customers?
3. What are the first steps for each known type of failure?
4. When do we decide to restore from a backup, and who owns that decision?
5. How do we document what happened once the failure is over, and what do we change so it doesn’t happen again?

Who owns the system after launch

We distinguish between launch and day-to-day operation. Launch is an event with a date; operation is ongoing work: security updates, reviewing backups, following up on alerts, and planned upgrades. What often harms systems after launch is that each side assumes these tasks belong to the other.

So before launch, it’s settled who takes charge of operation: your in-house team, the vendor, or a clear split between the two. The responsibilities are written down in one document that everyone knows: who reviews the backups, who receives the alerts, who approves updates, and who has access to the servers.

If the vendor is the one running the system, ask for the backups to be kept in an account your company owns, and for a regular report on restore test results and the alerts that occurred. The data is yours, and you must be able to reach it no matter who ends up managing the system.

A system whose operation no one owns works until the first failure, and then the search for whoever is responsible begins while the business stands still.

A post-launch checklist

• A full backup restore, tested in a separate environment, with the test repeated regularly.
• A written decision on how often backups are taken, how long they are kept, and what is an acceptable time for the system to be back.
• An off-site backup, with separate access and protection from deletion.
• Monitoring of availability, errors, slowness, disk space, background jobs, and certificate expiry.
• A named person responsible for each alert, and a stand-in.
• Logs stored off the server and protected from modification.
• A written failure runbook that the team has practiced.
• A known owner for day-to-day operation and for approving updates and upgrades.

These items call for clear decisions and known owners more than for complex tools. When we build backend services and APIs, we design backups, monitoring, and logs along with them, not after launch.

If your system runs today without tested backups or clear monitoring, or you’re preparing to launch a new system, you can discuss your situation with the ITS team, so that together we can review your operations plan before the first failure puts it to the test.