Backup and recovery architecture for client workloads hosted on your platform

Who this is for. MSPs who host client servers on a platform they run or rent and who are, in practice, the last line of defence when something is deleted, corrupted, encrypted or burned down. If a client assumes “hosted by our MSP” means “recoverable”, this article is about earning that assumption.

Begin with the failures, not the product

Backup design starts from what you need to survive. Walk each scenario and ask what you would restore, from where, in what order, and how long it would take. The same platform recovers very differently from each:

Scenario	What is lost	Recover from	What usually goes wrong
A tenant deletes or corrupts a file or database	Data, not infrastructure	Point-in-time backup, restored beside the original	Retention too short to cover when the damage was noticed
One virtual machine is lost or fails to boot	One server	Backup or replica of that VM	Restore target and network not ready
A host fails	Capacity	Other hosts, if headroom exists	Cluster cannot carry the load at N minus 1
Ransomware reaches a tenant	That tenant’s data and often its backups	Immutable or offline copy	Backups reachable with the same credentials as production
The shared storage array fails	Many tenants at once	Second array, replica or backup repository	Backup repository sits on the same array
The backup system itself is compromised or lost	Your ability to restore anything	Independent copy, rebuilt console	The console had no backup of its own
The whole site is lost	Everything	A second location	Assumes services (DNS, identity) that were in the lost site

If you cannot fill in the “recover from” column for a row, that row is the gap.

Set recovery objectives per workload class

Recovery point objective (RPO) is how much recent data you can afford to lose, measured in time. Recovery time objective (RTO) is how long a service can be unavailable. Both come from the client’s business, not from your tooling. Ask each tenant what an hour of lost work or an hour of downtime costs them, then group workloads into a small number of tiers. The table is an illustration of the shape; the numbers are not recommendations.

Tier	Example workloads	Example RPO	Example RTO	Typical mechanism
A	Line-of-business databases, identity	Minutes to one hour	One to four hours	Replication plus backup, rehearsed failover
B	Application servers, file services	Four to twenty-four hours	Four to twenty-four hours	Frequent backup, pre-staged restore capacity
C	Archives, test systems	Twenty-four hours or more	Days	Nightly backup

Three or four tiers are manageable. Per-tenant bespoke objectives are not, because every exception has to be tested and documented separately. Write the tiers into the contract so that nobody is surprised by what “backed up” means.

Replication is not backup. Replication copies a deletion or an encryption event to the second site as faithfully as it copies data. Use replication to meet short recovery times and backup to meet point-in-time recovery, and do not let one stand in for the other.

The copies: independence matters more than count

The classic 3-2-1 rule asks for three copies of the data, on two different kinds of media, with one copy off-site. A common extension, 3-2-1-1-0, adds one copy that is offline or immutable and zero errors after verification. The principle behind both is independence: a copy only counts if the event that destroys the first copy cannot also destroy it.

Test independence in three dimensions:

• Failure domain. Different storage system, different rack, ideally a different site.
• Credentials. A copy that your daily production administrator account can delete is not protected from an attacker who has that account. Use separate credentials, held by different people or systems.
• Mutability. Immutable storage (for example object storage with a write-once retention setting, a hardened repository with immutability enabled, or media that is physically offline) stops deletion and encryption for the retention period even by a privileged user. Read the vendor’s documentation carefully: some immutability modes can be shortened by an administrator, and some need to be enabled when the storage is created.

Separate tenants inside the backup system

A multi-tenant backup platform is itself a multi-tenant system, and it deserves the same isolation attention as the compute layer. Use a separate repository, or at minimum a separate logical container and set of credentials, for each tenant, so that one tenant’s compromised credentials, or one tenant’s offboarding, does not touch another’s data. Where tenants have encryption requirements, use per-tenant keys and decide in advance where those keys live and how they are recovered. A key that exists only inside the system it protects is lost when that system is.

Retention and deletion obligations vary by tenant and by contract and can conflict (one requires long retention, another requires prompt deletion on exit). Record them per tenant and check them against what the system actually does.

Application consistency

A backup of a running database taken without cooperation from the database is crash-consistent: it looks like the server lost power. Many databases recover from that, some do not, and none give you a clean transaction boundary. Application-consistent backups quiesce or snapshot through the application’s own mechanism. On Windows, this is typically the Volume Shadow Copy Service, and you can check the health of the writers it relies on:

# Run in an elevated prompt on the server being protected. Every writer should show State: Stable and Last error: No error.
vssadmin list writers

For databases that offer a native dump or log shipping facility, use it as well as image-level backup. Know which one you would actually restore from, and test that path.

Rehearsing restores

A backup that has never been restored is an assumption. Rehearsal serves two purposes: proving the data is good, and measuring how long restoration takes. The second matters more than most MSPs expect, because restore throughput, not backup throughput, usually limits the achievable RTO.

A simple, repeatable program:

1. Verify integrity on every job. Many tools can verify checksums or read-back automatically. Turn it on.
2. Restore a sample on a schedule you set. Pick tenants and workload classes in rotation, restore into an isolated network, check the application actually starts and the data looks right, and record the elapsed time.
3. Rehearse a bad day at least occasionally. Restore the largest tenant, or several tenants in parallel, to see where the bottleneck is. A restore that takes four hours alone could take far longer when six tenants restore at once through one repository.
4. Test the path that does not exist yet: restoring when the backup console is gone, using only the repository and your documented procedure.

For open-source tooling that you may use for file-level or application backups, the checks look like this. The repository location and password file are examples; use your own and do not paste secrets into shell history.

export RESTIC_REPOSITORY=/srv/backups/tenant-a      # example path
export RESTIC_PASSWORD_FILE=/etc/restic/tenant-a.pw   # file readable only by the backup service account

restic snapshots                                  # confirm recent snapshots exist
restic check --read-data-subset=10%               # verify structure and read a sample of the data
restic restore latest --target /tmp/restore-test  # restore into a scratch directory, never over production

Whatever tool you use, record the date, the tenant, what was restored, how long it took and who checked it. Those records are your evidence when a client or an auditor asks.

Disaster recovery design: what the second site needs

When the recovery tier calls for a second location, the work is not only copying data. The second site needs capacity to run the workloads, a network design that includes addressing and routing, name resolution and identity services that are available before the workloads are, a way to redirect users (often DNS with a short time-to-live set well in advance), and a rehearsed procedure for failing over and, later, failing back. Failback is the part most plans omit. Decide how data written at the second site returns to the primary, and who decides when.

State plainly what the design does not cover. A plan that restores virtual machines but not the firewall rules, certificates or licensing the application needs is a partial plan, and the partial part is what will be discovered during the incident.

Common mistakes

• Backups reachable with production credentials. The most common reason ransomware takes the backups too.
• Repository on the same storage as the data it protects.
• No backup of the backup console and its catalogue, so that a rebuild has to rediscover everything.
• RTO claims based on a single-server restore with no allowance for restoring many tenants at once.
• Retention that is shorter than the time it takes to notice a problem. Slow corruption is discovered late.
• Never testing the restore of the management plane. See dedicated hosting for how to treat it as its own tier.
• Quoting recovery objectives you have never measured in a client contract.

What to do next

Classify your workloads into three or four tiers, fill in the failure-scenario table for the platform you run or are evaluating, and schedule the first rehearsal. The requirements planner turns recovery point and recovery time answers into design considerations, and evaluating a hosting provider lists what to ask a supplier about their backup and recovery claims before you rely on them.