Skip to content
MSP SERVER HOSTING

Virtual servers for MSPs: a capacity planning method that survives a host failure

Who this is for. MSP technical leads who place client servers on shared virtualisation hosts, whether you rent the hosts or own them, and who need to answer two questions with evidence: how many hosts do we need, and when do we need the next one? The method is platform-neutral. The arithmetic is illustrative: every number in the worked examples is invented to show the calculation, not a benchmark or a recommendation.

The method in six steps

Capacity planning for a multi-tenant platform fails when it is done per virtual machine and passes when it is done per resource, per time window, and in the failed state. The sequence below is the one worth repeating each quarter.

  1. Group workloads into classes and measure each class.
  2. Sum demand the way it actually coincides, not the way it averages.
  3. Decide overcommit per resource, with the signal that will tell you it went wrong.
  4. Convert storage demand to what the storage actually has to do, not what the guest asks for.
  5. Add failure headroom, then check the arithmetic in the failed state.
  6. Turn growth into a runway in months, and compare it with your real lead time to add capacity.

Step 1: Classes and the right measurements

Do not size from a spreadsheet of server names. Group by behaviour: file and print, directory and identity, line-of-business application tiers, databases, hosted desktops, web. Each class has a characteristic shape. File servers are quiet and bursty at logon and backup time. Databases are steady on memory and unforgiving on storage latency. Hosted desktops spike on the morning logon wave.

For each class, record peak CPU, memory in use (the working set, not the allocated amount), storage IOPS, storage latency, throughput and network. Sampling interval matters: a five-minute average will hide a ninety-second spike that your users notice. If your monitoring cannot go finer than five minutes, say so in the plan and add margin you can justify.

Step 2: Sum of peaks versus peak of sums

The key idea is that tenants do not all peak at once, but they do not peak independently either. Month-end, the Monday logon wave, the nightly backup window, and patch Tuesday are coincident across many tenants.

Worked example (illustrative numbers). Three tenants peak at 8, 6 and 5 cores. The sum of individual peaks is 19 cores. If you plot all three on the same time axis and the combined load never exceeds 14 cores, the peak of sums is 14. Planning on 19 wastes money; planning on the average, say 6 cores, invites pain at 14. The honest number is the peak of sums measured over a period that includes your worst known coincidences, plus whatever growth margin you can defend.

For tenants you have not onboarded yet, you do not have a time series. Use the class profile, add a deliberately generous margin, and replace it with measured data after the first month.

Step 3: Overcommit is an output, not a setting

Overcommit means promising more virtual resource than the hardware has. For CPU it is routine, because most virtual CPUs are idle most of the time. There is no universal safe ratio, and you should be suspicious of anyone who quotes one without knowing the workloads. Treat the ratio as something you earn from observation:

  • Start conservative for latency-sensitive classes (databases, hosted desktops) and looser for file, print and identity.
  • Watch the platform’s CPU wait metric (CPU ready on VMware-style consoles; the equivalent run-queue and wait counters elsewhere; steal time inside a Linux guest). Rising wait with idle guests means you are over the line.
  • Right-size virtual machines. A VM with eight virtual CPUs that needs two does not just waste capacity: it must be scheduled across eight, and that raises wait for everyone on the host.

Memory deserves a stricter stance. Overcommitting memory converts demand spikes into ballooning, compression or host swapping, and the symptom appears as unexplained latency in a tenant that did nothing wrong. Many providers keep database and regulated tenants fully memory-backed and allow overcommit only where the consequence is bounded and visible. Always reserve host memory for the hypervisor and its management agents.

Step 4: Storage is three numbers, not one

Capacity in gigabytes is the easy part. A tenant also needs a number of operations per second (IOPS), a latency target, and a throughput. The pool must deliver all of them simultaneously for all tenants, including during backup.

Guest IOPS are not the same as the work the array does. Parity and mirror schemes multiply writes. The write penalty commonly cited for random writes is 2 backend operations for RAID 10, 4 for RAID 5 and 6 for RAID 6, though modern arrays with caching and full-stripe write behaviour can differ, so check your platform’s documentation.

Worked example (illustrative). The combined peak from the guests is 4,000 IOPS, 70 percent reads and 30 percent writes.

  • Reads: 4,000 x 0.70 = 2,800 backend operations.
  • Writes: 4,000 x 0.30 = 1,200 guest writes.
  • On RAID 10: 1,200 x 2 = 2,400, for a backend total of 5,200 IOPS.
  • On RAID 6: 1,200 x 6 = 7,200, for a backend total of 10,000 IOPS.

Same tenants, nearly double the backend load. That is why “how many IOPS do I need” cannot be answered without the layout.

Thin provisioning deserves its own warning: allocating more virtual disk than the pool holds is fine until the pool fills, at which point virtual machines can pause or fail. If you thin-provision, alarm on pool free space well before it matters and track the growth rate.

Step 5: Failure headroom

A cluster must carry its load when a host is down, since hosts fail and also get patched. The check is arithmetic:

required capacity  D  <=  (N - 1) x C x u

where N is the number of hosts, C is the usable capacity of one host in the resource you are checking, and u is the highest utilisation you are willing to run at while a host is out. The value of u is your design choice, not an industry constant: it reflects how much degradation you will tolerate during a failure.

Worked example (illustrative). Measured coincident peak demand is D = 90 cores’ worth. Each host provides C = 32 usable cores. You choose u = 0.8 for the failed state.

  • Required hosts that must remain: 90 / (32 x 0.8) = 3.52, so at least 4 hosts must survive.
  • N – 1 = 4, so N = 5.

Repeat this for memory, which is usually the binding constraint, and for storage and network where they are shared. The smallest N that satisfies every resource is your answer. Note what this implies for host size: with few large hosts, losing one removes a large fraction of capacity and N-1 gets expensive, while many small hosts carry more overhead. Neither is automatically right.

If your recovery objectives are tight or tenant count is large, ask whether N+1 is enough, since a second failure during repair is a real planning case. The requirements planner flags when to price N+2.

Step 6: Growth runway

Take your measured demand D, your failed-state ceiling (N – 1) x C x u, and your net monthly demand growth g (onboarding plus organic growth minus retirements). The runway in months is roughly (ceiling – D) / g. Compare it with the real lead time to add a host or storage shelf: ordering, delivery, racking, burn-in, joining the cluster. If runway is shorter than lead time plus a margin, you are already late. Make the lead time an explicit number you have asked your supplier for, rather than an assumption.

Measuring on a real host

These read-only commands help you gather evidence. Run them on hosts or guests you administer, and sample during a known busy period.

# Linux: CPU wait/steal, and pressure stall information (kernel 4.20+, if enabled)
vmstat 1 5
cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io

# Linux: per-device latency and queue depth (sysstat package)
iostat -x 1 5
# Windows: average read latency on all physical disks, 5 samples, 1 second apart
typeperf "\PhysicalDisk(_Total)\Avg. Disk sec/Read" -sc 5

If you need to characterise a storage pool before go-live, a synthetic test is fine, but only against a scratch file on a non-production volume, and never against a device node:

# Creates and then reads a 1 GiB test file under /mnt/scratch. Safe on an empty, non-production volume.
fio --name=randread --filename=/mnt/scratch/fio.test --size=1g --rw=randread --bs=4k \
    --iodepth=32 --direct=1 --ioengine=libaio --runtime=60 --time_based --group_reporting
rm /mnt/scratch/fio.test

Synthetic load on a shared pool affects every tenant on it. Run it before go-live or in an agreed window, never as a casual test on a production pool.

How capacity plans fail

  • Sizing to averages. The plan works every day except the day it matters.
  • Planning the healthy state only. The cluster that fits at N hosts and not at N minus 1 is a cluster with no failure tolerance.
  • Counting the backup window as free. Backups, snapshots and replication read from the same storage tenants use. Schedule or throttle them.
  • Ignoring the per-VM overheads. Hypervisor memory overhead, snapshots that never got merged, and monitoring agents all consume capacity.
  • Allowing one tenant to size the platform. A single large tenant can dominate the peak. Price it separately or place it on its own pool. See multi-tenant isolation.
  • No lead time in the plan. Capacity you cannot get for ten weeks must be ordered at ten weeks of runway, not zero.

What to do next

Gather a month of per-class peaks, run steps 2 to 6 for each resource, and write the failed-state ceiling and the runway on one page you revisit quarterly. Use the requirements planner to record the headroom posture your estate implies, then read hosting economics to turn the host count into a cost model per tenant.