Multi-tenant isolation: choosing the boundary for each client

Who this is for. MSP owners and technical directors who run, or are about to run, several clients on shared infrastructure and need to decide where one client ends and the next begins. It assumes you can read a hypervisor console; it does not assume you have a formal architecture background.

Isolation is a stack, not a switch

When a client asks “are we isolated from your other customers?”, the honest answer is a list. A tenant boundary has to hold in at least five places, and a platform is only as separate as the weakest of them.

• Compute. CPU time, memory and the scheduler that arbitrates between workloads.
• Storage. The pool, the queue in front of it, the cache, and the replication or snapshot machinery behind it.
• Network. Switching, routing, firewalling and the address plan.
• Identity and management. Who can administer what: hypervisor consoles, the RMM, the backup console, shared service accounts.
• Data at rest and in backup. Where copies live, under which keys, reachable with which credentials.

Most incidents in shared estates are not dramatic breakouts. They are a tenant’s month-end job saturating a storage queue, a firewall rule that was written for one client and matched another, or a shared admin credential. Plan for those first.

Three boundary types

Boundary	What tenants share	Main weakness	Reasonable when
Virtual machines on a shared hypervisor	Physical CPU, memory, storage, network	Contention; a flaw in the hypervisor or a hardware side channel	The default for most small and mid-size client workloads
Containers on a shared kernel	The operating system kernel as well as the hardware	A kernel flaw crosses every tenant; limits are policy, not walls	Your own application tiers or a single trusted client’s services, rarely mutually untrusting clients
Dedicated hardware or cluster per tenant	Only the management plane and, usually, the network upstream	Cost, utilisation, operational load	Strict compliance scope, or workloads that cannot tolerate neighbours

Virtual machines are the workhorse for a reason. The hypervisor presents each tenant with its own virtual hardware, and the attack surface between tenants is much smaller than a general-purpose kernel’s. Two caveats belong in your risk register. First, hypervisor escapes exist as a vulnerability class, and you should treat hypervisor patching as a tenant-wide emergency process. Second, the speculative-execution disclosures of 2018 (Spectre, Meltdown and related issues) demonstrated that sharing a physical core can leak data between workloads. Mitigations exist and cost performance; whether they matter to you depends on the platform, so ask whoever operates it.

Containers share one kernel. Linux control groups (cgroups) limit what each container can consume, and namespaces limit what it can see, but a kernel vulnerability is a vulnerability for every container on the host. This is fine for services you build and control, and a poor fit for running different clients’ arbitrary workloads side by side. Orchestrators add their own isolation vocabulary (namespaces, quotas, network policy), and those are administrative separation, not a hard boundary. Treat the namespace as a labelling convenience unless you have specifically hardened it.

Dedicated hardware is the strongest separation and the most expensive way to get it. It is often used for a subset of tenants: the one regulated client, the one with a database that cannot share spindles. The shared management plane then becomes the remaining common point, and it deserves the hardening the tenants no longer get from sharing nothing.

Noisy neighbours: detect first, then limit

A noisy neighbour is a tenant whose consumption degrades others. It almost never announces itself as “CPU 100%” on the victim. It shows up as symptoms on innocent workloads: slow logons, database queries that time out, backups that overrun.

What to watch, platform-neutrally:

1. CPU wait, meaning time a runnable virtual CPU spends waiting for a physical one. Different platforms name it differently (CPU ready on VMware-style consoles, steal time inside a Linux guest). Sustained waiting while the guest shows idle is the signature.
2. Storage latency per workload, not just pool throughput. A pool can look healthy on bandwidth while queue depth has already hurt everyone’s small writes.
3. Memory pressure at the host: ballooning, compression or swapping on the host side means guests are paying for someone else’s demand.
4. Network uplink saturation and dropped packets on shared uplinks.

Once you can see it, limit it. Limits exist at every common layer:

# Hyper-V: reserve and cap a VM's share of processor resources (percentages)
Set-VMProcessor -VMName "tnt03-sql01" -Reserve 20 -Maximum 80

# Hyper-V: cap and floor storage IOPS for one virtual disk
Set-VMHardDiskDrive -VMName "tnt03-sql01" -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 `
  -MinimumIOPS 100 -MaximumIOPS 1500

# KVM/libvirt: cap a running guest's disk IOPS (applies live; repeat in the domain XML to persist)
virsh blkdeviotune tnt03-sql01 vda --total-iops-sec 1500 --live

# Container on a shared Linux host
docker run --cpus=2 --memory=4g --pids-limit=512 --name tnt03-web example/image:tag

# systemd slice for a tenant's services; --runtime means it does not survive a reboot
sudo systemctl set-property --runtime tenant-a.slice CPUQuota=200% MemoryMax=8G

The numbers above are placeholders, not recommendations. Choose limits from the tenant’s measured peaks and your contract. Parameter names and units vary by platform and version, so confirm them in your platform’s documentation before applying anything to production.

Limits are policy you are imposing on a client’s workload. A cap set too low looks like an outage to the client. Apply new caps to a non-critical tenant first, watch the effect, and tell the client what you changed.

A worked example

Four tenants share a host cluster and a single storage pool. Tenant C runs a monthly reporting job against a database, and for two hours each month the other three tenants complain about slow file opens.

The monitoring shows nothing alarming at the host level: CPU is moderate, memory is fine. Per-workload storage latency tells the real story: Tenant C’s job issues a large number of reads at high queue depth and the pool’s latency rises for every workload using it. Three steps fix it in the right order:

1. Cap tenant C’s virtual disks at an IOPS ceiling so that the job takes longer but cannot saturate the pool. Tell tenant C to expect a longer run.
2. Alert on pool latency, not just per-VM latency, so the next tenant who does this is noticed in minutes.
3. Decide on structure: if tenant C’s job matters and cannot be capped without business harm, move its database to a separate pool or its own storage tier, and price it that way. Isolation you have to provide is a cost, and the tenant who needs it should be paying for it. The economics article covers how to structure that conversation.

Management-plane isolation

The most dangerous shared component is usually not the hypervisor but the tooling used to administer everything. A remote monitoring and management platform with agents on every tenant’s machines is, by design, an account that can run code everywhere. Incidents in which attackers compromised management software and used it to reach many downstream customers at once are well documented in the industry, and they are the reason this layer deserves separate treatment.

• Give every administrator a named account with multi-factor authentication. Retire shared logins.
• Use separate credentials for the management plane and for tenant environments, so a tenant-side compromise does not hand over the console.
• Restrict where administrative interfaces are reachable from; do not publish hypervisor or hardware consoles to the internet.
• Give each tenant the least access to its own resources that works, and keep tenant users out of your management network.

The dedicated hosting article covers where to run that tooling and how to recover it.

Checking that isolation holds

Do not rely on the diagram. Test with tenants you own, in a test or pre-production environment, and with authorisation for the networks you touch.

# From a test VM in tenant A: confirm tenant B's test range is NOT reachable on common management ports.
# Replace the address with a host in YOUR test tenant B; do not probe production or other organisations' ranges.
for port in 22 135 445 3389; do
  nc -vz -w 3 10.20.0.10 "$port" || echo "port $port blocked as expected"
done

Test the other layers in the same spirit. Run a deliberate load generator against a test tenant while measuring latency on others. Confirm that tenant A’s backup credentials cannot list tenant B’s repository. Confirm that removing a tenant’s administrator account does not leave a shared secret behind. Write down each test and repeat it after platform changes; isolation that was true at build time drifts.

Common mistakes

• Treating the VLAN as the whole story. Separate broadcast domains do nothing about shared storage queues or shared credentials.
• Reserving nothing. Without reservations or caps, a platform has isolation by luck.
• Sizing from averages. Contention happens at peaks. Averages hide it.
• Namespaces as walls. Container namespaces and orchestrator quotas are useful administrative boundaries; they are not equivalent to a hypervisor boundary.
• One flat management network shared by hardware controllers, hypervisors and tenant-facing portals.
• Never testing. Firewall rules, caps and permissions accumulate exceptions. If you do not re-test, you do not know.

What to do next

Decide the boundary tenant by tenant rather than platform-wide: most tenants may sit comfortably in the shared pool while one or two warrant dedicated capacity. The requirements planner turns your tenant count, workload mix and compliance pressure into an isolation model to ask suppliers for. Then work through virtual servers and capacity planning for the compute and storage side, and private cloud and network design for segmentation and address planning.