Who this is for. MSP engineers and directors building, or buying, a shared platform that several client environments will live on, and who will be responsible when the network design turns out to have a hole in it. The focus is the network and the physical failure structure, because that is where private clouds built by MSPs most often go wrong.
What “private cloud” means here
In an MSP setting the phrase usually describes a pool of virtualised compute and storage, run for you or by you, with some self-service and automation, used by many clients who are not meant to see each other. It does not have to resemble a public cloud’s feature set. What makes it a platform rather than a collection of servers is that you can add a tenant, change a tenant and remove a tenant by following a repeatable procedure, and that the separation between tenants is designed rather than accidental.
That repeatability is the real design goal. A network you can only keep consistent by hand will drift, and drift is where tenants end up reachable from each other.
Start with failure domains
Before drawing a single VLAN, write down what can fail together. A failure domain is anything whose failure takes out everything inside it: a power feed, a top-of-rack switch pair, a storage array, a hypervisor cluster, a site. Good designs make the domains explicit and keep the replicas of anything important in different ones.
A practical way to record it is a table with one row per component class and three columns: what it depends on, what depends on it, and what happens when it goes.
| Component | Depends on | Failure takes out | Mitigation to ask about |
|---|---|---|---|
| Top-of-rack switch pair | Power feed A and B | Hosts in that rack, if single-homed | Dual-homed hosts across a switch pair |
| Storage array | Controller pair, fabric | Every VM on its datastores | Second array or replication, tested |
| Cluster | Shared storage, management | Its tenants | Spread tenants across clusters by criticality |
| Management plane | Its own database and network | Ability to administer everything | Separate recovery plan; see dedicated hosting |
| Site | Power, cooling, carrier | Everything | Second location, with honest recovery objectives |
If a supplier runs the platform, ask for the equivalent table in their words. An answer that cannot name the shared failure domains is itself an answer.
Segmentation: VLANs, routing instances and firewalls
A VLAN (IEEE 802.1Q) separates broadcast domains on a shared switch fabric. The tag is 12 bits, which gives 4,094 usable VLAN IDs (0 and 4095 are reserved). For a few dozen tenants that is ample. At very large tenant counts, or when tenants need many segments each, overlay technologies such as VXLAN, whose network identifier is 24 bits (about 16.7 million values), remove the ceiling at the cost of more moving parts.
A VLAN alone is not a routing boundary. If two tenants’ VLANs end up on the same router interface or the same flat routing table, separation comes only from firewall rules, and rules accumulate mistakes. The stronger pattern is one routing instance per tenant (VRF), so that tenant routes cannot see each other by construction, with a deliberate, firewalled path to shared services such as backup and monitoring.
A minimal Linux illustration of the idea, for a lab or test host (it changes the network configuration, so do not run it over the only SSH session to a production machine):
# Create a routing instance for tenant 03 and attach its VLAN interface to it
ip link add vrf-tnt03 type vrf table 1003
ip link set vrf-tnt03 up
ip link add link eth1 name eth1.203 type vlan id 203
ip link set eth1.203 master vrf-tnt03
ip addr add 10.203.0.1/24 dev eth1.203
ip link set eth1.203 up
# Verify: routes for this tenant live only in its own table
ip route show vrf vrf-tnt03Commercial switches, routers and firewalls expose the same idea under their own syntax, so use this for understanding the model rather than as a recipe for your hardware.
Layer the controls rather than choosing one: VLAN or overlay for separation on the wire, a routing instance per tenant, a firewall policy per tenant with a default of deny, and explicit, documented exceptions. Review exceptions on a schedule; every unreviewed exception is a future incident.
Address planning when every client uses 192.168.1.0/24
Private IPv4 address space is limited to three ranges defined in RFC 1918: 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16. Your clients will pick from them, and many will pick the same handful of subnets. Sooner or later two tenants will want the same network and, worse, both will need a VPN to a client site that uses it too.
You have three honest options, and you should choose deliberately per tenant:
- Keep each tenant in its own routing instance and accept overlap. Tenants never talk to each other, so identical ranges do not collide. Overlap is only a problem at the edges where tenants meet shared services or client networks.
- Renumber on onboarding into a range you allocate. Clean, and a lot of work for the client.
- Translate addresses (NAT) at the tenant edge. It works, and it makes troubleshooting, logging and some applications harder. Use it where renumbering is not possible, and record every translation.
Whatever you choose, keep a single register of every range in use, on your side and on the clients’ side, with the owner and the purpose. A plain table is enough:
| Range | Owner | Purpose | Notes |
|---|---|---|---|
| 10.250.0.0/16 | Platform | Management network | Not routed to tenants |
| 10.251.0.0/16 | Platform | Backup and monitoring transit | Firewalled per tenant |
| 10.203.0.0/24 | Tenant 03 | Server VLAN 203 | Client side also uses 192.168.1.0/24 at site |
| 100.64.0.0/10 | Avoid | Carrier-grade NAT shared space (RFC 6598) | Often used by ISPs; do not assign to tenants |
(The ranges above are an example scheme, not a recommendation of specific numbers.)
Point-to-point links between your routers and a tenant edge can use /31 networks (RFC 3021) to avoid wasting addresses. If you plan IPv6, the usual guidance (RFC 6177) is to give an end site a /48 and carve /64s out of it, which keeps subnetting trivial. Decide this early; retrofitting an address plan is far harder than starting with one.
The management network and out-of-band access
Everything that administers the platform belongs on a management network that no tenant can reach, and that is not reachable from the internet: hypervisor management interfaces, storage controllers, switch consoles, and the baseboard management controllers (BMCs, with names like iDRAC, iLO or IPMI interfaces) in each server. BMCs deserve particular care. The IPMI 2.0 protocol has well-documented weaknesses in its authentication, and BMC firmware is an attractive place to hide. Keep them off tenant networks and off the internet, change defaults, and patch firmware on a schedule.
Add out-of-band access: a path into the platform that works when the production network does not. Common forms are a console server with a separate uplink, a cellular or secondary-carrier connection to the management network, or a supplier-run equivalent. Test it by disabling a production link in a maintenance window and confirming you can still get in. An out-of-band path you have not used is a hypothesis.
Monitoring that knows about tenants
Tag every metric and log line with a tenant identifier at the source. Alert on the shared things (uplink saturation, storage latency, CPU wait, hypervisor memory pressure) as well as on individual servers, because the shared resources are where one tenant hurts another. Keep per-tenant dashboards for reporting and for the first question every client asks after an incident: was it us?
How private clouds fail
- A trunk that allows every VLAN. Trunk ports should list the VLANs they carry. A “permit all” trunk exposes a tenant VLAN to any device that can reach the port.
- Using VLAN 1 or an untagged native VLAN carelessly on trunks, which has repeatedly been the foothold in VLAN-hopping findings.
- Overlapping ranges discovered at the first VPN. Build the address register before the first tenant, not after the tenth.
- A shared service reachable from all tenants without per-tenant firewall policy, so the backup server or monitoring collector becomes a bridge.
- Management reachable from tenant networks, usually through convenience rules that outlive their reason.
- Hand-built tenant networks. At more than a handful of tenants, build from templates and automation, then compare running configuration against the template.
- Failure domains that exist only on the diagram: two switches that share one power feed, a “second site” that depends on the first for DNS.
What to do next
Write the failure-domain table and the address register for the platform you have or the one you are evaluating. Then use the requirements planner to capture the segmentation, management and monitoring requirements that follow from your tenant count and support model, and read multi-tenant isolation for the compute and storage side of the same separation. If you are choosing a supplier, take both documents to the conversation described in evaluating a hosting provider.