Where to Run GPU Servers: Data Center Requirements and Three Ways to Deploy

9 min readPacketStream

GPU servers do not drop straight into a standard server rack. What to check on power, cooling and networking, and how renting, building and colocation differ.

On this page
  1. Outgrowing a standard rack
  2. Five things to check
  3. Networking and storage
  4. Data paths and residency
  5. Three ways to get GPUs
  6. Where each falls short
  7. Deciding by workload

Once you decide to bring in GPU servers, the first questions are which GPU and how many. But even with the hardware chosen, it cannot be installed until there is a place with the power to run it and the cooling to carry the heat away. Where GPU servers go and how you obtain them are tied together, so it is hard to settle one before the other. This article looks at what to check in a data center, and then at whether renting, commissioning a build or bringing your own hardware is the right fit.

Why GPU servers outgrow a standard server rack

A GPU server draws far more power than an ordinary server in the same rack space, and gives off that much more heat. Most of the power a server uses ends up as heat, so taking more power means changing how the heat is removed as well. Empty space in a rack does not tell you whether a GPU server can go into it.

NVIDIA's user guide for the DGX Vera Rubin NVL72 shows the gap.

Power shelves in each rack
4
Rating of one power shelf
110 kW
DC voltage distributed inside the rack
54 V
Power configuration of a rack-scale system, from NVIDIA's DGX Vera Rubin NVL72 user guide. These are not PacketStream product specifications.

The 110 kW figure is the rating of one power shelf. Adding the four ratings does not tell you how much power the rack actually draws. The power shelves convert AC power to DC and distribute it inside the rack through a bus bar. The compute trays are cooled by liquid running through manifolds and cold plates.

A PacketStream Colocation Full rack (47U) comes with 10 A at 220 V (2.2 kVA). This is the supply capacity included in the contract, not the power drawn by the equipment. Additional capacity is available in 1.1 kVA increments.

After that, there are five things to check.

Five things to check in a data center for GPU servers

A data center for GPU servers is checked on power, cooling, node-to-node networking, data paths and data residency.

RequirementWhat to askWhat happens when it does not fit
PowerPower available per rack, spare capacity at the data center, and options for expansionHardware is split across more racks, or you wait for an upgrade
CoolingAir cooling only, or liquid cooling infrastructure as wellHardware built for liquid cooling cannot be installed
Node-to-node networkingThe type and topology of the fabric that links GPU nodesAdding nodes does not shorten training time
Data pathsThe circuits that bring data in and out, and where storage sitsTime spent moving data becomes time jobs spend waiting
Data residencyThe country or region data must stay in, and the access pathsThe configuration is redrawn from the start to meet the requirement

Plan for power when adding equipment, too. The first rack may have enough power, but adding racks requires spare capacity at the data center. The hardware decides the cooling method. Equipment built for liquid cooling only goes into a facility with a CDU (Coolant Distribution Unit), the device that distributes coolant.

This is what PacketStream E1 provides.

Maximum power
14.2 MW
Maximum power per building
4.8 MW
Configuration
3 buildings

The 14.2 MW figure is the maximum for the entire E1 data center. To check whether a particular GPU server can be installed, compare its requirements with the power and cooling available at its rack. E1 natively supports CDU-based liquid cooling. It has an OCP-type design built with NVIDIA Rubin-based systems in mind, and it supports DC Power Shelf configurations.

Power and cooling are properties of the building. The other three depend on how things are connected inside and outside it.

Node-to-node networking and where storage sits

With two or more GPU nodes, the network between them and the location of storage set the pace of training. In distributed training the nodes exchange results constantly, so when that traffic is slow the GPUs sit idle waiting for the next data. It is like a kitchen that hired more cooks but still has a single corridor for bringing in ingredients.

That is why a GPU cluster has a fabric for node-to-node traffic that is designed separately from the service network. PacketStream GPU Server runs node-to-node traffic over InfiniBand NDR/XDR, with the topology determined by scale and communication pattern.

Storage sits close to the GPUs for the same reason. If the data read and written during training, and the checkpoints, live on distant storage, training waits every time it saves. PacketStream recommends splitting data across two places according to how it is used.

Parallel file storage (PFS)

  • NVMe-backed and connected to the same fabric as the GPU nodes
  • Holds the data read and written during training
  • Holds checkpoints

Object Storage

  • S3-compatible storage
  • Keeps rarely used dataset originals
  • Keeps output archives

What is easy to miss is that none of this appears on a facility spec sheet. Power and cooling are specifications of the building, but the fabric and storage depend on who designs the cluster and how. The same goes for the paths that leave the building.

Data paths in and out, and data residency

The connection to the outside decides how long it takes to move data and where that data is allowed to stay. New data coming in, and models and outputs going out, repeat for as long as the cluster runs.

If your data sits in another cloud or in on-premises storage, work out the bandwidth and transfer cost between there and the GPU cluster in advance. PacketStream carries data ingress and egress over its own backbone and direct ISP interconnects. For hardware in a colocation rack, Cloud Connect provides private links to public clouds and Metro Connect provides them to other data centers in the Seoul metro area.

This requirement also narrows the next choice: how to deploy.

Three ways to get GPU servers

There are three ways to get GPU servers: rent by the hour, commission a dedicated cluster, or place hardware you own in colocation. At PacketStream these are GPU Cloud, Cluster Build and Colocation.

CriterionHourly rental (GPU Cloud)Dedicated cluster build (Cluster Build)Colocation
HardwareInstances in a shared clusterA cluster dedicated to youHardware you own
Billing and contractHourly, or reserved for one month or moreBuild contract and operations contractMonthly fee by rack size and extra power
Network and topologyStandard configurationDesigned to your requirementsConfigured by you
Power and coolingThe facility as the provider prepared itMethod chosen for rack density, with the site preparedRack terms checked against the hardware before contract
LocationChoose from available regionsSpecified region or your siteChoose from data centers in Seoul
FailuresAffected node isolated, workload moved to another nodePart replacement within the operations contractYou operate the hardware
  • GPU Cloud: you pick a GPU in the Portal and create an instance. On-demand is billed by the hour from creation until deletion, carries no commitment and is offered within available capacity. A reserved instance secures capacity ahead of time for a fixed term of one month or more.
  • Cluster Build: you can hand over only the parts you need: GPU and node selection, interconnect design, storage, power and cooling, handover, and operations and monitoring. A build runs through requirements, design and quote, order and build, then handover and operations. The finished cluster is handed over either as bare metal with the OS and drivers installed, or with Kubernetes and a scheduler configured. It can be placed in a PacketStream data center, in the colocation you contract, or as an isolated network inside a region where data residency applies.
  • Colocation: you install your own servers and network equipment in a data center and run them yourself. PacketStream provides physical security, power, cooling and network connectivity, and you handle the configuration and operation of the hardware.

From the table it can look as though you simply pick one, but each option has cases it does not suit.

Where each of the three falls short

Each of the three covers a weakness of the others, so which one fails depends on the workload.

  1. Rent by the hour → Capacity may not be available on the day you need it. On PacketStream GPU Cloud too, requesting dozens of GPUs at once or a specific region can mean allocation takes a few days. Because the network is a standard configuration, a specific topology or an isolated-network requirement is hard to meet.
  2. Commission a dedicated cluster → It goes through design, ordering and site preparation, so it is not available right away. At the experiment stage, before scale and communication pattern are known, the numbers a design would rest on are missing.
  3. Place your own hardware in colocation → Procurement, fabric design, incident response and part replacement all become your operations team's work. You also check for yourself, before contract, that the rack's power and cooling match the hardware.

None of them fits every stage. So rather than picking only one, you can also plan to change options stage by stage.

  1. Experiment stageStart on demand, with no commitment.
  2. Once usage is steadyMove to reserved instances to secure capacity ahead of time.
  3. Once scale and communication pattern are clearDesign a dedicated cluster to your requirements.

There is also a middle ground between bringing your own hardware and commissioning a build. Cluster Build can be delivered into the colocation you contract, and in that case the site's power terms are checked first. What remains is working out which case your workload falls into.

Deciding in order, by workload

Narrow the place and the option in this order: data requirements, duration, scale, then operations staff. An option ruled out at one step does not need to be compared again at a later one.

  1. Data requirements: Check whether the data must stay in a given country or region. If so, keep only the configurations possible inside it.
  2. Duration: Experiments and short training runs suit hourly use with no commitment. For work that runs steadily for months, compare the total cost of reserved capacity and a dedicated build over the same period.
  3. Scale and communication pattern: Single-node jobs and inference can be fine on a standard configuration. For multi-node distributed training, put fabric topology and storage bandwidth into the requirements.
  4. Hardware and operations staff: If you already own the hardware and have people who can work on it, consider colocation. Check the hardware's power draw and cooling method against the rack terms.
  5. Data paths: Settle the circuit between where the datasets are and the GPUs, and where training storage and archive storage will sit.

GPU model and count are the first line of a quote. The quote becomes an operating plan only when the power to run the hardware, the cooling to carry the heat away, the fabric to link the nodes and the paths the data will travel are settled with it. Put where to run it and how to get it in one table, and the missing requirement shows up first.

View GPU Server

More insights

All insights

Ready to get started?

Tell us about your workload and we will help you choose the right products and plans.

  • Talk to our team

    Questions about anything you read here? Our team will help.

    Contact Sales
  • Insights

    Where PacketStream is heading, what we learn running the network, and the latest product news and notices.

    All insights

Privacy settings

Manage what you allow

Necessary cookies and similar technologies (such as local storage) are always active. You can choose the rest below and change your choice at any time from Cookie settings at the bottom of the site. See our Privacy Policy.

Necessary

Always active

Required for the site to work and stay secure. They cannot be turned off.

Examples: Theme setting, consent record, contact form security check (Cloudflare Turnstile)

Analytics and performance

Optional

Shows how visitors arrive and use our pages so we can improve the site.

Examples: Google Analytics, Microsoft Clarity

Marketing

Optional

Measures how our ads perform and helps show more relevant ads on other sites.

Examples: Google Ads