RentServersBy the RentServers team at Wells Software Consulting LLC · Updated
H100 specifications
| Architecture | Hopper |
|---|---|
| Memory | 80 GB HBM3 (SXM) |
| Memory bandwidth | 3.35 TB/s (SXM) |
| FP8 Tensor (dense) | ~1,979 TFLOPS |
| BF16 Tensor (dense) | ~989 TFLOPS |
| NVLink | 900 GB/s |
| Max power | Up to 700 W (SXM) |
Figures for the SXM/HGX version unless noted, from NVIDIA. Dense Tensor Core throughput; NVIDIA's headline numbers with sparsity are double.
SXM, PCIe and NVL are different products
Most rental listings are the SXM5 version in 8-GPU HGX servers, connected by NVLink and NVSwitch. That's what you want for multi-GPU training.
The PCIe version has HBM2e memory at about 2 TB/s, a 350 W power limit and no NVSwitch fabric. It's noticeably slower and should be priced lower. The H100 NVL pairs two PCIe cards with 94 GB each and is aimed at inference.
When you compare quotes, confirm the form factor first. A cheap "H100" hourly rate is often a PCIe card.
What fits on H100s
As a rough guide (weights only, before KV cache and activations):
- Inference, 70B model in FP8: about 70 GB of weights. That fills a single 80 GB card with almost no room for KV cache, so plan on 2 GPUs, or 1 H200.
- LoRA fine-tune of a 70B model in BF16: about 140 GB of frozen weights plus adapters and activations. 2–4 H100s with sharding.
- Full fine-tune or pre-training at 70B: mixed-precision Adam needs roughly 16 bytes per parameter, over 1.1 TB before activations. Plan on 16–32 H100s with FSDP or ZeRO-3, and an InfiniBand fabric between nodes.
- 7B–13B models: a single 8-GPU node is comfortable for full fine-tuning.
What drives H100 pricing
Term length matters most. The reserved median on GetDeploying's tracker sits about 10% below the on-demand median, and long reservations on large clusters go well below both.
After that: interconnect (InfiniBand clusters cost more than Ethernet-only nodes), region, and whether storage and support are bundled. Hyperscaler on-demand prices are typically the top of the market.
When to pick something else
If your model's weights plus KV cache won't fit in 80 GB, the H200 often costs less per useful token than adding more H100s. If you need FP4 or are training at frontier scale, look at the B200. For fine-tuning small models where FP8 doesn't matter, an A100 can be the better value.
H100 questions
How much does it cost to rent an H100?
On GetDeploying's tracker (October 16, 2026), the median H100 price is $3.74 per GPU-hour on demand and $3.34 reserved, with in-stock on-demand offers from $1.78. An 8-GPU node at $3.34 is about $19,500 a month.
Is an H100 enough to run a 70B model?
Not on one card at useful batch sizes. A 70B model in FP8 needs about 70 GB for weights alone, so most deployments use 2 H100s or a single H200.