Idle GPUs vs. cloud A100s: a 2025 inference case study

The PoC reported 37% higher peak throughput than cloud A100s

---

‍

This case study was first published on June 10, 2025. It reports a proof of concept (PoC) exploring whether a distributed GPU swarm using corporate hardware could support enterprise AI inference. The results below describe that test and its cost assumptions; they are not current rental prices or a performance guarantee for other workloads.

A different kind of cloud test

In collaboration with a global bank, we explored whether standard corporate GPUs could replace cloud-hosted AI inference. The PoC compared workstation GPUs (NVIDIA RTX 4500, RTX 4090, and dual RTX 6000 Ada) with Runpod’s high-performance A100 80 GB GPU instance.

Inside the GPU swarm

The PoC explored a distributed cluster of corporate GPUs managed through a gateway. This historical design should be distinguished from today’s Policloud-backed Hivenet infrastructure. It does not establish that the current Compute service pools customers’ idle desktops or includes automatic scaling and enterprise identity integration.

PoC compute results

The reported test measured generative AI inference throughput, latency, concurrency, and energy use. The original article did not provide the model, software versions, or a full reproducible test configuration. The figures are retained as reported results from that PoC.

In this PoC, the swarm's 2x dual RTX 6000 Ada GPUs notably outperformed Runpod's 2xA100 by achieving 37% higher throughput at peak load and maintaining a consistent 16% throughput advantage under continuous workloads. While the A100 held a slight latency advantage (11% better time-to-first-token at extremely high concurrency), the GPU cluster running on Hivenet tech provided impressive performance overall. Energy usage was initially higher for the consumer GPUs. But after factoring in typical data center overheads (PUE), the power efficiency gap significantly narrowed.

Cost efficiency and savings

The historical monthly total cost of ownership (TCO) calculation assumed three-year hardware amortization, €0.18/kWh for electricity using a 2024 price assumption, associated licensing or cloud fees, and 75% GPU utilization. The table retains the original dollar-denominated totals; the report did not specify its currency-conversion basis. These are modeled costs, not current offers.

Using the reported cost per million tokens, the swarm was approximately 49% cheaper than the cloud A100 comparison and 43% cheaper than the on-premises A100 comparison. Those percentages are not reductions in the monthly TCO totals. For a separate hardware comparison, see the article on RTX 4090 versus A100 workload trade-offs; it is not evidence of current RTX 4090 rental availability.

Why every CIO should care

The PoC gives teams a reason to test whether existing hardware can serve a defined inference workload. Any budget benefit depends on utilization, throughput, electricity, licensing, and the staff time needed to operate the system. The reported savings should not be applied to another deployment without recalculating those inputs.

Running inference on owned infrastructure can give a team more control over placement and operations. It does not guarantee stable latency or regulatory compliance. Measure performance under expected demand and review data handling, access, and applicable requirements for the actual deployment.

Reusing hardware can affect an infrastructure footprint, but the PoC does not establish a universal environmental saving. Its reported consumer-GPU energy use was initially higher, and the comparison depended on data center overhead assumptions. Assess the electricity mix, utilization, hardware lifecycle, and workload before making a sustainability claim.

A distributed deployment also needs an operating plan. Test integration, monitoring, and recovery when a node becomes unavailable; do not assume that automatic reassignment or a low-complexity migration is included. These checks are separate from the throughput and cost figures reported here.

"We saw immediate relief in our GPU budget," said a senior infrastructure lead at the participating bank after the test. "The transition was smoother than expected, and the performance surprised our engineering team."

The bank’s comment above is retained from the original report. The results support further workload-specific testing, not a promise that every enterprise will save money or reduce its environmental impact.

Strategic takeaway

Use this case study as a starting point for comparing existing hardware with rented capacity. Recheck the model, configuration, utilization, and operating costs for your workload. For a current rental decision, consult the Compute GPU reference and live console: the RTX 4090 fleet has been retired, and current presets depend on location and capacity.

Choose a GPU for your workload.

Check the available GPU configuration, region, and active price in the Compute console before launching. Capacity can vary.

Open the Compute console