Workstation specifications
- OS: Ubuntu 22.04 LTS
- CPU: AMD Ryzen Threadripper 3970X
- Motherboard: GIGABYTE TRX40 AORUS XTREME
- GPUs: 4 × NVIDIA GeForce RTX 3080 Ti
- RAM: 256 GB
- Storage:
- 1 × 4 TB NVMe SSD — OS and software
- 1 × 2 TB NVMe SSD — scratch storage
- 120 TB HDD storage — cold storage
- PSU: Sparkle Power FSP2000-52AGPBI, 2000 W
Background
This workstation was purchased by our laboratory to preprocess data before it is transferred to our HPC cluster. Historically, it has mainly been used for CPU-intensive workloads with relatively light GPU usage.
However, our processing pipeline has changed over time, and several of the applications we use have become increasingly GPU-intensive. I noticed that the workstation would occasionally shut down abruptly under heavy GPU load.
When these shutdowns occur:
- The machine loses power abruptly with no errors in the logs preceding the shutdown.
- After a shutdown, I have to wait ~20 minutes before the machine can be powered on again.
I stress-tested the CPU, RAM and storage individually and there were no problems. However, GPU stress testing always causes a shutdown, this led me to suspect either the GPU power delivery or the PSU itself.
Stress-test behaviour
Single GPU under sustained load:
The workstation may eventually shut down after a period of sustained GPU power draw.
All four GPUs under load:
The workstation shuts down after approximately 5 minutes.
I initially suspected that the 2000 W PSU might be faulty or unable to cope with the GPU load, so we purchased a new ASUS 2200 W PSU.
While disconnecting the original PSU, however, I noticed another potential problem.
Issue A: Existing GPU power cabling
Each RTX 3080 Ti was being powered using a single PCIe 8-pin cable from the PSU that splits into two 6+2-pin PCIe connectors at the GPU end. In other words, both 8-pin power sockets on each GPU were being supplied by a single daisy-chained/pigtail cable.
From what I have read, an RTX 3080 Ti should preferably be powered using two separate PCIe 8-pin cables from the PSU, rather than both connectors being supplied from the same cable.
I am therefore wondering whether the existing cabling could explain the shutdowns under sustained GPU load, rather than the PSU itself necessarily being faulty.
Issue B: Connecting four RTX 3080 Ti cards to the new PSU
The new 2200 W PSU presents another problem. It does not appear to have enough conventional PCIe 8-pin PSU outputs to provide two independent 8-pin cables to each of the four GPUs.
The newer PSU instead provides several 16-pin GPU power connectors, which I understand are 12VHPWR/12V-2x6 style connections.
I am not particularly experienced with high-power workstation hardware, so I can currently see several possible approaches, but I am unsure which is safest:
- Continue using the original 2000 W PSU, assuming it is still functional, and obtain the correct additional PCIe power cables so that each RTX 3080 Ti receives two separate PCIe 8-pin cables.
- Replace the original PSU with the new 2200 W PSU, and either:
- A. Power all four GPUs using the available PCIe 8-pin outputs in a similar configuration to the existing system; or
- B. Power two GPUs using four conventional PCIe 8-pin PSU outputs, and power the remaining two GPUs from the PSU’s 16-pin GPU outputs using appropriate 16-pin-to-dual-8-pin cables/adapters.
My main concern is making the safest configuration before powering the workstation again. Because this is laboratory equipment and replacing the workstation or four GPUs would be impossible given the state of hardware prices.
I would appreciate advice on:
- Is powering each 3080 Ti from a single daisy-chained PCIe cable could plausibly explain the shutdown behaviour.
- Whether the ~20 minute period before the workstation can be restarted is consistent with PSU over-current, over-power or thermal protection being triggered.
- Is it preferable to keep the existing 2000 W PSU and correct the cabling, or replace it with the new 2200 W PSU.
- Whether using the new PSU’s 16-pin GPU outputs with manufacturer-approved 16-pin-to-dual-8-pin cables is appropriate for RTX 3080 Ti cards.
- Any tests I should perform before reconnecting all four GPUs.
Thank you for any help that you can provide. For the sake of completeness I also posted this question on the Level1tech forums but I haven't really gotten an answer yet.