DeepSeek V4 Flash on 4x RTX Pro 6000: What the Benchmarks Actually Need
A look at what it takes to benchmark DeepSeek V4 Flash across four RTX Pro 6000 GPUs, and why the hardware behind the numbers matters as much as the tokens.
What does it take to benchmark an LLM across four GPUs?
Running a large language model across multiple GPUs isn’t just a software problem. Before any tokens-per-second number means anything, you need a platform that can actually feed four power-hungry accelerators enough PCIe bandwidth and CPU throughput to keep them busy. That’s the real story behind any serious multi-GPU inference benchmark: the rig underneath it. A setup built around four RTX Pro 6000 Blackwell GPUs, paired with an AMD Threadripper Pro workstation platform, is a common way builders are approaching local inference for large models like DeepSeek’s newer releases, where a single consumer GPU simply doesn’t have enough VRAM to hold the weights.
TL;DR
- Multi-GPU inference rigs for models like DeepSeek V4 Flash typically pair RTX Pro 6000 GPUs with a Threadripper Pro CPU to get enough PCIe lanes to avoid bottlenecking the cards.
- PCIe lane count matters more than raw CPU speed for this use case: a Threadripper Pro platform offers 128 PCIe 5.0 lanes, enough to give four or more GPUs full-bandwidth slots without lane splitting.
- Bigger CPU core counts don’t speed up token generation directly; they mainly help with data preprocessing, model loading, and serving multiple concurrent users at once, not single-stream inference speed.
- Power delivery is a real constraint: RTX Pro 6000 cards can draw up to 600 watts each in workstation form, which is why builders overhaul stock power supplies to 3000W-class units when running four of them.
- The Max-Q variant caps at 300 watts, trading some performance for a much easier thermal and power budget, which matters when stacking multiple cards in one case.
- VRAM capacity, not just GPU count, decides which models you can run locally, since a model the size of DeepSeek V4 Flash needs to be split or fully replicated across the available memory pool depending on the serving framework.
- Concurrency scaling (how throughput changes from 1 to 16 simultaneous requests) is the real benchmark that matters for anyone planning to serve more than one user, since single-user token speed rarely reflects real deployment conditions.
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
Why does DeepSeek V4 Flash need a 4-GPU setup in the first place?
DeepSeek’s models have grown from research curiosities to genuinely large systems that most single-card setups can’t hold in memory. A four-GPU rig using RTX Pro 6000 cards exists specifically to get enough combined VRAM and compute to run a model like DeepSeek V4 Flash locally instead of through an API. Each RTX Pro 6000 is a Blackwell-generation workstation GPU with a large VRAM pool, and running four of them together lets a serving framework shard the model’s weights and KV cache across the cards.
This is a different problem from gaming or single-model fine-tuning. Inference serving has to juggle weight loading, batching multiple requests, and managing memory across GPUs simultaneously. That’s why the platform choice (motherboard, CPU, power supply) ends up mattering nearly as much as the GPUs themselves. A consumer motherboard with limited PCIe lanes will bottleneck four full-power GPUs no matter how fast the cards are individually.
How does the underlying hardware affect tokens-per-second numbers?
Tokens-per-second figures for a model like DeepSeek V4 Flash depend on several layers stacking on top of each other:
PCIe bandwidth between GPUs. When a model is sharded across four cards, GPUs need to exchange activations and KV cache data constantly. A platform with 128 PCIe 5.0 lanes (as found on high-end Threadripper Pro workstation boards) lets every GPU run at full x16 bandwidth simultaneously, avoiding the lane-splitting that happens on consumer platforms with fewer lanes.
CPU core count. More cores don’t make a single token generate faster once the GPUs are already saturated, but they matter a lot for concurrency. A CPU with more threads can handle multiple simultaneous request streams, data preprocessing, and virtual machine overhead without becoming the bottleneck when concurrency climbs from 1 to 16 users.
Power headroom. RTX Pro 6000 workstation cards can pull up to 600 watts under sustained load, while the Max-Q edition caps at 300 watts for easier thermal management. Four full-power cards plus a high-core CPU can push total system draw well past what a standard 1000W to 1500W power supply can safely handle, which is why builders running four-card rigs often move to power supplies in the 3000W class.
Quantization format. Formats like FP4 and FP8 reduce the memory footprint and computational cost per token compared to FP16 or BF16, which is often what makes running a model of this size feasible on four cards instead of eight. Lower precision formats trade a small amount of output quality for significantly higher throughput and lower VRAM use, which is why they’re the default choice for benchmarking large modern models on workstation-class GPUs.
Is a 4x RTX Pro 6000 rig actually worth it for local inference?
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
For most individual developers, no. The GPUs alone represent a serious investment, and pairing them with a Threadripper Pro CPU, ECC memory, a high-wattage power supply, and a case built to handle the airflow and cabling for four large cards adds real cost. This is the kind of setup built by people who need dedicated capacity for running large open-weight models locally, testing multi-user concurrency, or avoiding per-token API costs at high volume.
For teams or businesses evaluating whether to self-host a model like DeepSeek V4 Flash, the calculation is different. If you’re serving many concurrent users and want to avoid ongoing API bills, or if data privacy requirements rule out cloud inference entirely, the up-front hardware cost can make sense against months or years of API spend. The concurrency behavior (how throughput holds up as simultaneous requests increase from 1 to 16) is the number that actually predicts whether the investment pays off, since a rig that looks great at a single request can fall apart once real traffic hits it.
What role does concurrency testing play in evaluating a model like this?
Single-stream tokens-per-second is the easiest number to report, but it rarely reflects how a model behaves in production. Testing throughput at concurrency levels from 1 up through 16 concurrent requests shows how well the serving stack handles batching, memory pressure, and GPU-to-GPU communication under real load. A rig can look fast with one user and then degrade sharply once several requests compete for the same KV cache memory and compute, especially if the underlying platform doesn’t have enough PCIe bandwidth or CPU threads to keep everything fed.
This is why the hardware platform choice matters as much as the GPUs. A Threadripper Pro system with full PCIe lane allocation and enough cores to manage multiple request queues is built specifically to keep the concurrency curve as flat as possible, so throughput per user doesn’t collapse as more requests arrive.
Frequently Asked Questions
What GPU is best for running DeepSeek V4 Flash locally?
The RTX Pro 6000 (Blackwell generation) is a common choice for local serving of large open-weight models because of its large VRAM pool and workstation-class reliability. Running multiple cards together, often four, is typical for models too large to fit on a single GPU.
Why use a Threadripper Pro CPU instead of a standard desktop CPU?
Threadripper Pro platforms offer significantly more PCIe lanes (up to 128 on the top-end chips), which lets every GPU in a multi-card rig run at full bandwidth. Standard desktop CPUs have far fewer lanes and would force GPUs to share bandwidth, creating bottlenecks in multi-GPU inference.
Does a bigger CPU make token generation faster?
Not directly. Once GPUs are handling the actual model computation, CPU core count mainly affects how well the system handles data preprocessing, request queuing, and multiple concurrent users, rather than the raw speed of generating tokens for a single request.
Why does power supply size matter for a 4-GPU rig?
Workstation GPUs like the RTX Pro 6000 can draw up to 600 watts each under load. Four cards plus a high-power CPU can easily exceed what standard power supplies handle safely, which is why multi-GPU builds often use power supplies rated in the 3000W range.
What’s the difference between FP4, FP8, and full precision for inference speed?
Other agents ship a demo. Remy ships an app.
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
Lower precision formats like FP4 and FP8 reduce memory footprint and computation cost per token, which increases throughput and reduces VRAM requirements at a small cost to output precision. They’re commonly used when running very large models on hardware with limited combined VRAM.