Distributed Inference for Local AI: One Model Across Multiple Machines (2026)

Published on February 26, 2026 • 24 min read

Quick Start: Run Llama 3.1 70B Across Two RTX 3090s on Two Boxes

If you have two machines, each with a 24GB GPU, you can serve a 70B model today. Three commands:

1. On the worker box: ./llama-rpc-server --host 0.0.0.0 --port 50052
2. On the head box: ./llama-cli -m llama-70b.Q4_K_M.gguf --rpc 192.168.1.42:50052 -ngl 99
3. Watch your prompt actually answer instead of OOM-ing.

That is the cheapest path to running a 70B-class model at home. The rest of this guide is the engineering behind that one-liner — the network topology, the parallelism strategies, when to use vLLM instead of llama.cpp, and the arithmetic that tells you whether going distributed is worth the complexity.

---

You will leave this guide knowing:

• The difference between tensor, pipeline, and data parallelism (and which one your toolchain actually does)
• Whether 1 Gbps Ethernet is enough (spoiler: for some workloads yes, for others absolutely not)
• How to set up llama.cpp RPC, vLLM tensor-parallel, exo, and Petals from scratch
• The throughput tax for going distributed vs single-machine
• Pitfalls that turn a clever cluster into a box that runs slower than one machine

This is one of the most actively developed corners of local AI in 2026. We will note where each tool's interfaces are still moving. If you have not benchmarked anything yet, start with our benchmark methodology so you can measure your own numbers against the ceilings in this article.

Table of Contents

1. When Distributed Inference Makes Sense
2. Parallelism Strategies, Demystified
3. Network Topology That Actually Works
4. Toolchain 1: llama.cpp RPC
5. Toolchain 2: vLLM Tensor Parallelism
6. Toolchain 3: exo for Heterogeneous Hardware
7. Toolchain 4: Petals for Public Networks
8. Expected Throughput Across Toolchains
9. Common Pitfalls
10. Frequently Asked Questions

---

When Distributed Inference Makes Sense

Distributed inference is not faster than fitting the model on one box. It is the option you reach for when you cannot fit the model on one box, or when you have already saturated one box and need to scale concurrency.

Good reasons to go distributed:

1. You own two 24GB GPUs (e.g., two used RTX 3090s) and want to run Llama 3.1 70B.
2. You have a Mac Studio with 192GB unified memory and a PC with a 4090, and you want both to contribute.
3. You are serving a team and a single GPU is throughput-bound at peak.
4. You want geographic redundancy — head node in your office, worker on a homelab box.

Bad reasons:

1. "It seems cool." Distributed adds 50-200ms of TTFT minimum and is operationally fragile.
2. You have one big box and one tiny box. The slow one will pin overall throughput.
3. You are running 13B or smaller. Just upgrade to a single GPU with enough VRAM.

If you have not yet picked your hardware, our used GPU buying guide covers the math on two 3090s vs one 4090.

---

Parallelism Strategies, Demystified

Every distributed inference toolchain implements one or more of three parallelism strategies. Picking the right one is half the battle.

Tensor Parallelism (TP)

The model's weight matrices are sliced horizontally. Each GPU holds a fraction (1/N) of every layer and contributes to every forward pass. After every matmul, partial results are summed via an all-reduce.

• Strength: Lowest latency for single-stream generation.
• Weakness: Network is on the critical path of every forward pass. Demands fast interconnect (NVLink ideal, 25 Gbps+ Ethernet acceptable).
• Used by: vLLM, TensorRT-LLM, DeepSpeed-Inference.

Pipeline Parallelism (PP)

The model's layers are sliced vertically. GPU 0 holds layers 1-40, GPU 1 holds layers 41-80. Tokens flow through the pipeline.

• Strength: Network only carries activations between layer boundaries — much lighter than TP. Works fine over 1 Gbps Ethernet.
• Weakness: Single-stream throughput is bottlenecked by the slowest GPU; pipeline bubble hurts unless you batch.
• Used by: llama.cpp RPC, exo (in part), Petals.

Data Parallelism (DP)

Each GPU holds a full copy of the model and serves a different batch of requests independently.

• Strength: Trivially parallel; throughput scales near-linearly with GPU count.
• Weakness: Does not help with models that do not fit on one GPU. Pure DP is a reverse proxy in disguise.
• Used by: Ollama / vLLM behind a load balancer.

Quick decision tree

If your problem is concurrency rather than capacity, our LiteLLM gateway guide walks through pure data parallelism instead.

---

Network Topology That Actually Works

The single biggest mistake in homelab distributed inference is underestimating network requirements.

Pipeline parallelism (llama.cpp RPC, exo, Petals)

Pipeline parallelism only sends activations across the layer boundary: one hidden-state vector per token, 8,192 values for a 70B Llama, or 16-32 KB at FP16/FP32. Even on 1 GbE that transfer is a fraction of a millisecond, against roughly 45 ms per token at the ~22 tok/s bandwidth ceiling of two RTX 3090s (936 GB/s / 42.5GB). What actually costs time is per-message latency and the RPC framework's own round trips, so a faster, lower-latency link helps less than you might expect. Measure on your own network before buying a 10 GbE switch.

Tensor parallelism (vLLM)

Tensor parallelism synchronises the GPUs with an all-reduce twice per transformer layer, so a 70B Llama (80 layers) needs about 160 synchronisations for every generated token. Each message is small, but each costs at least one network round trip; at a few hundred microseconds per round trip on commodity Ethernet, the waiting alone adds tens of milliseconds per token. If you are not running 10 GbE or better, ideally with RDMA, do not use vLLM TP across boxes; use pipeline instead.

Practical home setups that work

• Two desktops, one switch: an inexpensive 4-port 2.5 GbE switch (Mokerlink, TP-Link) is the cost-effective sweet spot.
• Mac Studio + PC: Direct Thunderbolt 4 cable, configure IP-over-Thunderbolt. Free 40 Gbps.
• Three+ nodes: a small 10 GbE SFP+ switch such as the MikroTik CRS305 is the usual budget path.

The Vulkan / NVIDIA team's NCCL documentation goes deep on tensor-parallel collective operations if you want the underlying theory.

---

Toolchain 1: llama.cpp RPC

llama.cpp ships a built-in RPC backend that pipeline-parallelizes any GGUF model across N machines. It is the easiest path from "one box" to "two boxes."

Setup

On every worker:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DGGML_RPC=ON
cmake --build build --config Release -j

# Start the RPC server
./build/bin/rpc-server --host 0.0.0.0 --port 50052 --threads 8

On the head node:

# Same build, with RPC enabled
./build/bin/llama-cli \
  -m /models/Meta-Llama-3.1-70B-Instruct.Q4_K_M.gguf \
  --rpc 192.168.1.42:50052,192.168.1.43:50052 \
  -ngl 99 -c 4096 -t 8

-ngl 99 means "offload all layers"; llama.cpp will distribute them across the local GPU + remote RPC servers based on available VRAM.

What works

• Drop-in for any GGUF model
• Mixed-quant support (Q4_K_M on one node, Q5_K_M on another for testing)
• Layer-aware split — handles GPUs with different VRAM gracefully

What hurts

• TTFT is high: prefill on a 4K-token prompt across 2 nodes takes 4-6 seconds vs 1.5s on a single 4090.
• No continuous batching — concurrency is single-stream + queue.
• RPC is unauthenticated by default. Run it inside a private network or behind WireGuard.

Production tip: run RPC under systemd

# /etc/systemd/system/llama-rpc.service
[Unit]
Description=llama.cpp RPC worker
After=network.target

[Service]
ExecStart=/opt/llama.cpp/build/bin/rpc-server --host 0.0.0.0 --port 50052 --threads 8
Restart=always
User=ai
Group=ai

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now llama-rpc.service

---

Toolchain 2: vLLM Tensor Parallelism

vLLM is the production answer when you need concurrent users on a model that does not fit on one GPU.

Single-machine, multi-GPU (best case)

pip install "vllm>=0.6.4"

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --quantization awq \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.92

If both GPUs are in the same box with NVLink or PCIe 4.0 x16, this is the fastest local 70B serving setup in this guide: tensor parallelism lets both cards read their half of the weights at once (a ~44 tok/s single-stream ceiling for two RTX 3090s on a 42.5GB file), and continuous batching raises aggregate throughput as more users connect.

Multi-machine (requires fast interconnect)

vLLM uses Ray for multi-node:

# Head node
ray start --head --port=6379

# Worker node
ray start --address='192.168.1.10:6379'

# Launch (from head)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 2 \
  --quantization awq

This requires 25 GbE+ between nodes to be useful. On 1 GbE, vLLM cross-machine TP is slower than llama.cpp RPC pipeline by 5-10x.

What works

• Continuous batching — best throughput for multi-user workloads
• Mature OpenAI-compatible API
• Excellent observability (Prometheus metrics out of the box)

What hurts

• High network demand for cross-machine TP
• Configuration is unforgiving — wrong tensor/pipeline ratio will crash on startup
• Requires homogeneous GPUs (all 24GB or all 48GB; mixing is unsupported)

---

Toolchain 3: exo for Heterogeneous Hardware

exo is the only mainstream tool that handles heterogeneous fleets gracefully. Mac Studio + iPhone + Linux PC, all participating in one inference.

Setup

git clone https://github.com/exo-explore/exo
cd exo
pip install -e .

Run on every node — exo auto-discovers peers via mDNS:

exo

Open the dashboard at http://localhost:8000 and you should see the cluster forming. Run a model:

curl http://localhost:8000/v1/chat/completions \
  -d '{
    "model": "llama-3.1-70b",
    "messages": [{"role":"user","content":"hi"}]
  }'

What works

• Genuinely heterogeneous: Apple Silicon Macs and CUDA boxes share a model
• Layer assignment respects each node's available memory, so a 64GB Mac takes a larger share of the layers than a 24GB 3090.
• Zero-config peer discovery is delightful when it works

What hurts

• More framework overhead than llama.cpp RPC; you pay for the convenience.
• Network errors cascade into full restarts.
• Newer project; expect interface churn.

Use exo when your cluster is mixed and you want it running tonight. For repeatable production, prefer llama.cpp RPC or vLLM.

---

Toolchain 4: Petals for Public Networks

Petals is BitTorrent for LLM inference — a public swarm where each volunteer hosts a slice of the model.

from petals import AutoDistributedModelForCausalLM
from transformers import AutoTokenizer

model_name = "petals-team/StableBeluga2"
tok = AutoTokenizer.from_pretrained(model_name)
model = AutoDistributedModelForCausalLM.from_pretrained(model_name)

inputs = tok("Privacy-preserving inference is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64)
print(tok.decode(out[0]))

What works

• Lets a laptop run 70B+ models you would never fit locally
• Privacy-preserving routing — peers cannot see prompts in plaintext
• Genuinely educational for understanding distributed LLM architecture

What hurts

• Latency is at the mercy of public peers, so speed varies from run to run
• Model selection is limited to what the swarm hosts
• Not appropriate for sensitive workloads despite the privacy claims — read the threat model carefully

Petals is a research toolkit. Treat it as one.

---

Expected Throughput Across Toolchains

We have not benchmarked these toolchains on our own cluster, so this section gives the arithmetic that bounds them and the method to measure your own. Reference setup: Llama 3.1 70B at 4-bit (Ollama's llama3.1:70b Q4_K_M file is 42.5GB) split across two RTX 3090s (936 GB/s each, NVIDIA spec).

Hardware

• Node A: RTX 3090 24GB in any modern desktop with 64GB RAM
• Node B: RTX 3090 24GB, same
• Network: 2.5 GbE switch and Cat6 cable, or a direct Thunderbolt link between Macs

Single-stream tokens/sec

8 concurrent streams

To measure your own: run the same 512-token prompt and 256-token output five times after a warm-up, record tokens per second and time to first token, then repeat with 8 concurrent requests. llama.cpp's llama-bench and vLLM's benchmark_serving.py both report these.

Take-aways

• For one user, two GPUs in different machines, Llama 70B: llama.cpp RPC over 2.5 GbE is the simplest path, and its ceiling is the same as two GPUs in one box with a layer split.
• For many users, two GPUs in the same machine: vLLM tensor parallelism with continuous batching is the strongest option.
• Cross-machine vLLM is for fast networking only. Do not build a homelab around it on commodity Ethernet.

If you want to compare these against a single-GPU rig, the single-machine benchmark guide uses the same prompts and method.

---

Common Pitfalls

1. Mixing GPU classes

Pipeline parallelism is bottlenecked by the slowest stage. Mixing an RTX 3090 with a GTX 1080 Ti drops your overall throughput to roughly 1080 Ti levels. Match your hardware or accept the loss.

2. Forgetting MTU and jumbo frames

On 10 GbE, the default MTU of 1500 means more packets and more CPU per byte; jumbo frames often help. Set MTU 9000 across the entire path (every NIC and the switch):

sudo ip link set eth0 mtu 9000

3. Running RPC over WAN unencrypted

llama.cpp's RPC protocol has no auth and no encryption. Always tunnel through WireGuard or Tailscale if any hop is untrusted.

4. Underestimating CPU cost

Pipeline parallelism uses CPU heavily for activation copies. An RTX 3090 paired with a 4-core CPU will be CPU-bound. Use 8C/16T or better on every node.

5. Forgetting to pin the model file across nodes

vLLM expects the same SHA256 of weights on every node. If one node has Llama 3.1 70B and another has Llama 3.1 70B-Instruct, vLLM will start, run garbage, and exit.

6. NUMA mishaps on dual-socket systems

Pin the inference process to the NUMA node where the GPU lives. numactl --cpunodebind=0 --membind=0 avoids cross-socket memory traffic on Epyc / Xeon dual-socket builds; measure before and after.

7. Trusting Wi-Fi

Wi-Fi adds milliseconds of latency and jitter to every round trip, and distributed inference makes many round trips per token. Always use Ethernet between nodes.

---

Putting It Together: A Recommended Homelab Stack

For a hobbyist with 2-4 boxes:

1. Two used RTX 3090s (resaleprices.com put the fair used range at $1,499-1,606 each on November 1, 2026)
2. Two mid-range PCs with 64GB RAM each (price RAM on the day; it rose sharply in 2026)
3. One 2.5 GbE switch
4. llama.cpp RPC for 70B / 8x22B models
5. One vLLM instance per node for high-concurrency 13B/8B serving
6. LiteLLM in front to route requests by model

That is a homelab that runs Llama 3.1 70B for one user and Mixtral 8x7B for ten concurrent users. For the gateway layer, our LiteLLM gateway guide covers the routing config.

---

Frequently Asked Questions

Q: Can I run Llama 3.1 70B on two used RTX 3090s in different boxes?

A: Yes. llama.cpp RPC splits the 42.5GB Q4_K_M file across both cards; the GPUs take turns, so the single-stream ceiling is ~22 tok/s (936 GB/s / 42.5GB), with network round trips and RPC overhead taking real speed below that. Two cards in one box under vLLM tensor parallelism have a ~44 tok/s ceiling because both read at once.

Q: Is 1 Gbps Ethernet enough for distributed inference?

A: For pipeline parallelism (llama.cpp RPC, exo), it works: one token's activations are only 16-32 KB. For tensor parallelism (vLLM), no: ~160 round trips per token make it unusable. 2.5 GbE is a cheap upgrade that lowers latency.

Q: What is the difference between tensor parallelism and pipeline parallelism?

A: Tensor parallelism slices each weight matrix horizontally across GPUs and all-reduces after every matmul (network-heavy, low latency). Pipeline parallelism slices the model vertically by layer and only passes activations at boundaries (network-light, higher latency).

Q: Do I need NVLink to do distributed inference?

A: Not for cross-machine setups (NVLink is GPU-to-GPU). For same-box multi-GPU, NVLink helps tensor parallelism a lot. For pipeline-parallel cross-machine work, you only need fast Ethernet.

Q: Can I mix Apple Silicon and NVIDIA in one cluster?

A: Yes, with exo. It is the only mainstream tool that handles heterogeneous hardware. Expect throughput somewhere between the two types, biased toward the slower node.

Q: How does Petals compare to a private cluster?

A: Petals lets a laptop run models that would otherwise be impossible, but throughput is at the mercy of public peers and privacy guarantees are weaker than they sound. Use it for experimentation, not production.

Q: What about CPU-only distributed inference?

A: llama.cpp RPC supports CPU nodes. On a 70B model it is slow (dual-channel DDR5 at ~90 GB/s gives a ~2 tok/s ceiling on the 42.5GB file) but works. Better used to add fallback capacity than to be the primary path.

Q: Is multi-machine vLLM worth setting up at home?

A: Only if you have 10 GbE or better between nodes. On 1-2.5 GbE, llama.cpp RPC will outperform multi-machine vLLM despite vLLM being the more advanced tool, simply because the network bottleneck dominates.

---

Conclusion

Distributed local inference is no longer exotic — it is a Saturday afternoon project if you have two GPUs and a switch. The trick is matching the parallelism strategy to your interconnect: pipeline for slow networks, tensor for fast ones, data parallel for pure concurrency scaling.

For the homelab reader, llama.cpp RPC on an inexpensive 2.5 GbE switch unlocks 70B-class models you could not otherwise run. For a small team, a single box with two GPUs running vLLM TP is dramatically simpler than two boxes and almost always faster.

Pick the smallest cluster that solves your problem. Do not build a four-node cluster because it is fun if a single beefier machine would do — operational cost is real, and the second box that breaks at 2 AM is the one you regret.

Pair this guide with our Ollama rate limiting article for the multi-user side and the hardware requirements overview before you start sourcing parts.

---

Want new distributed-inference guides as they land? Subscribe to the LocalAimaster newsletter.