Build a 48GB local AI server for under $5,000

2026-08-16

The best local-AI computer under $5,000 is not a tiny “AI supercomputer.” It is not a 5090. It is two used RTX 3090s in a large, well-ventilated Linux box.

That answer is inelegant, power-hungry, and much more useful than it sounds. Two 3090s give you 48 GB of GDDR6X. That is enough to keep a good 4-bit quantization of a dense 70B-class model on GPUs instead of spilling layers into system RAM. For interactive inference, crossing that fit/no-fit boundary matters more than buying one newer, faster 24 or 32 GB card.

The finished machine is not better at everything. A DGX Spark has 128 GB of unified memory, draws far less power, and arrives assembled. A 128 GB unified-memory mini PC can run models this build cannot fit at all. But for single-user generation from dense models that fit inside 48 GB, the dual-3090 machine buys substantially more memory bandwidth and the mature CUDA software path. That is the workload this build is designed to beat.

The build

Treat these as spending limits, not a frozen shopping cart. Used GPU prices move daily, and motherboard layouts matter more than logos.

Part	What to buy	Budget ceiling
GPUs	2× used, air-cooled RTX 3090 24 GB	$2,500
CPU	A modern 12-core desktop CPU compatible with the chosen board	$400
Motherboard	Two physical x16 slots that operate at PCIe 4.0 x8/x8	$500
Memory	128 GB RAM	$350
Power supply	Reputable 1600 W unit with enough native PCIe power leads	$450
Storage	2 TB NVMe SSD	$180
Case or frame	Full tower or open frame with room between the GPUs	$300
Cooling and incidentals	CPU cooler, high-static-pressure fans, supports, cables	$250
Maximum		$4,930

That is deliberately conservative. On August 16, 2026, our US eBay check found intact used air-cooled 3090s around $1,190–$1,250. We also found the listings you should reject: a card with a broken fan blade, one explicitly described as thermal-throttling, and several converted to water blocks. A cheap card that needs a cooler, loop, or repair is not a beginner bargain.

Do not buy every part in the table independently and assume it fits. Pick the two exact GPU models first, record their length and slot width, and then choose the board and case around them.

Why two old cards beat one new card here

Model weights have to fit somewhere. A nominal 70-billion-parameter model at 4 bits starts around 35 GB before quantization metadata, runtime buffers, and the KV cache. It does not fit in a 24 GB 4090. It does not fit in a 32 GB 5090. It does fit across two 24 GB 3090s when context and cache settings are sane.

Once a model spills into ordinary RAM, the GPU waits on a much slower memory path. The newer card still wins on models that fit entirely inside it; this is not an argument that a 3090 is secretly faster than a 5090. It is an argument that 48 GB on GPUs beats 32 GB on a GPU plus an emergency system-RAM offload.

NVIDIA specifies 24 GB on each 3090. The cards do not magically become one 48 GB device, and NVLink is not required to make the capacity useful. llama.cpp can split whole layers and their KV cache between CUDA devices. With a layer split, only activations cross between cards; the model weights stay put.

This distinction came up repeatedly in r/LocalLLaMA build threads. Owners reported 70B quantized models running completely across dual 3090s, while builders warned that the physical slot layout, power supply, and cooling, not the CPU, were the parts most likely to ruin the build. One early dual-3090 report measured roughly 8–10 generated tokens per second on its particular Llama 2 70B/GPTQ stack. Treat that as a community measurement, not a promise: model, quantization, context, runtime, and sampling settings all change the number.

The more useful comparison is architectural. DGX Spark has 128 GB of unified memory at 273 GB/s. A 3090 is specified at 936 GB/s, and a layer-split model reads each half of its weights from one card’s local memory. The dual-3090 build gives up the Spark’s huge capacity and compact 240 W envelope in return for a much faster local memory path on models that fit.

The three checks to make before buying anything

1. Verify that the board really runs x8/x8

Two long slots do not imply two useful GPU links. Motherboard product pages often hide the second slot behind the chipset at x4, or share its lanes with an M.2 socket. Read the lane table in the manual for the exact CPU you plan to install. You want the two GPU slots to negotiate PCIe 4.0 x8/x8 from the CPU.

Inference can work at x4 because weights stay resident after loading, but x8/x8 is the beginner-safe target: faster model loading, fewer surprises during prompt processing, and room to experiment with parallel split modes later. Do not spend workstation-platform money chasing x16/x16 for a layer-split chat server.

After assembly, verify instead of assuming:

nvidia-smi --query-gpu=index,name,memory.total,power.limit --format=csv
sudo lspci -vv | grep -A 20 -E "VGA|3D controller" | grep -E "LnkCap|LnkSta"

Both cards should appear with roughly 24 GB each. Under load, LnkSta should show the expected width and speed.

2. Design power delivery, do not improvise it

NVIDIA rates the RTX 3090 at 350 W. Two cards alone can therefore ask for 700 W before the CPU, drives, fans, and transient spikes enter the picture. That is why the list specifies 1600 W rather than “whatever passes an online calculator.”

Use native PSU leads. Do not feed multiple GPU sockets from one daisy-chained cable, do not use SATA-to-PCIe adapters, and do not put the machine on a cheap power strip. A standard US 120 V, 15 A circuit has a 1,800 W theoretical ceiling; continuous loads should stay well below that, and the server should not share the circuit with a heater or portable air conditioner.

After the system is stable at stock settings, cap both cards to 300 W and measure your own workload:

sudo nvidia-smi -pm 1
sudo nvidia-smi -pl 300 -i 0
sudo nvidia-smi -pl 300 -i 1
watch -n 1 nvidia-smi

This is a starting point, not a universal optimum. Run the same prompt and measure tokens per second before and after. Keep the cap only if the speed loss is acceptable and temperatures improve.

3. Make both cards breathe

Three-slot cards pressed face-to-back recycle hot exhaust. If the motherboard does not leave a slot of air between them, use a full tower with a vertical mount that preserves lane width, or an open frame with a quality PCIe 4.0 riser. Support the cards mechanically; a 3090 hanging from a riser is heavy enough to damage itself or the slot.

Watch GPU temperature, fan speed, power, and memory temperature during a long prompt plus generation run. Reject a used card that cannot hold clocks without thermal throttling. Do not make “replace the thermal pads” step one of a beginner build. Buy a healthy card and preserve your return window.

Install the boring software first

Use Ubuntu 24.04 LTS. Install the NVIDIA driver from Ubuntu’s supported path, reboot, and make sure nvidia-smi sees both cards before installing an inference runtime. If it sees only one, stop. Software flags cannot repair a bad riser, an unseated card, disabled lane bifurcation, or missing power.

For a first boot, use llama.cpp. It has the fewest moving parts and its server exposes an OpenAI-compatible API.

sudo apt update
sudo apt install -y build-essential cmake git curl
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j "$(nproc)"
./build/bin/llama-server --list-devices

Download a reputable GGUF quantization of the model you want to run. Start with Q4_K_M, an 8,192-token context, quantized KV cache, every layer on the GPUs, and the default layer split:

./build/bin/llama-server \
  --model /models/your-70b-instruct-Q4_K_M.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size 8192 \
  --n-gpu-layers all \
  --split-mode layer \
  --tensor-split 1,1 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --flash-attn on

Flags change as llama.cpp evolves, so check llama-server --help on the binary you built. The success condition is not merely that the server starts. Read its allocation log and confirm the model and cache landed on CUDA devices rather than silently spilling into host RAM.

Test the OpenAI-compatible endpoint directly:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "local-70b",
    "messages": [{"role": "user", "content": "Explain NUMA in one paragraph."}],
    "temperature": 0.2
  }'

Only after this works should you add a web UI, containers, remote access, or vLLM. Every extra layer makes a hardware problem harder to identify.

Benchmark it like you mean it

A screenshot of one fast response proves nothing. Record all of these:

exact model file and quantization;
prompt length and generated-token count;
context size and KV-cache types;
prompt-processing tokens per second;
generation tokens per second;
peak VRAM on each card;
GPU power and temperature;
runtime commit or version.

Warm the model once, then run the same prompt at least five times. Report the median. Test a short prompt and a long prompt separately because prompt processing and token generation stress different parts of the system.

This is also why we will not print a fake “2.7× faster than DGX Spark” badge. A recent single-Spark benchmark demonstrated how violently software choices move results: on the same hardware and harness, changing the serving stack and kernel path moved single-stream and concurrent throughput by large margins. The honest claim is narrower and more useful: for dense quantized models that fit in 48 GB, this build gives you a much wider GPU-memory path than compact unified-memory appliances; for models above 48 GB, those appliances win by fitting the model at all.

What you give up

This machine is large. At full stock GPU power it turns most of a kilowatt into heat. It can be loud. It asks you to inspect PCIe topology and used hardware instead of opening one box. It has 48 GB, not the 128 GB found in a DGX Spark or high-memory mini PC.

In exchange, you get a repairable CUDA workstation that runs serious 70B-class quantized models entirely on GPUs, exposes a local API to every machine in your house once you choose to network it, and can also handle image generation, embeddings, reranking, transcription, and CUDA experiments. You can sell one card, replace one card, add storage, change runtimes, and see exactly where every byte went.

That is the part the appliance cannot beat. You did not just buy local AI. You built the machine, measured it, and made it yours.