Unofficial Tutorials The guides the official docs won't write
Llms

Running LLMs Locally on a Budget — Dual GPU Setup

Published 2026-07-23 ~572 words Tags: ollama, llama, local-ai, gpu

You don't need a data center to run capable language models locally. With two used RTX 3080 Ti cards (about $600-700 total on the used market), you can run 26B-70B parameter models at usable speeds. Here's how to set it up.

The Hardware Reality

Two RTX 3080 Ti cards give you 24GB total VRAM (12GB each). With NVLink (if your motherboard supports it), both GPUs act as a single pool. Without NVLink, model layers split across cards — slightly slower but still very usable.

What fits in 24GB:

| Model Size | Quantization | VRAM Needed | Performance | |---|---|---|---| | 7B | Q4_K_M | ~6GB | Blazing fast, full context | | 13B | Q4_K_M | ~10GB | Very fast | | 26B | Q4_K_M | ~18GB | Good — daily driver | | 34B | Q4_K_M | ~24GB | Tight fit, usable | | 70B | Q3_K_M | ~35GB | Won't fit — use CPU offloading or smaller quant |

Ollama Setup

Ollama handles multi-GPU splitting automatically. Install it:

curl -fsSL https://ollama.com/install.sh | sh

Then configure it to use both GPUs. Edit /etc/systemd/system/ollama.service and add:

[Service]
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_MAX_LOADED_MODELS=2"

Ollama automatically discovers all CUDA-capable GPUs. Verify with:

ollama ps

You should see both GPUs listed with their memory allocation.

Model Selection for Dual GPU

The sweet spot for dual 3080 Ti is 26B at Q4 quantization. It's large enough to be genuinely capable (on par with GPT-3.5 in many tasks), but small enough to leave room for context.

Top picks:

  • Gemma 2 27B Q4 — Best all-rounder. Fast, coherent, follows instructions well.
  • Qwen 2.5 32B Q3 — Fits at lower quantization. Excellent coding ability.
  • Mistral Small 22B Q4 — Lightweight and fast, but less capable than the 26B+ models.

Pull and run any of these:

ollama pull gemma2:27b-q4_K_M
ollama run gemma2:27b-q4_K_M

Getting the Most Out of It

Context window. With 24GB VRAM, you can run 32K context on a 26B Q4 model. But that eats ~7GB just for the KV cache. For most tasks, 8K-16K is the practical sweet spot.

Parallel requests. The OLLAMA_NUM_PARALLEL setting lets you queue multiple requests. The GPU processes them sequentially but switches nearly instantly — useful for batch processing.

Temperature sampling. With local models, you can crank creativity. I run 0.7-0.9 most of the time. If you need factual precision, drop to 0.1-0.2.

GPU Splitting in Practice

Without NVLink, model layers split evenly across cards. The first GPU handles layers 0-N, the second handles N+1 to end. During inference:

  • Forward pass: first GPU processes its layers, passes the hidden state to the second
  • Second GPU finishes, generates the token

This adds ~5-10ms latency per token compared to single-GPU inference. For interactive use, you won't notice. For batch processing, the throughput is nearly double.

When to Use CPU Offloading

If you want to run a 70B model, you can offload some layers to system RAM with Ollama:

ollama run llama3:70b-q3_K_M

Ollama auto-detects when the model exceeds VRAM and offloads. Expect about 3-5 tokens/second with 16GB of system RAM backing it — slow but functional for one-off queries.

The Bottom Line

A dual RTX 3080 Ti setup running Ollama is the best bang-for-buck in local AI. You get GPT-3.5-class performance with zero API costs, no rate limits, and complete privacy. The 26B Q4 models are the sweet spot — capable, fast, and they fit comfortably in 24GB with room for context.