I Got a 70B Model Running on 36GB RAM — What the Latest LLM Compression Actually Looks Like

A 70B model in float16 requires approximately 140GB of RAM just for the weights, before KV cache, context overhead, or anything else. Running one on a consumer workstation sounds like a joke. In 2026, it's a solved engineering problem — with specific trade-offs you need to understand before committing to the hardware and the workflow.

Here's exactly how Llama 3.3 70B runs on an RTX 4090 (24GB VRAM) plus CPU offloading, the quantization methods that make it possible, what quality you're actually giving up, and whether the economics justify building the setup.

---

The Hardware Equation

The RTX 4090 is the current ceiling for consumer GPU VRAM at 24GB. Pair it with a workstation carrying 64GB of system RAM and llama.cpp's CPU offloading, and your effective inference memory budget expands significantly.

Here's what the hardware actually costs in mid-2026:

The 36GB figure in the headline: 24GB VRAM + 12GB system RAM actively paged during inference. The remaining ~30GB of system RAM is available for the OS, applications, and KV cache.

At Claude 3.5 Sonnet API pricing ($3.00 input / $15.00 output per million tokens), a developer running 10 million output tokens per month pays $150,000 per year. The $3,500 workstation pays for itself in three months at that volume. At 5 million output tokens per month, it's a six-month payback. The economics of local inference depend entirely on your usage volume.

---

The Quantization Methods, Explained

Full-precision weights (float16 or bfloat16) for a 70B model weigh approximately 140GB. The entire field of "quantization" is about reducing that weight while preserving enough quality to be useful.

GGUF Q4_K_M — The Practical Standard

GGUF is the file format created by the llama.cpp project for efficient CPU and hybrid CPU/GPU inference. Q4_K_M refers to 4-bit quantization with a specific "K-quant" method that quantizes different weight matrices at different precision levels based on their sensitivity.

Llama 3.3 70B at Q4_K_M weighs approximately 43GB. With 24GB VRAM, you load 28–34 transformer layers onto the GPU and offload the rest to CPU. The performance:

• Tokens per second (generation): 6–9 on RTX 4090 with 24 GPU layers
• Time to load model: ~45 seconds from NVMe
• Practical use: Interactive chat, single-user API, developer tools

For reference, a 4-bit quantized 7B model on the same hardware runs at 80–120 tokens per second. The 70B is dramatically slower — this isn't a streaming chat replacement for a fast API, it's a high-quality local model for workloads where latency matters less than capability.

AWQ — Better Quality at the Same Bit Width

AWQ (Activation-aware Weight Quantization) is a research method that identifies which weights have the highest impact on output quality and quantizes them less aggressively. The result: at the same 4-bit width as GGUF Q4, AWQ models tend to show smaller quality degradation on complex reasoning tasks.

The trade-off: AWQ models don't run efficiently in llama.cpp. They're best used with the transformers library or vLLM, which adds setup complexity but makes them practical for building API servers.

AWQ 70B models also weigh 38–40GB — slightly larger than Q4_K_M, so the 24GB VRAM budget gets tighter and you'll need more CPU offloading.

GPTQ — Best for API Server Use Cases

GPTQ is the most widely supported 4-bit quantization format in the HuggingFace ecosystem. If you're building an OpenAI-compatible API server using vLLM to serve a team rather than running personal interactive inference, GPTQ is the format with the most tooling support.

Performance on the RTX 4090 with GPTQ via exllama2: 8–14 tokens per second depending on batch size, with the higher end achieved on small batches.

---

Quality Trade-Offs, Honestly Measured

The benchmark impact of Q4_K_M versus float16 is 1–3 percentage points on MMLU and HumanEval — often within noise margin. But benchmark performance and practical task performance diverge in specific areas.

In my daily use — code generation, technical writing, document summarization — Q4_K_M output was indistinguishable from API-served float16 in roughly 85% of generations. The 15% where it showed up were concentrated in complex reasoning chains with many interdependent steps.

If those task types represent a small fraction of your workload, the quality gap is acceptable. If you're primarily running the model on complex multi-step reasoning at high precision, float16 via API may be worth the cost.

---

The Practical Stack in 2026

For interactive, single-user inference:

• Ollama — wraps llama.cpp, handles model downloading and management, provides an OpenAI-compatible REST API. ollama run llama3.3:70b-q4_K_M is the full setup command after installation.
• Layer offloading is configured automatically based on available VRAM.

For building a local API server serving a small team (2–5 users):

• llama.cpp server mode — llama-server -m model.gguf -ngl 24 --host 0.0.0.0 --port 8080 starts an OpenAI-compatible server. The -ngl 24 flag specifies GPU layers.

For higher-throughput batch processing:

• vLLM with AWQ or GPTQ model — more complex setup but significantly better throughput for batch workloads. Requires CUDA 12.1+ and a compatible Python environment.

---

When This Setup Makes Sense (and When It Doesn't)

Builds to local 70B inference:

• You're processing data that can't leave your network (legal, medical, financial)
• You're running enough volume that API costs exceed $500/month
• You need guaranteed uptime without dependency on a third-party service
• You want to fine-tune on proprietary data — local weights, full control

Doesn't build to local 70B inference:

• You need throughput above 20 tokens/second sustained
• You're serving more than 3–4 concurrent users
• Your workload is bursty — local hardware idles during off-hours while API scales elastically
• You'd rather spend the hardware budget on API credits for a frontier model (GPT-5, Claude 4) with meaningfully better capability

The decision is economic and practical, not ideological. Local 70B is a genuine option in 2026. It's the right option for a specific use case profile — and the wrong one if your profile doesn't fit.