How much intelligence can you run locally?

The AA Intelligence Index is the most useful single scale for this comparison, as long as all scores come from the same version. It moved from v4.1 in June to v4.3.2 in October, and scores from different versions are not comparable.

The open models that come closest to the frontier are too large for a desk. Kimi K3 has 2.8 trillion parameters with 104 billion active, and DeepSeek V4.1 Flash has 552 billion, which fits a 512 GB Mac Studio at best. Among models that do fit, newer beats bigger: Qwen3.8-27B outscores every 120B-class model that fits on a 128 GB machine by more than 20 points, except Qwen3.8-Flash-Next.1

Epoch AI estimates that the best open-weight models trail the best closed models by about 4 months (Edwards & Emberson, May 29, 2026). An earlier Epoch analysis put the lag for models that fit one consumer GPU at 6–12 months (Somala & Emberson, Aug 2025), and notes that small open models are more likely to be tuned for specific benchmarks, so their real-world lag may be longer.

Open models also cost much less through a hosted API. At the v4.1 launch in June, AA found DeepSeek V4 Pro cost $0.04 per Index task, against $1.78 for Claude Opus 4.8.24

Generating a token means reading every active weight from memory once. There is little arithmetic per byte, so the chip spends most of its time waiting on memory, and speed is capped at roughly memory bandwidth divided by the bytes read per token. On any model that fits in its 32 GB, an RTX 5090 at 1,792 GB/s is much faster than a 128 GB machine at 273 GB/s.

With gpt-oss-120b selected, the RTX 5090 drops out because the model needs about 66 GB. The 128 GB machines run it at about 55–88 tok/s, since each token reads only about 5 GB of the 66. On a dense 70B model, where every token reads all 40 GB, the Framework Desktop manages about 5 tok/s.

Dense models in this brief measure at about 55–70% of the bandwidth ceiling. Speculative decoding (MTP, DFlash, EAGLE and similar) can go above it: a small draft model proposes several tokens and the large model checks them in one pass. AA's configurations ran Qwen3.8-27B 30–120% above the ceiling, and a pinned llama.cpp build took an RTX 3090 from 37.4 to 60.3 tok/s with MTP.530

Reading the prompt (prefill) is limited by compute instead, because its tokens are processed in parallel. It takes a large share of agent time: AA found prefill took 22–41% of end-to-end time on Qwen3.8-27B, even with 73–93% of prompt tokens served from cache.5 Long context also slows decode. An RTX PRO 6000 generates 210 tok/s on gpt-oss-120b at 4K context and 100 tok/s at 128K.43

Weights need roughly parameters × bits ÷ 8 bytes. A 70B model is about 140 GB at 16-bit and 40–42 GB at Q4_K_M, which averages about 4.8 bits per weight.29 A mixture-of-experts model has to store every parameter, but each token reads only the active ones.

The KV cache comes on top of the weights and grows with context. Its per-token size is 2 (K and V) × layers × KV heads × head dimension × bytes. For Qwen3.8-27B that works out to about 64 KiB per token at f16, so 128K tokens need 8 GiB. One study of quantized reasoning models found a 4-bit KV cache roughly lossless and a 3-bit cache "severe".31

Quality stays almost flat down to about 4.8 bits per weight, then falls off quickly. At a similar size, Q4_K_M also beats the older Q4_0, because K-quants use a more precise block-scaling scheme.36

Perplexity can miss real damage. "The Illusion of Equivalency" (arXiv 2607.08734) finds that Q8_0 through Q4_K keep accuracy close to the base model yet still produce "noticeable behavioral divergence", and concludes that perplexity "is not a trusted metric".39

Small models and hard tasks lose the most. A comparison of methods at IJCAI 2025 found FP8 "the most reliable method for all model sizes" and AWQ consistently ahead of GPTQ at 70B. It also found quantization "considerably reduces performance in Coding and STEM tasks" on MT-Bench even when leaderboard averages looked stable.38

Tool calling is especially sensitive. Unsloth's write-up, cited by Pinggy, warns that tool calling breaks down below UD-Q2_K_XL. On Qwen3.8-27B, 32-token agreement with the full model fell from about 25% at UD-Q2_K_XL to under 10% at UD-IQ2_S.22

Unified-memory machines give the most memory per dollar, and NVIDIA's GDDR cards give the most bandwidth per dollar. The 96 GB M5 Ultra has the best bandwidth per dollar of any big-memory machine. At late-2026 street prices the RTX 5090 is no longer cheap VRAM: TensorRigs puts the card alone at about $156 per GB, close to professional cards.72

Every speed on this page is a published measurement with its source named. Sites that estimate speeds can be far off: LLMCheck lists 7 tok/s for gpt-oss-120b on the M5 Max, where MLX measured 88.4973

Two newer options sit near the DGX Spark. NVIDIA's RTX Spark N1X puts a GB10-class chip with 300 GB/s of bandwidth in a Windows laptop; a 128 GB prototype Surface Laptop Ultra ($5,900) finished AA's agent workload in 18.7 minutes.46 AMD's own 128 GB Ryzen AI Halo ($4,000) ran 1.4–1.7× slower than the DGX Spark on three of four models. That is far more than their roughly 7% bandwidth difference, and AA attributes it to the Spark's extra low-precision compute for prefill and its more mature CUDA stack.5 No verified current price was found for DGX Station.

llama.cpp, with Ollama and LM Studio on top of it, is built for one user on whatever hardware is at hand. It splits layers between GPU and CPU so oversized models still load, and it supports the widest range of quantization types.75 vLLM manages the KV cache in pages (PagedAttention) and admits new requests at every step, which is what serving many users at once needs.76

The gap widens with more users. Red Hat measured vLLM producing about 44× more tokens per second than llama.cpp at 64 users on one H200, with llama.cpp's P99 time to first token past 3 minutes.76 With little spare memory the gap closes: on a 24 GB RTX 4090 the two tied at 32 requests (306 vs. 300 tok/s) because vLLM had no VRAM left for batching.79 Two RTX PRO 6000s on vLLM served gpt-oss-120b at 230.5 tok/s to one user and peaked at 664 tok/s across 6 requests.80

Frontier models run on racks of accelerators with high-bandwidth memory. A GB200 or GB300 NVL72 rack links 72 GPUs with 1.8 TB/s of NVLink each (130 TB/s in total) and up to 576 TB/s of aggregate memory bandwidth.8990 Tensor, pipeline and expert parallelism spread one model across many chips.

Batching is the main reason datacenter serving costs less per token. A home machine spends its whole bandwidth budget on one user. A datacenter GPU reads the weights once per step and produces a token for every request in the batch. DeepSeek's published V3/R1 serving stats show each 8×H800 node averaging about 14.8k output tok/s during decoding while each user saw 20–22 tok/s.91

Several other techniques add to this. Running prefill and decode on separate GPU pools gave about 1.5× higher throughput for Llama 3.1 405B in NVIDIA's MLPerf results.92 Prefix caching and KV offload keep repeated context cheap (about 96% of tokens were cached input in AA's Vera Rubin test), and session-aware routing such as NVIDIA Dynamo sends returning users back to their cache. AA measured Vera Rubin NVL72 at 272 million tokens per second per megawatt on DeepSeek V4 Pro, against 239 million for GB300 NVL72, with the lead growing to about 5.7× at around 190 tok/s per user. NVIDIA submitted those results, and the MW figure counts GPU and HBM power only.86

• Anthropic says it trains and runs Claude on "AWS Trainium, Google TPUs, and NVIDIA GPUs".93
• Project Rainier, about 500,000 Trainium2 chips, came online in October 2025.94
• In October 2025, Anthropic and Google announced access to up to 1 million TPUs, bringing well over a gigawatt online in 2026.9596
• On April 6, 2026, Anthropic announced a Google and Broadcom deal for "multiple gigawatts" of TPU capacity from 2027. Tom's Hardware puts it at about 3.5 GW.9397
• Claude is available on AWS Bedrock, Google Vertex AI and Microsoft Azure Foundry.93

For one person, buy enough memory for the model you want, then as much bandwidth as the budget allows. Expect roughly Claude Haiku-class results, about a year behind the frontier. The gap from desk to frontier (about 24 AA points) is twice the gap from open weights to frontier (about 11).

• Use Q4_K_M, AWQ or NVFP4 as the default quantization.
• Move up to Q5, Q6 or FP8 for coding and math when memory allows.
• Avoid anything below about 3.5 bits for agents.
• Quantize the KV cache to Q8 for long context.
• Turn on speculative decoding.
• Switch to vLLM or SGLang only once you serve more than one user.

Anthropic, Qwen and DeepSeek figures are self-reported, use different harnesses and differ from independent runs. Qwen3.8-27B's SWE-bench Pro score of 61.7 uses Qwen's refined version of the benchmark and is not comparable to Anthropic's 89.9%.25 AA Index versions change weights and benchmarks, so only compare scores within v4.3.2. The same hardware can differ 4× by engine and settings, so cross-source speed comparisons are approximate.