Llama-modchip displays token throughput and memory usage while running DeepSeek‑284B on a pure CPU system.
LLAMA-MODCHIP RUNS 284B DEEPSEEK ON 64GB CPU, NO GPU REQUIRED
*A GitHub project compresses a 284‑billion‑parameter model to fit 64 GB RAM, delivering real‑time inference on pure CPU. The breakthrough challenges the GPU‑centric AI hardware narrative.*
The open‑source llama‑modchip project announced version 2.4.0, claiming it can run the 284.33B DeepSeek‑V4‑Flash model on a standard 64 GB RAM workstation without any GPU. The repo lists a 56.56 GiB checkpoint that peaks at 57.61 GiB VmRSS while holding a 256K token context in RAM. Token throughput reaches 44.36 tok/s during prefill and 9.1–14.7 tok/s on pure CPU decode, rising to 11.5–14.7 tok/s with Stockfish‑PVS assistance. The same code also runs 30 B MoE models at speeds comparable to GPU‑accelerated inference, all on CPUs ranging from Zen 5 to Threadripper. The claim directly confronts the prevailing belief that multi‑hundred‑billion‑parameter models demand high‑end GPUs or massive NVMe storage.
How does llama-modchip shrink the 284B model to fit 64 GB?
Llama‑modchip applies a stack of compression techniques: Analytical SwiGLU‑H2 second‑order Hessian compression, Fractal‑64 Q2_0 quantization at 3.75 bits per weight, and a four‑level register tiling that leverages AVX‑512BW registers. The pipeline also uses whole‑model tiling and $L_2$‑quadrature MoE pruning. These steps reduce the original 76.87 GiB DeepSeek checkpoint to a 56.56 GiB disk image and a 57.61–58.9 GiB resident memory footprint at full 256K context. The model therefore fits inside a 64 GB RAM box without triggering out‑of‑memory crashes that plagued the uncompressed version on Linux.
What performance does the CPU‑only setup deliver?
At 256K context the 284B model achieves 44.36 tokens per second during prefill and between 9.1 and 14.7 tokens per second on pure CPU decode. Enabling the Stockfish‑PVS drafter raises decode to 11.5–14.7 tok/s. The 30 B MoE variant runs at 25.0–60.4 tok/s on a 16‑core Zen 5, 80.8 tok/s on an AMD Vulkan hybrid, 111.6 tok/s on a Threadripper, and 44.97 tok/s on an AWS Graviton3 ARM64. Claude Code restore cuts re‑evaluation latency to 13.1 ms on DeepSeek‑284B and 18.9 ms on Nemotron‑30B, eliminating a 45‑second bottleneck.
Why does llama-modchip require a custom binary?
The project stores models in a custom GGML_TYPE_Q2_0 tensor format (type ID 40) that uses block‑64 BF16‑scale gray‑coded tensors. Unpatched llama.cpp, Ollama, or LM Studio cannot read this format and will error on unknown tensor type 40. Users must run the provided ./modchip binary or the bundled pre‑compiled llama‑server‑modchip‑linux‑x86_64 executable, both of which include the required deepseek4 and Stockfish‑PVS kernels.
What does this mean for the future of AI hardware?
If a 284 billion‑parameter model can run on a commodity CPU box, the economic calculus for AI deployment shifts. Enterprises can repurpose existing servers rather than invest in costly GPU clusters or NVMe‑heavy storage. The ability to sustain a 256K token context in RAM also opens doors for long‑form reasoning and agentic applications on modest hardware. The open‑source nature of llama‑modchip invites rapid iteration, potentially accelerating a broader move toward CPU‑centric large‑model inference.
Frequently asked questions
How can the 284B DeepSeek model run on a machine with only 64 GB of RAM?
Llama-modchip compresses the model using SwiGLU‑H2 Hessian compression, Fractal‑64 Q2_0 quantization and register tiling, shrinking the checkpoint to 56.56 GiB on disk and 57.61 GiB resident memory, allowing it to fit in 64 GB RAM.
What token throughput does llama-modchip achieve on CPU‑only hardware?
The 284B model reaches 44.36 tokens per second during prefill and 9.1–14.7 tokens per second on pure CPU decode, rising to 11.5–14.7 tok/s with Stockfish‑PVS; the 30B MoE variant hits up to 111.6 tok/s on Threadripper.
Do I need a GPU to use llama-modchip?
No. The repository explicitly states that zero GPUs are required; all supported models run on pure CPU with the provided binary.
Llama‑modchip demonstrates that aggressive compression and clever kernel engineering can democratize access to the largest language models. As the code matures, developers may bypass expensive GPU farms, reshaping the AI hardware market and widening the field of actors who can experiment with frontier‑scale inference.
Sources: [Hacker News – Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU]