180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
Streaming MoE experts directly from SSD allows running a 180B parameter model on a 32GB Mac with only a 6% performance drop compared to Strata. This breakthrough enables deploying large MoE models locally on consumer hardware, reducing reliance on expensive GPUs or cloud services for inference, and makes it feasible to run complex models like Qwen3.8-Flash-Next at practical speeds for coding agents and other applications.
32 GB unified memory can now run a 180B MoE model at 13.5 tok/s by streaming experts from SSD. This means you can ship agentic workflows on consumer laptops without cloud GPUs, cutting inference costs to near-zero, but latency will still break real-time autocomplete; expect to batch or pre-warm experts instead.