Rank #05 · Daily · Infra
FreeToken: Datacenter-Style MoE Serving on a Desktop GPU
Serve big MoE models from a gaming or workstation PC the way a datacenter would, with elastic memory, expert caching, and agent-ready endpoints.
What it does
The repository describes FreeToken as "datacenter-scale model serving to your desktop." The README's main pillars:
- Edge-native runtime. Bandwidth-adaptive CPU–GPU co-execution, streamed prefill, a global LRU expert cache, and a dedicated fast weight format (FTW).
- Semantic-aware caching. Checkpoints for KV and recurrent state, so agentic context edits (tool calls, thinking blocks) don't force a full recompute.
- Elastic memory. VRAM is reallocated between expert cache and KV at runtime without restarting the engine.
- Models and formats. Frontier open MoE families across MXFP4 / NVFP4 / FP8 / BF16. See
docs/models.mdfor the exact matrix. - Hardware. NVIDIA RTX 30/40/50-series laptops and desktops.
- APIs. OpenAI- and Anthropic-compatible, for coding agents such as Codex, Claude Code, and OpenCode.
Distribution: a desktop app (Windows/Linux, via flashml.ai) or a Python package (uv pip install "freetoken[accel]"). An accompanying arXiv paper describes the system. The README credits SGLang, vLLM, FlashInfer, llama.cpp, and others for design or code reuse.
Who it's for
Developers with a recent NVIDIA GPU who want a local, agent-compatible endpoint for large MoE models, and systems researchers interested in edge MoE scheduling.
Stars / language / license
16,070 stars · Python · Apache-2.0
Data verified: 2026-11-03 09:04 Asia/Shanghai (GitHub API get_repository + README get_file_contents)
Trending note: +653 stars today (Trending selection captured 09:02 Asia/Shanghai)
Category tags
Infra · inference · MoE · local-LLM · NVIDIA · serving · agents
How it differs from similar projects
- Compared with colibri (#04). Both target "huge MoE on consumer hardware," from opposite directions. colibri is dependency-free C that streams experts from disk and runs CPU-first. FreeToken is a GPU-centric serving engine that juggles VRAM, host RAM, and the interconnect, with caching aimed at agent workloads.
- Compared with llama.cpp. llama.cpp is the general-purpose, broad-format local runtime. FreeToken is narrower (big MoE, NVIDIA) and serving-oriented.
- Compared with vLLM/SGLang. Those are built for datacenter batching. FreeToken borrows from them but optimizes for a single consumer box.
Speed claims in the README are marketing until measured. Use the paper and your own hardware. TidyHow hasn't benchmarked any of the three engines. Weights keep their upstream licenses.