Rank #04 · Daily · AI
colibri: Pure-C Engine That Streams MoE Experts From Disk
"Tiny engine, immense model." It runs hundreds-of-billions-parameter MoE models on the computer you already own — GPU optional.
What it does
A MoE model activates only a small slice of its weights for each token. colibri places those weights hierarchically: hot experts on GPU (CUDA or Vulkan when available), warm ones in RAM, and cold ones streamed from NVMe when the router asks for them. The README's example is GLM-5.2 — roughly 744B parameters, about 40B used per token, and only about 11 GB of that changing between tokens. Its sample session shows GLM-5.2 int4 streaming on CPU with about 9.9 GB resident.
- Pure C, one file per model family, zero dependencies.
- Thirteen engines listed today: language models (GLM, Kimi, DeepSeek, Qwen, MiMo, OLMoE families and others), one image model (Qwen-Image), and decision models (Laya, GLiNER2.5-Decide, Clef).
- Reads existing GGUF files as-is. Runs on CPU, on any GPU via Vulkan, or on NVIDIA via CUDA.
- One-step setup (
START-HERE.bat/./start-here.sh) checks RAM, disk, CPU, and GPU, recommends a model that fits, builds or downloads the engine, downloads the model with resume support, and opens a dashboard. - Exposes OpenAI- and Anthropic-compatible local endpoints, plus an MCP server for setup/control.
Minimum per the README: 8 GB RAM (16 GB+ recommended) and about 22 GB of free disk for the smallest model. For large streamed models, disk speed sets the token rate, so a fast NVMe drive matters most.
Who it's for
Local-LLM users who want to try frontier-scale open MoE models without a multi-GPU rig, along with tinkerers who appreciate a small, readable C codebase. If you need high throughput, this isn't the tool — streaming from disk trades speed for reach.
Stars / language / license
47,249 stars · C · Apache-2.0
Data verified: 2026-11-04 09:13 Asia/Shanghai (GitHub API get_repository + README get_file_contents)
Trending note: +802 stars today (Trending selection captured 09:12 Asia/Shanghai)
Category tags
AI · local-LLM · MoE · inference · C · GGUF · edge
How it differs from similar projects
llama.cpp has broad model coverage and can mmap weights, but it's optimized around fitting the model in RAM or VRAM. colibri is built specifically around expert streaming / hierarchical placement for huge MoE models, with per-family engines. Compared with Python serving stacks aimed at GPU/CPU co-execution, colibri is dependency-free C that runs CPU-first and optionally accelerates with Vulkan or CUDA. The README publishes per-machine tok/s tables — those are the project's own figures and TidyHow hasn't reproduced them. Model weights keep their upstream licenses.