Rank #10 · Daily · AI
kimi-k3-in-c: Portable C99 MoE Inference for Huge Kimi K3 on CPU RAM
A from-scratch C99 MoE runtime that streams a multi-trillion-parameter Kimi K3 checkpoint from disk so the same model answers on an ordinary laptop RAM budget — slowly, but without a GPU cluster.
What it does
kimi-k3-in-c is a portable C99 inference engine for Moonshot's Kimi K3 Mixture-of-Experts model. Repository framing: 2.78T parameters, ~1.56 TB checkpoint on disk, no BLAS / no framework / no GPU, with a tiny engine binary. The design goal is byte-identical output across memory budgets; more RAM mostly buys speed by keeping more of the model resident.
Architectural reductions described in the README (high level):
- Experts already ship in packed low-bit form (MXFP4-style) and are multiplied without materializing full dense experts in RAM.
- Attention uses memory-bounded variants (KDA / MLA family ideas as documented) so the KV story doesn't explode.
- A dense trunk can be partially streamed; routed experts use an LRU cache sized by preset (
laptop/server/ etc.). - Cross-platform ports (Linux / macOS / Windows; AVX2 / NEON paths among contributions) stay in portable C99.
Presets trade resident set vs I/O. The README publishes author-measured tables (e.g. peak RSS ~8.24 GB on a laptop preset, multi-second-per-token timings that improve when the model fits in memory). TidyHow has not reproduced those timings or RSS figures. Treat them as project-reported measurements from docs/data/, not independent throughput claims — and do not invent faster numbers here.
Practical notes from the project:
- You still need the large checkpoint and a fast enough NVMe for streaming presets.
- Chat/REPL paths exist alongside raw continuation prompts.
pushed_aton verification was 2026-10-02 (about a month before this Daily capture) — active enough for a research engine, not a daily-commit product.
Homepage write-up linked from the repo: the author's Medium article on building Kimi K3 in C for consumer hardware.
Who it's for
- Systems / ML engineers who want to study or run a huge MoE on commodity CPUs without CUDA stacks.
- Educators and researchers interested in memory-hierarchy MoE inference (streaming experts, quantized kernels, identical-output contracts).
- Hobbyists with disk space and patience — interactive chat on an 8 GB laptop preset is a research demo, not a product latency target.
Not a drop-in OpenAI-compatible server replacement for production chat apps.
Stars / language / license
10,311 stars · C · Apache-2.0
Data verified: 2026-11-05 09:31 Asia/Shanghai (GitHub API get_repository + README get_file_contents)
Trending note: +351 stars today (Trending selection captured 09:26 Asia/Shanghai)
Category tags
AI · MoE · CPU-inference · C99 · quantization · systems · Kimi-K3 · from-scratch
How it differs from similar projects
Compared with llama.cpp / ggml family. Those target (mostly denser) models with a broad GGUF ecosystem and GPU offload. kimi-k3-in-c is a purpose-built MoE path for Kimi K3, emphasizing streaming experts and a no-dependency C99 core rather than a multi-model runtime.
Compared with PyTorch / vLLM / TensorRT-LLM. Framework stacks assume accelerators and large RAM/VRAM. This engine's thesis is the opposite: CPU + disk hierarchy, identical answers at different RSS budgets.
Compared with yesterday's pure-C MoE demos (e.g. colibri-style engines). Shared "tiny C, huge MoE" aesthetic; kimi-k3-in-c is specifically wired to the Kimi K3 checkpoint layout and documented reduction cascade. No cross-engine benchmark is claimed here.
Again: any seconds-per-token or RSS numbers in the upstream README are author measurements. This article does not restate them as TidyHow results and does not invent throughput figures.