Running Kimi K3 on a M1 Max

Install

Three commands, then you're generating. The only real decision is step 3.

The two modes

Every token reads 16 experts × 92 layers = 25.8 GB of expert data. From local disk that's about 4 seconds; over the network it's minutes. That single fact is the whole difference between the two columns.

Run setup_k3.py with no flag and it picks --full when the disk allows, otherwise falls back to streaming and tells you exactly how much space you'd need to free.

Starting with streaming and upgrading later

Streaming is a fine way to try Deltafin without committing 1.7 TB. Whenever you want the speed, one command finishes the job — no reinstall, no reconfiguration, and it picks up whatever is already cached:

Deltafin prints a reminder at startup — both for the CLI and the API server — whenever it's still in streaming mode, showing how much of the pool is local and what finishing would cost.

Optional: int8 spine

Halves per-token I/O for the non-expert weights, with no meaningful quality change in our checks. Takes a few minutes:

Usage

Tokens print as they are generated, so you always see the text as it comes. Ctrl-C stops cleanly at any point and prints the result so far; --max-new N caps the length. One honest warning: K3 thinks before it answers, and at about a token per minute a full chat answer can take a while — watching it stream is part of the experience.

Router selections are logged to router_trace.jsonl if you want to study K3's routing behaviour.

Deltafin can serve the standard OpenAI API, so chat interfaces, the openai SDK and coding agents can use it by changing a base URL:

/v1/chat/completions, /v1/completions and /v1/models are implemented, and streaming ("stream": true) works. Most tools that read OPENAI_BASE_URL and OPENAI_API_KEY will work by pointing those at the server.

Please read these caveats before pointing anything automated at it:

• Time. Answers arrive when they arrive — set your client's timeouts to hours, not seconds. Omitting max_tokens lets the model finish its answer (recommended); raw completions, which never end on their own, default to 256. Operators can set a hard ceiling with K3_SERVER_MAX_TOKENS.
• Streaming installs are much slower here. A chat-template prompt is 60 tokens or more and prefill touches many experts per layer, so on a partly filled cache a chat request can spend hours fetching. With a full install it's just normal (slow) inference. The server prints a warning at startup when you're in streaming mode.
• Greedy only. temperature and top_p are accepted and ignored, and one request runs at a time (a second concurrent request gets a 429).
• Agents are a curiosity, not a workflow. Coding assistants work in principle, but their long system prompts make prefill expensive.

Configuration

Everything works with no configuration: Deltafin picks the GPU when there is one and the int8 spine when it has been built, and says what it chose at startup. These variables exist for overriding that:

Requirements

• An Apple Silicon Mac. All published numbers are from an M1 Max with 64 GB — the slowest machine it has run on. More RAM is used automatically (a 128 GB machine pins several times more of the model), and newer chips bring higher memory bandwidth, more GPU cores and faster storage. See Why newer Macs should be faster.
• Xcode Command Line Tools, for clang (xcode-select --install).
• Python 3.12 or newer.
• Disk: ~1.7 TB for the full install, ~215 GB for streaming (see Install).
• Network access to Hugging Face.

How it works

K3's weights total about 1.56 TB, which is more than this machine's free disk, let alone its RAM. The observation that makes local inference possible anyway is that a Mixture-of-Experts model only touches a small fraction of itself per token.

• The resident spine (~114 GB: attention, shared experts, latent projections, embeddings) is downloaded once and read layer-by-layer from local NVMe each token, quantized to int8 and computed on the GPU.
• The 82,432 routed experts (~1.45 TB). For each token K3's router picks 16 experts per layer, and only those are read. Install them all locally if you can (recommended); otherwise Deltafin fetches them from Hugging Face on demand — one HTTP range request per expert — into a growing disk cache.
• The forward pass runs Moonshot's own modeling code, unmodified. A small pure-PyTorch shim stands in for the CUDA-only fla kernels it expects.

What to expect

All numbers below were measured on one M1 Max (10-core CPU, 32-core GPU, 64 GB, internal NVMe) with the full model installed locally, greedy decoding, on a quiet machine. This is the slowest machine Deltafin has run on — treat these as a floor.

Roughly 3.75 tokens per minute. Where those 16 seconds go, per token:

Decode is now bound by disk bandwidth on the resident spine. Those 53 GB are re-read every token, and at the ~7 GB/s this access pattern sustains that is about 7.5 s of the 15 — unavoidable without either more RAM (enough to hold the spine without displacing the page cache the expert reads need) or a smaller spine.

Why newer Macs should be faster

Every one of those lines is bound by something the M1 Max is simply the weakest at. Nothing here is tuned for a specific chip, so a newer machine should pick up the difference without any configuration:

• Memory bandwidth. The M1 Max has 400 GB/s. An M3/M4 Max is meaningfully higher and an Ultra roughly doubles it — that lands directly on the spine load and the expert matmuls.
• GPU. More cores execute the same Metal kernels faster; the dequant shader and the attention path both scale with it.
• SSD. Expert reads are the single biggest slice, and they run at whatever the internal drive delivers. Later Macs ship faster NVMe.
• RAM. This matters most. The 53 GB spine does not fit alongside everything else on a 64 GB machine, so it is re-read from disk every token — about half the total time. On a 128 GB machine it can simply stay in the page cache, and that cost largely disappears. Deltafin also pins more of the model there automatically, with no flags.

We have only ever run this on the one machine. If you try it on an M3, M4 or M5, or with 128 GB, we would genuinely like to see your numbers — open an issue with the output of K3_PROFILE=1 and your chip.

When n-gram speculation accepts a draft, one forward pass emits two tokens, so repetitive text runs proportionally faster. Speculation is lossless: accepted drafts reproduce the reference sequence exactly, and a rejected draft restores the model state bit-for-bit.

The gap between the two decode rows is the whole argument for the full install (see above): with the experts local, every prompt runs at the top-row speed instead of only the ones whose experts happen to be cached.

Output is greedy and reproducible: the same prompt yields the same tokens, run after run.

The capital of France is → Paris. The Eiffel Tower is located in Paris. The Louvre Museum is also in Paris. The Louvre has…

To be clear about the limitations: this is a research artifact, not a practical chat setup. Sixteen seconds a token is a long way from interactive, and long prompts are expensive because prefill touches many experts. We think it is interesting mainly as an existence proof, and as a testbed for streaming-inference techniques.

Techniques

Each of these was measured on real weights before we trusted it. The full experiment log — including the ideas that didn't survive measurement — is in BRAINSTORM-SPEED.md, and the original plan in PLAN.md. Little of this is novel on its own; most of it adapts ideas from the projects credited below to K3's particular shape.

I/O and streaming

• Coalesced expert fetch. Each expert's six tensors happen to be contiguous in the shard files (we checked all 82,432), so a whole expert is a single 17.55 MB range request over a small pool of keep-alive connections. That measured about 6.4× faster than fetching tensors individually. We also tried HTTP/2; it was slower than HTTP/1.1 keep-alive in this setting.
• Raw-span disk cache. Cache files are the shard bytes verbatim — no container format, no parsing.
• Parallel expert reads. A layer's 16 selected experts are read together by a thread pool using pread with F_NOCACHE, rather than being demand-faulted page by page while the kernel computes. Measured cold: 0.87 GB/s faulting versus 6.85 GB/s reading. This was worth 40 s → 4.3 s per token on the read path alone, and F_NOCACHE keeps 25 GB/token of expert traffic from evicting the page cache the spine needs.
• Double-buffered layer loading. A worker thread reads the next layer's spine data while the current layer computes.
• Previous-token prefetch. Consecutive tokens reuse about 31% of their expert selections, so each token's set is fetched in the background for the next one. (An earlier 39.7% figure counted repeated prompts in our own trace; on a deduplicated holdout it is 30.8%, below the 41.3% colibri reported for GLM.)

Compute

• Fused MXFP4 dequant+GEMV (tools/fused_gemv.c) — a NEON kernel that dequantizes and multiplies in one pass using a 16-entry table lookup, with the e8m0 scale applied as integer arithmetic on the fp32 exponent. It matches the reference implementation bit-for-bit and replaced a much slower dequantize-then-matmul path. A Metal version exists as a validated prototype.
• Template-layer buffer reuse. All 69 KDA layers share one set of tensor shapes and all 24 MLA layers another, so two persistent GPU-resident "template" layers can receive each layer's weights via copy_(). This avoids the allocator churn that profiling showed was a large share of per-token time.
• int8 resident spine. Halves the per-token resident I/O. In our checks the top-5 next-token candidates kept their order and the top logit moved by 0.07%.
• Custom Metal dequant kernel. Loading the spine spent most of its time in a row-broadcast multiply that MPS runs at 43 GB/s, against 334 GB/s for a plain copy of the same bytes. A small compile_shader kernel fusing int8→fp32, the row scale, and the copy reaches 297 GB/s; with persistent staging buffers and transfers hoisted out from between dispatches, per-layer load went 118 ms → 21 ms. Bit-exact: max|diff| = 0 on every tensor.
• Pure-PyTorch KDA shim (tools/fla/) — Kimi Delta Attention's recurrence, short convolution, and gated norm, ported from fla-core's semantics. Chunked and step-by-step execution agree to about 1e-9. At decode the recurrence runs on CPU, where its small state fits better than a series of GPU dispatches.

Decoding

• N-gram speculation. Drafts come free from suffix matching against the text so far, and are verified in a two-position batch whose fixed costs are shared. This is worthwhile here precisely because resident I/O and compute — not expert fetching — dominate a warm token. Accepted drafts reproduced the reference sequence exactly in our tests, and rejection restores the model state bit-for-bit (K3's recurrent states are small enough to snapshot cheaply).

• Two modes. exact (default) is logit-faithful and reproducible; we use it for all correctness work. approx (K3_APPROX=1) uses fp16 weights, so output stays coherent but near-tie tokens can differ between runs. We haven't measured approx to be reliably faster yet, which is why it isn't called "fast".

Scaling with RAM

• At startup Deltafin reserves memory for the OS (max(10 GB, 18%)) and pins as many resident layers as the remainder allows. A 128 GB machine pins several times more than a 64 GB one without any configuration, and the expert cache additionally benefits from whatever page cache is free.

What didn't work

We kept the failures, since they may save someone else the time:

Where this could go

Roughly in order: the Metal expert kernels (already prototyped), a proper quality harness — average NLL against the official API — so lossy speed/quality trade-offs can be measured rather than argued about, smarter expert prefetching, and eventually a native engine in the spirit of ds4, where most of the remaining overhead should disappear. Details in PLAN.md.

Thanks

Deltafin leans heavily on work that others published openly. In rough order of influence:

• colibri (JustVugg, Apache-2.0) — showed that a 744B MoE can run in 25 GB of RAM, and is where we learned about router-lookahead prefetch, learned expert pinning, F_NOCACHE and F_RDADVISE discipline on macOS, and the shard-by-shard conversion pattern. Its M5 Max performance report — CPU spin-waits starving the GPU of a shared power budget — changed how we schedule work.
• ds4 / DwarfStar (Salvatore Sanfilippo, MIT) — the clearest expert-streaming design we studied: zero-copy expert buffers, masked dispatch, selection-based cache eviction, session persistence, and a quality methodology (average NLL against official API outputs) that we adopted outright. Its stated philosophy — correctness before speed, hide I/O behind compute — is the sensible one, and we tried to follow it.
• Moonshot AI — for releasing K3's weights openly with readable modeling code, which Deltafin runs directly, and for the Kimi Delta Attention design, whose small recurrent state is what makes long context feasible on a laptop at all.
• flash-linear-attention (fla-org, MIT) — our KDA shim is a port of semantics from its kernels and reference implementations.
• llama.cpp / ggml — prior art for in-kernel dequantization and MXFP4 handling, and the foundation of most of what the local-inference community knows.
• PyTorch, Transformers, ml_dtypes (our bit-exactness reference for e2m1) and tiktoken.

License

Deltafin's own code is MIT. Two things in this repository are not ours:

• tools/fla/ is a pure-PyTorch port of semantics from flash-linear-attention (MIT, © 2023–2026 Songlin Yang, Yu Zhang, Zhiyuan Li). The attribution is repeated in the file header and in LICENSE.
• Kimi K3's weights and modeling code belong to Moonshot AI and are distributed under Moonshot's own license. They are downloaded at setup, never vendored here — please read that license before using them.

Deltafin is an independent project with no affiliation to Moonshot AI.