K2 Horizon: Six Fully Open Models and 21TB of Training Data
Most "open source" model releases mean open weights and a carefully filtered data card. K2 Horizon, released September 3, 2026 by the Institute of Foundation Models (IFM, the MBZUAI lab spun up in May 2025), does something closer to actual open science: six foundation models from 0.9B to 375B parameters, the training datasets themselves on Hugging Face, data recipes and mixtures, intermediate checkpoints, and fine-grained training logs on W&B. The claim attached to it, largest fully open-source model launch in AI history, is not obviously marketing. I went through the release the day after it dropped and checked the numbers myself.
What actually shipped
The fleet is six models, all sharing the same architecture family, tokenizer (except the 0.9B, which has a smaller vocab), training method, and tooling:
| Model | Type | Stored params | Context | The pitch |
|---|---|---|---|---|
| K2-Horizon-0.9B | dense | 1.08B with embeddings | 8K native, 131K via YaRN | Watch and edge class, SOTA under 1B |
| K2-Horizon-3.7B | dense | 5.06B | up to 524K | Phone and on-device, fine-tune friendly |
| K2-Horizon-7B | dense | 9B | 524K native | Best under 10B, strong SWE scores, runs on a phone |
| K2-Horizon-32B | dense | 35B | 512K | Most powerful dense, but stage-1 only (see caveats) |
| K2-Horizon-MoVA-36B-A4B | MoE + MoVA | 37B total, ~4B active | 512K | Near-32B quality at 4B active params |
| K2-Horizon-375B-A23B | MoE sparse | 379B total, ~23B active | 512K | Flagship for enterprise reasoning and agents |
Weights and code are Apache 2.0. The collection advertises FP8 variants and GGUF quants for llama.cpp, and there is day-zero support claimed in vLLM, SGLang, and Ollama across NVIDIA, AMD, and Cerebras hardware. One practical warning from running these myself, though: the architecture is custom (k2_horizon), and as of September 20 I could not get a GGUF build running at all. The reliable path is the official codebase through Hugging Face transformers 5.17+ with trust_remote_code=True in BF16. The lineage goes back through LLM360 and K2 in 2024, K2-Think 32B in September 2025, K2-V2 70B in December 2025, and K2-Think V2 in January 2026, so this is a mature line, not a first attempt.
The training data is real, and I checked
This is the part that makes the release unusual. A Hacker News commenter claimed the actual training data was sitting on Hugging Face: 3.3TB for code reasoning, 4.5TB for mathematical reasoning, 8.4TB of pre-training behaviors. I checked the dataset headers on September 4. The claim holds:
| Dataset | Size | Rows | License |
|---|---|---|---|
| Code-Reasoning | 3.28 TB | 451.5M | Apache 2.0 |
| Math-Reasoning | 4.46 TB | 2.02B | Apache 2.0 |
| Pretrain-Behaviors | 8.37 TB | 1.64B | Apache 2.0 |
| TxT360-v2 | 5.29 TB | 1.84B | CC-BY-4.0 |
| SFT-Reasoning | 159 GB | not listed | Apache 2.0 |
That is roughly 21.5TB across five repos and about 5.9 billion rows. Not the full 20T-token crawl per model, but the training-ready reasoning and synthetic slices, the part you would actually fine-tune with. Code-Reasoning alone has a code-thinking-hard subset of 34.3M thinking traces, the kind of slice that normally lives behind a closed lab's doors. Math-Reasoning includes 249M solutions generated with Qwen3-32B thinking and 248M with GPT-OSS-120B high reasoning, two different teachers you can compare. Pretrain-Behaviors has 502M reasoning rows and 290M planning trajectories. Each pretraining run used about 20T tokens, with around 10T of that synthetic and roughly 17% explicit reasoning trajectories.
One licensing detail worth caring about: TxT360-v2 is CC-BY-4.0, not Apache. If you build derivatives, keep it in a separate pool and carry the attribution.
The interesting tech
Two pieces stand out.
MoVA (Mixture-of-Value-Attention) extends MoE sparsity from the feedforward layers into the attention value projection. The 36B-A4B model is the reference artifact: near-32B quality while activating only about 4B parameters per token, and it stays compatible with FlashAttention and GQA.
Uno is a diffusion distillation trick shipped as a LoRA. Frozen autoregressive weights plus a lightweight adapter that generates token blocks in parallel, with a claimed ~3x speedup at no quality loss, and the gains persist across batch sizes. You attach the LoRA and go. Whether that holds up outside their benchmarks is exactly the kind of thing a fully open release lets everyone test.
The fleet also ships native 512K context on the 375B, 36B-A4B, and 32B from mid-training onward, and the 0.9B stretches its native 8K to 131K with a YaRN factor of 16. Every size includes reasoning effort controls (high, medium, low) as native chat template options. Every model is a thinking model: outputs carry <ifm|think>...</ifm|think> reasoning traces, so budget your tokens accordingly.
I benchmarked the three small ones
Since the sub-10B models are the practical ones for most readers, I put the 0.9B, 3.7B, and 7B through throughput and capability probes on a Kaggle VM with two Tesla T4s, BF16 through transformers. The 32B and the MoVA MoE did not fit (both need 65-75GB of weights, beyond 30GB of VRAM even attempting aggressive offload).
| Model | Setup | Prompt processing | Generation | VRAM |
|---|---|---|---|---|
| K2-Horizon-0.9B | single T4 | 65-75 tok/s | 19 tok/s | 2.2 GB |
| K2-Horizon-3.7B | single T4 | 47-79 tok/s | 16-18 tok/s | 10.1 GB |
| K2-Horizon-7B | 2x T4 split | 28-44 tok/s | 12.7 tok/s | 18 GB |
My probes were small (grade-school math, prime-checking and Fibonacci code, instruction following with exact-format constraints), so treat them as sanity checks against the card claims, not a suite verification. That said, the pattern was consistent: the card claims 0.9B at AIME 41.7/48.5 and 79.9 HumanEval-plus, and claims 70.6 SWE-bench Verified for the 7B, and everything I threw at all three sizes got the correct reasoning with the only failures being my own token caps cutting answers mid-thought.
The interesting findings were about shape, not score. All three are thinking models, and the family trait is that reasoning was correct at every size while answer completeness depended almost entirely on token budget, not model size. The 7B is the most deliberative and can over-think a trivial prompt until its budget runs out. The 3.7B had the crispest instruction following of the three, matching an exact format on the first try. And the 0.9B is the star for weak hardware: real reasoning at 19 tokens per second inside 2.2GB of VRAM is the best performance-per-gigabyte of any small model I have tested this year.
Two deployment lessons for anyone reproducing this. First, keep any model under 14GB of weights on a single GPU; splitting the 3.7B across two T4s dropped its generation speed (7.0 tok/s split versus 16-18 on one), while the 7B genuinely needs both. Second, generation was rock-steady once loaded, and load times were fine after first download (the 0.9B reloads in 3 seconds).
The caveats, and they matter
- Training code is still placeholder. The IFM GitHub org's repos for the training infra and post-training pipeline were empty as of September 4. The previous K2-Think V2 release did ship a full recipe eventually, so this is probably a fill-in-progress situation, but you cannot reproduce full pretraining today.
- The 32B is a stage-1 checkpoint, final pending. Early community benchmarks put it behind Qwen3.8-27B and even behind the 7B on some suites. If you are picking a model today, the 7B, 3.7B, and 0.9B are the ones to look at. Re-evaluate the 32B when the final lands.
- The reward-hacking self-audit is unusual and worth reading. The 375B scored 70.2% on TerminalBench 2.1, dropping to 66.9% after removing 24 of 500 flagged trials. The 7B once hit a big SWE-bench number partly by downloading answers from GitHub. IFM published this themselves, which is the open science posture in action, but treat headline scores with the same suspicion you would apply anywhere else.
- Scale is real. You are not downloading 21TB to a laptop. Stream and filter. The only repo small enough to fully materialize on a modest VM is SFT-Reasoning at 159GB.
- The SFT-Reasoning dataset viewer is broken on HF. Use streaming or file listings, and inspect the features per subset before building pipelines, since field names differ between subsets.
Why this release matters
Open weights let you run a model. Open data plus recipes plus checkpoints plus logs let you study how it was made, replicate parts of it, and build on the exact slices that made it work. That is the difference between "free to use" and "actually reproducible", and no previous release has shipped this much of it at once. Even if you never touch the 375B, the 21.5TB of Apache 2.0 reasoning traces is arguably the most valuable part of the whole thing, and it is just sitting there, streaming-ready.
The models live on the IFM Hugging Face org. My practical suggestion: ignore the flagship entirely unless you have a cluster, start with the 0.9B or 3.7B through transformers (they run on a single modest GPU), and treat the open datasets as the real release. One caveat to keep in mind as you deploy: the 512K context claims are architecture-level, and KV cache at those lengths needs far more memory than most setups have, so treat context as a scaling plan rather than a working spec until you have tested it on your own hardware.