Memory bandwidth is the target
Packed ternary weights reduce the bytes streamed per token; compute layouts spend RAM only when the hardware path benefits.
Software system 03 / Inference
Ternary networks on CPUs, with exact math and memory used deliberately.
Hyphae BitNet is the foundation repository for the current celiums-bitnet runtime: one-shot generation, native HTTP serving, an experimental C ABI, and prefill/decode benchmarks.
Make supported ternary models operational on ordinary CPU servers without turning approximate kernels, uncontrolled RAM growth, or inherited engine internals into hidden product behavior.
01 / Benefits
Why the runtime exists
Exact accumulation
I2_S uses (D−S)ρ and Q1 uses 2P−S with integer accumulation; approximate LUT-GEMM and T-MAC stay outside strict mode.
Fail-closed RAM budgets
Model load or session creation refuses to exceed the configured working-set cap instead of making the host unusable.
Family-gated loading
Supported architecture and file-type combinations are explicit; ordinary Llama/Qwen Q4 models and colliding Q2 formats are refused.
Product-owned interfaces
Applications use the celiums-bitnet CLI, C ABI, or native HTTP subset instead of binding to GGML internals.
Receipts before marketing
Benchmarks include model digests, RAM cap, CPU/ISA, prefill/decode split, raw JSON, and explicit non-claims.
02 / Architecture
How inference moves through the runtime
Packed GGUF remains the durable store. An optional, budgeted in-memory compute layout matches the CPU ISA; prefill and decode then use different kernels.
- 01
Validated GGUF
I2_S BitNet or Q1_0 Bonsai; architecture, type, revision, and digest remain operator-visible
- 02
Budget gate
ram_budget reserves host headroom and fails before allocation if the selected layout cannot fit
- 03
ISA compute image
ARM i8mm expands Q1 to int8 ±1; x86 VNNI keeps bit-packed 4×8 panels; I2_S uses strict tinyBLAS paths
- 04
Prefill
GEMM reuses each weight panel across eight then four activation rows
- 05
Decode
GEMV handles one token and streams the weight image once; prefill optimizations do not pretend to accelerate decode
- 06
Public runtime
Tokenize, prefill, decode, logits, sampling, cancellation, streaming callbacks, HTTP/SSE, and JSONL benches
03 / Surface
Supported product surface
Certified BitNet 2B
Microsoft BitNet b1.58 2B in MOSTLY_I2_S with pinned conversion identity and exactness gates.
Bonsai 27B CPU text
Pre-quantized Q1_0 Qwen35 family with 1-bit signs, FP16 block scales, DeltaNet plus attention, and explicit family selection.
Runtime and serving
One-shot generation, native HTTP completion/chat subset, SSE, API keys, metrics, stop sequences, and cooperative cancellation.
Optional Hyphae gateway
A separate Rust process can add retrieval, memory, receipts, registry, proofs, semantic cache, and MCP without linking Hyphae into GGML.
04 / Evidence
Measured hardware effects
Prefill and decode have different rooflines. The published receipts report both and preserve cases where a layout helps one phase but hurts another.
Graviton 4 prefill at one thread
Q1 expand-to-int8 vs packed mmap
Graviton 4 decode at one thread
high thread counts can regress
Xeon prefill at one thread
8-row vs 4-row GEMM
deliberate failing budget
refused before model allocation
The Graviton and Xeon figures apply only to the named models, digests, machines, thread counts, prompt/generation lengths, and 64 GiB cap. Expanding Q1 can reduce high-thread decode when DRAM saturates.
What it does now
- Runs certified BitNet 2B I2_S and the explicit Bonsai 27B Q1_0 CPU text family.
- Provides CLI run/serve/bench/validate/version commands, an experimental C ABI, and an OpenAI-shaped HTTP subset.
- Selects native, portable AVX2, or scalar CPU profiles and budgeted compute layouts.
- Validates exact kernel behavior through scalar/generic oracles, sanitizers, package tests, and model-backed release gates.
What it intends to do
- Compare additional exact kernel tiles and shared activation quantization.
- Explore exact zero-channel compaction and denser ternary encodings without approximate pruning.
- Evaluate AMX as a prefill-only path while keeping persisted weights packed.
- Mature multi-sequence scheduling and asynchronous request ownership.
- Move GPU paths from inherited experiments to generic, explicitly tested I2_S kernels before support claims.
What it deliberately refuses
- Not an arbitrary llama.cpp model runner; unsupported architecture/quantization combinations are rejected.
- No supported GPU inference, continuous batching, complete OpenAI API, embeddings, tools, logprobs, or built-in TLS in the current product.
- No claim of universal bitwise identity across ISAs or source-checkpoint certification.
- The current public binary and release artifacts retain the celiums-bitnet product name; the foundation repository does not erase upstream or model ownership.
Hyphae BitNet