Fair comparisons
Strategies share attention, MLP, tokenizer, data, training pipeline, and declared tuning budgets.
Software system 02 / Research
Transformer experiments that keep the hypothesis, budget, failure, and result together.
Hyphae Transformer is a from-scratch PyTorch framework for controlled research on residual strategies in decoder-only transformers.
Turn architecture research into a replayable process rather than a folder of scripts and selectively reported runs.
01 / Benefits
Why the framework matters
Identity-initialized control
Canonical, split, and RMS-normalized ReZero strategies expose explicit residual gates and trajectories.
Typed research objects
Hypotheses, stages, budgets, manifests, evidence, and reports are structured rather than implied by filenames.
Resumable and idempotent
Atomic checkpoints and append-only histories restore optimizer, data source, and RNG state; completed manifests do not rerun.
Negative results survive
Failed seeds, inconclusive intervals, invalid attempts, and stop decisions remain first-class evidence.
Cloud with a leash
Cloud promotion requires local invariants, an explicit cost/lifetime budget, artifact retrieval, and cleanup.
02 / Architecture
The experiment lifecycle
A result advances only through ordered gates. Interim curves do not authorize optional stopping.
- 01
Preregister
Question, strategy, baseline, seeds, data identity, effect threshold, budgets, and stop rules
- 02
Static validation
Schemas, source and corpus identity, model geometry, finite-state and contract checks
- 03
Mini-pilot
Correctness-only run before spending the full token or hardware budget
- 04
Pilot / full local
Paired observations, fixed validation grid, checkpoints, gate trajectories, failures, and confidence intervals
- 05
Promoted cloud rung
Exact manifest, allowlisted runner, hard time/cost cap, artifact retrieval, and resource destruction
- 06
Immutable report
Aggregate JSON/HTML, per-seed evidence, verdict, non-claims, protocol defects, and next decision
03 / Surface
What the laboratory contains
Core
ReZero gates, decoder-only model, grouped-query KV caches, RMSNorm, SwiGLU, RoPE, optimizers, and signal diagnostics.
Lab
Typed schemas, campaign runner, registry, memory, budgets, search, evaluator, serialization, and reports.
Governed training
Frozen-backbone control heads for answer, request-evidence, and abstain decisions with strict evidence-pointer bounds.
Data and execution
Checksum-pinned WikiText-2, canonical enwiki8 ranges, portable paths, containers, and an allowlisted manifest runner.
04 / Evidence
What the current evidence says
The strongest promoted result is narrow: one residual strategy under a specific 30M campaign, not universal transformer superiority.
paired final-NLL improvement
95% CI [+1.59%, +3.06%]
token reduction to 1.8 NLL
eight paired seeds
earlier threshold at 24/48 layers
secondary metric
cloud cap overrun retained
protocol defect remains visible
The eight-seed 30M campaign cleared its preregistered practical threshold. Earlier depth-rung primary verdicts remained inconclusive, and the cloud budget overrun is reported rather than erased.
What it does now
- Compares five residual strategies under one controlled decoder-only pipeline.
- Runs staged local and cloud campaigns from strict manifests.
- Captures checkpoints, histories, metrics, failures, confidence intervals, budgets, and reports.
- Supports autoregressive generation and a governed frozen-backbone control-training slice.
What it intends to do
- Test whether observed effects persist across larger parameter, sequence-length, corpus, and architecture rungs.
- Improve cloud runner and artifact automation before scaling cost.
- Expand independently annotated frozen-backbone evaluation before any production-quality claim.
- Keep research memory and publication workflows governed by explicit source and sufficiency policies.
What it does not establish
- Not a production model service or a universal claim for ReZero.
- The fixture backbone proves training and governance contracts, not production Gemma quality.
- No test-set optional stopping or unequal tuning budgets are allowed.
- Historical package names and manifest identifiers remain for compatibility and are not rewritten.
Hyphae Transformer