Vectors that truncate
A Matryoshka embedding is trained so that its first 64 dimensions are already a usable vector, and its first 128, and its first 256 — each prefix a complete representation rather than a fragment. One model then serves a cheap coarse index and an exact rerank without retraining or distillation.
legal-financial-modernbert-150m · pretraining from scratch · Matryoshka embeddings
View source All worksWhat nesting actually means
On the left, an ordinary embedding: cut it short and you have destroyed it, because no prefix was ever asked to stand alone. On the right, a Matryoshka objective applies the loss at every nested width at once, so information sorts itself coarse-to-fine along the axis. Drag the dimension slider and watch which one survives the cut.
A schematic of the training objective, not a measurement. Stage-1 pretraining is still running and the project makes no quality claims yet — so there are deliberately no retrieval scores on this page.
The training budget, to scale
The numbers this project is actually committed to. 124 GB of deduplicated text becomes 25.79B tokens; training runs 271,000 steps to 71B tokens, which is 2.76 epochs over the corpus and 475 tokens per parameter. Scrub the schedule to see where a given step lands.
Figures from the project README: 149.7M parameters at ModernBERT-base geometry from random init, 50,368-token vocabulary, ~28,400 tok/s in bf16 with sparse prediction and full torch.compile.
legal-financial-modernbert-150m
124 GB of deduplicated corpus becomes 25.79B tokens, 80% legal and financial against 20% FineWeb-Edu, under a 50,368-token vocabulary carrying 156 domain tokens. Stage-1 pretraining runs 271,000 steps to 71B tokens — 475 tokens per parameter — at roughly 28,400 tok/s in bf16 with sparse prediction and full torch.compile. The dedup pass is a Go program doing 604 MB/s, 24× the Python reference. All of it on a single desk-side GB10.
- Params149.7M, ModernBERT-base geometry, random init
- Corpus124 GB deduplicated → 25.79B tokens
- Schedule271k steps → 71B tokens → 2.76 epochs
- Hubsakshamio/legal-financial-modernbert-150m
Stage-1 pretraining in progress. Published checkpoints are early and no quality claims are made yet.
HPC Cluster Simulator
A drag-and-drop, gamified training environment for building and operating GPU clusters. Trainees design a cluster, run workloads against it, break it, and learn why it broke.
The Immortal Daemon
An RL environment where the agent must keep a realistic cloud application alive while chaos batters it — and every reward is machine-verifiable, not judged by a model.
Populace
A world-scale persona simulation: grounded personas on a real globe living daily routines and reacting to events, with a language model authoring what each kind of person makes of the news and a Go engine doing the per-capita work.