Seven engines, pulling on each other
A cluster simulator is only interesting where its parts collide. A scheduler that ignores racks spreads a job across the fabric, which makes its collectives slower, which keeps nodes busy longer, which heats the racks, which throttles the clock, which makes the job later still. Below, the same simulator is run four times, each time isolating one of those couplings.
HPC Cluster Simulator · simulation · PDES · React Flow
View source All worksOpen the live simulator
The real thing — the React Flow canvas, the rack theatre, the pedagogy catalog and the conductor's own PDES engine — is deployed and running. Build a cluster, submit a workload, break it.
What backfill is actually promising
Both machines get the identical arrival stream from the identical seed, so anything different on screen is the policy. Strict FCFS runs the queue in order: when the head job is too wide to fit, everything behind it waits and the idle nodes stay idle. Every other policy backfills — a later job may start early only if its estimate says it finishes before the reservation the head job already holds. Users pad their estimates, which is exactly why backfill leaves some of the gap unfilled.
Bounded slowdown is turnaround divided by the job's runtime, with the denominator floored at 60s. The floor matters: plain slowdown lets very short jobs dominate the mean, and since backfill completes far more short jobs it can score worse on the unbounded measure while being better on every dimension that counts.
Why locality is a performance argument
Collectives only cross the spine when a job straddles racks, and every job that crosses it competes with every other. The locality-blind scheduler takes whatever nodes are free; the topology-aware one fills the fullest rack first. Jobs spread across three or more racks are marked. Raise the communication intensity and watch the blind machine saturate its spine while the aware one does the same work over shorter wires.
A single degraded NIC taxes every collective its node takes part in — a collective runs at the speed of its slowest participant, which is why one bad cable can cost a whole allocation.
Cooling is a performance system
Rack inlet temperature rises with the work in the rack and falls with whatever the CRAC removes. Silicon does not fail at temperature, it throttles — so losing the cooling does not stop the machine, it quietly makes it slower, and the queue backs up behind a cluster that still looks fully busy. Throttling nodes are outlined. Turn the cooling off and watch utilisation stay high while completions collapse.
Failure hazard also rises with temperature here, so a hot machine is not only slower but likelier to lose a node — the coupling that turns a cooling incident into a resilience incident.
How often to checkpoint
Checkpoint too often and the machine spends its time writing state instead of computing. Checkpoint too rarely and every node failure throws away everything since the last one. There is an optimum in between, and it is not intuition — it falls out of the failure rate and the cost of a write. Each bar is a full nine-thousand-second simulation of the same workload at a different interval.
Run in the regime where the question is real: long jobs on a machine that fails often enough to interrupt them. With short jobs the answer is always "do not bother", which teaches nothing. This is the shape Young and Daly derive analytically — the optimum grows with the square root of the checkpoint cost and the mean time between failures.
HPC Cluster Simulator
Built around a custom conservative parallel-discrete-event core with seven coupled in-process engines — workload, scheduler, compute, fabric, storage, thermal and failure. Upstream simulators (Slurm, Astra-sim, Batsim/SimGrid) are used as offline validation oracles rather than live dependencies, so the runtime stays fast and self-contained. The fabric model is calibrated against Astra-sim; a 15-concept pedagogy catalog annotates every scheduling decision with what happened, why, and how it maps to real machines.
- CoreConservative PDES, 7 coupled engines
- SchedulingFCFS, EASY-backfill, fairshare, topology-aware
- FailuresMTBF Poisson, SDC, slow NIC, BMC hang, CRAC loss
- Front endReact Flow canvas + react-three-fiber rack theatre
The Immortal Daemon
An RL environment where the agent must keep a realistic cloud application alive while chaos batters it — and every reward is machine-verifiable, not judged by a model.
legal-financial-modernbert-150m
A 150M-parameter ModernBERT encoder trained from random init — no pretrained checkpoint, no distillation — on public legal and financial documents, then turned into a Matryoshka embedding model whose vectors truncate without retraining.
Populace
A world-scale persona simulation: grounded personas on a real globe living daily routines and reacting to events, with a language model authoring what each kind of person makes of the news and a Go engine doing the per-capita work.