Numbat Forensics: how to tell whether a checkpoint was trained

The claim and the question

A public repository presented itself as a mid-pretraining checkpoint at step 1,830,000 of a ~1.96M step run, roughly 14T parameters, 772 shards, 7.72 TB, quantised to NVFP4. Those numbers come from the uploader. The audit asked a narrower question: are the parameters genuinely trained, or can a deterministic program regenerate them?

Method: range requests, no downloads, no code execution

The audit never downloaded a shard and never executed repository code. It read safetensors headers with HTTP range requests, pinned the source revision in every script, and rebuilt the forward pass from config independently.

Findings

633 norm and convolution tensors, covering 11,374,080 values, regenerate bit-for-bit from SHA256(tensor_name) with zero fitted parameters. 373,620,632 signs across 12 expert matrices match the same name-seeded stream, again with zero fitted parameters.

The token embedding table is the exception. It carries real learned semantic structure and is the only component in the audited slice that looks trained.

The artifact itself is a truncated 1-of-114-layer model: 4.73B parameters, 21 tensors, 9.46 GB in BF16, built from the real tensor bytes so the reconstruction can be reproduced.

What this does not prove

• Parameter authenticity and provenance are different questions; only authenticity was measured.
• The artifact keeps layer 0 only — no MLA, no MoE, no MTP, no deep residual stream. It cannot produce coherent text.
• One per-tensor scalar S has no closed form yet (it ranges 106.2–143.7 across tensors), so reconstruction is not fully parameter-free.
• The 6.85T and ~14T parameter figures both come from the source repository; the discrepancy is unexplained.

The honest summary: the audited slice is reconstructible from names, the embedding table is not, and no claim is made about who uploaded the checkpoint.

Why this matters beyond one repo

Model cards are claims. Weight audits turn some claims into measurements. As checkpoints get larger and provenance gets harder to verify, cheap deterministic tests like this one become part of how the ecosystem checks itself — before a 7.7 TB download, not after.

The full methodology, per-test numbers and reproduction scripts are on Hugging Face.