Passes 5

Sanskrit DoneKeep

passes_p5_d60m

d60m (68.9M parameters) on plus_all v2 subset (155M tokens), trained on RTX 3060 12GB (Home GPU). Started 7 Oct 2026, ended 7 Oct 2026.

Hypothesis

A fifth pass still improves held-out scores at F10's data-to-size ratio.

What we learned

Best score but small step: ex-Gita 0.6574, 1.43% below three passes (clean_v1 -1.34%); the fifth pass adds 0.57% over four, at the noise bar, while Gita memorisation keeps rising (-4.4%) and Vedic stays flat. Gains shrink about 0.6x per pass.

Scores

Bits per byte, lower is better; change against the parent run, Passes 3.

ex-Gītā (headline)
0.6574
bits per byte
−1.4%
ex-Gītā, clean_v1
0.6799
bits per byte
−1.3%
Pooled, all five sets
0.6493
bits per byte
−1.5%
Validation split
0.6477
bits per byte
−1.6%
Bits per byte on each test set
Test setPasses 5Passes 3 (parent)Change
DCS gold (classical)0.66640.6754−1.3%
Bhagavad-gītā (memorisation)0.4360.456−4.4%
Out of domain0.66140.671−1.4%
Prose0.62450.6353−1.7%
Vedic (Ṛgveda)0.89240.8913+0.1%

Curves

Training loss

Cross-entropy per token, by step. The first few percent of the run, far higher, run off the top; hover or the table has every value.

Held-out bits per byte

The validation split, evaluated during training, by step. Lower is better.

Throughput

Tokens per second, by step.

Model

Preset
d60m
Parameters
68,924,160
Outside embeddings
56,636,160
Layers · heads · width
8 · 12 · 768
Tokenizer
SLP1 unigram 8k

Muon + AdamW, rope, qk_norm, relu^2, untied head, weight decay (F10 recipe)

Data

Slice
plus_all v2 subset (155M tokens)
Words
—
Training tokens
155,261,295
Passes
5 passes
Tokens seen
776,306,688

Compute

GPU
RTX 3060 12GB
Where
Home GPU
Steps
15,794 / 15,794
GPU hours
4.8
Cost
—
Spot restarts
—

Lineage

homed60m