f11_slp1_uni8k_d700m_plus_all_v2_4x

d700m (704.1M parameters) on plus_all v2 slice (489.9M words, 4 passes), trained on L40S 48GB (AWS). Started 8 Oct 2026, ended 11 Oct 2026.

Hypothesis

4 passes instead of 3 improves ex-Gita (~0.9% predicted by the passes proxy).

What we learned

Winner: ex-Gita 0.5262 vs F10's 0.5307 (-0.85%, as the passes proxy predicted), clean_v1 -0.61%, Vedic flat, verse recall 2.2% vs 0.7% (mostly Gita). Released as sansar-700m v0.2.0.

Scores

Bits per byte, lower is better; change against the parent run, F10.

ex-Gītā (headline)
0.5262
bits per byte
−0.8%
ex-Gītā, clean_v1
0.5507
bits per byte
−0.6%
Pooled, all five sets
0.5094
bits per byte
−0.9%
Validation split
0.4389
bits per byte
−2.2%
Bits per byte on each test set
Test setF11F10 (parent)Change
DCS gold (classical)0.54540.5513−1.1%
Bhagavad-gītā (memorisation)0.06830.0758−9.9%
Out of domain0.5240.5286−0.9%
Prose0.51440.5186−0.8%
Vedic (Ṛgveda)0.66790.6671+0.1%

Curves

Training loss

Cross-entropy per token, by step. The first few percent of the run, far higher, run off the top; hover or the table has every value.

Held-out bits per byte

The validation split, evaluated during training, by step. Lower is better.

Throughput

Tokens per second, by step.

Model

Preset
d700m
Parameters
704,128,512
Outside embeddings
679,552,512
Layers · heads · width
24 · 12 · 1,536
Tokenizer
SLP1 unigram 8k

Muon + AdamW, rope, qk_norm, relu^2, untied head, weight decay

Data

Slice
plus_all v2 slice
Words
489,850,000
Training tokens
1,586,326,034
Passes
4 passes
Tokens seen
6,345,326,592

Compute

GPU
L40S 48GB
Where
AWS
Steps
129,096 / 129,096
GPU hours
71.48
Cost
$133.97
Spot restarts
0

Lineage

awsl40sd700m