89

Source document

[2026-03-07 22:07] xenonfun: we are adding 2 more tokens to it just to make things easy for our code and not have to refactor. [2026-03-07 22:08] lisamegawatts: CustomModelMLXConversion/fineweb_edu/ [2026-03-07 22:08] lisamegawatts: (files: Screenshot_2026-03-07_22-05-59.png) [2026-03-07 22:09] lisamegawatts: check out pipeline maybe too, may need to tell it make sure we capture my live runs [2026-03-07 22:09] lisamegawatts: should load from wandb and have replays now [2026-03-07 22:10] xenonfun: 624MB of clean educational text — much richer than 126MB philosophy corpus. Now let me launch the 50K vocab training on FineWeb-Edu: [2026-03-07 22:10] xenonfun: ``` Model: AnchorKAN-GPT | 145.4M params d=768, layers=12, heads=12, anchors=64, hidden=3072, L=4096 chunk_size=None, vocab=50259 Tokenizing corpus (first run - this may take a few minutes)... Source: /Users/ms/MS/CustomModelMLXConversion/fineweb_edu/train.txt Processed 10000 lines, 402,773 tokens Processed 20000 lines, 832,593 tokens Processed 30000 lines, 1,264,598 tokens Processed 40000 lines, 1,689,781 tokens Processed 50000 lines, 2,100,511 tokens Processed 60000 lines, 2,511,539 tokens Processed 70000 lines, 2,920,415 tokens Processed 80000 lines, 3,336,199 tokens Processed 90000 lines, 3,746,896 tokens Processed 100000 lines, 4,167,040 tokens Processed 110000 lines, 4,623,727 tokens Processed 120000 lines, 5,036,864 token ``` be a bit 3M lines [2026-03-07 22:32] xenonfun: and we are running on Fineweb (files: Screenshot_2026-03-07_at_5.32.06_PM.png, Screenshot_2026-03-07_at_5.31.52_PM.png) [2026-03-07 22:34] xenonfun: Compared to 8K vocab at iter 500: loss=5.43, PPL=228. The higher initial PPL is expected — the output space is 6.3× larger (50K vs 8K vocab), so random-init cross-entropy starts much higher (ln(50259) ≈ 10.8 vs ln(8000) ≈ 9.0). Throughput is 7.8K tok/s vs 9.1K — about 14% slower, as predicted (larger logits matmul). Memory 23.5GB vs 20.7GB — 13% more. Both right in line with estimates. Yes — after this 10K iter comparison run, we can continue training to a full epoch. At 1.9 it/s and 133M tokens: - Full epoch = 133M / 4096 = ~32,470 iters → ~4.7 hours - The current 10K iters covers ~31% of the corpus - We can resume from the iter 10K checkpoint and run another 22.5K iters to complete the epoch - Or extend to 2+ epochs (~9.5 hours) for better convergence The checkpoints save every 2K iters, so we can always resume. [2026-03-07 22:35] xenonfun: Almost done — 2.79M/3.2M lines, 115.8M tokens. And interesting: 50K vocab encodes the same 624MB text in ~116M tokens vs 199M tokens with 8K vocab. That's a 1.7x compression ratio — each token carries 70% more information. Throughput is 7.8K tok/s vs 9.1K — about 14% slower, as predicted (larger logits matmul). Memory 23.5GB vs 20.7GB — 13% more. Both right in line with estimates. [2026-03-07 22:44] xenonfun: (files: Screenshot_2026-03-07_at_5.44.31_PM.png)