27

Source document

[2026-02-17 18:23] xenonfun: yeah is with bpe, 7.5M model, with ~40MB of data on that (Gutenburg free library) I am going to do full training that should be enouge sample data now: ``` It's running! 55.7M tokens — so 1 epoch = 50.1M / 4096 = 12,237 iters. Your 12,000 target is almost exactly right. The key numbers: 109MB text, 55.7M tokens, tokens.bin cached at 212.5MB. ETA ~143 minutes (~2.4 hours). This is the big run — one full epoch with 2.6× more data than before. Should push well below ppl 40. `` (files: Screenshot_2026-02-17_at_1.21.52_PM.png) [2026-02-17 18:23] xenonfun: (files: Screenshot_2026-02-17_at_1.23.52_PM.png) [2026-02-17 18:34] ajaxdavis: why did i see metalgpu in your logs but you are still running stuff on your cpu you said? [2026-02-17 19:01] xenonfun: so that is 2K into this 12K run with the big dataset. so BPE runs on CPU, tokenization also still CPU, both single threaded. The training is using metal GPU. For earilier iteration it was using SIMD acceleration stuff but not using GPU cores fully, Candle framework is handling most of the training details to get that. its got a while to cook still: ```Iter 2225 | Loss: 3.7794 | Val: 3.8677 (ppl 47.8) | LR: 0.000030 | Best val: 3.7910 | 869ms/iter | 177s elapsed | ETA 8494s | Patience: 3/20``` (files: Screenshot_2026-02-17_at_1.59.08_PM.png) [2026-02-17 19:04] xenonfun: there are probably better library options but that one was picked because it was mostly just drop in replacement for its write-your-own autograd attempt. it wanted to just write a custom shader to do it. [2026-02-17 19:06] ajaxdavis: (files: Screenshot_from_2026-02-18_05-04-24.png) [2026-02-17 19:06] ajaxdavis: 5 hour run with 250k words from novels.txt [2026-02-17 19:10] xenonfun: yeah I couldn't figure out exactly what your parms were was tryin to apple-to-apples but it couldn't backcalculate it well. It also pretty much told me I wasn't going to get much better than I had achieved, perhaps 15-20% more on my tiny attempt.