72

Source document

[2026-02-21 13:58] xenonfun: ``` 1. Persistent fp16 (not cast-per-matmul) Our fp16 failed because mm() does 3 dtype casts per matmul × ~120 matmuls = 360 extra GPU kernels. Instead: cast once at the start of each layer, keep intermediates in f16 through chains of matmuls, cast back only for norms/softmax. ~4 casts per layer instead of ~30. More invasive but would actually realize the fp16 benefit. 2. Eliminate the CPU↔GPU sync (biggest single bottleneck) Every layer does router_probs.to_vec1() — this stalls the GPU waiting for the router matmul to finish, copies data to CPU, does top-K sorting on CPU, then uploads index tensors back. Those GPU utilization dips in your graph are exactly this. 12 stalls per forward + backward. Could do the top-K on GPU using Candle's tensor ops instead. ``` [2026-02-21 14:23] xenonfun: Iter 2900 | Loss: 4.5760 | Val: 4.3665 (ppl 78.8) | LR: 0.000100 | Best val: 4.3195 | **2356ms/iter** | 183s elapsed | ETA 27034s | Pat: 3/30 LRx0 [2026-02-21 14:46] lisamegawatts: Hot tip, grab the data science/ml team from the plugin and ask them if there any problems to fix or optimizations to make before training [2026-02-21 19:25] xenonfun: Val is lower then prior non MoE, but quality bad, #1 still hasn't read thru as much, #2 retraining same material on resumes because it had fixed seed of 1337, now seeds with iter on RNG so windows randomly as intended. [2026-02-21 23:26] xenonfun: yeah loss was good quality was trash, but think RNG was core problem, restarting it from 0 so it gets proper sampling to be able to compare. [2026-02-22 01:55] ajaxdavis: whats your vocab size [2026-02-22 05:36] xenonfun: 2k all same, 256 block size/context window