73

Source document

[2026-02-22 15:56] xenonfun: ``` Yeah, this confirms what we saw earlier — MoE gets better val loss but significantly worse generation quality. The numbers tell the story: ┌────────────┬────────────────┬─────────────┐ │ │ ds2 (dense) │ MoE (fresh) │ ├────────────┼────────────────┼─────────────┤ │ Best val │ 3.82 │ 3.498 │ ├────────────┼────────────────┼─────────────┤ │ Best PPL │ 45.8 │ 33.1 │ ├────────────┼────────────────┼─────────────┤ │ Generation │ Coherent prose │ Gibberish │ └────────────┴────────────────┴─────────────┘ The val loss paradox: MoE is better at predicting the next token across the training distribution, but worse at generating text autoregressively. The core problem is compounding routing errors — each generated token feeds back in, and if the router makes a slightly wrong expert assignment, the next token's representation is off, which makes the next routing decision worse, and so on. Dense models don't have this failure mode since all parameters are used for every token. A few things that could help MoE generation quality: 1. Lower temperature (0.3-0.5) — reduce sampling noise so the router sees cleaner inputs 2. Expert dropout during training — force redundancy so no single routing mistake is catastrophic 3. Larger model — 128-dim may be too small for 4 experts to specialize meaningfully 4. Shared expert — one expert always active (like DeepSeek-MoE), guaranteeing a baseline For now, ds2 remains the better model for actual text generation despite the higher val loss. The MoE architecture needs more capacity to shine. ``` [2026-02-22 21:33] ajaxdavis: have you managed to get your gpu utilization humming at a decent level always [2026-02-22 21:42] xenonfun: Yeah after it's final optimizations it was never below 98% on GPU, only very brief blip on the eval every 25 iterations. [2026-02-26 01:54] ajaxdavis: more feature orientated <jmaaican voice> (files: safier-jamaican-promo-final.mp4) [2026-02-27 17:42] ajaxdavis: https://mates.symploke.dev/rsafier [2026-02-27 19:51] xenonfun: Kotlin expert I am, huh? [2026-02-28 00:20] ajaxdavis: ahh probably synthesized that you starreda kotlin repo, will push a fix [2026-04-24 15:26] xenonfun: well trying to max this claude had it try to make a days worth of work and split it across subagents. codex is doing review on the prs automagicly (files: Screenshot_2026-04-24_at_11.25.44_AM.png) [2026-04-24 15:40] xenonfun: (files: Screenshot_2026-04-24_at_11.40.45_AM.png)