45

Source document

[2026-03-02 23:55] xenonfun: Excellent results. The semi-fused approach is the winner: - Semi-fused: 130 tok/s (7.69 ms/tok, 311 GB/s effective bandwidth) - Non-fused: 93 tok/s (10.78 ms/tok, 222 GB/s) - Fused single-TG: 11 tok/s (dead last — can't use all GPU cores) 1.40x speedup from semi-fused over non-fused by halving dispatches (332 → 155) while keeping multi-core parallelism. The semi-fused achieves 311 GB/s effective bandwidth on the M3 Ultra (theoretical max ~800 GB/s). Key insights at 1B scale: - Single-threadgroup fusion kills performance — only uses 1 of 76 GPU cores → 27 GB/s - Multi-threadgroup semi-fusion saves ~3ms/tok in dispatch overhead while keeping full parallelism - Merged QKV (3 matvecs in 1 dispatch) and merged W1/W3+SiLU are the big wins - Memory bandwidth is the bottleneck at 311 GB/s (39% of theoretical 800 GB/s) Want me to try ANE prefill with --ane to see the full pipeline, or push further on the GPU decode optimization? [2026-03-02 23:57] xenonfun: yeah and that number is about what I got a threading test to do you cannot get past about 430GB overall (R/W) particularlly because you cannot pin cores in MacOS, so if you have loops that fit in L1/L2 the AMD stuff has quite a bit less latency. but the M-series latency across boundry is practically flat as a result, while on AMD you see a massive jump once your ouside of your L2 also the boundry of the chiplet. and only has ~32GB/s max write on DDR4 [2026-03-03 00:11] xenonfun: (files: Screenshot_2026-03-02_at_7.11.17_PM.png) [2026-03-03 00:13] xenonfun: at this point gonna have to have it build one for to run one of our wacker models to compare to pytouch/metal version. (files: Screenshot_2026-03-02_at_7.13.35_PM.png) [2026-03-03 00:18] ajaxdavis: do it [2026-03-03 00:33] xenonfun: its gonna try on a wacky one `Estimated ANE impact: ~1.5ms vs ~20ms PyTorch CPU → 13x speedup per the benchmark extrapolation in the ANE doc.` (files: Screenshot_2026-03-02_at_7.33.14_PM.png) [2026-03-03 00:35] xenonfun: oh that is new it realized clang would use useful and asked if I wanted to add the LSP