context

Part 74

ctx:discord/blah/safiersemantics/part-74

kind: custommode: permissive136 statements62 distinct subjects

No external document is attached to this context. (Many contexts are pure organisational labels.)

Facts in this context

Grouped by subject. Each subject links to its full article.

Xenonfun20 facts`ex:xenonfun`

announcesRestart	Training
believes	Rng Core Problem
criticizesPreviousFp16	true
estimatesPotential	almost a second
evaluatesNegatively	Memory Copying Operations
evaluatesNegatively	Quality
evaluatesPositively	Loss
expectsImprovementFromRestart	true
optimizing	Training Loop
postedAttachment	Screenshot 2026 02 21 at 6 43 42 Am Png
postedMessageAt	2026-02-21 19:25
postedMessageAt	2026-02-21 13:58
postedMessageAt	2026-02-21 13:56
postedMessageAt	2026-02-21 14:23
postedMessageAt	2026-02-22 05:36
postedMessageAt	2026-02-21 23:26
postedMessageAt	2026-02-21 11:47
reportsProgress	Training Iter 2900
suggests	Persistent Fp16
suggests	Eliminate Cpu Gpu Sync

Training Iter 29009 facts`ex:training-iter-2900`

hasBestVal	4.3195
hasElapsedSeconds	183
hasEtaSeconds	27034
hasLoss	4.576
hasLr	0.0001
hasMsPerIter	2356
hasPatience	3/30
hasPpl	78.8
hasValLoss	4.3665

Text8 facts`ex:text`

assumesAudienceKnowledgeOf	Moe Training
presupposesExistenceOf	Training Model
presupposesExistenceOf	Router
presupposesExistenceOf	Moe Architecture
presupposesExistenceOf	Gpu
references	Candle
references	Graph
references	Plugin

Proposed Fp167 facts`ex:proposed-fp16`

castsBackFor	Softmax
castsBackFor	Norms
castsOncePerLayerStart	true
castsPerLayer	4
isMoreInvasive	true
keepsIntermediatesInF16	Matmul Chains
realizesBenefitOf	Fp16

Training Loop7 facts`ex:training-loop`

achievedMsPerIter	3108
achievedMsPerIter	2816
achievedMsPerIter	3293
potentialMsPerIterReduction	1000
reduced	Memory Copying Operations
reduced	Stalls
stillStalling	true

Router Probs to Vec15 facts`ex:router-probs-to-vec1`

copiesDataTo	Cpu
performsTopKSortingOn	Cpu
precedes	Top K Sorting
stallsGpuWaitingFor	Router Matmul
uploadsIndexTensorsTo	Gpu

Lisamegawatts4 facts`ex:lisamegawatts`

advises	Consult Ds ML Team
postedMessageAt	2026-02-21 14:46
providedHotTip	Grab Ds ML Team
recommends	Grab Ds ML Team

Ms Per Iter3 facts`ex:ms-per-iter`

decreasedFrom	3293
decreasedTo	3108
decreasedTo	2816

Training3 facts`ex:training`

assumesGoalOf	Low Loss
restartingFrom	0
usesLrScheduler	true

Ajaxdavis2 facts`ex:ajaxdavis`

asked	Vocab Size Question
postedMessageAt	2026-02-22 01:55

Cpu Gpu Sync2 facts`ex:cpu-gpu-sync`

identifiedAs	Biggest Bottleneck
isBiggestBottleneck	true

Extra Gpu Kernels2 facts`ex:extra-gpu-kernels`

equals	360
resultFrom	Dtype Casts Per Matmul

Fixed Seed2 facts`ex:fixed-seed`

equals	1337
implicates	Non Random Windows

Gpu Utilization Dips2 facts`ex:gpu-utilization-dips`

causedBy	Cpu Gpu Sync
correspondTo	Router Probs to Vec1

Matmul Count2 facts`ex:matmul-count`

approximates	120
isApproximate	120

Mm Function2 facts`ex:mm-function`

appliesTo	Matmul
performsDtypeCasts	3

Model2 facts`ex:model`

hasBlockSize	256
hasContextWindow	256

Previous Fp162 facts`ex:previous-fp16`

castsPerLayer	30
failed	true

Quality2 facts`ex:quality`

prioritizedOverLoss	true
wasTrash	true

Quality Bad2 facts`ex:quality-bad`

causedBy	Reason 1 Quality Bad
causedBy	Reason 2 Quality Bad

Reason 2 Quality Bad2 facts`ex:reason-2-quality-bad`

causedBy	Fixed Seed 1337
retrainingSameMaterialOnResumes	true

Rng2 facts`ex:rng`

impacts	Sampling
wasCoreProblem	true

Rng Seeding2 facts`ex:rng-seeding`

fixedPreviously	1337
nowDynamicWithIter	true

Stalls2 facts`ex:stalls`

causedBy	Cpu Gpu Sync
remainSignificant	true

Top K Sorting2 facts`ex:top-k-sorting`

cpuBoundCurrently	true
precedes	Index Upload

Val Loss2 facts`ex:val-loss`

improvedRelativeTo	Prior Non Moe Val
lowerThan	Prior Non Moe

Backward Pass1 fact`ex:backward-pass`

hasStalls

Block Size1 fact`ex:block-size`

equalsContextWindow

true

Candle Tensor Ops1 fact`ex:candle-tensor-ops`

enablesGpuTopk

true

Casts Per Layer1 fact`ex:casts-per-layer`

isApproximate

Current Quality1 fact`ex:current-quality`

isBad

true

Current Seeding1 fact`ex:current-seeding`

usesIterOnRng

true

Data Science ML Team1 fact`ex:data-science-ml-team`

sourcedFrom

Plugin

Eta1 fact`ex:eta`

calculatedFrom

Current Speed

Every Layer1 fact`ex:every-layer`

performs

Router Probs to Vec1

Forward Pass1 fact`ex:forward-pass`

hasStalls

Fp161 fact`ex:fp16`

essentialFor

Performance Benefit

Grab Ds ML Team1 fact`ex:grab-ds-ml-team`

involvesAsking

Data Science ML Team

Intermediates1 fact`ex:intermediates`

maintainedIn

F16

Layers1 fact`ex:layers`

haveStartCast

Proposed Fp16

Lisamegawatts Message1 fact`ex:lisamegawatts-message`

isReplyTo

Training Iter 2900 Message

Loss1 fact`ex:loss`

wasGood

true

Lr1 fact`ex:lr`

multipliedBy

Matmuls1 fact`ex:matmuls`

chainedInLayers

true

Model Vocab1 fact`ex:model-vocab`

hasSize

2000

Moe1 fact`ex:moe`

contrastedWith

Non Moe

Number11 fact`ex:number1`

hasReadLess

true

Optimizations1 fact`ex:optimizations`

prioritizes

Speed

Patience1 fact`ex:patience`

atLevel

Previous Fp16 Failure1 fact`ex:previous-fp16-failure`

causedBy

Extra Gpu Kernels

Proposed Topk1 fact`ex:proposed-topk`

performedOnGpu

Candle Tensor Ops

Quality Trash1 fact`ex:quality-trash`

implicates

Insufficient Sampling

Reason 1 Quality Bad1 fact`ex:reason-1-quality-bad`

number1HasNotReadThruAsMuch

true

Restarting1 fact`ex:restarting`

enablesFairComparison

Prior Runs

Resumes1 fact`ex:resumes`

involvesRetrainingSameMaterial

true

Router Matmul1 fact`ex:router-matmul`

mustFinishBefore

Router Probs to Vec1

Sampling1 fact`ex:sampling`

requiresProperRng

true

Stalls Per Forward Backward1 fact`ex:stalls-per-forward-backward`

equals

Training Loop Optimizations1 fact`ex:training-loop-optimizations`

target

Stalls

Val Loss 4 36651 fact`ex:val-loss-4-3665`

worseThan

Best Val 4 3195

Vocab1 fact`ex:vocab`

sameForAll

true

Windows1 fact`ex:windows`

randomlyAsIntended

true

Part 74

Facts in this context

Xenonfun20 factsex:xenonfun

Training Iter 29009 factsex:training-iter-2900

Text8 factsex:text

Proposed Fp167 factsex:proposed-fp16

Training Loop7 factsex:training-loop

Router Probs to Vec15 factsex:router-probs-to-vec1

Lisamegawatts4 factsex:lisamegawatts

Ms Per Iter3 factsex:ms-per-iter

Training3 factsex:training

Ajaxdavis2 factsex:ajaxdavis

Cpu Gpu Sync2 factsex:cpu-gpu-sync

Extra Gpu Kernels2 factsex:extra-gpu-kernels

Fixed Seed2 factsex:fixed-seed

Gpu Utilization Dips2 factsex:gpu-utilization-dips

Matmul Count2 factsex:matmul-count

Mm Function2 factsex:mm-function

Model2 factsex:model

Previous Fp162 factsex:previous-fp16

Quality2 factsex:quality

Quality Bad2 factsex:quality-bad

Reason 2 Quality Bad2 factsex:reason-2-quality-bad

Rng2 factsex:rng

Rng Seeding2 factsex:rng-seeding

Stalls2 factsex:stalls

Top K Sorting2 factsex:top-k-sorting

Val Loss2 factsex:val-loss

Backward Pass1 factex:backward-pass

Block Size1 factex:block-size

Candle Tensor Ops1 factex:candle-tensor-ops

Casts Per Layer1 factex:casts-per-layer

Current Quality1 factex:current-quality

Current Seeding1 factex:current-seeding

Data Science ML Team1 factex:data-science-ml-team

Eta1 factex:eta

Every Layer1 factex:every-layer

Forward Pass1 factex:forward-pass

Fp161 factex:fp16

Grab Ds ML Team1 factex:grab-ds-ml-team

Intermediates1 factex:intermediates

Layers1 factex:layers

Lisamegawatts Message1 factex:lisamegawatts-message

Loss1 factex:loss

Lr1 factex:lr

Matmuls1 factex:matmuls

Model Vocab1 factex:model-vocab

Moe1 factex:moe

Number11 factex:number1

Optimizations1 factex:optimizations

Patience1 factex:patience

Previous Fp16 Failure1 factex:previous-fp16-failure

Proposed Topk1 factex:proposed-topk

Quality Trash1 factex:quality-trash

Reason 1 Quality Bad1 factex:reason-1-quality-bad

Restarting1 factex:restarting

Resumes1 factex:resumes

Router Matmul1 factex:router-matmul

Sampling1 factex:sampling

Stalls Per Forward Backward1 factex:stalls-per-forward-backward

Training Loop Optimizations1 factex:training-loop-optimizations

Val Loss 4 36651 factex:val-loss-4-3665

Vocab1 factex:vocab

Windows1 factex:windows

Xenonfun20 facts`ex:xenonfun`

Training Iter 29009 facts`ex:training-iter-2900`

Text8 facts`ex:text`

Proposed Fp167 facts`ex:proposed-fp16`

Training Loop7 facts`ex:training-loop`

Router Probs to Vec15 facts`ex:router-probs-to-vec1`

Lisamegawatts4 facts`ex:lisamegawatts`

Ms Per Iter3 facts`ex:ms-per-iter`

Training3 facts`ex:training`

Ajaxdavis2 facts`ex:ajaxdavis`

Cpu Gpu Sync2 facts`ex:cpu-gpu-sync`

Extra Gpu Kernels2 facts`ex:extra-gpu-kernels`

Fixed Seed2 facts`ex:fixed-seed`

Gpu Utilization Dips2 facts`ex:gpu-utilization-dips`

Matmul Count2 facts`ex:matmul-count`

Mm Function2 facts`ex:mm-function`

Model2 facts`ex:model`

Previous Fp162 facts`ex:previous-fp16`

Quality2 facts`ex:quality`

Quality Bad2 facts`ex:quality-bad`

Reason 2 Quality Bad2 facts`ex:reason-2-quality-bad`

Rng2 facts`ex:rng`

Rng Seeding2 facts`ex:rng-seeding`

Stalls2 facts`ex:stalls`

Top K Sorting2 facts`ex:top-k-sorting`

Val Loss2 facts`ex:val-loss`

Backward Pass1 fact`ex:backward-pass`

Block Size1 fact`ex:block-size`

Candle Tensor Ops1 fact`ex:candle-tensor-ops`

Casts Per Layer1 fact`ex:casts-per-layer`

Current Quality1 fact`ex:current-quality`

Current Seeding1 fact`ex:current-seeding`

Data Science ML Team1 fact`ex:data-science-ml-team`

Eta1 fact`ex:eta`

Every Layer1 fact`ex:every-layer`

Forward Pass1 fact`ex:forward-pass`

Fp161 fact`ex:fp16`

Grab Ds ML Team1 fact`ex:grab-ds-ml-team`

Intermediates1 fact`ex:intermediates`

Layers1 fact`ex:layers`

Lisamegawatts Message1 fact`ex:lisamegawatts-message`

Loss1 fact`ex:loss`

Lr1 fact`ex:lr`

Matmuls1 fact`ex:matmuls`

Model Vocab1 fact`ex:model-vocab`

Moe1 fact`ex:moe`

Number11 fact`ex:number1`

Optimizations1 fact`ex:optimizations`

Patience1 fact`ex:patience`

Previous Fp16 Failure1 fact`ex:previous-fp16-failure`

Proposed Topk1 fact`ex:proposed-topk`

Quality Trash1 fact`ex:quality-trash`

Reason 1 Quality Bad1 fact`ex:reason-1-quality-bad`

Restarting1 fact`ex:restarting`

Resumes1 fact`ex:resumes`

Router Matmul1 fact`ex:router-matmul`

Sampling1 fact`ex:sampling`

Stalls Per Forward Backward1 fact`ex:stalls-per-forward-backward`

Training Loop Optimizations1 fact`ex:training-loop-optimizations`

Val Loss 4 36651 fact`ex:val-loss-4-3665`

Vocab1 fact`ex:vocab`

Windows1 fact`ex:windows`