GPT-2 Small
From Dontopedia, the open, paraconsistent wiki. (Last updated 2026-06-06.)
GPT-2 Small has 64 facts recorded in Dontopedia across 12 references, with 6 live disagreements.
64 facts·51 predicates·12 sources·6 in dispute
Mostly:trained on dataset(4), has params(3), rdf:type(3)
Maturity scale
raw canonical shape-checked rule-derived certifiedInbound mentions (13)
Other subjects in dontopedia point AT this entity as a value. These are inverse relationships — e.g. "X motherOf this subject" — and answer questions the forward facts can't. Grouped by predicate.
referencesModelReferences Model(2)
- Log Entry 2
ex:log-entry-2 - This Assessment
ex:this-assessment
comparesSubjectCompares Subject(1)
- Comparison Claim
ex:comparison-claim
comparesToCompares to(1)
- Comparative Statistic
ex:comparative-statistic
comparesUnfavorablyToGpt2InTokensCompares Unfavorably to Gpt2 in Tokens(1)
- Assessed Model
ex:assessed-model
correspondsToLevelCorresponds to Level(1)
- Bpb Scale 1 0
ex:bpb-scale-1-0
dataVolumeComparedToData Volume Compared to(1)
- Anchorkan
ex:anchorkan
determinesCoherenceThresholdDetermines Coherence Threshold(1)
- Training Data Quantity
ex:training-data-quantity
hasSignificantlyFewerTokensThanHas Significantly Fewer Tokens Than(1)
- Anchorkan
ex:anchorkan
involvesEntityInvolves Entity(1)
- Comparison Method Ideal
ex:comparison-method-ideal
isSufficientIs Sufficient(1)
- Seq Len 2048
ex:seq-len-2048
muchSmallerThanMuch Smaller Than(1)
- Config G8 S4
ex:config-g8-s4
usedForComparisonUsed for Comparison(1)
- Size Breakdown Table
ex:size-breakdown-table
Other facts (61)
The long tail: predicates that appear too rarely to warrant their own section. Filter or scroll to find a specific one. Each row links to its source.
61 facts
| Predicate | Value | Ref |
|---|---|---|
| Trained on Dataset | Webtext | [1] |
| Trained on Dataset | Webtext | [3] |
| Trained on Dataset | Webtext | [8] |
| Trained on Dataset | Webtext | [9] |
| Has Params | 124000000 | [1] |
| Has Params | 124M | [1] |
| Has Params | 124000000 | [5] |
| Rdf:type | Model Architecture | [8] |
| Rdf:type | Model | [9] |
| Rdf:type | Language Model | [11] |
| Achieved Ppl on | Penn Treebank | [1] |
| Achieved Ppl on | Wikitest 103 | [1] |
| Parameter Count | 124000000 | [8] |
| Parameter Count | 117000000 | [10] |
| Performs on Benchmark | Wikitext 103 Test | [8] |
| Performs on Benchmark | Penn Treebank | [8] |
| Ppl Measured on | Held Out Test Set | [1] |
| Differs From Anchor Kan in Params | 124m Vs 145m | [1] |
| Uses Attention | Softmax O L2 | [1] |
| Trained on Tokens | 8-9B | [1] |
| Has Context Length | 1024 | [1] |
| Tokens Seen | ~8-9B | [1] |
| References Benchmark Datasets | Wikitest 103 Penn Treebank | [1] |
| Has Num Heads | 12 | [2] |
| Has Total Params | ~124M | [2] |
| Has Dimensionality | 768 | [2] |
| References Known Architecture | True | [2] |
| Has Num Layers | 12 | [2] |
| Needed Tokens to Become Coherent | 40000000000 | [3] |
| Has Parameters | 117M | [4] |
| Benchmarked at Bpb | ~1.0 | [4] |
| Is Baseline Model | true | [5] |
| Tokens Per Byte | 0.29 | [5] |
| Byte Normalized Nats Per Byte | 0.29 | [5] |
| Byte Normalized Nats Per Token | 1 | [5] |
| Uses Bpe Tokenizer | true | [5] |
| Achieves Bpb | 0.97 | [5] |
| Byte Normalized Bpb | 0.42 | [5] |
| Has Params Value | 124000000 | [6] |
| Has Model Size | 500 MB | [6] |
| Is Baseline | Comparison | [6] |
| Referenced As Benchmark | Seq Len 2048 | [7] |
| Context Length | 1024 | [8] |
| Evaluated on | Held Out Test Set | [8] |
| Uses Attention Mechanism | Softmax O L2 | [8] |
| Tokens at Completion | 8000000000 | [8] |
| Required Training Token Count | 40000000000 | [9] |
| Achieved Coherence With Token Count | ~40B tokens | [9] |
| Has Performance Metric | 1 | [10] |
| Performance Metric Unit | BPB | [10] |
| Parameter Multiplier Relative | 14 | [10] |
| Training Step Multiplier Relative | 15 | [10] |
| Has Parameter Count | 124000000 | [11] |
| Has Bpb | 0.97 | [11] |
| Tested on | WebText test set | [11] |
| Has Nats Per Token | 1 | [11] |
| Has Tokens Per Byte | 0.29 | [11] |
| Has Nats Per Byte | 0.29 | [11] |
| Has Byte Normalized Bpb | 0.42 | [11] |
| Param Count | 124000000 | [12] |
| Size | 524288000 | [12] |
Timeline
Timeline axis is valid_time — when each source says the fact was true in the world, not when Dontopedia learned about it. Retracted rows are kept for provenance; coloured stripes indicate the context kind.
—
pplMeasuredOnblah/watt-activation/part-92
ex:held-out-test-set
—
achievedPPLOnblah/watt-activation/part-92
ex:penn-treebank
—
achievedPPLOnblah/watt-activation/part-92
ex:wikitest-103
—
differsFromAnchorKANInParamsblah/watt-activation/part-92
ex:124m-vs-145m
—
usesAttentionblah/watt-activation/part-92
ex:softmax-o-l2
—
trainedOnTokensblah/watt-activation/part-92
8-9B
—
hasContextLengthblah/watt-activation/part-92
1024
—
trainedOnDatasetblah/watt-activation/part-92
ex:webtext
—
tokensSeenblah/watt-activation/part-92
~8-9B
—
hasParamsblah/watt-activation/part-92
124000000
—
hasParamsblah/watt-activation/part-92
124M
—
referencesBenchmarkDatasetsblah/watt-activation/part-92
ex:wikitest-103-penn-treebank
—
hasNumHeadsblah/watt-activation/part-110
12
—
hasTotalParamsblah/watt-activation/part-110
~124M
—
hasDimensionalityblah/watt-activation/part-110
768
—
referencesKnownArchitectureblah/watt-activation/part-110
ex:true
—
hasNumLayersblah/watt-activation/part-110
12
—
neededTokensToBecomeCoherentblah/watt-activation/part-162
40000000000
—
trainedOnDatasetblah/watt-activation/part-162
ex:webtext
—
hasParametersblah/watt-activation/part-326
117M
—
benchmarkedAtBpbblah/watt-activation/part-326
~1.0
—
isBaselineModelblah/watt-activation/part-338
true
—
tokensPerByteblah/watt-activation/part-338
0.29
—
byteNormalizedNatsPerByteblah/watt-activation/part-338
0.29
—
byteNormalizedNatsPerTokenblah/watt-activation/part-338
1
—
usesBpeTokenizerblah/watt-activation/part-338
true
—
hasParamsblah/watt-activation/part-338
124000000
—
achievesBpbblah/watt-activation/part-338
0.97
—
byteNormalizedBpbblah/watt-activation/part-338
0.42
—
hasParamsValueblah/watt-activation/part-467
124000000
—
hasModelSizeblah/watt-activation/part-467
500 MB
—
isBaselineblah/watt-activation/part-467
ex:comparison
—
referencedAsBenchmarkblah/watt-activation/part-61
ex:seq-len-2048
—
typeblah/watt-activation/92
ex:ModelArchitecture
—
labelblah/watt-activation/92
GPT-2 Small
—
parameterCountblah/watt-activation/92
124000000
—
performsOnBenchmarkblah/watt-activation/92
ex:wikitext-103-test
—
performsOnBenchmarkblah/watt-activation/92
ex:penn-treebank
—
trainedOnDatasetblah/watt-activation/92
ex:webtext
—
contextLengthblah/watt-activation/92
1024
—
evaluatedOnblah/watt-activation/92
ex:held-out-test-set
—
usesAttentionMechanismblah/watt-activation/92
ex:softmax-o-l2
—
tokensAtCompletionblah/watt-activation/92
8000000000
—
typeblah/watt-activation/162
ex:Model
—
labelblah/watt-activation/162
GPT-2 small
—
requiredTrainingTokenCountblah/watt-activation/162
40000000000
—
trainedOnDatasetblah/watt-activation/162
ex:webtext
—
achievedCoherenceWithTokenCountblah/watt-activation/162
~40B tokens
—
hasPerformanceMetricblah/watt-activation/324
1
—
performanceMetricUnitblah/watt-activation/324
BPB
—
parameterCountblah/watt-activation/324
117000000
—
parameterMultiplierRelativeblah/watt-activation/324
14
—
trainingStepMultiplierRelativeblah/watt-activation/324
15
—
typeblah/watt-activation/336
ex:LanguageModel
—
hasParameterCountblah/watt-activation/336
124000000
—
hasBPBblah/watt-activation/336
0.97
—
testedOnblah/watt-activation/336
WebText test set
—
hasNatsPerTokenblah/watt-activation/336
1
—
hasTokensPerByteblah/watt-activation/336
0.29
—
hasNatsPerByteblah/watt-activation/336
0.29
—
hasByteNormalizedBPBblah/watt-activation/336
0.42
—
labelblah/watt-activation/465
GPT-2 small
—
paramCountblah/watt-activation/465
124000000
—
sizeblah/watt-activation/465
524288000
References (12)
12 references
ctx:discord/blah/watt-activation/part-92ctx:discord/blah/watt-activation/part-110ctx:discord/blah/watt-activation/part-162ctx:discord/blah/watt-activation/part-326ctx:discord/blah/watt-activation/part-338ctx:discord/blah/watt-activation/part-467ctx:discord/blah/watt-activation/part-61ctx:discord/blah/watt-activation/92- full textwatt-activation-92text/plain3 KB
doc:agent/watt-activation-92/a597b55a-3d12-478b-951d-f09c655a8870Show excerpt
[2026-03-08 01:46] xenonfun: ``` Direct comparison is tricky but here are the reference points: GPT-2 Small (124M) published benchmarks: - WikiText-103 test: 29.4 PPL - Penn Treebank: 65.9 PPL - Trained on ~8-9B tokens of WebText …
ctx:discord/blah/watt-activation/162- full textwatt-activation-162text/plain2 KB
doc:agent/watt-activation-162/60b4e03a-418d-44da-a803-c9585844c73eShow excerpt
[2026-03-09 18:40] xenonfun: ⏺ Here's my assessment: Speed: Excellent - 265 tok/s decode on M2 Ultra (idle), 14-27ms prefill. Very fast for 108M params. The compiled O(1) recurrent decode is working well. …
ctx:discord/blah/watt-activation/324- full textwatt-activation-324text/plain2 KB
doc:agent/watt-activation-324/5428b43d-e96d-4178-8924-6d7c75574aa4Show excerpt
[2026-03-15 03:48] xenonfun: ## Design (files: Screenshot_2026-03-14_at_11.48.03_PM.png) [2026-03-15 03:49] xenonfun: (files: Screenshot_2026-03-14_at_11.49.32_PM.png) [2026-03-15 04:03] xenonfun: You went from 2.39 → 1.572 today. You cros…
ctx:discord/blah/watt-activation/336- full textwatt-activation-336text/plain3 KB
doc:agent/watt-activation-336/04f318bf-4029-460c-b2ce-82900263e51eShow excerpt
[2026-03-15 15:12] xenonfun: ⏺ Step 2000 results (bs=512 seq=256 (its pointless to use higher bandwidth cuts off hurts quality of mappings beyond this)) so trying optimal run, high BS smooth out variance considerable. Eval (mixed_bytes v…
ctx:discord/blah/watt-activation/465- full textwatt-activation-465text/plain2 KB
doc:agent/watt-activation-465/946aff72-543d-4656-a5d0-4ba8931d0674Show excerpt
[2026-03-21 18:30] xenonfun: (files: Screenshot_2026-03-21_at_2.30.06_PM.png) [2026-03-21 18:33] xenonfun: I don't think we can really get much smaller of a model than this: ``` ⏺ Here's the size breakdown for all configs we're testing: …
See also
Keep researching
Missing something or suspicious of what's here? Kick off a research session — a Claude agent will investigate, cite its sources, and file new facts into a dedicated context you can review before accepting into the shared view.