Dontopedia

Parameterized Activations

From Dontopedia, the open, paraconsistent wiki. (Last updated 2026-06-06.)

Parameterized Activations is parameterize a family that contains both as special cases.

24 facts·19 predicates·2 sources·2 in dispute

Mostly:similar to(4), has special case(3), parameterizes family(1)

Maturity scale raw canonical shape-checked rule-derived certified

Inbound mentions (2)

Other subjects in dontopedia point AT this entity as a value. These are inverse relationships — e.g. "X motherOf this subject" — and answer questions the forward facts can't. Grouped by predicate.

describesApproachDescribes Approach(1)

listsApproachesLists Approaches(1)

Other facts (24)

The long tail: predicates that appear too rarely to warrant their own section. Filter or scroll to find a specific one. Each row links to its source.

24 facts
PredicateValueRef
Similar toPau[1]
Similar toAcon[1]
Similar toAcon[2]
Similar toPau[2]
Has Special CaseSilu Activation[2]
Has Special CaseGelu Activation[2]
Has Special CaseRelu Activation[2]
Parameterizes FamilyActivation Function Family[1]
Defines Activationactivation(x; α, β) = x * σ(α * x + β)[1]
Has Value Levelhigh value[1]
Enables Gradient Descent OverActivation Shape[1]
Uses Learnable ParametersAlpha[1]
Has Complexity Leveleasy[1]
Transforms Evolutionary Search tocontinuous gradient descent over activation shape[1]
Complexity Leveleasy[2]
Value Levelhigh[2]
Descriptionparameterize a family that contains both as special cases[2]
Has Formulaactivation(x; α, β) = x * σ(α * x + β)[2]
Training Mechanismlearnable parameters per layer, trained via backprop alongside the weights[2]
Implementation ContextAlpha[2]
Requires Operationone new autograd op[2]
Operation Outputx * sigmoid(α*x + β)[2]
Has Featuregradients for all three[2]
Evolvesnaturally during training[2]

Timeline

Timeline axis is valid_time — when each source says the fact was true in the world, not when Dontopedia learned about it. Retracted rows are kept for provenance; coloured stripes indicate the context kind.

24 undated
—
parameterizesFamilyblah/training-and-evals/part-20
ex:activation-function-family
—
definesActivationblah/training-and-evals/part-20
activation(x; α, β) = x * σ(α * x + β)
—
hasValueLevelblah/training-and-evals/part-20
high value
—
enablesGradientDescentOverblah/training-and-evals/part-20
ex:activation-shape
—
usesLearnableParametersblah/training-and-evals/part-20
ex:alpha
—
hasComplexityLevelblah/training-and-evals/part-20
easy
—
transformsEvolutionarySearchToblah/training-and-evals/part-20
continuous gradient descent over activation shape
—
similarToblah/training-and-evals/part-20
ex:pau
—
similarToblah/training-and-evals/part-20
ex:acon
—
complexityLevelblah/training-and-evals/20
easy
—
valueLevelblah/training-and-evals/20
high
—
descriptionblah/training-and-evals/20
parameterize a family that contains both as special cases
—
hasFormulablah/training-and-evals/20
activation(x; α, β) = x * σ(α * x + β)
—
hasSpecialCaseblah/training-and-evals/20
ex:silu-activation
—
hasSpecialCaseblah/training-and-evals/20
ex:gelu-activation
—
hasSpecialCaseblah/training-and-evals/20
ex:relu-activation
—
trainingMechanismblah/training-and-evals/20
learnable parameters per layer, trained via backprop alongside the weights
—
similarToblah/training-and-evals/20
ex:acon
—
similarToblah/training-and-evals/20
ex:pau
—
implementationContextblah/training-and-evals/20
Alpha
—
requiresOperationblah/training-and-evals/20
one new autograd op
—
operationOutputblah/training-and-evals/20
x * sigmoid(α*x + β)
—
hasFeatureblah/training-and-evals/20
gradients for all three
—
evolvesblah/training-and-evals/20
naturally during training

References (2)

2 references
  1. [1]Part 209 facts
    ctx:discord/blah/training-and-evals/part-20
  2. [2]2015 facts
    ctx:discord/blah/training-and-evals/20
    • full texttraining-and-evals-20
      text/plain3 KBdoc:agent/training-and-evals-20/df884008-3d53-4aea-97bd-68748c59313f
      Show excerpt
      [2026-02-25 10:19] ajaxdavis: ``` There are a few concrete approaches, from least to most ambitious: 1. Parameterized activations (easy, high value) Instead of choosing between gelu and silu, parameterize a family that contains both a…

See also

Keep researching

Missing something or suspicious of what's here? Kick off a research session — a Claude agent will investigate, cite its sources, and file new facts into a dedicated context you can review before accepting into the shared view.

↑