Skip to content

Commit eaa8682

Browse files
committed
transformerless_lm: omniweight fluid form (tanh-backed standard)
Replace hard clamp [-pi*log(phi), +pi*log(phi)] with fluid form: fluid_delta = phi^pi * tanh(delta_acc / phi^pi) phi^pi ~ 4.53 is the substrate reserve standard (same constant as bigram blend alpha, recency, harmony scale). Small contributions pass nearly linear (tanh near origin ~ identity). Large contributions saturate gracefully toward +/- phi^pi. Key property: when primitives agree, sum grows naturally inside the linear region. When they disagree, contributions cancel within the sum -- no artificial ceiling restricting growth. User-named architecture (omniweight, ported from earlier robotics control work). Backed-standard not clamp.
1 parent 91c484f commit eaa8682

1 file changed

Lines changed: 20 additions & 7 deletions

File tree

experiments/transformerless_lm/train_self_recursive.py

Lines changed: 20 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1292,9 +1292,14 @@ def substrate_recency_penalty(history_tokens: torch.Tensor, logits: torch.Tensor
12921292

12931293
# OMNIWEIGHT: shared log-pressure ledger. Each primitive contributes
12941294
# delta_log_p to a single accumulator instead of chaining probs->probs
1295-
# transforms. Total contribution is clamped to [-pi*log(phi), +pi*log(phi)]
1296-
# (substrate-bounded), then applied once.
1297-
_OMNIWEIGHT_CLAMP = math.pi * math.log(_PHI_FOR_SAMPLING) # ~1.51
1295+
# transforms.
1296+
#
1297+
# FLUID backed-standard form (v72+): the substrate reserve phi^pi acts
1298+
# as a backing standard. Accumulator passes through tanh scaled by
1299+
# phi^pi -- small contributions pass nearly linear, large saturate
1300+
# gracefully to +/- phi^pi. No hard clamp; growth allowed in
1301+
# proportion to substrate trust.
1302+
_OMNIWEIGHT_RESERVE = _PHI_FOR_SAMPLING ** math.pi # ~4.53
12981303

12991304

13001305
def _omniweight_delta(base_probs: torch.Tensor,
@@ -1310,11 +1315,19 @@ def _omniweight_delta(base_probs: torch.Tensor,
13101315

13111316
def _omniweight_apply(base_probs: torch.Tensor,
13121317
delta_acc: torch.Tensor) -> torch.Tensor:
1313-
"""Apply accumulated log-pressure to base probs. Clamped to
1314-
substrate-bounded range, then renormalized.
1318+
"""Apply accumulated log-pressure via tanh-scaled substrate reserve.
1319+
1320+
fluid_delta = phi^pi * tanh(delta_acc / phi^pi)
1321+
1322+
Small contributions pass linear (tanh near origin ~ identity).
1323+
Large contributions saturate gracefully toward +/- phi^pi.
1324+
When primitives agree, deltas sum cleanly. When they disagree,
1325+
they cancel naturally within the sum.
1326+
1327+
Pure substrate (phi^pi as the reserve standard).
13151328
"""
1316-
delta_clamped = delta_acc.clamp(-_OMNIWEIGHT_CLAMP, _OMNIWEIGHT_CLAMP)
1317-
out = base_probs * torch.exp(delta_clamped)
1329+
fluid = _OMNIWEIGHT_RESERVE * torch.tanh(delta_acc / _OMNIWEIGHT_RESERVE)
1330+
out = base_probs * torch.exp(fluid)
13181331
return out / (out.sum() + 1e-8)
13191332

13201333

0 commit comments

Comments
 (0)