Skip to content

compile: declare input activation ranges so opt=1 keeps int8 when provably safe - #161

Merged
sbryngelson merged 2 commits into
sbryngelson:mainfrom
axiom-of-choice:feat-155-input-ranges
Aug 3, 2026
Merged

compile: declare input activation ranges so opt=1 keeps int8 when provably safe#161
sbryngelson merged 2 commits into
sbryngelson:mainfrom
axiom-of-choice:feat-155-input-ranges

Conversation

@axiom-of-choice

Copy link
Copy Markdown
Contributor

Implements #155, the piece you wanted landed before #156.

This supersedes #156 rather than stacking on it: it carries that gate plus the bound machinery that turns it into an optimization instead of a near-blanket decline, so there is no regression window between the two. Close #156 in favour of this, or say the word and I will.

Two pieces, per the issue

1. Declare a range. af.input(shape, max_abs=6.0) records a promise about |value| at runtime, stored on the node so _node_max_abs returns it instead of None. It is a promise, not a clamp, and the docstring says so. Undeclared inputs stay unbounded, so #153's fail-closed default is untouched.

2. Propagate it. _propagate_max_abs walks the graph conservatively. Anything not modelled returns None rather than a guess, since over-estimating only declines a lossy variant more often while under-estimating would wrongly keep one. Covered: non-expanding elementwise and shape ops, scalar mul/add, add/sub/mul/minimum/maximum over both operands, clip by its limits, the matmul/linear weight row-sum bound, bmm, and the normalizations when their affine is baked.

The part worth more than the declaration

The saturating activations are bounded whatever feeds them, so they rescue an undeclared graph with no annotation:

input undeclared            bound=None       int8 declined
input undeclared -> softmax bound=1.0        int8 KEPT
input undeclared -> sigmoid bound=1.0        int8 KEPT
input max_abs=6.0           bound=6.0        int8 KEPT
input max_abs=6.0 -> relu   bound=6.0        int8 KEPT
input max_abs=1e5           bound=100000.0   int8 declined

So anything downstream of a softmax, sigmoid, tanh, l2_norm or relu6 keeps the fast path for free. That covers a lot of real attention and classifier graphs without anyone touching their code, which is what makes this more than a fancier option 3.

_act_max_abs changed meaning

In #156 it bounded the graph output. That was wrong for the purpose: what an int8-weight variant must encode is the activation feeding each weight-consuming node. It now walks to every matmul/linear/bmm/conv and bounds srcs[0] per node, returning None if any one of them is unbounded.

Verification

Apple M2 Pro (Mac14,12 Mac mini), macOS 26.5.2.

result
bench/opt1_int8_saturation_repro.py 179/496 non-finite to 0/496
declared max_abs=8.0, real peak 7.1 int8 kept, finite, relerr 4.6e-3 (lossless path: 2.5e-4)
tests/test_input_ranges.py 21 passed
tests/test_opt1_act_ceiling.py (from #156) 10 passed, unchanged
test_routes, test_tune_guards, test_shapes 47 passed together
tests/run_corpus.py GATE: GREEN, 90/90
ruff / pyright aneforge / pre-commit clean (pyright's 9 are the pre-existing optional imports)

The declared-range case is the one that needed on-device proof rather than a build-level assertion: the gate lets int8 through there, so it has to actually be correct, not just permitted. relerr 4.6e-3 against the fp32 reference is the expected int8 cost and sits inside _ACCURACY_TOL.

Your review nit from #156

bench/opt1_int8_saturation_repro.py is committed here, since you said a committed on-device repro would be good to have, and the test docstring that referenced it while it was absent is fixed.

Judgement calls worth flagging

The _NORM_SIGMA = 8.0 cap. A normalized activation is O(1) but not provably bounded, since fp16 division by a small variance can overshoot, so bounding layer_norm/rms_norm output needs some assumption. I used max|gamma| * 8 + max|beta|, only when the affine is baked. 8 sigma is deliberately loose, but it is still an assumption rather than a proof, and it is the one part of this I would understand you rejecting. Dropping those four ops from _BOUNDED_OUT costs only bounds through a norm and changes nothing else.

Untested combination. I did not exercise max_abs together with compress= or a non-fp16 dtype; the declaration is inert in both paths as far as I can see, but I have not proven it.

…vably safe

Implements sbryngelson#155, and supersedes sbryngelson#156 by carrying its gate plus the bound machinery
that makes the gate an optimization instead of a near-blanket decline.

af.input(shape, max_abs=) records a promise about |value| at runtime. Undeclared
inputs stay unbounded, so the fail-closed default from sbryngelson#153 is unchanged.

_propagate_max_abs walks the graph conservatively. Anything not modelled returns
None rather than a guess, since over-estimating only declines a lossy variant more
often while under-estimating would wrongly keep one. Covers non-expanding
elementwise ops, scalar mul/add, add/sub/mul/min/max over both operands, the
matmul and linear weight row-sum bound, bmm, clip by its limits, and the
saturating activations that are bounded whatever feeds them.

That last group matters more than the declaration itself: sigmoid, tanh, softmax,
l2_norm and relu6 bound an undeclared input, so anything downstream of a softmax
or a sigmoid recovers int8 at opt=1 with no annotation at all.

_act_max_abs now bounds the activation feeding each weight-consuming node rather
than the graph output, which is the quantity a weight-quantizing variant actually
has to encode.

Verified on M2 Pro: the repro goes from 179/496 non-finite to 0/496, and a graph
declaring max_abs=8.0 with real activations peaking at 7.1 keeps int8 and stays
finite at relerr 4.6e-3, against 2.5e-4 for the lossless path. 21 new build-level
tests, the 10 from sbryngelson#156 still pass, corpus GATE GREEN 90/90.

Also commits bench/opt1_int8_saturation_repro.py, which the review on sbryngelson#156 asked
for, and fixes the test docstring that referenced it while it was absent.
@sbryngelson

Copy link
Copy Markdown
Owner

This is the right shape for #155, and the _BOUNDED_OUT insight is what makes it not a regression: softmax / sigmoid / tanh / l2_norm / relu6 bound the activation with no declaration, so anything downstream keeps int8 for free -- which covers most real graphs (every transformer has a softmax) without users touching their code. The propagation is appropriately conservative (norms via gamma*8sigma + beta, matmul via weight row-sums, over-estimation always safe, unmodelled -> None fail-closed), and max_abs as a promise-not-clamp with fail-closed default is exactly the contract. Test coverage is thorough, the repro is committed, and the docstring nit is fixed. It supersedes #156 -- I'm closing that in favor of this.

One blocker before merge: this branch is behind #157 and would revert it. Its measure() gate is back to if relerr > tol (head _optimize.py:364), where main now has if not (relerr <= tol) from #157 (merged). Merging as-is re-opens the exact nan fail-open #157 just closed. Please rebase on current main -- after that the diff should stop touching measure() entirely, since #161 doesn't intend to change it. (I'll hold CI approval until the rebase so it tests the right thing.)

Two smaller, non-blocking notes:

I built this branch's _optimize.py from the sbryngelson#156 branch, which predates sbryngelson#157, so
copying it reverted the fail-closed relerr check. sbryngelson#157's source-form test caught
it in CI, which is what that test exists for.

Verified the rest of the copy carried nothing else stale: the diff against main is
now only the ceiling gate and the rename.
@sbryngelson
sbryngelson merged commit 06ae749 into sbryngelson:main Aug 3, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants