compile: declare input activation ranges so opt=1 keeps int8 when provably safe - #161
Conversation
…vably safe Implements sbryngelson#155, and supersedes sbryngelson#156 by carrying its gate plus the bound machinery that makes the gate an optimization instead of a near-blanket decline. af.input(shape, max_abs=) records a promise about |value| at runtime. Undeclared inputs stay unbounded, so the fail-closed default from sbryngelson#153 is unchanged. _propagate_max_abs walks the graph conservatively. Anything not modelled returns None rather than a guess, since over-estimating only declines a lossy variant more often while under-estimating would wrongly keep one. Covers non-expanding elementwise ops, scalar mul/add, add/sub/mul/min/max over both operands, the matmul and linear weight row-sum bound, bmm, clip by its limits, and the saturating activations that are bounded whatever feeds them. That last group matters more than the declaration itself: sigmoid, tanh, softmax, l2_norm and relu6 bound an undeclared input, so anything downstream of a softmax or a sigmoid recovers int8 at opt=1 with no annotation at all. _act_max_abs now bounds the activation feeding each weight-consuming node rather than the graph output, which is the quantity a weight-quantizing variant actually has to encode. Verified on M2 Pro: the repro goes from 179/496 non-finite to 0/496, and a graph declaring max_abs=8.0 with real activations peaking at 7.1 keeps int8 and stays finite at relerr 4.6e-3, against 2.5e-4 for the lossless path. 21 new build-level tests, the 10 from sbryngelson#156 still pass, corpus GATE GREEN 90/90. Also commits bench/opt1_int8_saturation_repro.py, which the review on sbryngelson#156 asked for, and fixes the test docstring that referenced it while it was absent.
|
This is the right shape for #155, and the One blocker before merge: this branch is behind #157 and would revert it. Its Two smaller, non-blocking notes:
|
I built this branch's _optimize.py from the sbryngelson#156 branch, which predates sbryngelson#157, so copying it reverted the fail-closed relerr check. sbryngelson#157's source-form test caught it in CI, which is what that test exists for. Verified the rest of the copy carried nothing else stale: the diff against main is now only the ceiling gate and the rename.
Implements #155, the piece you wanted landed before #156.
This supersedes #156 rather than stacking on it: it carries that gate plus the bound machinery that turns it into an optimization instead of a near-blanket decline, so there is no regression window between the two. Close #156 in favour of this, or say the word and I will.
Two pieces, per the issue
1. Declare a range.
af.input(shape, max_abs=6.0)records a promise about|value|at runtime, stored on the node so_node_max_absreturns it instead ofNone. It is a promise, not a clamp, and the docstring says so. Undeclared inputs stay unbounded, so #153's fail-closed default is untouched.2. Propagate it.
_propagate_max_abswalks the graph conservatively. Anything not modelled returnsNonerather than a guess, since over-estimating only declines a lossy variant more often while under-estimating would wrongly keep one. Covered: non-expanding elementwise and shape ops, scalarmul/add,add/sub/mul/minimum/maximumover both operands,clipby its limits, thematmul/linearweight row-sum bound,bmm, and the normalizations when their affine is baked.The part worth more than the declaration
The saturating activations are bounded whatever feeds them, so they rescue an undeclared graph with no annotation:
So anything downstream of a
softmax,sigmoid,tanh,l2_normorrelu6keeps the fast path for free. That covers a lot of real attention and classifier graphs without anyone touching their code, which is what makes this more than a fancier option 3._act_max_abschanged meaningIn #156 it bounded the graph output. That was wrong for the purpose: what an int8-weight variant must encode is the activation feeding each weight-consuming node. It now walks to every
matmul/linear/bmm/convand boundssrcs[0]per node, returningNoneif any one of them is unbounded.Verification
Apple M2 Pro (Mac14,12 Mac mini), macOS 26.5.2.
bench/opt1_int8_saturation_repro.pymax_abs=8.0, real peak 7.1tests/test_input_ranges.pytests/test_opt1_act_ceiling.py(from #156)test_routes,test_tune_guards,test_shapestests/run_corpus.pyruff/pyright aneforge/ pre-commitThe declared-range case is the one that needed on-device proof rather than a build-level assertion: the gate lets int8 through there, so it has to actually be correct, not just permitted. relerr 4.6e-3 against the fp32 reference is the expected int8 cost and sits inside
_ACCURACY_TOL.Your review nit from #156
bench/opt1_int8_saturation_repro.pyis committed here, since you said a committed on-device repro would be good to have, and the test docstring that referenced it while it was absent is fixed.Judgement calls worth flagging
The
_NORM_SIGMA = 8.0cap. A normalized activation is O(1) but not provably bounded, since fp16 division by a small variance can overshoot, so boundinglayer_norm/rms_normoutput needs some assumption. I usedmax|gamma| * 8 + max|beta|, only when the affine is baked. 8 sigma is deliberately loose, but it is still an assumption rather than a proof, and it is the one part of this I would understand you rejecting. Dropping those four ops from_BOUNDED_OUTcosts only bounds through a norm and changes nothing else.Untested combination. I did not exercise
max_abstogether withcompress=or a non-fp16dtype; the declaration is inert in both paths as far as I can see, but I have not proven it.