[useScrollTrigger] Fix flip-flopping when a layout resize shifts the scroll position

[xxashxx-svg]

Fixes #46589

Problem

When useScrollTrigger drives content inside a position="sticky" AppBar (for example a Collapse that hides a secondary toolbar), the trigger flip-flops and the AppBar visibly shakes while scrolling slowly:

1. The user scrolls down past the threshold, the trigger turns true and the collapse starts.
2. The collapse shrinks the page. The browser's scroll anchoring keeps the visible content in place by reducing scrollY, which fires scroll events with a decreasing position.
3. defaultTrigger interprets the decreasing position as the user scrolling up and flips the trigger back to false.
4. The content expands again, scroll anchoring increases scrollY, the trigger flips to true… and the loop repeats.

As @abriginets noted in the issue, the resize itself changes pageYOffset/scrollTop, so the previous-vs-current comparison in defaultTrigger no longer reflects the user's scroll intent during the transition.

Fix

defaultTrigger now also records the scroll target's scrollHeight between scroll events (entirely internal state, no API change):

• If the scrollable area's height changed since the last scroll event, the position change is (at least partly) caused by the layout resize, so the event's direction is not trusted and the previous trigger value is kept.
• With hysteresis, once triggered, scrolling down no longer re-checks the threshold. A resize can legitimately leave the scroll position below the threshold while triggered (previously unreachable state); only a genuine upward scroll should untrigger. This keeps the trigger stable after the collapse settles.

Genuine scrolling behaves exactly as before: all existing tests pass unchanged, and the threshold/hysteresis semantics for a stable layout are untouched.

Tests

Added two regression tests (element target and window target) that replay the reported scenario: scroll down past the threshold, shrink the scrollable area while shifting the position up (collapse + scroll anchoring), keep scrolling, scroll up, grow the area back. Both fail on master with the exact oscillation from the issue (trigger flips on every resize frame) and pass with this change.

Ran: pnpm test:unit run useScrollTrigger (node: 16 passed; browser: 3 passed, 13 jsdom-only skipped), eslint, prettier and tsgo -p tsconfig.json for packages/mui-material.

Relation to #46751

The earlier attempt #46751 added a user-facing opt-in (enableReentrantLock + a duration that had to match the transition timing) and was closed after reviewers asked for the edge case to be fixed internally without new public API (review comment). This PR fixes it inside defaultTrigger with no new options, no timers, and no behavior change for stable layouts.

🤖 Generated with Claude Code

[code-infra-dashboard]

Deploy preview

https://deploy-preview-48801--material-ui.netlify.app/

Bundle size

Bundle	Parsed size	Gzip size
@mui/material	🔺+226B(+0.04%)	🔺+65B(+0.04%)
@mui/lab	0B(0.00%)	0B(0.00%)
@mui/private-theming	0B(0.00%)	0B(0.00%)
@mui/system	0B(0.00%)	0B(0.00%)
@mui/utils	0B(0.00%)	0B(0.00%)

Details of bundle changes

---

Check out the code infra dashboard for more information about this PR.