ENG-9028: Move event queue to backend

[masenf]

This patch moves the primary responsibility of chaining/queueing events to the backend. Previously the frontend was responsible for sending events to the backend. This created additional round-trip latency when chaining events using yield.

Summary: Event Processing & State Management Overhaul

Contexts are King

Instead of relying on ClassVar, module globals, and other hacks to track reflex objects, this PR introduces contextvars to the code base and leverages them to manage global state safely from anywhere in the app.

New Modules (in reflex_base)

• registry.py — ContextVar-based registry for BaseState classes and EventHandlers, replacing implicit global lookups. Currently used for states, events and stateful components. Eventually will be used for DECORATED_PAGES and other weird globals.
• event/context.py — EventContext (inherits BaseContext) holding per-event metadata: token, state manager ref, enqueue/emit/delta callbacks
• event/processor/ — New BaseStateEventProcessor and EventProcessor classes that own the full event lifecycle (previously spread across App and State methods)
• context/base.py — Generic BaseContext ContextVar wrapper

Breaking Changes (0.9.0)

• Event.token field removed — Event no longer carries a token; the token lives in EventContext
• Event.substate_token replaced by Event.state_cls property (resolved via the registry)
• Delta type refined from dict[str, Any] → dict[str, dict[str, Any]] (nested by substate name)
• StateManager.create() no longer takes a state= arg — state classes are discovered from the registry
• StateToken introduced (reflex/istate/manager/token.py) — typed generic token replacing raw "client_token_substate" strings in all state managers (disk, memory, redis)
• EventHandlerSetVar.state_cls → .state field rename to align with EventHandler change.
• fix_events removed from sortof public event API, replaced by Event.from_event_type()
• get_hydrate_event removed — replaced by internal rehydration; simulated pre-hydrated states removed
• App._background_tasks replaced by App.event_processor._tasks (processor manages background task lifecycle in the same way as normal event tasks)
• AppHarness (reflex/testing.py) simplified — state_manager property and related methods removed

State Manager Changes

• All three managers (disk, memory, redis) refactored to use StateToken[BaseState] instead of raw string tokens
• State managers now close() old locks properly
• OPLOCK_ENABLED support in state manager close/tests

Frontend (state.js)

• Params passed around as a ref instead of inline values
• StateUpdate only includes non-empty fields, reducing bytes over the wire.

Test Infrastructure

• Large test suite overhaul: new fixtures in conftest.py for EventProcessor, registry, and context
• get_app / mock_app dependencies removed from tests in favor of the new processor-based approach
• New unit tests for registry, event context, base state processor, and event processor
• Integration tests updated to replace all backend state assertions with in-app equivalents.

[masenf]

@greptile-apps re-review this PR now that first round of feedback has been applied and all tests are passing.

pay specific attention to things like potentially breaking changes that have not already been called out. pay attention to regressions. watch out for potential performance pitfalls.

[Copilot]

Pull request overview

This PR migrates event chaining/queueing responsibility from the frontend to the backend by introducing ContextVar-based registries/contexts and a dedicated backend event processor, alongside a typed StateToken API for state managers.

Changes:

• Introduces ContextVar-driven RegistrationContext and EventContext, plus a new backend EventProcessor / BaseStateEventProcessor lifecycle.
• Refactors state managers (memory/disk/redis) to use StateToken/BaseStateToken instead of legacy string tokens, and updates call sites accordingly.
• Updates unit/integration tests and frontend event loop behavior to align with backend-driven event chaining and smaller payloads.

Reviewed changes

Copilot reviewed 70 out of 73 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/units/test_state_tree.py	Updates Redis state access to use BaseStateToken.
tests/units/test_model.py	Switches event processing test to processor-based enqueue + emitted delta assertions.
tests/units/test_event.py	Updates tests for Event token removal and EventHandler signature/state changes.
tests/units/reflex_core/_internal/test_registry.py	Adds unit tests for the new RegistrationContext registry.
tests/units/reflex_core/_internal/event/test_context.py	Adds unit tests for EventContext behavior (fork/emit).
tests/units/reflex_core/_internal/event/processor/test_timeout.py	Adds tests for DrainTimeoutManager.
tests/units/reflex_core/_internal/event/processor/test_future.py	Adds tests for hierarchical EventFuture chaining/cancellation.
tests/units/reflex_core/_internal/event/processor/test_base_state_processor.py	Adds tests for _rehydrate path and delta emission.
tests/units/reflex_core/_internal/context/test_base.py	Adds tests for BaseContext ContextVar attach/reset semantics.
tests/units/reflex_core/_internal/context/init.py	Marks internal context test package.
tests/units/reflex_core/_internal/init.py	Marks internal test package.
tests/units/reflex_core/init.py	Marks reflex_core unit test package.
tests/units/middleware/test_hydrate_middleware.py	Migrates middleware test to registry-based state discovery.
tests/units/middleware/conftest.py	Removes Event.token usage from test event factory.
tests/units/istate/test_proxy.py	Updates proxy recovery test to use EventContext + state manager monkeypatching.
tests/units/istate/manager/test_token.py	Adds unit tests for new StateToken / BaseStateToken behaviors.
tests/units/istate/manager/test_redis.py	Refactors Redis state manager tests to BaseStateToken + new ctor.
tests/units/istate/manager/test_expiration.py	Refactors memory manager expiration tests to BaseStateToken + new ctor.
tests/units/conftest.py	Adds new fixtures for processors, contexts, registries, and parametrized state managers.
tests/integration/utils.py	Adds UI-driven helpers to assert event ordering without backend state reads.
tests/integration/test_memory_state_manager_expiration.py	Updates assertion to check app_instance.state_manager instead of harness API.
tests/integration/test_input.py	Adjusts integration test to clear state via backend event rather than direct state mutation.
tests/integration/test_form_submit.py	Makes form state reactive field and asserts backend receipt via rendered JSON text.
tests/integration/test_event_chain.py	Migrates event chain assertions to DOM-based order checks + strict app naming.
tests/integration/test_event_actions.py	Migrates event-order assertions to DOM and simplifies rendered list markup.
tests/integration/test_dynamic_routes.py	Migrates order checks to DOM and exposes params via computed var for assertions.
tests/integration/test_connection_banner.py	Switches “disconnect” simulation to browser offline mode and refactors Redis handling.
tests/integration/test_computed_vars.py	Removes direct backend state assertions that depended on old harness state access.
tests/integration/test_component_state.py	Migrates backend var assertions to explicit backend events triggered from UI.
tests/integration/test_client_storage.py	Refactors expiration/reset paths to avoid direct backend state manager manipulation.
tests/integration/test_background_task.py	Updates background task tracking assertion to processor-managed _tasks.
reflex/utils/token_manager.py	Updates keyspace notification enablement to new Redis manager ctor.
reflex/utils/tasks.py	Adds optional task_context support for task creation in a specific ContextVar context.
reflex/testing.py	Refactors AppHarness to manage RegistrationContext + removes direct state-manager access helpers.
reflex/istate/wrappers.py	Refactors get_state wrapper to use BaseStateToken and EventContext-sourced manager.
reflex/istate/shared.py	Updates shared state flows to use BaseStateToken and removes Event.token dependency.
reflex/istate/proxy.py	Refactors proxy to use EventContext + BaseStateToken and emit delta on exit.
reflex/istate/manager/token.py	Introduces StateToken/BaseStateToken (serialization, string forms, legacy parsing).
reflex/istate/manager/redis.py	Refactors Redis manager API to accept StateToken and support non-BaseState tokens.
reflex/istate/manager/memory.py	Refactors memory manager to accept StateToken and handle eviction/locking updates.
reflex/istate/manager/disk.py	Refactors disk manager to accept StateToken and generic queue items for persistence.
reflex/istate/manager/init.py	Updates StateManager API to be token-based and fetch manager from EventContext.
reflex/app.py	Integrates backend event processor, registry context plumbing, and token-based modify_state.
reflex/app_mixins/middleware.py	Removes automatic HydrateMiddleware injection (processor now owns event lifecycle).
reflex/init.py	Exposes StateToken and BaseStateToken in public namespace mapping.
pyproject.toml	Expands coverage source list and increases fail_under threshold.
pyi_hashes.json	Updates hash for generated reflex/__init__.pyi.
packages/reflex-core/src/reflex_core/utils/serializers.py	Adds a deserializers mapping (incl. UUID) for typed event arg transformation.
packages/reflex-core/src/reflex_core/utils/format.py	Updates handler formatting to use handler.state and makes query parsing more defensive.
packages/reflex-core/src/reflex_core/plugins/_screenshot.py	Refactors clone-state plugin to use BaseStateToken accessors.
packages/reflex-core/src/reflex_core/event.py	Removes Event.token, adds Event.from_event_type, updates EventHandler state binding semantics.
packages/reflex-core/src/reflex_core/constants/state.py	Removes FRONTEND_EVENT_STATE constant.
packages/reflex-core/src/reflex_core/components/tags/tag.py	Defers EventChain import to avoid import cycles.
packages/reflex-core/src/reflex_core/components/component.py	Moves stateful component memo cache into RegistrationContext.
packages/reflex-core/src/reflex_core/.templates/web/utils/state.js	Removes frontend event-processing lock; reduces payload size; adjusts router_data query inclusion.
packages/reflex-core/src/reflex_core/_internal/registry.py	Adds new ContextVar-backed registry for states/handlers/components.
packages/reflex-core/src/reflex_core/_internal/event/processor/timeout.py	Adds DrainTimeoutManager used for graceful shutdown/drain semantics.
packages/reflex-core/src/reflex_core/_internal/event/processor/future.py	Adds EventFuture for hierarchical chain tracking/cancellation.
packages/reflex-core/src/reflex_core/_internal/event/processor/compat.py	Adds as_completed shim for older Python versions.
packages/reflex-core/src/reflex_core/_internal/event/processor/base_state_processor.py	Introduces stateful event lifecycle handling, rehydration, payload transforms, chaining.
packages/reflex-core/src/reflex_core/_internal/event/processor/init.py	Exposes processor public internal API surface.
packages/reflex-core/src/reflex_core/_internal/event/context.py	Adds EventContext for per-event metadata and enqueue/emit callbacks.
packages/reflex-core/src/reflex_core/_internal/event/init.py	Marks internal event package.
packages/reflex-core/src/reflex_core/_internal/context/base.py	Adds BaseContext ContextVar wrapper + attach/reset safeguards.
packages/reflex-core/src/reflex_core/_internal/context/init.py	Marks internal context package.
packages/reflex-core/src/reflex_core/_internal/init.py	Marks internal reflex_core package.
packages/reflex-components-core/src/reflex_components_core/core/_upload.py	Refactors upload handling to enqueue via backend processor and stream deltas.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.