Refactor AppHarness to use reflex run subprocess instead of threads (#5555)

* Refactor AppHarness to use reflex run subprocess instead of threads

- Replace _start_backend and _start_frontend thread methods with _start_subprocess
- Use new_process from reflex/utils/processes.py for subprocess management
- Implement console output parsing to detect server startup
- Update port management using handle_port for dynamic port assignment
- Maintain API compatibility for state management methods
- Update process cleanup for proper subprocess termination
- Add compatibility attributes and methods for existing tests
- Fix type annotations to resolve pyright variable override errors
- Add port parameter to _start_backend for test compatibility
- Fix docstring documentation for darglint compliance

Fixes ENG-6317

Co-Authored-By: masen@reflex.dev <masen@reflex.dev>

* Fix subprocess implementation issues in AppHarness

- Add timeout and non-blocking stdout reading in _wait_for_servers
- Simplify process cleanup logic in stop() method
- Add Windows compatibility for subprocess stdout handling
- Use daemon threads to avoid hanging on cleanup
- Add proper error handling for process termination

Fixes CI timeout issues in integration tests

Co-Authored-By: masen@reflex.dev <masen@reflex.dev>

* Fix subprocess implementation with shell=True approach

- Change from subprocess.Popen with argument list to shell=True with command string
- Debug testing confirmed shell=True successfully captures both frontend and backend messages
- Add fallback backend detection using port checking for additional robustness
- Resolves core subprocess stdout capture issue causing all 12 integration test failures
- Remove all debugging print statements and fix ruff-check issues

Based on comprehensive subprocess behavior testing that showed:
- Basic subprocess approach: Frontend=False, Backend=False
- Shell=True approach: Frontend=True, Backend=True

Co-Authored-By: masen@reflex.dev <masen@reflex.dev>

* run prod servers with `python -m` to avoid PATH issues

* human fixups

* enable logging to stdout
* allow prod mode to work
* better port selection algorithm
* remove useless code
* cleaner teardown
* test fixups for compatibility

* Fix darglint error by adding docstring to consume_output function

- Add proper docstring to nested consume_output function in _wait_for_servers method
- Resolves darglint compliance issue for subprocess implementation
- All other pre-commit hooks (ruff-format, ruff-check, codespell, prettier) pass

Co-Authored-By: masen@reflex.dev <masen@reflex.dev>

* Fix pre-commit

* Fix test_lifespan import error

* Fix test_input against timing race

* Add REFLEX_ERROR_LOG_FILE

Allow writing the console.error messages to a separate log file. Mainly used for testing or monitoring for errors.

* AppHarness: simplify subprocess output handling

Remove complex loops and threads from subprocess management in favor of log files.

Use a separate error log to check if any errors should be raised in the test process.

Raise error if reflex exits on its own with a non-zero code

* test_exception_handlers use new-style methods to get reflex output

* Remove unused fields from AppHarness

* test_deploy_url: check url via sitemap.xml

* test_large_state: disable 10k

it throws an error with the source map builder, and OOM without node 22

* test_stateless_app: update check URLs

rx.config is no longer the local app config, since it runs in a subprocess

* Unset REFLEX_REACT_STRICT_MODE after harness exits

* also ignore AccessDenied and keep trying

* fix missing rx import

* extra delay before bringing backend back up

* test_connection_banner: wait for backend port to free up

Avoid issue where we try to restart reflex, but the port is still taken by the
process we just killed. To properly test the reconnect logic, the backend needs
to come back up on the same port as before.

* test_connection_banner: wait longer for port to free up

* test_connection_banner: wait even longer (120s) for the port to free up

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: masen@reflex.dev <masen@reflex.dev>
Co-authored-by: Masen Furer <m_github@0x26.net>

Author: devin-ai-integration[bot]

reflex/environment.py

@@ -650,6 +650,9 @@ class EnvironmentVariables:
     # Enable full logging of debug messages to reflex user directory.
     REFLEX_ENABLE_FULL_LOGGING: EnvVar[bool] = env_var(False)
 
+    # The path to the reflex errors log file. If not set, no separate error log will be used.
+    REFLEX_ERROR_LOG_FILE: EnvVar[Path | None] = env_var(None)
+
 
 environment = EnvironmentVariables()