Merge pull request #93 from Integration-Automation/dev

Split extended_features into 10 thematic subdocs

Author: JE-Chen

docs/source/Eng/doc/architecture/architecture_doc.rst

@@ -0,0 +1,38 @@
+============
+Architecture
+============
+
+System overview
+===============
+
+.. mermaid::
+
+   flowchart LR
+     A1["Action JSON"] --> EXE["Executor"]
+     A2["Recorder"] --> A1
+     A3["LLM NL → draft"] --> A1
+     EXE --> SEL["Selenium"]
+     EXE --> PW["Playwright"]
+     EXE --> APM["Appium"]
+     EXE --> HTTP["HTTP API"]
+     EXE --> DB["Database"]
+     SEL --> REC["Records"]
+     PW --> REC
+     REC --> REP["Reports"]
+     REC --> OBS["Observability"]
+     REC --> NOT["Notifiers"]
+
+Action lifecycle
+================
+
+.. mermaid::
+
+   flowchart LR
+     IN["[cmd, args, kwargs]"] --> VAL["Validator"]
+     VAL --> ENV["${ENV.X} / ${ROW.x}"]
+     ENV --> SPAN["OTel span"]
+     SPAN --> RETRY["Retry policy"]
+     RETRY --> GATE["Script gate"]
+     GATE --> DISP["event_dict[cmd]"]
+     DISP --> RECORD["records.append"]
+     DISP -- failure --> SHOT["Auto screenshot"]

docs/source/Eng/doc/backends/backends_doc.rst

@@ -0,0 +1,46 @@
+========
+Backends
+========
+
+Selenium (default)
+==================
+
+The original ``WebDriverWrapper`` plus ``WebElementWrapper``. All commands
+without a more specific prefix dispatch here.
+
+Playwright
+==========
+
+A full mirror of the Selenium surface lives under ``WR_pw_*``:
+
+* Lifecycle / pages / navigation
+* Find (with ``TestObject`` translation) and direct page-level shortcuts
+* Element-level wrapper
+* Mobile emulation, locale, timezone, geolocation, permissions, clock
+* HAR recording, route mocking, console + network event capture
+* Network throttling presets via CDP
+
+Switch is opt-in: existing scripts keep running on Selenium.
+
+Cloud Grid
+==========
+
+Provider helpers for BrowserStack, Sauce Labs, and LambdaTest:
+
+* ``connect_browserstack`` / ``connect_saucelabs`` / ``connect_lambdatest``
+* ``build_browserstack_capabilities`` / ``build_saucelabs_capabilities`` /
+  ``build_lambdatest_capabilities``
+* ``start_remote_driver`` for arbitrary hub URLs
+
+Appium (mobile)
+===============
+
+``start_appium_session`` builds an Appium WebDriver and registers it on the
+shared Selenium wrapper so existing ``WR_*`` commands keep working against a
+mobile session. Capability builders cover both Android (UiAutomator2) and
+iOS (XCUITest).
+
+``appium_integration.gestures`` adds higher-level mobile gestures —
+``swipe`` / ``scroll`` / ``long_press`` / ``pinch`` / ``double_tap``
+prefer Appium's ``mobile:`` named extensions and fall back to W3C Actions
+sequences.

docs/source/Eng/doc/browser_internals/browser_internals_doc.rst

@@ -0,0 +1,63 @@
+=================
+Browser internals
+=================
+
+* **CDP** — ``selenium_cdp`` / ``playwright_cdp`` raw passthroughs.
+* **Storage** — ``localStorage`` / ``sessionStorage`` / ``IndexedDB`` get /
+  set / clear via injected JS.
+* **Service worker / cache** — unregister / clear caches /
+  ``Network.setBypassServiceWorker``.
+* **Console + network capture** — Playwright event listeners with
+  assertions (``no console errors`` / ``no 5xx``).
+* **Shadow DOM** — selector chains pierce nested shadow roots.
+* **iframes** — switch chains and Playwright frame-locator chains.
+* **File upload / download** — element ``send_keys`` / ``set_input_files``
+  for upload; ``wait_for_download`` polls a directory for completed files.
+* **Browser extension loaders** — Chrome ``add_extension`` / Playwright
+  ``--load-extension``.
+
+Browser & locale
+================
+
+* ``device_emulation`` — ``available_presets`` /
+  ``playwright_kwargs("iPhone 15 Pro")`` /
+  ``apply_to_chrome_options(opts, "Desktop 1080p")`` /
+  ``cdp_emulation_command(name)``.
+* ``geo_locale.GeoOverride`` — yields both
+  ``cdp_payloads(override)`` and ``playwright_context_kwargs(override)``.
+* ``multi_tab.TabChoreographer`` — track tabs by alias;
+  ``register_current`` / ``open_new`` / ``switch_to`` / ``with_tab`` /
+  ``close``.
+* ``webauthn.enable_virtual_authenticator(driver)`` — CDP
+  ``WebAuthn.addVirtualAuthenticator`` for passkey simulation.
+
+Storybook / shadow DOM
+======================
+
+* ``storybook.discover_stories(index_or_path)`` reads Storybook 7+
+  ``index.json``;
+  ``plan_actions_for_stories(stories, base_url, run_a11y=True,
+  capture_screenshot=True, extra_per_story=...)`` builds a flat action
+  plan that visits each story under ``iframe.html?id=...`` and runs
+  axe / screenshot.
+* ``storybook.visual_snapshots.capture_story_snapshots(stories,
+  base_url, take_screenshot, navigate, baseline_dir=...)`` — per-story
+  PNG capture with byte-level baseline comparison.
+* ``dom_traversal.shadow_pierce.find_first(driver, css_selector)`` /
+  ``find_all`` walk open shadow roots recursively. ``execute_script``
+  for Selenium, ``evaluate`` for Playwright; ``assert_pierced_visible``
+  raises if the selector doesn't match anywhere.
+
+CDP tap / cross-browser / state diff
+====================================
+
+* ``cdp_tap.CdpRecorder(output_path).attach(driver)`` — wraps
+  ``execute_cdp_cmd`` so every call is appended to an ndjson log;
+  ``CdpReplayer(load_recording(path))`` plays the same sequence back.
+* ``cross_browser.diff_runs([chromium_run, firefox_run, webkit_run])``
+  — buckets findings into ``major`` / ``minor`` (5xx → major,
+  screenshot hash → minor); ``assert_parity(report, only_major=True)``
+  is the CI gate.
+* ``state_diff.capture_state(driver)`` snapshots cookies +
+  localStorage + sessionStorage; ``diff_states(before, after)`` reports
+  added / removed / changed keys per section.

docs/source/Eng/doc/cookbook/cookbook_doc.rst

@@ -0,0 +1,49 @@
+==============================
+Cookbook, tiers & API façade
+==============================
+
+Cookbook examples
+=================
+
+The ``examples/`` directory ships runnable recipes that drive real Chrome
+end-to-end. Each found a real bug the unit suite missed:
+
+* ``counting_stars.{py,json}`` — open YouTube and play OneRepublic
+  Counting Stars; revealed the bug where
+  ``webdriver_wrapper.execute_script`` was swallowing return values.
+* ``google_search.py`` — consent dismissal + result heading scrape.
+* ``form_submit.py`` — ``form_autofill.plan_fill_actions`` +
+  ``state_diff.capture_state`` round trip against ``httpbin``.
+* ``smart_wait_demo.py`` — ``wait_for_fetch_idle``,
+  ``wait_for_spa_route_stable``, ``memory_leak.detect_growth``.
+* ``fanout_demo.py`` — parallel HTTP preflights via ``run_fan_out``.
+* ``pii_redact_demo.py`` — pure-logic ``scan_text`` / ``redact_text``.
+
+Test tiers
+==========
+
+* ``test/unit_test/`` — 1200 mock-based unit tests, ~12s.
+* ``test/integration_test/`` — 30 wired-modules tests with real I/O
+  (in-memory SQLite, in-process HTTP servers, real subprocesses for
+  the MCP / LSP). Surfaced the Windows LSP CRLF framing bug.
+* ``test/e2e_test/`` — six real-browser smoke tests; skips cleanly when
+  ``WEBRUNNER_E2E_HUB`` doesn't resolve. Use
+  ``cd docker && docker compose up -d`` locally;
+  ``.github/workflows/e2e_browser.yml`` runs them daily / on demand.
+
+Thematic façade
+===============
+
+The 80+ helpers under ``je_web_runner.utils.<area>`` are also re-exported
+under ``je_web_runner.api`` grouped by theme:
+
+.. code-block:: python
+
+   from je_web_runner.api import (
+       authoring, debugging, frontend, infra, mobile,
+       networking, observability, quality, reliability,
+       security, test_data,
+   )
+
+The original Selenium-flavoured top-level surface stays unchanged so
+existing user code keeps working.

docs/source/Eng/doc/data_auth_api/data_auth_api_doc.rst

@@ -0,0 +1,52 @@
+=========
+Test data
+=========
+
+* **Faker integration** — ``fake_email`` / ``fake_name`` / ``fake_value`` and
+  friends; ``faker`` is a soft dependency.
+* **Factories** — ``Factory(defaults)`` evaluates callable defaults per
+  ``build()``; pre-built ``user_factory`` / ``order_factory`` /
+  ``product_factory``.
+* **Testcontainers** — ``start_postgres`` / ``start_redis`` /
+  ``start_generic`` wrap testcontainers-python.
+* **.env loader + ${ENV.X}** — ``load_env`` / ``expand_in_action`` so the
+  same actions can target dev / staging / prod.
+* **Data-driven runner** — ``load_dataset_csv`` / ``load_dataset_json`` /
+  ``run_with_dataset`` with ``${ROW.col}`` placeholder expansion.
+
+Test data & determinism
+=======================
+
+* ``snapshot.fixture_record.FixtureRecorder("fx.json", mode="auto")`` —
+  record once, replay forever; modes ``record`` / ``replay`` / ``auto``.
+* ``database.fixtures.load_fixture_file("seed.json")`` +
+  ``load_into_connection(conn, fixture)`` — seed Postgres / MySQL /
+  SQLite from ``{table: [rows]}`` JSON.
+
+Auth, API, database
+===================
+
+* **OAuth2 / OIDC** — ``client_credentials_token`` / ``password_grant_token``
+  / ``refresh_token_grant`` with in-process token cache that refreshes 30 s
+  before expiry.
+* **HTTP API** — ``http_get`` / ``http_post`` / ``http_put`` / ``http_patch``
+  / ``http_delete`` plus ``http_assert_status`` and
+  ``http_assert_json_contains``.
+* **Database** — ``db_query`` / ``db_assert_count`` / ``db_assert_value`` /
+  ``db_assert_exists`` / ``db_assert_empty``; SQLAlchemy soft dependency,
+  bound parameters only.
+
+API & contract testing
+======================
+
+* ``api_mock.MockRouter().add(method, url_pattern, body=, status=, times=)``
+  — supports literal, glob, and ``re:`` regex URL patterns; attach to a
+  Playwright page with ``attach_to_page(page)``.
+* ``contract_testing.validate_response(body, schema)`` — JSON-Schema
+  subset (type / properties / required / items / enum / oneOf /
+  additionalProperties); ``validate_against_openapi`` resolves
+  ``$ref`` and looks up ``paths[…].responses[…]``.
+* ``graphql.GraphQLClient(endpoint).execute(query, variables=)`` +
+  ``extract_field(payload, "users[0].name")``.
+* ``mock_services`` — ``MockOAuthServer``, ``MockSmtpServer``,
+  ``MockS3Storage`` for offline CI runs.