feat(live): refactor AI self-heal to use keyword-based ReAct loop and improve locator logging by chinmayajha · Pull Request #303 · mozarkai/optics-framework

chinmayajha · 2026-06-15T16:37:02Z

Summary

This PR refactors the AI self-heal mechanism from coordinate-guessing (which frequently tapped blind/guessed screen coordinates and often missed targets) to a keyword-based execution model. The AI now acts as a ReAct loop over the framework's own action keywords, utilizing more reliable, targeted actions that route through the locating strategies ladder (XPath, text, OCR, image-matching).

Additionally, it cleans up noisy expected element-not-found log output from Appium element sources to prevent the live TUI silent error checker from overriding successful self-heals with failure statuses.

Changes

1. AI Self-Heal Refactor (`optics_framework/common/ai_self_heal.py`)

Rewrote AISelfHealHandler to run a keyword-based loop (similar to the NL agent but lighter).
Replaced the coordinate schema (percent_x, percent_y, text, direction) in HEAL_ACTION_SCHEMA with keyword name, parameter string list, and a completed status flag.
Added catalog introspection to the prompt so the LLM knows available signatures.
Instructed the LLM to prefer targeting elements by visible text (using the condensed UI hierarchy) via press_element instead of guessing percentages.
Added explicit prompt guidelines for gesture directions to align the LLM's spatial logical scroll actions with the physical swipe inputs expected by the driver.

2. Action Keyword Integration (`optics_framework/api/action_keyword.py`)

Implemented _heal_execute() to parse and execute keywords generated by the LLM during the self-heal process.
Temporarily disables the self-heal toggle during internal keyword calls to prevent infinite recursion.
Added _heal_catalog() to serve a focused keyword catalog: press_element, enter_text, scroll, swipe_by_percentage, press_keycode, press_by_percentage.

3. Log Cleanup for Silent Error Checker

optics_framework/engines/elementsources/appium_find_element.py
optics_framework/engines/elementsources/appium_page_source.py
- Caught NoSuchElementException specifically and raised OpticsError without logging stack tracebacks via internal_logger.exception (which is treated as a silent error/failure in live.py).

4. Tests

tests/units/common/test_ai_self_heal.py
- Updated unit tests to use FakeExecutor instead of FakeDriver.
- Added new coverage for keyword execution, multi-step navigation, catalog prompts, done/give_up loops, and ActionKeyword integrations.

Verification

Ran full test suite:

poetry run pytest --ignore=tests/feature

Add ElementSourceInterface.capture_screenshot_bytes(), an optional fast path that returns the device's natively-encoded screenshot (PNG/JPEG) so callers needing only bytes skip the numpy decode -> re-encode round-trip. Implement it across every backend: - Appium / Selenium: base64 endpoint, with a backoff retry for truncated responses. - Playwright: returns page.screenshot()'s PNG bytes verbatim (no decode at all). Each backend's capture_screenshot_as_numpy() now reuses capture_screenshot_bytes(), so the bytes and numpy paths share one fetch+retry implementation. StrategyManager.capture_screenshot_bytes() prefers any source implementing the native path and otherwise encodes the numpy frame, so every backend keeps working. LiveController.screenshot_png_bytes() (the NL agent's screenshot provider) now delegates to it, keeping the live layer driver-agnostic.

…back StrategyManager prefers native bytes, falls back to numpy encode, and skips a failing native source. Per-backend coverage: Appium/Selenium base64 decode with retry, Playwright raw-PNG passthrough, and capture_screenshot_as_numpy reusing the bytes path.

…erface default - ElementSourceInterface.capture_screenshot_bytes() is now concrete: the default encodes capture()'s numpy frame via cv2.imencode, so every source automatically supports the bytes path without raising NotImplementedError. Backends with a native encoded path (Appium, Playwright, Selenium) still override for efficiency. - StrategyManager.capture_screenshot_bytes() is simplified: the NotImplementedError branch is removed (no longer needed), and a final OpticsError is raised if all strategies fail rather than silently falling back to a secondary numpy encode path that could itself fail. - execution_tracer.log_attempt() is now called for both success and failure in capture_screenshot_bytes(), consistent with capture_screenshot() and _try_strategy_locate(). - Tests updated to reflect new semantics: exhausted strategies raise OpticsError, and a new test verifies the interface default encode path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…clean up locator logging

sonarqubecloud · 2026-07-02T03:56:20Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

kun-codes · 2026-07-13T12:16:39Z

+
+    try:
+        results_list = list(results)
+    except OpticsError:


why is it swallowing OpticsError? Wouldn't this make debugging harder?

kun-codes · 2026-07-13T12:17:40Z

+        for attempt in range(2):
+            try:
+                return base64.b64decode(driver.get_screenshot_as_base64())
+            except Exception as e:


This exception is too wide as well

kun-codes · 2026-07-13T12:18:47Z

+        """
+        driver = self._require_driver()
+        last_exc: Optional[Exception] = None
+        for attempt in range(2):


I would suggest not using magic numbers like 2. Use a named constant for this

kun-codes · 2026-07-13T12:24:59Z

+        for attempt in range(2):
+            try:
+                return base64.b64decode(driver.get_screenshot_as_base64())
+            except Exception as e:


Exception is too wide

kun-codes · 2026-07-13T12:25:15Z

+        """
+        driver = self._require_driver()
+        last_exc: Optional[Exception] = None
+        for attempt in range(2):


We should use a named constant instead of 2 here

kun-codes · 2026-07-13T12:45:03Z

+                except NoSuchElementException:
+                    return None
+                except Exception as e:
+                    internal_logger.debug("Error finding element by index: %s: %s", xpath, e)


This should be an exception. Its for an error

kun-codes · 2026-07-13T13:03:58Z

        internal_logger.exception("SeleniumScreenshot does not support getting interactive elements.")
        raise NotImplementedError("SeleniumScreenshot does not support getting interactive elements.")

+    def capture_screenshot_bytes(self) -> bytes:


this and capture_screenshot_bytes() of appium_screenshot look similar

This should be extracted to a common place

kun-codes · 2026-07-13T13:04:25Z

 from optics_framework.common.logging_config import internal_logger

+# Delay between screenshot retries (Selenium Grid nodes can be remote/flaky too).
+_SCREENSHOT_RETRY_BACKOFF_S = 0.5


This constant and the one in appium_screenshot.py are duplicated

kun-codes · 2026-07-13T13:08:02Z

        try:
-            return utils.encode_numpy_to_png_bytes(image)
+            return self._action_keyword.strategy_manager.capture_screenshot_bytes()
        except ValueError as exc:  # pragma: no cover - defensive


the capture_screenshot_bytes() never raises a ValueError. It will raise a OpticsError with code E303

kun-codes · 2026-07-13T13:10:56Z

@@ -1,14 +1,16 @@
 from functools import wraps


Nothing wrong with the code but commits f079ea1 and 2b545fa do not use conventional commits

chinmayajha requested a review from thakur-patel June 15, 2026 16:37

chinmayajha self-assigned this Jun 22, 2026

malto101 reviewed Jun 26, 2026

View reviewed changes

Comment thread optics_framework/common/elementsource_interface.py

thakur-patel requested changes Jun 27, 2026

View reviewed changes

Comment thread optics_framework/common/strategies.py

chinmayajha changed the title ~~feat(live): native screenshot-bytes fast path for the NL LLM agent (driver-agnostic)~~ feat(live): refactor AI self-heal to use keyword-based ReAct loop and improve locator logging Jun 29, 2026

chinmayajha force-pushed the live-native-screenshot-bytes branch 2 times, most recently from bf85cfe to 80e721e Compare June 29, 2026 18:27

thakur-patel requested review from malto101 and thakur-patel June 30, 2026 20:00

chinmayajha force-pushed the live-native-screenshot-bytes branch from efbe0f8 to 555d18c Compare July 1, 2026 03:15

chinmayajha and others added 7 commits July 2, 2026 09:25

feat(live): refactor AI self-heal to use keyword-based execution and …

f302f41

…clean up locator logging

Apply pre-commit fixes

f079ea1

refactoring functions to reduce cognitive complexity

2b545fa

fix: remove trailing whitespace from ai_self_heal.py

e8b0d7d

chinmayajha force-pushed the live-native-screenshot-bytes branch from 97943f0 to e8b0d7d Compare July 2, 2026 03:55

kun-codes suggested changes Jul 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(live): refactor AI self-heal to use keyword-based ReAct loop and improve locator logging#303

feat(live): refactor AI self-heal to use keyword-based ReAct loop and improve locator logging#303
chinmayajha wants to merge 7 commits into
mozarkai:mainfrom
chinmayajha:live-native-screenshot-bytes

chinmayajha commented Jun 15, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud Bot commented Jul 2, 2026

Uh oh!

kun-codes Jul 13, 2026

Uh oh!

kun-codes Jul 13, 2026

Uh oh!

kun-codes Jul 13, 2026

Uh oh!

kun-codes Jul 13, 2026

Uh oh!

kun-codes Jul 13, 2026

Uh oh!

kun-codes Jul 13, 2026

Uh oh!

kun-codes Jul 13, 2026

Uh oh!

kun-codes Jul 13, 2026

Uh oh!

kun-codes Jul 13, 2026

Uh oh!

kun-codes Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

chinmayajha commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

1. AI Self-Heal Refactor (optics_framework/common/ai_self_heal.py)

2. Action Keyword Integration (optics_framework/api/action_keyword.py)

3. Log Cleanup for Silent Error Checker

4. Tests

Verification

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud Bot commented Jul 2, 2026

Quality Gate passed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

chinmayajha commented Jun 15, 2026 •

edited

Loading

1. AI Self-Heal Refactor (`optics_framework/common/ai_self_heal.py`)

2. Action Keyword Integration (`optics_framework/api/action_keyword.py`)