feat(live): incremental /save and NL curation for cleaner recordings#327
Open
chinmayajha wants to merge 3 commits into
Open
feat(live): incremental /save and NL curation for cleaner recordings#327chinmayajha wants to merge 3 commits into
chinmayajha wants to merge 3 commits into
Conversation
…onflicts The /save command now takes two arguments: /save <test_case> <module_name> and writes to fixed-name CSVs (modules/modules.csv, test_cases/test_cases.csv, elements/elements.csv) that are APPENDED to when they already exist, enabling a 'one module at a time' pitch workflow. If a name conflicts, the save is refused with a prompt; re-running the identical /save confirms the append. The recording buffer is cleared after each /save, so the next keywords form the next module. This prevents cross-module duplication and enforces a clean separation of concerns. A header-only elements.csv stub is created if missing, so the standard file structure is always present for manual editing. Implements the user's requested workflow: build a test suite interactively, one module per /save, without accidental clobbering or name confusion.
When an AI instruction completes (status='done'), run an extra text-only LLM pass that prunes the recorded steps to the minimal subset reproducing the goal. This drops the agent's dead-ends, backtracks (e.g., navigate then press back), and no-op gestures—all of which report ok=True mechanically but don't contribute to the outcome. The result is a clean, replayable test module, not a transcript of every attempt. Curation is behind a curate_on_done flag (default=False on NaturalLanguageAgent to keep it a pure primitive; enabled in LiveController so /save gets a clean script). Safe fallback: any curation error or unusable response falls back to keeping all successful steps, so a working recording is never lost. Incomplete runs (failed/exhausted/aborted) record nothing, as before. Includes: - CURATION_SCHEMA: flat JSON schema (Gemini-safe, no anyOf). - _curate_successful_steps: text-only LLM call; validates indices; safely falls back to "keep all" on any anomaly. - _build_curation_prompt: candidate successful steps numbered 1-based (mapping to state.successful indices); failed steps as non-selectable context. - Tests: 13 new cases covering prune, normalisation, all fallback paths, LLM errors, skip conditions, toggle-off, and failed-step exclusion. The "kept N of M steps" note appears in the NLSummary message so the TUI shows pruning happened.
This was referenced Jul 1, 2026
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…s in test The _Controller test class intentionally bypasses LiveController.__init__ since we're unit-testing save() without a real session. Add a noqa comment and docstring to clarify this is deliberate.
|
| pytestmark = pytest.mark.white_box | ||
|
|
||
|
|
||
| class _Controller(LiveController): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two complementary improvements to the optics live workflow:
Incremental /save with fixed filenames:
/save <test_case> <module_name>now writes to standard files (modules/modules.csv, test_cases/test_cases.csv, elements/elements.csv) that APPEND when they exist. This enables building a test suite one module at a time without name confusion. If a name conflicts, you're prompted to choose a different one or confirm the append. The recording buffer is cleared after each /save so the next actions form the next module.NL-mode curation on completion: When an AI instruction finishes successfully, an extra LLM pass prunes the recorded steps to a minimal replayable script—dropping dead-ends, backtracks, and no-op gestures. If curation fails or produces no usable result, it safely falls back to keeping all successful steps.
Together, these make it practical to interactively build a clean test suite from AI-driven recordings, one focused module per instruction.
Test Plan
Follow-up Work
Two enhancements identified for future PRs:
doneto confirm the model's claim actually matches the screen, catching hallucinations before they're saved.assert_presencestep to saved modules so the test validates the outcome, not just the actions.These would make AI-generated tests even more robust for CI/automation.