Skip to content

Commit 65f7adc

Browse files
ferpasriferpasri
committed
Make cli agent to use SKILL.md instructions
1 parent 3cbcc1c commit 65f7adc

4 files changed

Lines changed: 200 additions & 23 deletions

File tree

‎agent/resources/skills/testar-cli/SKILL.md‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
---
2+
name: testar-cli
3+
description: "Use when operating the distributed testar-cli launcher to execute goals through CLI commands, keep one session alive per goal, and drive SUTs step by step."
4+
---
5+
6+
# TESTAR CLI
7+
8+
Use this skill for the distributed `testar-cli` runtime that is bundled with TESTAR.
9+
10+
Treat the CLI distribution as an operational environment, not as a source-code workspace.
11+
12+
## When to use
13+
14+
- The user wants a goal executed through the installed `testar-cli` launcher.
15+
- The task requires step-by-step control of a live Windows, WebDriver, or Android session.
16+
- The task requires using `getState`, `getStateScreenshot`, and `getDerivedActions` as execution evidence.
17+
18+
## Runtime layout
19+
20+
- Distribution root contains the public launcher.
21+
- This skill is copied into `.agents/skills/testar-cli/` inside the CLI distribution.
22+
23+
## Public launcher paths
24+
25+
- Windows: `testar-cli.bat`
26+
- Linux or macOS: `./testar-cli`
27+
28+
## Public commands
29+
30+
- `sessionStatus`
31+
- `startSession windows <path>`
32+
- `startSession webdriver <url>`
33+
- `startSession android <apk>`
34+
- `getState`
35+
- `getStateScreenshot`
36+
- `getDerivedActions`
37+
- `executeAction click <semanticText>`
38+
- `executeAction type <semanticText> <inputText>`
39+
- `executeAction select <semanticText> <value>`
40+
- `stopSession`
41+
- `shutdownDaemon`
42+
43+
## Semantic action matching
44+
45+
For `executeAction click`, `executeAction type`, and WebDriver-only `executeAction select`, `<semanticText>` is intended to match the derived action description text.
46+
47+
Use short meaningful text that is likely to appear in the action description, for example:
48+
49+
- `executeAction click Open`
50+
- `executeAction type input_contact This_is_a_message`
51+
- `executeAction select select_amount 99999`
52+
53+
## Action availability
54+
55+
- `executeAction click` is available in Windows, WebDriver, and Android sessions.
56+
- `executeAction type` is available in Windows, WebDriver, and Android sessions.
57+
- `executeAction select` only makes sense in WebDriver sessions, where HTML select/dropdown elements can expose selectable values.
58+
- Do not use `executeAction select` in Windows or Android sessions. For Windows or Android dropdowns or combo boxes, inspect `getDerivedActions` and use the available derived `click` or `type` actions instead.
59+
60+
## Workflow
61+
62+
1. Work from the CLI distribution root.
63+
2. Start one session with the selected platform and target.
64+
3. Use `getState`, `getStateScreenshot`, and `getDerivedActions` to inspect the live UI.
65+
4. Execute actions one at a time.
66+
5. Re-check state for execution evidence.
67+
6. Stop the session when the goal is finished.
68+
7. Shutdown the daemon after stopping the session.
69+
70+
## Resilience and retry behavior rules
71+
72+
Some SUTs may need time to start, load screens, populate state information, or expose derived actions after an action is executed.
73+
74+
When `getState`, `getStateScreenshot`, or `getDerivedActions` fails, returns incomplete information, or does not yet reflect the expected UI change:
75+
76+
- Do not assume the goal failed immediately.
77+
- Retry the same observation command after a short wait.
78+
- Prefer re-running `getState` before `getDerivedActions` when the UI may still be loading.
79+
- Retry a small number of times before changing strategy.
80+
- Keep retries sequential; do not run overlapping commands.
81+
- Ground decisions in the latest successful command output.
82+
83+
If derived actions are missing or stale, first refresh the state with:
84+
85+
- `getState`
86+
- `getStateScreenshot`
87+
- `getDerivedActions`
88+
89+
Only execute an action after the relevant action appears in the latest `getDerivedActions` output.
90+
91+
## Session examples
92+
93+
Windows testing:
94+
95+
- `testar-cli.bat sessionStatus`
96+
- `testar-cli.bat startSession windows notepad.exe`
97+
- `testar-cli.bat getState`
98+
- `testar-cli.bat getStateScreenshot`
99+
- `testar-cli.bat getDerivedActions`
100+
- `testar-cli.bat executeAction click Open`
101+
- `testar-cli.bat executeAction type Editor Writing_text_in_Notepad`
102+
- `testar-cli.bat stopSession`
103+
- `testar-cli.bat shutdownDaemon`
104+
105+
WebDriver testing:
106+
107+
- `testar-cli.bat sessionStatus`
108+
- `testar-cli.bat startSession webdriver https://para.testar.org/`
109+
- `testar-cli.bat getState`
110+
- `testar-cli.bat getStateScreenshot`
111+
- `testar-cli.bat getDerivedActions`
112+
- `testar-cli.bat executeAction click a_about`
113+
- `testar-cli.bat executeAction type input_contact This_is_a_message`
114+
- `testar-cli.bat executeAction select select_amount 99999`
115+
- `testar-cli.bat stopSession`
116+
- `testar-cli.bat shutdownDaemon`
117+
118+
Android testing:
119+
120+
- `testar-cli.bat sessionStatus`
121+
- `testar-cli.bat startSession android ApiDemos-debug.apk`
122+
- `testar-cli.bat getState`
123+
- `testar-cli.bat getStateScreenshot`
124+
- `testar-cli.bat getDerivedActions`
125+
- `testar-cli.bat executeAction click Graphics`
126+
- `testar-cli.bat executeAction type username Writing_text_in_Android`
127+
- `testar-cli.bat stopSession`
128+
- `testar-cli.bat shutdownDaemon`
129+
130+
## Operational rules
131+
132+
- Use one stable session per goal when possible.
133+
- Run commands sequentially.
134+
- Treat `startSession`, `stopSession`, and `shutdownDaemon` as the public lifecycle commands.
135+
- Prefer observing the live state through CLI commands.
136+
- Keep evidence grounded in command outputs.

‎cli/build.gradle‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,10 @@ task prepareCliDistribution(type: Sync) {
4949
into 'settings'
5050
}
5151

52+
from(project(':agent').file('resources/skills/testar-cli')) {
53+
into '.agents/skills/testar-cli'
54+
}
55+
5256
from('resources/log4j2.xml') {
5357
into '.runtime'
5458
}

‎webstudio/frontend/src/App.svelte‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,8 @@
2626
approvalPolicy: "never",
2727
networkAccessEnabled: false,
2828
skipGitRepoCheck: true,
29-
promptTitle: "CLI Agent Goal",
30-
promptText: ""
29+
promptTitle: "Test Parabank Login",
30+
promptText: "As a test agent verify that you can log in with the credentials john/demo. Then the Welcome John Smith message is shown."
3131
};
3232
let scriptlessStatus = null;
3333
let loading = false;

‎webstudio/src/org/testar/webstudio/execution/CliExecutionAdapter.java‎

Lines changed: 58 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -433,16 +433,16 @@ private CliAgentSettingsDto normalizeAgentSettings(CliAgentSettingsDto settings)
433433

434434
private CliAgentSettingsDto defaultAgentSettings() {
435435
return new CliAgentSettingsDto(
436-
"OPENAI_API_KEY",
436+
"OPENAI_API",
437437
"",
438-
"gpt-5.4",
438+
"gpt-5.4-mini",
439439
"medium",
440-
"workspace-write",
440+
"danger-full-access",
441441
"never",
442442
Boolean.FALSE,
443443
Boolean.TRUE,
444-
"CLI Agent Goal",
445-
""
444+
"Test Parabank Login",
445+
"As a test agent verify that you can log in with the credentials john/demo. Then the Welcome John Smith message is shown."
446446
);
447447
}
448448

@@ -646,34 +646,28 @@ private String buildAgentPrompt(CliAgentSettingsDto settings,
646646
String platform,
647647
String target,
648648
Path cliLauncher) {
649+
Path skillDirectory = resolveCliSkillDirectory();
650+
Path skillFile = skillDirectory.resolve("SKILL.md");
651+
String skillInstructions = readCliSkillInstructions(skillDirectory);
649652
StringBuilder prompt = new StringBuilder();
650653
prompt.append(settings.promptTitle()).append('\n').append('\n');
651654
prompt.append("You are controlling TESTAR through testar-cli.").append('\n');
652-
prompt.append("Selected CLI profile: ").append(workspaceName).append('\n');
653655
prompt.append("Platform: ").append(platform).append('\n');
654656
prompt.append("Target: ").append(target).append('\n');
655657
prompt.append("TESTAR CLI launcher: ").append(cliLauncher).append('\n');
656-
prompt.append("Repository root: ").append(resolveProjectRoot()).append('\n').append('\n');
657-
prompt.append("Required execution rules:").append('\n');
658-
prompt.append("1. Start the session with the selected platform, target, and profile.").append('\n');
659-
prompt.append("2. Use only testar-cli commands to inspect state and execute actions.").append('\n');
660-
prompt.append("3. Prefer semantic commands such as getState, getDerivedActions, and executeAction.").append('\n');
661-
prompt.append("4. If you start a session, finish with stopSession and then shutdownDaemon.").append('\n');
662-
prompt.append("5. Report what you did and why.").append('\n').append('\n');
663-
prompt.append("Useful command examples:").append('\n');
664-
prompt.append("\"").append(cliLauncher).append("\" startSession ").append(platform).append(" ").append(target).append(" ").append(workspaceName).append('\n');
665-
prompt.append("\"").append(cliLauncher).append("\" sessionStatus").append('\n');
666-
prompt.append("\"").append(cliLauncher).append("\" getState").append('\n');
667-
prompt.append("\"").append(cliLauncher).append("\" getDerivedActions").append('\n');
668-
prompt.append("\"").append(cliLauncher).append("\" executeAction click \"Log In\"").append('\n');
669-
prompt.append("\"").append(cliLauncher).append("\" stopSession").append('\n');
670-
prompt.append("\"").append(cliLauncher).append("\" shutdownDaemon").append('\n').append('\n');
658+
prompt.append("Repository root: ").append(resolveProjectRoot()).append('\n');
659+
prompt.append("CLI skill directory: ").append(skillDirectory).append('\n');
660+
prompt.append("CLI skill file: ").append(skillFile).append('\n').append('\n');
661+
prompt.append("Execution rule: read and follow the TESTAR CLI skill instructions before issuing commands.").append('\n');
662+
prompt.append("Use the selected platform and target for this run.").append('\n').append('\n');
671663
if (!settings.apiKeyEnvVarName().isBlank()) {
672664
prompt.append("Expected API key environment variable for Codex authentication: ")
673665
.append(settings.apiKeyEnvVarName())
674666
.append('\n')
675667
.append('\n');
676668
}
669+
prompt.append("Authoritative TESTAR CLI skill instructions:").append('\n');
670+
prompt.append(skillInstructions).append('\n').append('\n');
677671
prompt.append("User goal:").append('\n');
678672
prompt.append(
679673
settings.promptText().isBlank()
@@ -687,6 +681,49 @@ private Path resolveProjectRoot() {
687681
return Paths.get(System.getProperty("user.dir")).toAbsolutePath().normalize();
688682
}
689683

684+
private Path resolveCliSkillDirectory() {
685+
Path skillDirectory = resolveCliInstallDirectory()
686+
.resolve(".agents")
687+
.resolve("skills")
688+
.resolve("testar-cli");
689+
690+
if (!Files.isDirectory(skillDirectory)) {
691+
throw new IllegalStateException(
692+
"CLI skill directory not found: " + skillDirectory + ". Run :cli:cliDistribution again."
693+
);
694+
}
695+
696+
return skillDirectory;
697+
}
698+
699+
private String readCliSkillInstructions(Path skillDirectory) {
700+
try (Stream<Path> files = Files.list(skillDirectory)) {
701+
List<Path> markdownFiles = files
702+
.filter(Files::isRegularFile)
703+
.filter(path -> path.getFileName().toString().toLowerCase().endsWith(".md"))
704+
.sorted(Comparator.comparing(path -> path.getFileName().toString()))
705+
.collect(Collectors.toList());
706+
707+
if (markdownFiles.isEmpty()) {
708+
throw new IllegalStateException("No CLI skill markdown files found in: " + skillDirectory);
709+
}
710+
711+
StringBuilder content = new StringBuilder();
712+
for (Path markdownFile : markdownFiles) {
713+
if (content.length() > 0) {
714+
content.append('\n').append('\n');
715+
}
716+
717+
content.append("File: ").append(markdownFile.getFileName()).append('\n').append('\n');
718+
content.append(Files.readString(markdownFile, StandardCharsets.UTF_8).trim());
719+
}
720+
721+
return content.toString();
722+
} catch (IOException exception) {
723+
throw new IllegalStateException("Unable to read CLI skill instructions from: " + skillDirectory, exception);
724+
}
725+
}
726+
690727
private Path resolveCliInstallDirectory() {
691728
debugLog.log("resolveCliInstallDirectory invoked");
692729
Path workingDirectory = Paths.get(System.getProperty("user.dir")).toAbsolutePath().normalize();

0 commit comments

Comments
 (0)