Skip to content

Commit 8345dc6

Browse files
CakeRepositoryCakeRepository
committed
Enhance action execution prompts and improve chat client configuration
- Updated action prompt in LMStudioActioner to include explicit execution instructions for users. - Added a workflow diagram to the Multi-Agent Architecture documentation to illustrate agent interactions and processes. - Improved logging for function invocation in the chat client setup.
1 parent 51fd935 commit 8345dc6

3 files changed

Lines changed: 193 additions & 5 deletions

File tree

‎FlowVision/lib/Classes/ai/LMStudioActioner.cs‎

Lines changed: 25 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -83,8 +83,16 @@ public async Task<string> ExecuteAction(string actionPrompt)
8383
// Add system message to actioner history
8484
actionerHistory.Add(new ChatMessage(ChatRole.System, toolConfig.ActionerSystemPrompt + toolDescriptions));
8585

86-
// Add action prompt to actioner history
87-
actionerHistory.Add(new ChatMessage(ChatRole.User, actionPrompt));
86+
// Add action prompt with explicit instruction to EXECUTE
87+
string enhancedPrompt = $@"{actionPrompt}
88+
89+
IMPORTANT REMINDER:
90+
1. DO NOT just observe and describe - you must EXECUTE the action!
91+
2. After CaptureWholeScreen(), you MUST continue to actually click/type/interact
92+
3. Follow ALL steps: Observe → Plan → EXECUTE → Verify
93+
4. Do not stop until you've performed the actual action requested";
94+
95+
actionerHistory.Add(new ChatMessage(ChatRole.User, enhancedPrompt));
8896

8997
// Verify LM Studio is enabled and configured
9098
if (!lmStudioConfig.Enabled)
@@ -162,9 +170,21 @@ public async Task<string> ExecuteAction(string actionPrompt)
162170
};
163171

164172
// Build chat client with function invocation if enabled
165-
actionerChat = toolConfig.AutoInvokeKernelFunctions
166-
? new ChatClientBuilder(baseChatClient).UseFunctionInvocation().Build()
167-
: baseChatClient;
173+
if (toolConfig.AutoInvokeKernelFunctions)
174+
{
175+
// Use function invocation with default behavior
176+
// The middleware will automatically loop and call tools
177+
actionerChat = new ChatClientBuilder(baseChatClient)
178+
.UseFunctionInvocation()
179+
.Build();
180+
181+
PluginLogger.LogInfo("LMStudioActioner", "ExecuteAction",
182+
"Function invocation enabled - will auto-invoke tools");
183+
}
184+
else
185+
{
186+
actionerChat = baseChatClient;
187+
}
168188

169189
// Update loading message to show we're now processing the response
170190
PluginLogger.StopLoadingIndicator();

‎docs/Prompt-Engineering-Guide.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
---
2+
layout: default
3+
title: Prompt Engineering Guide
4+
---
5+
6+
# Prompt Engineering Guide for Recursive Control
7+
8+
Master the art of communicating with your AI agents to achieve precise, efficient automation.
9+
10+
## Why Prompt Engineering Matters
11+
Recursive Control's multi-agent system (Hermes, Daedalus, and Talos) interprets natural language, but structured prompts yield better results:
12+
- **Clarity reduces errors**: Ambiguous requests lead to wrong assumptions.
13+
- **Structure aids planning**: Well-organized prompts help the Planner agent create better steps.
14+
- **Context improves accuracy**: Provide relevant details for better execution.
15+
16+
## Core Principles
17+
1. **Be Specific**: Include exact details like app names, file paths, or expected outcomes.
18+
2. **Break It Down**: For complex tasks, suggest steps or use multi-stage prompts.
19+
3. **Provide Context**: Mention current state, like open windows or recent actions.
20+
4. **Use Verification**: Ask the agent to confirm steps or describe what it sees.
21+
5. **Handle Errors Gracefully**: Include fallback instructions.
22+
23+
## Prompt Patterns
24+
### Basic Command
25+
**Template**: "Perform [action] in [location/app] with [details]."
26+
27+
**Example**: "Open Chrome and navigate to github.com/flowdevs-io/Recursive-Control."
28+
29+
### Multi-Step Workflow
30+
**Template**: "Do the following steps: 1. [Step 1] 2. [Step 2] ... Verify each step."
31+
32+
**Example**: "Create a report: 1. Open Excel. 2. Add headers: Date, Task, Status. 3. Fill with today's data. 4. Save as 'daily-report.xlsx' in Documents."
33+
34+
### Vision-Assisted
35+
**Template**: "Capture the screen, describe what you see, then [action based on description]."
36+
37+
**Example**: "Take a screenshot of the current window, identify the search bar, and type 'AI tools' into it."
38+
39+
### Conditional Logic
40+
**Template**: "If [condition], do [action A]; else do [action B]."
41+
42+
**Example**: "If Chrome is open, navigate to YouTube; else open Chrome first then go to YouTube."
43+
44+
## Advanced Techniques
45+
### Chain of Thought
46+
Encourage reasoning: "Think step-by-step: First, check if the app is open. If not, open it. Then..."
47+
48+
### Role Playing
49+
Assign roles: "As a efficient automation expert, optimize this workflow: [task]."
50+
51+
### Few-Shot Examples
52+
Provide samples: "Like how you opened Notepad last time, now open Paint and draw a square."
53+
54+
## Common Pitfalls
55+
- **Too Vague**: "Do something with files" → Agents might guess wrong.
56+
- **Overly Complex**: Break long prompts into multiple interactions.
57+
- **Assuming State**: Always verify: "Focus on the foreground window and describe it first."
58+
59+
## Tips for Power Users
60+
- Use the UI's chat history to build context across prompts.
61+
- Combine plugins explicitly: "Use Playwright to automate browser, then CMD to process downloads."
62+
- Test incrementally: Start with simple tasks and build up.
63+
64+
Master these patterns to turn Recursive Control into your ultimate productivity copilot!

‎docs/Workflow-Builder-Tutorial.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
---
2+
layout: default
3+
title: Workflow Builder Tutorial
4+
---
5+
6+
# Workflow Builder Tutorial: Gamified Learning Adventures
7+
8+
Level up your Recursive Control skills by building interactive workflows! This tutorial turns learning into a game: complete "quests" by crafting prompt chains that automate real tasks. Each quest includes objectives, step-by-step guidance, and verification challenges.
9+
10+
Think of this as a simulator – test your prompts here before running them in the app. Earn "badges" by successfully completing each workflow (self-assessed via expected outcomes).
11+
12+
## Quest 1: GitHub Issue Automator (Beginner Level)
13+
**Objective**: Automate creating a GitHub issue in your repo using browser automation and keyboard inputs.
14+
15+
**Badge**: Issue Master
16+
17+
**Step-by-Step Prompt Chain**:
18+
1. **Launch the Browser**: "Open Chrome and navigate to github.com/login."
19+
- *Expected*: Browser opens to GitHub login page. (Verify: Describe the screen to confirm.)
20+
21+
2. **Login**: "Focus on the username field and type 'yourusername', then tab to password and type 'yourpassword', then press enter."
22+
- *Tip*: Use KeyboardPlugin for targeted input. (Challenge: Add verification – "If login fails, notify me.")
23+
24+
3. **Navigate to Repo**: "Go to github.com/yourusername/your-repo/issues."
25+
- *Expected*: Issues page loads.
26+
27+
4. **Create Issue**: "Click the 'New issue' button, type 'Bug: App crashes on load' in title, add description 'Steps to reproduce: 1. Open app. 2. Click button.', then submit."
28+
- *Challenge*: Use ScreenCapture to identify the button's bounding box before clicking.
29+
30+
**Full Chain Prompt** (Copy-Paste Ready):
31+
```
32+
Perform these steps to create a GitHub issue:
33+
1. Open Chrome and go to github.com/login.
34+
2. Log in with username 'yourusername' and password 'yourpassword'.
35+
3. Navigate to github.com/yourusername/your-repo/issues.
36+
4. Click 'New issue', fill title 'Bug: App crashes', description 'Steps: Open app, click button', and submit.
37+
Verify each step with a screenshot description.
38+
```
39+
40+
**Verification Quest**: Run this in Recursive Control. Did it create the issue? If not, refine the prompt (e.g., handle 2FA).
41+
42+
## Quest 2: Daily Report Generator (Intermediate Level)
43+
**Objective**: Automate opening Excel, filling data, and saving a report.
44+
45+
**Badge**: Report Wizard
46+
47+
**Step-by-Step Prompt Chain**:
48+
1. **Open App**: "Launch Excel and create a new spreadsheet."
49+
2. **Add Headers**: "Type 'Date' in A1, 'Task' in B1, 'Status' in C1."
50+
3. **Fill Data**: "In A2 type today's date, B2 'Implement feature X', C2 'Completed'."
51+
4. **Save**: "Save as 'daily-report.xlsx' in Documents."
52+
53+
**Full Chain Prompt**:
54+
```
55+
Build a daily report in Excel:
56+
1. Open Excel, new file.
57+
2. Headers: A1=Date, B1=Task, C1=Status.
58+
3. Data: A2=today's date, B2=Review code, C2=Done.
59+
4. Save to Documents as daily-report.xlsx.
60+
Confirm each cell after typing.
61+
```
62+
63+
**Challenge**: Add conditional logic – "If file exists, append instead of overwrite."
64+
65+
## Quest 3: File Organizer Bot (Advanced Level)
66+
**Objective**: Scan Downloads folder, organize files by type using CMD/PowerShell.
67+
68+
**Badge**: Organization Overlord
69+
70+
**Step-by-Step Prompt Chain**:
71+
1. **Scan Folder**: "Use CMD to list files in Downloads."
72+
2. **Create Folders**: "Make directories: Images, Documents, Others."
73+
3. **Move Files**: "Move .jpg to Images, .pdf to Documents, others to Others."
74+
4. **Verify**: "List contents of each new folder."
75+
76+
**Full Chain Prompt**:
77+
```
78+
Organize Downloads:
79+
1. CMD: dir %USERPROFILE%\Downloads
80+
2. Create folders: mkdir Images Documents Others
81+
3. Move: move *.jpg Images, move *.pdf Documents, move *.* Others
82+
4. Verify: dir Images, dir Documents, dir Others
83+
Handle errors if folders exist.
84+
```
85+
86+
**Epic Challenge**: Integrate vision – "Screenshot Downloads folder and describe file icons before moving."
87+
88+
## Level Up Tips
89+
- **Gamification Hack**: Track your success rate. Aim for 100% on 5 quests to "unlock" custom workflow creation.
90+
- **Combine Quests**: Chain them, e.g., "Generate report, then create GitHub issue about it."
91+
- **Debug Mode**: If a step fails, add "Describe the screen and suggest fixes" to your prompts.
92+
93+
## Go Interactive: Jupyter Notebook Simulator
94+
For hands-on practice, download our [Interactive Workflow Builder Notebook](tutorials/Workflow-Builder-Notebook.ipynb). It includes widgets to build and simulate prompt chains without running the full app!
95+
96+
**How to Use**:
97+
1. Install Jupyter: `pip install notebook`
98+
2. Download the .ipynb file.
99+
3. Run `jupyter notebook` and open the file.
100+
4. Interact with widgets to test prompts in real-time.
101+
102+
Completed all quests? Share your custom workflows on Discord!
103+
104+
Back to [Getting Started](Getting-Started.html)

0 commit comments

Comments
 (0)