Migrate act() to conversation-based architecture with Speaker pattern and add caching v2 features.

.gitignore

@@ -169,5 +169,6 @@ reports/
 /askui_chat.db-shm
 /askui_chat.db-wal
 .cache/
+.askui_cache/*
 
 bom.json

README.md

@@ -123,6 +123,7 @@ Ready to build your first agent? Check out our documentation:
 9. **[Reporting](docs/08_reporting.md)** - Obtain agent logs as execution reports and summaries as test reports
 10. **[Observability](docs/09_observability_telemetry_tracing.md)** - Monitor and debug agents
 11. **[Extracting Data](docs/10_extracting_data.md)** - Extracting structured data from screenshots and files
+12. **[Callbacks](docs/11_callbacks.md)** - Inject custom logic into the control loop
 
 **Official documentation:** [docs.askui.com](https://docs.askui.com)
 

docs/00_overview.md

@@ -90,6 +90,9 @@ Understand what data is collected and how to opt out.
 ### 10 - Extracting Data
 **Topics**: Using `get()`, file support (PDF. Excel, Word, CSV), structured data extraction, response schemas
 
+### 11 - Callbacks
+**Topics**: Inject custom logic at different positions of the control loop through callbacks
+
 Extract information from screens and files using the `get()` method with Pydantic models.
 
 ## Additional Resources

docs/06_caching.md

@@ -8,12 +8,12 @@ The caching system works by recording all tool use actions (mouse movements, cli
 
 ## Caching Strategies
 
-The caching mechanism supports four strategies, configured via the `caching_settings` parameter in the `act()` method:
+The caching mechanism supports three strategies, configured via the `caching_settings` parameter in the `act()` method:
 
-- **`"no"`** (default): No caching is used. The agent executes normally without recording or replaying actions.
-- **`"write"`**: Records all agent actions to a cache file for future replay.
-- **`"read"`**: Provides tools to the agent to list and execute previously cached trajectories.
-- **`"both"`**: Combines read and write modes - the agent can use existing cached trajectories and will also record new ones.
+- **`None`** (default): No caching is used. The agent executes normally without recording or replaying actions.
+- **`"record"`**: Records all agent actions to a cache file for future replay.
+- **`"execute"`**: Provides tools to the agent to list and execute previously cached trajectories.
+- **`"auto"`**: Combines execute and record modes - the agent can use existing cached trajectories and will also record new ones.
 
 ## Configuration
 
@@ -23,20 +23,20 @@ Caching is configured using the `CachingSettings` class:
 from askui.models.shared.settings import CachingSettings, CachedExecutionToolSettings
 
 caching_settings = CachingSettings(
-    strategy="write",        # One of: "read", "write", "both", "no"
+    strategy="record",       # One of: "execute", "record", "auto", or None
     cache_dir=".cache",      # Directory to store cache files
-    filename="my_test.json", # Filename for the cache file (optional for write mode)
+    filename="my_test.json", # Filename for the cache file (optional for record mode)
     execute_cached_trajectory_tool_settings=CachedExecutionToolSettings(
-        delay_time_between_action=0.5  # Delay in seconds between each cached action
+        delay_time_between_actions=0.5  # Delay in seconds between each cached action
     )
 )
 ```
 
 ### Parameters
 
-- **`strategy`**: The caching strategy to use (`"read"`, `"write"`, `"both"`, or `"no"`).
+- **`strategy`**: The caching strategy to use (`"execute"`, `"record"`, `"auto"`, or `None`).
 - **`cache_dir`**: Directory where cache files are stored. Defaults to `".cache"`.
-- **`filename`**: Name of the cache file to write to or read from. If not specified in write mode, a timestamped filename will be generated automatically (format: `cached_trajectory_YYYYMMDDHHMMSSffffff.json`).
+- **`filename`**: Name of the cache file to write to or read from. If not specified in record mode, a timestamped filename will be generated automatically (format: `cached_trajectory_YYYYMMDDHHMMSSffffff.json`).
 - **`execute_cached_trajectory_tool_settings`**: Configuration for the trajectory execution tool (optional). See [Execution Settings](#execution-settings) below.
 
 ### Execution Settings
@@ -47,21 +47,21 @@ The `CachedExecutionToolSettings` class allows you to configure how cached traje
 from askui.models.shared.settings import CachedExecutionToolSettings
 
 execution_settings = CachedExecutionToolSettings(
-    delay_time_between_action=0.5  # Delay in seconds between each action (default: 0.5)
+    delay_time_between_actions=0.5  # Delay in seconds between each action (default: 0.5)
 )
 ```
 
 #### Parameters
 
-- **`delay_time_between_action`**: The time to wait (in seconds) between executing consecutive cached actions. This delay helps ensure UI elements can materialize before the next action is executed. Defaults to `0.5` seconds.
+- **`delay_time_between_actions`**: The time to wait (in seconds) between executing consecutive cached actions. This delay helps ensure UI elements can materialize before the next action is executed. Defaults to `0.5` seconds.
 
 You can adjust this value based on your application's responsiveness:
 - For faster applications or quick interactions, you might use a smaller delay (e.g., `0.1` or `0.2` seconds)
 - For slower applications or complex UI updates, you might need a longer delay (e.g., `1.0` or `2.0` seconds)
 
 ## Usage Examples
 
-### Writing a Cache (Recording)
+### Recording a Cache
 
 Record agent actions to a cache file for later replay:
 
@@ -73,7 +73,7 @@ with ComputerAgent() as agent:
     agent.act(
         goal="Fill out the login form with username 'admin' and password 'secret123'",
         caching_settings=CachingSettings(
-            strategy="write", # you could also use "both" here
+            strategy="record", # you could also use "auto" here
             cache_dir=".cache",
             filename="login_test.json"
         )
@@ -82,7 +82,7 @@ with ComputerAgent() as agent:
 
 After execution, a cache file will be created at `.cache/login_test.json` containing all the tool use actions performed by the agent.
 
-### Reading from Cache (Replaying)
+### Executing from Cache (Replaying)
 
 Provide the agent with access to previously recorded trajectories:
 
@@ -94,13 +94,13 @@ with ComputerAgent() as agent:
     agent.act(
         goal="Fill out the login form",
         caching_settings=CachingSettings(
-            strategy="read", # you could also use "both" here
+            strategy="execute", # you could also use "auto" here
             cache_dir=".cache"
         )
     )
 ```
 
-When using `strategy="read"`, the agent receives two additional tools:
+When using `strategy="execute"`, the agent receives two additional tools:
 
 1. **`retrieve_available_trajectories_tool`**: Lists all available cache files in the cache directory
 2. **`execute_cached_executions_tool`**: Executes a specific cached trajectory
@@ -109,7 +109,7 @@ The agent will automatically check if a relevant cached trajectory exists and us
 
 ### Referencing Cache Files in Goal Prompts
 
-When using `strategy="read"` or `strategy="both"`, **you need to inform the agent about which cache files are available and when to use them**. This is done by including cache file information directly in your goal prompt.
+When using `strategy="execute"` or `strategy="auto"`, **you need to inform the agent about which cache files are available and when to use them**. This is done by including cache file information directly in your goal prompt.
 
 #### Explicit Cache File References
 
@@ -126,7 +126,7 @@ with ComputerAgent() as agent:
         If the cache file "open_website_in_chrome.json" is available, please use it
         for this execution. It will open a new window in Chrome and navigate to the website.""",
         caching_settings=CachingSettings(
-            strategy="read",
+            strategy="execute",
             cache_dir=".cache"
         )
     )
@@ -149,7 +149,7 @@ with ComputerAgent() as agent:
         Check if a cache file named "{test_id}.json" exists. If it does, use it to
         replay the test actions, then verify the results.""",
         caching_settings=CachingSettings(
-            strategy="read",
+            strategy="execute",
             cache_dir="test_cache"
         )
     )
@@ -171,7 +171,7 @@ with ComputerAgent() as agent:
         Choose the most recent one if multiple are available, as it likely contains
         the most up-to-date interaction sequence.""",
         caching_settings=CachingSettings(
-            strategy="read",
+            strategy="execute",
             cache_dir=".cache"
         )
     )
@@ -195,7 +195,7 @@ with ComputerAgent() as agent:
 
         After each cached execution, verify the step completed successfully before proceeding.""",
         caching_settings=CachingSettings(
-            strategy="read",
+            strategy="execute",
             cache_dir=".cache"
         )
     )
@@ -219,10 +219,10 @@ with ComputerAgent() as agent:
     agent.act(
         goal="Fill out the login form",
         caching_settings=CachingSettings(
-            strategy="read",
+            strategy="execute",
             cache_dir=".cache",
             execute_cached_trajectory_tool_settings=CachedExecutionToolSettings(
-                delay_time_between_action=1.0  # Wait 1 second between each action
+                delay_time_between_actions=1.0  # Wait 1 second between each action
             )
         )
     )
@@ -233,7 +233,7 @@ This is particularly useful when:
 - UI elements take time to become interactive after appearing
 - You're testing on slower hardware or environments
 
-### Using Both Strategies
+### Using Auto Strategy
 
 Enable both reading and writing simultaneously:
 
@@ -245,7 +245,7 @@ with ComputerAgent() as agent:
     agent.act(
         goal="Complete the checkout process",
         caching_settings=CachingSettings(
-            strategy="both",
+            strategy="auto",
             cache_dir=".cache",
             filename="checkout_test.json"
         )
@@ -323,7 +323,7 @@ The delay between actions can be customized using `CachedExecutionToolSettings`
 ## Limitations
 
 - **UI State Sensitivity**: Cached trajectories assume the UI is in the same state as when they were recorded. If the UI has changed, the replay may fail or produce incorrect results.
-- **No on_message Callback**: When using `strategy="write"` or `strategy="both"`, you cannot provide a custom `on_message` callback, as the caching system uses this callback to record actions.
+- **No on_message Callback**: When using `strategy="record"` or `strategy="auto"`, you cannot provide a custom `on_message` callback, as the caching system uses this callback to record actions.
 - **Verification Required**: After executing a cached trajectory, the agent should verify that the results are correct, as UI changes may cause partial failures.
 
 ## Example: Complete Test Workflow
@@ -340,7 +340,7 @@ with ComputerAgent() as agent:
     agent.act(
         goal="Navigate to the login page and log in with username 'testuser' and password 'testpass123'",
         caching_settings=CachingSettings(
-            strategy="write",
+            strategy="record",
             cache_dir="test_cache",
             filename="user_login.json"
         )
@@ -356,10 +356,10 @@ with ComputerAgent() as agent:
         the login sequence. It contains the steps to navigate to the login page and
         authenticate with the test credentials.""",
         caching_settings=CachingSettings(
-            strategy="read",
+            strategy="execute",
             cache_dir="test_cache",
             execute_cached_trajectory_tool_settings=CachedExecutionToolSettings(
-                delay_time_between_action=1.0
+                delay_time_between_actions=1.0
             )
         )
     )

docs/11_callbacks.md

@@ -0,0 +1,82 @@
+# Callbacks
+
+Callbacks provide hooks into the agent's conversation lifecycle, similar to PyTorch Lightning's callback system. Use them for logging, monitoring, custom metrics, or extending agent behavior.
+
+## Usage
+
+Subclass `ConversationCallback` and override the hooks you need:
+
+```python
+from askui import ComputerAgent, ConversationCallback
+
+class MetricsCallback(ConversationCallback):
+    def on_step_start(self, conversation, step_index):
+        print(f"Step {step_index} starting...")
+
+    def on_step_end(self, conversation, step_index, result):
+        print(f"Step {step_index} finished: {result.status}")
+
+with ComputerAgent(callbacks=[MetricsCallback()]) as agent:
+    agent.act("Open the settings menu")
+```
+
+## Available Hooks
+
+| Hook | When Called | Parameters |
+|------|-------------|------------|
+| `on_conversation_start` | After setup, before control loop | `conversation` |
+| `on_conversation_end` | After control loop, before cleanup | `conversation` |
+| `on_control_loop_start` | Before the iteration loop begins | `conversation` |
+| `on_control_loop_end` | After the iteration loop ends | `conversation` |
+| `on_step_start` | Before each step execution | `conversation`, `step_index` |
+| `on_step_end` | After each step execution | `conversation`, `step_index`, `result` |
+| `on_tool_execution_start` | Before tools are executed | `conversation`, `tool_names` |
+| `on_tool_execution_end` | After tools are executed | `conversation`, `tool_names` |
+
+### Parameters
+
+- **`conversation`**: The `Conversation` instance with access to messages, settings, and state
+- **`step_index`**: Zero-based index of the current step
+- **`result`**: `SpeakerResult` containing `status`, `messages_to_add`, and `usage`
+- **`tool_names`**: List of tool names being executed
+
+## Example: Timing Callback
+
+```python
+import time
+from askui import ComputerAgent, ConversationCallback
+
+class TimingCallback(ConversationCallback):
+    def __init__(self):
+        self.start_time = None
+        self.step_times = []
+
+    def on_conversation_start(self, conversation):
+        self.start_time = time.time()
+
+    def on_step_start(self, conversation, step_index):
+        self._step_start = time.time()
+
+    def on_step_end(self, conversation, step_index, result):
+        elapsed = time.time() - self._step_start
+        self.step_times.append(elapsed)
+        print(f"Step {step_index}: {elapsed:.2f}s")
+
+    def on_conversation_end(self, conversation):
+        total = time.time() - self.start_time
+        print(f"Total: {total:.2f}s across {len(self.step_times)} steps")
+
+with ComputerAgent(callbacks=[TimingCallback()]) as agent:
+    agent.act("Search for documents")
+```
+
+## Multiple Callbacks
+
+Pass multiple callbacks to combine behaviors:
+
+```python
+with ComputerAgent(callbacks=[TimingCallback(), MetricsCallback()]) as agent:
+    agent.act("Complete the form")
+```
+
+Callbacks are called in the order they are provided.