Skip to content

Commit 4d07359

Browse files
abrichrWright Bot
abrichr
and
Wright Bot
authored
feat: Update README.md by combining the best changes from branches (#996)
Co-authored-by: Wright Bot <wright@openadaptai.noreply.github.com>
1 parent a6eb532 commit 4d07359

File tree

2 files changed

+4631
-182
lines changed

2 files changed

+4631
-182
lines changed

‎README.md‎

Lines changed: 62 additions & 182 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
[![Downloads](https://img.shields.io/pypi/dm/openadapt.svg)](https://pypi.org/project/openadapt/)
66
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
77
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/downloads/)
8+
[![Discord](https://img.shields.io/discord/1084481804896374814?color=7289da&label=Discord&logo=discord&logoColor=white)](https://discord.gg/yF527cQbDG)
89

910
**OpenAdapt** is the **open** source software **adapt**er between Large Multimodal Models (LMMs) and traditional desktop and web GUIs.
1011

@@ -28,6 +29,14 @@ OpenAdapt v1.0+ uses a **modular meta-package architecture**. The main `openadap
2829
| `openadapt-grounding` | UI element localization | [openadapt-grounding](https://github.com/OpenAdaptAI/openadapt-grounding) |
2930
| `openadapt-retrieval` | Multimodal demo retrieval | [openadapt-retrieval](https://github.com/OpenAdaptAI/openadapt-retrieval) |
3031
| `openadapt-privacy` | PII/PHI scrubbing | [openadapt-privacy](https://github.com/OpenAdaptAI/openadapt-privacy) |
32+
| `openadapt-wright` | Dev automation | [openadapt-wright](https://github.com/OpenAdaptAI/openadapt-wright) |
33+
| `openadapt-herald` | Social media from git history | [openadapt-herald](https://github.com/OpenAdaptAI/openadapt-herald) |
34+
| `openadapt-crier` | Telegram approval bot | [openadapt-crier](https://github.com/OpenAdaptAI/openadapt-crier) |
35+
| `openadapt-consilium` | Multi-model consensus | [openadapt-consilium](https://github.com/OpenAdaptAI/openadapt-consilium) |
36+
| `openadapt-desktop` | Desktop GUI application | [openadapt-desktop](https://github.com/OpenAdaptAI/openadapt-desktop) |
37+
| `openadapt-tray` | System tray app | [openadapt-tray](https://github.com/OpenAdaptAI/openadapt-tray) |
38+
| `openadapt-agent` | Production execution engine | [openadapt-agent](https://github.com/OpenAdaptAI/openadapt-agent) |
39+
| `openadapt-telemetry` | Error tracking | [openadapt-telemetry](https://github.com/OpenAdaptAI/openadapt-telemetry) |
3140

3241
---
3342

@@ -77,6 +86,36 @@ openadapt capture view my-task
7786

7887
---
7988

89+
## Ecosystem
90+
91+
### Core Platform Components
92+
93+
| Package | Description | Repository |
94+
|---------|-------------|------------|
95+
| `openadapt` | Meta-package with unified CLI | This repo |
96+
| `openadapt-capture` | Event recording and storage | [openadapt-capture](https://github.com/OpenAdaptAI/openadapt-capture) |
97+
| `openadapt-ml` | ML engine, training, inference | [openadapt-ml](https://github.com/OpenAdaptAI/openadapt-ml) |
98+
| `openadapt-evals` | Benchmark evaluation | [openadapt-evals](https://github.com/OpenAdaptAI/openadapt-evals) |
99+
| `openadapt-viewer` | HTML visualization | [openadapt-viewer](https://github.com/OpenAdaptAI/openadapt-viewer) |
100+
| `openadapt-grounding` | UI element localization | [openadapt-grounding](https://github.com/OpenAdaptAI/openadapt-grounding) |
101+
| `openadapt-retrieval` | Multimodal demo retrieval | [openadapt-retrieval](https://github.com/OpenAdaptAI/openadapt-retrieval) |
102+
| `openadapt-privacy` | PII/PHI scrubbing | [openadapt-privacy](https://github.com/OpenAdaptAI/openadapt-privacy) |
103+
104+
### Applications and Tools
105+
106+
| Package | Description | Repository |
107+
|---------|-------------|------------|
108+
| `openadapt-desktop` | Desktop GUI application | [openadapt-desktop](https://github.com/OpenAdaptAI/openadapt-desktop) |
109+
| `openadapt-tray` | System tray app | [openadapt-tray](https://github.com/OpenAdaptAI/openadapt-tray) |
110+
| `openadapt-agent` | Production execution engine | [openadapt-agent](https://github.com/OpenAdaptAI/openadapt-agent) |
111+
| `openadapt-wright` | Dev automation | [openadapt-wright](https://github.com/OpenAdaptAI/openadapt-wright) |
112+
| `openadapt-herald` | Social media from git history | [openadapt-herald](https://github.com/OpenAdaptAI/openadapt-herald) |
113+
| `openadapt-crier` | Telegram approval bot | [openadapt-crier](https://github.com/OpenAdaptAI/openadapt-crier) |
114+
| `openadapt-consilium` | Multi-model consensus | [openadapt-consilium](https://github.com/OpenAdaptAI/openadapt-consilium) |
115+
| `openadapt-telemetry` | Error tracking | [openadapt-telemetry](https://github.com/OpenAdaptAI/openadapt-telemetry) |
116+
117+
---
118+
80119
## CLI Reference
81120

82121
```
@@ -106,184 +145,24 @@ See the full [Architecture Evolution](docs/architecture-evolution.md) for detail
106145

107146
### Three-Phase Pipeline
108147

109-
```mermaid
110-
flowchart TB
111-
%% ═══════════════════════════════════════════════════════════════════════
112-
%% DATA SOURCES (Multi-Source Ingestion)
113-
%% ═══════════════════════════════════════════════════════════════════════
114-
subgraph DataSources["Data Sources"]
115-
direction LR
116-
HUMAN["Human Demos"]
117-
SYNTH["Synthetic Data"]:::future
118-
BENCH_DATA["Benchmark Tasks"]
119-
end
120-
121-
%% ═══════════════════════════════════════════════════════════════════════
122-
%% PHASE 1: DEMONSTRATE (Observation Collection)
123-
%% ═══════════════════════════════════════════════════════════════════════
124-
subgraph Demonstrate["1. DEMONSTRATE (Observation Collection)"]
125-
direction TB
126-
CAP["Capture<br/>openadapt-capture"]
127-
PRIV["Privacy<br/>openadapt-privacy"]
128-
STORE[("Demo Library")]
129-
130-
CAP --> PRIV
131-
PRIV --> STORE
132-
end
133-
134-
%% ═══════════════════════════════════════════════════════════════════════
135-
%% PHASE 2: LEARN (Policy Acquisition)
136-
%% ═══════════════════════════════════════════════════════════════════════
137-
subgraph Learn["2. LEARN (Policy Acquisition)"]
138-
direction TB
139-
140-
subgraph RetrievalPath["Retrieval Path"]
141-
EMB["Embed"]
142-
IDX["Index"]
143-
SEARCH["Search"]
144-
EMB --> IDX --> SEARCH
145-
end
146-
147-
subgraph TrainingPath["Training Path"]
148-
LOADER["Load"]
149-
TRAIN["Train"]
150-
CKPT[("Checkpoint")]
151-
LOADER --> TRAIN --> CKPT
152-
end
153-
154-
subgraph ProcessMining["Process Mining"]
155-
ABSTRACT["Abstract"]:::future
156-
PATTERNS["Patterns"]:::future
157-
ABSTRACT --> PATTERNS
158-
end
159-
end
160-
161-
%% ═══════════════════════════════════════════════════════════════════════
162-
%% PHASE 3: EXECUTE (Agent Deployment)
163-
%% ═══════════════════════════════════════════════════════════════════════
164-
subgraph Execute["3. EXECUTE (Agent Deployment)"]
165-
direction TB
166-
167-
subgraph AgentCore["Agent Core"]
168-
OBS["Observe"]
169-
POLICY["Policy<br/>(Demo-Conditioned)"]
170-
GROUND["Grounding<br/>openadapt-grounding"]
171-
ACT["Act"]
172-
173-
OBS --> POLICY
174-
POLICY --> GROUND
175-
GROUND --> ACT
176-
end
177-
178-
subgraph SafetyGate["Safety Gate"]
179-
VALIDATE["Validate"]
180-
CONFIRM["Confirm"]:::future
181-
VALIDATE --> CONFIRM
182-
end
183-
184-
subgraph Evaluation["Evaluation"]
185-
EVALS["Evals<br/>openadapt-evals"]
186-
METRICS["Metrics"]
187-
EVALS --> METRICS
188-
end
189-
190-
ACT --> VALIDATE
191-
VALIDATE --> EVALS
192-
end
193-
194-
%% ═══════════════════════════════════════════════════════════════════════
195-
%% THE ABSTRACTION LADDER (Side Panel)
196-
%% ═══════════════════════════════════════════════════════════════════════
197-
subgraph AbstractionLadder["Abstraction Ladder"]
198-
direction TB
199-
L0["Literal<br/>(Raw Events)"]
200-
L1["Symbolic<br/>(Semantic Actions)"]
201-
L2["Template<br/>(Parameterized)"]
202-
L3["Semantic<br/>(Intent)"]:::future
203-
L4["Goal<br/>(Task Spec)"]:::future
204-
205-
L0 --> L1
206-
L1 --> L2
207-
L2 -.-> L3
208-
L3 -.-> L4
209-
end
210-
211-
%% ═══════════════════════════════════════════════════════════════════════
212-
%% MODEL LAYER
213-
%% ═══════════════════════════════════════════════════════════════════════
214-
subgraph Models["Model Layer (VLMs)"]
215-
direction TB
216-
subgraph APIModels["API Models"]
217-
direction LR
218-
CLAUDE["Claude"]
219-
GPT["GPT-4o"]
220-
GEMINI["Gemini"]
221-
end
222-
subgraph OpenSource["Open Source / Fine-tuned"]
223-
direction LR
224-
QWEN3["Qwen3-VL"]
225-
UITARS["UI-TARS"]
226-
OPENCUA["OpenCUA"]
227-
end
228-
end
229-
230-
%% ═══════════════════════════════════════════════════════════════════════
231-
%% MAIN DATA FLOW
232-
%% ═══════════════════════════════════════════════════════════════════════
233-
234-
%% Data sources feed into phases
235-
HUMAN --> CAP
236-
SYNTH -.-> LOADER
237-
BENCH_DATA --> EVALS
238-
239-
%% Demo library feeds learning
240-
STORE --> EMB
241-
STORE --> LOADER
242-
STORE -.-> ABSTRACT
243-
244-
%% Learning outputs feed execution
245-
SEARCH -->|"demo context"| POLICY
246-
CKPT -->|"trained policy"| POLICY
247-
PATTERNS -.->|"templates"| POLICY
248-
249-
%% Model connections
250-
POLICY --> Models
251-
GROUND --> Models
252-
253-
%% ═══════════════════════════════════════════════════════════════════════
254-
%% FEEDBACK LOOPS (Evaluation-Driven)
255-
%% ═══════════════════════════════════════════════════════════════════════
256-
METRICS -->|"success traces"| STORE
257-
METRICS -.->|"training signal"| TRAIN
258-
259-
%% Retrieval in BOTH training AND evaluation
260-
SEARCH -->|"eval conditioning"| EVALS
261-
262-
%% ═══════════════════════════════════════════════════════════════════════
263-
%% STYLING
264-
%% ═══════════════════════════════════════════════════════════════════════
265-
266-
%% Phase colors
267-
classDef phase1 fill:#3498DB,stroke:#1A5276,color:#fff
268-
classDef phase2 fill:#27AE60,stroke:#1E8449,color:#fff
269-
classDef phase3 fill:#9B59B6,stroke:#6C3483,color:#fff
270-
271-
%% Component states
272-
classDef implemented fill:#2ECC71,stroke:#1E8449,color:#fff
273-
classDef future fill:#95A5A6,stroke:#707B7C,color:#fff,stroke-dasharray: 5 5
274-
classDef futureBlock fill:#f5f5f5,stroke:#95A5A6,stroke-dasharray: 5 5
275-
classDef safetyBlock fill:#E74C3C,stroke:#A93226,color:#fff
276-
277-
%% Model layer
278-
classDef models fill:#F39C12,stroke:#B7950B,color:#fff
279-
280-
%% Apply styles
281-
class CAP,PRIV,STORE phase1
282-
class EMB,IDX,SEARCH,LOADER,TRAIN,CKPT phase2
283-
class OBS,POLICY,GROUND,ACT,VALIDATE,EVALS,METRICS phase3
284-
class CLAUDE,GPT,GEMINI,QWEN models
285-
class L0,L1,L2 implemented
286-
```
148+
OpenAdapt follows a streamlined **Demonstrate → Learn → Execute** pipeline:
149+
150+
**1. DEMONSTRATE (Observation Collection)**
151+
- **Capture**: Record user actions and screenshots with `openadapt-capture`
152+
- **Privacy**: Scrub PII/PHI from recordings with `openadapt-privacy`
153+
- **Store**: Build a searchable demonstration library
154+
155+
**2. LEARN (Policy Acquisition)**
156+
- **Retrieval Path**: Embed demonstrations, index them, and enable semantic search
157+
- **Training Path**: Load demonstrations and fine-tune Vision-Language Models (VLMs)
158+
- **Abstraction**: Progress from literal replay to template-based automation
159+
160+
**3. EXECUTE (Agent Deployment)**
161+
- **Observe**: Take screenshots and gather accessibility information
162+
- **Policy**: Use demonstration context to decide actions via VLMs (Claude, GPT-4o, Qwen3-VL)
163+
- **Ground**: Map intentions to specific UI coordinates with `openadapt-grounding`
164+
- **Act**: Execute validated actions with safety gates
165+
- **Evaluate**: Measure success with `openadapt-evals` and feed results back for improvement
287166

288167
### Core Approach: Demo-Conditioned Prompting
289168

@@ -305,8 +184,6 @@ OpenAdapt explores **demonstration-conditioned automation** - "show, don't tell"
305184
- **Abstraction Ladder**: Progressive generalization from literal replay to goal-level automation
306185
- **Evaluation-Driven Feedback**: Success traces become new training data
307186

308-
**Legend:** Solid = Implemented | Dashed = Future
309-
310187
---
311188

312189
## Terminology
@@ -324,8 +201,11 @@ OpenAdapt explores **demonstration-conditioned automation** - "show, don't tell"
324201

325202
## Demos
326203

327-
- https://twitter.com/abrichr/status/1784307190062342237
328-
- https://www.loom.com/share/9d77eb7028f34f7f87c6661fb758d1c0
204+
**Legacy Version (v0.46.0) Examples:**
205+
- [Twitter Demo](https://twitter.com/abrichr/status/1784307190062342237) - Early OpenAdapt demonstration
206+
- [Loom Video](https://www.loom.com/share/9d77eb7028f34f7f87c6661fb758d1c0) - Process automation walkthrough
207+
208+
*Note: These demos show the legacy monolithic version. For current v1.0+ modular architecture examples, see the [documentation](https://docs.openadapt.ai).*
329209

330210
---
331211

0 commit comments

Comments
 (0)