Skip to content

Commit fbd60ae

Browse files
committed
evals: add evaluation framework docs
1 parent 40b17b7 commit fbd60ae

1 file changed

Lines changed: 34 additions & 0 deletions

File tree

docs/evaluations/README.md

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Evaluations
2+
3+
Evaluations prove that AI-OS improves work quality instead of only adding process.
4+
5+
## Evaluation loop
6+
7+
```mermaid
8+
flowchart TD
9+
Task[Task] --> Baseline[Run without AI-OS]
10+
Baseline --> AIOS[Run with AI-OS]
11+
AIOS --> Score[Score outcomes]
12+
Score --> Compare[Compare]
13+
Compare --> Improve[Improve prompt or loop]
14+
Improve --> Task
15+
```
16+
17+
## Suggested metrics
18+
19+
- goal completion
20+
- verifier pass rate
21+
- number of recovery loops
22+
- documentation completeness
23+
- security review completeness
24+
- time to usable result
25+
- number of unverified claims
26+
- human approval correctness
27+
28+
## First eval set
29+
30+
- feature documentation task
31+
- bugfix reasoning task
32+
- CI failure triage task
33+
- release checklist task
34+
- wiki sync task

0 commit comments

Comments
 (0)