File tree Expand file tree Collapse file tree
Expand file tree Collapse file tree Original file line number Diff line number Diff line change 1+ # Evaluations
2+
3+ Evaluations prove that AI-OS improves work quality instead of only adding process.
4+
5+ ## Evaluation loop
6+
7+ ``` mermaid
8+ flowchart TD
9+ Task[Task] --> Baseline[Run without AI-OS]
10+ Baseline --> AIOS[Run with AI-OS]
11+ AIOS --> Score[Score outcomes]
12+ Score --> Compare[Compare]
13+ Compare --> Improve[Improve prompt or loop]
14+ Improve --> Task
15+ ```
16+
17+ ## Suggested metrics
18+
19+ - goal completion
20+ - verifier pass rate
21+ - number of recovery loops
22+ - documentation completeness
23+ - security review completeness
24+ - time to usable result
25+ - number of unverified claims
26+ - human approval correctness
27+
28+ ## First eval set
29+
30+ - feature documentation task
31+ - bugfix reasoning task
32+ - CI failure triage task
33+ - release checklist task
34+ - wiki sync task
You can’t perform that action at this time.
0 commit comments