Skip to content

Commit fcd0894

Browse files
authored
Add files via upload
Signed-off-by: Jonathan Harrison <145727918+Raiff1982@users.noreply.github.com>
1 parent 83c0de2 commit fcd0894

13 files changed

Lines changed: 448 additions & 0 deletions

CONTRIBUTING.md

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
2+
# Contributing
3+
4+
Thanks for contributing to the AGI Governance Benchmark!
5+
6+
## Branching & PRs
7+
- Create feature branches from `main`.
8+
- Add a clear PR title and fill in the PR template.
9+
- Keep diffs focused — avoid committing generated artifacts unless necessary.
10+
11+
## Code Style
12+
- Python: PEP8. Prefer type hints and docstrings.
13+
- TypeScript: strict types, small modules.
14+
15+
## Tests / CI
16+
- Ensure `run_harness.py` and `score_results.py` run locally before PR.
17+
- CI will execute a short demo run against `tasks.jsonl` and publish artifacts.
18+
19+
## Structure (proposed)
20+
- `tasks.jsonl` — tasks
21+
- `trace_schema.yaml` — trace schema
22+
- `run_harness.py` / `run_harness.ts` — reference harness
23+
- `score_results.py` — scoring
24+
- `eval_utils.py`, `fairness_metrics.py`, `security_utils.py`, `visualization_utils.py` — utilities
25+
- `error_recovery_handler.py` — recovery orchestration
26+
- `docs/` — executive summaries, partner readmes
27+
- `.github/workflows/` — CI
28+
29+
## Large Files
30+
- Avoid adding large models/binaries to the repo. Use release artifacts if needed.
31+
32+
## License
33+
- MIT. By contributing you agree to license your contributions under MIT.

EXEC_SUMMARY.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
2+
# Executive Summary — AGI Governance Benchmark
3+
4+
**What it is**: A portable, reproducible benchmark for evaluating AI reliability across:
5+
- Adversarial Fact Verification (AFV)
6+
- Multi-Step Tool-Augmented Reasoning (MSR)
7+
- Constrained Policy Generation (CPG)
8+
9+
**Why it matters**: It measures determinism, hallucination, source selection, constraint adherence, fairness, and recovery — the pillars of governed cognition.
10+
11+
**What’s included**:
12+
- Tasks (`tasks.jsonl`), Trace schema (`trace_schema.yaml`)
13+
- Harnesses (Python/TS), Scorer (`score_results.py`)
14+
- Utilities (evaluation, fairness, security, recovery, visualization)
15+
- CI workflow that runs a short demo and publishes artifacts
16+
17+
**What’s new (v1.2)**:
18+
- Error recovery orchestrator with retry/fallback/escalate/safe-fail
19+
- Optional fairness proxy check and encrypted recovery logs
20+
- Partner-ready docs and demo artifacts
21+
22+
**How to use**:
23+
1. Run the harness: `python run_harness.py --tasks tasks.jsonl --runs 10 --out results.jsonl`
24+
2. Score: `python score_results.py --tasks tasks.jsonl --results results.jsonl --out scores.json`
25+
3. (Optional) Recovery demo: `python run_harness_recovery.py --tasks tasks.jsonl --simulate_errors`
26+
4. Review CI artifacts for a quick sanity check
27+
28+
**Roadmap**:
29+
- Graded hallucination metric by default
30+
- Expanded task bank with multilingual/noisy/contradictory cases
31+
- Dashboard visualizations and fairness auditing hooks

Makefile

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
2+
.PHONY: demo score ci validate
3+
4+
demo:
5+
python run_harness.py --tasks tasks.jsonl --runs 10 --out results.jsonl
6+
7+
score:
8+
python score_results.py --tasks tasks.jsonl --results results.jsonl --out scores.json
9+
10+
recovery:
11+
python run_harness_recovery.py --tasks tasks.jsonl --simulate_errors --out results_recovery.jsonl
12+
13+
validate:
14+
./scripts/validate_tasks.py tasks.jsonl
15+
16+
ci: demo score

README_REORG.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
2+
# Repository Reorg Plan
3+
4+
This repo currently includes multiple *update variants* (e.g., `eval_utilsUpdate.py`, `fairness_metrics2.py`, `tasksUpdate.jsonl`, `trace_schemaUpdate.yaml`). To reduce confusion:
5+
6+
## Keep (canonical)
7+
- `tasks.jsonl`
8+
- `trace_schema.yaml`
9+
- `run_harness.py`, `run_harness.ts`, `run_harness_recovery.py`
10+
- `score_results.py`, `eval_utils.py`
11+
- `error_recovery_handler.py`
12+
- `fairness_metrics.py`, `security_utils.py`, `visualization_utils.py`
13+
- `README.md`, `README_UPDATED.md`, `README_RECOVERY.md`, `PARTNER_README.md`
14+
- `CHANGELOG.md`
15+
16+
## Remove or archive
17+
- `*Update*.py`, `*Update*.ts`, `tasksUpdate.jsonl`, `trace_schemaUpdate.yaml`, `trace_schemaBC.yaml`, duplicates like `fairness_metrics2.py`, `security_utils2.py`, `error_recovery2.py`.
18+
- If content differs, merge improvements back into the canonical files first.
19+
20+
## Structure (suggested)
21+
- `/docs` — executive summaries & partner docs
22+
- `/scripts` — validation and helper scripts
23+
- `/.github/workflows` — CI
24+
- root — tasks, schema, harnesses, scorers, core utilities
25+
26+
## Next steps
27+
1. Run `scripts/validate_tasks.py tasks.jsonl` to ensure integrity.
28+
2. Merge any improvements from `*Update*` files into canonical ones.
29+
3. Delete superseded variants in a single PR titled **Repo cleanup: consolidate duplicates**.

benchmark-ci.yml

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
2+
name: benchmark-ci
3+
4+
on:
5+
push:
6+
branches: [ main ]
7+
pull_request:
8+
branches: [ main ]
9+
10+
jobs:
11+
run-benchmark:
12+
runs-on: ubuntu-latest
13+
steps:
14+
- name: Checkout
15+
uses: actions/checkout@v4
16+
17+
- name: Set up Python
18+
uses: actions/setup-python@v5
19+
with:
20+
python-version: '3.11'
21+
22+
- name: Install dependencies
23+
run: |
24+
python -m pip install --upgrade pip
25+
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
26+
if [ -f requirements-optional.txt ]; then pip install -r requirements-optional.txt || true; fi
27+
28+
- name: Run demo harness
29+
run: |
30+
python run_harness.py --tasks tasks.jsonl --runs 5 --out results_ci.jsonl
31+
32+
- name: Score results
33+
run: |
34+
python score_results.py --tasks tasks.jsonl --results results_ci.jsonl --out scores_ci.json
35+
36+
- name: Upload artifacts
37+
uses: actions/upload-artifact@v4
38+
with:
39+
name: benchmark-results
40+
path: |
41+
results_ci.jsonl
42+
scores_ci.json

error_recovery3.py

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
import json
2+
import time
3+
import random
4+
import hashlib
5+
from typing import Dict, Any, Optional
6+
import requests
7+
try:
8+
from cryptography.fernet import Fernet
9+
except ImportError:
10+
Fernet = None
11+
try:
12+
from sentence_transformers import SentenceTransformer, util
13+
except ImportError:
14+
SentenceTransformer, util = None, None
15+
16+
class ErrorRecoveryHandler:
17+
def __init__(self, encryption_key: Optional[bytes] = None, fairness_check: bool = False, api_key: Optional[str] = None):
18+
self.categories = ["retry", "fallback", "escalate", "safe-fail"]
19+
self.encryption_key = encryption_key
20+
self.cipher = Fernet(encryption_key) if Fernet and encryption_key else None
21+
self.recovery_log = []
22+
self.fairness_check = fairness_check
23+
self.fairness_model = SentenceTransformer('all-MiniLM-L6-v2') if SentenceTransformer else None
24+
self.api_key = api_key
25+
random.seed(42) # Reproducibility
26+
27+
def classify_and_recover(self, task: Dict[str, Any], error_type: str, trace: Dict[str, Any]) -> Dict[str, Any]:
28+
start_time = time.time()
29+
pattern = "none"
30+
31+
weights = {
32+
"TimeoutError": 0.75,
33+
"HTTPError_429": 0.75,
34+
"HTTPError_400": 0.5,
35+
"ConstraintViolation": 0.3
36+
}
37+
prob = random.uniform(0, 1)
38+
if prob < weights.get(error_type, 0.5):
39+
pattern = "retry"
40+
outcome = self._retry(task, trace)
41+
elif prob < 0.85:
42+
pattern = "fallback"
43+
outcome = self._fallback(task, trace)
44+
else:
45+
pattern = "escalate" if "CPG" in task["task_set"] and "Violation" in error_type else "safe-fail"
46+
outcome = self._escalate_or_fail(task, trace, pattern)
47+
48+
trace["error_recovery_pattern"] = pattern
49+
trace["recovery_outcome"] = outcome
50+
trace["performance_efficiency"]["latency_ms"] = trace.get("performance_efficiency", {}).get("latency_ms", 0) + int((time.time() - start_time) * 1000)
51+
self._log_securely(trace)
52+
return trace
53+
54+
def _retry(self, task: Dict[str, Any], trace: Dict[str, Any]) -> str:
55+
trace["retry_attempt"] = trace.get("retry_attempt", 0) + 1
56+
if trace["retry_attempt"] > 2:
57+
return f"Retry {trace['retry_attempt']} failed for {task['id']}: Max retries exceeded"
58+
try:
59+
output = self._call_grok_api(task)
60+
trace["model_output"] = output
61+
return f"Retry {trace['retry_attempt']} successful for {task['id']}"
62+
except Exception as e:
63+
return f"Retry {trace['retry_attempt']} failed for {task['id']}: {str(e)}"
64+
65+
def _fallback(self, task: Dict[str, Any], trace: Dict[str, Any]) -> str:
66+
default = task.get("ground_truth", "No ground truth available")
67+
trace["fallback_used"] = default
68+
return f"Fallback to {default} for {task['id']}"
69+
70+
def _escalate_or_fail(self, task: Dict[str, Any], trace: Dict[str, Any], pattern: str) -> str:
71+
if pattern == "escalate" and self.fairness_check and self.fairness_model:
72+
output = trace.get("model_output", "")
73+
constraints = task.get("constraints", [])
74+
embeddings = self.fairness_model.encode([output] + constraints)
75+
similarities = util.cos_sim(embeddings[0], embeddings[1:])
76+
if any(sim > 0.7 for sim in similarities):
77+
trace["fairness_audit"] = "Potential proxy violation detected"
78+
else:
79+
trace["fairness_audit"] = "No proxy violations"
80+
trace["escalation_flag"] = (pattern == "escalate")
81+
return f"{pattern.capitalize()} for {task['id']}"
82+
83+
def _call_grok_api(self, task: Dict[str, Any]) -> str:
84+
# Mock API responses based on task ID (real calls need API key)
85+
if task["id"] == "AFV-001":
86+
return "GDP grew at 2.1%"
87+
elif task["id"] == "MSR-001":
88+
raise Exception("HTTPError_400") # Invalid JSON
89+
elif task["id"] == "CPG-001":
90+
raise Exception("HTTPError_429") # Rate limit
91+
return "[error]"
92+
93+
def _log_securely(self, trace: Dict[str, Any]):
94+
log_entry = json.dumps(trace)
95+
if self.cipher:
96+
self.recovery_log.append(self.cipher.encrypt(log_entry.encode()))
97+
else:
98+
self.recovery_log.append(log_entry)
99+
100+
def get_recovery_stats(self) -> Dict[str, float]:
101+
if not self.recovery_log:
102+
return {"total_recoveries": 0, "success_rate": 0.0}
103+
patterns = []
104+
successes = 0
105+
for entry in self.recovery_log:
106+
entry_data = json.loads(entry.decode() if isinstance(entry, bytes) else entry)
107+
patterns.append(entry_data["error_recovery_pattern"])
108+
if entry_data["recovery_outcome"].startswith(("Retry successful", "Fallback to")):
109+
successes += 1
110+
return {
111+
"total_recoveries": len(patterns),
112+
"success_rate": successes / len(patterns) if patterns else 0.0,
113+
**{cat: patterns.count(cat) / len(patterns) for cat in self.categories}
114+
}

gitignore

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
2+
# Byte-compiled / cache
3+
__pycache__/
4+
*.pyc
5+
*.pyo
6+
7+
# Virtual envs
8+
.venv/
9+
.env/
10+
11+
# Results / artifacts
12+
results*.jsonl
13+
scores*.json
14+
recovery_stats.json
15+
*.enc
16+
17+
# OS files
18+
.DS_Store

pull_request_template.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
2+
## Summary
3+
<!-- What does this PR change and why? -->
4+
5+
## Changes
6+
- [ ] Tasks / schema
7+
- [ ] Harness / scorer
8+
- [ ] Recovery / fairness / security
9+
- [ ] Docs / CI
10+
11+
## Validation
12+
- [ ] `python run_harness.py --tasks tasks.jsonl --runs 5 --out results_ci.jsonl`
13+
- [ ] `python score_results.py --tasks tasks.jsonl --results results_ci.jsonl --out scores_ci.json`
14+
15+
## Screenshots / Artifacts
16+
<!-- Attach or link to results_ci.jsonl / scores_ci.json if relevant -->
17+
18+
## Checklist
19+
- [ ] Focused diff
20+
- [ ] Docs updated
21+
- [ ] CI green

pyproject.toml

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
[project]
2+
name = "agi-governance-benchmark"
3+
version = "1.2.0"
4+
description = "Governance-first benchmark for AFV, MSR, and CPG with recovery, fairness, and transparency."
5+
requires-python = ">=3.10"
6+
authors = [{name="Codette x SIM-ONE", email="devnull@example.com"}]
7+
readme = "README.md"
8+
license = {text = "MIT"}
9+
dependencies = [
10+
"pyyaml>=6.0.1",
11+
"cryptography>=42.0.0",
12+
"matplotlib>=3.8.0"
13+
]
14+
15+
[project.optional-dependencies]
16+
fairness = ["sentence-transformers>=2.7.0"]
17+
dev = ["pytest>=8.0.0", "ruff>=0.5.0", "black>=24.4.0", "mypy>=1.10.0"]
18+
19+
[tool.ruff]
20+
line-length = 100
21+
target-version = "py311"
22+
select = ["E","F","I","UP"]
23+
ignore = []
24+
25+
[tool.black]
26+
line-length = 100
27+
target-version = ["py311"]
28+
29+
[tool.mypy]
30+
python_version = "3.11"
31+
ignore_missing_imports = true

requirements-optional.txt

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
2+
# Optional dependencies (heavy); install if you want fairness proxy checks
3+
sentence-transformers>=2.2.2
4+
torch>=2.2.0

0 commit comments

Comments
 (0)