Skip to content

Commit 99c87ca

Browse files
Jonathan Harrisonclaude
andcommitted
Add tests, pin deps, update README and CHANGELOG for April 2026 quality wave
Tests: - tests/test_aegis.py: 17 tests covering evaluate(), quick_check(), alignment_trend(), veto counting, serialization round-trip - tests/test_cocoon_synthesizer.py: 19 tests covering extract_patterns(), forge_strategy(), apply_and_compare(), run_full_synthesis() standalone and with live UnifiedMemory - tests/test_web_research.py: 20 tests covering SSRF/URL safety edge cases, fetch_url_text() mocked HTTP, search_web() error paths, query classification Dependencies: - requirements.txt: added upper-bound version pins (torch<3.0, transformers<5.0, numpy<2.0, etc.), added missing core deps (llama-cpp-python, trl, datasets, accelerate, huggingface-hub, aiohttp), removed unused legacy deps Docs: - README.md: updated to paper v7, added ablation study section, new Recent Improvements table, updated architecture tree, CHANGELOG links - docs/CHANGELOG_2026-04-26.md: full record of all April 26 changes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 2fd8aed commit 99c87ca

6 files changed

Lines changed: 899 additions & 18 deletions

File tree

README.md

Lines changed: 49 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ Codette is a production-ready AI reasoning system that thinks from multiple angl
5353

5454
Created by **Jonathan Harrison** (Raiff1982)
5555

56-
> **New in v5**: Publishable benchmark suite with 17 problems across 6 categories demonstrates **93.1% improvement** over single-perspective baseline (p < 0.0001, Cohen's d = 7.88). Meta-cognitive CocoonSynthesizer discovers cross-domain reasoning patterns and forges new strategies. Full academic paper: [`paper/codette_paper_v5.pdf`](paper/codette_paper_v5.pdf) and [`paper/codette_paper_v5.tex`](paper/codette_paper_v5.tex)
56+
> **Paper v7 available**: [`paper/codette_paper_v7.tex`](paper/codette_paper_v7.tex) — includes rebuttal changes, updated tables, and Kaggle notebook. Benchmark suite (17 problems, 6 categories) demonstrates **93.1% improvement** over single-perspective baseline (p < 0.0001, Cohen's d = 7.88). New **ablation study** isolates each component's contribution. Full v5 paper: [`paper/codette_paper_v5.pdf`](paper/codette_paper_v5.pdf)
5757
5858
### 🚀 Public Landing Page
5959

@@ -72,7 +72,7 @@ Codette is a modular reasoning system with published demos, tests, benchmarks, a
7272
- **Automated tests**: [tests](tests)
7373
- **Benchmark suites**: [benchmarks](benchmarks)
7474
- **Saved benchmark reports**: [data/results](data/results)
75-
- **Change transparency**: [docs/CHANGELOG_2026-04-02.md](docs/CHANGELOG_2026-04-02.md)
75+
- **Change transparency**: [docs/CHANGELOG_2026-04-26.md](docs/CHANGELOG_2026-04-26.md) · [docs/CHANGELOG_2026-04-02.md](docs/CHANGELOG_2026-04-02.md)
7676
- **Contributing guide**: [CONTRIBUTING.md](CONTRIBUTING.md)
7777

7878
Quick evidence links:
@@ -107,8 +107,11 @@ This repository includes reproducible evidence of:
107107
| **Cocoon Introspection** | Statistical self-analysis of her own reasoning history — real patterns, not generated text |
108108
| **Meta-Cognitive Synthesis** | CocoonSynthesizer discovers cross-domain patterns in reasoning history and forges new strategies |
109109
| **Publishable Benchmarks** | 17-problem suite across 6 categories with 7-dimension scoring (93.1% improvement, p<0.0001) |
110-
| **AEGIS Ethics** | 6-framework ethical evaluation (utilitarian, deontological, virtue, care, ubuntu, indigenous) |
111-
| **Code7eCQURE** | Quantum emotional context enrichment on every query (Layer 2.5) |
110+
| **Ablation Study** | `benchmarks/ablation_study.py` isolates each component's contribution — memory, ethical layer, sycophancy guard, single-agent baseline |
111+
| **AEGIS Ethics** | 6-framework ethical evaluation (utilitarian, deontological, virtue, care, ubuntu, indigenous) with differentiable ethics potential field |
112+
| **Code7eCQURE** | Stochastic multi-perspective reasoning with named cognitive frames (Newton, DaVinci, Ethical, Quantum, Memory) — metaphorical "quantum" framing for probabilistic reasoning |
113+
| **WAL-mode SQLite** | Write-Ahead Logging + thread-safe write lock prevents serialization under concurrent requests |
114+
| **Hardened Model Loading** | Path validation before load + 5-minute timeout prevent silent hangs on missing or corrupted GGUF |
112115
| **Real Self-Diagnostic** | Health checks return measured values from 9 subsystems, not LLM-generated guesses |
113116
| **Phase 6/7 Routing** | Query complexity classification, domain detection, executive control |
114117
| **Session Continuity** | Active continuity summaries, decision landmarks, and recall markers reduce drift in long sessions |
@@ -237,7 +240,8 @@ codette-clean/
237240
| +-- semantic_tension.py # Embedding-based conflict measurement
238241
|
239242
|-- benchmarks/ # Publishable evaluation suite
240-
| +-- codette_benchmark_suite.py # 17 problems x 4 conditions x 7 dimensions
243+
| |-- codette_benchmark_suite.py # 17 problems x 4 conditions x 7 dimensions
244+
| +-- ablation_study.py # Component contribution analysis (run standalone)
241245
|
242246
|-- demo/ # Reproducible local demos
243247
| |-- README.md # Demo index
@@ -583,6 +587,46 @@ That document explains:
583587

584588
---
585589

590+
## Run the Ablation Study
591+
592+
```bash
593+
python benchmarks/ablation_study.py
594+
```
595+
596+
Outputs a table showing how much each component contributes to the full-system score:
597+
598+
```
599+
Condition Mean Drop Cohen d p-value
600+
------------------------------------------------------------
601+
full 0.652 +0.000 0.000 1.0000
602+
no_memory 0.636 +0.016 x.xxx x.xxxx
603+
no_ethical 0.xxx +0.xxx x.xxx x.xxxx **
604+
no_sycophancy 0.xxx +0.xxx x.xxx x.xxxx
605+
single_agent 0.338 +0.314 7.520 <0.0001 **
606+
```
607+
608+
Results are saved to `benchmarks/results/ablation_results.json`.
609+
610+
---
611+
612+
## Recent Improvements (April 2026)
613+
614+
| Area | Change |
615+
|------|--------|
616+
| **Session race condition** | Session captured once per request — eliminates mid-request swap when `/api/new_session` fires concurrently |
617+
| **Model load hang** | GGUF path validated before load; 5-min `ThreadPoolExecutor` timeout prevents indefinite hang on corrupt files |
618+
| **SQLite concurrency** | WAL journal mode + `threading.Lock` on all writes — concurrent reads no longer block writes |
619+
| **Memory consolidation** | Removed orphaned `memory_kernel_local.py`; `memory_kernel.py` is canonical |
620+
| **Ablation study** | `benchmarks/ablation_study.py` added — isolates contribution of memory, ethical layer, sycophancy guard |
621+
| **Honest quantum docs** | `code7e_cqure.py` now documents that "quantum" is a metaphor for stochastic multi-perspective reasoning |
622+
| **Test coverage** | Added `test_aegis.py`, `test_cocoon_synthesizer.py`, `test_web_research.py` |
623+
| **Dependencies** | `requirements.txt` now has upper-bound version pins; removed bloated unused deps |
624+
| **Legacy module** | `universal_reasoning.py` fixed (broken imports, botbuilder removed, memory destroy hardened) and archived |
625+
| **Bare except** | `hallucination_guard.py` bare `except:` narrowed to `(ValueError, AttributeError, IndexError)` |
626+
| **Paper v7** | `paper/codette_paper_v7.tex`, rebuttal, tables, and Kaggle notebook added |
627+
628+
---
629+
586630
## License
587631

588632
MIT -- Created by **Jonathan Harrison** (Raiff1982)

docs/CHANGELOG_2026-04-26.md

Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
# Changelog — 2026-04-26
2+
3+
This changelog records the April 26, 2026 quality and reliability wave.
4+
Focus areas: concurrency safety, test coverage, ablation study for publication,
5+
dependency hygiene, legacy module cleanup, and paper v7.
6+
7+
---
8+
9+
## Bug Fixes
10+
11+
### Session Race Condition (Critical)
12+
- **File**: `inference/codette_server.py`
13+
- **Problem**: `_get_active_session()` was called three separate times during
14+
one request (lines ~1098, ~1384, ~1572). A concurrent `/api/new_session`
15+
call between those reads could swap the global session, causing one request
16+
to read context from session A but write metrics to session B.
17+
- **Fix**: Session captured once (`session = _get_active_session()`) at the
18+
top of request processing and reused throughout. The three duplicate calls
19+
were removed.
20+
21+
### Model Load Hang (High)
22+
- **File**: `inference/codette_server.py`
23+
- **Problem**: If the GGUF model path did not exist, the server would start
24+
successfully but hang indefinitely on the first request (lazy-load blocked
25+
waiting for a model that could never load). No error was surfaced to the UI.
26+
- **Fix**:
27+
1. Path existence check added before attempting load — returns a clear error
28+
message immediately if the file is missing.
29+
2. `ThreadPoolExecutor` with a 5-minute timeout wraps the `CodetteOrchestrator`
30+
constructor — a corrupted or stalled GGUF now raises `RuntimeError` instead
31+
of hanging forever.
32+
33+
### Bare `except:` in Hallucination Guard
34+
- **File**: `reasoning_forge/hallucination_guard.py:242`
35+
- **Problem**: `except:` (no exception type) silently caught `KeyboardInterrupt`
36+
and `SystemExit` in addition to real parse errors.
37+
- **Fix**: Narrowed to `except (ValueError, AttributeError, IndexError)`.
38+
39+
---
40+
41+
## Concurrency Hardening
42+
43+
### SQLite WAL Mode + Write Lock
44+
- **File**: `reasoning_forge/unified_memory.py`
45+
- **Changes**:
46+
- Enabled `PRAGMA journal_mode=WAL` on database open — readers no longer
47+
block writers and writers no longer block readers under concurrent load.
48+
- Set `PRAGMA synchronous=NORMAL` for a safe balance of durability vs speed.
49+
- Added `threading.Lock` (`self._write_lock`) wrapping all write operations
50+
(`store()`, `mark_success()`) — prevents concurrent writes from producing
51+
corrupted rows when multiple worker threads fire simultaneously.
52+
- Added `import threading` (was missing).
53+
54+
---
55+
56+
## Code Cleanup
57+
58+
### Removed Orphaned Memory Kernel
59+
- **File removed**: `reasoning_forge/memory_kernel_local.py`
60+
- A 150-line stripped-down duplicate of `memory_kernel.py`. Zero imports
61+
anywhere in the live codebase. `memory_kernel.py` (410 lines, with disk
62+
persistence, `EthicalAnchor`, type hints) is the canonical file.
63+
64+
### Fixed and Archived Legacy Entry Point
65+
- **File fixed**: `consciousness/universal_reasoning.py`
66+
**Moved to**: `archive/consciousness/universal_reasoning.py`
67+
- This was an old standalone Azure Bot Framework entry point that predated
68+
the current server architecture. Fixed before archiving:
69+
- Removed unused `botbuilder` and `dialog_helper` imports (Azure Bot
70+
Framework, no longer in the system).
71+
- Replaced missing `perspectives` module with compatibility shims mapping
72+
old `*Perspective` API → current `reasoning_forge/agents/*Agent` classes.
73+
- `destroy_sensitive_data()` now overwrites bytearray data before `del`
74+
(reduces memory exposure window; Python strings are immutable so
75+
full zero-fill is not possible).
76+
- Hardcoded `https://api.example.com/data` URL made configurable via
77+
`config['real_time_data_url']`.
78+
- Added deprecation header documenting why the file is archived and where
79+
current equivalents live.
80+
81+
### Documented Code7eCQURE "Quantum" Framing
82+
- **File**: `reasoning_forge/code7e_cqure.py`
83+
- Added module-level docstring clarifying that "quantum" is a **metaphor**
84+
for stochastic multi-perspective reasoning, not quantum computing.
85+
- Mechanism: named perspective functions apply labeled prefix transforms;
86+
`random()` and `random.choice()` simulate epistemic uncertainty and
87+
superposition of outcomes.
88+
- Added inline comments on `quantum_spiderweb()` and `quantum_superposition()`.
89+
90+
---
91+
92+
## New Features
93+
94+
### Ablation Study Runner
95+
- **File**: `benchmarks/ablation_study.py`
96+
- `AblationRunner` class runs 5 experimental conditions, each disabling
97+
exactly one component from the full system:
98+
- `full` — all components active (baseline)
99+
- `no_memory` — cocoon recall disabled
100+
- `no_ethical` — ethical dimension weight zeroed
101+
- `no_sycophancy` — sycophancy guard pass skipped
102+
- `single_agent` — single perspective only (worst-case baseline)
103+
- Reports mean composite score, drop from full, Cohen's d, and p-value per
104+
condition — ready for inclusion in the paper's ablation section.
105+
- Saves JSON results to `benchmarks/results/ablation_results.json`.
106+
- Run with: `python benchmarks/ablation_study.py`
107+
108+
---
109+
110+
## Test Coverage
111+
112+
### New: `tests/test_aegis.py`
113+
Covers the full AEGIS ethical governance API:
114+
- `evaluate()` — benign content passes, harmful content vetoed, all keys present
115+
- Framework count (6 frameworks verified)
116+
- EMA eta updates across multiple calls
117+
- Veto count and total evaluation tracking
118+
- `quick_check()` — safe content passes, harmful patterns blocked
119+
- `alignment_trend()` — trend after benign vs. mixed inputs
120+
- `to_dict()` / `from_dict()` serialization round-trip
121+
- `get_state()` summary structure
122+
123+
### New: `tests/test_cocoon_synthesizer.py`
124+
Covers the CocoonSynthesizer meta-cognitive engine:
125+
- `extract_patterns()` — cross-domain detection, single-domain produces no
126+
cross-domain patterns, empty input, field structure validation
127+
- `forge_strategy()` — returns `ReasoningStrategy`, references source patterns,
128+
default strategy on empty input, strategy history growth
129+
- `apply_and_compare()` — returns `StrategyComparison`, improvement delta,
130+
readable output, dict serialization
131+
- `run_full_synthesis()` — standalone mode, valuation context embedding,
132+
with live `UnifiedMemory` (temp DB)
133+
134+
### New: `tests/test_web_research.py`
135+
Covers web research safety and fetch behaviour:
136+
- `_is_safe_url()` — blocks localhost, loopback IPv4/v6, private ranges
137+
(10.x, 192.168.x, 172.16.x), link-local (169.254.x), file/ftp schemes,
138+
empty string, missing hostname
139+
- `fetch_url_text()` — unsafe URL returns empty, safe URL returns text,
140+
network error returns empty, output capped at `max_chars`
141+
- `search_web()` — empty/whitespace query, network error, timeout (all mocked)
142+
- `query_requests_web_research()` — edge cases: lookup phrasing, vague short
143+
query, internal reflection phrasing
144+
- `query_benefits_from_web_research()` — current events, price queries, static
145+
knowledge (no benefit), personal questions, recent release queries
146+
147+
---
148+
149+
## Dependencies
150+
151+
### `requirements.txt` — Version Pins Added
152+
- Added upper-bound version constraints to all packages to prevent silent
153+
breaking changes from major version bumps:
154+
- `torch>=2.1.0,<3.0.0` (PyTorch 3.x not yet validated)
155+
- `transformers>=4.40.0,<5.0.0`
156+
- `peft>=0.10.0,<1.0.0`
157+
- `numpy>=1.24.0,<2.0.0`
158+
- etc.
159+
- Added missing core dependencies: `llama-cpp-python`, `trl`, `datasets`,
160+
`accelerate`, `huggingface-hub`, `aiohttp`.
161+
- Removed bloated/unused deps that were only needed by the archived
162+
`universal_reasoning.py`: `botbuilder`, `speech_recognition`, `PIL`,
163+
`nltk`, `numba`, `dataclasses-json`.
164+
- Added comments clarifying which deps are optional (Gradio = demo only,
165+
vaderSentiment = legacy module only).
166+
167+
---
168+
169+
## .gitignore Updates
170+
171+
Added:
172+
- `.claude/settings.local.json` — machine-specific Claude IDE settings
173+
- `demo/outputs/*.mp4` — large demo recordings
174+
- `*.mp4` — any video files
175+
176+
---
177+
178+
## Paper
179+
180+
- Added `paper/codette_paper_v7.tex` — updated paper with rebuttal changes
181+
- Added `paper/rebuttal_changes.tex` — point-by-point rebuttal document
182+
- Added `paper/tables/` — 10 formatted result tables
183+
- Added `paper/codette_kaggle_notebook.ipynb` — Kaggle submission notebook
184+
- Added `paper/kaggle_writeup.md` — Kaggle competition writeup
185+
- Added `paper/benchmark_tasks/` — 17 individual benchmark task notebooks
186+
187+
---
188+
189+
## Commits This Wave
190+
191+
| Hash | Message |
192+
|------|---------|
193+
| `601d128` | Add benchmark results, paper v7, new reasoning modules, and training pipeline updates |
194+
| `7bcd33d` | Fix session race, model load hang, SQLite concurrency, and add ablation study |
195+
| `2fd8aed` | Fix and archive universal_reasoning.py; patch bare except; update .gitignore |
196+
| *(this wave)* | Tests, README, CHANGELOG, requirements pins |

requirements.txt

Lines changed: 36 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,36 @@
1-
torch>=2.8.0
2-
transformers>=4.30.0
3-
peft>=0.4.0
4-
numpy>=1.24.0
5-
scipy>=1.10.0
6-
dataclasses-json>=0.5.14
7-
pydantic>=2.0.0
8-
numba>=0.57.0
9-
tqdm>=4.65.0
10-
gradio>=3.50.0
11-
requests>=2.31.0
12-
python-dotenv>=1.0.0
13-
cryptography>=42.0.0
1+
# Core inference — llama.cpp backend
2+
llama-cpp-python>=0.2.90,<0.3.0
3+
4+
# PyTorch — pinned to 2.x; 3.x API is not yet validated
5+
torch>=2.1.0,<3.0.0
6+
torchvision>=0.16.0,<1.0.0
7+
8+
# HuggingFace stack — LoRA training pipeline
9+
transformers>=4.40.0,<5.0.0
10+
peft>=0.10.0,<1.0.0
11+
trl>=0.8.0,<1.0.0
12+
datasets>=2.18.0,<3.0.0
13+
accelerate>=0.27.0,<1.0.0
14+
huggingface-hub>=0.22.0,<1.0.0
15+
16+
# Numerical
17+
numpy>=1.24.0,<2.0.0
18+
scipy>=1.11.0,<2.0.0
19+
20+
# Async + HTTP
21+
aiohttp>=3.9.0,<4.0.0
22+
requests>=2.31.0,<3.0.0
23+
24+
# Utilities
25+
python-dotenv>=1.0.0,<2.0.0
26+
cryptography>=42.0.0,<44.0.0
27+
tqdm>=4.65.0,<5.0.0
28+
pydantic>=2.0.0,<3.0.0
29+
30+
# Sentiment (used by universal_reasoning legacy module — optional)
31+
vaderSentiment>=3.3.2,<4.0.0
32+
33+
# Gradio (demo space UI only — not required for server)
34+
gradio>=4.0.0,<5.0.0
35+
36+
# SQLite FTS5 is built into Python's standard library sqlite3 (no extra install needed)

0 commit comments

Comments
 (0)