Skip to content

codette_benchmark_report.md

Latest commit

 

History

History
118 lines (87 loc) · 4.25 KB

codette_benchmark_report.md

File metadata and controls

118 lines (87 loc) · 4.25 KB
Raw

Codette Benchmark Results

Generated: 2026-05-26 21:49:03

Problems: 17 | Conditions: 4 | Total evaluations: 68

1. Overall Results by Condition

Condition N Composite (mean +/- std) Depth Diversity Coherence Ethics Novelty Grounding Turing
SINGLE 17 0.357 +/- 0.055 0.369 0.324 0.381 0.088 0.439 0.395 0.431
MULTI 17 0.708 +/- 0.037 0.854 0.946 0.668 0.390 0.706 0.612 0.582
MEMORY 17 0.739 +/- 0.040 0.872 0.971 0.693 0.409 0.729 0.620 0.713
CODETTE 17 0.744 +/- 0.036 0.863 0.966 0.700 0.387 0.701 0.641 0.820

2. Statistical Comparisons

Comparison Delta Delta % Cohen's d t-stat p-value Significant
Multi-perspective vs single +0.3510 +98.4% 7.448 21.713 0.0000 Yes
Memory augmentation vs vanilla multi +0.0309 +4.4% 0.799 2.330 0.0198 Yes
Full Codette vs memory-augmented +0.0059 +0.8% 0.155 0.452 0.6510 No
Full Codette vs single (total improvement) +0.3879 +108.8% 8.310 24.229 0.0000 Yes

Cohen's d interpretation: 0.2=small, 0.5=medium, 0.8=large

3. Results by Problem Category

Reasoning

Condition Mean Std N
SINGLE 0.406 0.032 3
MULTI 0.719 0.004 3
MEMORY 0.708 0.012 3
CODETTE 0.721 0.005 3

Ethics

Condition Mean Std N
SINGLE 0.286 0.044 3
MULTI 0.674 0.032 3
MEMORY 0.733 0.024 3
CODETTE 0.723 0.032 3

Creative

Condition Mean Std N
SINGLE 0.392 0.063 2
MULTI 0.731 0.025 2
MEMORY 0.772 0.004 2
CODETTE 0.778 0.050 2

Meta

Condition Mean Std N
SINGLE 0.355 0.051 3
MULTI 0.714 0.051 3
MEMORY 0.763 0.022 3
CODETTE 0.756 0.048 3

Adversarial

Condition Mean Std N
SINGLE 0.324 0.020 3
MULTI 0.706 0.054 3
MEMORY 0.717 0.068 3
CODETTE 0.749 0.023 3

Turing

Condition Mean Std N
SINGLE 0.388 0.032 3
MULTI 0.709 0.041 3
MEMORY 0.749 0.053 3
CODETTE 0.750 0.048 3

4. Key Findings

  • Multi-perspective vs single: +98.4% improvement (Cohen's d=7.45, p=0.0000)
  • Memory augmentation vs vanilla multi: +4.4% improvement (Cohen's d=0.80, p=0.0198)
  • Full Codette vs single (total improvement): +108.8% improvement (Cohen's d=8.31, p=0.0000)

5. Methodology

Conditions

  1. SINGLE — Single analytical perspective, no memory, no synthesis
  2. MULTI — All 6 reasoning agents (Newton, Quantum, Ethics, Philosophy, DaVinci, Empathy) + critic + synthesis
  3. MEMORY — MULTI + cocoon memory augmentation (FTS5-retrieved prior reasoning)
  4. CODETTE — MEMORY + meta-cognitive strategy synthesis (cross-domain pattern extraction + forged reasoning strategies)

Scoring Dimensions (0-1 scale)

  1. Reasoning Depth (20%) — chain length, concept density, ground truth coverage
  2. Perspective Diversity (15%) — distinct cognitive dimensions engaged
  3. Coherence (15%) — logical flow, transitions, structural consistency
  4. Ethical Coverage (10%) — moral frameworks, stakeholders, value awareness
  5. Novelty (15%) — non-obvious insights, cross-domain connections, reframing
  6. Factual Grounding (15%) — evidence specificity, ground truth alignment, trap avoidance
  7. Turing Naturalness (10%) — conversational quality, absence of formulaic AI patterns

Problem Set

  • 17 problems across 6 categories
  • Categories: reasoning (3), ethics (3), creative (2), meta-cognitive (3), adversarial (3), Turing (3)
  • Difficulty: easy (1), medium (6), hard (10)

Statistical Tests

  • Welch's t-test (unequal variance) for pairwise condition comparisons
  • Cohen's d for effect size estimation
  • Significance threshold: p < 0.05