docs(solutions): document spec validation workflow and critical patterns

Created comprehensive troubleshooting documentation for kiro specification
validation workflow:

Documentation Added:
- docs/solutions/documentation-gaps/documentation-drift-spec-validation-kiro-spec-system-20251126.md
  * Documents validation findings from market-data-kafka-producer Phase 5
  * Covers design.md drift, E2E test gaps, architecture diagram updates
  * Provides step-by-step resolution with code examples
  * Includes prevention strategies for future specifications

- docs/solutions/patterns/kiro-spec-critical-patterns.md (Required Reading)
  * Pattern #1: Always Run Multi-Agent Validation Before Production
  * Pattern #2: Track Validation Findings in Spec.json
  * Pattern #3: Test Default Behavior, Not Legacy Options
  * Formatted as ❌ WRONG vs ✅ CORRECT with code examples

Cross-references established between troubleshooting doc and critical patterns.

Validation Workflow Documented:
1. /kiro:spec-status - Check overall completion
2. /kiro:validate-design - Check requirements ↔ design alignment
3. /kiro:validate-impl - Check design ↔ implementation alignment
4. Fix all findings atomically
5. Track in spec.json post_validation_refinements
6. Verify 100% test pass rate

Related: market-data-kafka-producer validation (commits 53f9e54, b244e6f)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

Author: tommy-ca

docs/solutions/documentation-gaps/documentation-drift-spec-validation-kiro-spec-system-20251126.md

@@ -0,0 +1,215 @@
+---
+module: Kiro Specification System
+date: 2025-11-26
+problem_type: documentation_gap
+component: documentation
+symptoms:
+  - "Design.md migration strategy conflicted with approved requirements (dual-write vs Blue-Green)"
+  - "Performance targets in design.md misaligned with validated metrics (10k vs 150k+ msg/s)"
+  - "E2E test expected per-symbol topics but implementation used consolidated strategy"
+  - "Message headers missing from architecture diagrams"
+root_cause: inadequate_documentation
+resolution_type: documentation_update
+severity: medium
+tags: [kiro-spec, validation, documentation-drift, design-requirements-alignment, multi-agent-validation]
+---
+
+# Troubleshooting: Documentation Drift Between Requirements and Design After Spec Validation
+
+## Problem
+
+After completing implementation of the market-data-kafka-producer specification (Phase 5 ready), multi-agent validation discovered that design.md had drifted from approved requirements.md, and E2E tests were validating legacy behavior instead of the implemented default strategy. This caused confusion about the actual production behavior and could have led to incorrect deployment assumptions.
+
+## Environment
+
+- Module: Kiro Specification System (.kiro/specs/)
+- Specification: market-data-kafka-producer (Phase 5)
+- Affected Components:
+  - `.kiro/specs/market-data-kafka-producer/design.md`
+  - `tests/e2e/test_kafka_callback_e2e.py`
+- Date: 2025-11-26
+- Branch: feature/kafka-proto-backend
+
+## Symptoms
+
+- **Migration Strategy Conflict**: Design.md §6.2 described dual-write migration approach, but requirements.md had approved Blue-Green cutover (4-week timeline)
+- **Performance Target Misalignment**: Design.md §7.1 showed 10k msg/s targets, but implementation had been validated at 150k+ msg/s
+- **E2E Test Gap**: `test_kafka_callback_e2e.py` expected per-symbol topics (`cryptofeed.trades.coinbase.btc-usd`), but implementation defaulted to consolidated topics (`cryptofeed.trade`)
+- **Architecture Diagram Incompleteness**: Design.md §2.2 and §3.4.1 didn't explicitly show message headers in data flow diagrams
+
+## What Didn't Work
+
+**Attempted Solution 1:** Running `/kiro:spec-status` to check completion
+- **Why it failed:** Spec status only checks task completion counts and test pass rates. It doesn't validate alignment between requirements, design, and implementation.
+
+**Attempted Solution 2:** Manual review of implementation code
+- **Why it failed:** Code review confirmed implementation was correct, but didn't surface that design documentation had become stale during development.
+
+## Solution
+
+Used kiro multi-agent validation commands to systematically discover gaps, then fixed all issues atomically:
+
+**1. Discovery Phase (Multi-Agent Validation):**
+
+```bash
+# Phase 1: Check overall spec status
+/kiro:spec-status market-data-kafka-producer
+# Result: High completion (19/19 tasks), but no design validation
+
+# Phase 2: Validate design against requirements
+/kiro:validate-design market-data-kafka-producer
+# Result: Subagent found 3 critical documentation misalignments (C-001, C-002, C-003)
+
+# Phase 3: Validate implementation against design
+/kiro:validate-impl market-data-kafka-producer
+# Result: Subagent found 1 E2E test gap (W-001)
+```
+
+**2. Fix Phase (Atomic Commits):**
+
+**E2E Test Fix** (`tests/e2e/test_kafka_callback_e2e.py`):
+
+```python
+# Before (incorrect - expected per-symbol topics):
+assert "cryptofeed.trades.coinbase.btc-usd" in topics
+assert "cryptofeed.trades.binance.eth-usdt" in topics
+
+# After (correct - validates consolidated topic strategy):
+topics = {message.topic for message in producer.messages}
+# Consolidated topic strategy (default): all trades go to single topic
+assert "cryptofeed.trade" in topics
+assert len(topics) == 1  # All messages use consolidated topic
+```
+
+**Design.md Migration Strategy** (§6.2):
+
+```markdown
+# Before (incorrect - dual-write not approved):
+### 6.2 Migration Strategy: Dual-Write Mode
+**Approach**: Run both old and new backends simultaneously
+
+# After (correct - matches approved requirements):
+### 6.2 Migration Strategy: Blue-Green Cutover (4 Weeks)
+**Approach**: Direct migration with parallel deployment and per-exchange consumer cutover.
+**NO dual-write mode** - new backend is production-ready and can replace legacy immediately.
+```
+
+**Design.md Performance Targets** (§7.1):
+
+```markdown
+# Before (incorrect - outdated targets):
+Sustained Throughput: 10,000 msg/s → p99 <100ms latency
+
+# After (correct - validated metrics):
+Sustained Throughput (production validated):
+  150,000+ msg/s → p99 <5ms latency (consolidated topics)
+  200,000+ msg/s → p99 <10ms (multi-instance horizontal scaling)
+```
+
+**Design.md Architecture Diagrams** (§2.2 and §3.4.1):
+
+Added explicit message header specifications to data flow diagram:
+
+```markdown
+│  │ [Enrich] → (add message headers for routing)              │  │
+│  │   • exchange: "coinbase" (source exchange)                │  │
+│  │   • symbol: "BTC-USD" (trading pair)                      │  │
+│  │   • data_type: "trade" (message type)                     │  │
+│  │   • schema_version: "1.0" (protobuf schema version)       │  │
+│  │   • timestamp: RFC3339 (message generation time)          │  │
+```
+
+**3. Tracking Phase (Spec Metadata):**
+
+Updated `.kiro/specs/market-data-kafka-producer/spec.json`:
+
+```json
+"post_validation_refinements": {
+  "date": "2025-11-26",
+  "findings_addressed": 2,
+  "changes": [
+    "Fixed E2E test topic naming expectations (consolidated vs per-symbol)",
+    "Aligned design.md with approved Blue-Green migration strategy",
+    "Updated performance targets to validated 150k+ msg/s",
+    "Enhanced architecture diagrams with message header specifications"
+  ],
+  "commit": "53f9e548",
+  "test_pass_rate": "100%"
+}
+```
+
+**Commits Created:**
+- `53f9e548` - Fixed all validation findings (E2E test + design.md updates)
+- `b244e6f0` - Updated spec.json with post-validation refinements
+- `30d4136e` - Removed standalone /todos files after integrating into spec.json
+
+## Why This Works
+
+**Root Cause Analysis:**
+
+1. **Design Documentation Drift**: Design.md was drafted early in the specification process (before requirements were finalized). When requirements changed (migration strategy: dual-write → Blue-Green), the design document wasn't updated systematically.
+
+2. **Test Legacy Behavior**: E2E test was written before the consolidated topic strategy became the default. The test validated per-symbol topic naming (legacy behavior) instead of consolidated topics (actual default).
+
+3. **Missing Validation Step**: The development workflow lacked systematic validation between requirements ↔ design ↔ implementation before production deployment.
+
+**Why the Solution Works:**
+
+1. **Multi-Agent Validation**: Using dedicated validation subagents (`validate-design-agent`, `validate-impl-agent`) systematically checks alignment across all specification artifacts.
+
+2. **Atomic Fixes**: All related changes fixed in a single commit (53f9e548) ensures consistency and traceability.
+
+3. **Metadata Tracking**: `spec.json` metadata provides permanent record of validation findings and resolutions, making the process auditable.
+
+4. **Test Validation**: E2E test now validates the actual default behavior (consolidated topics), not legacy behavior.
+
+## Prevention
+
+**How to avoid this problem in future specification development:**
+
+1. **Always Run Validation Before Production**:
+   ```bash
+   # Required workflow before declaring "production ready"
+   /kiro:validate-design {feature}     # Checks requirements ↔ design alignment
+   /kiro:validate-impl {feature}       # Checks design ↔ implementation alignment
+   ```
+
+2. **Update Design.md When Requirements Change**:
+   - If requirements.md is modified after design approval, immediately update design.md
+   - Run `/kiro:validate-design` after any requirements change to surface drift
+
+3. **Write Tests for Default Behavior**:
+   - E2E tests should validate the default configuration, not legacy/optional behavior
+   - Use comments to document why specific behavior is tested: `# Validates consolidated topic strategy (default)`
+
+4. **Track Validation Findings in Spec.json**:
+   - Don't use standalone /todos files for validation findings
+   - Use `post_validation_refinements` section in spec.json for permanent tracking
+   - Include commit hash for traceability
+
+5. **Establish Validation Gates**:
+   - Phase 1-4: Implementation and testing
+   - Phase 5: Pre-production validation (run all validation subagents)
+   - Phase 6: Production deployment only after 100% test pass rate + zero validation findings
+
+6. **Keep Architecture Diagrams Current**:
+   - When adding features (like message headers), update all relevant diagram sections
+   - Check both high-level diagrams (§2.2) and implementation details (§3.4.1)
+
+## Related Issues
+
+**Promoted to Required Reading:**
+- See **[Kiro Specification Critical Patterns](../patterns/kiro-spec-critical-patterns.md)** - This solution has been promoted to required reading as patterns #1, #2, and #3:
+  - Pattern #1: Always Run Multi-Agent Validation Before Production
+  - Pattern #2: Track Validation Findings in Spec.json
+  - Pattern #3: Test Default Behavior, Not Legacy Options
+
+No other related issues documented yet.
+
+---
+
+**Confidence Note**: After applying these fixes, the market-data-kafka-producer specification achieved:
+- ✅ 100% test pass rate (629 tests)
+- ✅ Zero validation findings (all resolved)
+- ✅ HIGH confidence (95%) for production deployment
+- ✅ GO decision for Phase 5 execution