feat: implement deliberate context compaction with goal-aware compression

[kimjune01]

Summary

Implements deliberate context compaction - a goal-aware conversation compression system that intelligently preserves context relevant to the user's current work.

This PR implements the complete TDD work plan outlined in TDD_WORK_PLAN.md. The feature uses a hybrid trigger system, extracts user goals via LLM, prompts for goal selection, and performs goal-focused compression.

This is not yet production-grade code, given the time constraints for the coding exercise

📖 How to Review This PR

Recommended Review Order

1. Start with the design documentation (5 min)

• Read docs/DELIBERATE_CONTEXT_COMPACTION.md - Architecture overview and design decisions
• Read TDD_WORK_PLAN.md - Implementation plan with acceptance criteria for each phase

2. Review the test files first (15-20 min)
Following TDD principles, tests define the behavior. Review in this order:

• packages/core/src/services/goalExtractionService.test.ts - Goal extraction from conversation
• packages/core/src/services/deliberateCompressionHandler.test.ts - Opt-out decision logic
• packages/core/src/services/deliberateCompressionOrchestrator.test.ts - Integration orchestration
• packages/core/src/core/client.test.ts (lines 2410-2525) - Trigger system tests
• packages/cli/src/ui/components/GoalSelectionPrompt.test.tsx - UI component tests
• packages/core/src/telemetry/deliberate-compression-telemetry.test.ts - Telemetry tests

3. Review the implementation (20-25 min)
After understanding the tests, review the implementation:

• packages/core/src/services/goalExtractionService.ts - Goal extraction service
• packages/core/src/services/deliberateCompressionHandler.ts - Opt-out handling
• packages/core/src/services/deliberateCompressionOrchestrator.ts - Main orchestrator
• packages/core/src/core/client.ts (lines 117-874) - Trigger system & integration
• packages/cli/src/ui/components/GoalSelectionPrompt.tsx - Goal selection UI
• packages/core/src/core/prompts.ts (getChatCompressionPrompt) - Enhanced prompt
• packages/core/src/services/chatCompressionService.ts (compress method) - Integration points

4. Review configuration and telemetry (10 min)

• packages/core/src/config/config.ts (lines 1217-1280) - 6 new config methods
• packages/core/src/code_assist/experiments/flagNames.ts - Experiment flags
• packages/core/src/telemetry/types.ts (ChatCompressionEvent) - Enhanced telemetry

🤖 Prompts for AI-Assisted Review

Use these prompts in your AI assistant to speed up review:

For architecture understanding:

Review the deliberate context compaction feature. Explain:
1. How the hybrid trigger system works (absolute threshold vs safety valve)
2. The goal extraction and selection flow
3. How goal-focused compression differs from standard compression

For test coverage analysis:

Analyze test coverage for deliberate compression:
1. Are all edge cases covered in deliberateCompressionHandler.test.ts?
2. Do the orchestrator tests validate the complete flow?
3. Are there any missing integration tests?

For code quality check:

Review the TypeScript implementation for:
1. Type safety - are all types properly defined?
2. Error handling - are errors properly caught and logged?
3. Concurrency - are race conditions handled in the trigger system?

For configuration review:

Review the configuration approach:
1. Are default values sensible?
2. Is the experiment flag integration correct?
3. Can these settings be overridden appropriately?

🧪 Testing Instructions

Manual Testing

Prerequisites:

npm install
npm run build
npm start

Test Scenario 1: Goal extraction and selection

1. Start a long conversation (40+ messages) about implementing a feature
2. Wait for compression to trigger
3. Verify goal selection prompt appears with extracted goals
4. Select a goal and verify compression completes

Test Scenario 2: Safety valve trigger

1. Configure a small token limit model
2. Fill the context to >50% utilization
3. Verify safety valve triggers compression even below absolute threshold

Test Scenario 3: Opt-out handling

1. Enable auto-skip: Set experiment flag DELIBERATE_COMPRESSION_AUTO_SKIP=true
2. Verify compression proceeds without prompting

Automated Testing

npm test -- packages/core/src/services/goalExtractionService.test.ts
npm test -- packages/core/src/services/deliberateCompressionHandler.test.ts
npm test -- packages/core/src/services/deliberateCompressionOrchestrator.test.ts
npm test -- packages/core/src/core/client.test.ts
npm test -- packages/cli/src/ui/components/GoalSelectionPrompt.test.tsx
npm test -- packages/core/src/telemetry/deliberate-compression-telemetry.test.ts

📊 Implementation Stats

• Total commits: 12 (11 implementation + 1 compilation fix)
• Test files created: 6
• Total tests added: 50+
• Implementation files: 7 new services/components
• Modified files: 6 existing files enhanced
• Experiment flags: 6 new configuration flags
• Lines of code: ~2,500+ (including tests)

🔍 Key Changes by Category

New Services

• GoalExtractionService - Extracts user goals from conversation using LLM
• DeliberateCompressionHandler - Manages opt-out logic and user prompting decisions
• DeliberateCompressionOrchestrator - Orchestrates the complete deliberate compression flow

Enhanced Existing Services

• ChatCompressionService - Added split strategies, goal-focused prompts, discarded context extraction
• GeminiClient - Integrated hybrid trigger system, goal extraction, and compression orchestration

UI Components

• GoalSelectionPrompt - Ink/React component for goal selection interface

Configuration

• 6 new experiment flags for controlling compression behavior
• Default values tuned for production use

Telemetry

• Enhanced ChatCompressionEvent with goal selection, message counts, and trigger reason

🚀 Feature Flags

All features are controlled by experiment flags (default values shown):

• COMPRESSION_TRIGGER_TOKENS - Absolute token threshold (default: 40000)
• COMPRESSION_TRIGGER_UTILIZATION - Safety valve utilization (default: 0.5)
• COMPRESSION_MIN_MESSAGES - Min messages between compressions (default: 25)
• COMPRESSION_MIN_TIME_BETWEEN_PROMPTS - Min time between compressions (default: 300s)
• DELIBERATE_COMPRESSION_ENABLED - Enable deliberate compression (default: true)
• DELIBERATE_COMPRESSION_AUTO_SKIP - Auto-skip goal selection (default: false)

✅ Acceptance Criteria

All acceptance criteria from the TDD work plan have been met:

• ✅ Goal extraction with XML parsing
• ✅ Hybrid trigger system with guards
• ✅ Goal selection UI component
• ✅ Opt-out handling for all edge cases
• ✅ Integration with existing compression service
• ✅ Configuration via experiment flags
• ✅ Enhanced telemetry tracking
• ✅ Comprehensive test coverage (50+ tests)
• ✅ All tests passing
• ✅ Build succeeds with no TypeScript errors

📝 Notes

• Removed streaming guard in trigger system since Turn.isStreaming() doesn't exist yet (noted in code comment)
• Used TDD methodology: all tests written before implementation
• Each phase committed incrementally for easier review
• All default values are conservative and tuned for production safety

🤖 Generated with Claude Code

[kimjune01]

Superseded by google-gemini#24736