Skip to content

TESTING_GUIDE.md

Latest commit

 

History

History
353 lines (274 loc) · 9.47 KB

TESTING_GUIDE.md

File metadata and controls

353 lines (274 loc) · 9.47 KB
Raw

PyFlowGraph Testing Guide

Quick Start

Installation

First, install the testing dependencies:

pip install -r requirements.txt

Basic Testing Workflow

# Run fast tests (recommended for development)
python test_runner.py --fast --format claude

# Run complete test suite with analysis
python test_runner.py --coverage --analyze --format claude

# Check test suite health
python test_analyzer.py --format claude

Testing Infrastructure Overview

New Files Added

PyFlowGraph/
├── pytest.ini                     # Pytest configuration with parallel execution
├── test_runner.py                  # Advanced test runner with performance tracking
├── test_analyzer.py               # Test failure analysis and coverage reporting
├── test_generator.py              # Automated test generation from coverage gaps
├── claude_agents/
│   └── test_analysis_agent.md     # Claude Code test analysis agent
├── claude_commands/
│   ├── test_command.md            # Enhanced /test command
│   ├── fix_tests_command.md       # /fix-tests command
│   └── test_health_command.md     # /test-health command
└── TESTING_GUIDE.md               # This guide

Enhanced Features

  • 67-81% faster execution through parallel testing
  • Token-efficient reporting for Claude Code integration
  • Automated failure analysis with fix suggestions
  • Coverage gap identification with test generation
  • Performance monitoring with bottleneck detection

Test Runner (test_runner.py)

Basic Usage

# Default: Run all tests in parallel
python test_runner.py

# Fast development cycle (unit + headless tests only)
python test_runner.py --fast

# GUI tests only (sequential to avoid QApplication conflicts)
python test_runner.py --gui-only --no-parallel

# Coverage analysis
python test_runner.py --coverage --format claude

Advanced Options

# Test only changed files (requires git)
python test_runner.py --changed --fast

# Performance analysis
python test_runner.py --workers 4 --profile --timeout 15

# Save results for analysis
python test_runner.py --save results.json --format json

Example Output (Claude Format)

=== TEST EXECUTION REPORT ===
Total: 47 | Pass: 45 | Fail: 2 | Skip: 0
Duration: 23.4s | Workers: 4

=== FAILURES ===
✗ test_node_deletion (2.1s) - AssertionError: Node not removed
✗ test_gui_startup (5.8s) - QApplication RuntimeError

=== PERFORMANCE ===
Slow tests: 3 | Avg: 0.8s | Max: 5.8s
Categories: gui:2.1s | unit:0.3s | integration:1.2s

Test Analyzer (test_analyzer.py)

Basic Analysis

# Analyze latest test results
python test_analyzer.py --format claude

# Focus on coverage gaps only
python test_analyzer.py --coverage-only

# Save detailed report
python test_analyzer.py --format detailed --output-file analysis.md

Features

  • Failure Pattern Recognition: Categorizes failures (Qt issues, imports, timeouts)
  • Coverage Gap Analysis: Identifies untested functions with priority scoring
  • Performance Bottlenecks: Detects slow tests and optimization opportunities
  • Flaky Test Detection: Statistical analysis across multiple runs

Example Output (Claude Format)

=== TEST ANALYSIS REPORT ===
Health Score: 84.1/100
Analysis Time: 2025-01-18 10:30:45

=== TOP FAILURE PATTERNS ===
• qt_application: 3 tests
  Fix: Use class-level QApplication setup
• import_error: 1 test
  Fix: Check PYTHONPATH and module dependencies

=== HIGH PRIORITY COVERAGE GAPS ===
• src/core/node.py::calculate_bounds (8 lines)
• src/execution/graph_executor.py::handle_timeout (12 lines)

=== RECOMMENDATIONS ===
1. Fix QApplication lifecycle conflicts in GUI tests
2. Add tests for NodeGraph.clear_graph() method
3. Optimize test_gui_startup performance (<5s target)

Test Generator (test_generator.py)

Coverage-Driven Test Generation

# Generate tests for top 10 complex functions
python test_generator.py

# Generate tests for high-complexity functions only
python test_generator.py --min-complexity 2.0 --max-functions 5

# Analyze coverage gaps without generating tests
python test_generator.py --analyze-only --format claude

Features

  • AST-based analysis of source code structure
  • PyFlowGraph-specific templates for Node, Pin, Connection tests
  • Smart categorization (unit, integration, GUI, headless)
  • Pattern learning from existing test conventions

Example Generated Test

def test_update_position(self):
    """Test update_position functionality."""
    # Arrange
    node = Node("TestNode")
    # Add setup code as needed
    
    # Act
    result = node.update_position(QPointF(100, 100))
    
    # Assert
    self.assertIsNotNone(result)
    # TODO: Add specific assertions for this function

Claude Code Integration

Enhanced /test Command

# Fast development workflow
/test fast --parallel --format claude

# Coverage-driven development  
/test changed --coverage --analyze

# Performance optimization
/test slow --profile --analyze

# CI/CD integration
/test all --parallel --format json --save results.json

/fix-tests Command

# Auto-fix common issues
/fix-tests auto --confidence 0.8

# Interactive guided fixes
/fix-tests guided --pattern qt_application

# Preview fixes without applying
/fix-tests all --dry-run --format claude

/test-health Command

# Quick health overview
/test-health overview --format claude

# Detailed health analysis
/test-health detailed --period 30

# Performance-focused health check
/test-health performance --alerts

Development Workflows

Daily Development Cycle

# 1. Quick feedback during coding
python test_runner.py --fast --format claude

# 2. Coverage check before commit
python test_runner.py --changed --coverage

# 3. Health check weekly
python test_analyzer.py --format claude

Bug Investigation Workflow

# 1. Reproduce and analyze failure
python test_runner.py --test specific_test --verbose

# 2. Analyze failure patterns
python test_analyzer.py --format detailed

# 3. Generate missing tests if needed
python test_generator.py --analyze-only

Performance Optimization Workflow

# 1. Identify slow tests
python test_runner.py --profile --format claude

# 2. Analyze bottlenecks
python test_analyzer.py --format claude

# 3. Monitor improvements
python test_runner.py --benchmark --save before.json
# ... make optimizations ...
python test_runner.py --benchmark --save after.json

Best Practices

Test Organization

  • Unit tests: Fast, isolated component testing
  • Integration tests: Component interaction testing
  • GUI tests: User interface and workflow testing
  • Headless tests: Logic testing without GUI dependencies

Performance Guidelines

  • Unit tests: <0.5s each
  • Integration tests: <2.0s each
  • GUI tests: <5.0s each (enforced by timeout)
  • Total suite: <120s with parallel execution

Coverage Targets

  • Critical components: >90% coverage
  • Core functionality: >80% coverage
  • UI components: >70% coverage
  • Utility functions: >60% coverage

Troubleshooting

Common Issues

QApplication Conflicts

Error: QApplication RuntimeError
Fix: Use class-level QApplication setup in GUI tests

Import Path Issues

Error: ModuleNotFoundError
Fix: Use standardized src path setup:
src_path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'src')
sys.path.insert(0, src_path)

Test Timeouts

Error: Test timeout after 10s
Fix: Optimize setup/teardown or use mocking for expensive operations

Debug Commands

# Verbose test output
python test_runner.py --verbose --no-parallel

# Analyze specific failure pattern
python test_analyzer.py --pattern timeout

# Check test isolation
python test_runner.py --workers 1 --repeat 5

Migration from Existing Tests

Gradual Adoption

  1. Install dependencies: Add pytest and related packages
  2. Run existing tests: Use test_runner.py with current test files
  3. Add markers: Gradually add pytest markers for categorization
  4. Enable parallel: Test parallel execution with --no-parallel fallback
  5. Integrate analysis: Use test_analyzer.py for insights

Compatibility

  • Existing test_*.py files: Work unchanged with new infrastructure
  • run_test_gui.bat: Continues to work alongside new tools
  • Test patterns: Existing setUp/tearDown patterns are preserved
  • Import styles: Current import patterns are maintained

Integration with CI/CD

GitHub Actions Example

- name: Run Tests with Analysis
  run: |
    python test_runner.py --parallel --format json --save results.json
    python test_analyzer.py --results results.json --format summary

Quality Gates

# Fail build if health score too low
python test_analyzer.py --format json | jq '.test_health_score < 70' && exit 1

# Fail build if coverage drops
python test_runner.py --coverage --format json | jq '.coverage.total < 80' && exit 1

Future Enhancements

Planned Features

  • Automatic test generation from code changes
  • Machine learning for failure prediction
  • Visual test reports with interactive dashboards
  • Integration with external monitoring tools

Community Contributions

  • Custom fix patterns for domain-specific issues
  • Additional test templates for common PyFlowGraph patterns
  • Performance optimizations for large test suites
  • Enhanced reporting formats for different use cases