Skip to content

test-suite-organization.md

Latest commit

 

History

History
214 lines (165 loc) · 6.67 KB

test-suite-organization.md

File metadata and controls

214 lines (165 loc) · 6.67 KB

PyFlowGraph Test Suite Organization

Problem Solved

You were absolutely right - the existing tests were mostly pseudo-GUI tests that created QApplication but didn't actually test real user interactions. This is why obvious GUI errors were slipping through despite tests passing.

New Test Organization

📁 tests/headless/ - Fast Unit Tests

  • Purpose: Core logic validation without GUI overhead
  • Speed: ⚡ Fast execution (< 10 seconds total)
  • When to run: During development, CI/CD, quick validation
  • Coverage: Business logic, data structures, algorithms

Files:

  • test_node_system.py - Node creation, properties, code management
  • test_pin_system.py - Pin generation, type detection, positioning
  • test_connection_system.py - Connection logic, validation, serialization

📁 tests/gui/ - Real GUI Integration Tests

  • Purpose: Actual user interaction testing with visible windows
  • Speed: 🐌 Slower execution (30+ seconds) but catches real issues
  • When to run: Before releases, when GUI bugs suspected, comprehensive validation
  • Coverage: User workflows, visual behavior, integration issues

Files:

  • test_full_gui_integration.py - Complete GUI component testing
  • test_end_to_end_workflows.py - Real user workflow simulation
  • test_gui_node_deletion.py - Specific GUI interaction tests
  • test_user_scenario.py - Bug reproduction scenarios

Test Runners

🚀 Quick Development Testing

# Fast headless tests only
run_headless_tests.bat

🖥️ Full GUI Testing

# Complete GUI integration tests
run_gui_tests.bat

🎛️ Enhanced Test Runner

# Visual test management with categories
run_enhanced_test_gui.bat

What Makes These Tests Different

❌ Old Approach (Pseudo-GUI)

# Creates QApplication but doesn't show windows
app = QApplication([])
node = Node("Test")
# Tests logic but misses visual/interaction issues

✅ New Approach (Real GUI)

# Actually shows windows and tests user interactions
self.window = NodeEditorWindow()
self.window.show()          # Window is visible!
self.window.resize(1200, 800)
QApplication.processEvents()  # Real event processing

# Simulates actual user actions
node.setSelected(True)
delete_event = QKeyEvent(QKeyEvent.KeyPress, Qt.Key_Delete, Qt.NoModifier)
self.graph.keyPressEvent(delete_event)

# Verifies visual results
self.assertTrue(node.isVisible())

Key Test Categories

1. Application Startup (TestApplicationStartup)

  • Window opens correctly
  • Menu bar exists and functional
  • Components initialize properly

2. Node Creation Workflows (TestNodeCreationWorkflow)

  • Context menu node creation
  • Node selection and properties
  • Code editing workflow

3. Connection Workflows (TestConnectionWorkflow)

  • Creating connections between compatible nodes
  • Visual feedback during connection
  • Connection validation

4. Reroute Node Testing (TestRerouteNodeWorkflow)

  • Creation and deletion of reroute nodes
  • Critical: Undo/redo cycle (addresses user-reported bug)

5. End-to-End Workflows (TestEnd2EndWorkflows)

  • Complete data processing pipelines
  • Save/load file operations
  • Error recovery scenarios

Critical Tests That Catch Real Issues

Reroute Node Bug Test

def test_reroute_node_undo_redo_cycle(self):
    """This test specifically addresses the user-reported bug"""
    
    # Create reroute node
    reroute = RerouteNode()
    self.assertIsInstance(reroute, RerouteNode)
    
    # Delete it
    delete_event = QKeyEvent(QKeyEvent.KeyPress, Qt.Key_Delete, Qt.NoModifier)
    self.graph.keyPressEvent(delete_event)
    
    # Undo deletion (this is where the bug occurred)
    undo_event = QKeyEvent(QKeyEvent.KeyPress, Qt.Key_Z, Qt.ControlModifier)
    self.view.keyPressEvent(undo_event)
    
    # CRITICAL TEST: Restored node should still be a RerouteNode
    restored_node = find_restored_node("Reroute")
    self.assertIsInstance(restored_node, RerouteNode, 
                         "CRITICAL BUG: RerouteNode was restored as regular Node!")

Data Processing Pipeline Test

def test_create_simple_data_pipeline(self):
    """Tests complete user workflow: create -> connect -> execute"""
    
    # User creates input node
    input_node = self.graph.create_node("Data Generator", pos=(100, 200))
    input_node.set_code('@node_entry\ndef generate() -> list:\n    return [1,2,3]')
    
    # User creates processing node  
    process_node = self.graph.create_node("Processor", pos=(400, 200))
    process_node.set_code('@node_entry\ndef process(data: list) -> list:\n    return [x*2 for x in data]')
    
    # User connects nodes
    connection = self.graph.create_connection(output_pin, input_pin)
    
    # Verify complete pipeline works
    self.assertTrue(connection.isVisible())
    self.assertEqual(len(self.graph.connections), 1)

When to Run Which Tests

🔄 During Development

# Quick feedback loop
run_headless_tests.bat

🚀 Before Committing

# Full validation
run_enhanced_test_gui.bat
# Select both Headless and GUI categories

🐛 When Investigating GUI Bugs

# Focus on GUI tests
run_gui_tests.bat

📦 Before Releases

# Complete test suite
run_enhanced_test_gui.bat
# Run ALL tests and ensure 100% pass rate

Enhanced Test Runner Features

The new run_enhanced_test_gui.bat provides:

  • 📊 Visual test organization by category
  • ⚡ Category-specific timeouts (GUI tests get more time)
  • 🎯 Selective execution (run only what you need)
  • 📝 Detailed output with real-time status
  • ⚠️ GUI test warnings (don't interact with test windows)

Test Results Interpretation

✅ Headless Tests Pass + GUI Tests Pass

  • Application is working correctly
  • Safe to release/deploy

✅ Headless Tests Pass + ❌ GUI Tests Fail

  • Your exact situation! Logic works but user experience is broken
  • Focus on GUI test failures to find integration issues

❌ Headless Tests Fail + GUI Tests Any

  • Core logic problems
  • Fix headless issues first

❌ Both Test Categories Fail

  • Major issues requiring comprehensive fixes

Benefits

  1. 🎯 Catches Real Issues: GUI tests find problems users actually encounter
  2. ⚡ Fast Development: Headless tests provide quick feedback
  3. 🔧 Organized Debugging: Know exactly which layer has issues
  4. 📊 Better Coverage: Tests both logic AND user experience
  5. 🚀 Confidence: Release knowing the GUI actually works

This testing approach ensures that when tests pass, the application actually works for real users, not just in theory!