Code Reorganization Migration Plan

Document Version: 1.0
Created: 2024-08-16
Author: Winston (Architect)
Status: Ready for Implementation

Overview

This document outlines a comprehensive migration plan to reorganize the PyFlowGraph codebase from a flat structure to a well-organized, modular architecture. The current structure has 24 files in the main src/ directory with only the commands/ subdirectory properly organized.

Current Structure Analysis

Functional Areas Identified

Core Graph Components: node.py, pin.py, connection.py, reroute_node.py, node_graph.py
UI/Editor Components: node_editor_window.py, node_editor_view.py, various dialogs
Execution & Environment: graph_executor.py, execution_controller.py, environment managers
File & Data Operations: file_operations.py, flow_format.py, view_state_manager.py
Utilities: color_utils.py, ui_utils.py, debug_config.py, event_system.py
Code Editing: python_code_editor.py, python_syntax_highlighter.py, code_editor_dialog.py

Proposed Target Structure

src/
├── __init__.py
├── main.py                 # Entry point (stays at root)
├── core/                   # Core graph engine
│   ├── __init__.py
│   ├── node.py
│   ├── pin.py
│   ├── connection.py
│   ├── reroute_node.py
│   ├── node_graph.py
│   └── event_system.py
├── ui/                     # User interface components
│   ├── __init__.py
│   ├── editor/
│   │   ├── __init__.py
│   │   ├── node_editor_window.py
│   │   ├── node_editor_view.py
│   │   └── view_state_manager.py
│   ├── dialogs/
│   │   ├── __init__.py
│   │   ├── code_editor_dialog.py
│   │   ├── node_properties_dialog.py
│   │   ├── graph_properties_dialog.py
│   │   ├── settings_dialog.py
│   │   └── environment_selection_dialog.py
│   ├── code_editing/
│   │   ├── __init__.py
│   │   ├── python_code_editor.py
│   │   └── python_syntax_highlighter.py
│   └── utils/
│       ├── __init__.py
│       └── ui_utils.py
├── execution/              # Code execution and environments
│   ├── __init__.py
│   ├── graph_executor.py
│   ├── execution_controller.py
│   ├── environment_manager.py
│   └── default_environment_manager.py
├── data/                   # Data persistence and formats
│   ├── __init__.py
│   ├── file_operations.py
│   └── flow_format.py
├── commands/               # Command pattern (already organized)
│   ├── __init__.py
│   ├── command_base.py
│   ├── command_history.py
│   ├── connection_commands.py
│   └── node_commands.py
├── utils/                  # Shared utilities
│   ├── __init__.py
│   ├── color_utils.py
│   └── debug_config.py
├── testing/                # Test infrastructure
│   ├── __init__.py
│   └── test_runner_gui.py
└── resources/              # Static resources (already organized)
    ├── Font Awesome 6 Free-Solid-900.otf
    └── Font Awesome 7 Free-Regular-400.otf

Migration Strategy

Architectural Benefits

Clear Separation of Concerns: Each directory has a single responsibility
Logical Grouping: Related files are co-located
Scalability: Easy to add new components within appropriate categories
Import Clarity: Import paths clearly indicate component relationships
Team Development: Different developers can work on different areas with minimal conflicts

Phase 1: Preparation & Analysis

Duration: 30-60 minutes

Step 1: Create Migration Backup

# Create a backup branch
git checkout -b backup/pre-reorganization
git push origin backup/pre-reorganization

# Create working branch
git checkout -b refactor/code-organization

Step 2: Dependency Analysis

Before moving files, map all import relationships:

# Analyze current imports (run from project root)
grep -r "from.*import" src/ > import_analysis.txt
grep -r "import.*" src/ >> import_analysis.txt

Key imports to track:

Cross-module dependencies (e.g., node.py importing pin.py)
Circular imports (potential issues)
External library imports (remain unchanged)
Relative vs absolute imports

Phase 2: Directory Structure Creation

Duration: 15 minutes

Step 3: Create New Directory Structure

# Create all new directories
mkdir -p src/core
mkdir -p src/ui/editor
mkdir -p src/ui/dialogs  
mkdir -p src/ui/code_editing
mkdir -p src/ui/utils
mkdir -p src/execution
mkdir -p src/data
mkdir -p src/utils
mkdir -p src/testing

Step 4: Create `init.py` Files

src/core/init.py

"""Core graph engine components."""
from .node import Node
from .pin import Pin
from .connection import Connection
from .reroute_node import RerouteNode
from .node_graph import NodeGraph
from .event_system import EventSystem

__all__ = [
    'Node', 'Pin', 'Connection', 'RerouteNode', 
    'NodeGraph', 'EventSystem'
]

src/ui/init.py

"""User interface components."""
from .editor import NodeEditorWindow, NodeEditorView
from .dialogs import (
    CodeEditorDialog, NodePropertiesDialog, 
    GraphPropertiesDialog, SettingsDialog
)

__all__ = [
    'NodeEditorWindow', 'NodeEditorView',
    'CodeEditorDialog', 'NodePropertiesDialog',
    'GraphPropertiesDialog', 'SettingsDialog'
]

src/ui/editor/init.py

"""Node editor UI components."""
from .node_editor_window import NodeEditorWindow
from .node_editor_view import NodeEditorView
from .view_state_manager import ViewStateManager

__all__ = ['NodeEditorWindow', 'NodeEditorView', 'ViewStateManager']

src/ui/dialogs/init.py

"""Dialog components."""
from .code_editor_dialog import CodeEditorDialog
from .node_properties_dialog import NodePropertiesDialog
from .graph_properties_dialog import GraphPropertiesDialog
from .settings_dialog import SettingsDialog
from .environment_selection_dialog import EnvironmentSelectionDialog

__all__ = [
    'CodeEditorDialog', 'NodePropertiesDialog', 'GraphPropertiesDialog',
    'SettingsDialog', 'EnvironmentSelectionDialog'
]

src/ui/code_editing/init.py

"""Code editing components."""
from .python_code_editor import PythonCodeEditor
from .python_syntax_highlighter import PythonSyntaxHighlighter

__all__ = ['PythonCodeEditor', 'PythonSyntaxHighlighter']

src/ui/utils/init.py

"""UI utility functions."""
from .ui_utils import *

__all__ = ['ui_utils']

src/execution/init.py

"""Code execution and environment management."""
from .graph_executor import GraphExecutor
from .execution_controller import ExecutionController
from .environment_manager import EnvironmentManager
from .default_environment_manager import DefaultEnvironmentManager

__all__ = [
    'GraphExecutor', 'ExecutionController', 
    'EnvironmentManager', 'DefaultEnvironmentManager'
]

src/data/init.py

"""Data persistence and format handling."""
from .file_operations import FileOperationsManager
from .flow_format import FlowFormatHandler

__all__ = ['FileOperationsManager', 'FlowFormatHandler']

src/utils/init.py

"""Shared utility functions."""
from . import color_utils
from . import debug_config

__all__ = ['color_utils', 'debug_config']

src/testing/init.py

"""Testing infrastructure."""
from .test_runner_gui import TestRunnerGUI

__all__ = ['TestRunnerGUI']

Phase 3: File Migration

Duration: 45-60 minutes

Step 5: Move Files in Dependency Order

Move files in this specific order to minimize import issues:

Phase 3A: Utilities First (No dependencies)

# Move utilities (these have no internal dependencies)
mv src/color_utils.py src/utils/
mv src/debug_config.py src/utils/
mv src/ui_utils.py src/ui/utils/
mv src/test_runner_gui.py src/testing/

Phase 3B: Core Components

# Move core graph components
mv src/event_system.py src/core/
mv src/pin.py src/core/
mv src/connection.py src/core/
mv src/reroute_node.py src/core/
mv src/node.py src/core/
mv src/node_graph.py src/core/

Phase 3C: Execution Components

# Move execution-related files
mv src/graph_executor.py src/execution/
mv src/execution_controller.py src/execution/
mv src/environment_manager.py src/execution/
mv src/default_environment_manager.py src/execution/
mv src/environment_selection_dialog.py src/ui/dialogs/

Phase 3D: Data & File Operations

# Move data handling
mv src/file_operations.py src/data/
mv src/flow_format.py src/data/

Phase 3E: UI Components

# Move UI components
mv src/node_editor_window.py src/ui/editor/
mv src/node_editor_view.py src/ui/editor/
mv src/view_state_manager.py src/ui/editor/

# Move dialogs
mv src/code_editor_dialog.py src/ui/dialogs/
mv src/node_properties_dialog.py src/ui/dialogs/
mv src/graph_properties_dialog.py src/ui/dialogs/
mv src/settings_dialog.py src/ui/dialogs/

# Move code editing
mv src/python_code_editor.py src/ui/code_editing/
mv src/python_syntax_highlighter.py src/ui/code_editing/

Phase 4: Import Updates

Duration: 60-90 minutes

Step 6: Update Import Statements

This is the most critical phase. Update imports systematically:

Pattern for updates:

# OLD
from node import Node
from pin import Pin

# NEW  
from src.core.node import Node
from src.core.pin import Pin

# OR (preferred for cleaner imports)
from src.core import Node, Pin

Key files requiring major import updates:

main.py - Entry point, imports many modules
node_editor_window.py - Central UI component
node_graph.py - Core component with many dependencies
Commands - Already organized, but need path updates

Step 7: Update Relative Imports

Convert relative imports to absolute imports using the new structure:

# Before
from commands import CommandHistory

# After
from src.commands import CommandHistory

Common Import Update Patterns:

# Core components
from src.core import Node, Pin, Connection, NodeGraph
from src.core.node import Node
from src.core.pin import Pin

# UI components  
from src.ui.editor import NodeEditorWindow, NodeEditorView
from src.ui.dialogs import CodeEditorDialog, NodePropertiesDialog
from src.ui.code_editing import PythonCodeEditor

# Execution
from src.execution import GraphExecutor, ExecutionController

# Data handling
from src.data import FileOperationsManager, FlowFormatHandler

# Utilities
from src.utils import color_utils, debug_config
from src.utils.color_utils import generate_color_from_string

# Commands (updated paths)
from src.commands import CommandHistory, DeleteNodeCommand

Phase 5: Testing & Validation

Duration: 30-45 minutes

Step 8: Incremental Testing

Test after each major group of changes:

# Test basic import functionality
python -c "import src.core; print('Core imports OK')"
python -c "import src.ui; print('UI imports OK')"
python -c "import src.execution; print('Execution imports OK')"
python -c "import src.data; print('Data imports OK')"
python -c "import src.utils; print('Utils imports OK')"

# Test application startup
python src/main.py

Step 9: Run Full Test Suite

# Run your existing tests
python src/testing/test_runner_gui.py

# Manual smoke tests:
# 1. Create a node
# 2. Delete and undo
# 3. Save and load a file
# 4. Execute a simple graph
# 5. Test GUI functionality

Step 10: Validate Core Functionality

✅ Application starts without import errors
✅ Node creation and manipulation
✅ Pin connections work correctly
✅ Undo/redo system functions
✅ File save/load operations
✅ Code execution works
✅ GUI dialogs open properly

Phase 6: Cleanup & Documentation

Duration: 15-30 minutes

Step 11: Clean Up Old References

Remove any empty directories
Update any documentation referencing old paths
Update any scripts or build configurations
Update requirements.txt if needed

Step 12: Update Import Guidelines

Create development guidelines documenting the new import patterns:

# Recommended import patterns for new structure

# Core components
from src.core import Node, Pin, Connection

# UI components  
from src.ui.editor import NodeEditorWindow
from src.ui.dialogs import CodeEditorDialog

# Utilities
from src.utils import color_utils, debug_config

Risk Mitigation Strategies

1. Checkpoint Strategy

Commit after each major phase
Tag working versions: git tag checkpoint-phase-3
Keep backup branch for quick rollback

2. Import Compatibility Layer (Temporary)

If needed, create temporary compatibility imports in old locations:

# src/node.py (temporary compatibility)
from src.core.node import Node

3. Gradual Migration Alternative

If issues arise, consider gradual migration:

Move one module at a time
Test thoroughly before next move
Keep old imports working via compatibility layer

Expected Challenges & Solutions

Challenge 1: Circular Imports

Solution: Use late imports or restructure dependencies

# Instead of top-level import
def get_node_graph():
    from src.core import NodeGraph
    return NodeGraph

Challenge 2: Command Pattern Dependencies

Solution: Update command imports to use new paths

# In command files
from src.core import Node, Connection
from src.ui.editor import NodeEditorView

Challenge 3: Main Entry Point

Solution: Update main.py to use new structure

# main.py updates
from src.ui.editor import NodeEditorWindow
from src.utils import debug_config

Challenge 4: Resource Path References

Solution: Update any hardcoded paths to resources

# Update resource references
from src.resources import font_path

Success Metrics

✅ Application starts without import errors
✅ All existing functionality works
✅ Test suite passes
✅ No circular import warnings
✅ Clean, logical import statements
✅ Improved code maintainability
✅ Clear module boundaries

Timeline Summary

Phase 1-2: 1 hour (Prep + Structure)
Phase 3: 1 hour (File moves)
Phase 4: 1.5 hours (Import updates)
Phase 5: 45 minutes (Testing)
Phase 6: 30 minutes (Cleanup)

Total Estimated Time: 4-5 hours

Post-Migration Benefits

Developer Experience

Clearer Code Organization: Easier to find related functionality
Reduced Cognitive Load: Logical grouping reduces mental overhead
Better IDE Support: IDEs can better understand module relationships
Improved Code Navigation: Related files are co-located

Maintenance Benefits

Easier Debugging: Clear separation helps isolate issues
Simplified Testing: Can test modules in isolation
Better Documentation: Clear module boundaries for API docs
Reduced Merge Conflicts: Different teams can work on different modules

Future Development

Scalable Architecture: Easy to add new features within appropriate modules
Plugin Architecture: Clear boundaries enable plugin development
Performance Optimization: Can optimize modules independently
Code Reuse: Well-defined modules can be reused across projects

Implementation Notes

Backup Everything: This is a major structural change
Test Frequently: After each phase, ensure functionality works
Commit Often: Small commits make rollback easier if needed
Update Documentation: Keep docs in sync with new structure
Team Coordination: If working with others, coordinate the migration

Document Status: Ready for Implementation
Next Steps: Begin with Phase 1 preparation and analysis

This migration plan follows Python packaging best practices and maintains backward compatibility during the transition period.

Uh oh!

FilesExpand file tree

code-reorganization-migration-plan.md

Latest commit

History