Developer Documentation

📋 Document Metadata
Type: Development Guide | Audience: Developers & Contributors | Complexity: Intermediate-Advanced
Cross-References: Testing Guide | API Documentation | AGENTS.md | doc/INDEX.md | doc/SPEC.md (versioning policy)

Overview

This guide provides information for developers contributing to the GNN project, including code organization, development workflows, and architecture patterns.

Documentation audit tooling

From the repository root:

# Scan maintained Markdown for broken relative links (generated output/ is skipped)
uv run --extra dev python doc/development/docs_audit.py
# Fail the shell if any issues (for CI / pre-commit)
uv run --extra dev python doc/development/docs_audit.py --strict
# With --strict and failures, full per-issue lines go to stderr by default (fix loop). Use -q for summary only.
uv run --extra dev python doc/development/docs_audit.py --strict -q
# Optional: validate #fragments against heading slugs (heuristic; can be noisy)
uv run --extra dev python doc/development/docs_audit.py --check-anchors
# Optional: log markdown file count and other diagnostics to stderr
uv run --extra dev python doc/development/docs_audit.py --verbose
# Check maintained docs for stale PyMDP and policy terminology
uv run --extra dev python scripts/check_maintained_doc_terms.py --strict

Writes docs_audit_report.md: broken relative links, AGENTS.md→SPEC.md consistency, src/ dirs with .py but no AGENTS.md, maintained doc/ dirs missing AGENTS.md or README.md, AGENTS.md/README.md pairing under src/, doc/, .github/, and the repo root, and doc/**/AGENTS.md orientation (## Overview, ## Purpose, or ## Directory Identity; substantive ## Purpose when present). Generated snapshots under doc/ are excluded from pairing where noted. After moving files under doc/gnn/, run the rewriter (idempotent on already-fixed links):

uv run python doc/development/rewrite_gnn_doc_links.py

Source: docs_audit.py, rewrite_gnn_doc_links.py, ../../scripts/check_maintained_doc_terms.py. Documentation improvement tracker: ../DOCS_TO_IMPROVE.md.

Quick Start for Developers

Environment Setup

# Clone repository
git clone https://github.com/ActiveInferenceInstitute/GeneralizedNotationNotation.git
cd GeneralizedNotationNotation

# Run setup (Step 1: env + dependencies; use --dev for dev extras)
uv run python src/main.py --only-steps 1 --dev

# Or sync only: uv sync / uv sync --extra dev

# Activate virtual environment (if not using `uv run` for every command)
source .venv/bin/activate  # Linux/Mac (repo-root `.venv` after `uv sync`)
# .venv\Scripts\activate    # Windows

# Run tests to verify setup (Step 2)
uv run python src/main.py --only-steps 2 --verbose

Development Workflow

# Create feature branch
git checkout -b feature/my-new-feature

# Make changes, add tests
# ...

# Run quality checks
just quality
just test-pymdp-focused

# Commit and push
git add .
git commit -m "feat: add new feature"
git push origin feature/my-new-feature

Project Architecture

Module Organization

src/
├── main.py                    # Pipeline orchestrator
├── pipeline/                  # Pipeline configuration, execution adapters, DAG/context helpers
│   ├── config.py             # Step metadata and output directory mapping
│   ├── execution.py          # Programmatic API delegating to main.py/numbered steps
│   └── step_timeouts.py      # Per-step timeout policy
├── gnn/                      # Core GNN processing
│   ├── processor.py          # Directory processing helpers
│   ├── schema.py             # Core schema parsing/validation
│   └── parsers/              # Format-specific parsers
├── export/                   # Multi-format export
│   ├── processor.py          # Export orchestration
│   └── format_exporters.py   # Export implementations
├── visualization/            # Model visualization
│   ├── core/                 # Core visualization processing
│   ├── graph/                # Graph visualizers
│   └── matrix/               # Matrix visualizers
├── render/                   # Code generation
│   ├── processor.py          # Main rendering processor
│   ├── pymdp/                # PyMDP code generation
│   ├── rxinfer/              # RxInfer.jl generation
│   └── stan/                 # Stan rendering support
├── execute/                  # Simulator execution
│   ├── processor.py          # Execute orchestration
│   ├── pymdp/                # PyMDP execution
│   └── rxinfer/              # RxInfer.jl execution
├── llm/                     # LLM integration
│   ├── providers/           # LLM provider implementations
│   ├── processor.py         # Step 13 processing
│   └── mcp.py               # LLM MCP tools
├── mcp/                     # Model Context Protocol
│   ├── mcp.py               # Core MCP instance
│   ├── server_stdio.py      # STDIO server
│   └── processor.py         # Step 21 processing
├── ontology/                # Ontology processing
│   ├── processor.py         # Core processing
│   └── act_inf_ontology_terms.json # Ontology data
├── utils/                   # Shared utilities
│   ├── argument_utils.py    # CLI/pipeline argument model
│   └── pipeline_template.py # Thin orchestrator wrapper
└── tests/                   # 171 pytest files, mirrored by module

Design Patterns

Pipeline Step Pattern

Each pipeline step follows this pattern:

#!/usr/bin/env python3
"""
GNN Processing Pipeline - Step N: Description

Purpose description and usage information.
"""

import logging
import argparse
from pathlib import Path

logger = logging.getLogger(__name__)

def process_main_logic(args: argparse.Namespace) -> bool:
    """Main processing logic for this step."""
    # Implementation here
    return success

def main(args: argparse.Namespace) -> int:
    """Entry point called by main.py pipeline."""
    logger.info(f"Starting Step N: Description")
    
    if not process_main_logic(args):
        logger.error("Step N failed")
        return 1
        
    logger.info("Step N completed successfully")
    return 0

if __name__ == "__main__":
    # Standalone execution setup
    parser = argparse.ArgumentParser(description="Step N (Standalone)")
    # Add arguments
    args = parser.parse_args()
    
    # Setup logging for standalone
    from utils.logging_utils import setup_standalone_logging
    setup_standalone_logging(
        level=logging.DEBUG if args.verbose else logging.INFO,
        logger_name=__name__
    )
    
    sys.exit(main(args))

MCP Tool Pattern

Each module can provide MCP tools:

# In module_name/mcp.py
def tool_function(param1: str, param2: int = 10) -> dict:
    """
    Tool description for MCP schema generation.
    
    Args:
        param1: Parameter description
        param2: Optional parameter with default
        
    Returns:
        Result dictionary
    """
    # Implementation
    return {"result": result}

def register_tools():
    """Register all tools from this module."""
    from src.mcp import mcp_instance
    
    tools = {
        "module_tool_name": tool_function,
        # Add more tools
    }
    
    for name, func in tools.items():
        mcp_instance.register_tool(name, func)

Development Guidelines

Code Quality Standards

Type Hints

# Required for all function signatures
def process_gnn_file(file_path: Path, strict: bool = False) -> Dict[str, Any]:
    """Process a GNN file and return parsed data."""
    pass

# Use Union types where appropriate
from typing import Union, Optional, List, Dict

def validate_model(model: Union[str, Path, Dict]) -> Optional[List[str]]:
    """Validate a model, return errors if any."""
    pass

Documentation Standards

def complex_function(param1: str, param2: Optional[int] = None) -> Dict[str, Any]:
    """
    One-line summary of the function.
    
    More detailed description if needed. Explain the purpose,
    key behaviors, and any important considerations.
    
    Args:
        param1: Description of what this parameter does
        param2: Optional parameter, explain default behavior
        
    Returns:
        Dictionary containing:
            - key1: Description of what this contains
            - key2: Description of this field
            
    Raises:
        ValueError: When param1 is invalid
        FileNotFoundError: When required files are missing
        
    Example:
        >>> result = complex_function("test", 42)
        >>> print(result["key1"])
        expected_output
    """
    pass

Error Handling

# Use specific exceptions
def parse_gnn_file(file_path: Path) -> Dict[str, Any]:
    """Parse GNN file with proper error handling."""
    try:
        if not file_path.exists():
            raise FileNotFoundError(f"GNN file not found: {file_path}")
            
        with open(file_path, 'r', encoding='utf-8') as f:
            content = f.read()
            
        # Processing logic
        return parse_content(content)
        
    except json.JSONDecodeError as e:
        raise ValueError(f"Invalid JSON in GNN file: {e}")
    except Exception as e:
        logger.error(f"Unexpected error parsing {file_path}: {e}")
        raise

Testing Standards

Test Organization

tests/
├── unit/                    # Unit tests
│   ├── test_gnn_parser.py  # Test individual modules
│   ├── test_exporters.py   
│   └── test_visualization.py
├── integration/             # Integration tests  
│   ├── test_pipeline_steps.py # Test step interactions
│   ├── test_export_formats.py # Test full export workflows
│   └── test_mcp_integration.py # Test MCP functionality
├── fixtures/                # Test data
│   ├── valid_gnn_files/    # Valid test models
│   ├── invalid_gnn_files/  # Invalid models for error testing
│   └── expected_outputs/   # Expected results for validation
└── conftest.py             # Pytest configuration

Test Writing Patterns

import pytest
from pathlib import Path
from src.gnn.parser import parse_gnn_file

class TestGNNParser:
    """Test suite for GNN file parsing."""
    
    @pytest.fixture
    def sample_gnn_file(self, tmp_path):
        """Create a sample GNN file for testing."""
        content = """
        # ModelName
        Test Model
        
        # StateSpaceBlock
        s_f0[3,1,type=float]
        """
        file_path = tmp_path / "test_model.md"
        file_path.write_text(content)
        return file_path
    
    def test_parse_valid_file(self, sample_gnn_file):
        """Test parsing a valid GNN file."""
        result = parse_gnn_file(sample_gnn_file)
        
        assert result["model_name"] == "Test Model"
        assert "s_f0" in result["state_space"]
        assert result["state_space"]["s_f0"]["dimensions"] == [3, 1]
    
    def test_parse_missing_file(self):
        """Test error handling for missing files."""
        with pytest.raises(FileNotFoundError):
            parse_gnn_file(Path("nonexistent.md"))
            
    @pytest.mark.parametrize("invalid_content,expected_error", [
        ("# InvalidSection\nContent", "Unknown section"),
        ("# StateSpaceBlock\ninvalid[format", "Invalid dimension"),
    ])
    def test_parse_invalid_content(self, tmp_path, invalid_content, expected_error):
        """Test error handling for various invalid content."""
        file_path = tmp_path / "invalid.md"
        file_path.write_text(invalid_content)
        
        with pytest.raises(ValueError, match=expected_error):
            parse_gnn_file(file_path)

Running Tests

# Run all tests
python src/main.py --only-steps 3

# Run specific test categories
uv run --extra dev python -m pytest src/tests/unit/ -v
uv run --extra dev python -m pytest src/tests/integration/ -v

# Run with coverage
pytest --cov=src --cov-report=html:output/coverage

# Run performance tests
uv run --extra dev python -m pytest src/tests/performance/ --benchmark-only

Adding New Features

Adding a New Pipeline Step

Create the script: src/N_description.py
Add configuration: Update src/config.py
Add tests: Create tests in tests/unit/ and tests/integration/
Add documentation: Update pipeline documentation
Add MCP tools: Create module/mcp.py if exposing APIs

Adding New Export Formats

Implement exporter: Add to src/export/format_exporters.py
Register format: Update AVAILABLE_EXPORT_FUNCTIONS
Add tests: Test export functionality and output validation
Update documentation: Add format to documentation

Adding New MCP Tools

Implement tools: Create functions in appropriate module/mcp.py
Register tools: Call registration in module initialization
Add schemas: Ensure proper type hints for auto-schema generation
Test integration: Test via MCP CLI and HTTP server
Document APIs: Update MCP documentation

Performance Considerations

Memory Management

Use generators for large file processing
Implement streaming for export formats
Monitor memory usage in resource estimation

Optimization Guidelines

Profile performance-critical paths
Use caching appropriately
Consider JAX for numerical computations
Implement timeout handling for long operations

Monitoring

Log performance metrics
Track memory usage per pipeline step
Monitor export file sizes
Measure visualization generation times

Release Process

Versioning

Follow semantic versioning (MAJOR.MINOR.PATCH)
Tag releases corresponding to paper submissions
Maintain CHANGELOG.md with clear categories

Quality Gates

Formatting and lint pass (uv run --extra dev ruff format --check src scripts; uv run --extra dev ruff check src scripts)
Terminology and documentation audits pass (scripts/check_repo_terminology.py, scripts/check_maintained_doc_terms.py, doc/development/docs_audit.py, scripts/check_gnn_doc_patterns.py)
Type checking passes (uv run --extra dev mypy src --show-error-codes)
Security scan passes (uv run --extra dev bandit -r src -c pyproject.toml -q)
Focused PyMDP/POMDP tests, collect-only, and the full suite command of record pass

Release Checklist

Troubleshooting Development Issues

Common Problems

Import Errors
- Check virtual environment activation
- Verify src/ is in Python path
- Run setup step: python src/main.py --only-steps 2
Test Failures
- Check test data fixtures
- Verify dependencies installed
- Run individual test files for debugging
Pipeline Issues
- Check step configuration in src/config.py
- Verify argument passing between steps
- Check timeout settings for slow operations
MCP Integration Problems
- Verify tool registration in module mcp.py files
- Check MCP server startup logs
- Test tools via CLI before HTTP integration

Debug Tools

# Verbose pipeline execution
python src/main.py --verbose --only-steps 1,4

# MCP tool debugging
python src/mcp/cli.py --debug list-tools

# Test debugging
pytest -vvv --pdb src/tests/unit/test_specific.py

# Type checking
uv run --extra dev mypy src --show-error-codes

Contributing Guidelines

Code Review Process

Create feature branch from main
Implement changes with tests
Submit pull request with clear description
Address review feedback
Merge after approval and CI passes

Documentation Requirements

Update relevant documentation for all changes
Add docstrings for new functions/classes
Include examples for new features
Update troubleshooting guides for new error cases

Testing Requirements

Unit tests for all new functions
Integration tests for new pipeline steps
Performance tests for computationally intensive features
Regression tests for bug fixes

FilesExpand file tree

README.md

Latest commit

History