Skip to content

README.md

Latest commit

 

History

History
553 lines (427 loc) · 16.5 KB

README.md

File metadata and controls

553 lines (427 loc) · 16.5 KB
Raw

GNN (Generalized Notation Notation) Core Module

Core infrastructure for GNN: file discovery, multi-format parsing, serialization, round-trip validation, and schema / cross-format consistency checks.

Format Interoperability Status

Canonical numbers (enum size, serializer count, round-trip scope) are defined in SPEC.md.

  • GNNFormat: 23 values (parsers/common.py).
  • Serializers: 22 registered (parsers/system.py); PNML is parse-only in the registry sense (no dedicated serializer entry).
  • Round-trip tests: testing/test_round_trip.py exercises markdown plus 20 target formats (21 entries total). EBNF and PNML are outside the default round-trip list.

For the reference model used in tests, those 21 formats achieve 100% round-trip success in that suite.

Supported Format Categories (round-trip suite)

  • Schema: JSON, XML, YAML, Protobuf, XSD, ASN.1, PKL (7)
  • Languages: Python, Scala, Lean, Coq, Isabelle, Haskell (6)
  • Formal / grammar: TLA+, Agda, Alloy, Z-notation, BNF (5); EBNF shares machinery with BNF but is not separately listed in the default round-trip config
  • Other: Maxima, Pickle (2)
  • Reference: Markdown (1)

Overview

GNN enables researchers and practitioners to:

  • Specify generative models in a standardized, machine-readable format
  • Convert between all supported formats with semantic preservation
  • Validate model specifications with multiple validation levels
  • Parse and analyze model structures across different paradigms
  • Export models to simulation environments (PyMDP, RxInfer.jl)
  • Generate visualizations and documentation

GNN Processing Workflow

graph TD
    Input[Input .md File] -->|Parser| Parsed[Parsed Model]
    Parsed -->|Schema Validator| Valid{Valid?}
    Valid -->|No| Error[Error Report]
    Valid -->|Yes| IR[Intermediate Representation]
    
    IR -->|Cross-Format Validator| Consistent{Consistent?}
    Consistent -->|No| Warn[Consistency Warning]
    Consistent -->|Yes| Serializer[Multi-Format Serializer]
    
    Serializer -->|Export| JSON[JSON]
    Serializer -->|Export| XML[XML]
    Serializer -->|Export| Python[Python]
    Serializer -->|Export| Others[Other 20+ Formats]
    
    subgraph "Validation Core"
    Parsed
    Valid
    IR
    Consistent
    end
Loading

Format Conversion Architecture

graph LR
    subgraph "Input Formats"
        MD[Markdown]
        JSON_IN[JSON]
        XML_IN[XML]
        YAML_IN[YAML]
    end
    
    subgraph "Unified Parser"
        Parser[GNNParsingSystem]
        IR[Internal Representation]
    end
    
    subgraph "Format Serializers"
        Serializer[Format Serializers]
    end
    
    subgraph "Output Formats"
        SCALA[Scala]
        LEAN[Lean]
        COQ[Coq]
        PYTHON[Python]
        HASKELL[Haskell]
        PROTOBUF[Protobuf]
        XSD[XSD]
        ASN1[ASN.1]
        OTHERS[15+ More Formats]
    end
    
    MD --> Parser
    JSON_IN --> Parser
    XML_IN --> Parser
    YAML_IN --> Parser
    
    Parser --> IR
    IR --> Serializer
    
    Serializer --> SCALA
    Serializer --> LEAN
    Serializer --> COQ
    Serializer --> PYTHON
    Serializer --> HASKELL
    Serializer --> PROTOBUF
    Serializer --> XSD
    Serializer --> ASN1
    Serializer --> OTHERS
Loading

Parser Architecture

graph TB
    subgraph "Parser Registry"
        Registry[GNNParsingSystem]
    end
    
    subgraph "Format Parsers"
        MarkdownP[Markdown Parser]
        JSONP[JSON Parser]
        XMLP[XML Parser]
        YAMLP[YAML Parser]
        SchemaP[Schema Parsers]
        GrammarP[Grammar Parsers]
        BinaryP[Binary Parser]
    end
    
    subgraph "Format Serializers"
        MarkdownS[Markdown Serializer]
        JSONS[JSON Serializer]
        XMLS[XML Serializer]
        YAMLS[YAML Serializer]
        ScalaS[Scala Serializer]
        LeanS[Lean Serializer]
        CoqS[Coq Serializer]
        PythonS[Python Serializer]
        OthersS[15+ More Serializers]
    end
    
    subgraph "Validation"
        SchemaV[Schema Validator]
        CrossV[Cross-Format Validator]
        RoundTrip[Round-Trip Tester]
    end
    
    Registry --> MarkdownP
    Registry --> JSONP
    Registry --> XMLP
    Registry --> YAMLP
    Registry --> SchemaP
    Registry --> GrammarP
    Registry --> BinaryP
    
    MarkdownP --> SchemaV
    JSONP --> SchemaV
    XMLP --> SchemaV
    
    SchemaV --> CrossV
    CrossV --> RoundTrip
    
    Registry --> MarkdownS
    Registry --> JSONS
    Registry --> XMLS
    Registry --> YAMLS
    Registry --> ScalaS
    Registry --> LeanS
    Registry --> CoqS
    Registry --> PythonS
    Registry --> OthersS
Loading

Module Integration Flow

flowchart LR
    subgraph "Pipeline Step 3"
        Step3[3_gnn.py Orchestrator]
    end
    
    subgraph "GNN Module"
        MultiFormat[multi_format_processor.py]
        Processor[processor.py]
        Parser[parser.py]
        ParsingSystem[parsers/GNNParsingSystem]
    end
    
    subgraph "Downstream Steps"
        Step5[Step 5: Type Checker]
        Step6[Step 6: Validation]
        Step7[Step 7: Export]
        Step8[Step 8: Visualization]
        Step10[Step 10: Ontology]
        Step11[Step 11: Render]
    end
    
    Step3 --> MultiFormat
    MultiFormat --> Processor
    MultiFormat --> ParsingSystem
    Processor --> Parser
    
    MultiFormat -->|Parsed Models| Step5
    MultiFormat -->|Parsed Models| Step6
    MultiFormat -->|Parsed Models| Step7
    MultiFormat -->|Parsed Models| Step8
    MultiFormat -->|Parsed Models| Step10
    MultiFormat -->|Parsed Models| Step11
Loading

Module Structure

src/gnn/
├── __init__.py                    # Module initialization with format ecosystem
├── README.md                      # This documentation
├── SPEC.md                        # Canonical format counts and architecture
├── mcp.py                         # Model Context Protocol integration
├── schema_validator.py            # Enhanced validator with multiple validation levels
├── cross_format_validator.py      # Cross-format consistency validation
├── processors.py                  # Enhanced processing with comprehensive testing
├── alignment_status.md            # Format compatibility status tracking
│
├── parsers/                       # Parser ecosystem (see SPEC.md for 23/22 counts)
│   ├── __init__.py               # Package exports
│   ├── system.py                 # PARSER_REGISTRY / SERIALIZER_REGISTRY, GNNParsingSystem
│   ├── unified_parser.py         # Unified parsing entry
│   ├── *_parser.py / *_serializer.py  # Per-format parsers and serializers
│   ├── grammar_parser.py         # BNF/EBNF parsers
│   ├── schema_parser.py          # Schema parsers (XSD, ASN.1, PKL, etc.)
│   ├── xml_parser.py             # XML / PNML
│   ├── binary_parser.py          # Pickle and binary paths
│   └── common.py                 # GNNFormat enum, protocols, shared types
│
├── testing/                       # Testing infrastructure
│   ├── test_round_trip.py        # Round-trip suite (see SPEC.md)
│   ├── README_round_trip.md      # Testing methodology and results
│   └── round_trip_reports/       # Test reports and analysis
│
├── schemas/                       # Schema definitions
│   ├── json.json                 # JSON Schema with Unicode support
│   ├── yaml.yaml                 # YAML Schema with validation guidance
│   ├── xsd.xsd                   # XML Schema
│   ├── asn1.asn1                 # ASN.1 schema
│   ├── pkl.pkl                   # PKL schema
│   └── [additional schemas...]   # Additional schema files
│
├── gnn_examples/                  # Reference Markdown models (e.g. actinf_pomdp_agent.md)
└── (repo root) input/gnn_files/   # Pipeline input examples; tests often use input/gnn_files/actinf_pomdp_agent.md

Validation System

Validation Levels

  1. BASIC - File structure and syntax validation
  2. STANDARD - Semantic validation + Active Inference compliance
  3. STRICT - Cross-format consistency + research standards
  4. RESEARCH - Complete documentation + provenance tracking
  5. ROUND_TRIP - Format conversion validation with data preservation
from gnn.schema_validator import GNNValidator, ValidationLevel

validator = GNNValidator(
    validation_level=ValidationLevel.STANDARD,
    enable_round_trip_testing=False
)

result = validator.validate_file('model.md')
print(f"Valid: {result.is_valid}")
print(f"Errors: {len(result.errors)}")

Round-Trip Testing

Suite status

For the reference model and the formats listed in testing/test_round_trip.py, the suite reports 100% round-trip success. See SPEC.md for how that relates to all 23 enum formats (PNML/EBNF not in the default list).

Embedded Data Architecture

The system uses embedded data in format-specific comments to preserve model semantics during format conversion:

# JSON format with embedded model data
{
    "model_name": "Example",
    "variables": [...],
    /* MODEL_DATA: {"complete":"model","data":"embedded"} */
}

# XML format with embedded preservation
<model>
    <variables>...</variables>
    <!-- MODEL_DATA: {"complete":"model","data":"embedded"} -->
</model>

Usage

from gnn.testing.test_round_trip import GNNRoundTripTester

tester = GNNRoundTripTester()
report = tester.run_comprehensive_tests()

print(f"Success rate: {report.get_success_rate():.1f}%")
print(f"Tests passed: {report.successful_tests}/{report.total_tests}")

Processing Capabilities

Multi-Level Processing

from gnn.processors import process_gnn_folder

success = process_gnn_folder(
    target_dir=Path("models/"),
    output_dir=Path("results/"),
    logger=logger,
    validation_level="standard",
    enable_round_trip=False,
    recursive=True
)

Cross-Format Consistency

from gnn.cross_format_validator import CrossFormatValidator

validator = CrossFormatValidator()
result = validator.validate_cross_format_consistency(gnn_content)

print(f"Consistency rate: {result.get_consistency_rate():.1f}%")
print(f"Formats tested: {len(result.schema_formats)}")

Enhanced Components

Schema Validator

Features:

  • Multiple validation levels
  • Multi-format parsing with automatic detection
  • Binary format support
  • Performance metrics and reporting
  • Cross-format semantic validation
from gnn.schema_validator import GNNValidator, ValidationLevel

validator = GNNValidator(validation_level=ValidationLevel.STRICT)
result = validator.validate_file('model.md')

# Result analysis
print(f"Validation level: {result.validation_level.value}")
print(f"Format detected: {result.format_tested}")
print(f"Performance: {result.performance_metrics}")

Testing Infrastructure

The testing/test_round_trip.py module provides comprehensive testing:

from gnn.testing.test_round_trip import GNNRoundTripTester

tester = GNNRoundTripTester()
report = tester.run_comprehensive_tests()

# Results analysis
for result in report.round_trip_results:
    status = "PASS" if result.success else "FAIL"
    print(f"{result.target_format.value}: {status}")

GNN Syntax

Variable Definitions

# StateSpaceBlock
A[3,3,type=float]                    # 3x3 transition matrix
s_f0[2,1,type=float]                 # Hidden state factor 0
o_m0[3,1,type=int]                   # Observation modality 0
learning_rate[1,type=float]          # Scalar learning rate
π_policy[4,2,type=categorical]       # Policy matrix (Unicode supported)

Connections

# Connections
A>B                                  # Directed influence
(A,B)-C                             # Multi-source undirected
X|Y                                  # Conditional dependency
(s_f0,s_f1)>(A_m0,A_m1,A_m2)       # Multi-variable connection

Parameters

# InitialParameterization
A=[[1.0, 0.0], [0.0, 1.0]]         # Matrix initialization
learning_rate=0.01                   # Scalar value
enabled=true                         # Boolean value
metadata={"version": "1.0"}          # Complex object

Active Inference Conventions

Standard Active Inference naming patterns:

  • A matrices: A_m0, A_m1 (Likelihood/observation matrices)
  • B matrices: B_f0, B_f1 (Transition dynamics)
  • C vectors: C_m0, C_m1 (Preferences/goals)
  • D vectors: D_f0, D_f1 (Priors over initial states)
  • Hidden states: s_f0, s_f1 (State factors)
  • Observations: o_m0, o_m1 (Observation modalities)
  • Actions: u_c0, u_c1 (Control factors)
  • Policies: π_c0, π_c1 (Policy variables)

Error Handling

Common Error Patterns

Format Detection

Error: Could not detect format for file.unknown
Solution: Use format_hint parameter: validator.validate_file('model.txt', format_hint='markdown')

Round-Trip Issues

Error: Round-trip test failed for XML format: 3 semantic differences
Solution: Check embedded data preservation in XML serializer

Cross-Format Inconsistencies

Error: Semantic checksums differ across formats
Solution: Review format-specific serialization patterns

Pipeline Integration

Integration with the broader GNN pipeline:

  1. Discovery: Multi-format file discovery with automatic detection
  2. Type Checking: Multi-level validation with optional round-trip testing
  3. Export: Semantic preservation across all formats
  4. Visualization: Format-aware graph generation
  5. Rendering: Code generation for multiple backends

Pipeline Usage

from gnn.processors import run_comprehensive_gnn_testing

success = run_comprehensive_gnn_testing(
    target_dir=Path("models/"),
    output_dir=Path("results/"),
    logger=logger,
    validation_level="standard",
    enable_round_trip=False
)

Performance

Benchmarks

  • File Processing: 50+ files/second with full validation
  • Round-Trip Testing: 20 formats in ~0.07 seconds per model
  • Cross-Format Validation: Sub-second consistency checks
  • Memory Usage: <100MB for complex multi-format models

Performance Monitoring

result = validator.validate_file('model.md')
metrics = result.performance_metrics

print(f"Validation time: {metrics.get('validation_time', 0):.3f}s")
print(f"Content length: {metrics.get('content_length', 0)} chars")

Development

Adding Validation Rules

class CustomGNNValidator(GNNValidator):
    def _validate_custom_requirements(self, parsed_gnn, result):
        if not parsed_gnn.ontology_mappings:
            result.suggestions.append("Consider adding ontology mappings")

validator = CustomGNNValidator(validation_level=ValidationLevel.STRICT)

Supporting New Formats

from gnn.parsers.common import BaseGNNParser

class NewFormatParser(BaseGNNParser):
    def get_supported_extensions(self) -> List[str]:
        return ['.newext']
    
    def parse_file(self, file_path: str) -> ParseResult:
        return self._parse_with_embedded_data(file_path)

# Register with system
parsing_system.register_parser(GNNFormat.NEW_FORMAT, NewFormatParser)

Documentation

Available Resources

  • testing/README_round_trip.md: Testing methodology and results
  • alignment_status.md: Format compatibility status
  • Format-specific guides: Documentation for each supported format
  • Performance guides: Optimization best practices
  • SPEC.md: Canonical format counts and architecture

Summary

The GNN module provides infrastructure for Active Inference model specification: 23 GNNFormat values, 22 serializers, and a round-trip test suite (see SPEC.md) that achieves 100% pass rate for its configured formats. Key features include multi-level validation, cross-format consistency checking, and round-trip testing with embedded model data where formats allow it.

License and Citation

This implementation follows the GNN specification v1.0+ and is part of the GeneralizedNotationNotation project. See the main repository for license and citation information.

References

  • Project overview: ../../README.md
  • Comprehensive docs: ../../DOCS.md
  • Architecture guide: ../../ARCHITECTURE.md
  • Pipeline details: ../../doc/pipeline/README.md

Documentation

  • README: Module Overview
  • AGENTS: Agentic Workflows
  • SPEC: Architectural Specification
  • SKILL: Capability API