Skip to content

diagnostics.md

Latest commit

 

History

History
203 lines (144 loc) · 7.03 KB

diagnostics.md

File metadata and controls

203 lines (144 loc) · 7.03 KB

MCP Tree-sitter Server Diagnostics

This document describes the diagnostic testing approach for the MCP Tree-sitter Server project.

Overview

The diagnostics suite consists of targeted pytest tests that isolate and document specific issues in the codebase. These tests are designed to:

  1. Document current behavior with proper pass/fail results
  2. Isolate failure points to specific functions or modules
  3. Provide detailed error information and stack traces
  4. Create a foundation for developing targeted fixes

The diagnostic framework combines standard pytest behavior with enhanced diagnostic capabilities:

  • Tests properly pass or fail based on assertions
  • Comprehensive diagnostic data is captured for debugging
  • Diagnostic information is saved to JSON for further analysis

Running Diagnostics

The Makefile includes several targets for running diagnostics:

# Run all diagnostic tests
make test-diagnostics

# CI-friendly version (won't fail the build on diagnostic issues)
make test-diagnostics-ci

For running diagnostics alongside regular tests:

# Run both regular tests and diagnostics
make test-all

Using the Diagnostic Framework

Basic Test Structure

import pytest
from mcp_server_tree_sitter.testing import diagnostic

@pytest.mark.diagnostic  # Mark the test as producing diagnostic data
def test_some_feature(diagnostic):  # Use the diagnostic fixture
    # Add details to diagnostic data
    diagnostic.add_detail("key", "value")
    
    try:
        # Test your functionality
        result = some_functionality()
        
        # Use standard assertions - the test will fail if they don't pass
        assert result is not None, "Result should not be None"
        
    except Exception as e:
        # Record the error in diagnostic data
        diagnostic.add_error("ErrorType", str(e))
        
        # Add any artifacts you want to save
        diagnostic.add_artifact("error_artifact", {"error": str(e)})
        
        # Re-raise to fail the test
        raise

Diagnostic Operations

The diagnostic fixture provides several methods:

  • add_detail(key, value): Add a key-value pair to diagnostic details
  • add_error(error_type, message, traceback=None): Add an error
  • add_artifact(name, content): Add an artifact (e.g., JSON data)
  • finalize(status="completed"): Mark the diagnostic as complete

Key Issues Identified and Fixed

The following issues were identified during the diagnostic process and have since been fixed in the current implementation:

1. Language Registry Issues (FIXED)

  • list_languages() previously returned empty lists despite languages being available
  • Language detection through install_language() worked, but languages didn't appear in available lists

2. AST Parsing Failures (FIXED)

  • get_ast() previously failed with errors when attempting to build the tree
  • Core AST parsing functionality is now operational with efficient cursor-based traversal

3. "Too Many Values to Unpack" Errors (FIXED)

  • Several analysis functions failed with "too many values to unpack (expected 2)"
  • Affected get_symbols(), get_dependencies(), and analyze_complexity()
  • These issues were resolved by fixing query captures handling

4. Tree-sitter Language Pack Integration (FIXED)

  • Integration with tree-sitter-language-pack is now complete and stable
  • All supported languages are correctly recognized and available for analysis

Diagnostic Results

The diagnostic tests generate detailed JSON result files in the diagnostic_results directory with timestamps. These files contain valuable information for debugging:

  • Error messages and stack traces
  • Current behavior documentation
  • Environment and configuration details
  • Detailed information about tree-sitter integration

In addition, the test output includes a diagnostic summary:

============================== Diagnostic Summary ==============================
Collected 4 diagnostics, 2 with errors
-------------------------------- Error Details ---------------------------------
- /path/to/test.py::test_function
  Error 1: ErrorType: Error message

Recommended Debugging Approach

  1. Run the diagnostic tests to verify current issues

    make test-diagnostics

  2. Examine the diagnostic results in the terminal output and the diagnostic_results directory

  3. Review specific error patterns to identify the root cause:

    • For unpacking errors, check the query capture processing code
    • For AST parsing, examine the tree-sitter integration layer
    • For language registry issues, check the initialization sequence

  4. Make targeted fixes to address specific issues, using the diagnostic tests to verify repairs

  5. After fixes, run both diagnostics and regular tests to ensure no regressions

    make test-all

Previous Issue Priority (Now Resolved)

The following priority was used to address the previously identified issues, which have all been resolved:

  1. ✅ Language Registry Issues - Fixed language listing to enable proper language detection
  2. ✅ AST Parsing - Fixed core parsing functionality with efficient cursor-based traversal
  3. ✅ Query Handling - Resolved unpacking errors in query captures to enable analysis tools
  4. ✅ Incremental Improvements - Core functionality is working correctly and ready for further refinement

All 90 tests are now passing, including the diagnostic tests.

Integrating with Development Workflow

Diagnostics should be run:

  • After any significant changes to core tree-sitter integration code
  • Before submitting pull requests that touch language or AST handling
  • When investigating specific failures in higher-level functionality
  • As part of debugging for issues reported by users

Continuous Integration

For CI environments, the diagnostic tests have special considerations:

CI-Friendly Targets

The Makefile includes CI-friendly targets that won't fail the build due to known issues:

  • make test-diagnostics-ci: Runs diagnostics but always returns success

CI Setup Recommendations

  1. Primary CI Pipeline: Use make test for regression testing of working functionality

    test:
  script:
    - make test

  2. Diagnostic Job: Add a separate, optional job for diagnostics

    diagnostics:
  script:
    - make test-diagnostics-ci
  artifacts:
    paths:
      - diagnostic_results/
  allow_failure: true

Benefits of the Pytest-based Approach

The pytest-based diagnostic framework offers significant advantages:

  1. Unified framework: All tests use pytest with consistent behavior
  2. Clear pass/fail: Tests fail when they should, making issues obvious
  3. Rich diagnostics: Detailed diagnostic information is still collected
  4. Standard integration: Works with pytest's fixtures, plugins, and reporting

Future Improvements

In the future, we plan to:

  1. Enhance the diagnostic plugin with more features
  2. Integrate with CI/CD pipelines for better reporting
  3. Add automatic visualization of diagnostic data
  4. Improve the organization of diagnostic tests