Skip to content

e2e-testing.md

Latest commit

 

History

History
345 lines (251 loc) · 8.05 KB

e2e-testing.md

File metadata and controls

345 lines (251 loc) · 8.05 KB
Raw

E2E Testing Guide

This guide covers end-to-end testing for the QPanda3 Runtime MCP Server.

Overview

E2E tests verify the complete functionality of the MCP server, including:

  • MCP protocol compliance (tool registration, invocation, response format)
  • Complete quantum computing workflows (account setup → task execution → results)
  • CircuitObservableBinding for multi-objective decisions
  • Batch operations (batch sampling, batch estimation)
  • Error handling and edge cases
  • MCP resources (circuit examples)

Test Structure

tests/
├── e2e/                           # End-to-end tests
│   ├── __init__.py
│   ├── conftest.py               # E2E fixtures
│   ├── test_mcp_protocol.py      # MCP protocol tests
│   ├── test_quantum_workflow.py  # Quantum computing workflows
│   ├── test_batch_operations.py  # Batch operation tests
│   ├── test_circuit_observable_binding.py  # Multi-objective tests
│   ├── test_mcp_resources.py     # MCP resource tests
│   └── test_error_handling.py    # Error handling tests
├── conftest.py                   # Unit test fixtures
├── test_server.py                # Server unit tests
└── test_runtime.py               # Runtime unit tests

Running E2E Tests

All E2E Tests

# Using uv
uv run pytest tests/e2e/ -v

# With markers
uv run pytest -m e2e -v

Specific Test Files

# MCP protocol tests
uv run pytest tests/e2e/test_mcp_protocol.py -v

# Quantum workflow tests
uv run pytest tests/e2e/test_quantum_workflow.py -v

# Error handling tests
uv run pytest tests/e2e/test_error_handling.py -v

With Coverage

uv run pytest tests/e2e/ --cov=src --cov-report=html

Test Categories

1. MCP Protocol Tests (test_mcp_protocol.py)

Tests for MCP protocol compliance:

  • Server initialization and configuration
  • Tool registration and metadata
  • Tool invocation and response format
  • Resource accessibility
  • Tool schema validation
uv run pytest tests/e2e/test_mcp_protocol.py -v

2. Quantum Workflow Tests (test_quantum_workflow.py)

Tests for complete quantum computing user journeys:

  • Account management (setup, listing, info)
  • Device management (list, properties)
  • Sampling workflow (submit → status → results)
  • Estimation workflow (submit → status → results)
  • Task management (status, cancel, list)
  • Full quantum computing journey
uv run pytest tests/e2e/test_quantum_workflow.py -v

3. Batch Operations Tests (test_batch_operations.py)

Tests for batch execution:

  • Batch sampling with multiple circuits
  • Batch estimation with single observable
  • VQE-style parameterized circuits
  • Batch vs individual comparison
uv run pytest tests/e2e/test_batch_operations.py -v

4. CircuitObservableBinding Tests (test_circuit_observable_binding.py)

Tests for multi-objective decision making:

  • Binding creation
  • Product rule (Cartesian combinations)
  • Zip rule (one-to-one pairs)
  • Estimation with binding
  • Binding management (list, delete)
uv run pytest tests/e2e/test_circuit_observable_binding.py -v

5. MCP Resources Tests (test_mcp_resources.py)

Tests for MCP resource endpoints:

  • Bell state circuit resource
  • GHZ state circuit resource
  • Random circuit resource
  • Superposition circuit resource
  • Service status resource
  • Resource usability with tools
uv run pytest tests/e2e/test_mcp_resources.py -v

6. Error Handling Tests (test_error_handling.py)

Tests for error scenarios:

  • Account errors (missing credentials, invalid URL)
  • Device errors (non-existent device, no service)
  • Task errors (non-existent task, pending results)
  • Sampling errors (invalid circuit, invalid device, invalid shots)
  • Estimation errors (invalid observable)
  • Binding errors (empty lists, invalid indices)
  • Service connection errors
uv run pytest tests/e2e/test_error_handling.py -v

Test Fixtures

E2E Fixtures (tests/e2e/conftest.py)

Fixture Description
e2e_env_vars Environment variables for E2E testing
mock_runtime_service_e2e Comprehensive mock RuntimeService
sample_bell_state_circuit Bell state circuit string
sample_ghz_circuit GHZ state circuit string
sample_superposition_circuit Superposition circuit string
sample_observable_dict Observable as dictionary
sample_observable_str Observable as string
sample_circuits_batch List of circuits for batch testing

Writing New E2E Tests

Basic Test Pattern

import pytest
from unittest.mock import MagicMock

class TestMyFeature:
    """Tests for my feature."""

    @pytest.mark.asyncio
    async def test_feature_success(
        self,
        mock_runtime_service_e2e: MagicMock,
        e2e_env_vars: dict[str, str],
    ) -> None:
        """Test successful feature execution."""
        from qpanda3_runtime_mcp_server import runtime

        runtime._service = mock_runtime_service_e2e

        # Execute feature
        result = await runtime.my_feature()

        # Verify result
        assert result["status"] == "success"

Error Handling Test Pattern

@pytest.mark.asyncio
async def test_feature_error(self) -> None:
    """Test feature error handling."""
    from qpanda3_runtime_mcp_server import runtime

    runtime._service = None  # Simulate no service

    result = await runtime.my_feature()

    assert result["status"] == "error"

Workflow Test Pattern

@pytest.mark.asyncio
async def test_complete_workflow(
    self,
    mock_runtime_service_e2e: MagicMock,
    e2e_env_vars: dict[str, str],
) -> None:
    """Test complete workflow."""
    from qpanda3_runtime_mcp_server import runtime

    runtime._service = mock_runtime_service_e2e

    # Step 1: Setup
    setup_result = await runtime.setup()
    assert setup_result["status"] == "success"

    # Step 2: Execute
    exec_result = await runtime.execute()
    assert exec_result["status"] == "success"

    # Step 3: Verify
    verify_result = await runtime.verify()
    assert verify_result["status"] == "success"

CI/CD Integration

GitHub Actions

name: E2E Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  e2e-tests:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Install uv
        uses: astral-sh/setup-uv@v4

      - name: Install dependencies
        run: uv sync --extra test

      - name: Run E2E tests
        run: uv run pytest tests/e2e/ -v --tb=short

      - name: Upload coverage
        uses: codecov/codecov-action@v4
        with:
          files: ./coverage.xml

Test Coverage Goals

Category Target Coverage
MCP Protocol 90%+
Quantum Workflow 85%+
Error Handling 95%+
Resources 90%+
Overall 85%+

Best Practices

DO

  • ✅ Use fixtures for common setup
  • ✅ Test both success and error paths
  • ✅ Verify response structure (status, message, data)
  • ✅ Test complete workflows end-to-end
  • ✅ Use descriptive test names
  • ✅ Group related tests in classes

DON'T

  • ❌ Skip error handling tests
  • ❌ Use hardcoded credentials
  • ❌ Test implementation details
  • ❌ Ignore flaky tests
  • ❌ Mix unit tests with E2E tests

Debugging Failed Tests

Verbose Output

uv run pytest tests/e2e/test_quantum_workflow.py -v --tb=long

Debug Mode

uv run pytest tests/e2e/ --pdb

Capture Logs

uv run pytest tests/e2e/ --capture=no -v

Integration with Real Credentials

For testing with real QPanda3 Runtime credentials:

# Set environment variables
export QPANDA3_API_KEY="your_real_api_key"

# Run integration tests
uv run pytest -m integration tests/e2e/

Warning: Integration tests with real credentials will:

  • Consume quantum computing quota
  • Create real tasks on QPU devices
  • Take longer to complete

Related Documentation