Skip to content

Latest commit

 

History

History
754 lines (538 loc) · 28 KB

File metadata and controls

754 lines (538 loc) · 28 KB

AI-Powered Linux Testing at Scale: How the LISA MCP Server Changes Everything for Distro Providers

A practical deep-dive for Red Hat, Canonical, SUSE, and every team that ships and validates Linux on cloud or bare metal.


Table of Contents

  1. The Problem with Linux Testing Today
  2. What is the Model Context Protocol (MCP)?
  3. Why MCP Servers Matter for Infrastructure Teams
  4. Where to Register an MCP Server (VS Code, Claude, and more)
  5. Introducing the LISA MCP Server
  6. The 18 Tools — What Each One Does
  7. AI-Powered Failure Analysis with Any LLM
  8. Real Workflows for RHEL, Ubuntu, SLES, and More
  9. CI/CD Integration — Gate Your Releases with AI
  10. Getting Started in 5 Minutes
  11. Frequently Asked Questions

1. The Problem with Linux Testing Today

If you work at a company that ships or validates Linux — Red Hat, Canonical, SUSE, Oracle, Amazon, or any cloud provider — you know this pain intimately:

The test matrix never stops growing.

Every new kernel version, every hypervisor update, every hardware generation adds more combinations to validate. The typical workflow looks like this:

  1. An engineer manually searches through hundreds of Python test files to find the right test cases.
  2. They hand-write a YAML runbook from documentation, hoping they remember all the required fields.
  3. Tests run overnight. JUnit XML lands in a results folder.
  4. Someone parses the XML by hand — or with a brittle grep script — to identify failures.
  5. A senior engineer spends hours triaging root causes before a bug is even filed.
  6. The same investigation happens again next week with a slightly different failure.

This is slow, error-prone, and doesn't scale. The institutional knowledge about why specific failures happen lives in people's heads, not in systems.

The LISA MCP Server solves all of this by connecting an AI assistant directly to the Microsoft LISA testing framework — so your team can run, analyze, and report on Linux tests through natural language, in seconds.


2. What is the Model Context Protocol (MCP)?

The Model Context Protocol (MCP) is an open standard developed by Anthropic that defines how AI assistants (like GitHub Copilot, Claude, or any LLM-powered tool) connect to external systems.

Think of it like this:

MCP is to AI assistants what REST APIs are to web applications.

Instead of an AI assistant only knowing what's in its training data, MCP lets it call tools — real code that runs on your machine or server — to fetch live information, execute commands, and return structured results.

How it works

User in VS Code
    │
    ▼
GitHub Copilot / Claude / AI Client
    │  speaks Model Context Protocol
    ▼
MCP Server (running locally or remotely)
    │  executes tools
    ▼
Your systems: LISA repo, test results, file system, external APIs

An MCP server is just a process that:

  • Declares a list of tools (with names, descriptions, and typed parameters)
  • Receives JSON-RPC calls from an MCP client (the AI)
  • Returns structured results the AI can reason over

The AI doesn't need special training. It reads your tool descriptions and figures out when and how to use them — just like it reads documentation.


3. Why MCP Servers Matter for Infrastructure Teams

For a Linux distro provider or cloud validation team, an MCP server is a force multiplier:

Traditional Approach With an MCP Server
Engineer grepping Python source to find test names AI instantly searches and filters thousands of tests
Manually writing YAML runbooks AI generates complete, valid runbooks from a one-line request
Parsing JUnit XML with scripts AI reads results, categorizes failures, explains root causes
Tribal knowledge for triage AI applies consistent root cause classification across every failure
Reports written manually AI generates executive summaries and HTML reports in seconds
Different engineers, different formats Every analysis follows the same structured schema

MCP servers don't replace engineers — they eliminate the low-value repetitive work so engineers can focus on the hard problems that actually require expertise.

For organizations validating Linux on Azure, AWS, GCP, HyperV, or bare-metal SBOM images — this is the difference between a 4-hour triage cycle and a 4-minute one.


4. Where to Register an MCP Server

MCP servers can be registered in multiple AI clients. Here are the most common options:

Option A — VS Code with GitHub Copilot (Recommended)

This is the easiest path. VS Code 1.99+ supports MCP natively.

Step 1: Create .vscode/mcp.json in your project root:

{
  "servers": {
    "lisa": {
      "command": "python3",
      "args": ["-m", "lisa_mcp.server"],
      "cwd": "/absolute/path/to/lisa-mcp-server"
    }
  }
}

Step 2: Open the workspace in VS Code. The MCP server starts automatically.

Step 3: Open Copilot Chat (Ctrl+Alt+I), switch to Agent mode, and the 18 LISA tools are immediately available.

The LISA MCP Server repository ships with a pre-configured .vscode/mcp.json. Just update the cwd path and you're done.


Option B — Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "lisa": {
      "command": "python3",
      "args": ["-m", "lisa_mcp.server"],
      "cwd": "/path/to/lisa-mcp-server"
    }
  }
}

Restart Claude Desktop. The tools appear in the tool selector automatically.


Option C — Any OpenAI-Compatible MCP Client

The LISA MCP Server speaks the stdio MCP transport. Any client that supports stdio-based MCP servers can use it. Point the client at:

command: python3 -m lisa_mcp.server

Option D — Remote / Docker Deployment

For teams that want a shared MCP server accessed by multiple engineers:

FROM python:3.12-slim
WORKDIR /app
COPY . .
RUN pip install -e .
EXPOSE 8080
CMD ["python3", "-m", "lisa_mcp.server", "--transport", "sse", "--port", "8080"]

Register via HTTP SSE transport in your client:

{
  "servers": {
    "lisa": {
      "url": "http://your-server:8080/sse"
    }
  }
}

5. Introducing the LISA MCP Server

GitHub: github.com/kkkashan/LISA_MCP_Server

The LISA MCP Server connects any MCP-compatible AI directly to Microsoft LISA — the Linux Integration Services Automation framework used to validate Linux on Azure, HyperV, and bare metal at scale.

What is LISA?

LISA (Linux Integration Services Automation) is Microsoft's quality validation system for Linux. It provides:

  • A declarative Python test framework with hundreds of built-in test suites
  • Coverage across networking, storage, CPU, memory, kernel, hypervisor integration, security, performance
  • Multi-platform support: Azure, HyperV, QEMU, bare metal
  • Runbook-driven execution via YAML configuration files
  • Tier-based test selection (T0–T4) for different validation stages:
    • T0 — Smoke tests (< 5 min, runs on every build)
    • T1 — Core functional tests
    • T2 — Extended coverage
    • T3 — Edge cases and stress tests
    • T4 — Full regression

LISA is open source at github.com/microsoft/lisa and is actively used by Microsoft, and increasingly adopted by Linux distro providers to validate their images on Azure Marketplace.

What the MCP server adds

The MCP server wraps all of LISA's complexity in 18 AI-callable tools organized into 5 categories:

Discovery → Generation → Validation → Execution → AI Analysis

You ask questions in natural language. The AI calls the right tools in the right order. You get structured results.


6. The 18 Tools

Discovery Tools — Find the Right Tests

Tool What it does
discover_test_cases Scan a LISA repo and filter tests by area, tier, priority, or platform. Returns structured metadata for every matching test.
list_test_areas List all functional areas available in a LISA installation (networking, storage, cpu, memory, kdump, nvme, gpu, etc.)
get_test_case_details Get complete metadata for a single test — description, requirements, parameters, known issues
search_tests Free-text search across test names and descriptions — great for finding all tests that touch a specific feature

Example prompt:

Show me all network-related tests in ~/lisa that run at priority 0 (smoke level)

The AI calls discover_test_cases(lisa_path="~/lisa", area="network", tier="T0") and presents a filtered, formatted list.


Generation Tools — Build Runbooks and Test Code

Tool What it does
generate_test_suite_code Generate complete Python source code for a new LISA test suite, including class structure, decorators, setup/teardown, and parameterization
build_runbook Generate a ready-to-use YAML runbook with the correct schema, test criteria, and platform configuration
build_tier_runbook_file Quick tier-based runbook — just specify tier, platform, and image

Example prompt:

Build a T1 Azure runbook for validating RHEL 9.4 storage performance.
Include tests for disk I/O, NVMe, and SCSI.

The AI calls build_runbook(...) and returns a complete, schema-validated YAML file you can run immediately.


Validation Tools — Check Your Configuration

Tool What it does
validate_runbook_file Parse and validate a YAML runbook — catches schema errors, missing fields, and invalid test references before you run
add_test_to_existing_runbook Add include or exclude criteria to a runbook you already have
check_lisa_environment Verify your LISA CLI installation, Python version, and dependencies
get_tier_info Get the definition of each tier (T0–T4), priority ranges, and recommended use cases

Execution Tools — Run Tests and Parse Results

Tool What it does
run_lisa_tests Execute LISA tests via the lisa CLI subprocess. Streams output and captures the result path
parse_test_results Parse JUnit XML or console output into structured pass/fail/skip counts with per-test details

Example prompt:

Run the T0 smoke tests in ~/lisa using runbook.yml and show me the results

AI Analysis Tools — Understand Failures

Tool What it does
analyze_test_run_with_llm Send all failures from a test run to an LLM. Get root cause categories, fix recommendations, and severity for each failure
analyze_failure_root_cause Deep-dive AI analysis for a single failure — useful for complex or ambiguous failures you want to investigate specifically
generate_analysis_report Full pipeline: parse results → analyze with AI → generate HTML + Markdown report
run_and_analyze End-to-end: run tests → analyze → report in one step
list_llm_providers List all 6 supported LLM providers with endpoint templates and usage instructions

7. AI-Powered Failure Analysis with Any LLM

One of the most powerful features of the LISA MCP Server is structured AI failure analysis. When tests fail, the AI doesn't just tell you what failed — it tells you why and what to do.

Root Cause Categories

The analysis engine classifies every failure into one of these categories:

  • NETWORK_TIMEOUT — DNS, TCP, routing, firewall issues
  • DISK_IO_ERROR — Storage controller, SCSI, NVMe errors
  • KERNEL_PANIC — Kernel crashes, oops, soft lockups
  • MEMORY_EXHAUSTION — OOM killer, huge pages, swap
  • HYPERVISOR_INTEGRATION — Hyper-V VSS, KVP, balloon driver
  • CONFIGURATION_ERROR — Missing packages, wrong permissions, misconfiguration
  • TEST_INFRASTRUCTURE — SSH, agent, or test harness problems
  • UNKNOWN — Fallback for ambiguous failures

Each analysis returns:

  • Root cause category + description
  • Recommended fix (specific commands, paths, or kernel params)
  • Severity level (critical / high / medium / low)
  • Relevant log lines
  • Confidence score

Choosing Your LLM Provider

The server works with any OpenAI-compatible LLM. You're not locked in.

Call list_llm_providers in chat to see all options, then pass the endpoint parameter to any analysis tool:

Provider When to use
Azure OpenAI (default) Enterprise teams with Azure subscriptions. Best performance with sonnet-4.6.
OpenAI Teams with OpenAI API access (platform.openai.com).
Azure AI Foundry / GitHub Models Free tier available with a GitHub PAT. Great for open-source contributors.
Ollama (local) Air-gapped environments or teams with data sovereignty requirements. Run llama3, mistral, phi-3, etc. locally — completely free, no API key needed.
LM Studio (local) Windows/Mac users who want a GUI for local model management.
Azure OpenAI Chat Completions Classic Azure deployments using the older chat completions API format.

Using Ollama (free, local):

Analyze failures in ~/lisa/lisa_results.xml
Endpoint: http://localhost:11434/v1/chat/completions
API key: ollama
Model: llama3
Save the report to ~/reports/

No cloud API key needed. Your test results never leave your machine.

Using OpenAI:

Analyze failures in ~/lisa/lisa_results.xml
API key: sk-your-openai-key
Endpoint: https://api.openai.com/v1/chat/completions
Model: gpt-4o

8. Real Workflows for Distro Providers

Red Hat Enterprise Linux (RHEL)

Scenario: Validating RHEL 9.4 for Azure Marketplace submission.

You: "I need to validate RHEL 9.4 for Azure. Show me all Tier 1 and Tier 2 tests
      covering networking, storage, and kernel integration, then build a runbook."

AI:
  1. discover_test_cases(area="network", tier="T1")
  2. discover_test_cases(area="storage", tier="T1")
  3. discover_test_cases(area="network", tier="T2")
  4. build_runbook(
       name="RHEL94-Azure-Validation",
       platform_type="azure",
       tier="T1",
       image="redhat rhel 9-4 latest",
       include_areas=["network", "storage", "core"]
     )

Result: A filtered test list and a complete runbook YAML, ready to upload to your Azure validation pipeline.


Scenario: RHEL 9.4 validation run just finished and 12 tests failed.

You: "The RHEL 9.4 run finished. Results are in ~/runs/rhel94_results.xml.
      Analyze all failures, tell me root causes, and generate a report for
      the image team."

AI:
  1. parse_test_results(result_path="~/runs/rhel94_results.xml")
     → 127 passed, 12 failed, 3 skipped
  2. analyze_test_run_with_llm(...)
     → 8 × NETWORK_TIMEOUT (likely NSG misconfiguration)
     → 3 × HYPERVISOR_INTEGRATION (Hyper-V VSS driver version)
     → 1 × CONFIGURATION_ERROR (missing waagent package)
  3. generate_analysis_report(output_dir="~/reports/rhel94/")
     → rhel94_analysis.html + rhel94_analysis.md

You now have a stakeholder-ready report in under 2 minutes, with specific recommended fixes for each failure category.


Canonical Ubuntu

Scenario: Validating Ubuntu 24.04 LTS (Noble) across multiple Azure VM sizes.

You: "Build me separate runbooks for Ubuntu 24.04 LTS Azure validation at
      Tier 0 (smoke), Tier 1 (core), and Tier 2 (extended). I want to
      include GPU tests for the Tier 2 runbook."

AI generates three runbooks:
  - ubuntu2404-t0-smoke.yml     (< 10 tests, runs in 5 min)
  - ubuntu2404-t1-core.yml      (full networking + storage + compute)
  - ubuntu2404-t2-extended.yml  (adds GPU, NVMe, kdump, live migration)

Scenario: Ubuntu kernel team wants to add tests for their new io_uring optimization.

You: "Generate a LISA test suite for io_uring performance benchmarking on
     Ubuntu 24.04. Include tests for sequential read, random read/write,
     and mixed I/O patterns. Make them T2 priority."

AI calls generate_test_suite_code(...) and returns:
# Generated by LISA MCP Server
import logging
from dataclasses import dataclass
from typing import Any

from lisa import (
    Node,
    TestCaseMetadata,
    TestSuite,
    TestSuiteMetadata,
    simple_requirement,
)
from lisa.operating_system import Ubuntu
from lisa.testsuite import TestResult

@TestSuiteMetadata(
    area="storage",
    category="performance",
    description="io_uring performance benchmarking for Ubuntu 24.04",
    requirement=simple_requirement(min_core_count=4, min_memory_mb=8192),
)
class IoUringPerformance(TestSuite):

    @TestCaseMetadata(
        description="Verify io_uring sequential read throughput",
        priority=2,
        requirement=simple_requirement(supported_os=[Ubuntu]),
    )
    def verify_iou_sequential_read(self, node: Node, result: TestResult) -> None:
        ...

Your kernel team has a proper, schema-compliant test suite in seconds — not hours.


SUSE Linux Enterprise (SLES)

Scenario: SUSE is preparing SLES 15 SP6 for the Azure Marketplace. The previous SLES 15 SP5 submission had issues with the Hyper-V VSS (Volume Shadow Copy) integration. They want targeted regression testing.

You: "Search for all Hyper-V VSS and backup-related tests in ~/lisa"

AI: search_tests(query="hyper-v VSS backup snapshot")
→ Returns: HypervFloppy, HypervVssBackup, HypervKvp, StorageHotPlug, ...

You: "Build a SLES 15 SP6 runbook that specifically includes these VSS tests
     plus the core storage tier T1 tests"

AI: build_runbook(
      name="SLES15SP6-HypervVSS-regression",
      image="suse sles 15-sp6 latest",
      include_tests=["HypervVssBackup", "HypervKvp", "HypervFloppy"],
      include_areas=["storage"],
      tier="T1"
    )

Targeted regression testing — no manual YAML authoring needed.


Oracle Linux / Amazon Linux / Other Enterprise Distros

The same workflow applies for any distro. The AI understands the LISA test taxonomy regardless of what image you're validating.

One-liner end-to-end run:

You: "Run the T0 smoke tests for Oracle Linux 9 using ~/runbooks/ol9.yml,
     analyze any failures, and email me the report."

AI: run_and_analyze(
      runbook_path="~/runbooks/ol9.yml",
      api_key="<your-key>",
      output_dir="~/reports/ol9/"
    )

For Kernel Developer Teams

Even if you're not a distro provider, LISA MCP is valuable for kernel developers:

You: "I just landed a patch in the virtio-net driver. What LISA tests
     cover virtio network devices? Show me T0 and T1 tests only."

AI: discover_test_cases(area="network", tier="T1", keyword="virtio")

Get a targeted test plan in seconds. Know exactly which tests to run before pushing to a CI gate.


9. CI/CD Integration

The LISA MCP Server is designed to work with both interactive and automated workflows.

GitHub Actions Example

name: LISA Validation
on:
  push:
    branches: [main]
  schedule:
    - cron: '0 2 * * *'   # nightly

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup LISA MCP Server
        run: |
          git clone https://github.com/kkkashan/LISA_MCP_Server.git ~/lisa-mcp
          pip install -e ~/lisa-mcp

      - name: Run and Analyze Tests
        env:
          AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
        run: |
          python3 << 'EOF'
          from lisa_mcp.tools.test_runner import run_lisa_tests
          from lisa_mcp.tools.result_parser import parse_test_results
          from lisa_mcp.tools.llm_analyzer import analyze_run
          from lisa_mcp.tools.report_generator import generate_report
          import os, sys

          run_result = run_lisa_tests(runbook_path="runbook.yml", timeout=3600)
          parsed   = parse_test_results(result_path=run_result["result_path"])
          analysis = analyze_run(
                       test_results=parsed,
                       api_key=os.environ["AZURE_OPENAI_API_KEY"]
                     )
          report   = generate_report(analysis, output_dir="reports/")

          # Fail the build if any critical failures
          if analysis.critical_failure_count > 0:
              print(f"CRITICAL: {analysis.critical_failure_count} critical failures found")
              sys.exit(1)
          EOF

      - name: Upload Report
        uses: actions/upload-artifact@v4
        with:
          name: lisa-analysis-report
          path: reports/

Quality Gate Pattern

Use the LISA MCP Server as a release gate:

  • T0 gate — Must pass before any PR merge
  • T1 gate — Must pass before a nightly build is promoted
  • T2 gate — Must pass before an image is submitted to Marketplace
  • T3/T4 gate — Must pass before a major version release

The AI analysis ensures consistent severity scoring — no more "it's fine, it's probably a flake" judgment calls.


10. Getting Started in 5 Minutes

Prerequisites

  • Python 3.10 or later
  • VS Code with GitHub Copilot extension (for interactive use)
  • A LISA installation (optional — the MCP server also works standalone for runbook authoring and code generation)

Step 1 — Clone and Install

git clone https://github.com/kkkashan/LISA_MCP_Server.git ~/lisa-mcp-server
cd ~/lisa-mcp-server
pip install -e .

Step 2 — Configure the MCP Server in VS Code

The repo ships with .vscode/mcp.json. Open the project in VS Code:

code ~/lisa-mcp-server

VS Code will detect the MCP config and prompt you to start the server. Click Start.

If you're registering manually, edit .vscode/mcp.json and update the cwd path to your absolute path:

{
  "servers": {
    "lisa": {
      "command": "python3",
      "args": ["-m", "lisa_mcp.server"],
      "cwd": "/home/yourname/lisa-mcp-server"
    }
  }
}

Step 3 — Verify the Server

Open Copilot Chat (Ctrl+Alt+I), switch to Agent mode, and type:

Check the LISA environment and list available test tiers

The AI will call check_lisa_environment and get_tier_info. If you see a structured response with tier definitions, the server is running correctly.

Step 4 — Your First Analysis

If you have an existing LISA results file (JUnit XML), try:

Parse results in ~/lisa/results.xml and analyze failures.
Use endpoint: https://api.openai.com/v1/chat/completions
API key: sk-...
Model: gpt-4o

If you don't have an API key yet, use Ollama locally (free):

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3
ollama serve

Then:

Parse results in ~/lisa/results.xml and analyze failures.
Endpoint: http://localhost:11434/v1/chat/completions
API key: ollama
Model: llama3

Step 5 — Build Your First Runbook

Build a Tier 1 Azure runbook for Ubuntu 22.04 covering networking and storage.
Save it to ~/runbooks/ubuntu2204-t1.yml

In 10 seconds you have a production-ready runbook. Validate it:

Validate the runbook at ~/runbooks/ubuntu2204-t1.yml

11. Frequently Asked Questions

Q: Do I need a LISA installation to use this?

For test discovery, execution, and result parsing — yes. For runbook generation, test code generation, and AI failure analysis of existing result files — no. You can use the generation and analysis tools independently.

Q: Can I use this with distros that aren't Ubuntu, RHEL, or SLES?

Absolutely. LISA supports any Linux distribution. The MCP server works with any image: Oracle Linux, Amazon Linux, Flatcar Container Linux, AlmaLinux, Rocky Linux, Debian, openSUSE, and more. Just use the appropriate image name in your runbook.

Q: Does the AI have access to my test results or infrastructure?

Only what you explicitly provide. The AI calls tools on your local machine. Test results, runbooks, and logs stay on your system unless you configure a remote endpoint. Failure analysis text is sent to whichever LLM provider you configure (or stays fully local if you use Ollama/LM Studio).

Q: What if I don't have an OpenAI or Azure account?

Use Ollama for free local inference. Install it, pull any model (llama3, mistral, phi-3), and point the endpoint at http://localhost:11434/v1/chat/completions. No account, no cost, no data leaves your machine. Quality is lower than GPT-4o but sufficient for root cause classification.

Q: Can multiple engineers share one MCP server?

Yes — deploy the server in SSE/HTTP transport mode (instead of stdio) and all engineers can connect to the same instance. This is useful for teams that run tests in a shared CI/CD environment.

Q: How does the server handle large test runs with hundreds of failures?

The server chunks analysis into batches and summarizes at the run level. The analyze_test_run_with_llm tool groups failures by category before sending to the LLM, keeping context windows manageable even for large runs.

Q: Is the LISA framework only for Azure?

No. LISA supports Azure, Hyper-V, QEMU, and bare metal. The MCP server works across all platforms. You can validate the same image on bare metal and Azure and compare results using the same tools.

Q: What about Windows Subsystem for Linux (WSL)?

The MCP server runs inside WSL. Configure VS Code's .vscode/mcp.json with the WSL path (as shown in the Getting Started section). The server will work identically.

Q: How does the AI know which tools to call?

The MCP protocol sends the AI all 18 tool names, descriptions, and typed parameter schemas. The AI reasons over these just like it reasons over documentation. You don't need to say "call discover_test_cases" — just describe what you want in plain English.

Q: Can I contribute new test cases through the MCP server?

Yes. Use generate_test_suite_code to scaffold a new test suite, then check it into your LISA fork. The code generator follows LISA's decorator patterns and Pydantic models, producing tests that pass LISA's own linting.


Closing Thoughts

The Linux testing ecosystem is mature and comprehensive — LISA has hundreds of battle-tested tests, well-defined tiers, and years of signal from production Azure runs. What's been missing is accessibility: the ability for any engineer, regardless of their familiarity with the framework, to compose, execute, and interpret a validation run without a week of onboarding.

That's what the LISA MCP Server provides. A natural language interface to a world-class testing framework — paired with AI that can read failure logs and tell you exactly what broke and how to fix it.

For distro providers shipping to Azure Marketplace, this is the difference between a 2-week validation cycle and a 2-day one. For kernel developers landing upstream patches, it's the ability to know in minutes whether your change broke anything in production-like conditions.

The test infrastructure was already there. Now the AI can use it.


Resources


Written for teams who validate Linux at scale. Feedback and contributions welcome at the GitHub repo above.