|
| 1 | +# PROJECT KNOWLEDGE BASE |
| 2 | + |
| 3 | +**Updated:** 2026-06-23 |
| 4 | + |
| 5 | +## OVERVIEW |
| 6 | + |
| 7 | +**Project:** teradata-mcp-evals |
| 8 | + |
| 9 | +**Purpose:** Eval suite for the [Teradata MCP Server](https://github.com/Teradata/teradata-mcp-server) community edition. Primary goal is **MCP tool description quality** — tests whether an LLM agent selects the right tool and forms valid parameters from natural language. Failures on `ambiguous_selection` cases usually indicate overlapping or unclear tool descriptions in the MCP server. |
| 10 | + |
| 11 | +**Stack:** |
| 12 | +- **Language:** Python ≥ 3.11 |
| 13 | +- **Package manager / build tool:** `uv` (recommended) or standard `venv` + `pip` |
| 14 | +- **Core dependencies:** |
| 15 | + - `deepeval>=1.4.0` |
| 16 | + - `mcp>=1.0.0` |
| 17 | + - `boto3>=1.34.0` |
| 18 | + - `python-dotenv>=1.0.0` |
| 19 | + - `pytest>=8.0.0` |
| 20 | + - `pytest-asyncio>=0.23.0` |
| 21 | + - `teradatasql>=20.0.0` |
| 22 | +- **Auxiliary tools:** `uv`, `dotenv`, deepeval CLI, `teradatasql` driver, AWS Bedrock (Claude) for agent + judge LLM calls |
| 23 | + |
| 24 | +**Case types:** |
| 25 | + |
| 26 | +| Type | Purpose | |
| 27 | +|------|---------| |
| 28 | +| `happy_path` | Unambiguous prompt → correct tool + params | |
| 29 | +| `ambiguous_selection` | Two tools could apply → tests description distinctness | |
| 30 | +| `missing_parameter` | Vague prompt → agent must ask, not hallucinate | |
| 31 | +| `multi_tool` | Chained tool calls in order | |
| 32 | +| `multi_turn` (optional) | 2–7 turns: clarify first, then correct tool after user supplies info | |
| 33 | + |
| 34 | +**Module priority:** P0 `base` · P1 `dba`, `sec` · P2 `qlty` · maintained `chat`, `plot`, `tmpl` |
| 35 | + |
| 36 | +## STRUCTURE |
| 37 | + |
| 38 | +``` |
| 39 | +agent/ |
| 40 | + client.py # MCP agent; optional description_overrides (opt-in) |
| 41 | +judge/ |
| 42 | + bedrock_llm.py # DeepEvalBaseLLM wrapper for Bedrock Converse API |
| 43 | + checks.py # Deterministic structural checks (fail before LLM judge) |
| 44 | + metrics.py # ToolCorrectnessMetric + Clarification GEval |
| 45 | + report.py # Eval summaries → results/latest_summary.* |
| 46 | + suggest_overrides.py # LLM draft overrides from failed ambiguous_selection cases |
| 47 | +cases/ |
| 48 | + <module>.json # Eval case definitions per MCP module prefix |
| 49 | +tests/ |
| 50 | + conftest.py # Fixtures, {EVALS_DATABASE} substitution, assert_eval_case() |
| 51 | + case_runner.py # Single/multi-turn execution and per-turn scoring |
| 52 | + test_<module>.py # One pytest file per module (all use assert_eval_case) |
| 53 | + test_checks.py # Unit tests for judge/checks.py |
| 54 | + test_multi_turn.py # Unit tests for multi-turn schema validation |
| 55 | +backup/ # Optional bootstrap/audit scripts — see backup/README.md |
| 56 | +run_evals.py # CLI entry point (deepeval + pytest) |
| 57 | +suggest_overrides.py # CLI: draft description_overrides from eval failures |
| 58 | +setup_test_data.py # Create evals_employees / evals_orders in EVALS_DATABASE |
| 59 | +preflight.py # Verify eval tables exist before live eval runs |
| 60 | +teardown_test_data.py |
| 61 | +docs/ # Extended user documentation |
| 62 | +README.md # Quick start and doc index |
| 63 | +.env.example # Required environment variables |
| 64 | +pyproject.toml # Project metadata, Ruff config, pytest paths |
| 65 | +``` |
| 66 | + |
| 67 | +## COMMANDS |
| 68 | + |
| 69 | +| Action | Command | |
| 70 | +|--------|---------| |
| 71 | +| **Install dependencies** (recommended) | `curl -LsSf https://astral.sh/uv/install.sh \| sh && uv venv && uv sync` | |
| 72 | +| **Install dependencies** (standard) | `python3 -m venv .venv && source .venv/bin/activate && pip install -e .` | |
| 73 | +| **Run test data setup** | `python setup_test_data.py` (or `uv run python setup_test_data.py`) | |
| 74 | +| **Verify eval tables** | `python preflight.py` (also runs automatically via `run_evals.py`) | |
| 75 | +| **Generate happy-path cases** | `uv run python backup/generate_cases.py` | |
| 76 | +| **Regenerate after description change** | `uv run python backup/generate_cases.py --module base --overwrite` | |
| 77 | +| **Audit ambiguous pair coverage** | `uv run python backup/audit_cases.py --strict` | |
| 78 | +| **Audit against live MCP tools** | `uv run python backup/audit_cases.py --live-mcp --strict` | |
| 79 | +| **List eval runs** | `uv run python run_evals.py --list-runs` | |
| 80 | +| **Suggest description overrides** | `uv run python suggest_overrides.py` (after baseline eval) | |
| 81 | +| **Apply description overrides** | `uv run python suggest_overrides.py --apply` (after reviewing draft) | |
| 82 | +| **Test description overrides** | `uv run python run_evals.py --with-description-overrides` | |
| 83 | +| **Unit tests (local)** | `uv run pytest tests/test_checks.py tests/test_multi_turn.py tests/test_report.py tests/test_suggest_overrides.py tests/test_description_overrides.py -v` | |
| 84 | +| **Run full eval suite** | `python run_evals.py` (or `uv run python run_evals.py`) | |
| 85 | +| **Run one module** | `python run_evals.py --module base` | |
| 86 | +| **Filter by case type** | `python run_evals.py --module base --type ambiguous_selection` | |
| 87 | +| **Run multi-turn cases only** | `python run_evals.py --module base --type multi_turn` | |
| 88 | +| **Verbose output** | `python run_evals.py --verbose` | |
| 89 | +| **Unit tests (no MCP/Bedrock)** | `uv run pytest tests/test_checks.py tests/test_multi_turn.py tests/test_report.py tests/test_suggest_overrides.py tests/test_description_overrides.py` | |
| 90 | +| **Run tests directly (pytest)** | `pytest` (or `uv run pytest`) | |
| 91 | +| **Tear down test data** | `python teardown_test_data.py` | |
| 92 | +| **Package script** | `run-evals` (installed via `[project.scripts]` in `pyproject.toml`) | |
| 93 | + |
| 94 | +### `--type` filter keywords (maps to pytest `-k` on case IDs) |
| 95 | + |
| 96 | +| `--type` | Matches IDs containing | |
| 97 | +|----------|------------------------| |
| 98 | +| `happy_path` | `happy` | |
| 99 | +| `ambiguous_selection` | `ambiguous` | |
| 100 | +| `missing_parameter` | `missing` | |
| 101 | +| `multi_tool` | `multi_tool` | |
| 102 | +| `multi_turn` | `clarify_then_call` | |
| 103 | + |
| 104 | +## CODING STANDARDS |
| 105 | + |
| 106 | +- **Language:** Python 3.11+ with full type hints (`typing`, `dataclasses`). |
| 107 | +- **Style:** PEP 8; line length 120 (Ruff in `pyproject.toml`). |
| 108 | +- **Docstrings:** Triple-quoted module- and function-level docstrings. |
| 109 | +- **Error handling:** Exceptions caught and surfaced as readable messages (e.g. tool errors in `agent/client.py`). |
| 110 | +- **Configuration:** `python-dotenv`; secrets in `.env` (never commit). Key vars: |
| 111 | + - `MCP_SERVER_URL`, `EVALS_DATABASE`, `BEDROCK_MODEL_ID` |
| 112 | + - `BEDROCK_JUDGE_MODEL_ID` (optional, defaults to agent model) |
| 113 | + - `USE_DESCRIPTION_OVERRIDES` / `DESCRIPTION_OVERRIDES_FILE` (optional — default evals use live MCP descriptions) |
| 114 | + - `AGENT_MAX_STEPS` (default 5, single-turn) |
| 115 | + - `AGENT_MAX_STEPS_PER_TURN` (default 3, multi-turn) |
| 116 | +- **Testing:** Pytest + DeepEval for integration evals; pure unit tests for `judge/checks.py` and schema validation without Bedrock/MCP. |
| 117 | + |
| 118 | +## WHERE TO LOOK |
| 119 | + |
| 120 | +| Task | Location | |
| 121 | +|------|----------| |
| 122 | +| Agent loop / MCP session | `agent/client.py` | |
| 123 | +| Deterministic checks | `judge/checks.py` | |
| 124 | +| LLM judge metrics | `judge/metrics.py` | |
| 125 | +| Run + score any case | `tests/case_runner.py` → `assert_eval_case()` | |
| 126 | +| Eval run summaries | `judge/report.py` → `results/runs/<run_id>/` + `results/latest.json` | |
| 127 | +| List recent runs | `run_evals.py --list-runs` | |
| 128 | +| Override suggestions | `suggest_overrides.py` → run's `suggested_overrides.json` | |
| 129 | +| Apply override draft | `suggest_overrides.py --apply` → `description_overrides.json` | |
| 130 | +| Apply overrides during evals | `description_overrides.json` + `--with-description-overrides` (`agent/client.py`) | |
| 131 | +| Case JSON definitions | `cases/<module>.json` | |
| 132 | +| Ambiguous pair registry / live MCP diff | `backup/audit_cases.py` → `AMBIGUOUS_PAIRS`, `--live-mcp` | |
| 133 | +| Happy-path generator | `backup/generate_cases.py` | |
| 134 | +| Test data setup / preflight | `setup_test_data.py`, `preflight.py` | |
| 135 | +| `{EVALS_DATABASE}` substitution | `tests/conftest.py` → `_substitute()` |s |
| 136 | +| User-facing docs | `README.md`, `docs/` | |
| 137 | + |
| 138 | +## EVAL FLOW |
| 139 | + |
| 140 | +**Description quality loop** (see `docs/workflow.md`): |
| 141 | + |
| 142 | +1. Baseline: `run_evals.py` — agent uses live MCP descriptions. |
| 143 | +2. Summary: `judge/report.py` writes `results/runs/<run_id>/` and updates `results/latest.json`. |
| 144 | +3. Draft: `suggest_overrides.py` — LLM rewrites for failed `ambiguous_selection` cases. |
| 145 | +4. Test: `suggest_overrides.py --apply`, then re-run with `--with-description-overrides`. |
| 146 | +5. Ship: promote to MCP server, re-run baseline. |
| 147 | + |
| 148 | +**Per-case scoring:** |
| 149 | + |
| 150 | +1. Test loads case from `cases/<module>.json` via `load_cases()`. |
| 151 | +2. `{EVALS_DATABASE}` placeholder substituted at runtime. |
| 152 | +3. **Single-turn:** `run_agent()` → deterministic checks → deepeval metrics. |
| 153 | +4. **Multi-turn:** `run_agent_turns()` (one MCP session, max 7 turns) → per-turn checks: |
| 154 | + - Clarification turns: no tools + Clarification GEval |
| 155 | + - Tool turns: deterministic checks + ToolCorrectnessMetric |
| 156 | +5. Structural failures in `judge/checks.py` raise `AssertionError` before the judge runs. |
| 157 | +6. Outcomes recorded under `results/runs/<run_id>/` via `judge/report.py`. |
| 158 | + |
| 159 | +## CASE JSON CONVENTIONS |
| 160 | + |
| 161 | +**Single-turn:** top-level `input`, `type`, `expected_tools`. |
| 162 | + |
| 163 | +**Multi-turn:** top-level `turns` (2–7 entries), each with `input` and exactly one of: |
| 164 | +- `"expect": "clarification"` |
| 165 | +- `"expected_tools": [...]` |
| 166 | + |
| 167 | +Multi-turn case IDs should contain `clarify_then_call` for `--type multi_turn` filtering. |
| 168 | + |
| 169 | +Param names in `expected_tools` must match live MCP schemas (e.g. `sql` for `base_readQuery`, `user_name` for sec tools). Use `{EVALS_DATABASE}.evals_*` tables for deterministic grounding in base, qlty, plot, and dba cases where applicable. |
| 170 | + |
| 171 | +## MCP DESCRIPTION CHURN WORKFLOW |
| 172 | + |
| 173 | +When a tool description changes in the MCP server: |
| 174 | + |
| 175 | +1. `backup/generate_cases.py --module <m> --overwrite` for happy paths (optional) |
| 176 | +2. Manually review `ambiguous_selection` cases in that module |
| 177 | +3. `backup/audit_cases.py --module <m> --strict` (optional) |
| 178 | +4. Run baseline evals; use `suggest_overrides.py` on failures; test with `--with-description-overrides` |
| 179 | +5. Promote accepted wording to the MCP server repo; re-run baseline evals |
| 180 | +6. Add new tool pairs to `AMBIGUOUS_PAIRS` in `backup/audit_cases.py` when overlap is discovered |
| 181 | + |
| 182 | +## NOTES |
| 183 | + |
| 184 | +- Requires running **Teradata MCP Server** at `MCP_SERVER_URL` and **ClearScape** database `EVALS_DATABASE` with demo tables from `setup_test_data.py`. |
| 185 | +- AWS Bedrock credentials via standard boto3 chain. Default model: `anthropic.claude-3-5-sonnet-20241022-v2:0`. |
| 186 | +- Eval results stored in `results/` (not committed). Each run gets `results/runs/<run_id>/` with summary, manifest, and optional suggestion draft. `results/latest.json` points at the newest run; `results/index.json` lists recent runs. |
| 187 | +- Do not commit `.env` or credentials. |
| 188 | +- Hand-author edge cases (`ambiguous_selection`, `missing_parameter`, `multi_tool`, `multi_turn`) after generating happy paths — generator cannot replace human judgement on description boundaries. |
| 189 | +- Prompt vocabulary should differ from tool descriptions (same rule enforced in `backup/generate_cases.py` DRAFT_PROMPT). |
0 commit comments