refactor(sql): swing consumers to mcpg.sql, delete the vendored kernel (18.1, PR 2)

CHANGELOG.md

@@ -6,6 +6,22 @@ adheres to [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+### Changed
+
+- **De-vendored the SQL-safety kernel (roadmap 18.1).** The formerly-vendored
+  `crystaldba/postgres-mcp` `sql/` subpackage (`src/mcpg/_vendor/`) is
+  replaced by a first-party `src/mcpg/sql/` package — `allowlist.py` (the
+  permitted statement/node/function/extension policy, as data), `safety.py`
+  (`SafeSqlDriver`, the `pglast` allowlist validator), and `driver.py`
+  (`SqlDriver` / `DbConnPool` / `obfuscate_password`). Behaviour is
+  **identical** (proven by a differential parity harness — 0 divergence —
+  plus the ported 760-LOC adversarial suite, a fuzz pass, and a
+  `/security-review` with no findings); the public seam is unchanged, so no
+  tool signatures moved. The kernel is now inside the coverage gate,
+  `mypy --strict`, `ruff`, and `bandit` (the vendored code was excluded from
+  all four). MCPg now ships no vendored runtime code. Supersedes ADR-0001
+  with ADR-0007.
+
 ### Added
 
 - **Three more built-in NL→SQL providers (19 → 22).** `translate_nl_to_sql`

CLAUDE.md

@@ -48,16 +48,27 @@ a read-only HF Spaces demo.
   `GITHUB_TOKEN`, `DEEPINFRA_TOKEN`). The provider table in
   `docs/user-guide.md` mirrors it.
 
-## The vendored SQL-safety kernel
+## The SQL-safety kernel (first-party)
 
-`src/mcpg/_vendor/sql/` is copied near-verbatim from
-[`crystaldba/postgres-mcp`](https://github.com/crystaldba/postgres-mcp)
-(MIT, pinned commit `07eb329`; see ADR-0001 and
-`src/mcpg/_vendor/README.md`): the `pglast` allowlist validator
-(`safe_sql.py`), async pool/driver (`sql_driver.py`), param binding.
-It's excluded from the coverage gate and `mypy --strict`; its own
-adversarial tests live in `tests/vendor/sql/`. **Never hand-edit** —
-local mods are tracked in the vendor README and re-applied on re-sync.
+`src/mcpg/sql/` is MCPg's own SQL-safety kernel (roadmap 18.1 de-vendored
+the former `crystaldba/postgres-mcp` copy; see ADR-0007, which supersedes
+ADR-0001). Three modules, policy separated from mechanism:
+
+- `sql/allowlist.py` — the SQL-safety **policy as data**: permitted
+  statement / `pglast` AST node / function / extension sets. The single
+  auditable decision surface.
+- `sql/safety.py` — `SafeSqlDriver`: the `pglast` parse + AST-walker +
+  read-only execute path. Reads policy from `allowlist.py`; can't widen it.
+- `sql/driver.py` — `SqlDriver` / `DbConnPool` / `obfuscate_password`
+  (pool + execution + credential redaction; no policy).
+
+Exports `SqlDriver`, `SafeSqlDriver`, `DbConnPool`, `obfuscate_password`
+(the seam ~74 `mcpg.*` modules depend on). Fully inside the coverage gate
++ `mypy --strict` + `ruff` + `bandit` like the rest of `mcpg`. Adversarial
++ fuzz tests: `tests/unit/test_sql_kernel_*.py`. Changing the allowlist is
+a **security-sensitive** edit — run the adversarial suite and treat the
+`docs/reviews/devendor-sql-kernel-security-review.md` threat model as the
+bar.
 
 ## Workflow conventions
 

NOTICE

@@ -6,19 +6,20 @@ the repository root for the full text. (Relicensed from AGPL-3.0-or-later
 in v0.5.x.)
 
 ------------------------------------------------------------------------
-Third-party vendored code
+Attribution — SQL-safety kernel lineage
 ------------------------------------------------------------------------
 
-This product includes software developed by Crystal Corp.
+MCPg's SQL-safety kernel (`src/mcpg/sql/`) is first-party code, but it was
+re-authored from — and its allowlist policy derives from — the MIT-licensed
+`sql/` subpackage of:
 
-  Component : PostgreSQL SQL-safety kernel (the `sql/` subpackage)
-  Location  : src/mcpg/_vendor/sql/
-  Source    : https://github.com/crystaldba/postgres-mcp
-  Commit    : 07eb329c8c48e49640e0d1b5b35465d4d024c3ee
-  License   : MIT License, Copyright (c) 2025, Crystal Corp.
-              See src/mcpg/_vendor/LICENSE for the full license text.
+  Project : crystaldba/postgres-mcp ("Postgres MCP Pro")
+  Source  : https://github.com/crystaldba/postgres-mcp
+  Origin  : commit 07eb329c8c48e49640e0d1b5b35465d4d024c3ee (Jan 2026)
+  License : MIT License, Copyright (c) 2025, Crystal Corp.
 
-The MIT-licensed code above is incorporated under the terms of the MIT
-License; both this project and the vendored kernel are now MIT-licensed.
-See src/mcpg/_vendor/README.md for provenance details and the re-sync
-procedure.
+The original kernel was vendored under src/mcpg/_vendor/sql/ from v0.x
+until roadmap 18.1 replaced it with the first-party implementation (see
+docs/adr/0007-first-party-sql-kernel.md, which supersedes ADR-0001). This
+attribution acknowledges that MIT-licensed lineage; MCPg itself remains
+MIT-licensed.

README.md

@@ -590,9 +590,9 @@ checklist.
 
 ## License
 
-MIT — see [`LICENSE`](https://github.com/devopam/MCPg/blob/main/LICENSE). The vendored SQL-safety kernel at
-`src/mcpg/_vendor/sql/` is also MIT-licensed; see [`NOTICE`](https://github.com/devopam/MCPg/blob/main/NOTICE)
-for provenance.
+MIT — see [`LICENSE`](https://github.com/devopam/MCPg/blob/main/LICENSE). The SQL-safety kernel
+(`src/mcpg/sql/`) is first-party, re-authored from the MIT-licensed
+`crystaldba/postgres-mcp`; see [`NOTICE`](https://github.com/devopam/MCPg/blob/main/NOTICE) for the lineage.
 
 ### Wrapped extensions — licenses you should know about
 

SECURITY.md

@@ -34,7 +34,8 @@ release notes unless they prefer otherwise.
 
 **In scope:**
 
-- The MCPg server code under `src/mcpg/` (excluding `src/mcpg/_vendor/`)
+- The MCPg server code under `src/mcpg/` — including the first-party
+  SQL-safety kernel `src/mcpg/sql/` (the `pglast` allowlist validator)
 - Authentication & authorisation paths (bearer-token, OIDC,
   multi-tenancy `SET LOCAL ROLE`)
 - The capability gates that restrict tool surfaces by access mode
@@ -45,9 +46,6 @@ release notes unless they prefer otherwise.
 **Out of scope:**
 
 - Vulnerabilities in PostgreSQL itself
-- Vulnerabilities in the vendored SQL-safety kernel at
-  `src/mcpg/_vendor/sql/` — those go upstream to
-  `crystaldba/postgres-mcp`
 - Issues that require an attacker to already have `unrestricted`
   access mode AND `MCPG_ALLOW_DDL=true` (that combination is
   by-design root access)

docs/adr/0001-build-approach.md

@@ -1,8 +1,13 @@
 # ADR-0001: Build approach — adopt `crystaldba/postgres-mcp` as a hard-forked base
 
-- **Status:** accepted
+- **Status:** superseded by [ADR-0007](0007-first-party-sql-kernel.md) (2026-07-08)
 - **Date:** 2026-05-20
 
+> **Superseded.** The vendored SQL-safety kernel this ADR introduced was
+> replaced by a first-party implementation in roadmap 18.1 — see
+> [ADR-0007](0007-first-party-sql-kernel.md). The historical rationale
+> below is retained as the record of the original decision.
+
 ## Context
 
 The official `@modelcontextprotocol/server-postgres` is deprecated and archived

docs/adr/0007-first-party-sql-kernel.md

@@ -0,0 +1,81 @@
+# ADR-0007: First-party SQL-safety kernel (de-vendor `crystaldba/postgres-mcp`)
+
+- **Status:** accepted
+- **Date:** 2026-07-08
+- **Supersedes:** [ADR-0001](0001-build-approach.md) (the hard-fork / vendor decision)
+
+## Context
+
+[ADR-0001](0001-build-approach.md) chose to hard-fork MCPg from
+`crystaldba/postgres-mcp` and **vendor** its `sql/` subpackage — the
+`pglast`-AST SQL-safety allowlist validator plus the async connection
+pool / driver — near-verbatim under `src/mcpg/_vendor/sql/` (MIT, pinned
+commit `07eb329`). That was the right call to ship fast, but it left two
+standing costs:
+
+1. **~1.3k lines of security-critical code outside every quality gate.**
+   The vendored kernel — including the SQL allowlist itself — was excluded
+   from the coverage gate, `mypy --strict`, `ruff`, and `bandit`
+   (`pyproject.toml` carve-outs).
+2. **Manual re-syncs.** Upstream changes had to be re-applied by hand with
+   local modifications tracked in a vendor README.
+
+It was also the last third-party runtime *code* MCPg shipped. Roadmap 18.1
+set out to own it.
+
+## Decision
+
+Replace the vendored kernel with a **first-party** `src/mcpg/sql/` package,
+re-authored from the MIT-licensed original (attribution retained in
+`NOTICE`). Recon showed three of the vendored modules (`bind_params.py`,
+`extension_utils.py`, `index.py`) had no consumer outside `_vendor/`, so
+they were **deleted, not rebuilt**. The kernel is re-architected to
+**separate policy from mechanism**:
+
+- `sql/allowlist.py` — the permitted statement / AST-node / function /
+  extension sets, as **data** (the single auditable decision surface).
+- `sql/safety.py` — `SafeSqlDriver`: the `pglast` parse + AST-walker +
+  read-only execute path. Reads policy from `allowlist.py`; cannot widen it.
+- `sql/driver.py` — `SqlDriver` / `DbConnPool` / `obfuscate_password`
+  (pool + execution + credential redaction; no policy).
+
+The public seam (`SqlDriver`, `SafeSqlDriver`, `DbConnPool`,
+`obfuscate_password`) is unchanged, so the ~74 consuming modules only
+changed an import path.
+
+**Faithful re-author, not a redesign.** The security *behaviour* is pinned
+identical to the vendored validator:
+
+- A **differential parity harness** ran a safe/unsafe/malformed corpus
+  through both validators — **0 divergence**.
+- The 760-LOC adversarial suite (ported) passes 100% against `mcpg.sql`.
+- A **fuzz pass** confirms the validator only ever returns a clean verdict
+  (accept / `ValueError`) — never a crash / hang — on adversarial input.
+- A dedicated security-review gate (allowlist audit, `/security-review`,
+  threat model) passed — see
+  [`../reviews/devendor-sql-kernel-security-review.md`](../reviews/devendor-sql-kernel-security-review.md).
+
+## Consequences
+
+- The SQL-safety kernel is now inside the coverage gate (90%),
+  `mypy --strict`, `ruff`, and `bandit` — a net security improvement over
+  the vendored state.
+- `src/mcpg/_vendor/` and `tests/vendor/` are deleted; the five
+  `pyproject.toml` `_vendor` carve-outs are removed; the module map and
+  docs (`architecture.md`, `CLAUDE.md`, `SECURITY.md`, `NOTICE`) are
+  first-party.
+- MCPg ships **no vendored runtime code**. (`pglast`, `psycopg`, and
+  `psycopg-pool` remain upstream *library* dependencies — not vendored
+  source.)
+- Attribution to `crystaldba/postgres-mcp` (MIT) is retained in `NOTICE`
+  as the design lineage; the in-query `/* crystaldba */` marker is kept
+  for now (a cosmetic rename is a possible follow-up).
+
+## Alternatives considered
+
+- **Keep vendoring.** Rejected — perpetuates the gate carve-outs and the
+  manual re-sync burden; ADR-0001's "ship fast" rationale no longer applies.
+- **Redesign the validator (stricter/simpler allowlist).** Deferred — a
+  behaviour change would need its own threat model + review. This ADR is a
+  faithful re-author; any policy tightening is a separate, reviewed change
+  (see the pre-existing observations in the security-review note).

docs/architecture.md

@@ -51,7 +51,7 @@ flowchart TD
    runs the SQL and maps rows to typed dataclasses.
 4. The driver stack decides exactly which pool the SQL hits:
    - **SafeSqlDriver** — agent-supplied SQL is parsed and
-     allowlisted via the vendored kernel before execution.
+     allowlisted via `mcpg.sql` (the first-party kernel) before execution.
    - **RoutedSqlDriver** — when `MCPG_REPLICA_URLS` is set,
      `force_readonly=True` queries round-robin across healthy
      replicas; writes always go to the primary.
@@ -64,7 +64,7 @@ flowchart TD
 ---
 
 <!-- BEGIN generated: module-map (python tools/generate_doc_tables.py --modules) -->
-## Module map (104 modules)
+## Module map (103 modules)
 
 Every `mcpg.*` module and what it owns, alphabetical. The layered
 request path through these lives in the [Overview](#overview) diagram;
@@ -73,7 +73,6 @@ this table is the exhaustive index. Regenerate with
 
 | Module | Responsibility |
 |---|---|
-| `mcpg._vendor` | Vendored MIT-licensed `SafeSqlDriver` + connection-pool kernel (SQL parse / allowlist / bind). |
 | `mcpg.about` | MCPg self-description. |
 | `mcpg.advisors` | Schema advisors — codified lint rules over the PG catalog. |
 | `mcpg.aio` | `AIO` — PG 19 asynchronous-I/O subsystem coverage. |
@@ -184,17 +183,28 @@ this table is the exhaustive index. Regenerate with
 
 ---
 
-## The vendored SQL-safety kernel
-
-`src/mcpg/_vendor/sql/` is a pinned copy of the SQL-safety
-subpackage from
-[`crystaldba/postgres-mcp`](https://github.com/crystaldba/postgres-mcp)
-(MIT). It provides `SafeSqlDriver` — a `pglast`-AST allowlist
-validator — and the base connection pool / driver.
-
-The kernel is kept near-verbatim, excluded from the coverage gate
-and `mypy`, and re-synced via the procedure in
-`src/mcpg/_vendor/README.md`. See [ADR-0001](adr/0001-build-approach.md).
+## The SQL-safety kernel (first-party)
+
+`src/mcpg/sql/` is MCPg's own SQL-safety kernel, split into policy
+and mechanism:
+
+- `sql/allowlist.py` — the permitted statement / `pglast` AST node /
+  function / extension sets, **as data** (the single auditable policy
+  surface).
+- `sql/safety.py` — `SafeSqlDriver`: the `pglast` parse + AST-walker
+  + read-only execute path. Reads policy from `allowlist.py`; the
+  walker can't widen it.
+- `sql/driver.py` — `SqlDriver` / `DbConnPool` / `obfuscate_password`
+  (pool + execution + credential redaction; no policy).
+
+It's fully inside the coverage gate + `mypy --strict` + `ruff` +
+`bandit`. Adversarial + fuzz tests live in
+`tests/unit/test_sql_kernel_*.py`; the threat model and security
+sign-off are in
+[`reviews/devendor-sql-kernel-security-review.md`](reviews/devendor-sql-kernel-security-review.md).
+It was de-vendored from `crystaldba/postgres-mcp` (MIT) — see
+[ADR-0007](adr/0007-first-party-sql-kernel.md), which supersedes
+[ADR-0001](adr/0001-build-approach.md).
 
 ---
 
@@ -294,14 +304,13 @@ otherwise.
 
 ## Testing approach
 
-MCPg is test-driven across four suites:
+MCPg is test-driven across three suites:
 
 | Suite | Scope |
 |---|---|
-| `tests/unit/` | Fake-driver tests with a 90% coverage gate. Authored code only — the vendored kernel keeps its own tests. |
+| `tests/unit/` | Fake-driver tests with a 90% coverage gate — all first-party code, including the SQL-safety kernel (`test_sql_kernel_*.py`: the adversarial allowlist suite + a fuzz/robustness pass). |
 | `tests/integration/` | Real PostgreSQL — requires `MCPG_TEST_DATABASE_URL`. CI runs the matrix against PostgreSQL 14, 15, 16, 17, 18 on every push (pgvector + PostGIS + AGE image), plus an **experimental PG 19** lane (pgvector built from source against `postgres:19beta1`) and a **WarehousePG** (Greenplum-derived MPP) characterisation lane. |
 | `tests/contract/` | Tool-surface snapshot (`tool_surface.snapshot.json`) + the doc-table drift-guard (`test_doc_tables.py`), so a new tool or module can't ship undocumented. |
-| `tests/vendor/` | The vendored kernel's own upstream tests, kept for adversarial SQL-injection coverage. |
 
 The integration container is built from
 `.github/ci-postgres.Dockerfile` and includes `pgvector`,

docs/contributing/adding-tools.md

@@ -41,7 +41,7 @@ from __future__ import annotations
 
 from dataclasses import dataclass, field
 
-from mcpg._vendor.sql import SqlDriver
+from mcpg.sql import SqlDriver
 from mcpg.extensions import extension_installed
 
 # Module-level constants (mode allowlists, defaults). Inline so the

docs/feature-shortlist.md

@@ -270,19 +270,15 @@ empty scratch database), which shows off none of the tool surface.
 
 ## 18. De-vendor the SQL-safety kernel — build our own
 
-MCPg's SQL-safety core is currently **vendored** from a third party
-(see [ADR-0001](adr/) and [`src/mcpg/_vendor/README.md`](../src/mcpg/_vendor/README.md)):
-the `pglast`-based allowlist validator, async connection pool / driver,
-and parameter binding all live under `src/mcpg/_vendor/sql/`, copied
-near-verbatim from
-[`crystaldba/postgres-mcp`](https://github.com/crystaldba/postgres-mcp)
-(MIT, pinned commit `07eb329`, Copyright © 2025 Crystal Corp). The goal
-is to **own this layer outright** — remove the upstream dependency and
-replace it with a first-party implementation MCPg fully controls.
+MCPg's SQL-safety core was **vendored** from `crystaldba/postgres-mcp`
+(MIT) under `src/mcpg/_vendor/sql/`. Roadmap 18.1 replaced it with a
+first-party implementation MCPg fully controls — removing the last
+vendored runtime code and folding the SQL allowlist into every quality
+gate.
 
 | # | Item | Effort | Value | Notes |
 |---|---|---|---|---|
-| 18.1 | ⬜ **Planned (agenda, no commitment yet).** **Replace the vendored `crystaldba/postgres-mcp` SQL-safety kernel with a first-party implementation.** In scope: `safe_sql.py` (the `pglast` parse + statement/keyword allowlist that every agent-supplied query passes through), `sql_driver.py` (the async psycopg pool + `DbConnPool`, incl. MCPg's ADR-0003 pool-sizing local mod), `bind_params.py`, `extension_utils.py`, `index.py`. Constraints: (a) **no safety regression** — the adversarial SQL-injection suite in `tests/vendor/sql/test_safe_sql.py` must be preserved (or re-implemented) and kept green, and the tool-surface contract test must not change; (b) preserve the `SafeSqlDriver` / pool public API the rest of `mcpg.*` depends on, or migrate call sites in the same effort; (c) fold the code into the coverage gate + `mypy --strict` once it's ours (the `_vendor/` carve-outs go away); (d) drop the vendored dir, its `README.md`/`LICENSE`, and the ADR-0001 rationale, and record the change in a new ADR. Likely a multi-PR effort: first re-home + test-parity, then simplify/own. **Phased plan: [`plans/devendor-sql-kernel.md`](plans/devendor-sql-kernel.md)** — recon found `bind_params.py` / `extension_utils.py` / `index.py` are **dead** (no consumer outside `_vendor/`), so they're deleted not rebuilt; the real rebuild is just `safe_sql.py` + `sql_driver.py`, and the consumer seam is 4 names (`SqlDriver` / `SafeSqlDriver` / `DbConnPool` / `obfuscate_password`). | L | Medium | Removes the last third-party runtime core; full control over the safety semantics. Revisit ADR-0001's "why vendor" reasoning when we start. |
+| 18.1 | ✅ **Shipped.** Replaced the vendored `crystaldba/postgres-mcp` kernel with a first-party `src/mcpg/sql/` package — `allowlist.py` (policy as data), `safety.py` (`SafeSqlDriver` walker), `driver.py` (`SqlDriver`/`DbConnPool`/`obfuscate_password`). The dead `bind_params.py` / `extension_utils.py` / `index.py` were deleted, not rebuilt. All ~74 consumers swung to `mcpg.sql`; `_vendor/` + `tests/vendor/` deleted; the 5 quality-gate carve-outs removed (the kernel is now inside coverage + `mypy --strict` + `ruff` + `bandit`). Faithful re-author, proven identical by a differential parity harness (0 divergence) + the ported 760-LOC adversarial suite + a fuzz pass + `/security-review` (no findings). Supersedes ADR-0001 with [ADR-0007](adr/0007-first-party-sql-kernel.md); plan in [`plans/devendor-sql-kernel.md`](plans/devendor-sql-kernel.md), sign-off in [`reviews/devendor-sql-kernel-security-review.md`](reviews/devendor-sql-kernel-security-review.md). | L | Medium | Removes the last third-party runtime core; full control over the safety semantics. |
 
 ---
 

docs/release-process.md

@@ -144,7 +144,7 @@ description = "A production-grade PostgreSQL Model Context Protocol (MCP) server
 readme = "README.md"
 requires-python = ">=3.12"
 license = "MIT"
-license-files = ["LICENSE", "src/mcpg/_vendor/LICENSE"]
+license-files = ["LICENSE"]
 ```
 
 ### 3.2 What to add before the first release
@@ -160,7 +160,7 @@ description = "A production-grade PostgreSQL Model Context Protocol (MCP) server
 readme = "README.md"                              # → "Project description" tab
 requires-python = ">=3.12"
 license = "MIT"
-license-files = ["LICENSE", "src/mcpg/_vendor/LICENSE"]
+license-files = ["LICENSE"]
 
 # Surfaces under "Author" on the project page + in `pip show`.
 authors = [