Introduces local RAG Ingest MCP server (rag-ingest-mcp) on top of unstructured-ingest that pairs with transform-mcp (supports Chroma, Qdrant, pgvector)#750
Conversation
A FastMCP STDIO server that lands Unstructured Transform MCP output into a local vector store and serves similarity search. Transform parses/chunks/optionally embeds on the platform; this server fetches the Element JSON out of band, embeds it locally when Transform did not, upserts into a local Chroma collection, and answers queries embedded into that same space. Reuses unstructured-ingest's Chroma stager/uploader and embedding encoders, so the write path matches the ingest pipeline's. A collection is pinned to one embedding space (provider, model, dimension); later loads with a different model are refused, keeping corpus and query vectors comparable. Tools: load_transform_output (embed_policy auto/passthrough/local), search, list_collections. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A source may now be a directory; every *.json file under it is ingested as its own document.
…g-3-small The 1536-dim default fits every candidate backend's ANN-index limits (pgvector caps indexable vectors at 2000 dims). Passthrough loads must name the model Transform actually used via embed_model; documented.
The store is now a package: a VectorStore contract (space pin, auto-provision, idempotent upsert, cosine scores, connector-shaped writes) with one implementation per backend, selected by URAG_STORE_BACKEND. - chroma (default): unchanged behavior, moved to stores/chroma.py - qdrant: embedded local mode or a server; space pins live in a reserved meta collection since Qdrant collections carry no metadata - pgvector: psycopg 3; a table per collection with a JSONB metadata column and a cosine HNSW index when the dimension is indexable; pins live in a rag_ingest_spaces table A conformance pytest suite (test/unit/mcp) runs identically against every backend and defines what supported means; a server-level smoke suite drives both embedded backends through FastMCP in-memory.
An overview of the local agentic RAG loop (Transform MCP + this server) plus one self-contained guide each for Chroma, Qdrant, and pgvector: setup, docker-compose where a service is needed, the agent workflow, CLI inspection of each database, and when the DB vendor's own MCP server does or does not pair safely with a corpus built here.
The wire-it-up section now links each backend's getting-started guide and its ready-to-copy host config instead of a single Chroma-only example; the standalone example_mcp_config.json is superseded by the per-backend examples/*/mcp-config.json files.
The unit-test CI environment installs only locked base dependencies, so fastmcp, chromadb, and qdrant_client are not importable there. Gate the server suite and each conformance backend with pytest.importorskip — the repo's existing pattern for optional connector dependencies.
CodesLikeIcarus
changed the title
CodesLikeIcarus
changed the title
Security reviewNice, well-structured PR — the space-pinning invariant, the streaming byte cap, deterministic idempotent upserts, and the parameterized pgvector SQL are all carefully done. A few security notes, one worth addressing before merge. 1. SSRF: the out-of-band fetch has no destination guard (please address)
with httpx.stream("GET", url, timeout=_FETCH_TIMEOUT_S, follow_redirects=True) as resp:There's no block on private / loopback / link-local / cloud-metadata address ranges, and Because Suggestion: resolve the hostname once, reject private/loopback/link-local/metadata ranges, connect pinned to the validated IP (preserving Host + TLS SNI), and re-validate on every redirect hop. 2. Arbitrary local file / directory read via tool args
3. pgvector SQL — clean ✅ (one non-security footgun)The SQL construction is solid: strict identifier regex ( Footgun (not security): 4. Note: stored
|
The pgvector VectorStore hand-rolled a narrow table of its own (id text, record_id, text, metadata jsonb, embedding) instead of the Unstructured-compliant schema the SQL connector and downstream readers (e.g. Dragon Lab's pgvector index) expect. A reader pointed at a rag-ingest-loaded table therefore found none of its expected columns (embeddings, type, element_id, …), so the two could not share a table. Rewrite ensure_space/write/search to: - adopt a pre-created table — verify the vector dimension and record the space pin without recreating it — or, standalone, auto-provision an equivalent id/type/record_id/element_id/text/embeddings schema with a cosine HNSW index; - fit staged rows to the table's actual columns (union across rows); - cast the vector to the column's own vector/halfvec type, which the generic SQLUploader does not (it adapts the list to a Postgres array). Chroma and Qdrant stores already reuse their connector stagers and build connector-shaped writes, so they are unchanged. The backend conformance suite (test/unit/mcp/test_stores.py) passes for all three. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2 participants
What
A local STDIO MCP server (
unstructured_ingest/mcp/, run aspython -m unstructured_ingest.mcp) that turns Transform MCP output into a queryable local RAG corpus. It pairs with the remote Transform MCP: Transform parses/chunks/(optionally) embeds and returns Element JSON; this server fetches that JSON out of band (it never enters the agent's context), embeds it locally iff Transform didn't, upserts it into a local vector store, and serves matched-space similarity search.Three tools:
load_transform_output(URLs, file paths, or directories;embed_policyauto/passthrough/local),search,list_collections.Design
(provider, model, dimension)on first load; loads with a different model are refused (SpaceMismatch), and every query is embedded into the pinned space. Corpus/query symmetry is structural, not a discipline — mixing embedding models silently ruins retrieval, so the server makes it impossible.VectorStorecontract (stores/), selected byURAG_STORE_BACKEND:chroma(default) — embedded, zero-config; pin in collection metadataqdrant— embedded local mode or a server (URAG_QDRANT_URL); pin in a reserved meta collectionpgvector— psycopg 3; auto-provisions the table + cosine HNSW index (≤2000 dims) on first load; pins in arag_ingest_spacestableChromaUploadStager,QdrantUploadStager,SQLUploadStager), so what lands in each store matches what the ingest pipeline's own connector would produce, and element ids are deterministic (re-loading upserts, never duplicates).Testing
test/unit/mcp/test_stores.py) runs identically against every backend and is the working definition of "supported backend": load → cosine ranking → space-guard refusal → idempotent re-load → metadata round-trip. Chroma and Qdrant run embedded with no services; pgvector runs againstURAG_TEST_PG_DSN(verified vs a dockerizedpgvector/pgvector:pg17).test/unit/mcp/test_server.py) drives the FastMCP server in-memory over both embedded backends: load/list, directory expansion, per-source space-guard errors.chunk_by_characterjob →download_url→load_transform_output(local embed,text-embedding-3-small/1536) →search. Three real files (PDF/MD/HTML, 92 chunks); three topical queries each returned the expected document as the top hit with identical scores across backends.Docs
unstructured_ingest/mcp/README.mdplusexamples/with a getting-started guide per backend: setup, docker-compose where a service is needed, the agentic loop with Transform MCP, CLI inspection, and when (not) to pair the DB vendor's own MCP server with a corpus built here.🤖 Generated with Claude Code