Local-first RAG system for searching, understanding, and organizing academic literature
6 RAG modes · Unified content pipeline · Hybrid vector+BM25 retrieval · MCP server · REST API
Perspicacité (French for "insight") helps scientists, researchers, and students search, understand, and organize academic literature using AI grounded in real research papers. It works entirely on your machine — only LLM inference calls leave your environment.
- docs/index.md — documentation home with links to all sections
- docs/VISION.md — framework vision, design philosophy, architecture overview
- docs/getting-started.md — install, configure, first KB, first question
- Multi-database search — Semantic Scholar, OpenAlex, PubMed, arXiv, HAL, DBLP via SciLEx
- Unified content pipeline — PMC JATS XML, arXiv HTML, OA PDFs, publisher APIs, and institutional-access via browser-cookie replay; quality-priority routing
- 6 RAG modes — Basic, Advanced, Profound, Agentic, Literature Survey, Contradiction; per-stage LLM tiering (Haiku routing/screening, Sonnet synthesis)
- Knowledge base management — BibTeX import, DOI bulk-add, local document ingest, Zotero-collection import; async ingestion with SSE progress streaming
- Citation-graph expansion — forward + backward snowball over OpenAlex; automatic Semantic Scholar fallback for arXiv-seeded papers (see docs/concepts/citation-graph.md)
- Honest sourcing —
PaperSourceenum records the true origin of every paper (OPENALEX,PUBMED,ARXIV,CROSSREF,SEMANTIC_SCHOLAR,BIBTEX,LOCAL) — no genericWEB_SEARCHcatch-all (see docs/reference/paper-source-enum.md) - MCP server — 32 tools at
/mcpfor integration with Mimosa-AI, Claude Code, SmolAgents, Codex - REST API — full JSON API with SSE streaming for async jobs
- Provenance tracking — per-answer retrieval trace stored in SQLite; exportable as RO-Crate 1.1 zip
- Auto KB routing —
kb_name: "auto"scores all KBs against your query (BM25 or LLM) and fans across the top-N in parallel - Capsule enrichment — per-paper figures, references, code snippets, and supplementary files indexed alongside main text
- Long-term preservation — PDF byte cache, Zotero attachment push (4-step file-upload protocol), BibTeX + PDF folder export, Obsidian vault export
- Flexible LLM routing — direct API (Anthropic with prompt caching, OpenAI, DeepSeek, Gemini), Ollama (local), agent-CLI subprocess (Claude Code, Codex, OpenClaw, Hermes), MCP sampling
- Local-first — data stays on your machine; no telemetry
- Python 3.12+
- uv (recommended) or pip
git clone https://github.com/HolobiomicsLab/Perspicacite-AI.git
cd Perspicacite-AI
uv synccp config.example.yml config.yml
cp .env.example .env
# Edit .env — add at least one LLM API key| Provider | Key variable | Get a key at |
|---|---|---|
| DeepSeek (default) | DEEPSEEK_API_KEY |
platform.deepseek.com |
| OpenAI | OPENAI_API_KEY |
platform.openai.com |
| Anthropic | ANTHROPIC_API_KEY |
console.anthropic.com |
Also set pdf_download.unpaywall_email in config.yml for open-access PDF discovery.
./dev.shdev.sh starts both processes with a single command and stops both on Ctrl+C:
:8000— Python/FastAPI backend (REST API + MCP server):3000— Next.js frontend ← open this in your browser
Frontend dependencies (node_modules) are installed automatically on the first
run; subsequent starts skip that step.
Heads-up — the backend is slow to start. It loads ML models (PyTorch, sentence-transformers) on boot, so expect roughly a minute before it is ready (longer on the very first run). On Windows-mounted / WSL filesystems it is slower still — keeping the repo on a native Linux filesystem speeds startup up substantially. The UI shows connection errors until the backend finishes booting; just wait for it.
http://localhost:3000 is the main UI: streaming chat, CNRS visual identity,
configurable database picker, KB management, and the similarity-expansion
workflow. It proxies /api/* to the backend at :8000 (overridable via
PERSPICACITE_BACKEND_URL). Use this for all everyday research.
If you need to start them in separate terminals (e.g. to attach a debugger):
# Terminal 1 — backend: REST API on :8000, MCP at /mcp
uv run perspicacite -c config.yml serve
# Terminal 2 — frontend: http://localhost:3000
cd frontend
npm install # first time only; dev.sh handles this automatically
npm run dev
# To point the frontend at a different backend (e.g. the OpenAI-embedding port):
PERSPICACITE_BACKEND_URL=http://localhost:8002 npm run devhttp://localhost:8000 still serves a Jinja-rendered fallback UI that requires
no Node.js. It is deprecated — new features are only added to the Next.js
frontend — and will be removed in a future release. Use it only as a last resort
when Node.js is unavailable. The historical pre-fork snapshot is preserved on the
backup/gui-vanilla-jinja branch.
Google Scholar is opt-in because it relies on either the SerpApi
managed scraper (recommended, requires SERPAPI_API_KEY; free tier ~100/mo)
or headless Chromium via Playwright.
To enable headless Chromium scraping:
uv sync --extra browser # installs the playwright Python package
uv run playwright install chromium # downloads the browser binary (~150 MB)Then in config.yml:
search:
enabled_providers:
# ...
- google_scholar
google_scholar:
enabled: true
headless: trueWithout both steps the server logs google_scholar_playwright_missing and
Scholar silently returns zero results. SerpApi-only setups can skip the
uv sync --extra browser and playwright install commands — set
SERPAPI_API_KEY instead and Scholar will route through SerpApi.
Keys can live in either the shell environment (
export ANTHROPIC_API_KEY=...) or in.envat the project root — the CLI loads.envon startup. Shell vars take precedence over.env. For offline / mocked-LLM development, bypass the preflight withPERSPICACITE_ALLOW_MISSING_LLM_KEYS=1.
The MCP server is also accessible at http://localhost:8000/mcp (streamable HTTP). MCP clients connect there to call tools like search_knowledge_base, generate_report, and ingest_asb_run. See docs/MCP.md for the envelope contract and per-tool latency expectations.
# From a BibTeX file
uv run perspicacite -c config.yml create-kb my-kb --from-bibtex refs.bib
# Or from a live literature search (requires SciLEx extra)
uv pip install -e ".[scilex]"
uv run perspicacite -c config.yml search-to-kb --query "diamond magnetometry" --kb sensors --max-results 20Full walkthrough: docs/getting-started.md.
| Section | What it covers |
|---|---|
| docs/concepts/ | Knowledge bases, RAG modes, capsules, provenance, citation graph |
| docs/guides/ | BibTeX import, search-to-KB, citation expansion, Zotero, Obsidian, institutional PDFs |
| docs/reference/ | CLI flags, REST endpoints, MCP tools, config schema, PaperSource enum |
| docs/development/ | Contributing, architecture tour, testing, design workflow |
Six modes covering different cost/depth trade-offs. See docs/concepts/rag-modes.md for when to use each.
| Mode | Best for | Speed |
|---|---|---|
| Basic | Quick answers from a well-curated KB | Fast |
| Advanced | Broader KB search with query expansion and WRRF fusion | Medium |
| Profound | Complex multi-faceted questions (3 retrieval cycles) | Slower |
| Agentic | Questions requiring live discovery beyond the KB | Variable |
| Literature Survey | Mapping a research field (checkpoint/resume supported) | Slowest |
| Contradiction | Comparing conflicting findings across papers | Medium |
32 tools at http://localhost:8000/mcp. Key tools:
search_literature · get_paper_content · search_knowledge_base · generate_report · add_dois_to_kb · build_kb_from_search · expand_kb_via_citations · push_to_zotero · build_capsules_for_kb · export_kb
Full tool catalog: docs/reference/mcp-tools.md.
For use with Mimosa-AI, Claude Code, SmolAgents, or any MCP-compatible agent, point at http://localhost:8000/mcp (streamable-HTTP transport).
Full JSON API with SSE streaming. Key endpoints:
POST /api/chat · GET /api/kb · POST /api/kb/{name}/bibtex/async · POST /api/kb/{name}/dois/async · GET /api/jobs/{id}/events · GET /api/conversations/{id}/export
Full endpoint list: docs/reference/rest-api.md.
perspicacite serve # Start server (web UI + MCP)
perspicacite create-kb NAME # Create KB (from BibTeX with --from-bibtex)
perspicacite add-to-kb NAME # Add papers to existing KB
perspicacite list-kb # List all KBs
perspicacite query QUESTION # Ask a question against a KB
perspicacite search-to-kb # Build KB from a literature search
perspicacite expand-kb # Grow KB via citation graph
perspicacite export-kb # Export KB (BibTeX / Obsidian vault)
perspicacite screen-papers # Score candidates by relevance (no server needed)
perspicacite pubmed-search # Search PubMed to BibTeX (no server needed)
perspicacite import-browser-cookies # Export session cookies for institutional PDF access
perspicacite check-cookies # Check cookie freshness
perspicacite build-capsule # Build per-paper capsule (figures, SI, code)
perspicacite build-capsules # Build capsules for all papers in a KB
perspicacite version # Show installed versionFull flags and usage: docs/reference/cli.md.
Copy config.example.yml and edit. Key sections:
llm:
default_provider: "deepseek"
default_model: "deepseek-chat"
knowledge_base:
embedding_model: "text-embedding-3-small"
chunk_size: 1000
pdf_download:
unpaywall_email: "your@email.com"
mcp:
enabled: trueFull schema: docs/reference/config.md.
Alternative config presets: config.ollama.example.yml (local Ollama),
config.claude_code.example.yml (Claude Code CLI), config.codex.example.yml,
config.hermes.example.yml, config.openclaw.example.yml.
- Your data stays local — KBs stored in ChromaDB and SQLite on your machine
- API calls only — queries sent to your configured LLM provider; no other data leaves your environment
- No tracking — no usage analytics
See CONTRIBUTING.md for the CLA workflow and contribution guidelines.
Dev-loop setup, test structure, and architecture: docs/development/.
Apache License 2.0 — see LICENSE and NOTICE.
Citation: An AI Pipeline for Scientific Literacy and Discovery: a Demonstration of Perspicacité-AI Integration with Knowledge Graphs
L. Pradi, T. Jiang, M. Feraud, M. Bekbergenova, Y. Taghzouti, L.-F. Nothias — ISWC-C 2025
@inproceedings{pradi2025perspicacite,
title = {An AI Pipeline for Scientific Literacy and Discovery: a Demonstration of Perspicacit\'{e}-AI Integration with Knowledge Graphs},
author = {Pradi, Lucas and Jiang, Tao and Feraud, Matthieu and Bekbergenova, Madina and Taghzouti, Yousouf and Nothias, Louis-Felix},
booktitle = {ISWC-C 2025},
year = {2025}
}@softwareversion{scilex2026,
title = {SciLEx, Science Literature Exploration Toolkit},
author = {Ringwald, C\'{e}lian and Navet, Benjamin},
url = {https://github.com/Wimmics/SciLEx},
year = {2026}
}- SciLEx — literature exploration toolkit powering multi-database search
- ChromaDB — local vector storage
- OpenAlex and Semantic Scholar — academic search and citation graphs
- Unpaywall — open access discovery