Skip to content

README.md

Latest commit

 

History

History
233 lines (169 loc) · 10.4 KB

README.md

File metadata and controls

233 lines (169 loc) · 10.4 KB
Raw

Knowledge Wiki

English · 한국어 · 日本語 · 简体中文

WikiBrain: a young researcher and a friendly brain mascot explore a linked knowledge map

A local-first, Markdown-native knowledge system with reproducible research and local semantic retrieval.

Why · Architecture · Quick start · Knowledge model · Development

Status: early but working. The authoritative knowledge base is ordinary OKF-flavored Markdown. PostgreSQL and pgvector are derived, replaceable search infrastructure.

Why this exists

Most personal knowledge tools make the database, a proprietary editor, or an embedding vendor the source of truth. This project makes the durable artifact plain Markdown instead:

  • Own the corpus. Keep concepts, comparisons, and source captures under local data/wiki/; the repository intentionally ships no personal knowledge documents.
  • Keep research auditable. Captured sources retain their URL, UTC timestamp, and SHA-256 provenance; interpretations link back to evidence.
  • Search locally. The API runs intfloat/multilingual-e5-small locally and stores 384-dimensional vectors in PostgreSQL/pgvector—no paid LLM or embedding API is required.
  • Treat indexes as disposable. Restarting the API rebuilds the catalog from Markdown; the database is never the only copy of knowledge.
  • Curate rather than merely collect. The current corpus explores self-hosted feature configuration, remote configuration, OpenFeature, LINE/NAVER-adjacent projects, Spring/JVM, and Python ecosystems.

Architecture

OKF Markdown ──> FastAPI cataloger ──> PostgreSQL + pgvector ──> React/Vite browser
     │                   │                       │
     │                   └── local multilingual embeddings (384d)
     └── source provenance, internal links, human-readable Git history
Layer Role Main technology
data/wiki/ Canonical knowledge bundle Markdown + YAML frontmatter / OKF v0.1 conventions
server/ Catalog projection, URL capture, semantic retrieval FastAPI, Psycopg, Sentence Transformers
infra/postgres/ Search schema and vector index PostgreSQL 16 + pgvector
web/ Browser for documents, graph, and search React, TypeScript, Vite
docker-compose.yml Local development/runtime topology Docker Compose

Retrieval model

The initial retrieval path is deliberately simple and local:

  1. Parse Markdown files with non-empty type frontmatter.
  2. Prefix documents with passage: and embed using multilingual-e5-small.
  3. Persist a 384-dimensional vector projection in pgvector with cosine HNSW search.
  4. Prefix queries with query: and return the most relevant documents.

Markdown remains canonical; deleting the derived database does not lose knowledge.

Quick start

Prerequisites

  • Docker Desktop / Docker Compose
  • Optional for local development: uv and Node.js
git clone https://github.com/hungrytech/knowledge-wiki.git
cd knowledge-wiki

test -f .env || cp .env.example .env
test -f .env.api || cp .env.api.example .env.api
docker compose up -d --build
docker compose ps
curl -fsS http://localhost:8000/api/health

The example credentials are for local use only. Before a shared or non-local deployment, replace them with a strong password and keep the password portion of NEO4J_AUTH in .env equal to NEO4J_PASSWORD in .env.api:

NEO4J_AUTH=neo4j/replace-with-a-strong-password
NEO4J_PASSWORD=replace-with-a-strong-password

Open:

The first API startup downloads the public multilingual embedding model and persists it in the Docker knowledge-models volume. Later starts reuse it.

This repository deliberately contains no personal corpus. Create your local data/wiki/ documents after installation; they are ignored by Git so they stay on your machine.

Verify semantic search

curl -fsS --get http://localhost:8000/api/semantic-search \
  --data-urlencode 'q=self hosted feature flags remote configuration' \
  --data-urlencode 'limit=5'

Add knowledge manually

Create a Markdown document beneath data/wiki/ with YAML frontmatter. Concept, Comparison, Project, and Source are useful document types.

---
type: Concept
title: Example concept
description: A concise explanation of the concept.
tags: [example]
---

# What it is

Write the durable explanation here.

Then refresh the derived catalog:

docker compose restart api

Knowledge model

data/wiki/
├── index.md                   # human navigation
├── log.md                     # curation history
├── concepts/                  # durable explanations
├── comparisons/               # decision-oriented analyses
├── projects/                  # systems and implementations
└── sources/                   # immutable captured primary material

Evidence boundary

A Source document preserves external material and provenance. A Concept or Comparison document adds interpretation. Keeping these separate makes claims reviewable and lets the corpus evolve without silently rewriting source evidence.

Typed context relations

Durable context can declare directed, typed relations in frontmatter. The compact form matches the corpus's existing convention; the structured form adds edge-level provenance or time metadata:

relations:
  - "supported-by [[sources/primary-evidence]]"
  - kind: supersedes
    target: concepts/previous-decision
    context: ADR-0042
    timestamp: 2026-07-26T10:30:00Z

Internal Markdown links are projected as links-to; frontmatter relations keep their explicit kind. The UI's Context graph shows incoming and outgoing edges around the selected document. Neo4j remains a rebuildable projection—the Markdown relation is canonical.

How context becomes durable knowledge

A graph edge does not make a note trustworthy. WikiBrain separates capture, interpretation, and action so a temporary observation does not silently become a durable fact.

Stage What it means Typical document or relation
0. Intake A conversation, URL, handoff, or observation has not been accepted as knowledge yet Outside the canonical bundle until a curator captures it
1. Evidence The original material is captured with its URL, time, and hash Source
2. Understanding A claim is distilled, checked, and connected to its evidence Concept or Comparison with supported-by
3. Use Reviewed knowledge informs an implementation or durable decision Project, or a decision-focused Concept, linked with extends, adopts, or another explicit kind
4. Revision New evidence changes an older conclusion without rewriting history A new document with supersedes; the old document remains traceable

Promotion is review-driven today. A curator checks provenance, removes duplicates, separates fact from interpretation, and writes the relation explicitly. The graph records why the durable document exists and what it replaced.

flowchart LR
    A[Conversation, URL,<br/>handoff, or observation] -->|curator captures| B[Source<br/>immutable evidence]
    B --> C{Provenance present?<br/>Relevant and readable?}
    C -- No --> X[Keep outside the durable corpus<br/>or request better evidence]
    C -- Yes --> D[Concept or Comparison<br/>distilled understanding]
    D -->|supported-by| B
    D --> E{Reviewed and useful<br/>for future work?}
    E -- Not yet --> D
    E -- Yes --> F[Project or durable<br/>decision Concept]
    F -->|extends / adopts| D
    G[New evidence] --> H[New interpretation]
    H -->|supersedes| D
    H --> F
Loading

What is implemented now

WikiBrain parses and preserves typed relations, returns them from /api/graph, projects them into Neo4j when enabled, and renders incoming and outgoing context in the UI. Markdown remains canonical; PostgreSQL and Neo4j can be rebuilt.

WikiBrain does not currently ingest chat sessions automatically, run a retention clock, calculate a popularity score, infer relation kinds, or provide a brainctl remember command. Automatic promotion may be added later, but it must retain provenance, expose conflicts, and require a reviewable policy before it can change canonical Markdown.

Development

# Python API tests
uv run --project server pytest server/tests -q

# Web tests and production build
npm --prefix web test -- --run
npm --prefix web run build

After editing the knowledge bundle, verify that every projected document has an embedding:

docker compose exec -T postgres psql -U knowledge -d knowledge_wiki -Atc \
  "SELECT count(*) FILTER (WHERE embedding IS NOT NULL) || '/' || count(*) FROM documents;"

Privacy and security notes

  • This project is designed for local use. Do not expose the API, database port, or Docker Compose credentials directly to the public internet.
  • The Compose PostgreSQL password is a development-only placeholder. Replace it and use managed secret handling before any shared or production deployment.
  • URL ingestion connects only to validated public IPv4 targets. IPv6-only/NAT64 targets, compressed responses, responses over 5 MiB, and fetches exceeding the 20-second total deadline are rejected. Preserve sources and inspect fetched content before trusting it.
  • Knowledge may contain personal notes or licensed material. Review data/wiki/ before making a fork or derivative repository public.

Roadmap

  • Persist heading-aware chunks for more precise retrieval
  • Add PostgreSQL full-text search and reciprocal-rank-fusion hybrid retrieval
  • Improve graph navigation and provenance views
  • Add import/export and corpus-validation tooling
  • Expand primary-source Spring/JVM OSS curation

License

This project is released under the MIT License.

Built around a simple conviction: your knowledge should outlive the tool that indexes it.