Skip to content

DEVELOPMENT.md

Latest commit

 

History

History
179 lines (136 loc) · 5.65 KB

DEVELOPMENT.md

File metadata and controls

179 lines (136 loc) · 5.65 KB

ChatCrystal Development Guide

English | 简体中文

This guide covers repository structure, architecture, development commands, testing, and release workflows.

Project Overview

ChatCrystal is a local-first AI conversation crystallization tool. It imports conversations from AI coding tools, generates structured notes with LLMs, builds embeddings for semantic search, and exposes both UI and MCP workflows.

Monorepo Layout

ChatCrystal/
├── shared/                  # Shared TypeScript types
├── server/                  # Fastify backend, CLI, MCP server
├── client/                  # React SPA
├── electron/                # Electron main and preload processes
├── skills/                  # Publishable ChatCrystal agent skills
├── docs/                    # Maintainer and user documentation
├── scripts/                 # Release and utility scripts
└── site/                    # Project website

Tech Stack

Layer Technology
Backend Node.js, Fastify v5, TypeScript
Frontend Vite v8, React 19, Tailwind CSS v4, TanStack React Query v5
Desktop Electron, electron-builder
Database sql.js WASM SQLite
LLM Vercel AI SDK v6
Embeddings vectra local vector index
Queue p-queue
File watching chokidar

Development Commands

npm run dev                   # Server 3721 + client 13721
npm run build                 # Build server and client
npm start                     # Production server
npm run lint                  # Biome + client ESLint
npm run lint:fix              # Apply safe lint fixes
npm run test -w server        # Server tests
npm run dev:electron          # Electron dev mode
npm run build:electron        # Build Windows installer
npm run pack:electron         # Build unpacked Electron app
npm run eval:experience -w server

npm run eval:experience -w server runs the offline calibration suite for the experience quality gate.

Runtime Data

Runtime data is stored in config.json and chatcrystal.db under the active data directory.

Default data directory:

  • CLI, MCP, npm package, repository checkout, and Electron: ~/.chatcrystal/data
  • Explicit override: DATA_DIR

Electron sets ELECTRON=true, DATA_DIR, and ELECTRON_PACKAGED when applicable.

Data Flow

AI tool conversation files
  -> SourceAdapter scan/parse
  -> Import service deduplication
  -> SQLite conversations/messages
  -> Summarization queue
  -> LLM structured note generation
  -> Embedding generation
  -> vectra semantic index
  -> REST API, UI, CLI, MCP

Summarization Pipeline

ChatCrystal uses turn-based transcript preparation before summarization:

  1. Split messages into user-assistant turns.
  2. Keep the user instruction plus the first and last substantial assistant replies in each turn.
  3. Score turns by instruction length and assistant engagement.
  4. Always include the first turn and final turns.
  5. Fill the remaining budget with high-value middle turns.
  6. Compress skipped turns into one-line previews.

Structured output uses Vercel AI SDK generateObject() with Zod schemas. This avoids fragile JSON extraction and lets schema validation retry invalid model output.

Source Adapters

Add a new source by implementing SourceAdapter:

interface SourceAdapter {
  name: string;
  displayName: string;
  detect(): Promise<SourceInfo | null>;
  scan(): Promise<ConversationMeta[]>;
  parse(meta: ConversationMeta): Promise<ParsedConversation>;
}

Built-in adapters:

Adapter Data Source Format
claude-code ~/.claude/projects/**/*.jsonl JSONL conversation log
codex ~/.codex/sessions/**/rollout-*.jsonl JSONL event stream
cursor Cursor workspaceStorage/state.vscdb SQLite KV store
trae Trae workspaceStorage/state.vscdb SQLite KV store
copilot VS Code workspaceStorage/chatSessions/*.jsonl JSONL snapshots

Create the adapter under server/src/parser/adapters/ and register it in server/src/parser/index.ts.

API Surface

Key REST endpoints:

Method Path Description
GET /api/status Server status and statistics
GET /api/config Current config with secrets redacted
POST /api/config Update provider config
POST /api/import/scan Trigger import
GET /api/conversations List conversations
GET /api/conversations/:id Conversation detail
POST /api/conversations/:id/summarize Summarize one conversation
POST /api/summarize/batch Batch summarization
GET /api/notes List notes
GET /api/notes/:id Note detail
GET /api/search?q=...&expand=true Semantic search
GET /api/relations/graph Knowledge graph data
GET /api/queue/status Queue status

Knowledge Graph

The relation system supports these relation types:

Relation Meaning
CAUSED_BY Causation
LEADS_TO Leads to
RESOLVED_BY Resolved by
SIMILAR_TO Similar topic
CONTRADICTS Contradiction
DEPENDS_ON Dependency
EXTENDS Extension
REFERENCES Reference

Relations can be discovered by LLM, added manually, or followed during semantic search expansion.

Testing

Primary verification:

npm run test -w server
npm run build
npm run lint
npm run eval:experience -w server

Use focused server tests while iterating, then run the full commands before committing.

Release

npm run release
npm run release -- minor
npm run release -- major
npm run release -- 1.0.0

The release script bumps version, creates a git tag, and pushes so CI can build and publish artifacts.