Merge branch 'main' into add-platform-builds

Author: Gioee

.gitignore

@@ -47,12 +47,7 @@ yarn-error.log*
 
 # System
 .DS_Store
-Thumbs.db
-CLAUDE.md
-
-# Dart/Flutter
-.dart_tool/
-pubspec.lock
-.flutter-plugins
-.flutter-plugins-dependencies
-packages/flutter/native_libraries/
+test/unittest.dSYM/Contents/Info.plist
+test/unittest.dSYM/Contents/Resources/DWARF/unittest
+test/unittest.dSYM/Contents/Resources/Relocations/aarch64/unittest.yml
+/build

API.md

@@ -5,6 +5,7 @@ A SQLite extension that provides semantic memory capabilities with hybrid search
 ## Table of Contents
 
 - [Overview](#overview)
+- [Sync Behavior](#sync-behavior)
 - [Loading the Extension](#loading-the-extension)
 - [SQL Functions](#sql-functions)
   - [General Functions](#general-functions)
@@ -29,6 +30,31 @@ sqlite-memory enables semantic search over text content stored in SQLite. It:
 
 ---
 
+## Sync Behavior
+
+All `memory_sync_*` functions use **content-hash change detection** to avoid redundant embedding computation. Each piece of content is hashed before processing — if the hash already exists in the database, the content is skipped.
+
+### Change Detection
+
+| Scenario | Behavior |
+|----------|----------|
+| New content | Chunked, embedded, and indexed |
+| Unchanged content | Skipped (hash match) |
+| Modified file | Old entry atomically deleted, new content reindexed |
+| Deleted file | Entry removed during directory sync |
+
+### Transactional Safety
+
+Every sync operation is wrapped in a SQLite **SAVEPOINT** transaction. If any step fails (embedding error, disk issue, constraint violation), the entire operation rolls back. This guarantees:
+
+- **No partially-indexed files** — content is either fully indexed or not at all
+- **No orphaned chunks** — embeddings and FTS entries are always consistent with `dbmem_content`
+- **Safe to retry** — a failed sync leaves the database in its previous valid state
+
+This makes all sync functions idempotent and safe to call repeatedly (e.g., on a schedule or at application startup).
+
+---
+
 ## Loading the Extension
 
 ### Dynamic Loading (Recommended)
@@ -174,9 +200,9 @@ SELECT memory_get_option('provider');
 
 ### Memory Management Functions
 
-#### `memory_add_text(content TEXT [, context TEXT])`
+#### `memory_sync_text(content TEXT [, context TEXT])`
 
-Adds text content to memory.
+Syncs text content to memory. Duplicate content (same hash) is skipped automatically.
 
 **Parameters:**
 | Parameter | Type | Required | Description |
@@ -189,23 +215,24 @@ Adds text content to memory.
 **Notes:**
 - Content is chunked based on `max_tokens` and `overlay_tokens` settings
 - Each chunk is embedded and stored in `dbmem_vault`
-- Content hash prevents duplicate storage
+- Content hash prevents duplicate storage — calling with the same content is a no-op
+- Runs inside a SAVEPOINT transaction (see [Sync Behavior](#sync-behavior))
 - Sets `created_at` timestamp automatically
 
 **Example:**
 ```sql
 -- Add text without context
-SELECT memory_add_text('SQLite is a C-language library that implements a small, fast, self-contained SQL database engine.');
+SELECT memory_sync_text('SQLite is a C-language library that implements a small, fast, self-contained SQL database engine.');
 
 -- Add text with context
-SELECT memory_add_text('Important meeting notes from 2024-01-15...', 'meetings');
+SELECT memory_sync_text('Important meeting notes from 2024-01-15...', 'meetings');
 ```
 
 ---
 
-#### `memory_add_file(path TEXT [, context TEXT])`
+#### `memory_sync_file(path TEXT [, context TEXT])`
 
-Adds a file to memory.
+Syncs a file to memory. Unchanged files are skipped; modified files are atomically replaced.
 
 **Parameters:**
 | Parameter | Type | Required | Description |
@@ -218,39 +245,51 @@ Adds a file to memory.
 **Notes:**
 - Only processes files matching configured extensions (default: `md,mdx`)
 - File path is stored in `dbmem_content.path`
+- If the file was previously indexed with different content, the old entry (chunks, embeddings, FTS) is deleted and new content is reindexed — all within a single SAVEPOINT transaction (see [Sync Behavior](#sync-behavior))
 - Not available when compiled with `DBMEM_OMIT_IO`
 
 **Example:**
 ```sql
-SELECT memory_add_file('/docs/readme.md');
-SELECT memory_add_file('/docs/api.md', 'documentation');
+SELECT memory_sync_file('/docs/readme.md');
+SELECT memory_sync_file('/docs/api.md', 'documentation');
 ```
 
 ---
 
-#### `memory_add_directory(path TEXT [, context TEXT])`
+#### `memory_sync_directory(path TEXT [, context TEXT])`
 
-Recursively adds all matching files from a directory.
+Synchronizes a directory with memory. Adds new files, reindexes modified files, and removes entries for deleted files.
 
 **Parameters:**
 | Parameter | Type | Required | Description |
 |-----------|------|----------|-------------|
 | `path` | TEXT | Yes | Full path to the directory |
 | `context` | TEXT | No | Optional context label applied to all files |
 
-**Returns:** INTEGER - Number of files processed
+**Returns:** INTEGER - Number of new files processed
 
 **Notes:**
 - Recursively scans subdirectories
 - Only processes files matching configured extensions
+- **Phase 1 — Cleanup**: Removes entries for files that no longer exist on disk
+- **Phase 2 — Scan**: Processes all matching files:
+  - **New files** are chunked, embedded, and added to the index
+  - **Unchanged files** are skipped (content hash match)
+  - **Modified files** have their old entries atomically replaced with new content
+- Each file is processed inside its own SAVEPOINT transaction (see [Sync Behavior](#sync-behavior))
+- Safe to call repeatedly — only changed content triggers embedding computation
 - Not available when compiled with `DBMEM_OMIT_IO`
 
 **Example:**
 ```sql
-SELECT memory_add_directory('/path/to/docs');
--- Returns: 42 (number of files added)
+SELECT memory_sync_directory('/path/to/docs');
+-- Returns: 42 (number of new files processed)
+
+SELECT memory_sync_directory('/project/notes', 'project-notes');
 
-SELECT memory_add_directory('/project/notes', 'project-notes');
+-- Safe to call again — unchanged files are skipped
+SELECT memory_sync_directory('/path/to/docs');
+-- Returns: 0 (nothing changed)
 ```
 
 ---
@@ -319,6 +358,7 @@ Deletes all memories from the database.
 **Notes:**
 - Clears `dbmem_content`, `dbmem_vault`, and `dbmem_vault_fts`
 - Does not delete settings from `dbmem_settings`
+- Does not clear the embedding cache (`dbmem_cache`)
 - Uses SAVEPOINT transaction for atomicity
 
 **Example:**
@@ -328,6 +368,35 @@ SELECT memory_clear();
 
 ---
 
+#### `memory_cache_clear([provider TEXT, model TEXT])`
+
+Clears the embedding cache.
+
+**Parameters:**
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `provider` | TEXT | No | Provider name to clear cache for |
+| `model` | TEXT | No | Model name to clear cache for |
+
+**Returns:** INTEGER - Number of cache entries deleted
+
+**Notes:**
+- With 0 arguments: clears the entire embedding cache
+- With 2 arguments: clears cache entries for a specific provider/model combination
+- The embedding cache stores computed embeddings keyed by (text hash, provider, model) to avoid redundant computation
+- Safe to call at any time — does not affect stored memories
+
+**Example:**
+```sql
+-- Clear entire cache
+SELECT memory_cache_clear();
+
+-- Clear cache for a specific provider/model
+SELECT memory_cache_clear('openai', 'text-embedding-3-small');
+```
+
+---
+
 ### `memory_search`
 
 A virtual table for performing hybrid semantic search.
@@ -395,6 +464,9 @@ AND context = 'meetings';
 | `text_weight` | REAL | 0.5 | Weight for FTS in scoring |
 | `min_score` | REAL | 0.7 | Minimum score threshold for results |
 | `update_access` | INTEGER | 1 | Update last_accessed on search |
+| `embedding_cache` | INTEGER | 1 | Cache embeddings to avoid redundant computation |
+| `cache_max_entries` | INTEGER | 0 | Max cache entries (0 = no limit). When exceeded, oldest entries are evicted |
+| `search_oversample` | INTEGER | 0 | Search oversampling multiplier (0 = no oversampling). When set, retrieves N * multiplier candidates from each index before merging down to N final results |
 
 ---
 
@@ -404,7 +476,7 @@ The extension tracks two timestamps for each memory:
 
 ### `created_at`
 
-- Set automatically when content is added via `memory_add_text`, `memory_add_file`, or `memory_add_directory`
+- Set automatically when content is added via `memory_sync_text`, `memory_sync_file`, or `memory_sync_directory`
 - Stored as Unix timestamp (seconds since 1970-01-01 00:00:00 UTC)
 - Never updated after initial creation
 
@@ -445,8 +517,8 @@ SELECT memory_set_option('max_tokens', 512);
 SELECT memory_set_option('min_score', 0.75);
 
 -- Add content
-SELECT memory_add_text('SQLite is a C library that provides a lightweight disk-based database.', 'sqlite-docs');
-SELECT memory_add_directory('/docs/sqlite', 'sqlite-docs');
+SELECT memory_sync_text('SQLite is a C library that provides a lightweight disk-based database.', 'sqlite-docs');
+SELECT memory_sync_directory('/docs/sqlite', 'sqlite-docs');
 
 -- Search
 SELECT path, snippet, ranking
@@ -474,9 +546,9 @@ SELECT memory_clear();
 
 ```sql
 -- Add memories with different contexts
-SELECT memory_add_text('Meeting notes...', 'meetings');
-SELECT memory_add_text('API documentation...', 'api-docs');
-SELECT memory_add_text('Tutorial content...', 'tutorials');
+SELECT memory_sync_text('Meeting notes...', 'meetings');
+SELECT memory_sync_text('API documentation...', 'api-docs');
+SELECT memory_sync_text('Tutorial content...', 'tutorials');
 
 -- Search within a context
 SELECT * FROM memory_search
@@ -546,6 +618,6 @@ Errors can be caught using standard SQLite error handling mechanisms.
 
 ```sql
 -- Example error handling in application code
-SELECT memory_add_text(123);  -- Error: expects TEXT parameter
+SELECT memory_sync_text(123);  -- Error: expects TEXT parameter
 SELECT memory_delete('abc');  -- Error: expects INTEGER parameter
 ```

CLAUDE.md

@@ -0,0 +1,65 @@
+# CLAUDE.md
+
+Behavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed.
+
+**Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment.
+
+## 1. Think Before Coding
+
+**Don't assume. Don't hide confusion. Surface tradeoffs.**
+
+Before implementing:
+- State your assumptions explicitly. If uncertain, ask.
+- If multiple interpretations exist, present them - don't pick silently.
+- If a simpler approach exists, say so. Push back when warranted.
+- If something is unclear, stop. Name what's confusing. Ask.
+
+## 2. Simplicity First
+
+**Minimum code that solves the problem. Nothing speculative.**
+
+- No features beyond what was asked.
+- No abstractions for single-use code.
+- No "flexibility" or "configurability" that wasn't requested.
+- No error handling for impossible scenarios.
+- If you write 200 lines and it could be 50, rewrite it.
+
+Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify.
+
+## 3. Surgical Changes
+
+**Touch only what you must. Clean up only your own mess.**
+
+When editing existing code:
+- Don't "improve" adjacent code, comments, or formatting.
+- Don't refactor things that aren't broken.
+- Match existing style, even if you'd do it differently.
+- If you notice unrelated dead code, mention it - don't delete it.
+
+When your changes create orphans:
+- Remove imports/variables/functions that YOUR changes made unused.
+- Don't remove pre-existing dead code unless asked.
+
+The test: Every changed line should trace directly to the user's request.
+
+## 4. Goal-Driven Execution
+
+**Define success criteria. Loop until verified.**
+
+Transform tasks into verifiable goals:
+- "Add validation" → "Write tests for invalid inputs, then make them pass"
+- "Fix the bug" → "Write a test that reproduces it, then make it pass"
+- "Refactor X" → "Ensure tests pass before and after"
+
+For multi-step tasks, state a brief plan:
+```
+1. [Step] → verify: [check]
+2. [Step] → verify: [check]
+3. [Step] → verify: [check]
+```
+
+Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification.
+
+---
+
+**These guidelines are working if:** fewer unnecessary changes in diffs, fewer rewrites due to overcomplication, and clarifying questions come before implementation rather than after mistakes.