feat(flowctl): migrate rusqlite → libSQL async + native vectors (#9)

* wip(flowctl-db): libsql foundation — async pool + schema (isolated tests pass)

Task 1 of fn-19 migration: libsql 0.9 installed as workspace dep,
new pool_async.rs with async open_memory_async/open_async, schema
consolidated into schema_libsql.sql with F32_BLOB(384) on memory table
and native libsql_vector_idx for semantic search.

ISSUE FOUND: libsql 0.9 cannot coexist with rusqlite in the same
test binary. 'Once instance has previously been poisoned' panic
when both init in same process. pool_async tests pass in isolation
(cargo test pool_async::) but fail when run with rusqlite tests.

Root cause: libsql bundles its own libsql-core (SQLite fork) and
rusqlite links libsqlite3. The two static inits collide.

Options for next step:
  A. Split into separate crate (flowctl-db-libsql)
  B. Feature-flag XOR (rusqlite OR libsql, never both)
  C. Big cutover: delete rusqlite entirely NOW, break everything, rebuild
  D. Use libsql-rusqlite compat shim instead of native libsql API

Committing scaffolding + schema + 6 passing tests as checkpoint.
Waiting for architectural decision before proceeding to task 2.

fn-19-migrate-flowctl-to-libsql-async-native.1

* feat(flowctl-db-lsql): new isolated crate for async libSQL data layer

Task 1 of fn-19 migration (strategy pivot — separate crate, Option A).

Why separate crate: libsql 0.9 and rusqlite cannot coexist in the same
test binary due to C-level static init collision ('Once instance has
previously been poisoned'). Splitting into its own crate gives clean
test isolation — each crate's tests run in a separate binary without
the other's SQLite init running.

New crate contents:
- pool.rs: async open_async() + open_memory_async() + PRAGMAs
- schema.sql: all 14 tables + memory.embedding F32_BLOB(384) +
  libsql_vector_idx for native semantic search
- error.rs: self-contained DbError enum (libsql::Error + serde_json + etc.)
- lib.rs: re-exports Connection, Database

Test isolation verified:
- cargo test -p flowctl-db --release: 57 passed
- cargo test -p flowctl-db-lsql --release: 4 passed
- cargo test --all --release: all green (separate test binaries)

Next: tasks 2-5 add repos in flowctl-db-lsql. Task 6 migrates callers.
Task 7 deletes flowctl-db entirely.

fn-19-migrate-flowctl-to-libsql-async-native.1

* feat(flowctl-db-lsql): async EpicRepo + TaskRepo + DepRepo + FileOwnershipRepo

Port core relational repos from flowctl-db (sync rusqlite) to
flowctl-db-lsql (async libsql). All methods take owned Connection
(cheap clone) and return DbError.

- EpicRepo: upsert/upsert_with_body/get/get_with_body/list/
  update_status/delete + epic_deps upsert
- TaskRepo: upsert/upsert_with_body/get/get_with_body/list_by_epic/
  list_all (status+domain filters)/list_by_status/update_status/
  delete + task_deps + file_ownership upsert
- DepRepo: add/remove/list for task_deps and epic_deps
- FileOwnershipRepo: add/remove/list_for_task/list_for_file
- parse_* helpers inlined at top of file
- 9 new async tests; empty-body preservation verified;
  NotFound mapping verified

All 13 tests pass (4 pool + 9 repo) on cargo test --release.
cargo check --all --release green.

fn-19-migrate-flowctl-to-libsql-async-native.2

* feat(flowctl-db-lsql): async Runtime + Evidence + FileLock + PhaseProgress repos

Appends four async repos to flowctl-db-lsql, completing the runtime
state surface area (Task 3 of fn-19). Each repo owns a libsql::Connection
by value (cheap clone) and exposes async methods returning DbError.

- RuntimeRepo: upsert/get on runtime_state (Teams assignment + timing)
- EvidenceRepo: upsert/get with JSON commit/test arrays
- FileLockRepo: Teams-mode file locking — acquire returns
  DbError::Constraint on duplicate file_path (load-bearing for
  concurrency). libsql Error is string-matched on
  'unique constraint' / 'constraint failed' / 'primary key'
- PhaseProgressRepo: mark_done/get_completed/reset on phase_progress

Tests: 21 passing (13 existing + 8 new). Notable: file_lock_acquire_twice
verifies the Constraint error path.

Verified: cargo test -p flowctl-db-lsql --release passes;
cargo check --all --release green.

fn-19-migrate-flowctl-to-libsql-async-native.3

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(flowctl-db-lsql): async EventLog + StatsQuery + EventRepo

Port events.rs, metrics.rs, and EventRepo from flowctl-db to async libSQL:

- events.rs: EventLog with query/query_by_type/record_token_usage/
  tokens_by_task/tokens_by_epic/count_by_type. Exports EventRow,
  TaskTokenSummary, TokenRecord, TokenUsageRow.
- metrics.rs: StatsQuery with summary/epic_stats/weekly_trends/
  token_breakdown/bottlenecks/dora_metrics/domain_duration_stats/
  generate_monthly_rollups.
- repo.rs: append EventRepo + EventRow (async, owned Connection)
  with insert/list_by_epic/list_by_type using last_insert_rowid.
- lib.rs: module decls + re-exports of new types.

All methods take owned libsql::Connection (cheap Clone). Uses the
async .query(..).await + rows.next().await pattern consistent with
Tasks 2/3. 11 new tests added (32 total, up from 21).

fn-19-migrate-flowctl-to-libsql-async-native.4

* feat(flowctl-db-lsql): MemoryRepo with fastembed + vector search

Semantic memory search via BGE-small embeddings + libSQL native
vector_top_k. Offline fallback to literal substring search.

- New `memory` module with MemoryRepo, MemoryEntry, MemoryFilter
- Methods: add, get, list, search_literal, search_semantic,
  delete, increment_refs
- fastembed 5.12 lazy-initialized via OnceCell + Mutex, CPU work
  pushed to blocking pool; first call downloads ~130MB BGE-small
- Embedding failures are non-fatal: add() still inserts with NULL
  embedding and logs a warning; search_semantic() returns
  DbError::Schema so callers can fall back to search_literal
- Vector storage via SQL `vector32(?1)` literal; semantic search
  uses `vector_top_k('memory_emb_idx', ...)` native libSQL index
- 5 new unit tests (CRUD, literal search, filter, refs, hash dedup)
  + 2 gated tests (#[ignore]) for model download + semantic lookup
- Verified: `cargo test -p flowctl-db-lsql --release` → 37 passing,
  2 ignored; `cargo check --all --release` → green
- Ignored tests verified manually against real model:
  `test_embedder_loads` (384-dim output), `test_search_semantic`
  (query "javascript frontend framework" → React entry)

fn-19-migrate-flowctl-to-libsql-async-native.5

* chore: ignore fastembed model cache

* feat(flowctl-service,daemon): async cascade to libsql

Task 6 of fn-19: migrate service + daemon layers to async libsql.

Service layer (crates/flowctl-service):
- Swap flowctl-db → flowctl-db-lsql, add libsql + tokio deps
- All lifecycle functions (start/done/block/fail/restart) become async
- Accept Option<&libsql::Connection> instead of Option<&rusqlite::Connection>
- All repo calls go through flowctl_db_lsql with .await
- ServiceError wraps flowctl_db_lsql::DbError
- connection.rs: drop ConnectionProvider trait (rusqlite-only), provide
  async open_async() + FileConnectionProvider::connect() async method

Daemon layer (crates/flowctl-daemon):
- Swap flowctl-db → flowctl-db-lsql, add libsql; drop rusqlite
- AppState.db: libsql::Connection (owned, cheap Clone) — no Arc<Mutex<>>
- Remove db_lock() — callers clone via state.db.clone()
- Remove all 5 tokio::task::spawn_blocking wrappers in handlers/task.rs
  (start/start_rest/done_rest/block_rest/restart_rest/done/block/restart)
- handlers/epic.rs, dag.rs, mod.rs: async repo/query cascade
- create_state() becomes async (opens db via open_async)
- AppError: add NotFound variant (HTTP 404), map DbError variants to
  appropriate HTTP status (NotFound→404, Constraint/InvalidInput→400)
- Migrate in-test inserts from state.db.lock() to state.db.execute().await

CLI (crates/flowctl-cli):
- Add flowctl-db-lsql + libsql deps, make tokio required (was optional)
- Add try_open_lsql_conn() + block_on() helpers in workflow/mod.rs
- workflow/lifecycle.rs: wrap each service call in block_on(...)
- mcp.rs: open libsql conn via tokio runtime, wrap service calls
- main.rs: .await on create_state()
- tests/parity_test.rs: 3 tests stubbed with #[ignore] (need rewrite with
  libsql async; tracked for fn-19 task 7)

flowctl-db-lsql drive-by fix:
- apply_pragmas(): use conn.query() instead of conn.execute() for PRAGMAs
  that return rows (journal_mode = WAL returns the resulting mode, which
  caused "Execute returned rows" errors on file-backed DBs). Fixes
  daemon + service file-backed test setup.

Workspace:
- Add flowctl-db-lsql to workspace deps in root Cargo.toml

Verification:
- cargo check --all --release: green
- cargo test -p flowctl-db-lsql --release: 37 passed (unchanged)
- cargo test -p flowctl-service --release: 2 passed
- cargo test -p flowctl-daemon --features daemon --release: 29 passed
- cargo build --release -p flowctl-daemon --features daemon: binary builds

Deferred to task 7:
- 3 parity tests in flowctl-cli/tests/parity_test.rs (stubbed + #[ignore])
- CLI workflow scheduling.rs/phase.rs still use flowctl-db (rusqlite) via
  try_open_db() helper — unchanged, only lifecycle.rs migrated
- flowctl-db rusqlite crate itself remains for CLI read paths

fn-19-migrate-flowctl-to-libsql-async-native.6

* chore(flowctl): delete rusqlite stack, libsql-only

Completes fn-19 migration. flowctl-db (rusqlite) crate deleted;
all 14 CLI command files migrated to flowctl-db-lsql via a small
sync shim (crates/flowctl-cli/src/commands/db_shim.rs) that wraps
the async repos in per-call current-thread tokio runtimes, keeping
call sites syntactically unchanged.

Changes:
- crates/flowctl-db/ deleted (968-line indexer included)
- crates/flowctl-db-lsql/src/indexer.rs: ported from old crate
  (async reindex + Python-legacy .json/.md fallback parsers)
- crates/flowctl-db-lsql/src/pool.rs: added cleanup(), resolve_db_path()
- crates/flowctl-cli/src/commands/db_shim.rs: new sync shim module
- flowctl-scheduler: dropped unused flowctl-db workspace dep
- workspace Cargo.toml: removed rusqlite, rusqlite_migration, include_dir
  shared deps; removed flowctl-db path dep; removed crate from members
- tests: 3 ignored parity placeholders removed; export_import_test.rs
  and parity_test.rs rewritten with async libsql (tokio::test)

Verification:
- grep rusqlite     crates/ --include '*.toml' --include '*.rs' → 0 code refs
- grep flowctl_db\\b crates/ --include '*.rs'                   → 0 code refs
- grep spawn_blocking crates/flowctl-daemon/                    → 0
- cargo test --all --release                                    → 269 passed

Follow-up: renaming flowctl-db-lsql → flowctl-db left as a follow-up
(would touch 6 Cargo.toml + ~40 .rs files and is mechanically safe
but not required for functional completeness).

fn-19-migrate-flowctl-to-libsql-async-native.7

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: z23cc

CLAUDE.md

@@ -56,6 +56,8 @@ bash scripts/ralph_e2e_short_rp_test.sh
 
 All tests create temp directories and clean up after themselves. They must NOT be run from the plugin repo root (safety check enforced).
 
+**Storage runtime**: flowctl is libSQL-only (async, native vector search via `F32_BLOB(384)`). The `flowctl-db` rusqlite crate was deleted in fn-19 — `flowctl-db-lsql` is the sole storage crate. First build downloads the fastembed ONNX model (~130MB) to `.fastembed_cache/` for semantic memory search; subsequent builds/tests reuse the cache.
+
 ## Code Quality
 
 ```bash

flowctl/.gitignore

@@ -1 +1,3 @@
 /target/
+.fastembed_cache/
+**/.fastembed_cache/