Merge branch 'dev'

Author: cpdata

.github/workflows/ci.yml

@@ -0,0 +1,50 @@
+name: CI
+
+on:
+  push:
+    branches: ["main", "review", "review-1"]
+  pull_request:
+
+env:
+  PIP_DISABLE_PIP_VERSION_CHECK: "1"
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out
+        uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Install toolchain
+        run: |
+          pip install uv
+          uv pip install --system -e .
+          uv pip install --system ruff pyright typeguard toml-sort yamllint
+      - name: Lint and format checks
+        run: |
+          make fmt-check
+          make protos-check
+      - name: Docs guard
+        env:
+          BASE_REF: ${{ github.event.pull_request.base.sha || 'HEAD~1' }}
+        run: make docs-guard
+
+  tests:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out
+        uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Install dependencies
+        run: |
+          pip install uv
+          uv pip install --system -e .
+          uv pip install --system pytest
+      - name: Run pytest
+        run: make test

AGENTS.md

@@ -0,0 +1,20 @@
+# Agent Instructions
+
+## Documentation Workflow
+- After each batch of changes, add a `CHANGELOG.md` entry with an ISO 8601 date/time stamp in United States Eastern time (include the timezone code, e.g., `America/New_York` or `ET`) and developer-facing detail (files, modules, functions, variables, and rationale). Every commit should correspond to a fresh entry.
+- Maintain `README.md` as the canonical description of the project; update it whenever behaviour or workflows change. Archive older versions separately when requested.
+- Keep the `docs/` wiki and provisioning guides (`SETUP.md`, `ENVIRONMENT_NEEDS.md`) in sync with code updates; add or revise the
+  relevant page whenever features, modules, or workflows change.
+- After each iteration, refresh `ISSUES.md`, `SOT.md`, `PLAN.md`, `RECOMMENDATIONS.md`, `TODO.md`, and related documentation to stay in sync with the codebase.
+- Ensure `TODO.md` retains the `Completed`, `Priority Tasks`, and `Recommended Waiting for Approval Tasks` sections, moving finished items under `Completed` at the end of every turn.
+- Make every task in `TODO.md` atomic: each entry must describe a single, self-contained deliverable with enough detail to execute and verify without cross-referencing additional context.
+- Update `RESUME_NOTES.md` at the end of every turn so the next session starts with accurate context.
+- When beginning a turn, review `README.md`, `PROJECT.md`, `PLAN.md`, `RECOMMENDATIONS.md`, `ISSUES.md`, `SOT.md`, `ROADMAP.md`, `PLANNING_THOUGHTS.md`, and the `research/` wiki to harvest new actionable work. Maintain at least ten quantifiable, prioritised items in the `Priority Tasks` section of `TODO.md`, adding context or links when needed.
+- Keep `ROADMAP.md`, `PLANNING_THOUGHTS.md`, and the `research/` docs aligned with each iteration when new discoveries or decisions occur.
+- After completing any task, immediately update `TODO.md`, check for the next actionable item, and continue iterating until all unblocked `Priority Tasks` are exhausted for the session.
+- Continuously loop through planning and execution: finish a task, document it, surface new follow-ups, and resume implementation so long as environment blockers allow. If extra guidance would improve throughput, extend these instructions proactively.
+
+## Style Guidelines
+- Use descriptive Markdown headings starting at level 1 for top-level documents.
+- Keep lines to 120 characters or fewer when practical.
+- Prefer bullet lists for enumerations instead of inline commas.

CHANGELOG.md

@@ -0,0 +1,83 @@
+# Cleanup Plan for Temporary Workarounds
+
+The following actions should be executed once the development environment consistently provides internet access and all
+optional dependencies can be installed. Each item references the affected files, the temporary workaround currently in
+place, and the exact remediation steps required to bring the implementation in line with production expectations.
+
+## Compatibility Layers
+
+### Replace Pydantic Shim (Completed)
+- **Files**: `meshmind/_compat/pydantic.py`, modules importing from `meshmind._compat.pydantic`.
+- **Status**: Completed – the shim has been removed, `pydantic>=2.11` is a required dependency, and all models/tests now consume the official APIs directly.
+
+### Retire FastAPI Stub (Completed)
+- **Files**: `meshmind/api/rest.py`, `meshmind/api/service.py`, `docs/api.md`, `SETUP.md`.
+- **Status**: Completed – the REST layer now requires FastAPI, tests rely on
+  `fastapi.testclient.TestClient`, and documentation reflects the production
+  stack.
+
+### Replace gRPC Dataclass Shim (Completed)
+- **Files**: `meshmind/api/grpc.py`, `meshmind/protos/memory_service.proto`, `meshmind/tests/test_service_interfaces.py`, `docs/api.md`.
+- **Status**: Completed – protobuf definitions now live under `meshmind/protos`, generated modules back the Python stub, setup scripts install `grpcio`/`grpcio-tools`, and tests exercise the canonical schema.
+
+### Promote gRPC Server Implementation
+- **Files**: `meshmind/api/grpc.py`, `meshmind/api/grpc_server.py`, CLI entry points,
+  integration tests.
+- **Current State**: An asyncio server helper and CLI (`meshmind serve-grpc`) now
+  run the service; Docker Compose provisions a gRPC container. Integration tests
+  against a live server remain outstanding.
+- **Action**:
+  1. Add integration tests (possibly using `grpc.aio.insecure_channel`) that cover
+     ingestion, search, and memory counts against the running server.
+  2. Publish generated client artifacts for downstream consumers once
+     infrastructure is available.
+
+## Task Scheduling Workarounds
+
+### Remove Celery Dummy App and Beat Fallback (Completed)
+- **Files**: `meshmind/tasks/celery_app.py`, `meshmind/tasks/scheduled.py`.
+- **Status**: Completed – Celery is a required dependency, the runtime imports the
+  real app/beat scheduler, and docker-compose stacks provision Redis for workers.
+
+## Testing Fakes
+
+### Evaluate Graph/Embedding/Test Fakes
+- **Files**: `meshmind/testing/fakes.py`, `meshmind/tests/conftest.py`, `meshmind/tests/test_*`.
+- **Current State**: In-memory drivers and dummy encoders enable offline unit tests.
+- **Action**:
+  1. Keep unit-test fakes for isolation but ensure parallel integration suites use live services.
+  2. Document expectations in `docs/testing.md` and mark fakes clearly as test-only utilities.
+  3. Remove any production code paths that accidentally import fakes.
+
+### Replace Memgraph Client Mock
+- **Files**: `meshmind/tests/test_memgraph_driver.py`.
+- **Current State**: Monkeypatches a fake `mgclient` module for driver tests.
+- **Action**:
+  1. Install `pymgclient` and run tests against a dockerized Memgraph instance.
+  2. Retain the fake only for negative/offline coverage; guard it behind an explicit fixture flag.
+
+## Observability and Telemetry
+
+### Confirm Logging Metrics Alignment
+- **Files**: `meshmind/core/observability.py`, `meshmind/pipeline/*`, `docs/telemetry.md`.
+- **Current State**: Logging/metrics rely on standard library fallbacks.
+- **Action**:
+  1. Integrate preferred observability stack (e.g., OpenTelemetry) once dependencies are greenlit.
+  2. Remove or demote placeholders that mimic external exporters.
+
+## Tooling and Automation
+
+### Finalize Setup Scripts
+- **Files**: `run/install_setup.sh`, `run/maintenance_setup.sh`.
+- **Current State**: Scripts install packages opportunistically.
+- **Action**:
+  1. Review installed package list after environment unblock and remove conditional guards.
+  2. Ensure scripts configure CLI tools (e.g., `grpcurl`, `neo4j-admin`) that were previously skipped offline.
+
+### Lock Dependency Graph
+- **Files**: `pyproject.toml`, `uv.lock`.
+- **Current State**: Lockfile may exclude previously unavailable packages.
+- **Action**:
+  1. Regenerate `uv.lock` after all dependencies are installed.
+  2. Commit the updated lockfile and note differences in `CHANGELOG.md`.
+

CLEANUP.md

@@ -0,0 +1,53 @@
+# README vs Implementation Discrepancies
+
+## Overview
+- The legacy README has been superseded by `README.md`, which now reflects the implemented feature set.
+- The current codebase delivers extraction, preprocessing, triplet persistence, CRUD helpers, and expanded retrieval strategies
+  that were missing when the README was written.
+- Remaining gaps primarily involve pushing retrieval workloads into the graph backend, exporting observability to external sinks, and automated infrastructure provisioning.
+
+## API Surface
+- ✅ `MeshMind` now exposes CRUD helpers (`create_memory`, `update_memory`, `delete_memory`, `list_memories`, triplet helpers)
+  that the README referenced implicitly.
+- ✅ Triplet storage routes through `store_triplets` and `MemoryManager.add_triplet`, calling `GraphDriver.upsert_edge`.
+- ⚠️ The README still references `register_entity`, `register_allowed_predicates`, and `add_predicate`; predicate management is
+  handled automatically but there is no public API matching those method names.
+- ⚠️ README snippets showing `mesh_mind.store_memory(memory)` should be updated to call `store_memories([memory])` or the new
+  CRUD helpers.
+
+## Retrieval Capabilities
+- ✅ Vector-only, regex, exact-match, hybrid, BM25, fuzzy, and optional LLM rerank searches exist in `meshmind.retrieval.search`
+  and are surfaced through `MeshMind` helpers.
+- ⚠️ README implies retrieval queries the graph directly. Search helpers now fetch candidates from the configured driver when no
+  list is supplied but still score results in Python; Memgraph/Neo4j-native search remains future work.
+- ⚠️ Named helpers like `search_facts` or `search_procedures` never existed; the README should reference the dispatcher plus
+  specialized helpers now available.
+
+## Data & Relationship Modeling
+- ✅ Predicates are persisted automatically when storing triplets and tracked in `PredicateRegistry`.
+- ⚠️ README examples that look up subjects/objects by name still do not match the implementation, which expects UUIDs. Add
+  documentation explaining how to resolve names to UUIDs before storing edges.
+- ⚠️ Consolidation and expiry run via Celery jobs; README narratives should highlight that heuristics require further validation even though persistence is now wired up.
+
+## Configuration & Dependencies
+- ✅ `README.md` and `ENVIRONMENT_NEEDS.md` document required environment variables, dependency guards, and setup steps.
+- ⚠️ README still omits optional tooling now required by the Makefile/CI (ruff, pyright, typeguard, toml-sort, yamllint);
+  highlight these prerequisites more prominently.
+- ✅ Python version support in `pyproject.toml` now pins `>=3.11,<3.13`, matching the dependency landscape documented in the README.
+
+## Example Code Paths
+- ✅ Updated example scripts demonstrate extraction, triplet creation, and multiple retrieval strategies.
+- ⚠️ Legacy README code that instantiates custom Pydantic entities remains inaccurate; extraction returns `Memory` objects and
+  validates `entity_label` names only.
+- ⚠️ Search examples should be updated to show the new helper functions and optional rerank usage instead of nonexistent
+  `search_facts`/`search_procedures` calls.
+
+## Tooling & Operations
+- ✅ Makefile and CI workflows now exist, aligning with README promises about automation once the README is refreshed.
+- ✅ Docker Compose now provisions Memgraph, Redis, and a Celery worker; README sections should highlight the workflow and
+  caveats for environments lacking container tooling.
+- ⚠️ Celery tasks still depend on optional infrastructure; README should clarify that heuristics and scheduling need production hardening even though persistence now works.
+
+## Documentation State
+- Continue promoting `README.md` as the authoritative guide and propagate updates to supporting docs
+  (`SOT.md`, `PLAN.md`, `ENVIRONMENT_NEEDS.md`, `docs/`).

DISCREPANCIES.md

@@ -0,0 +1,19 @@
+# Dummies and Compatibility Artifacts
+
+The following table tracks all temporary shims, fake implementations, and stub services that were introduced while the
+sandbox lacked internet access or external infrastructure. Each row notes what the component does today and the
+recommended next step now that full dependencies can be installed.
+
+| Component | Location | Purpose | Current Usage | Recommended Action |
+| --- | --- | --- | --- | --- |
+| gRPC stub implementation | `meshmind/api/grpc.py` (`GrpcServiceStub` + generated protobuf helpers) | Provides an in-process service implementation backed by the canonical proto schema so tests can exercise the service layer without spinning up gRPC infrastructure. | Unit tests and docs rely on the stub while production traffic should flow through the new asyncio helpers in `meshmind.api.grpc_server`. | Keep the stub for tests; package the new server behind a CLI entry point and retire any ad-hoc wrappers once infrastructure is provisioned. |
+| Fake graph/storage drivers | `meshmind/testing/fakes.py` (`FakeMemgraphDriver`, `FakeRedisBroker`, `FakeEmbeddingEncoder`) | Provide offline stand-ins for Memgraph, Redis, and embedding models. | Pytest fixtures and documentation rely on these for isolation. | Keep as long as offline tests are desired; supplement with integration suites that use real services. |
+| Fake LLM client | `meshmind/testing/fakes.py` (`FakeLLMClient`) | Records per-request overrides and emits deterministic responses so tests exercise reranking without installing the OpenAI SDK. | Service/interface tests (`meshmind/tests/test_service_interfaces.py`, `test_client.py`) and the CLI fixtures inject this stub when `openai` is unavailable. | Keep for unit tests; add integration tests with real providers once keys and network access are provisioned. |
+| Dummy encoder fixture | `meshmind/tests/conftest.py` (`dummy_encoder`) and dependent tests | Supplies a lightweight embedding encoder for search tests. | Used across retrieval and service tests to avoid network calls. | Keep for unit tests; add integration coverage with real encoders once APIs are configured. |
+| Fake mgclient module | `meshmind/tests/test_memgraph_driver.py` (monkeypatch of `mgclient`) | Simulates the Memgraph client so driver code runs without the native binary. | Enables driver unit tests without installing `mgclient`. | Replace with real `mgclient`-backed tests once package access is ensured; keep shim for fallback coverage. |
+
+## Retired Items
+
+- **Pydantic compatibility shim (`meshmind/_compat/pydantic.py`)** – Removed now that the project installs `pydantic>=2.11` during setup. All models import directly from Pydantic and tests confirm no fallback is required.
+- **FastAPI REST stub (`meshmind/api/rest.py`)** – Removed in favour of the concrete FastAPI application now that `fastapi` is a required dependency and tests exercise it via `TestClient`.
+- **Celery dummy app and beat fallback (`meshmind/tasks/celery_app.py`, `meshmind/tasks/scheduled.py`)** – Retired because Celery is now installed by default; the real app and `crontab` integration are always imported.

DUMMIES.md

@@ -0,0 +1,24 @@
+FROM python:3.11-slim
+
+ENV PIP_NO_CACHE_DIR=1 \
+    PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1
+
+WORKDIR /app
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends \
+        build-essential \
+        cmake \
+        libssl-dev \
+        libkrb5-dev \
+        curl \
+        git \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY . /app
+
+RUN pip install uv \
+    && uv pip install --system -e .[dev,docs,testing]
+
+CMD ["bash"]

Dockerfile

@@ -0,0 +1,34 @@
+# Tasks for Human Project Manager
+
+- Keep the Python package layer aligned with the project extras during base image
+  refreshes. The `run/install_setup.sh` and `run/maintenance_setup.sh` scripts now
+  install the full optional stack (neo4j driver, `pymgclient`, Redis, Celery extras,
+  FastAPI/Uvicorn, LLM tooling, and developer linters/testers). Ensure cached
+  environments either run the maintenance script or bake these dependencies into the
+  image so cold starts do not regress coverage.
+- Provide system-level build dependencies for the graph drivers (e.g., `build-essential`,
+  `cmake`, `libssl-dev`, `libkrb5-dev`) so `pymgclient` (and its `mgclient` module) install cleanly.
+- Provision external services and credentials (compose files now exist under the project
+  root and `meshmind/tests/docker/`):
+  - Neo4j instance reachable from the execution environment with `NEO4J_URI`,
+    `NEO4J_USERNAME`, `NEO4J_PASSWORD` (defaults to `neo4j` / `meshminD123`).
+  - Memgraph instance with `MEMGRAPH_URI`, `MEMGRAPH_USERNAME`, `MEMGRAPH_PASSWORD`
+    (anonymous access acceptable locally).
+  - Redis instance with `REDIS_URL`.
+  - LLM provider API key (`LLM_API_KEY` or fallback `OPENAI_API_KEY`) plus any
+    alternative base URLs/models required for OpenRouter, Azure, or Google-hosted
+    endpoints so the new `llm_client` overrides can be exercised end-to-end.
+  - Default maintenance retry configuration (`MAINTENANCE_MAX_ATTEMPTS`, `MAINTENANCE_BASE_DELAY_SECONDS`) tuned for the deployed graph backend; surface recommended values once integration tests run against live clusters. *(Future refinement request once infra is available.)*
+- Supply datasets/fixtures (future request) representing large knowledge graphs to
+  stress-test consolidation heuristics and pagination under load.
+- Maintain outbound package download access to PyPI and vendor repositories; this
+  session confirmed package installation works when the network is open, and future
+  sessions need the same capability to refresh locks or install new optional
+  integrations.
+- Enable Docker or container runtime access (future request) so the provided
+  `docker-compose.yml` files can run inside this environment; alternatively, provision
+  remote services accessible to CI.
+- Document credential management procedures and rotation cadence so secrets stay current.
+- Keep gRPC tooling (`grpcio`, `grpcio-tools`, protobuf compiler) available in cached environments; the proto definitions now
+  back the production stubs and runtime server (`meshmind.api.grpc_server`). `scripts/generate_protos.py` regenerates bindings
+  and `scripts/check_protos.py` enforces drift in CI, so ensure the toolchain runs successfully during maintenance scripts.