Replace REST stubs with FastAPI and wire gRPC CLI runtime

Author: cpdata

CHANGELOG.md

@@ -1,5 +1,33 @@
 # Changelog
 
+## [2025-10-16T16:35:00-04:00 (America/New_York)]
+### Added
+- Added a `serve-grpc` CLI subcommand (`meshmind/cli/__main__.py`) that instantiates
+  `MemoryService` via `create_graph_driver`, delegates to
+  `meshmind.api.grpc_server.serve_forever`, and is exercised by
+  `meshmind/tests/test_cli_admin.py` alongside the new docker-compose service
+  definitions (`docker-compose.yml`, `meshmind/tests/docker/full-stack.yml`).
+- Recorded CLI runtime coverage and protobuf drift checks by extending
+  `meshmind/tests/test_cli_admin.py` and `meshmind/tests/test_protos_packaging.py`,
+  ensuring the new `serve-grpc` command and `scripts/check_protos.py` guard remain
+  functional.
+
+### Changed
+- Removed the legacy REST stub and Celery fallbacks by requiring real FastAPI and
+  Celery imports (`meshmind/api/rest.py`, `meshmind/tasks/celery_app.py`,
+  `meshmind/tasks/scheduled.py`) and updated smoke tests to exercise the FastAPI
+  app via `fastapi.testclient.TestClient` (`meshmind/tests/test_service_interfaces.py`,
+  `meshmind/tests/test_counts_smoke.py`).
+- Hardened protobuf utilities (`scripts/generate_protos.py`, `scripts/check_protos.py`)
+  to normalise relative imports so package builds remain import-safe.
+- Updated provisioning assets and documentation (README.md, SETUP.md,
+  docs/api.md, docs/operations.md, docs/testing.md, PROJECT.md, PLAN.md,
+  RECOMMENDATIONS.md, FINDINGS.md, ROADMAP.md, DUMMIES.md, CLEANUP.md,
+  ENVIRONMENT_NEEDS.md, NEEDED_FOR_TESTING.md, SOT.md, TODO.md, RESUME_NOTES.md)
+  to describe the new gRPC workflow, retired shims, and compose stacks.
+- Expanded the documentation guard mapping (`scripts/check_docs_sync.py`) so CLI
+  changes require updates to the API/operations guides.
+
 ## [2025-10-16T12:06:05-04:00 (America/New_York)]
 ### Added
 - Introduced `meshmind/api/grpc_server.py` with `create_server`, `serve`, and `serve_forever` helpers so production deployments

CLEANUP.md

@@ -10,37 +10,34 @@ place, and the exact remediation steps required to bring the implementation in l
 - **Files**: `meshmind/_compat/pydantic.py`, modules importing from `meshmind._compat.pydantic`.
 - **Status**: Completed – the shim has been removed, `pydantic>=2.11` is a required dependency, and all models/tests now consume the official APIs directly.
 
-### Retire FastAPI Stub
+### Retire FastAPI Stub (Completed)
 - **Files**: `meshmind/api/rest.py`, `meshmind/api/service.py`, `docs/api.md`, `SETUP.md`.
-- **Current State**: A lightweight FastAPI replacement exposes REST endpoints without requiring the framework.
-- **Action**:
-  1. Install `fastapi`, `uvicorn`, and related extras via the setup scripts.
-  2. Replace the stub implementation with a true FastAPI app using pydantic request/response models.
-  3. Update tests to spin up the FastAPI test client instead of the stub.
-  4. Refresh documentation to reflect the production stack only.
+- **Status**: Completed – the REST layer now requires FastAPI, tests rely on
+  `fastapi.testclient.TestClient`, and documentation reflects the production
+  stack.
 
 ### Replace gRPC Dataclass Shim (Completed)
 - **Files**: `meshmind/api/grpc.py`, `meshmind/protos/memory_service.proto`, `meshmind/tests/test_service_interfaces.py`, `docs/api.md`.
 - **Status**: Completed – protobuf definitions now live under `meshmind/protos`, generated modules back the Python stub, setup scripts install `grpcio`/`grpcio-tools`, and tests exercise the canonical schema.
 
 ### Promote gRPC Server Implementation
-- **Files**: `meshmind/api/grpc.py`, future server entry point, integration tests.
-- **Current State**: Tests rely on the in-process stub; there is no deployable server or channel bootstrap yet.
+- **Files**: `meshmind/api/grpc.py`, `meshmind/api/grpc_server.py`, CLI entry points,
+  integration tests.
+- **Current State**: An asyncio server helper and CLI (`meshmind serve-grpc`) now
+  run the service; Docker Compose provisions a gRPC container. Integration tests
+  against a live server remain outstanding.
 - **Action**:
-  1. Implement a gRPC server (synchronous or `grpc.aio`) that binds the generated service to an actual channel.
-  2. Add integration tests (possibly using `grpc.aio.insecure_channel`) that cover ingestion, search, and memory counts against the running server.
-  3. Document provisioning steps (`SETUP.md`, `docs/api.md`) and add CLI helpers or docker-compose services if required.
+  1. Add integration tests (possibly using `grpc.aio.insecure_channel`) that cover
+     ingestion, search, and memory counts against the running server.
+  2. Publish generated client artifacts for downstream consumers once
+     infrastructure is available.
 
 ## Task Scheduling Workarounds
 
-### Remove Celery Dummy App and Beat Fallback
+### Remove Celery Dummy App and Beat Fallback (Completed)
 - **Files**: `meshmind/tasks/celery_app.py`, `meshmind/tasks/scheduled.py`.
-- **Current State**: Custom placeholders allow imports without Celery and Celery Beat.
-- **Action**:
-  1. Make `celery` a required dependency for maintenance tasks.
-  2. Refactor scheduling utilities to import real Celery constructs and fail fast when misconfigured.
-  3. Add integration tests that execute Celery workers against Redis or RabbitMQ in docker-compose.
-  4. Delete fallback classes once coverage exists.
+- **Status**: Completed – Celery is a required dependency, the runtime imports the
+  real app/beat scheduler, and docker-compose stacks provision Redis for workers.
 
 ## Testing Fakes
 

DUMMIES.md

@@ -6,10 +6,7 @@ recommended next step now that full dependencies can be installed.
 
 | Component | Location | Purpose | Current Usage | Recommended Action |
 | --- | --- | --- | --- | --- |
-| FastAPI REST stub | `meshmind/api/rest.py` (`RestAPIStub`, `create_app`) | Exposes REST behaviour without the FastAPI dependency. | Tests use the stub when FastAPI is not installed; production should use FastAPI. | Keep temporarily for offline tests but plan to gate it behind an explicit test flag once FastAPI becomes mandatory. |
 | gRPC stub implementation | `meshmind/api/grpc.py` (`GrpcServiceStub` + generated protobuf helpers) | Provides an in-process service implementation backed by the canonical proto schema so tests can exercise the service layer without spinning up gRPC infrastructure. | Unit tests and docs rely on the stub while production traffic should flow through the new asyncio helpers in `meshmind.api.grpc_server`. | Keep the stub for tests; package the new server behind a CLI entry point and retire any ad-hoc wrappers once infrastructure is provisioned. |
-| Celery dummy app | `meshmind/tasks/celery_app.py` (`_DummyCeleryApp`) | Allows module imports when Celery is missing. | Unit tests rely on the dummy to avoid Celery; production should use real Celery. | Retire dummy once Celery is a hard dependency; until then ensure tests explicitly exercise the real app when installed. |
-| Celery beat fallback | `meshmind/tasks/scheduled.py` (`crontab` shim) | Supplies a no-op crontab when Celery beat is missing. | Prevents import errors in test environments without Celery. | Remove once Celery is required; otherwise guard usage with feature flags. |
 | Fake graph/storage drivers | `meshmind/testing/fakes.py` (`FakeMemgraphDriver`, `FakeRedisBroker`, `FakeEmbeddingEncoder`) | Provide offline stand-ins for Memgraph, Redis, and embedding models. | Pytest fixtures and documentation rely on these for isolation. | Keep as long as offline tests are desired; supplement with integration suites that use real services. |
 | Fake LLM client | `meshmind/testing/fakes.py` (`FakeLLMClient`) | Records per-request overrides and emits deterministic responses so tests exercise reranking without installing the OpenAI SDK. | Service/interface tests (`meshmind/tests/test_service_interfaces.py`, `test_client.py`) and the CLI fixtures inject this stub when `openai` is unavailable. | Keep for unit tests; add integration tests with real providers once keys and network access are provisioned. |
 | Dummy encoder fixture | `meshmind/tests/conftest.py` (`dummy_encoder`) and dependent tests | Supplies a lightweight embedding encoder for search tests. | Used across retrieval and service tests to avoid network calls. | Keep for unit tests; add integration coverage with real encoders once APIs are configured. |
@@ -18,3 +15,5 @@ recommended next step now that full dependencies can be installed.
 ## Retired Items
 
 - **Pydantic compatibility shim (`meshmind/_compat/pydantic.py`)** – Removed now that the project installs `pydantic>=2.11` during setup. All models import directly from Pydantic and tests confirm no fallback is required.
+- **FastAPI REST stub (`meshmind/api/rest.py`)** – Removed in favour of the concrete FastAPI application now that `fastapi` is a required dependency and tests exercise it via `TestClient`.
+- **Celery dummy app and beat fallback (`meshmind/tasks/celery_app.py`, `meshmind/tasks/scheduled.py`)** – Retired because Celery is now installed by default; the real app and `crontab` integration are always imported.

FINDINGS.md

@@ -2,7 +2,7 @@
 
 ## General Observations
 - Core modules are now wired through the `MeshMind` client, including CRUD, triplet storage, and retrieval helpers. Graph-backed wrappers fetch namespace/entity-label filtered candidates from the configured driver automatically; remaining integration work focuses on server-side query optimisation and heuristic evaluation loops.
-- Optional dependencies are largely guarded behind lazy imports or factory functions, improving portability. Environments still need to install tooling referenced by the Makefile and CI (ruff, pyright, typeguard, toml-sort, yamllint). `DUMMIES.md` now tracks remaining shims (FastAPI/gRPC/Celery) and documents retired items such as the former Pydantic fallback.
+- Optional dependencies are largely guarded behind lazy imports or factory functions, improving portability. Environments still need to install tooling referenced by the Makefile and CI (ruff, pyright, typeguard, toml-sort, yamllint). `DUMMIES.md` now tracks remaining shims (gRPC stub, offline fakes) and documents retired items such as the former Pydantic fallback.
 - LLM usage is now centralized in `meshmind.llm_client` with per-operation defaults cascading from `LLM_*` environment variables and CLI overrides, reducing the risk of divergent configurations across pipelines and retrieval. REST/gRPC payloads now provide matching override dictionaries so services can experiment per request.
 - Timestamp helpers default to timezone-aware UTC, eliminating naive datetime outputs that previously leaked into maintenance pipelines and compatibility shims.
 - The gRPC interface is now defined by `meshmind/protos/memory_service.proto`; generated Python modules back the `GrpcServiceStub`
@@ -22,7 +22,7 @@
   embedding modules already encapsulate the SDK behind optional imports, easing the upcoming refactor to a dedicated
   wrapper.
 - Celery tasks initialize lazily, yet Redis/Memgraph services are still required at runtime. Docker Compose now provisions
-  Memgraph, Neo4j, and Redis, while targeted stacks under `meshmind/tests/docker/` support integration testing. `SETUP.md`
+  Memgraph, Neo4j, Redis, the gRPC server, and the Celery worker, while targeted stacks under `meshmind/tests/docker/` support integration testing. `SETUP.md`
   explains provisioning and teardown commands, and `scripts/benchmark_pagination.py` offers an offline way to measure driver throughput before integrating external services.
 
 ## Data Flow & Persistence

NEEDED_FOR_TESTING.md

@@ -62,7 +62,10 @@
 ## Local Configuration Steps
 - Ensure an embedding encoder is registered before extraction or hybrid search. The bootstrap utilities invoked by the CLI and
   `MeshMind` constructor handle this, but custom scripts must call `bootstrap_encoders()`.
-- For REST/gRPC testing, instantiate the `RestAPIStub`/`GrpcServiceStub` with the in-memory driver to avoid external services.
+- For REST/gRPC testing, instantiate the FastAPI app via `meshmind.api.rest.create_app`
+  and exercise it with `fastapi.testclient.TestClient` (requires the `httpx`
+  package); pair it with the `GrpcServiceStub` for lightweight gRPC coverage when
+  external services are unavailable.
 - Use `meshmind/testing` fakes (`FakeMemgraphDriver`, `FakeRedisBroker`, `FakeEmbeddingEncoder`, `FakeLLMClient`) in tests or demos to eliminate external infrastructure requirements.
 - Invoke `meshmind admin predicates` and `meshmind admin maintenance --max-attempts <n> --base-delay <seconds> --run <task>` during local runs to inspect predicate registries, telemetry, and tune maintenance retries without external services.
 - Use the benchmarking utilities in `scripts/` (`evaluate_importance.py`, `consolidation_benchmark.py`, `benchmark_pagination.py`) to validate heuristics and driver performance offline before connecting to live infrastructure.

PLAN.md

@@ -35,14 +35,17 @@
    incrementally.
 2. **Automation & CI** – Makefile provides lint/format/type/test/docs-guard targets and CI runs fmt-check, docs guard, and
    pytest. Protobuf drift now fails CI via `make protos-check`. Add caching and matrix builds when dependencies stabilize.
-3. **Environment Provisioning** – Docker Compose now provisions Memgraph, Neo4j, and Redis (with targeted stacks for tests).
-   Track multi-backend examples, document the new `SETUP.md`, keep docs current, and distribute the new `run/install_setup.sh`
-   and `run/maintenance_setup.sh` automation scripts for environment bootstrap.
+3. **Environment Provisioning** – Docker Compose now provisions Memgraph, Neo4j,
+   Redis, the gRPC server, and the Celery worker (with targeted stacks for tests).
+   Track multi-backend examples, document the new `SETUP.md`, keep docs current,
+   and distribute the new `run/install_setup.sh` and `run/maintenance_setup.sh`
+   automation scripts for environment bootstrap.
 
 ## Phase 5 – Strategic Enhancements (Planned)
 1. **Graph-Backed Retrieval** – Extend the new driver-side filtering/pagination to full vector/lexical execution using backend-native indexes to avoid round-tripping candidate embeddings.
 2. **Operational Observability** – Export telemetry to Prometheus/OpenTelemetry and surface dashboards/alerts.
 3. **Celery Hardening** – Stress test consolidation/compression heuristics at scale and codify retry/backoff policies.
-4. **Service Contracts** – Generated protobuf modules (`meshmind/protos/memory_service.proto`) back both the Python stub and the
-   new asyncio gRPC server helpers. Next: package a deployable entry point, publish generated clients, and add integration tests
-   once infrastructure is ready.
+4. **Service Contracts** – Generated protobuf modules (`meshmind/protos/memory_service.proto`)
+   back both the Python stub and the asyncio gRPC server helpers. A dedicated CLI
+   entry point (`meshmind serve-grpc`) now launches the runtime. Next: publish
+   generated clients and add integration tests once infrastructure is ready.

PROJECT.md

@@ -19,9 +19,10 @@
   initialize lazily so import-time failures are avoided.
 - **Support code**: `meshmind.core` provides configuration, data models, embeddings, similarity math, and optional dependency
   guards around tokenization.
-- **Service adapters**: `meshmind.api.rest` and `.grpc` expose REST/gRPC entry points. The gRPC surface now uses generated
-  protobuf messages (`meshmind/protos/memory_service.proto`) and runtime helpers in `meshmind.api.grpc_server` so tests,
-  scripts, and deployments share a canonical schema while retaining the in-process stub for fast feedback.
+- **Service adapters**: `meshmind.api.rest` exposes a FastAPI application, while
+  `.grpc` ships generated protobuf stubs alongside asyncio server helpers and a CLI
+  (`meshmind serve-grpc`) so deployments share a canonical schema while retaining
+  the in-process stub for fast feedback.
 - **Observability**: `meshmind.core.observability` collects metrics, gauges, and structured log events across pipelines and
   scheduled tasks.
 - **Tooling**: The CLI ingest command (`meshmind ingest`), updated example script, Makefile automation, CI workflow, and Docker
@@ -54,8 +55,8 @@
 - Graph-backed retrieval still hydrates namespace/entity-label filtered candidates client-side; pushing ranking into the graph store is future work.
 - Predicate management remains internal to the bootstrap process; external administration APIs are still missing.
 - Metrics remain in-memory; external exporters (Prometheus/OpenTelemetry) are not wired up.
-- gRPC deployments now rely on the asyncio server helpers; packaging an executable entry point and adding end-to-end integration
-  tests remain future work.
+- gRPC deployments now rely on the asyncio server helpers; end-to-end integration
+  tests against a live CLI-managed server remain future work.
 
 ## External Services & Dependencies
 - **Graph backend**: Choose via `GRAPH_BACKEND`. In-memory and SQLite require no external services. Memgraph needs the `pymgclient` package (which exposes the `mgclient` module);

README.md

@@ -174,6 +174,7 @@ meshmind admin predicates --add RELATED_TO
 meshmind admin maintenance --max-attempts 5 --base-delay 2.5 --run consolidate
 meshmind admin graph --backend neo4j
 meshmind admin counts --namespace demo
+meshmind serve-grpc --host 0.0.0.0 --port 50051 --backend memgraph
 ```
 
 ## Maintenance Tasks
@@ -190,8 +191,8 @@ Tasks instantiate the driver lazily, emit structured logs/metrics, and persist c
   to ensure code changes keep the wiki up to date.
 - **Examples** – `examples/extract_preprocess_store_example.py` demonstrates ingestion, triplet creation, and multiple retrieval
   strategies.
-- **Dockerfile / docker-compose** – Container definition and orchestration files that provision Memgraph, Neo4j, Redis, and the
-  Celery worker stacks documented in `SETUP.md` and `meshmind/tests/docker/`.
+- **Dockerfile / docker-compose** – Container definition and orchestration files that provision Memgraph, Neo4j, Redis, the
+  Celery worker, and the gRPC server documented in `SETUP.md` and `meshmind/tests/docker/`.
 - **Provisioning scripts** – `run/install_setup.sh` and `run/maintenance_setup.sh` validate that optional packages (`fastapi`,
   `neo4j`, `pymgclient`, `uvicorn`) are present and respect `MESH_SKIP_SYSTEM_PACKAGES=1` / `MESH_SKIP_PYTHON_SYNC=1` when you
   need a dry run without network access.
@@ -206,7 +207,7 @@ Tasks instantiate the driver lazily, emit structured logs/metrics, and persist c
   Use `make benchmarks` to run all three scripts with synthetic defaults and capture JSON summaries under `build/benchmarks/`.
 
 ## Service Interfaces
-- **REST** – `meshmind.api.rest.create_app` returns a FastAPI app (or lightweight stub) that exposes `/memories`, `/triplets`,
+- **REST** – `meshmind.api.rest.create_app` returns a FastAPI app that exposes `/memories`, `/triplets`,
   `/search`, and `/memories/counts` endpoints. Search payloads accept:
   - `use_llm_rerank` to toggle LLM-based reranking,
   - `llm_models`, `llm_base_urls`, and `llm_api_key` dictionaries for per-request overrides,
@@ -264,8 +265,9 @@ Tasks instantiate the driver lazily, emit structured logs/metrics, and persist c
 - MeshMind now depends on first-party Pydantic 2.x models; the legacy compatibility shim has been removed.
 - `meshmind/retrieval/bm25.py`, `meshmind/retrieval/fuzzy.py`, and `meshmind/core/similarity.py` include pure-Python fallbacks for scikit-learn, rapidfuzz, and numpy.
 - `meshmind/testing` exports fake Memgraph, Redis, and embedding drivers that power the pytest suite and examples without external infrastructure.
-- `DUMMIES.md` lists every remaining stub (REST/gRPC service adapters, Celery fallbacks, compatibility layers) with guidance
-  on whether to remove or preserve them once external services are provisioned.
+- `DUMMIES.md` lists every remaining stub (for example the gRPC in-process adapter
+  and offline test doubles) with guidance on whether to remove or preserve them
+  once external services are provisioned.
 
 ## Testing
 - Run `pytest` to execute the suite; tests rely on fixtures and fake drivers so they do not require external services or optional libraries.