Add local daemon Git layered indexing

Author: rudimar.ronsoni

AGENTS.md

@@ -0,0 +1,48 @@
+This is built on top of [CocoIndex v1](https://cocoindex.io/docs-v1/llms.txt).
+
+
+## Build and Test Commands
+
+This project uses [uv](https://docs.astral.sh/uv/) for project management.
+
+```bash
+uv run mypy .           # Type check Python code
+uv run pytest tests/    # Run Python tests
+```
+
+## Code Conventions
+
+### Internal vs External Modules
+
+We distinguish between **internal modules** (under packages with `_` prefix, e.g. `_internal.*` or `connectors.*._source`) and **external modules** (which users can directly import).
+
+**External modules** (user-facing, e.g. `cocoindex/ops/sentence_transformers.py`):
+
+* Be strict about not leaking implementation details
+* Use `__all__` to explicitly list public exports
+* Prefix ALL non-public symbols with `_`, including:
+  * Standard library imports: `import threading as _threading`, `import typing as _typing`
+  * Third-party imports: `import numpy as _np`, `from numpy.typing import NDArray as _NDArray`
+  * Internal package imports: `from cocoindex.resources import schema as _schema`
+* Exception: `TYPE_CHECKING` imports for type hints don't need prefixing
+
+**Internal modules** (e.g. `cocoindex/_internal/component_ctx.py`):
+
+* Less strict since users shouldn't import these directly
+* Standard library and internal imports don't need underscore prefix
+* Only prefix symbols that are truly private to the module itself (e.g. `_context_var` for a module-private ContextVar)
+
+### General principles (also covered by `/review-changes`)
+
+- **Top-level imports.** Defer to in-function only for a real circular dependency or a heavy import that isn't always needed.
+- **Specific types over `Any`.** When a value enters as a weaker form (`str`, `Any`), convert to the strong type at the earliest point. Don't propagate the weak form.
+- **`NamedTuple`/small dataclass for multi-value returns.** Access fields by name at call sites.
+- **Single source of truth.** When the same value or logic appears in multiple places, consolidate it.
+- **Delete dead code and dead config.** When a change makes something unreachable, the code, the tests, and the knobs all go.
+- **Honest names.** The name describes what the code does today.
+
+### Testing Guidelines
+
+We prefer end-to-end tests on user-facing APIs, over unit tests on smaller internal functions. With this said, there're cases where unit tests are necessary, e.g. for internal logic with various situations and edge cases, in which case it's usually easier to cover various scenarios with unit tests.
+
+When tests fail, fix the underlying issue. Don't skip, ignore, or exclude to get a green result.

README.md

@@ -61,6 +61,10 @@ Two install styles — they mirror the Docker image variants of the same names:
 
 Next, set up your [coding agent integration](#coding-agent-integration) — or jump to [Manual CLI Usage](#manual-cli-usage) if you prefer direct control.
 
+Docs:
+- [Git Layered Indexing](./docs/layered-indexing.md): configure reusable `base > branch > dirty` Git layers for root clones and linked worktrees.
+- [Docker Layered Indexing](./docs/docker-layered-indexing.md): run the layered daemon in Docker with persistent native state.
+
 ## Coding Agent Integration
 
 ### Skill (Recommended)
@@ -162,6 +166,16 @@ The background daemon starts automatically on first use.
 
 > **Tip:** `ccc index` auto-initializes if you haven't run `ccc init` yet, so you can skip straight to indexing.
 
+For Git repositories, you can configure layered indexing once from the root clone:
+
+```bash
+ccc init --base main                    # share a base layer across linked worktrees
+ccc index                               # builds base + branch + dirty layers as needed
+ccc overlay status                      # inspect the current layer stack
+```
+
+Linked worktrees reuse the same daemon-owned base layer and only index branch and dirty deltas. See [Git Layered Indexing](./docs/layered-indexing.md) for the full configuration model.
+
 ### CLI Reference
 
 | Command | Description |
@@ -170,6 +184,8 @@ The background daemon starts automatically on first use.
 | `ccc index` | Build or update the index (auto-inits if needed). Shows streaming progress. |
 | `ccc search <query>` | Semantic search across the codebase |
 | `ccc status` | Show index stats (chunk count, file count, language breakdown) |
+| `ccc overlay status` | Inspect Git layered indexing state for the current worktree |
+| `ccc overlay prune` | Prune expired branch and dirty layers |
 | `ccc mcp` | Run as MCP server in stdio mode |
 | `ccc doctor` | Run diagnostics — checks settings, daemon, model, file matching, and index health |
 | `ccc reset` | Delete index databases. `--all` also removes settings. `-f` skips confirmation. |
@@ -185,6 +201,7 @@ ccc search --lang python --lang markdown schema      # filter by language
 ccc search --path 'src/utils/*' query handler        # filter by path
 ccc search --offset 10 --limit 5 database schema     # pagination
 ccc search --refresh database schema                 # update index first, then search
+ccc index --base release/1.2                         # override Git overlay base ref once
 ```
 
 By default, `ccc search` scopes results to your current working directory (relative to the project root). Use `--path` to override.
@@ -231,11 +248,12 @@ PUID=$(id -u) PGID=$(id -g) docker compose -f <(curl -L https://raw.githubuserco
 
 Or grab [`docker/docker-compose.yml`](./docker/docker-compose.yml) and run `docker compose up -d` next to it (works on any shell, including Windows cmd / PowerShell).
 
-By default your home directory is mounted into the container (set
-`COCOINDEX_HOST_WORKSPACE` to narrow this to a specific code folder). Index
-data and the embedding model cache persist in a Docker volume across
-restarts. Your global settings file at `$HOME/.cocoindex_code/global_settings.yml`
-is visible and editable on the host; edits take effect on your next `ccc` command.
+By default your home directory is mounted into the container. For team setups,
+prefer a narrower mount such as `COCOINDEX_HOST_WORKSPACE=$HOME/src` or one
+repo path. Index data, daemon Git-layer state, and the embedding model cache
+persist in the `cocoindex-data` Docker volume under `/var/cocoindex`. Your
+global settings file at `$HOME/.cocoindex_code/global_settings.yml` is visible
+and editable on the host; edits take effect on your next `ccc` command.
 
 > **Pick a different image:** set `COCOINDEX_CODE_IMAGE` to override the
 > default. For example, the `:full` variant or GHCR:
@@ -254,6 +272,9 @@ docker run -d --name cocoindex-code \
   --volume "$HOME:/workspace" \
   --volume cocoindex-data:/var/cocoindex \
   -e COCOINDEX_CODE_HOST_PATH_MAPPING="/workspace=$HOME" \
+  -e COCOINDEX_CODE_STATE_DIR=/var/cocoindex/state \
+  -e COCOINDEX_CODE_RUNTIME_DIR=/var/run/cocoindex_code \
+  -e COCOINDEX_CODE_DB_PATH_MAPPING=/workspace=/var/cocoindex/db \
   cocoindex/cocoindex-code:latest
 ```
 </details>
@@ -267,18 +288,35 @@ docker run -d --name cocoindex-code \
   --volume "$HOME:/workspace" \
   --volume cocoindex-data:/var/cocoindex \
   -e COCOINDEX_CODE_HOST_PATH_MAPPING="/workspace=$HOME" \
+  -e COCOINDEX_CODE_STATE_DIR=/var/cocoindex/state \
+  -e COCOINDEX_CODE_RUNTIME_DIR=/var/run/cocoindex_code \
+  -e COCOINDEX_CODE_DB_PATH_MAPPING=/workspace=/var/cocoindex/db \
   cocoindex/cocoindex-code:latest
 ```
 </details>
 
 ### Shell wrapper for `ccc` commands
 
-Paste this into `~/.bashrc` / `~/.zshrc` so `ccc` feels native on the host
-and picks up the right project based on your current directory:
+Paste this into `~/.bashrc` / `~/.zshrc` so `ccc` feels native on the host,
+picks up the right project based on your current directory, and uses the right
+TTY mode for interactive commands vs. MCP or piped stdin:
 
 ```bash
 ccc() {
-  docker exec -it -e COCOINDEX_CODE_HOST_CWD="$PWD" cocoindex-code ccc "$@"
+  local container="${COCOINDEX_CODE_CONTAINER_NAME:-cocoindex-code}"
+  if [ "$(docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null)" != "true" ]; then
+    echo "cocoindex-code container is not running. Start it with: docker compose -f docker/docker-compose.yml up -d" >&2
+    return 1
+  fi
+
+  local flags=(-i)
+  if [ "${1:-}" != "mcp" ] && [ -t 0 ] && [ -t 1 ]; then
+    flags=(-it)
+  fi
+
+  docker exec "${flags[@]}" \
+    -e COCOINDEX_CODE_HOST_CWD="$PWD" \
+    "$container" ccc "$@"
 }
 ```
 
@@ -346,6 +384,86 @@ docker rm -f cocoindex-code
 docker volume rm cocoindex-db cocoindex-model-cache
 ```
 
+For regular upgrades, keep the volume and recreate the container:
+
+```bash
+docker compose -f docker/docker-compose.yml pull
+docker compose -f docker/docker-compose.yml up -d
+```
+
+Switch between the slim and full images by changing `COCOINDEX_CODE_IMAGE`:
+
+```bash
+COCOINDEX_CODE_IMAGE=cocoindex/cocoindex-code:latest docker compose -f docker/docker-compose.yml up -d
+COCOINDEX_CODE_IMAGE=cocoindex/cocoindex-code:full docker compose -f docker/docker-compose.yml up -d
+```
+
+### Docker debugging
+
+Useful commands:
+
+```bash
+# Logs from the daemon supervisor and daemon process
+docker logs -f cocoindex-code
+
+# Shell inside the container
+docker exec -it cocoindex-code sh
+
+# Daemon readiness/status
+docker exec cocoindex-code ccc daemon status
+docker exec cocoindex-code test -S /var/run/cocoindex_code/daemon.sock
+
+# Restart the container
+docker restart cocoindex-code
+
+# Stop and remove the container, preserving index/state/cache volume
+docker rm -f cocoindex-code
+
+# Reset all Docker-managed index/state/cache data
+docker compose -f docker/docker-compose.yml down -v
+```
+
+Docker paths:
+
+| Data | Default path |
+|---|---|
+| Host workspace mount | `/workspace` |
+| Settings on the mounted workspace | `/workspace/.cocoindex_code/global_settings.yml` |
+| DB/index files | `/var/cocoindex/db` via `COCOINDEX_CODE_DB_PATH_MAPPING=/workspace=/var/cocoindex/db` |
+| Durable daemon Git-layer state | `/var/cocoindex/state` via `COCOINDEX_CODE_STATE_DIR` |
+| Runtime socket, PID, and daemon log | `/var/run/cocoindex_code` via `COCOINDEX_CODE_RUNTIME_DIR` |
+| Embedding/cache data | `/var/cocoindex/cache` |
+
+Local Git worktrees that use the same Docker container should share the Docker
+daemon state in `/var/cocoindex/state`. That lets the local daemon Git-layer
+feature reuse daemon-owned layer metadata and materialized layer sources across
+projects while keeping transient sockets under `/var/run/cocoindex_code`.
+
+For layered indexing in Docker, initialize the base ref from the root clone and
+then use linked worktrees through the same wrapper/container:
+
+```bash
+cd $HOME/src/github/cocoindex-io/cocoindex-code
+ccc init --base main
+ccc index
+
+git worktree add ../cocoindex-code.worktrees/feature-1 -b feature-1 main
+cd ../cocoindex-code.worktrees/feature-1
+ccc index
+ccc overlay status
+```
+
+Mount a workspace parent that contains both the root clone and linked
+worktrees. For example:
+
+```bash
+COCOINDEX_HOST_WORKSPACE=$HOME/src/github/cocoindex-io \
+  docker compose -f docker/docker-compose.yml up -d
+```
+
+See [Docker Layered Indexing](./docs/docker-layered-indexing.md) for the full
+Docker setup and troubleshooting guide.
+
 ### Configuration via environment variables
 
 Pass configuration to `docker run` / compose with `-e`:
@@ -365,6 +483,25 @@ Pass configuration to `docker run` / compose with `-e`:
 > to everything under it. If that's too broad, bind-mount a narrower
 > directory instead (`COCOINDEX_HOST_WORKSPACE=/path/to/code`).
 
+Supported Docker environment variables:
+
+| Variable | Purpose |
+|---|---|
+| `COCOINDEX_CODE_IMAGE` | Compose image, e.g. `cocoindex/cocoindex-code:full`. |
+| `COCOINDEX_CODE_CONTAINER_NAME` | Compose container name, default `cocoindex-code`. |
+| `COCOINDEX_HOST_WORKSPACE` | Host directory mounted at `/workspace`, default `${HOME}`. |
+| `COCOINDEX_CODE_HOST_PATH_MAPPING` | Container-to-host path mapping for displayed paths. |
+| `COCOINDEX_CODE_HOST_CWD` | Host current directory forwarded by `docker exec` wrappers. |
+| `COCOINDEX_CODE_STATE_DIR` | Durable daemon state directory, default `/var/cocoindex/state`. |
+| `COCOINDEX_CODE_RUNTIME_DIR` | Runtime socket/PID/log directory, default `/var/run/cocoindex_code`. |
+| `COCOINDEX_CODE_DB_PATH_MAPPING` | DB/index storage remapping, default `/workspace=/var/cocoindex/db`. |
+| `PUID`, `PGID` | Linux UID/GID used to chown Docker-managed paths and write host-owned workspace files. |
+
+`COCOINDEX_CODE_STATE_DIR` is where repository/worktree metadata, overlay
+policy, layer manifests, and materialized layer sources are stored. Keep it on
+the persistent Docker volume if you want base layers to survive container
+recreation.
+
 ### Build the image locally
 
 ```bash
@@ -373,6 +510,7 @@ docker build -t cocoindex-code:local -f docker/Dockerfile .
 
 ## Features
 - **Semantic Code Search**: Find relevant code using natural language queries when grep doesn't work well, and save tokens immediately.
+- **Git Layered Indexing**: Reuse a shared base index across root clones and linked worktrees, then index only branch and dirty deltas. Configure it with `ccc init --base main`; see [Git Layered Indexing](./docs/layered-indexing.md).
 - **Ultra Performant**: ⚡ Built on top of ultra performant [Rust indexing engine](https://github.com/cocoindex-io/cocoindex). Only re-indexes changed files for fast updates.
 - **Multi-Language Support**: Python, JavaScript/TypeScript, Rust, Go, Java, C/C++, C#, SQL, Shell, and more.
 - **Embedded**: Portable and just works, no database setup required!
@@ -493,6 +631,23 @@ def my_chunker(path: Path, content: str) -> tuple[str | None, list[Chunk]]:
 
 See [`src/cocoindex_code/chunking.py`](./src/cocoindex_code/chunking.py) for the public types and [`tests/example_toml_chunker.py`](./tests/example_toml_chunker.py) for a complete example.
 
+### Git Layered Indexing Configuration
+
+For Git repositories, `ccc init --base <ref>` stores a repository-level overlay
+policy in daemon state. The checkout-local `settings.yml` still controls file
+matching and chunking, while daemon state controls the shared base ref used by
+root clones and linked worktrees.
+
+```bash
+ccc init --base main
+ccc index
+ccc overlay status
+```
+
+The daemon stores durable layer metadata under `COCOINDEX_CODE_STATE_DIR` and
+uses stable hash IDs, so moving a repository or linked worktree does not
+invalidate reusable base and branch layers. See [Git Layered Indexing](./docs/layered-indexing.md) for details.
+
 ## Embedding Models
 
 With the `[full]` extra installed, `ccc init` defaults to a local SentenceTransformers model ([Snowflake/snowflake-arctic-embed-xs](https://huggingface.co/Snowflake/snowflake-arctic-embed-xs)) — no API key required. To use a different model, edit `~/.cocoindex_code/global_settings.yml`.

docker/Dockerfile

@@ -3,15 +3,15 @@
 #
 # Stable layers (reuse across releases — digest reproducible from the RUN
 # command string + base image, so users keep them in local cache):
-#   1. apt install gosu + create coco user
+#   1. apt install gosu + git + create coco user
 #   2. install uv
 #   3. (full only) `uv pip install sentence-transformers` — ~1 GB of torch +
 #      transformers. This is the heavy, slow-changing layer we're optimizing
 #      around.
 #   4. (full only) pre-bake the default embedding model under
 #      /var/cocoindex/cache/... so the named volume's copy-up populates it
 #      on first start without a network fetch.
-#   5. writable-path setup (mkdir /var/cocoindex/db + /var/run/cocoindex_code,
+#   5. writable-path setup (mkdir /var/cocoindex/state + /var/cocoindex/db + /var/run/cocoindex_code,
 #      chown to coco) + env vars + entrypoint copy.
 #
 # Per-release layers (invalidate when the source tree changes):
@@ -36,7 +36,7 @@
 FROM python:3.12-slim
 
 RUN apt-get update \
-    && apt-get install -y --no-install-recommends gosu \
+    && apt-get install -y --no-install-recommends git gosu \
     && rm -rf /var/lib/apt/lists/* \
     && groupadd -g 1000 coco \
     && useradd -u 1000 -g 1000 -m coco
@@ -65,14 +65,15 @@ RUN mkdir -p /var/cocoindex/cache/huggingface /var/cocoindex/cache/sentence-tran
 # entrypoint re-chowns to the host user; under root (Docker Desktop
 # default) coco-ownership is harmless since processes run as root and can
 # write anywhere.
-RUN mkdir -p /var/cocoindex/db /var/run/cocoindex_code \
+RUN mkdir -p /var/cocoindex/state /var/cocoindex/db /var/cocoindex/cache/huggingface /var/cocoindex/cache/sentence-transformers /var/run/cocoindex_code \
     && chown -R coco:coco /var/cocoindex /var/run/cocoindex_code
 
 WORKDIR /workspace
 
 # Runtime defaults — see the spec for what each does. All overridable at
 # `docker run -e ...` time.
 ENV COCOINDEX_CODE_DIR=/workspace/.cocoindex_code \
+    COCOINDEX_CODE_STATE_DIR=/var/cocoindex/state \
     COCOINDEX_CODE_RUNTIME_DIR=/var/run/cocoindex_code \
     COCOINDEX_CODE_DB_PATH_MAPPING=/workspace=/var/cocoindex/db \
     COCOINDEX_CODE_DAEMON_SUPERVISED=1

docker/docker-compose.yml

@@ -13,23 +13,47 @@
 # Override the image via COCOINDEX_CODE_IMAGE — for example:
 #   COCOINDEX_CODE_IMAGE=cocoindex/cocoindex-code:full docker compose up -d
 #   COCOINDEX_CODE_IMAGE=ghcr.io/cocoindex-io/cocoindex-code:latest docker compose up -d
+#
+# Optional knobs:
+#   COCOINDEX_CODE_CONTAINER_NAME=my-ccc
+#   COCOINDEX_CODE_STATE_DIR=/var/cocoindex/state
+#   COCOINDEX_CODE_RUNTIME_DIR=/var/run/cocoindex_code
+#   COCOINDEX_CODE_DB_PATH_MAPPING=/workspace=/var/cocoindex/db
+#
+# For Git layered indexing, mount a workspace parent that contains both the
+# root clone and linked worktrees. Keep COCOINDEX_CODE_STATE_DIR on the
+# persistent cocoindex-data volume so base/branch layers survive container
+# recreation. See docs/docker-layered-indexing.md.
 
 services:
   cocoindex-code:
     image: ${COCOINDEX_CODE_IMAGE:-cocoindex/cocoindex-code:latest}
-    container_name: cocoindex-code
+    container_name: ${COCOINDEX_CODE_CONTAINER_NAME:-cocoindex-code}
     volumes:
       - ${COCOINDEX_HOST_WORKSPACE:-${HOME}}:/workspace
       - cocoindex-data:/var/cocoindex
     environment:
+      COCOINDEX_CODE_STATE_DIR: ${COCOINDEX_CODE_STATE_DIR:-/var/cocoindex/state}
+      COCOINDEX_CODE_RUNTIME_DIR: ${COCOINDEX_CODE_RUNTIME_DIR:-/var/run/cocoindex_code}
+      COCOINDEX_CODE_DB_PATH_MAPPING: ${COCOINDEX_CODE_DB_PATH_MAPPING:-/workspace=/var/cocoindex/db}
       # Makes CLI and MCP output show your real paths
       # (e.g. `/Users/you/myproject/...`) instead of container paths
       # (e.g. `/workspace/myproject/...`).
-      COCOINDEX_CODE_HOST_PATH_MAPPING: /workspace=${COCOINDEX_HOST_WORKSPACE:-${HOME}}
+      COCOINDEX_CODE_HOST_PATH_MAPPING: ${COCOINDEX_CODE_HOST_PATH_MAPPING:-/workspace=${COCOINDEX_HOST_WORKSPACE:-${HOME}}}
       # Linux only: set these so files written to your workspace are owned by
       # you rather than root. Not needed on macOS / Windows — leave empty.
       PUID: ${PUID:-}
       PGID: ${PGID:-}
+    healthcheck:
+      test:
+        [
+          "CMD-SHELL",
+          "ccc daemon status >/dev/null 2>&1 || test -S /var/run/cocoindex_code/daemon.sock",
+        ]
+      interval: 10s
+      timeout: 5s
+      retries: 12
+      start_period: 10s
 
 volumes:
   cocoindex-data: