Document Git layered indexing

Author: rudironsoni

README.md

@@ -61,6 +61,10 @@ Two install styles — they mirror the Docker image variants of the same names:
 
 Next, set up your [coding agent integration](#coding-agent-integration) — or jump to [Manual CLI Usage](#manual-cli-usage) if you prefer direct control.
 
+Docs:
+- [Git Layered Indexing](./docs/layered-indexing.md): configure reusable `base > branch > dirty` Git layers for root clones and linked worktrees.
+- [Docker Layered Indexing](./docs/docker-layered-indexing.md): run the layered daemon in Docker with persistent native state.
+
 ## Coding Agent Integration
 
 ### Skill (Recommended)
@@ -162,6 +166,16 @@ The background daemon starts automatically on first use.
 
 > **Tip:** `ccc index` auto-initializes if you haven't run `ccc init` yet, so you can skip straight to indexing.
 
+For Git repositories, you can configure layered indexing once from the root clone:
+
+```bash
+ccc init --base main                    # share a base layer across linked worktrees
+ccc index                               # builds base + branch + dirty layers as needed
+ccc overlay status                      # inspect the current layer stack
+```
+
+Linked worktrees reuse the same daemon-owned base layer and only index branch and dirty deltas. See [Git Layered Indexing](./docs/layered-indexing.md) for the full configuration model.
+
 ### CLI Reference
 
 | Command | Description |
@@ -170,6 +184,8 @@ The background daemon starts automatically on first use.
 | `ccc index` | Build or update the index (auto-inits if needed). Shows streaming progress. |
 | `ccc search <query>` | Semantic search across the codebase |
 | `ccc status` | Show index stats (chunk count, file count, language breakdown) |
+| `ccc overlay status` | Inspect Git layered indexing state for the current worktree |
+| `ccc overlay prune` | Prune expired branch and dirty layers |
 | `ccc mcp` | Run as MCP server in stdio mode |
 | `ccc doctor` | Run diagnostics — checks settings, daemon, model, file matching, and index health |
 | `ccc reset` | Delete index databases. `--all` also removes settings. `-f` skips confirmation. |
@@ -185,6 +201,7 @@ ccc search --lang python --lang markdown schema      # filter by language
 ccc search --path 'src/utils/*' query handler        # filter by path
 ccc search --offset 10 --limit 5 database schema     # pagination
 ccc search --refresh database schema                 # update index first, then search
+ccc index --base release/1.2                         # override Git overlay base ref once
 ```
 
 By default, `ccc search` scopes results to your current working directory (relative to the project root). Use `--path` to override.
@@ -422,6 +439,31 @@ daemon state in `/var/cocoindex/state`. That lets the local daemon Git-layer
 feature reuse daemon-owned layer metadata and materialized layer sources across
 projects while keeping transient sockets under `/var/run/cocoindex_code`.
 
+For layered indexing in Docker, initialize the base ref from the root clone and
+then use linked worktrees through the same wrapper/container:
+
+```bash
+cd $HOME/src/github/cocoindex-io/cocoindex-code
+ccc init --base main
+ccc index
+
+git worktree add ../cocoindex-code.worktrees/feature-1 -b feature-1 main
+cd ../cocoindex-code.worktrees/feature-1
+ccc index
+ccc overlay status
+```
+
+Mount a workspace parent that contains both the root clone and linked
+worktrees. For example:
+
+```bash
+COCOINDEX_HOST_WORKSPACE=$HOME/src/github/cocoindex-io \
+  docker compose -f docker/docker-compose.yml up -d
+```
+
+See [Docker Layered Indexing](./docs/docker-layered-indexing.md) for the full
+Docker setup and troubleshooting guide.
+
 ### Configuration via environment variables
 
 Pass configuration to `docker run` / compose with `-e`:
@@ -455,6 +497,11 @@ Supported Docker environment variables:
 | `COCOINDEX_CODE_DB_PATH_MAPPING` | DB/index storage remapping, default `/workspace=/var/cocoindex/db`. |
 | `PUID`, `PGID` | Linux UID/GID used to chown Docker-managed paths and write host-owned workspace files. |
 
+`COCOINDEX_CODE_STATE_DIR` is where repository/worktree metadata, overlay
+policy, layer manifests, and materialized layer sources are stored. Keep it on
+the persistent Docker volume if you want base layers to survive container
+recreation.
+
 ### Build the image locally
 
 ```bash
@@ -463,6 +510,7 @@ docker build -t cocoindex-code:local -f docker/Dockerfile .
 
 ## Features
 - **Semantic Code Search**: Find relevant code using natural language queries when grep doesn't work well, and save tokens immediately.
+- **Git Layered Indexing**: Reuse a shared base index across root clones and linked worktrees, then index only branch and dirty deltas. Configure it with `ccc init --base main`; see [Git Layered Indexing](./docs/layered-indexing.md).
 - **Ultra Performant**: ⚡ Built on top of ultra performant [Rust indexing engine](https://github.com/cocoindex-io/cocoindex). Only re-indexes changed files for fast updates.
 - **Multi-Language Support**: Python, JavaScript/TypeScript, Rust, Go, Java, C/C++, C#, SQL, Shell, and more.
 - **Embedded**: Portable and just works, no database setup required!
@@ -583,6 +631,23 @@ def my_chunker(path: Path, content: str) -> tuple[str | None, list[Chunk]]:
 
 See [`src/cocoindex_code/chunking.py`](./src/cocoindex_code/chunking.py) for the public types and [`tests/example_toml_chunker.py`](./tests/example_toml_chunker.py) for a complete example.
 
+### Git Layered Indexing Configuration
+
+For Git repositories, `ccc init --base <ref>` stores a repository-level overlay
+policy in daemon state. The checkout-local `settings.yml` still controls file
+matching and chunking, while daemon state controls the shared base ref used by
+root clones and linked worktrees.
+
+```bash
+ccc init --base main
+ccc index
+ccc overlay status
+```
+
+The daemon stores durable layer metadata under `COCOINDEX_CODE_STATE_DIR` and
+uses stable hash IDs, so moving a repository or linked worktree does not
+invalidate reusable base and branch layers. See [Git Layered Indexing](./docs/layered-indexing.md) for details.
+
 ## Embedding Models
 
 With the `[full]` extra installed, `ccc init` defaults to a local SentenceTransformers model ([Snowflake/snowflake-arctic-embed-xs](https://huggingface.co/Snowflake/snowflake-arctic-embed-xs)) — no API key required. To use a different model, edit `~/.cocoindex_code/global_settings.yml`.

docker/docker-compose.yml

@@ -19,6 +19,11 @@
 #   COCOINDEX_CODE_STATE_DIR=/var/cocoindex/state
 #   COCOINDEX_CODE_RUNTIME_DIR=/var/run/cocoindex_code
 #   COCOINDEX_CODE_DB_PATH_MAPPING=/workspace=/var/cocoindex/db
+#
+# For Git layered indexing, mount a workspace parent that contains both the
+# root clone and linked worktrees. Keep COCOINDEX_CODE_STATE_DIR on the
+# persistent cocoindex-data volume so base/branch layers survive container
+# recreation. See docs/docker-layered-indexing.md.
 
 services:
   cocoindex-code:

docs/README.md

@@ -0,0 +1,4 @@
+# CocoIndex Code Docs
+
+- [Git Layered Indexing](./layered-indexing.md): configuration model, stable IDs, layer stack behavior, and commands.
+- [Docker Layered Indexing](./docker-layered-indexing.md): Docker-specific state layout, wrapper, and linked worktree setup.

docs/docker-layered-indexing.md

@@ -0,0 +1,143 @@
+# Docker Layered Indexing
+
+This guide covers the Docker-specific configuration for Git layered indexing. For the core model, see [Git Layered Indexing](./layered-indexing.md).
+
+## Recommended Compose Setup
+
+Use the repository compose file:
+
+```bash
+docker compose -f docker/docker-compose.yml up -d
+```
+
+The compose defaults are designed for layered indexing:
+
+```yaml
+COCOINDEX_CODE_STATE_DIR: /var/cocoindex/state
+COCOINDEX_CODE_RUNTIME_DIR: /var/run/cocoindex_code
+COCOINDEX_CODE_DB_PATH_MAPPING: /workspace=/var/cocoindex/db
+COCOINDEX_CODE_HOST_PATH_MAPPING: /workspace=$HOME
+```
+
+The important split is:
+
+- source code and settings live on the bind mount under `/workspace`
+- durable daemon layer metadata lives under `/var/cocoindex/state`
+- per-project non-layer DB paths are remapped to `/var/cocoindex/db`
+- sockets, PID files, and logs stay under `/var/run/cocoindex_code`
+
+## Mount the Right Workspace
+
+The default compose file mounts your home directory:
+
+```bash
+COCOINDEX_HOST_WORKSPACE=$HOME docker compose -f docker/docker-compose.yml up -d
+```
+
+For a narrower mount, point it at the parent containing both the root clone and linked worktrees:
+
+```bash
+COCOINDEX_HOST_WORKSPACE=$HOME/src/github/cocoindex-io \
+  docker compose -f docker/docker-compose.yml up -d
+```
+
+Example host layout:
+
+```text
+$HOME/src/github/cocoindex-io/
+  cocoindex-code/
+  cocoindex-code.worktrees/
+    feature-1/
+```
+
+Both paths must be visible inside the same container mount for the daemon to reuse repository and layer state across them.
+
+## Host Wrapper
+
+Use this wrapper so Docker commands resolve the host current directory correctly:
+
+```bash
+ccc() {
+  local container="${COCOINDEX_CODE_CONTAINER_NAME:-cocoindex-code}"
+  if [ "$(docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null)" != "true" ]; then
+    echo "cocoindex-code container is not running. Start it with: docker compose -f docker/docker-compose.yml up -d" >&2
+    return 1
+  fi
+
+  local flags=(-i)
+  if [ "${1:-}" != "mcp" ] && [ -t 0 ] && [ -t 1 ]; then
+    flags=(-it)
+  fi
+
+  docker exec "${flags[@]}" \
+    -e COCOINDEX_CODE_HOST_CWD="$PWD" \
+    "$container" ccc "$@"
+}
+```
+
+`COCOINDEX_CODE_HOST_CWD` is required for linked worktrees. It tells the container-side CLI which host directory you are actually in, then the path mapping translates it to `/workspace/...`.
+
+## Layered Workflow in Docker
+
+Root clone:
+
+```bash
+cd $HOME/src/github/cocoindex-io/cocoindex-code
+ccc init --base main
+ccc index
+```
+
+Linked worktree:
+
+```bash
+git worktree add ../cocoindex-code.worktrees/feature-1 -b feature-1 main
+cd ../cocoindex-code.worktrees/feature-1
+ccc index
+ccc search "query planner"
+ccc overlay status
+```
+
+The base layer is stored once under `/var/cocoindex/state` and reused by the linked worktree.
+
+## Environment Variables
+
+| Variable | Purpose |
+|---|---|
+| `COCOINDEX_CODE_IMAGE` | Image used by compose, e.g. `cocoindex/cocoindex-code:full`. |
+| `COCOINDEX_CODE_CONTAINER_NAME` | Container name used by compose and the wrapper. |
+| `COCOINDEX_HOST_WORKSPACE` | Host directory mounted at `/workspace`. Mount a parent that contains all worktrees you want to share. |
+| `COCOINDEX_CODE_HOST_PATH_MAPPING` | Container-to-host path mapping for display and host CWD translation. |
+| `COCOINDEX_CODE_HOST_CWD` | Host current directory passed per `docker exec` invocation. |
+| `COCOINDEX_CODE_STATE_DIR` | Durable daemon layer state. Default: `/var/cocoindex/state`. |
+| `COCOINDEX_CODE_RUNTIME_DIR` | Runtime socket/PID/log directory. Default: `/var/run/cocoindex_code`. |
+| `COCOINDEX_CODE_DB_PATH_MAPPING` | Non-layer project DB remapping. Default: `/workspace=/var/cocoindex/db`. |
+| `PUID`, `PGID` | Linux-only ownership mapping for bind-mounted files and Docker-managed state. |
+
+## Debugging
+
+Check daemon status:
+
+```bash
+docker exec cocoindex-code ccc daemon status
+```
+
+Inspect overlay status from the current host directory:
+
+```bash
+ccc overlay status
+```
+
+Inspect state in the container:
+
+```bash
+docker exec -it cocoindex-code sh
+ls -R /var/cocoindex/state
+```
+
+Reset all Docker-managed index, layer, and cache state:
+
+```bash
+docker compose -f docker/docker-compose.yml down -v
+```
+
+This preserves your source tree because it is bind-mounted from the host.