auto_index uses tens of GB of memory in a non-git folder containing code

[zy-aaron]

Version

codebase-memory-mcp 0.8.1

Platform

Linux (x64)

Install channel

GitHub release archive / install.sh / install.ps1

Binary variant

standard

What happened, and what did you expect?

With auto_index=true, opening an agent in a non-git folder that contains code indexes the whole tree and pushes RSS into the tens of GB. auto_index_limit=50000 does not help.

The limit guard counts files with git ls-files | wc -l (src/mcp/mcp.c:4953). In a non-git directory that returns 0, so the limit is never reached and auto-index proceeds with a full filesystem walk.

Expected: the limit applies to non-git roots as well, or auto-index skips a root that has no .git.

Reproduction

The same 60,000 files are indexed when the directory is non-git, but skipped once it is a git repo:

mkdir -p /tmp/ab-nongit /tmp/ab-git
seq 1 60000 | xargs -P8 -I{} touch /tmp/ab-nongit/f{}.py
cp /tmp/ab-nongit/*.py /tmp/ab-git/
( cd /tmp/ab-git && git init -q && git add -A && git -c user.email=t@t -c user.name=t commit -qm x )

codebase-memory-mcp config set auto_index true
INIT='{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"t","version":"1"}}}'

# non-git root: a project .db is created in the cache dir (full index ran)
( cd /tmp/ab-nongit && printf '%s\n' "$INIT" | timeout 8 codebase-memory-mcp )

# git root, same files: no .db (auto-index skipped on the file-count limit)
( cd /tmp/ab-git && printf '%s\n' "$INIT" | timeout 8 codebase-memory-mcp )

Logs

Diagnostics trajectory (memory / performance / leak issues)

Project scale (if relevant)

No response

Confirmations

• I searched existing issues and this is not a duplicate.
• My reproduction uses shareable code (a dummy snippet or a public OSS repository), not proprietary code.

[DeusData]

Hey, yes we seen this unfortunate behavior also from other reporters. This is kinda architecturally, so need to think about this a bit. For these cases the "easy" fix is to basically make a hook when you commit sth that this triggers an incremental index rather than using the auto indexer (u can turn it off as you have seen already). This will be patched once we found a good concept

[DeusData]

Adding this to the large-repo performance epic. The core acceptance criterion is that auto_index_limit must protect non-git roots too, or non-git auto-index should fail closed, so opening an agent in a large plain directory cannot accidentally walk the whole tree.