Skip to content

Commit 0b30f16

Browse files
georgeh0claude
andauthored
feat: assorted improvements for CLI/daemon to be resilient and ergonomic (#72)
* feat: assorted improvements for CLI/daemon to be resilient and ergonomic * tests: add more e2e tests * fix: resolve path comparison and LMDB cleanup issues on Windows/3.14t Three root causes fixed: 1. Path resolution mismatch on Windows: init and auto_init_project used unresolved Path.cwd() but find_parent_with_marker/find_project_root resolve internally, causing comparison failures and daemon key mismatches. 2. LMDB not released on remove_project: The daemon ProjectRegistry dropped the Project from dicts without closing its SQLite connection or forcing GC of the Rust LMDB environment. On free-threaded Python (3.14t) and Windows, deferred GC kept the LMDB open, causing environment already open errors and PermissionErrors when deleting db files. 3. Silent connection close on streaming errors: When update_index async iteration failed in the daemon, the connection was closed without sending an ErrorResponse, causing the client to get an unhelpful EOFError. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: break LMDB reference chain explicitly and resolve cwd in path filter On 3.14t (free-threaded Python), gc.collect() alone does not release the Rust LMDB environment because deferred reference counting keeps core.Environment alive through App._core_env_app, ContextProvider._core_env, and Environment._core_env. Explicitly null these internal references in Project.close() before gc.collect() so the Rust object is freed promptly. Also resolve Path.cwd() in resolve_default_path() — on Windows, the unresolved cwd did not match the resolved project_root, causing relative_to() to fail and the subdirectory path filter to be skipped. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use double gc.collect() for layered deferred refcount on 3.14t Revert fragile internal attribute clearing. On free-threaded Python, the first gc.collect() frees Python wrappers whose Rust Drop implementations issue further deferred Py_DECREF calls on core.Environment; a second gc.collect() flushes those and actually drops the LMDB handle. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add sleep in tests for LMDB release on free-threaded Python On 3.14t, deferred reference counting means the Rust LMDB environment is not released immediately after remove_project + gc.collect(). Add a 1-second sleep in the two tests that reset and re-index, giving the runtime time to process pending deferred Py_DECREF calls. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: restart daemon after reset in tests, increase daemon wait timeout Instead of sleeping, restart the daemon after reset in the two tests that re-index after removing databases. This reliably releases the LMDB environment on all platforms including free-threaded Python (3.14t). Also increase _wait_for_daemon timeout from 5s to 10s — Windows CI runners occasionally need longer to start the daemon subprocess. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 5e83126 commit 0b30f16

9 files changed

Lines changed: 713 additions & 161 deletions

File tree

src/cocoindex_code/cli.py

Lines changed: 224 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@
3232

3333

3434
# ---------------------------------------------------------------------------
35-
# Shared CLI helpers (G1)
35+
# Shared CLI helpers
3636
# ---------------------------------------------------------------------------
3737

3838

@@ -70,7 +70,7 @@ def require_daemon_for_project() -> tuple[DaemonClient, str]:
7070

7171
def resolve_default_path(project_root: Path) -> str | None:
7272
"""Compute default ``--path`` filter from CWD relative to project root."""
73-
cwd = Path.cwd()
73+
cwd = Path.cwd().resolve()
7474
try:
7575
rel = cwd.relative_to(project_root)
7676
except ValueError:
@@ -120,8 +120,126 @@ def print_search_results(response: SearchResponse) -> None:
120120
_typer.echo(r.content)
121121

122122

123+
def _run_index_with_progress(client: DaemonClient, project_root: str) -> None:
124+
"""Run indexing with streaming progress display. Exits on failure."""
125+
from rich.console import Console as _Console
126+
from rich.live import Live as _Live
127+
from rich.spinner import Spinner as _Spinner
128+
129+
err_console = _Console(stderr=True)
130+
last_progress_line: str | None = None
131+
132+
with _Live(_Spinner("dots", "Indexing..."), console=err_console, transient=True) as live:
133+
134+
def _on_waiting() -> None:
135+
live.update(
136+
_Spinner(
137+
"dots",
138+
"Another indexing is ongoing, waiting for it to finish...",
139+
)
140+
)
141+
142+
def _on_progress(progress: IndexingProgress) -> None:
143+
nonlocal last_progress_line
144+
last_progress_line = f"Indexing: {_format_progress(progress)}"
145+
live.update(_Spinner("dots", last_progress_line))
146+
147+
try:
148+
resp = client.index(project_root, on_progress=_on_progress, on_waiting=_on_waiting)
149+
except RuntimeError as e:
150+
live.stop()
151+
_typer.echo(f"Indexing failed: {e}", err=True)
152+
raise _typer.Exit(code=1)
153+
154+
# Print the final progress line so it remains visible after the spinner clears
155+
if last_progress_line is not None:
156+
_typer.echo(last_progress_line, err=True)
157+
158+
if not resp.success:
159+
_typer.echo(f"Indexing failed: {resp.message}", err=True)
160+
raise _typer.Exit(code=1)
161+
162+
163+
_GITIGNORE_COMMENT = "# cocoindex-code"
164+
_GITIGNORE_ENTRY = "/.cocoindex_code/"
165+
166+
167+
def add_to_gitignore(project_root: Path) -> None:
168+
"""Add ``/.cocoindex_code/`` to ``.gitignore`` if ``.git`` exists.
169+
170+
Creates ``.gitignore`` if it doesn't exist. Skips if the entry is already
171+
present.
172+
"""
173+
if not (project_root / ".git").is_dir():
174+
return
175+
176+
gitignore = project_root / ".gitignore"
177+
if gitignore.is_file():
178+
content = gitignore.read_text()
179+
if _GITIGNORE_ENTRY in content.splitlines():
180+
return # already present
181+
# Ensure a trailing newline before appending
182+
if content and not content.endswith("\n"):
183+
content += "\n"
184+
content += f"{_GITIGNORE_COMMENT}\n{_GITIGNORE_ENTRY}\n"
185+
gitignore.write_text(content)
186+
else:
187+
gitignore.write_text(f"{_GITIGNORE_COMMENT}\n{_GITIGNORE_ENTRY}\n")
188+
189+
190+
def remove_from_gitignore(project_root: Path) -> None:
191+
"""Remove ``/.cocoindex_code/`` entry and its comment from ``.gitignore``."""
192+
gitignore = project_root / ".gitignore"
193+
if not gitignore.is_file():
194+
return
195+
196+
lines = gitignore.read_text().splitlines(keepends=True)
197+
new_lines: list[str] = []
198+
i = 0
199+
while i < len(lines):
200+
stripped = lines[i].rstrip("\n\r")
201+
if stripped == _GITIGNORE_ENTRY:
202+
# Skip this line; also remove preceding comment if it matches
203+
if new_lines and new_lines[-1].rstrip("\n\r") == _GITIGNORE_COMMENT:
204+
new_lines.pop()
205+
i += 1
206+
continue
207+
new_lines.append(lines[i])
208+
i += 1
209+
gitignore.write_text("".join(new_lines))
210+
211+
212+
def auto_init_project() -> Path:
213+
"""Auto-initialize project from CWD.
214+
215+
Runs core ``init`` logic without parent-directory confirmation and without
216+
the "run ``ccc index``" prompt. Returns the project root (CWD).
217+
"""
218+
from .settings import project_settings_path
219+
220+
cwd = Path.cwd().resolve()
221+
settings_file = project_settings_path(cwd)
222+
223+
if not settings_file.is_file():
224+
# Create user settings if missing
225+
user_path = user_settings_path()
226+
if not user_path.is_file():
227+
save_user_settings(default_user_settings())
228+
_typer.echo(f"Created user settings: {user_path}")
229+
230+
# Create project settings
231+
save_project_settings(cwd, default_project_settings())
232+
_typer.echo(f"Created project settings: {settings_file}")
233+
_typer.echo("You can edit the settings files to customize indexing behavior.")
234+
235+
# Update .gitignore
236+
add_to_gitignore(cwd)
237+
238+
return cwd
239+
240+
123241
# ---------------------------------------------------------------------------
124-
# Commands (G2-G5)
242+
# Commands
125243
# ---------------------------------------------------------------------------
126244

127245

@@ -130,10 +248,10 @@ def init(
130248
force: bool = _typer.Option(False, "-f", "--force", help="Skip parent directory warning"),
131249
) -> None:
132250
"""Initialize a project for cocoindex-code."""
133-
from .settings import project_settings_path
251+
from .settings import project_settings_path as _project_settings_path
134252

135-
cwd = Path.cwd()
136-
settings_file = project_settings_path(cwd)
253+
cwd = Path.cwd().resolve()
254+
settings_file = _project_settings_path(cwd)
137255

138256
# Check if already initialized
139257
if settings_file.is_file():
@@ -160,49 +278,32 @@ def init(
160278
# Create project settings
161279
save_project_settings(cwd, default_project_settings())
162280
_typer.echo(f"Created project settings: {settings_file}")
163-
_typer.echo("Project initialized. Run `ccc index` to build the index.")
281+
282+
# Add to .gitignore
283+
add_to_gitignore(cwd)
284+
285+
_typer.echo("You can edit the settings files to customize indexing behavior.")
286+
_typer.echo("Run `ccc index` to build the index.")
164287

165288

166289
@app.command()
167290
def index() -> None:
168291
"""Create/update index for the codebase."""
169-
from rich.console import Console as _Console
170-
from rich.live import Live as _Live
171-
from rich.spinner import Spinner as _Spinner
172-
173-
client, project_root = require_daemon_for_project()
174-
err_console = _Console(stderr=True)
175-
last_progress_line: str | None = None
176-
177-
with _Live(_Spinner("dots", "Indexing..."), console=err_console, transient=True) as live:
178-
179-
def _on_waiting() -> None:
180-
live.update(
181-
_Spinner(
182-
"dots",
183-
"Another indexing is ongoing, waiting for it to finish...",
184-
)
185-
)
186-
187-
def _on_progress(progress: IndexingProgress) -> None:
188-
nonlocal last_progress_line
189-
last_progress_line = f"Indexing: {_format_progress(progress)}"
190-
live.update(_Spinner("dots", last_progress_line))
191-
192-
try:
193-
resp = client.index(project_root, on_progress=_on_progress, on_waiting=_on_waiting)
194-
except RuntimeError as e:
195-
live.stop()
196-
_typer.echo(f"Indexing failed: {e}", err=True)
197-
raise _typer.Exit(code=1)
292+
from .client import ensure_daemon
198293

199-
# Print the final progress line so it remains visible after the spinner clears
200-
if last_progress_line is not None:
201-
_typer.echo(last_progress_line, err=True)
294+
# Auto-init if not in an initialized project
295+
root = find_project_root(Path.cwd())
296+
if root is None:
297+
root = auto_init_project()
202298

203-
if not resp.success:
204-
_typer.echo(f"Indexing failed: {resp.message}", err=True)
299+
try:
300+
client = ensure_daemon()
301+
except Exception as e:
302+
_typer.echo(f"Error: Failed to connect to daemon: {e}", err=True)
205303
raise _typer.Exit(code=1)
304+
project_root = str(root)
305+
306+
_run_index_with_progress(client, project_root)
206307

207308
status = client.project_status(project_root)
208309
print_index_stats(status)
@@ -221,6 +322,10 @@ def search(
221322
client, project_root = require_daemon_for_project()
222323
query_str = " ".join(query)
223324

325+
# Refresh index with progress display before searching
326+
if refresh:
327+
_run_index_with_progress(client, project_root)
328+
224329
# Default path filter from CWD
225330
paths: list[str] | None = None
226331
if path is not None:
@@ -237,7 +342,7 @@ def search(
237342
paths=paths,
238343
limit=limit,
239344
offset=offset,
240-
refresh=refresh,
345+
refresh=False,
241346
)
242347
print_search_results(resp)
243348

@@ -250,6 +355,82 @@ def status() -> None:
250355
print_index_stats(resp)
251356

252357

358+
@app.command()
359+
def reset(
360+
all_: bool = _typer.Option(False, "--all", help="Also remove settings and .gitignore entry"),
361+
force: bool = _typer.Option(False, "-f", "--force", help="Skip confirmation"),
362+
) -> None:
363+
"""Reset project databases and optionally remove settings."""
364+
project_root = require_project_root()
365+
cocoindex_dir = project_root / ".cocoindex_code"
366+
367+
db_files = [
368+
cocoindex_dir / "cocoindex.db",
369+
cocoindex_dir / "target_sqlite.db",
370+
]
371+
settings_file = cocoindex_dir / "settings.yml"
372+
373+
# Determine what will be deleted
374+
to_delete = [f for f in db_files if f.exists()]
375+
if all_:
376+
if settings_file.exists():
377+
to_delete.append(settings_file)
378+
379+
if not to_delete and not all_:
380+
_typer.echo("Nothing to reset.")
381+
return
382+
383+
# Show what will be deleted
384+
if to_delete:
385+
_typer.echo("The following files will be deleted:")
386+
for f in to_delete:
387+
_typer.echo(f" {f}")
388+
389+
# Confirm
390+
if not force:
391+
if not _typer.confirm("Proceed?"):
392+
_typer.echo("Aborted.")
393+
raise _typer.Exit(code=0)
394+
395+
# Remove project from daemon first so it releases file handles
396+
try:
397+
from .client import DaemonClient
398+
399+
client = DaemonClient.connect()
400+
client.handshake()
401+
client.remove_project(str(project_root))
402+
client.close()
403+
except (ConnectionRefusedError, OSError, RuntimeError):
404+
pass # Daemon not running — that's fine
405+
406+
# Delete files/directories
407+
import shutil as _shutil
408+
409+
for f in to_delete:
410+
if f.is_dir():
411+
_shutil.rmtree(f)
412+
else:
413+
f.unlink(missing_ok=True)
414+
415+
if all_:
416+
# Remove .cocoindex_code/ if empty
417+
try:
418+
cocoindex_dir.rmdir()
419+
except OSError:
420+
pass # Not empty
421+
422+
# Remove from .gitignore
423+
remove_from_gitignore(project_root)
424+
_typer.echo("Project fully reset.")
425+
else:
426+
_typer.echo("Databases deleted.")
427+
if settings_file.exists():
428+
_typer.echo(
429+
"Settings file still exists. Run `ccc reset --all` to remove it too,\n"
430+
"or edit it manually."
431+
)
432+
433+
253434
@app.command()
254435
def mcp() -> None:
255436
"""Run as MCP server (stdio mode)."""
@@ -279,7 +460,7 @@ async def _bg_index(client, project_root: str) -> None: # type: ignore[no-untyp
279460
pass
280461

281462

282-
# --- Daemon subcommands (G5) ---
463+
# --- Daemon subcommands ---
283464

284465

285466
@daemon_app.command("status")

src/cocoindex_code/client.py

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,8 @@
2626
IndexWaitingNotice,
2727
ProjectStatusRequest,
2828
ProjectStatusResponse,
29+
RemoveProjectRequest,
30+
RemoveProjectResponse,
2931
Request,
3032
Response,
3133
SearchRequest,
@@ -72,7 +74,10 @@ def index(
7274
"""Request indexing with streaming progress. Blocks until complete."""
7375
self._conn.send_bytes(encode_request(IndexRequest(project_root=project_root)))
7476
while True:
75-
data = self._conn.recv_bytes()
77+
try:
78+
data = self._conn.recv_bytes()
79+
except EOFError:
80+
raise RuntimeError("Connection to daemon lost during indexing")
7681
resp = decode_response(data)
7782
if isinstance(resp, ErrorResponse):
7883
raise RuntimeError(f"Daemon error: {resp.message}")
@@ -121,6 +126,11 @@ def daemon_status(self) -> DaemonStatusResponse:
121126

122127
return self._send(DaemonStatusRequest()) # type: ignore[return-value]
123128

129+
def remove_project(self, project_root: str) -> RemoveProjectResponse:
130+
return self._send( # type: ignore[return-value]
131+
RemoveProjectRequest(project_root=project_root)
132+
)
133+
124134
def stop(self) -> StopResponse:
125135
return self._send(StopRequest()) # type: ignore[return-value]
126136

@@ -236,7 +246,7 @@ def stop_daemon() -> None:
236246
pass
237247

238248

239-
def _wait_for_daemon(timeout: float = 5.0) -> None:
249+
def _wait_for_daemon(timeout: float = 10.0) -> None:
240250
"""Wait for the daemon socket/pipe to become available."""
241251
deadline = time.monotonic() + timeout
242252
while time.monotonic() < deadline:

0 commit comments

Comments
 (0)