[BUG] Cloud database indexing fails after successful bisync - files in S3 but not indexed

[DonaldoDes]

[BUG] Cloud database indexing fails after successful bisync - files in S3 but not indexed

Bug Description

After a successful bm project bisync --resync, files are correctly synced to S3 storage but the cloud database indexing does not process the files. The CLI reports "Database sync initiated" but MCP tools (list_directory, search_notes, etc.) return empty or incomplete results despite all files being present in cloud storage.

Steps To Reproduce

1. Install version 0.16.2 via Homebrew
2. Authenticate with bm cloud login
3. Create a cloud project with local sync:

   bm project add myproject --local-path ~/path/to/vault

4. Run initial bisync:

   bm project bisync --name myproject --resync

5. Bisync completes successfully with message:

   myproject bisync completed successfully
   Database sync initiated: Filesystem sync initiated for project 'myproject'
   

6. Verify files are in S3:

   rclone ls "basic-memory-cloud:production-basic-memory-tenant-XXXXX/myproject/" | wc -l
   # Returns: 365 files

7. Check indexed entities:

   bm project info myproject --json | python3 -c "import sys,json; d=json.load(sys.stdin); print('Entities:', d['statistics']['total_entities'])"
   # Returns: 0

8. MCP tools return empty/incomplete results

Expected Behavior

After bisync completes and "Database sync initiated" message appears, the cloud database should index all synced files within a reasonable time. MCP tools should then be able to query and list all ~365 files.

Actual Behavior

• Files are correctly synced to S3 storage (verified via rclone ls)
• "Database sync initiated" message appears
• bm status --project myproject shows "No changes"
• bm project info myproject shows 0 entities (or very few, ~32 out of 365)
• MCP list_directory returns incomplete results
• MCP search_notes returns empty results for terms that exist in synced files
• Waiting 30+ minutes does not resolve the issue
• Deleting and recreating the project does not resolve the issue

Environment

• OS: macOS (Apple Silicon)
• Python version: 3.14
• Basic Memory version: 0.16.2
• Installation method: Homebrew
• Claude Desktop version: N/A (using Claude.ai with Remote MCP)

Additional Context

What works:

• bm cloud login and authentication
• bm cloud setup and rclone configuration
• bm project bisync file transfer to S3
• rclone ls confirms all 365 files in bucket

What doesn't work:

• Cloud database indexing after bisync
• MCP tools querying the indexed data

Code path analysis:

Looking at the CLI source (project.py), after successful bisync it calls:

response = await call_post(client, f"/{permalink}/project/sync?force_full=true", json={})

This triggers sync_service.sync() as a background task in project_router.py. The background task appears to either not execute properly, not find the files in the mounted S3 bucket, or fail silently.

Attempted remediation:

• Multiple bisync attempts with --resync flag
• Waiting extended periods (30+ minutes)
• Full project deletion and recreation (bm project remove --delete-notes then bm project add)
• All attempts result in same behavior: files in S3, no indexing

Possible Solution

The issue seems to be server-side in the background sync task that runs after bisync. Possible causes:

1. The background task may not be finding/scanning files in the mounted S3 bucket
2. There may be a watermark/cache issue preventing full scan even with force_full=true
3. The task may be timing out or hitting a circuit breaker on large file counts
4. The task may be completing but not committing changes to the database

A way to check server-side logs or manually trigger a full reindex would help diagnose. Happy to provide additional debugging information or test specific scenarios if needed.

---

Note: I'm available for further discussion and debugging if that would help.

[groksrc]

Duplicate of #460 - root cause identified as missing TigrisFS folder markers. See #460 for details and fix options.