feat: Implement cloud mount CLI commands for local file access (#306)

Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>

Author: phernandez

justfile

@@ -18,9 +18,12 @@ test-int:
 # Run all tests
 test: test-unit test-int
 
+# Lint and fix code (calls fix)
+lint: fix
+
 # Lint and fix code
-lint:
-    uv run ruff check . --fix
+fix:
+    uv run ruff check --fix --unsafe-fixes src tests
 
 # Type check code
 typecheck:

specs/SPEC-7 POC to spike Tigris Turso for local access to cloud data.md

@@ -0,0 +1,193 @@
+---
+title: 'SPEC-7: POC to spike Tigris/Turso for local access to cloud data'
+type: spec
+permalink: specs/spec-7-poc-tigris-turso-local-access-cloud-data
+tags:
+- poc
+- tigris
+- turso
+- cloud-storage
+- architecture
+- proof-of-concept
+---
+
+# SPEC-7: POC to spike Tigris/Turso for local access to cloud data
+
+## Why
+
+Current basic-memory-cloud architecture uses Fly volumes for tenant file storage, which creates several limitations:
+
+1. **Storage Scalability**: Fly volumes require pre-provisioning and don't auto-scale with usage
+2. **Cost Model**: Volume pricing vs object storage pricing may be less favorable at scale
+3. **Local Development**: No way for users to mount their cloud tenant files locally for real-time editing
+4. **Multi-Region**: Volumes are region-locked, limiting global deployment flexibility
+5. **Backup/Disaster Recovery**: Object storage provides better durability and replication options
+
+The core insight is that Basic Memory requires POSIX filesystem semantics but could benefit from object storage durability and accessibility. By combining:
+- **Tigris object storage** for file persistence (via rclone mount)
+- **Turso/libSQL** for SQLite indexing (replacing local .db files)
+
+We could enable a revolutionary user experience: **local editing of cloud-stored files** while maintaining Basic Memory's existing filesystem assumptions.
+
+## What
+
+This specification defines a proof-of-concept to validate the technical feasibility of the Tigris/Turso architecture for basic-memory-cloud tenants.
+
+**Affected Areas:**
+- **Storage Architecture**: Replace Fly volumes with Tigris object storage
+- **Database Architecture**: Replace local SQLite with Turso remote database
+- **Container Setup**: Add rclone mounting in tenant containers
+- **Local Development**: Enable local mounting of cloud tenant data
+- **Basic Memory Core**: Validate unchanged operation over mounted filesystems
+
+**Key Components:**
+- **Tigris Storage**: S3-compatible object storage via Fly.io integration
+- **rclone NFS Mount**: Native NFS mounting without FUSE dependencies
+- **Turso Database**: Hosted libSQL for SQLite replacement
+- **Single-Tenant Model**: One bucket + one database per tenant (simplified isolation)
+
+## How (High Level)
+
+### Phase 1: Local POC Validation
+- [ ] Set up Tigris bucket with test data
+- [ ] Configure rclone NFS mount locally
+- [ ] Test Basic Memory operations over mounted filesystem
+- [ ] Measure performance characteristics and identify issues
+- [ ] Validate file watching, sync operations, and concurrent access patterns
+
+### Phase 2: Database Migration
+- [ ] Set up Turso account and test database
+- [ ] Modify Basic Memory to accept external DATABASE_URL
+- [ ] Test all operations with remote SQLite via Turso
+- [ ] Validate performance and functionality parity
+
+### Phase 3: Container Integration
+- [ ] Create container image with rclone + NFS support
+- [ ] Implement tenant-specific credential management
+- [ ] Test container startup with automatic mounting
+- [ ] Validate isolation between tenant containers
+
+### Phase 4: Local Access Validation
+- [ ] Test local rclone mounting of tenant data
+- [ ] Validate real-time file editing experience
+- [ ] Test conflict resolution and sync behavior
+- [ ] Measure latency impact on user experience
+
+### Architecture Overview
+```
+Local Development:
+┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
+│ Local rclone    │───▶│ Tigris Bucket   │◀───│ Tenant Container│
+│ NFS Mount       │    │ (S3 storage)    │    │ rclone mount    │
+└─────────────────┘    └─────────────────┘    └─────────────────┘
+         │                                              │
+         ▼                                              ▼
+┌─────────────────┐                            ┌─────────────────┐
+│ Basic Memory    │                            │ Basic Memory    │
+│ (local files)   │                            │ (mounted files) │
+└─────────────────┘                            └─────────────────┘
+         │                                              │
+         ▼                                              ▼
+┌─────────────────┐                            ┌─────────────────┐
+│ Turso Database  │◀───────────────────────────│ Turso Database  │
+│ (shared index)  │                            │ (shared index)  │
+└─────────────────┘                            └─────────────────┘
+```
+
+## How to Evaluate
+
+### Success Criteria
+- [ ] **Filesystem Compatibility**: Basic Memory operates without modification over rclone-mounted Tigris storage
+- [ ] **Performance Acceptable**: File operations complete within 2x local filesystem latency
+- [ ] **Database Functionality**: All Basic Memory features work with Turso remote SQLite
+- [ ] **Container Reliability**: Tenant containers start successfully with automatic mounting
+- [ ] **Local Access**: Users can mount and edit cloud files locally with real-time sync
+- [ ] **Data Isolation**: Tenant data remains properly isolated using bucket/database separation
+
+### Testing Procedure
+1. **Local Filesystem Test**:
+   ```bash
+   # Mount Tigris bucket locally
+   rclone nfsmount tigris:test-bucket ~/tigris-test --vfs-cache-mode writes
+
+   # Run Basic Memory operations
+   cd ~/tigris-test && basic-memory sync --watch
+   # Test: create notes, search, file watching, bulk operations
+   ```
+
+2. **Database Migration Test**:
+   ```bash
+   # Configure Turso connection
+   export DATABASE_URL="libsql://test-db.turso.io?authToken=..."
+
+   # Test all MCP tools with remote database
+   basic-memory tools # Test each tool functionality
+   ```
+
+3. **Container Integration Test**:
+   ```dockerfile
+   # Test container with rclone mounting
+   FROM python:3.12
+   RUN apt-get update && apt-get install -y rclone nfs-common
+   # ... test startup and mounting process
+   ```
+
+4. **Performance Benchmarking**:
+   - File creation/read/write operations (target: <2x local latency)
+   - Search query performance (target: comparable to local SQLite)
+   - File watching responsiveness (target: events within 1-2 seconds)
+   - Concurrent operation handling
+
+### Risk Assessment
+**High Risk Items**:
+- [ ] NFS-over-S3 performance may be insufficient for real-time operations
+- [ ] File watching (`inotify`) over NFS may be unreliable
+- [ ] Network interruptions could cause filesystem errors
+- [ ] Concurrent access patterns might hit S3 rate limits
+
+**Mitigation Strategies**:
+- Comprehensive performance testing before committing to architecture
+- Fallback plan to S3-native storage backend if filesystem approach fails
+- Extensive error handling and retry logic for network issues
+
+### Metrics to Track
+- **Latency**: File operation response times (read/write/watch)
+- **Reliability**: Success rate of file operations over time
+- **Throughput**: Concurrent file operations and search queries
+- **User Experience**: Perceived performance for local mounting use case
+
+## Notes
+
+### Key Architectural Decisions
+- **Single tenant per bucket/database**: Simplifies isolation and credential management
+- **Maintain POSIX compatibility**: Preserve Basic Memory's existing filesystem assumptions
+- **NFS over FUSE**: Better compatibility and performance characteristics
+- **Turso for SQLite**: Leverages specialized remote SQLite expertise
+
+### Alternative Approaches Considered
+- **S3-native storage backend**: Would require Basic Memory architecture changes
+- **Hybrid approach**: Local files + cloud sync (adds complexity)
+- **FUSE mounting**: More platform dependencies and kernel requirements
+
+### Integration Points
+- [ ] Fly.io Tigris integration for bucket provisioning
+- [ ] Turso account setup and database provisioning
+- [ ] Container image modifications for rclone support
+- [ ] Credential management for tenant isolation
+
+## Observations
+
+- [architecture] Tigris/Turso split cleanly separates file storage from indexing concerns #storage-separation
+- [user-experience] Local mounting of cloud files could be revolutionary for knowledge management #local-cloud-hybrid
+- [compatibility] Maintaining POSIX filesystem assumptions preserves Basic Memory's local/cloud compatibility #architecture-preservation
+- [simplification] Single tenant per bucket eliminates complex multi-tenancy in storage layer #tenant-isolation
+- [risk] NFS-over-S3 performance characteristics are unproven for real-time operations #performance-risk
+- [benefit] Object storage pricing model could be more favorable than volume pricing #cost-optimization
+- [innovation] Real-time local editing of cloud-stored files addresses major SaaS limitation #competitive-advantage
+
+## Relations
+
+- implements [[SPEC-6 Explicit Project Parameter Architecture]]
+- requires [[Fly.io Tigris Integration]]
+- enables [[Local Cloud File Access]]
+- alternative_to [[Fly Volume Storage]]

src/basic_memory/cli/commands/cloud.py

@@ -11,78 +11,25 @@
 
 from basic_memory.cli.app import cloud_app
 from basic_memory.cli.auth import CLIAuth
-from basic_memory.config import ConfigManager
+from basic_memory.cli.commands.cloud.api_client import (
+    CloudAPIError,
+    get_cloud_config,
+    make_api_request,
+    get_authenticated_headers,
+)
+from basic_memory.cli.commands.cloud.mount_commands import (
+    mount_cloud_files,
+    setup_cloud_mount,
+    show_mount_status,
+    unmount_cloud_files,
+)
+from basic_memory.cli.commands.cloud.rclone_config import MOUNT_PROFILES
 from basic_memory.ignore_utils import load_gitignore_patterns, should_ignore_path
 from basic_memory.utils import generate_permalink
 
 console = Console()
 
 
-class CloudAPIError(Exception):
-    """Exception raised for cloud API errors."""
-
-    pass
-
-
-def get_cloud_config() -> tuple[str, str, str]:
-    """Get cloud OAuth configuration from config."""
-    config_manager = ConfigManager()
-    config = config_manager.config
-    return config.cloud_client_id, config.cloud_domain, config.cloud_host
-
-
-async def make_api_request(
-    method: str,
-    url: str,
-    headers: Optional[dict] = None,
-    json_data: Optional[dict] = None,
-    timeout: float = 30.0,
-) -> httpx.Response:
-    """Make an API request to the cloud service."""
-    headers = headers or {}
-    auth_headers = await get_authenticated_headers()
-    headers.update(auth_headers)
-    # Add debug headers to help with compression issues
-    headers.setdefault("Accept-Encoding", "identity")  # Disable compression for debugging
-
-    async with httpx.AsyncClient(timeout=timeout) as client:
-        try:
-            console.print(f"[dim]Making {method} request to {url}[/dim]")
-            console.print(f"[dim]Headers: {dict(headers)}[/dim]")
-
-            response = await client.request(method=method, url=url, headers=headers, json=json_data)
-
-            console.print(f"[dim]Response status: {response.status_code}[/dim]")
-            console.print(f"[dim]Response headers: {dict(response.headers)}[/dim]")
-
-            response.raise_for_status()
-            return response
-        except httpx.HTTPError as e:
-            console.print(f"[red]HTTP Error details: {e}[/red]")
-            # Check if this is a response error with response details
-            if hasattr(e, "response") and e.response is not None:  # pyright: ignore [reportAttributeAccessIssue]
-                response = e.response  # type: ignore
-                console.print(f"[red]Response status: {response.status_code}[/red]")
-                console.print(f"[red]Response headers: {dict(response.headers)}[/red]")
-                try:
-                    console.print(f"[red]Response text: {response.text}[/red]")
-                except Exception:
-                    console.print("[red]Could not read response text[/red]")
-            raise CloudAPIError(f"API request failed: {e}") from e
-
-
-async def get_authenticated_headers() -> dict[str, str]:
-    """Get authentication headers with JWT token."""
-    client_id, domain, _ = get_cloud_config()
-    auth = CLIAuth(client_id=client_id, authkit_domain=domain)
-    token = await auth.get_valid_token()
-    if not token:
-        console.print("[red]Not authenticated. Please run 'tenant login' first.[/red]")
-        raise typer.Exit(1)
-
-    return {"Authorization": f"Bearer {token}"}
-
-
 @cloud_app.command()
 def login():
     """Authenticate with WorkOS using OAuth Device Authorization flow."""
@@ -407,3 +354,45 @@ def status() -> None:
     except Exception as e:
         console.print(f"[red]Unexpected error: {e}[/red]")
         raise typer.Exit(1)
+
+
+# Mount commands
+
+
+@cloud_app.command("setup")
+def setup() -> None:
+    """Set up local file access with automatic rclone installation and configuration."""
+    setup_cloud_mount()
+
+
+@cloud_app.command("mount")
+def mount(
+    profile: str = typer.Option(
+        "balanced", help=f"Mount profile: {', '.join(MOUNT_PROFILES.keys())}"
+    ),
+    path: Optional[str] = typer.Option(
+        None, help="Custom mount path (default: ~/basic-memory-{tenant-id})"
+    ),
+) -> None:
+    """Mount cloud files locally for editing."""
+    try:
+        mount_cloud_files(profile_name=profile)
+    except Exception as e:
+        console.print(f"[red]Mount failed: {e}[/red]")
+        raise typer.Exit(1)
+
+
+@cloud_app.command("unmount")
+def unmount() -> None:
+    """Unmount cloud files."""
+    try:
+        unmount_cloud_files()
+    except Exception as e:
+        console.print(f"[red]Unmount failed: {e}[/red]")
+        raise typer.Exit(1)
+
+
+@cloud_app.command("mount-status")
+def mount_status() -> None:
+    """Show current mount status."""
+    show_mount_status()

src/basic_memory/cli/commands/cloud/__init__.py

@@ -0,0 +1 @@
+"""Cloud commands package."""