Skip to content

SCALING_AND_HARDENING.md

Latest commit

 

History

History
375 lines (266 loc) · 14.5 KB

SCALING_AND_HARDENING.md

File metadata and controls

375 lines (266 loc) · 14.5 KB
Raw

Scaling and Hardening CortexGraph

This report summarizes practical strategies to scale CortexGraph for heavier workloads and harden it for production use, while respecting its design constraints:

  • Local-first, JSONL-only storage (no databases)
  • MCP server model (short-lived tool calls, client-controlled lifecycle)
  • Strict security posture (no implicit network access, least privilege on disk)

It is organized as:

  1. Deployment and topology options
  2. Storage layout and scaling patterns
  3. Performance tuning and concurrency
  4. Security hardening and operational controls
  5. Observability, testing, and upgrade strategy

Where relevant, file paths and configuration options refer to the current codebase and docs (e.g., docs/deployment.md, docs/configuration.md, PERFORMANCE_OPTIMIZATIONS.md).

1. Deployment and Topology

1.1 Recommended Baseline Deployment

For a single user or small team:

  • Install via UV tool (preferred for isolation and updates):
    • uv tool install cortexgraph
  • Configuration location:
    • Primary: ~/.config/cortexgraph/.env
    • Fallback (dev only): ./.env in the project directory
  • Integrate as an MCP server:
    • Claude Desktop / VS Code: "command": "cortexgraph" for tool installs
    • Development: "command": "uv", "args": ["--directory", "/path/to/cortexgraph", "run", "cortexgraph"]

This baseline already isolates CortexGraph in a dedicated environment and keeps all user data local (~/.config/cortexgraph/jsonl and your Obsidian vault).

1.2 Topology Patterns

Because CortexGraph uses local JSONL and Markdown files rather than a central database, scaling is primarily a matter of topology and filesystem layout.

Pattern A: Per-User Instances (Recommended)

  • Each human user has:
    • Their own MCP client (Claude Desktop, VS Code, etc.)
    • Their own CortexGraph configuration and storage directory:
      • ~/.config/cortexgraph/jsonl for STM
      • LTM_VAULT_PATH pointing to a personal Obsidian vault
  • Benefits:
    • Strong isolation between users
    • No cross-user contention on JSONL files
    • Simple backup and restore per user
  • Use when:
    • You have multiple people on the same machine or shared environment
    • You do not need shared memory between users

Pattern B: Shared “Team Memory” Instance

  • One CortexGraph instance with:
    • Shared CORTEXGRAPH_STORAGE_PATH on a team-accessible filesystem
    • Shared LTM_VAULT_PATH pointing to a team Obsidian vault
  • Accessed by multiple MCP clients using the same command path.
  • Requirements:
    • Underlying filesystem supports safe concurrent access (e.g., local disk or properly configured network storage with file locking semantics).
    • Cooperative usage model (avoid running multiple long-lived CortexGraph processes writing to the same JSONL directory simultaneously).
  • Use when:
    • You want a shared memory graph for a team or project
    • Trust boundaries among users are relatively strong

Pattern C: Multi-Instance with Sharded Storage

  • You run N CortexGraph instances, each with its own storage root:
    • Example shard keys: user ID, project name, or tenant ID.
  • The MCP client or a higher-level orchestration layer chooses which CortexGraph instance to talk to based on the shard key.
  • Benefits:
    • Horizontal scaling without modifying CortexGraph’s storage layer
    • Clear boundaries between tenants/projects
  • Use when:
    • You have many independent tenants
    • Storage volumes per tenant may grow large

Note: Because CortexGraph is designed as a local MCP server with JSONL storage, do not introduce SQLite or other external databases. Scale out by topology (more instances, separate storage paths), not by replacing the storage backend.

2. Storage Layout and Scaling

2.1 Storage Paths

Key paths (see docs/configuration.md and docs/deployment.md):

  • STM JSONL directory:
    • CORTEXGRAPH_STORAGE_PATH=~/.config/cortexgraph/jsonl
  • LTM vault:
    • LTM_VAULT_PATH=~/Documents/Obsidian/Vault
  • LTM index:
    • LTM_INDEX_PATH=~/.config/cortexgraph/ltm_index.jsonl

For scaling and safety:

  • Use absolute paths in .env for shared or non-default locations.
  • Keep STM and LTM on the same physical disk for simplicity unless you have a strong reason to separate them (e.g., SSD for STM, larger HDD for LTM).

2.2 Storage Sizing and Growth Controls

JSONL STM files (memories.jsonl, relations.jsonl):

  • Growth is controlled by:
    • Decay and garbage collection thresholds:
      • CORTEXGRAPH_FORGET_THRESHOLD
      • Decay model parameters (CORTEXGRAPH_DECAY_MODEL, CORTEXGRAPH_*).
    • Promotion rules:
      • CORTEXGRAPH_PROMOTE_THRESHOLD
      • CORTEXGRAPH_PROMOTE_USE_COUNT
      • CORTEXGRAPH_PROMOTE_TIME_WINDOW
  • For heavy usage or long-lived systems:
    • Prefer slightly higher forget thresholds (e.g., 0.06–0.08) to keep STM manageable.
    • Run compaction and maintenance regularly (see cortexgraph-maintenance).

LTM Vault:

  • Markdown files accumulate as memories are promoted.
  • Scaling mainly depends on:
    • Git backups and pruning strategy
    • Obsidian vault organization (folders, tags)

2.3 Maintenance and Compaction

Use the maintenance CLI (see docs/deployment.md):

  • cortexgraph-maintenance stats
    • Inspect JSONL file sizes, active counts, and compaction hints.
  • cortexgraph-maintenance compact
    • Rewrite JSONL files without tombstones/duplicates.

Recommended schedule:

  • Daily or Weekly:
    • Run stats and compact if file sizes or tombstones grow significantly.
  • Monthly:
    • Review archived memories and promotion behavior.
    • Adjust thresholds if STM is growing too fast or too slow.

For large stores (thousands of memories or more):

  • Increase compaction frequency.
  • Consider per-user or per-tenant shard directories to limit individual file sizes.

3. Performance Tuning and Concurrency

PERFORMANCE_OPTIMIZATIONS.md describes several optimizations already implemented:

  • Embedding model caching
  • Lazy loading of LTM index
  • Buffered JSONL I/O and batch operations
  • Tag-based indexing and pagination
  • Async I/O support
  • Background task manager
  • Performance monitoring utilities

The following recommendations focus on how to use and configure these capabilities in a scalable deployment.

3.1 Configuration-Level Tuning

Key performance-related configuration options (see PERFORMANCE_OPTIMIZATIONS.md and config.py):

  • batch_size — batch size for bulk operations (default: 100)
  • cache_size — in-memory cache size (default: 1000)
  • enable_async_io — enable async I/O for better concurrency
  • search_timeout — search timeout in seconds (default: 5.0)

Guidelines:

  • For heavier search workloads:
    • Increase cache_size to keep more frequently accessed memories and tags in memory.
    • Ensure enable_async_io is true to avoid blocking operations.
    • Tune search_timeout to a value that balances responsiveness with completeness.
  • For heavy write workloads (many memories per session):
    • Use batch operations where possible (e.g., batch saving in tools/clients).
    • Keep batch_size moderate (50–200) to avoid large spikes in memory usage.

3.2 MCP-Level Concurrency

In MCP usage, CortexGraph is invoked via tools that run requests in isolation:

  • Each request should be treated as short-lived; the server’s internal caching smooths repeated operations within the same process.
  • To avoid JSONL write contention:
    • Prefer a single long-lived CortexGraph process per storage directory, rather than many short-lived processes writing concurrently.
    • If multiple MCP clients share a storage path, route them to the same cortexgraph command, not multiple independent processes each writing to the same files.

If you need true parallelism with separate processes:

  • Use sharding or per-user paths so each process writes to its own JSONL files.
  • Avoid multiple processes writing to the same JSONL directory unless you have verified filesystem-level locking semantics and are comfortable with the risk.

3.3 Embeddings and Semantic Search

Embeddings are optional and can be resource-intensive:

  • Enable:
    • CORTEXGRAPH_ENABLE_EMBEDDINGS=true
    • CORTEXGRAPH_EMBED_MODEL=all-MiniLM-L6-v2 (recommended lightweight model)
  • Considerations:
    • First run will download the model (~50MB).
    • Embedding computations can be CPU-heavy; on constrained machines, limit embedding usage or batch embedding operations.
  • Scaling strategies:
    • Use embeddings only where they add clear value (semantic search, clustering).
    • For low-end systems or large multi-tenant setups, consider disabling embeddings or limiting them to specific shards/users.

3.4 Background Tasks and Long-Running Operations

For expensive operations (index building, compaction, clustering):

  • Use the background task manager (see background.py) to avoid blocking active requests.
  • Schedule heavy jobs during off-peak hours:
    • LTM index rebuild
    • Large-scale clustering and consolidation
    • Full compaction of JSONL storage

Monitor performance metrics via:

  • get_performance_metrics() and reset_performance_metrics() (see performance.py and tools/performance.py).
  • Track average latencies and operation counts over time to validate tuning changes.

4. Security Hardening

CortexGraph already follows a strong security posture (see docs/security.md and SECURITY.md). The recommendations below focus on operationalizing and extending that posture.

4.1 Process Isolation and Privileges

  • Run CortexGraph under a dedicated OS user where feasible:
    • Only this user should have read/write access to:
      • ~/.config/cortexgraph/
      • The configured LTM_VAULT_PATH (or a subdirectory).
    • Avoid running under privileged accounts (e.g., root) unless absolutely necessary.
  • For containerized deployments:
    • Use non-root containers.
    • Mount ~/.config/cortexgraph/ and the Obsidian vault as volumes with minimal necessary permissions.

4.2 Filesystem Permissions

Enforce restrictive permissions consistently:

  • Config:
    • .env files should be 0600 (owner read/write only).
  • Storage directories:
    • JSONL storage (CORTEXGRAPH_STORAGE_PATH) should be 0700.
    • LTM vault should match your Obsidian practices; at minimum, only trusted users should have access.

On shared systems:

  • Avoid placing CORTEXGRAPH_STORAGE_PATH or LTM_VAULT_PATH in world-readable locations.
  • If using a shared “team memory” instance, treat the storage directory as containing sensitive personal data and restrict access accordingly.

4.3 Configuration and Secrets

  • Never commit .env files or storage data to version control.
  • When sharing diagnostics:
    • Redact file paths, personal data, and any sensitive content.
    • Prefer synthetic or anonymized examples.
  • For Git-based backups of the Obsidian vault:
    • Use private repositories.
    • Consider client-side encryption if storing sensitive data off-device.

4.4 Network and Integration Boundaries

  • CortexGraph is designed to run locally with no outbound network access by default:
    • Do not add remote API calls inside core modules or tools.
    • If you need network functionality (e.g., downloading models), confine it to explicit setup steps or separate scripts.
  • Integration with MCP clients:
    • Limit which tools are exposed to the assistant if you have stricter security requirements.
    • Review tool definitions for potential data exfiltration paths (e.g., anything that can read arbitrary files).

4.5 Supply Chain and CI Hardening

Current measures (see docs/security.md):

  • Dependabot, pip-audit, Bandit, CodeQL, SBOM generation roadmap.

Additional hardening steps:

  • Pin dependencies more tightly in pyproject.toml once the ecosystem stabilizes.
  • Enable SBOM-based scanning in CI (using snyk_sbom_scan or similar).
  • Periodically review:
    • GitHub security advisories
    • CI logs for Bandit, CodeQL, and pip-audit

5. Observability, Testing, and Upgrade Strategy

5.1 Logging and Metrics

  • Capture server logs:

    cortexgraph 2>&1 | tee ~/.config/cortexgraph/server.log

  • Use performance metrics APIs:

    • Track operation latencies (search, save, clustering).
    • Watch for regressions after configuration changes or upgrades.

For multi-instance setups:

  • Standardize log locations and formats so you can aggregate or inspect logs across instances.

5.2 Testing Under Load

Before rolling out configuration or topology changes:

  • Generate synthetic workloads:
    • Scripts that save many memories, run diverse searches, and exercise consolidation/GC.
    • Use realistic tags and content sizes.
  • Measure:
    • End-to-end latencies for key operations.
    • Growth in JSONL file sizes.
    • Memory usage of the cortexgraph process.

5.3 Upgrade and Migration Strategy

Because storage is JSONL and Markdown:

  • Upgrades are primarily schema-additive and backward compatible.
  • Good practice for upgrades:
    • Backup ~/.config/cortexgraph/jsonl/ and relevant parts of your Obsidian vault.
    • Upgrade CortexGraph (via uv tool install or pip).
    • Run tests if you are on a dev install (PYTHONPATH=src pytest).
    • Check cortexgraph-maintenance stats for inconsistencies.

For multi-instance deployments:

  • Roll out updates gradually:
    • Upgrade one instance.
    • Verify behavior under typical workload.
    • Then upgrade remaining instances.

6. Summary Checklist

Topology and Storage

  • Choose per-user, team, or sharded instances.
  • Set explicit CORTEXGRAPH_STORAGE_PATH, LTM_VAULT_PATH, LTM_INDEX_PATH.
  • Tune decay and promotion thresholds for expected load.
  • Schedule regular cortexgraph-maintenance stats and compact.

Performance

  • Enable async I/O and tune batch_size, cache_size, search_timeout.
  • Use a single long-lived process per storage directory.
  • Decide whether embeddings are necessary and configure accordingly.
  • Run heavy operations via background tasks during off-peak hours.

Security

  • Run under a dedicated OS user with minimal privileges.
  • Enforce 0600 on .env and 0700 on storage directories.
  • Keep all data local and avoid adding implicit network calls.
  • Review integration/tool exposure for potential data leaks.

Operations

  • Capture logs and monitor performance metrics.
  • Test configuration changes under synthetic load.
  • Back up JSONL and vault content before upgrades.
  • Roll out upgrades gradually, especially in multi-instance deployments.

This checklist, combined with the detailed sections above, provides a concrete path to scaling CortexGraph responsibly while maintaining its privacy-preserving, local-first design and strong security posture.