Migration Plan: Path-Based to ID-Based API (v1 → v2)

[jope-bm]

Migration Plan: Path-Based to ID-Based API (v1 → v2)

Overview

Migrate the Basic Memory API from path-based entity identifiers to integer ID-based identifiers for improved performance, stability, and developer experience. This will be implemented as a new /v2 API namespace with immediate v1 deprecation.

Problem Statement

Current v1 API uses path-based identifiers (/{project}/knowledge/entities/{identifier:path}) which:

• Require cascading database lookups (permalink → title → file_path → search)
• Change when files are moved or renamed
• Are difficult to cache effectively
• Have performance overhead from string comparisons

Solution

Create a new /v2 API namespace that uses integer entity IDs as primary identifiers:

• Direct database lookups via indexed integer primary keys
• Stable references that don't change with file moves
• Better caching support
• Improved performance

Migration Strategy

Phase 1: Create /api/v2 + Deprecate V1 (3-4 weeks)

Build v2 API with ID-based endpoints:

• Create v2 router structure: src/basic_memory/api/v2/
• Implement ID-based knowledge router
• Implement ID-based memory router
• Implement ID-based search router
• Implement ID-based resource router
• Create v2 schemas emphasizing entity IDs
• Add identifier resolution endpoint: POST /v2/{project}/knowledge/resolve
• Register v2 routers in app.py with /v2 prefix

Add v1 deprecation markers immediately:

• Create deprecation middleware with HTTP headers
  • Deprecation: true
  • Sunset: <6 months from release>
  • Link: </v2>; rel="successor-version"
• Add deprecation notices to v1 router docstrings
• Mark v1 routers as deprecated in OpenAPI
• Create deprecation info endpoint (/management/deprecation-info)
• Add deprecation metrics tracking
• Create migration guide documentation (docs/migration/v1-to-v2.md)
• Update CLAUDE.md with deprecation notice
• Add deprecation warning logging

Testing:

• Full test coverage for v2 endpoints
• Verify v1 deprecation headers present
• Validate deprecation metrics collection
• Ensure v1 tests still pass

Deliverables:

• v2 API fully functional with ID-based operations
• v1 API marked deprecated with 6-month sunset date
• Migration documentation ready
• Metrics tracking v1/v2 usage

Phase 2: Update MCP Tools to Use V2 (3-4 weeks)

Migrate MCP tools to v2:

• Create async_client_v2.py with /v2 prefix support
• Update read_note() to use v2 API
• Update edit_note() to use v2 API
• Update delete_note() to use v2 API
• Update move_note() to use v2 API
• Update view_note() to use v2 API
• Update build_context() to use v2 API
• Add numeric ID detection in tools
• Implement identifier-to-ID resolution in tools
• Update tool responses to prominently show entity IDs

Update memory:// URL handling:

• Support memory://id/123 format
• Support memory://123 shorthand format
• Update normalize_memory_url() for ID patterns
• Keep backward compatibility with path-based URLs
• Update context building to return IDs

Documentation updates:

• Update MCP tool descriptions to recommend IDs
• Update ai_assistant_guide with v2 examples
• Add ID-based examples to tool documentation
• Document performance benefits

Testing:

• MCP tools work with numeric IDs
• MCP tools work with legacy identifiers (via resolution)
• Performance benchmarks show improvement
• Integration tests updated

Deliverables:

• All MCP tools using v2 API
• Tools accept both IDs and legacy identifiers
• Updated documentation and examples

Phase 3: Monitor Usage & Support Migration (6 months)

Monitoring:

• Set up weekly metrics review process
• Track v2 adoption rate over time
• Identify and reach out to heavy v1 users
• Monitor error rates for both versions

User support:

• Enhance basic-memory migrate-to-v2 CLI command
• Add validation and dry-run modes to migration tool
• Create migration report generator
• Monthly reminder communications about deprecation
• Create migration tutorials and examples
• Provide migration support channel

Client library updates:

• Update Python client to default to v2
• Add use_v1=True flag for backward compatibility
• Update all client examples to v2

Success criteria:

• v2 adoption > 80% by month 3
• v2 adoption > 95% by month 5
• No critical v2 bugs
• Ready to remove v1 at 6 months

Deliverables:

• 95%+ v2 adoption rate
• Migration tooling and documentation proven effective
• User feedback incorporated

Phase 4: Remove V1 API (2-3 weeks)

Final preparations:

• Send final 30-day warning to users
• Return 410 Gone for endpoints past sunset date
• Provide emergency support for late migrators

Code cleanup:

• Remove v1 router files
• Remove deprecation middleware
• Simplify LinkResolver (keep for convenience)
• Update services to prioritize ID operations
• Clean up v1-specific code paths
• Move v2 schemas to main schema directory
• Update all imports

Version and release:

• Update package version to 2.0.0
• Create v2.0.0 release tag
• Update API documentation
• Archive v1 migration guide

Testing:

• All v1 endpoints return 404/410
• V2 endpoints fully functional
• Performance benchmarks meet targets
• 100% test coverage maintained

Deliverables:

• Clean v2-only codebase
• Basic Memory 2.0.0 release
• Performance improvements realized

Key Benefits

1. Performance: Direct integer lookups vs cascading string comparisons
2. Stability: IDs don't change when files move
3. Caching: Stable IDs enable effective caching strategies
4. Developer Experience: Clear, predictable API patterns
5. Database Optimization: Integer indexes more efficient than string indexes

API Comparison

Feature	V1 (Deprecated)	V2 (New)
URL Pattern	/{project}/knowledge/entities/{path:path}	/v2/{project}/knowledge/entities/{id}
Identifier	Path/Permalink/Title	Integer ID
Lookup Speed	Multiple queries (cascade)	Single indexed query
Stability	Changes with file moves	Immutable
Memory URLs	memory://path/to/note	memory://123
Caching	Difficult	Easy

Timeline

• Phase 1: 3-4 weeks (v2 build + v1 deprecation)
• Phase 2: 3-4 weeks (MCP tool migration)
• Phase 3: 6 months (monitoring & support)
• Phase 4: 2-3 weeks (v1 removal)

Total: ~2-3 months development + 6 months deprecation period

Sunset Date

V1 API End of Life: 6 months after v2 release (tentatively June 30, 2025)

Migration Support

• Migration guide: docs/migration/v1-to-v2.md
• CLI tool: basic-memory migrate-to-v2
• Deprecation endpoint: /management/deprecation-info
• Metrics endpoint: /management/metrics/deprecation

References

• Database models: src/basic_memory/models/knowledge.py
• Current v1 routers: src/basic_memory/api/routers/
• LinkResolver: src/basic_memory/services/link_resolver.py

---

This is a comprehensive migration that will significantly improve API performance and developer experience while maintaining backward compatibility during the transition period.

[claude]

Claude finished @jope-bm's task —— View job

---

Issue Triage Analysis

Classification:

• Type: Enhancement (comprehensive architectural improvement)
• Priority: High
• Components: API, MCP tools, Database, Documentation
• Complexity: Complex (multi-phase migration over 6-8 months)

Analysis:

This is an excellent, well-structured migration plan that addresses significant architectural and performance concerns identified in SPEC-11. The proposal to migrate from path-based to ID-based entity identifiers is well-justified and aligns with database best practices.

Key Strengths:

1. Performance Benefits: Direct integer lookups vs cascading string comparisons (addresses SPEC-11 findings)
2. Stability: Immutable IDs prevent broken references when files move
3. Phased Approach: 4-phase plan with clear deliverables and success criteria
4. Migration Support: Includes deprecation period, migration tooling, and backward compatibility
5. Comprehensive Scope: Covers API, MCP tools, docs, and testing

Alignment with Existing Work:

• Builds on SPEC-11's performance optimization work (database connection caching already implemented)
• Complements the repository pattern and dependency injection architecture
• Follows the specification-driven development process (SPEC-1)

Observations:

1. No Duplicates Found: This is a unique proposal with no conflicting issues

2. Timeline Realistic: 2-3 months dev + 6 months deprecation is appropriate for this scope

3. Related Performance Work: SPEC-11 has already optimized connection pooling and initialization, so v2 will benefit from those improvements

4. Migration Complexity Considered: The plan accounts for:

   • Backward compatibility during transition
   • Migration tooling and documentation
   • Monitoring and metrics
   • User support and communication

Recommended Next Steps:

1. Create Specification Document: Following SPEC-1 process, create SPEC-21: Path-Based to ID-Based API Migration with technical implementation details

2. Break Down into Issues: Create individual issues for each phase:

   • Phase 1: Create v2 API + Deprecate v1 (feat: Implement API v2 with ID-based endpoints (Phase 1) #441)
   • Phase 2: Update MCP Tools to v2 (draft: Add batch processing for Postgres sync optimization #442)
   • Phase 3: Monitor & Support Migration ([BUG] Sync stops working with "cannot schedule new futures after shutdown" errors #443)
   • Phase 4: Remove v1 API (Option to inline JSON Schema $defs for broader MCP client compatibility #444)

3. Proof of Concept: Consider a small PoC for Phase 1 to validate:

   • v2 API structure
   • ID resolution endpoint design
   • Performance improvements vs v1
   • Backward compatibility approach

4. Stakeholder Review: Since this affects all API consumers, review with:

   • MCP tool users
   • Cloud deployment team
   • Documentation team

Label Applied:

• ✅ enhancement (architectural improvement)

Note: Additional labels like api, mcp, or performance are not available in the repository. Consider creating these labels for better issue organization.

---

Summary: This is a well-thought-out, high-priority enhancement that will significantly improve API performance and stability. The phased approach with deprecation period demonstrates strong software engineering practices. Recommend proceeding with SPEC creation and breaking down into phase-specific implementation issues.