VCL Backwards Compatibility Strategy

Strategy for maintaining backwards compatibility across VCL versions, schema evolution, and API changes.

Table of Contents

Overview
Compatibility Guarantees
- What We Guarantee
- What We Don’t Guarantee
Versioning Strategy
Breaking vs Non-Breaking Changes
- Non-Breaking Changes (Minor Version)
- Breaking Changes (Major Version)
Deprecation Process
- Timeline
- Deprecation Warnings
Migration Strategies
Compatibility Testing
- Version Compatibility Matrix
- Upgrade Testing
Data Format Versioning
- Octad Storage Format
- Migration on Read vs Write
Federation Compatibility
- Mixed Version Networks
- Feature Detection
Client Library Compatibility
- Language Bindings
- Client API Stability
Compatibility Checklist
- Before Releasing Minor Version (1.x.x → 1.y.x)
- Before Releasing Major Version (1.x.x → 2.x.x)
Summary

Overview

VeriSimDB follows semantic versioning (SemVer) for VCL and API compatibility:

Major version (1.x.x → 2.x.x): Breaking changes allowed
Minor version (1.1.x → 1.2.x): New features, backwards compatible
Patch version (1.1.1 → 1.1.2): Bug fixes only, backwards compatible

Current VCL version: 1.0.0

Compatibility Guarantees

What We Guarantee

Query Syntax (VCL):

Valid VCL 1.x queries will parse and execute correctly in all 1.x releases
New keywords/operators are additive (won’t conflict with existing identifiers)
Deprecations announced at least 2 minor versions before removal

API Endpoints:

/api/v1/* endpoints remain stable within major version
New endpoints added without removing old ones
Response format changes are backwards compatible (new fields added, never removed)

Data Formats:

Octad storage format versioned independently
Old format readers maintained for at least 1 year after new format introduction
Automatic migration on read for legacy formats

What We Don’t Guarantee

Performance:

Query performance may change between versions (optimizations, regressions)
Federation latency depends on network and remote store versions

Error Messages:

Error message text may change (error codes remain stable)
Additional validation may be added (making previously accepted queries invalid)

Internal APIs:

ReScript/Elixir module APIs may change without notice
Use public HTTP API for external integrations

Versioning Strategy

VCL Syntax Versions

Each query can optionally specify a version:

VERSION 1.0;

FROM verisim:semantic
WHERE octad.types INCLUDES "https://schema.org/Person"
LIMIT 10;

Default Behavior:

If VERSION is omitted, use latest stable version (currently 1.0)
Parser detects version and applies appropriate syntax rules
Mixing versions in a single query is NOT allowed

API Versioning

URL-based versioning:

/api/v1/octads          # Version 1 (current)
/api/v2/octads          # Version 2 (future)

Multiple versions supported concurrently:

Old API versions maintained for at least 12 months after new version release
Deprecation warnings in response headers: X-API-Deprecated: true; sunset=2026-12-31

Schema Evolution

Octad schema changes:

;; STATE.scm tracks schema version
(metadata
  (schema-version "1.2.0")  ; Current Octad schema
  (min-compatible-version "1.0.0"))  ; Oldest readable format

Format Evolution:

Adding fields (backwards compatible):
- Old readers ignore unknown fields
- New readers provide defaults for missing fields
Removing fields (breaking change):
- Deprecated for 2+ minor versions first
- Removal only in major version bump
Changing field types (breaking change):
- Add new field with new type
- Deprecate old field
- Remove old field in major version bump

Breaking vs Non-Breaking Changes

Non-Breaking Changes (Minor Version)

Safe to add in 1.x.x releases:

New VCL keywords (if not previously valid identifiers)
New modality types
New query operators (INTERSECT, UNION ALL)
New API endpoints
New response fields (optional)
New error codes
Performance optimizations

Example: Adding INTERSECT operator

-- Version 1.2.0 introduces INTERSECT
-- Old queries (1.0.0) continue to work

-- New query using INTERSECT (requires 1.2.0+)
VERSION 1.2;
(FROM verisim:graph WHERE octad.id IN @set1)
INTERSECT
(FROM verisim:graph WHERE octad.id IN @set2);

Breaking Changes (Major Version)

Requires 2.0.0:

Removing VCL keywords
Changing query semantics (e.g., LIMIT behavior)
Removing API endpoints
Removing response fields
Changing error codes

Example: Removing deprecated USING clause

-- Version 1.0.0 syntax (deprecated in 1.2.0)
FROM verisim:graph
USING CONTRACT CitationContract
WHERE octad.types INCLUDES "Paper";

-- Version 2.0.0 syntax (USING removed, VERIFY required)
FROM verisim:graph
VERIFY CitationContract
WHERE octad.types INCLUDES "Paper";

Deprecation Process

Timeline

Announce deprecation (version N.x.x):
- Add deprecation warning to docs
- Log warning when deprecated feature is used
- Add X-Deprecated: true header to API responses
Maintain support (versions N.x.x through N+1.x.x):
- Feature still works
- Warnings continue
Remove (version N+2.0.0):
- Feature removed
- Queries using deprecated feature return error

Minimum timeline: 6 months between deprecation and removal

Deprecation Warnings

VCL Parser:

// VQLParser.res
let parseUsingClause = (tokens: array<token>): result<ast, parseError> => {
  Logger.warn("USING clause is deprecated since 1.2.0. Use VERIFY instead. Will be removed in 2.0.0.")

  // Still parse correctly
  parseUsingClauseImpl(tokens)
}

API Response Headers:

HTTP/1.1 200 OK
X-Deprecated: true
X-Deprecated-Since: 1.2.0
X-Sunset: 2026-12-31
Sunset: Wed, 31 Dec 2026 23:59:59 GMT
Link: <https://verisimdb.org/docs/migration-guide>; rel="sunset"

Migration Strategies

Strategy 1: Version Pinning

Use case: Stability-critical systems

# config.exs
config :verisim,
  vcl_version: "1.0.0",  # Pin to specific version
  api_version: "v1"

Trade-offs:

✅ No surprises from version updates
❌ Miss performance improvements
❌ Miss security fixes

Strategy 2: Conservative Upgrades

Use case: Production systems with good test coverage

Upgrade to latest minor version automatically (1.1.x → 1.2.x)
Upgrade to latest major version manually (1.x.x → 2.x.x)

# Dockerfile
ENV VCL_VERSION="1.x"  # Auto-upgrade within major version

Trade-offs:

✅ Get bug fixes and features
✅ No breaking changes
⚠️ Require testing before major version upgrades

Strategy 3: Gradual Migration

Use case: Large codebases with many queries

Identify deprecated features: bash grep -r "USING" queries/ # Find all queries using deprecated syntax
Update queries gradually: ```vcl — Old query (1.0.0) FROM verisim:graph USING CONTRACT CitationContract WHERE …;
```
-- New query (1.2.0+)
VERSION 1.2;
FROM verisim:graph
VERIFY CitationContract
WHERE ...;
```
```
Test both versions side-by-side: ```elixir old_result = VCL.parse_and_execute(old_query, version: "1.0.0") new_result = VCL.parse_and_execute(new_query, version: "1.2.0")
```
assert old_result.data == new_result.data
```
```
Switch cutover: elixir # Deploy new queries # Monitor for 1 week # Remove old queries

Compatibility Testing

Version Compatibility Matrix

Test all VCL versions against current parser:

# test/vcl_compatibility_test.exs
defmodule VeriSim.VCLCompatibilityTest do
  use ExUnit.Case

  @vcl_versions ["1.0.0", "1.1.0", "1.2.0"]
  @test_queries [
    # Basic SELECT
    """
    VERSION 1.0;
    FROM verisim:graph
    WHERE octad.id = @id;
    """,

    # WITH clause (added 1.1.0)
    """
    VERSION 1.1;
    WITH VERIFICATION CitationContract
    FROM verisim:graph
    WHERE octad.types INCLUDES "Paper";
    """,

    # INTERSECT (added 1.2.0)
    """
    VERSION 1.2;
    (FROM verisim:graph WHERE ...) INTERSECT (FROM verisim:vector WHERE ...);
    """
  ]

  test "parse all versions" do
    for query <- @test_queries do
      assert {:ok, _ast} = VCL.parse(query)
    end
  end

  test "1.0.0 queries work in all versions" do
    query_1_0 = """
    VERSION 1.0;
    FROM verisim:graph LIMIT 10;
    """

    for version <- @vcl_versions do
      config = %{vcl_version: version}
      assert {:ok, _result} = VCL.execute(query_1_0, config)
    end
  end
end

Upgrade Testing

Test suite for major version upgrades:

# test/upgrade_test.exs
defmodule VeriSim.UpgradeTest do
  use ExUnit.Case

  @tag :upgrade
  test "1.x data readable in 2.x" do
    # Create octad with 1.x format
    octad_1_x = create_octad_v1()

    # Upgrade to 2.x
    Application.put_env(:verisim, :schema_version, "2.0.0")

    # Read octad (should auto-migrate)
    assert {:ok, octad_2_x} = VeriSim.Octad.read(octad_1_x.id)

    # Verify data integrity
    assert octad_1_x.title == octad_2_x.title
    assert octad_1_x.body == octad_2_x.body
  end

  @tag :upgrade
  test "1.x queries fail gracefully in 2.x if deprecated features used" do
    deprecated_query = """
    VERSION 1.0;
    FROM verisim:graph
    USING CONTRACT CitationContract
    WHERE octad.id = @id;
    """

    Application.put_env(:verisim, :vcl_version, "2.0.0")

    assert {:error, {:deprecated_syntax, "USING clause removed in 2.0.0"}} =
      VCL.parse(deprecated_query)
  end
end

Data Format Versioning

Octad Storage Format

Format version embedded in each octad:

{
  "_format_version": "1.2.0",
  "octad_id": "550e8400-e29b-41d4-a716-446655440000",
  "title": "Machine Learning Paper",
  "body": "...",
  "created_at": "2025-01-15T10:30:00Z"
}

Format evolution example:

// rust-core/verisim-octad/src/format.rs

#[derive(Serialize, Deserialize)]
#[serde(tag = "_format_version")]
pub enum OctadFormat {
    #[serde(rename = "1.0.0")]
    V1_0_0(OctadV1_0_0),

    #[serde(rename = "1.1.0")]
    V1_1_0(OctadV1_1_0),

    #[serde(rename = "1.2.0")]
    V1_2_0(OctadV1_2_0),
}

impl OctadFormat {
    /// Read octad in any format, auto-migrate to latest
    pub fn read_and_migrate(data: &[u8]) -> Result<OctadV1_2_0, Error> {
        let format: OctadFormat = serde_json::from_slice(data)?;

        match format {
            OctadFormat::V1_0_0(octad) => octad.migrate_to_1_1_0()?.migrate_to_1_2_0(),
            OctadFormat::V1_1_0(octad) => octad.migrate_to_1_2_0(),
            OctadFormat::V1_2_0(octad) => Ok(octad),
        }
    }
}

impl OctadV1_0_0 {
    fn migrate_to_1_1_0(self) -> Result<OctadV1_1_0, Error> {
        Ok(OctadV1_1_0 {
            octad_id: self.octad_id,
            title: self.title,
            body: self.body,
            created_at: self.created_at,
            // New field in 1.1.0 (default value)
            provenance: None,
        })
    }
}

Migration on Read vs Write

Read migration (preferred):

Old format stored on disk
Auto-migrate to new format when reading
No disk writes required
Lazy migration (only migrates what’s accessed)

Write migration (for breaking changes):

Background job migrates all octads
Required when old format becomes unsupported
Progress tracked: migration_status.json

# lib/verisim/migration_worker.ex
defmodule VeriSim.MigrationWorker do
  @doc """
  Migrate all octads from old format to new format.
  """
  def migrate_all_octads(from_version, to_version) do
    total = count_octads(from_version)

    Stream.iterate(0, &(&1 + 100))
    |> Stream.take_while(&(&1 < total))
    |> Enum.each(fn offset ->
      octads = fetch_octads(from_version, limit: 100, offset: offset)

      octads
      |> Enum.each(fn octad ->
        migrated = migrate_octad(octad, from_version, to_version)
        write_octad(migrated)
      end)

      update_progress(offset + length(octads), total)
    end)
  end
end

Federation Compatibility

Mixed Version Networks

Challenge: Federated stores may run different VeriSimDB versions.

Solution: Version negotiation

POST /api/v1/federation/query HTTP/1.1
Host: remote-store.edu
X-VeriSimDB-Version: 1.2.0
X-VQL-Version: 1.0.0

{
  "query": "FROM verisim:graph WHERE octad.id = @id"
}

Response:

HTTP/1.1 200 OK
X-VeriSimDB-Version: 1.1.0
X-VQL-Version-Supported: 1.0.0, 1.1.0

{
  "data": [...]
}

Version compatibility rules:

VCL version: Query must use syntax supported by remote store
- If remote supports [1.0.0, 1.1.0] and query uses 1.2.0 → error
- Client downgrades query syntax if possible
API version: Client uses highest common API version
- Local has v1, remote has v1, v2 → use v1
Data format: Responses auto-migrated by receiving store

Feature Detection

Query store capabilities before sending complex queries:

def query_federation(stores, query) do
  # Check if all stores support required features
  required_features = VCL.detect_features(query)
  # [:zkp_verification, :tensor_modality, :drift_detection]

  compatible_stores = Enum.filter(stores, fn store ->
    store_features = fetch_capabilities(store)
    MapSet.subset?(required_features, store_features)
  end)

  if Enum.empty?(compatible_stores) do
    {:error, {:insufficient_stores, "No stores support required features"}}
  else
    execute_federated_query(compatible_stores, query)
  end
end

Store capabilities endpoint:

GET /api/v1/capabilities HTTP/1.1

Response:
{
  "verisimdb_version": "1.2.0",
  "vcl_versions": ["1.0.0", "1.1.0", "1.2.0"],
  "modalities": ["graph", "vector", "semantic", "document", "temporal"],
  "features": [
    "zkp_verification",
    "drift_detection",
    "cache_sharing",
    "reversibility"
  ],
  "limits": {
    "max_query_size": 1048576,
    "max_results": 10000,
    "timeout_ms": 60000
  }
}

Client Library Compatibility

Language Bindings

Support multiple language clients:

Rust client: verisim-client-rs (official)
Python client: verisim-client-py (community)
JavaScript client: verisim-client-js (official, Deno)

Version compatibility:

# Cargo.toml
[dependencies]
verisim-client = "1.2.0"  # Client version matches server minor version

# Server compatibility:
# verisim-client 1.x.x works with verisim-server 1.y.z (any y, z)
# verisim-client 2.x.x works with verisim-server 2.y.z (any y, z)

Client API Stability

Stable client API (no breaking changes in minor versions):

// Stable API
pub trait VeriSimClient {
    fn query(&self, vcl: &str) -> Result<QueryResult>;
    fn create_octad(&self, octad: Octad) -> Result<OctadId>;
    fn get_octad(&self, id: OctadId) -> Result<Octad>;
}

// Adding new methods is OK (minor version bump)
pub trait VeriSimClient {
    fn query(&self, vcl: &str) -> Result<QueryResult>;
    fn create_octad(&self, octad: Octad) -> Result<OctadId>;
    fn get_octad(&self, id: OctadId) -> Result<Octad>;

    // New in 1.2.0
    fn batch_query(&self, queries: &[&str]) -> Result<Vec<QueryResult>>;
}

Compatibility Checklist

Before Releasing Minor Version (1.x.x → 1.y.x)

❏ All existing tests pass with new changes
❏ New features have tests
❏ Deprecation warnings added for features planned for removal
❏ Documentation updated with new features
❏ CHANGELOG.md updated with new features
❏ No breaking changes to API, VCL syntax, or data formats
❏ Federation compatibility tested with previous minor version

Before Releasing Major Version (1.x.x → 2.x.x)

❏ Migration guide written for all breaking changes
❏ Deprecated features removed (with at least 6 months notice)
❏ Data migration tools provided
❏ Previous major version supported for at least 12 months
❏ All examples and tutorials updated
❏ Client libraries updated
❏ Federation compatibility tested
❏ Rollback procedure documented

Summary

VeriSimDB maintains backwards compatibility through:

Semantic versioning - Clear breaking vs non-breaking change policy
Deprecation timeline - At least 6 months notice before removal
Format versioning - Auto-migration for data formats
API versioning - Multiple API versions supported concurrently
VCL versioning - Explicit version markers in queries
Feature detection - Capability negotiation for federation
Comprehensive testing - Compatibility test suite

For users:

Pin to major version for stability
Upgrade to minor versions for bug fixes and features
Plan major version upgrades carefully with migration guide

For developers:

Follow deprecation process for all breaking changes
Maintain compatibility test suite
Document version changes in CHANGELOG.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

VCL Backwards Compatibility Strategy

Overview

Compatibility Guarantees

What We Guarantee

What We Don’t Guarantee

Versioning Strategy

VCL Syntax Versions

API Versioning

Schema Evolution

Breaking vs Non-Breaking Changes

Non-Breaking Changes (Minor Version)

Breaking Changes (Major Version)

Deprecation Process

Timeline

Deprecation Warnings

Migration Strategies

Strategy 1: Version Pinning

Strategy 2: Conservative Upgrades

Strategy 3: Gradual Migration

Compatibility Testing

Version Compatibility Matrix

Upgrade Testing

Data Format Versioning

Octad Storage Format

Migration on Read vs Write

Federation Compatibility

Mixed Version Networks

Feature Detection

Client Library Compatibility

Language Bindings

Client API Stability

Compatibility Checklist

Before Releasing Minor Version (1.x.x → 1.y.x)

Before Releasing Major Version (1.x.x → 2.x.x)

Summary

Uh oh!

FilesExpand file tree

backwards-compatibility.adoc

Latest commit

History

backwards-compatibility.adoc

File metadata and controls

VCL Backwards Compatibility Strategy

Overview

Compatibility Guarantees

What We Guarantee

What We Don’t Guarantee

Versioning Strategy

VCL Syntax Versions

API Versioning

Schema Evolution

Breaking vs Non-Breaking Changes

Non-Breaking Changes (Minor Version)

Breaking Changes (Major Version)

Deprecation Process

Timeline

Deprecation Warnings

Migration Strategies

Strategy 1: Version Pinning

Strategy 2: Conservative Upgrades

Strategy 3: Gradual Migration

Compatibility Testing

Version Compatibility Matrix

Upgrade Testing

Data Format Versioning

Octad Storage Format

Migration on Read vs Write

Federation Compatibility

Mixed Version Networks

Feature Detection

Client Library Compatibility

Language Bindings

Client API Stability

Compatibility Checklist

Before Releasing Minor Version (1.x.x → 1.y.x)

Before Releasing Major Version (1.x.x → 2.x.x)

Summary