v2.5.13: bump CosmoKv 2.1.9 → 3.2.0, add encryption + admin surface

Brings the driver onto CosmoKv's audited 3.2 release line and exposes
the v2.2 / v3.0 / v3.1 / v3.2 features the previous pin couldn't reach.

CosmoSQLClient.CosmoKv
  - csproj: CosmoKv 2.1.9 → 3.2.0.
  - CosmoKvConfiguration: new EncryptionKey (byte[]?) opens or creates
    the store as an AES-256-GCM encrypted database; length validated by
    CosmoKv's own DbOptions check (32 bytes). New AllowPlaintextBackup
    (bool, default false) is the explicit opt-in before a plaintext
    BackupAsync is allowed against an encrypted DB. Connection-string
    parser accepts EncryptionKey=<base64> and AllowPlaintextBackup=true;
    programmatic construction is preferred for key material — connection
    strings are commonly logged.
  - CosmoKvConnection.OpenAsync now builds the full DbOptions (was
    DbOptions.Default(path)), threading EncryptionKey +
    AllowPlaintextBackup through.
  - New BackupAsync(Stream, byte[]? backupKey, CT) overload — non-null
    key produces an AEAD-encrypted COSMOBAK stream; null requires
    AllowPlaintextBackup on encrypted sources.
  - New RunValueLogGcAsync(CT) wrapper exposing vlog space reclaim
    without reaching for RawDb.
  - Storage/Catalog.cs: documented Item.ReadValueAsync lifetime at the
    two HydrateAsync scan loops — CosmoKv v3.1 made the iterator-only
    validity an explicit invariant; the call sites are already safe,
    but a future refactor that caches Item past the loop body would
    break.

CosmoSQLClient.CosmoKv.Cli
  - .backup FILENAME writes a COSMOBAK snapshot (refuses to overwrite).
  - .vacuum runs RunValueLogGcAsync and reports files reclaimed.
  - .help + README updated. Opening an encrypted store from the shell
    works today via the connection-string EncryptionKey=<base64> form.

Tests
  - +3 connection tests: COSMOBAK magic on BackupAsync, no-throw zero
    return on RunValueLogGcAsync against an empty store, base64-decoded
    EncryptionKey + AllowPlaintextBackup round-trip through Parse.
  - 276/276 driver pass; 6/6 Http and 10/10 Pipes pass against the
    transitive bump.

Migration note: encrypted databases written by the previous driver
(CosmoKv 2.x) are NOT readable here. CosmoKv v3.0 changed the WAL/vlog
AEAD AAD scheme from fileId to (fileId, frameOffset) to defeat
intra-file frame-relocation attacks. To migrate, take a backup with
the previous driver, then restore into a fresh v3.x store. Plaintext
databases upgrade transparently.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: vkuttyp

Directory.Build.props

@@ -7,6 +7,6 @@
     <Authors>vkuttyp</Authors>
     <PackageLicenseExpression>MIT</PackageLicenseExpression>
     <RepositoryUrl>https://github.com/vkuttyp/CosmoSQLClient-Dotnet</RepositoryUrl>
-    <Version>2.5.12</Version>
+    <Version>2.5.13</Version>
   </PropertyGroup>
 </Project>

src/CosmoSQLClient.CosmoKv.Cli/Cli.cs

@@ -286,6 +286,8 @@ .tables                List user tables
                     .schema [TABLE]        CREATE TABLE script (all if omitted)
                     .indexes [TABLE]       List indexes (all if omitted)
                     .dump                  Schema + INSERTs for the whole database
+                    .backup FILENAME       Write a COSMOBAK snapshot to FILENAME
+                    .vacuum                Run value-log GC to reclaim space
                     .format FMT            Switch output (table|csv|tsv|json)
                     .quit / .exit          Leave the shell
                     """);
@@ -332,6 +334,18 @@ .format FMT            Switch output (table|csv|tsv|json)
                 await DumpAsync();
                 return true;
 
+            case ".backup":
+                if (arg is null) { await Console.Error.WriteLineAsync("usage: .backup FILENAME"); return true; }
+                await BackupAsync(arg);
+                return true;
+
+            case ".vacuum":
+                {
+                    int dropped = await _conn.RunValueLogGcAsync();
+                    Console.WriteLine($"vacuum: {dropped} vlog file(s) reclaimed");
+                }
+                return true;
+
             case ".format":
                 if (arg is null) Console.WriteLine($"format: {Format.ToString().ToLowerInvariant()}");
                 else
@@ -347,6 +361,15 @@ .format FMT            Switch output (table|csv|tsv|json)
         }
     }
 
+    private async Task BackupAsync(string path)
+    {
+        // FileMode.CreateNew so a typo on the destination doesn't silently
+        // overwrite a prior backup. Users can rm + retry if they meant it.
+        await using var fs = new FileStream(path, FileMode.CreateNew, FileAccess.Write, FileShare.None);
+        long bytes = await _conn.BackupAsync(fs);
+        Console.WriteLine($"backup: {bytes:n0} bytes written to {path}");
+    }
+
     private async Task DumpAsync()
     {
         Console.WriteLine("-- cosmokv .dump");

src/CosmoSQLClient.CosmoKv.Cli/README.md

@@ -92,6 +92,8 @@ Dot-commands work only at the start of a fresh line (not mid-statement) and cons
 | `.schema [TABLE]` | `CREATE TABLE` script for `TABLE` (or every table if omitted). Round-trips through the parser, so you can replay it. |
 | `.indexes [TABLE]` | `CREATE INDEX` scripts for `TABLE` (or every table). |
 | `.dump` | Schema + `INSERT` statements for the entire database — like `sqlite3`'s `.dump`. Safe to replay against an empty CosmoKv. |
+| `.backup FILENAME` | Write a COSMOBAK snapshot to `FILENAME`. MVCC — writes continue uninterrupted. Refuses to overwrite an existing file. |
+| `.vacuum` | Run value-log GC to reclaim space from tombstoned/overwritten large values. Prints the number of vlog files dropped. |
 | `.format FMT` | Switch the output format mid-session. |
 | `.quit` / `.exit` / `Ctrl-D` | Leave the shell. |
 
@@ -121,7 +123,9 @@ cosmokv ./mydb - < migrations/001_schema.sql
 
 - Statement splitting is lexical only — it understands single-quoted strings but not block comments or T-SQL `GO`. A trailing `;` ends the current statement.
 - No `readline`-style history or autocomplete; if you want those, wrap the CLI with `rlwrap`.
-- `.dump` rewrites every row as a fully-qualified `INSERT`. For tables with millions of rows, prefer `BackupAsync` (in code) to a COSMOBAK snapshot.
+- `.dump` rewrites every row as a fully-qualified `INSERT`. For tables with millions of rows, prefer `.backup FILENAME` — it writes a COSMOBAK snapshot directly without the parse/encode round-trip.
+- `.backup` only writes plaintext snapshots. To re-encrypt with a different key, use `BackupAsync(stream, backupKey)` in code.
+- Opening an encrypted store from the shell requires a connection string with `EncryptionKey=<base64>` (32-byte AES-256 key, base64-encoded). Bare-path `.open ./mydb` opens unencrypted.
 
 ## Related
 

src/CosmoSQLClient.CosmoKv/CosmoKvConfiguration.cs

@@ -21,6 +21,32 @@ public sealed record CosmoKvConfiguration
     /// </summary>
     public bool CreateIfMissing { get; init; } = true;
 
+    /// <summary>
+    /// 32-byte master key for AES-256-GCM encryption at rest. When set, the
+    /// CosmoKv directory is opened (or created) as an encrypted store: every
+    /// SST block, WAL frame, and vlog frame is AEAD-encrypted. The key never
+    /// touches disk — only the wrapped DEK in <c>KEYREGISTRY</c> does.
+    /// <para>
+    /// Prefer programmatic construction over <see cref="Parse"/> for this
+    /// field: connection strings are commonly logged, and the parser accepts
+    /// base64 only as a convenience.
+    /// </para>
+    /// <para>
+    /// CosmoKv v3.0 changed the AEAD AAD scheme; encrypted databases written
+    /// by CosmoKv v2.x are <b>not</b> readable here. Reseed from a backup
+    /// taken with the old driver before upgrading.
+    /// </para>
+    /// </summary>
+    public byte[]? EncryptionKey { get; init; }
+
+    /// <summary>
+    /// When <c>true</c>, <c>BackupAsync(stream, backupKey: null)</c> on an
+    /// encrypted database is allowed to emit plaintext bytes. Default
+    /// <c>false</c> — encrypted DBs must supply a backup key. Set to true
+    /// only for explicit export-to-plaintext workflows.
+    /// </summary>
+    public bool AllowPlaintextBackup { get; init; }
+
     public string ConnectionString => $"Data Source={DataSource};CreateIfMissing={CreateIfMissing}";
 
     /// <summary>
@@ -55,10 +81,28 @@ public static CosmoKvConfiguration Parse(string connectionString)
         if (Get("CreateIfMissing", "Create") is string s && bool.TryParse(s, out var b))
             create = b;
 
+        byte[]? key = null;
+        if (Get("EncryptionKey", "Key") is string keyStr && keyStr.Length > 0)
+        {
+            try { key = Convert.FromBase64String(keyStr); }
+            catch (FormatException fe)
+            {
+                throw new ArgumentException(
+                    "EncryptionKey must be base64-encoded 32-byte AES-256 key material.",
+                    nameof(connectionString), fe);
+            }
+        }
+
+        bool allowPlaintextBackup = false;
+        if (Get("AllowPlaintextBackup") is string apb && bool.TryParse(apb, out var apbB))
+            allowPlaintextBackup = apbB;
+
         return new CosmoKvConfiguration
         {
-            DataSource      = dataSource,
-            CreateIfMissing = create,
+            DataSource           = dataSource,
+            CreateIfMissing      = create,
+            EncryptionKey        = key,
+            AllowPlaintextBackup = allowPlaintextBackup,
         };
     }
 }

src/CosmoSQLClient.CosmoKv/CosmoKvConnection.cs

@@ -85,13 +85,46 @@ public bool IdentityInsertEnabled
     /// Online backup of the underlying CosmoKv store. Writes a COSMOBAK
     /// snapshot to <paramref name="output"/> using MVCC, so writes
     /// continue uninterrupted. Returns the number of bytes written.
+    /// <para>
+    /// On encrypted databases this overload emits plaintext bytes and
+    /// therefore requires <see cref="CosmoKvConfiguration.AllowPlaintextBackup"/>
+    /// to be set; otherwise CosmoKv throws. Use
+    /// <see cref="BackupAsync(Stream, byte[], CancellationToken)"/> to
+    /// re-encrypt the backup with a different key.
+    /// </para>
     /// </summary>
     public Task<long> BackupAsync(Stream output, CancellationToken ct = default)
     {
         EnsureOpen();
         return _db!.BackupAsync(output, ct);
     }
 
+    /// <summary>
+    /// Online backup with an explicit backup key. When non-null, the COSMOBAK
+    /// stream is AEAD-encrypted with <paramref name="backupKey"/> (32 bytes,
+    /// AES-256-GCM) — independent of the source database's master key, so
+    /// restore can rewrap with a fresh key. Pass <c>null</c> for plaintext
+    /// (requires <see cref="CosmoKvConfiguration.AllowPlaintextBackup"/> on
+    /// encrypted sources).
+    /// </summary>
+    public Task<long> BackupAsync(Stream output, byte[]? backupKey, CancellationToken ct = default)
+    {
+        EnsureOpen();
+        return _db!.BackupAsync(output, backupKey, ct);
+    }
+
+    /// <summary>
+    /// Reclaim space in the value log by rewriting live entries out of
+    /// vlog files whose tombstone density crosses CosmoKv's GC threshold.
+    /// Safe to call concurrently with reads and writes; runs at most one
+    /// GC pass at a time. Returns the number of vlog files dropped.
+    /// </summary>
+    public Task<int> RunValueLogGcAsync(CancellationToken ct = default)
+    {
+        EnsureOpen();
+        return _db!.RunValueLogGcAsync(ct);
+    }
+
     /// <summary>
     /// Pass-through to <see cref="CosmoKv.Db.GetStats"/>. Useful for
     /// admin/health surfaces (LSM level fan-out, table counts, optional
@@ -222,7 +255,14 @@ public static async Task<CosmoKvConnection> OpenAsync(
             throw new DirectoryNotFoundException(
                 $"Data directory '{config.DataSource}' does not exist and CreateIfMissing=false.");
 
-        var db = await CosmoKvDb.OpenAsync(CosmoKvDbOptions.Default(config.DataSource));
+        var dbOptions = CosmoKvDbOptions.Default(config.DataSource) with
+        {
+            // Length is validated by DbOptions (32 bytes for AES-256-GCM);
+            // surfacing the throw here would just duplicate that check.
+            EncryptionKey        = config.EncryptionKey,
+            AllowPlaintextBackup = config.AllowPlaintextBackup,
+        };
+        var db = await CosmoKvDb.OpenAsync(dbOptions);
         ct.ThrowIfCancellationRequested();
         var catalog = await Catalog.LoadAsync(db, ct);
         return new CosmoKvConnection(config, db, catalog);

src/CosmoSQLClient.CosmoKv/CosmoSQLClient.CosmoKv.csproj

@@ -5,7 +5,7 @@
   </ItemGroup>
 
   <ItemGroup>
-    <PackageReference Include="CosmoKv" Version="2.1.9" />
+    <PackageReference Include="CosmoKv" Version="3.2.0" />
     <!-- Microsoft.VisualStudio.Threading.AsyncReaderWriterLock — battle-
          tested writer-priority RW lock for the CosmoKvConnection
          hot path. The v2.5.7 attempt at rolling our own failed under

src/CosmoSQLClient.CosmoKv/Storage/Catalog.cs

@@ -45,6 +45,9 @@ private async Task HydrateAsync(CancellationToken ct)
         };
         await foreach (var item in _db.IterateAsync(iterOpts).WithCancellation(ct))
         {
+            // Resolve the value inside the iterator scope: CosmoKv's Item may
+            // hold a vlog pointer that is only valid for this iterator's
+            // lifetime. Never cache `item` past the loop body.
             var bytes = await item.ReadValueAsync();
             var schema = JsonSerializer.Deserialize<TableSchemaDto>(bytes, Json)!;
             var resolved = schema.ToSchema();
@@ -57,6 +60,7 @@ private async Task HydrateAsync(CancellationToken ct)
         };
         await foreach (var item in _db.IterateAsync(idxOpts).WithCancellation(ct))
         {
+            // Same lifetime contract as above — resolve before iteration advances.
             var bytes = await item.ReadValueAsync();
             var schema = JsonSerializer.Deserialize<IndexSchema>(bytes, Json)!;
             _indexes[schema.Name] = schema;

tests/CosmoSQLClient.CosmoKv.Tests/CosmoKvConnectionTests.cs

@@ -92,4 +92,48 @@ public void Configuration_Parse_MissingDataSource_Throws()
     {
         Assert.Throws<ArgumentException>(() => CosmoKvConfiguration.Parse("CreateIfMissing=true"));
     }
+
+    [Fact]
+    public async Task BackupAsync_WritesCosmobakHeader()
+    {
+        await using var conn = await CosmoKvConnection.OpenAsync(
+            new CosmoKvConfiguration { DataSource = _dir });
+        await conn.ExecuteAsync("CREATE TABLE t (id INT PRIMARY KEY, name NVARCHAR(50))");
+        await conn.ExecuteAsync("INSERT INTO t (id, name) VALUES (1, 'alice')");
+
+        using var ms = new MemoryStream();
+        long bytes = await conn.BackupAsync(ms);
+
+        Assert.True(bytes > 0);
+        Assert.Equal(bytes, ms.Length);
+        // Backup stream is CRC64-protected — first 8 bytes are the COSMOBAK magic.
+        Assert.True(ms.Length >= 8);
+        var magic = System.Text.Encoding.ASCII.GetString(ms.GetBuffer(), 0, 8);
+        Assert.Equal("COSMOBAK", magic);
+    }
+
+    [Fact]
+    public async Task RunValueLogGcAsync_OnEmptyStore_ReturnsZero()
+    {
+        await using var conn = await CosmoKvConnection.OpenAsync(
+            new CosmoKvConfiguration { DataSource = _dir });
+        int dropped = await conn.RunValueLogGcAsync();
+        // Empty store has no garbage to reclaim — must not throw, must report 0.
+        Assert.Equal(0, dropped);
+    }
+
+    [Fact]
+    public void Configuration_Parse_EncryptionKey_FromBase64()
+    {
+        var key = new byte[32];
+        for (int i = 0; i < 32; i++) key[i] = (byte)i;
+        var b64 = Convert.ToBase64String(key);
+
+        var c = CosmoKvConfiguration.Parse(
+            $"Data Source=/tmp/x;EncryptionKey={b64};AllowPlaintextBackup=true");
+
+        Assert.NotNull(c.EncryptionKey);
+        Assert.Equal(key, c.EncryptionKey);
+        Assert.True(c.AllowPlaintextBackup);
+    }
 }