Merge pull request #485 from tidesdb/latest-minors-5

guycipher · web-flow · commit cd4fc90c2186 · 2026-04-27T06:58:49.000-04:00
align kafka and admintool references with latest versions
diff --git a/src/content/docs/reference/admintool.md b/src/content/docs/reference/admintool.md
@@ -65,13 +65,25 @@ open <path> [options]
 |--------|-------------|
 | `--unified` | Enable unified memtable mode |
 | `--unified-buffer-size <n>` | Unified memtable write buffer size (bytes) |
+| `--unified-skip-list-max-level <n>` | Skip list max level for unified memtable (0 = default 12) |
+| `--unified-skip-list-probability <f>` | Skip list probability for unified memtable (0 = default 0.25) |
 | `--unified-sync <mode>` | Unified WAL sync mode (`none`\|`interval`\|`full`) |
 | `--unified-sync-interval <us>` | Unified WAL sync interval (microseconds) |
 | `--cache-size <n>` | Block cache size (bytes, 0 to disable) |
 | `--max-open-sstables <n>` | Max cached SSTable structures |
 | `--flush-threads <n>` | Flush thread pool size |
 | `--compaction-threads <n>` | Compaction thread pool size |
+| `--max-concurrent-flushes <n>` | Global cap on in-flight memtable flushes across all CFs (0 = default) |
 | `--max-memory <n>` | Global memory limit (bytes, 0 = auto) |
+| `--log-level <level>` | Log level: `debug`\|`info`\|`warn`\|`error`\|`fatal`\|`none` (default: `none`) |
+| `--log-to-file` | Write logs to `<db_path>/LOG` instead of stderr |
+| `--log-truncation-at <n>` | Log truncation size in bytes (0 = no truncation) |
+| `--object-store-fs <root_dir>` | Enable filesystem object-store connector. Auto-enables unified memtable mode |
+| `--obj-cache-path <dir>` | Local cache directory for object store (default: `<db_path>`) |
+| `--obj-cache-max-bytes <n>` | Maximum local cache size in bytes (0 = unlimited) |
+| `--obj-replica-mode` | Open the database in read-only replica mode |
+| `--obj-replica-sync-interval <us>` | Replica MANIFEST poll interval (default: 5000000) |
+| `--obj-wal-sync-on-commit` | Upload the WAL to the object store after every commit (RPO=0) |
 
 **Examples**
 ```
@@ -83,12 +95,22 @@ Opened database at './mydb' (unified memtable)
 
 admintool> open ./mydb --unified --cache-size 134217728 --flush-threads 4
 Opened database at './mydb' (unified memtable)
+
+admintool> open ./mydb --log-level info --log-to-file
+Opened database at './mydb'
+
+admintool> open ./mydb --object-store-fs /mnt/objects
+Opened database at './mydb' (object-store fs:/mnt/objects)
 ```
 
 :::note[Unified Memtable Mode]
 When `--unified` is set, all column families share a single skip list and WAL instead of each column family maintaining its own. A transaction touching N column families results in 1 WAL write instead of N. On-disk SSTables remain per-column-family. This mode is beneficial for write-heavy multi-CF workloads.
 :::
 
+:::note[Filesystem Object Store]
+`--object-store-fs <root_dir>` attaches a filesystem-backed object-store connector that mirrors objects under the given directory. This is intended for testing and local replication scenarios. Object-store mode automatically enables unified memtable mode. The connector is destroyed when the database is closed. Combine with `--obj-cache-max-bytes`, `--obj-replica-mode`, `--obj-replica-sync-interval`, and `--obj-wal-sync-on-commit` to tune behavior.
+:::
+
 ### close
 Close the current database.
 ```
@@ -158,6 +180,11 @@ cf-create <name> [options]
 | `--l0-stall <n>` | L0 queue stall threshold |
 | `--level-size-ratio <n>` | Level size multiplier |
 | `--min-levels <n>` | Minimum LSM levels |
+| `--dividing-level-offset <n>` | Compaction dividing level offset |
+| `--tombstone-density-trigger <ratio>` | Tombstone density above which compaction priority escalates (`0.0` to `1.0`, `0` disables) |
+| `--tombstone-density-min-entries <n>` | Minimum entry count for an SSTable to be considered by the density trigger |
+| `--object-lazy-compaction` / `--no-object-lazy-compaction` | Lazy compaction in object-store mode (doubles the L1 trigger, reducing remote I/O) |
+| `--object-prefetch-compaction` / `--no-object-prefetch-compaction` | Prefetch all input SSTables in parallel before a compaction merge (default: enabled) |
 
 **Examples**
 ```
@@ -242,8 +269,9 @@ cf-stats <name>
 - Memtable size, levels, total keys
 - Data size, avg key/value sizes
 - Read amplification, cache hit rate
-- Full configuration (compression, bloom filter, sync mode, etc.)
-- Per-level SSTable counts and sizes
+- Full configuration (compression, bloom filter, sync mode, dividing level offset, tombstone density trigger and minimum, object-store lazy/prefetch compaction, etc.)
+- Per-level SSTable counts, sizes, key counts, and tombstone counts
+- Tombstone observability: total tombstones, database-wide ratio, and the worst single-SSTable tombstone density (with the level it lives on)
 - B+tree statistics (if enabled)
 
 ### cf-status
@@ -276,8 +304,10 @@ All options from `cf-create` are supported except `--btree` and `--comparator` (
 - `--block-index-prefix-len`, `--index-sample-ratio`, `--no-block-indexes` · Block index settings for new SSTables
 - `--skip-list-max-level`, `--skip-list-probability` · Skip list settings for new memtables
 - `--isolation` · Default transaction isolation level
-- `--level-size-ratio`, `--min-levels` · LSM level sizing
+- `--level-size-ratio`, `--min-levels`, `--dividing-level-offset` · LSM level sizing
 - `--l1-trigger`, `--l0-stall` · Compaction and backpressure thresholds
+- `--tombstone-density-trigger`, `--tombstone-density-min-entries` · Tombstone-density compaction escalation
+- `--object-lazy-compaction` / `--no-object-lazy-compaction`, `--object-prefetch-compaction` / `--no-object-prefetch-compaction` · Object-store mode tuning
 - `--min-disk-space` · Minimum disk space required
 
 **Non-updatable settings**
@@ -293,6 +323,30 @@ admintool(./mydb)> cf-update cache --write-buffer-size 268435456 --no-persist
 Configuration updated for 'cache' (in-memory only)
 ```
 
+### cf-config-save
+Save the current column family configuration to an INI file. The optional `section_name` defaults to the column family name.
+```
+cf-config-save <cf> <ini_file> [section_name]
+```
+
+**Example**
+```
+admintool(./mydb)> cf-config-save users ./users.ini
+Saved configuration of 'users' to './users.ini' (section [users])
+```
+
+### cf-config-load
+Create a column family by loading its configuration from an INI file section. Useful for replicating a tuned configuration across databases or for templating column families.
+```
+cf-config-load <ini_file> <section_name> <cf_name>
+```
+
+**Example**
+```
+admintool(./mydb)> cf-config-load ./users.ini users users_replica
+Created column family 'users_replica' from './users.ini' [users]
+```
+
 ## Key-Value Operations
 
 ### put
@@ -323,6 +377,22 @@ Delete a key.
 delete <cf> <key>
 ```
 
+### single-delete
+Tombstone a key with single-delete semantics. Compaction can drop the put and the tombstone together as soon as both appear in the same merge input, rather than carrying the tombstone forward to the largest level.
+```
+single-delete <cf> <key>
+```
+
+:::caution[When to use]
+`single-delete` is only safe when each key is put at most once between any two single-deletes (and at most once before the first single-delete on that key). Violating the contract can leave older puts visible after the tombstone. Prefer `delete` for any workload that issues repeated updates to the same key.
+:::
+
+**Example**
+```
+admintool(./mydb)> single-delete users user:1
+OK
+```
+
 ### scan
 Scan all keys in a column family.
 ```
@@ -483,6 +553,31 @@ Trigger compaction on a column family.
 compact <cf>
 ```
 
+### compact-range
+Run a synchronous compaction over a specific key range. Only SSTables whose minimum and maximum keys overlap the requested range participate in the merge, so the work and I/O are bounded to the affected portion of the LSM tree rather than the whole column family.
+```
+compact-range <cf> <start_key> <end_key>
+```
+
+**Behavior**
+- Synchronous, blocks the caller until the merge commits or fails
+- Selects only SSTables whose key range overlaps `[start_key, end_key)`
+- Applies the same emit-loop logic as background compactions (tombstone reclamation, single-delete pair cancellation, sequence-based deduplication, value recompression)
+- Output SSTables are committed to the manifest atomically and old inputs are marked for deletion
+
+**Use cases**
+- Bulk reclaim after a large range delete, where waiting for natural compaction would leave tombstones on disk
+- Tenant eviction or sliding-window expiration that does not fit TTL semantics
+- Post-import cleanup of a known key range
+- Operational counterpart to the automatic tombstone density trigger when an operator wants reclaim now rather than at the next threshold crossing
+
+**Example**
+```
+admintool(./mydb)> compact-range users tenant_42: tenant_42;
+Compacting range 'users' [tenant_42: .. tenant_42;)...
+Range compaction completed for 'users'
+```
+
 ### flush
 Flush memtable to disk.
 ```
diff --git a/src/content/docs/reference/kafka.md b/src/content/docs/reference/kafka.md
@@ -34,7 +34,7 @@ Switching from RocksDB to TidesDB requires no changes to your stream topology. Y
 <dependency>
     <groupId>com.tidesdb</groupId>
     <artifactId>tidesdb-kafka</artifactId>
-    <version>0.3.1</version>
+    <version>0.4.0</version>
 </dependency>
 ```
 
@@ -134,6 +134,7 @@ These settings control the TidesDB database instance that backs the state store.
 | `maxMemoryUsage` | long | 0 (auto) | Global memory limit in bytes; 0 lets TidesDB auto-detect (50% of system RAM) |
 | `logToFile` | boolean | false | Write TidesDB logs to a file instead of stderr |
 | `logTruncationAt` | long | 24 MB | Log file truncation size; 0 disables truncation |
+| `maxConcurrentFlushes` | int | 0 (library default) | Global semaphore on in-flight memtable flushes across all column families; 0 inherits the engine default |
 | `objectStoreFsPath` | String | null | Filesystem path for object store connector; null disables object store mode |
 | `objectStoreConfig` | ObjectStoreConfig | null | Object store behavior configuration; null uses defaults |
 | `unifiedMemtable` | boolean | false | Enable unified memtable mode (single memtable + WAL for all CFs) |
@@ -171,6 +172,8 @@ Each state store maps to a single TidesDB column family. These settings control
 | `dividingLevelOffset` | int | 2 | Compaction dividing level offset |
 | `minDiskSpace` | long | 100 MB | Minimum free disk space required before writes are rejected |
 | `comparatorName` | String | "" (memcmp) | Custom comparator name; empty uses default binary comparison |
+| `tombstoneDensityTrigger` | double | 0.0 (disabled) | Per-SSTable tombstone density above which compaction priority escalates; range 0.0 to 1.0 |
+| `tombstoneDensityMinEntries` | long | 1024 | Minimum entry count for an SSTable to be considered by the tombstone density trigger |
 | `objectLazyCompaction` | boolean | false | Compact less aggressively in object store mode |
 | `objectPrefetchCompaction` | boolean | true | Download all inputs before merge in object store mode |
 
@@ -337,8 +340,17 @@ System.out.println("Data size: " + stats.getTotalDataSize());
 System.out.println("Memtable size: " + stats.getMemtableSize());
 System.out.println("Read amplification: " + stats.getReadAmp());
 System.out.println("Cache hit rate: " + stats.getHitRate());
+
+// Tombstone observability
+System.out.println("Total tombstones: " + stats.getTotalTombstones());
+System.out.printf("Tombstone ratio: %.2f%%%n", stats.getTombstoneRatio() * 100.0);
+System.out.printf("Worst SSTable density: %.2f%% at level %d%n",
+    stats.getMaxSstDensity() * 100.0, stats.getMaxSstDensityLevel());
+long[] levelTombstoneCounts = stats.getLevelTombstoneCounts();
 ```
 
+`getTotalKeys()` includes the active memtable plus every SSTable, so writes are visible to stats without a prior flush. The tombstone observability fields (`getTotalTombstones`, `getTombstoneRatio`, `getMaxSstDensity`, `getMaxSstDensityLevel`, `getLevelTombstoneCounts`) pair with the column family `tombstoneDensityTrigger` and `tombstoneDensityMinEntries` settings to monitor and drive density-based compaction escalation.
+
 **Database-level statistics**
 ```java
 DbStats dbStats = store.getDbStats();
@@ -381,6 +393,19 @@ boolean compacting = store.isCompacting();
 
 Use `purge()` before backup, after bulk deletes, or during maintenance windows. Use `compact()` and `flush()` for non-blocking background work.
 
+### Targeted Range Compaction
+
+`compactRange` runs a synchronous compaction over a specific key range. Only SSTables whose key range overlaps the bounds participate in the merge, so the work and I/O are bounded to the affected portion of the LSM tree.
+
+```java
+byte[] start = "tenant_42:".getBytes(StandardCharsets.UTF_8);
+byte[] end   = "tenant_42;".getBytes(StandardCharsets.UTF_8);
+
+store.compactRange(start, end);
+```
+
+A null or empty endpoint means unbounded on that side; both null/empty is rejected (use `compact()` for full-CF compaction). Use this after a large range delete, tenant eviction, or when you want to reclaim space immediately rather than waiting for the natural tombstone density trigger to fire.
+
 ### WAL Sync
 
 Force an immediate fsync of the write-ahead log:
@@ -427,6 +452,17 @@ store.updateRuntimeConfig(newConfig, true);
 
 Updatable settings include `writeBufferSize`, `skipListMaxLevel`, `skipListProbability`, `bloomFPR`, `indexSampleRatio`, `syncMode`, and `syncIntervalUs`.
 
+### Single-Delete
+
+`singleDelete` writes a tombstone with the same read semantics as `delete`, but lets compaction drop the put and tombstone together as soon as both appear in the same merge input - rather than carrying the tombstone forward until it reaches the largest active level.
+
+```java
+TidesDBStore store = (TidesDBStore) context.getStateStore("my-store");
+store.singleDelete(Bytes.wrap("session:42".getBytes()));
+```
+
+**Caller contract**: between any two single-deletes on the same key, the key has been put **at most once**. The engine cannot verify this; violating the contract can leave older puts visible after the single-delete. Use only for insert-once / delete-once workloads (session caches, secondary-index entries that are never updated, log-style state with scheduled purges). When in doubt, prefer `delete`.
+
 ### Range Cost Estimation
 
 Estimate the cost of iterating between two keys without performing any disk I/O:
@@ -559,6 +595,13 @@ store.renameColumnFamily("old_name", "new_name");
 store.dropColumnFamily("old_cf");
 ```
 
+A handle-based variant accepts a `ColumnFamily` object directly:
+
+```java
+ColumnFamily cf = store.getDb().getColumnFamily("old_cf");
+store.deleteColumnFamily(cf);
+```
+
 ### Custom Comparators
 
 Register a custom comparator for use with column families. Built-in comparators include `"memcmp"` (default), `"lexicographic"`, `"uint64"`, `"int64"`, `"reverse"`, and `"case_insensitive"`.
