Skip to content

Commit 173ff4c

Browse files
committed
update building reference and tidesql reference
1 parent 50a3aca commit 173ff4c

2 files changed

Lines changed: 87 additions & 8 deletions

File tree

src/content/docs/reference/building.md

Lines changed: 50 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -79,15 +79,21 @@ TidesDB is tested and supported on the following platforms:
7979
- Clang 6.0+ (macOS/Linux)
8080
- MSVC 2019 16.8+ (Windows with `/experimental:c11atomics`)
8181

82-
### Required Dependencies
83-
TidesDB requires the following libraries for compression and threading:
82+
### Dependencies
83+
84+
The only hard requirements are a C11 toolchain, CMake, and threading support:
8485

85-
- **[Snappy](https://github.com/google/snappy)** · Fast compression/decompression (not available on SunOS/Illumos)
86-
- **[LZ4](https://github.com/lz4/lz4)** · Extremely fast compression
87-
- **[Zstandard](https://github.com/facebook/zstd)** · High compression ratio
8886
- **pthreads** · POSIX threads (Linux/macOS/BSD: built-in, Windows: PThreads4W via vcpkg)
8987
- **libatomic** · Atomic operations library (required for 32-bit architectures like PowerPC)
90-
- **C++ standard library** · Required by Snappy (automatically linked on Linux)
88+
89+
#### Compression backends (optional, enabled by default)
90+
91+
Each of the three compression libraries is optional. All three are built in by default; drop any with `-DTIDESDB_WITH_<NAME>=OFF`. A build with all three disabled has no compression dependencies at all and supports only `TDB_COMPRESS_NONE`, so TidesDB can be installed with nothing beyond libc and a thread library. See [Compression Backends](#compression-backends) under Build Options.
92+
93+
- **[Snappy](https://github.com/google/snappy)** · Fast compression/decompression (`TIDESDB_WITH_SNAPPY`; not packaged on SunOS/Illumos, so disabled there by default)
94+
- **[LZ4](https://github.com/lz4/lz4)** · Extremely fast compression (`TIDESDB_WITH_LZ4`)
95+
- **[Zstandard](https://github.com/facebook/zstd)** · High compression ratio (`TIDESDB_WITH_ZSTD`)
96+
- **C++ standard library** · Pulled in only when Snappy is enabled (Snappy's implementation is C++); a build without Snappy carries no libstdc++ dependency
9197

9298
## Installing Dependencies
9399

@@ -463,12 +469,50 @@ TidesDB provides several CMake options to customize the build:
463469
| `TIDESDB_BUILD_TESTS` | Build test suite | `ON` |
464470
| `BUILD_SHARED_LIBS` | Build shared libraries instead of static | `ON` (Unix), `OFF` (Windows) |
465471
| `ENABLE_READ_PROFILING` | Enable read profiling instrumentation | `OFF` |
472+
| `TIDESDB_WITH_SNAPPY` | Build with Snappy compression support | `ON` (`OFF` on SunOS) |
473+
| `TIDESDB_WITH_LZ4` | Build with LZ4 compression support | `ON` |
474+
| `TIDESDB_WITH_ZSTD` | Build with Zstandard compression support | `ON` |
475+
| `TIDESDB_SNAPPY_TARGET` | External Snappy link target/item to use instead of autodetect | (empty) |
476+
| `TIDESDB_LZ4_TARGET` | External LZ4 link target/item to use instead of autodetect | (empty) |
477+
| `TIDESDB_ZSTD_TARGET` | External Zstandard link target/item to use instead of autodetect | (empty) |
466478
| `TIDESDB_WITH_S3` | Build S3-compatible object store connector (requires libcurl + OpenSSL) | `OFF` |
467479
| `TIDESDB_WITH_MIMALLOC` | Use mimalloc as the memory allocator | `OFF` |
468480
| `TIDESDB_WITH_TCMALLOC` | Use tcmalloc as the memory allocator | `OFF` |
469481
| `TIDESDB_WITH_JEMALLOC` | Use jemalloc as the memory allocator | `OFF` |
470482
| `CMAKE_BUILD_TYPE` | Build type (Debug/Release/RelWithDebInfo) | (unset) |
471483

484+
### Compression Backends
485+
486+
The Snappy, LZ4, and Zstandard backends are independently optional and all enabled by default. Disable any you do not need:
487+
488+
```bash
489+
# build without Zstandard
490+
cmake -S . -B build -DTIDESDB_WITH_ZSTD=OFF
491+
492+
# zero-dependency build -- no compression libraries linked, TDB_COMPRESS_NONE only
493+
cmake -S . -B build -DTIDESDB_WITH_SNAPPY=OFF -DTIDESDB_WITH_LZ4=OFF -DTIDESDB_WITH_ZSTD=OFF
494+
```
495+
496+
The on-disk compression algorithm IDs are stable regardless of which backends are compiled in, so an SSTable written with an algorithm a given build lacks is reported with a clear error rather than misread. The default column family compression resolves to the best available backend at runtime (LZ4, then Zstandard, then Snappy, then none), so a default configuration always works against whatever was built.
497+
498+
:::note[Consumers need no compression headers]
499+
The third-party compression headers are included only inside the library, so an application that links `libtidesdb` and includes `<tidesdb/tidesdb.h>` does not need the Snappy/LZ4/Zstandard development headers itself.
500+
:::
501+
502+
#### Linking against a vendored compression library
503+
504+
By default each backend links a known CMake target if one exists in the build (for example one provided by a parent project via `FetchContent` or `add_subdirectory`), and otherwise falls back to a system `-l<name>`. If you vendor a compression library under a target name the autodetect cannot guess, point TidesDB at it explicitly:
505+
506+
```bash
507+
# an exact CMake target from your dependency
508+
cmake -S . -B build -DTIDESDB_ZSTD_TARGET=zstd::libzstd_static
509+
510+
# or any target/link item your build produces
511+
cmake -S . -B build -DTIDESDB_ZSTD_TARGET=my_zstd_target
512+
```
513+
514+
This avoids the forced system `-l` link that otherwise fails when the library is only available as an in-tree target rather than installed system-wide. The same applies to `TIDESDB_SNAPPY_TARGET` and `TIDESDB_LZ4_TARGET`.
515+
472516
### Mimalloc Memory Allocator
473517

474518
TidesDB optionally supports [mimalloc](https://github.com/microsoft/mimalloc), Microsoft's high-performance memory allocator. When enabled, mimalloc replaces the standard malloc/free at link time, providing improved allocation performance for memory-intensive workloads.

src/content/docs/reference/tidesql.md

Lines changed: 37 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1184,6 +1184,14 @@ MariaDB [demo]> ANALYZE TABLE products;
11841184

11851185
The output includes the total number of keys, data size, memtable size, number of LSM levels, read amplification factor, cache hit rate, average key and value sizes, and per-level SSTable counts and sizes. For tables with secondary indexes, each index CF's statistics are reported separately.
11861186

1187+
When the column family has logged any user bytes, a per-CF write-amplification note is also emitted:
1188+
1189+
```
1190+
| demo.products | analyze | Note | [TIDESDB] WA user=4096 wal=4096 flush=8192 (1 ssts) compact_write=12288 (1 ssts) compact_read=8192 ratio=6.00x |
1191+
```
1192+
1193+
`user` is the logical bytes the plugin wrote through tidesdb's API, `wal` is the per-CF WAL bytes for that family (zero under unified memtable mode, which uses a single shared WAL accounted for separately in `SHOW ENGINE TIDESDB STATUS`), `flush` and `compact_write` are bytes emitted to SSTables by flush jobs and compaction jobs respectively, along with how many SSTables each produced, `compact_read` is bytes pulled in as compaction input, and `ratio` is `(wal + flush + compact_write) / user` rounded to two decimals. A high ratio with a low SSTable count tends to point at oversized flush memtables, a high ratio with many compactions points at unnecessary L0 churn or an under-sized `L1_FILE_COUNT_TRIGGER`.
1194+
11871195
This information is useful for understanding the physical layout of your data and diagnosing performance characteristics.
11881196

11891197

@@ -1197,6 +1205,12 @@ SHOW ENGINE TIDESDB STATUS\G
11971205

11981206
The output includes the data directory path, number of column families, global sequence number, memory usage (total system memory, resolved memory limit, memory pressure level, memtable bytes, transaction memory bytes), storage metrics (total SSTables, open SSTable handles, total data size, immutable memtable count), background queue sizes (flush pending, flush queue, compaction queue), and block cache statistics (enabled, entries, size, hits, misses, hit rate, partitions).
11991207

1208+
When `tidesdb_unified_memtable=ON` an additional **Unified Memtable** block reports the active skip-list bytes, the number of immutables queued, whether a flush is currently in flight, the current WAL generation, and the round-robin next-CF index that the library uses when it scans for CF state to fold into the shared memtable.
1209+
1210+
A **Write Amplification** block reports cumulative user bytes written through the plugin alongside unified-WAL bytes, per-CF WAL bytes, flush bytes written, compaction bytes written and read, and a derived total WA ratio computed as `(unified_wal + per_cf_wal + flush + compaction_write) / user_bytes_written`. The flush and compaction rows also include their SSTable counts, so the ratio between bytes and SSTable count gives a coarse view of average output-SSTable size at each stage.
1211+
1212+
When an object-store backend is configured an **Object Store** block is included, and the last successfully uploaded unified-WAL generation is reported alongside the local cache footprint, upload queue depth, and lifetime upload counters.
1213+
12001214
When `tidesdb_print_all_conflicts` is enabled, the last conflict event is also displayed under a Conflicts section.
12011215

12021216
### Status Variables
@@ -1209,8 +1223,8 @@ SHOW GLOBAL STATUS LIKE 'tidesdb%';
12091223

12101224
| Variable | Description |
12111225
|----------|-------------|
1212-
| `Tidesdb_version` | TideSQL plugin version string (e.g. `4.5.4`) |
1213-
| `Tidesdb_version_hex` | TideSQL plugin version as integer (e.g. `263940` = `0x40504`) |
1226+
| `Tidesdb_version` | TideSQL plugin version string (e.g. `4.5.5`) |
1227+
| `Tidesdb_version_hex` | TideSQL plugin version as integer (e.g. `263941` = `0x40505`) |
12141228
| `Tidesdb_column_families` | Number of active column families |
12151229
| `Tidesdb_global_sequence` | Global MVCC sequence number |
12161230
| `Tidesdb_memtable_bytes` | Total memtable memory usage in bytes |
@@ -1236,6 +1250,27 @@ SHOW GLOBAL STATUS LIKE 'tidesdb%';
12361250
| `Tidesdb_max_sst_tombstone_density_level` | LSM level of the SSTable with the maximum density |
12371251
| `Tidesdb_backpressure_waits` | Number of times a writer blocked on TidesDB back-pressure inside the plugin |
12381252
| `Tidesdb_backpressure_wait_us` | Total microseconds spent blocked on back-pressure |
1253+
| `Tidesdb_unified_memtable_enabled` | 1 when the unified memtable is active, 0 when each column family carries its own memtable and WAL |
1254+
| `Tidesdb_unified_memtable_bytes` | Live byte usage of the unified memtable's active skip list |
1255+
| `Tidesdb_unified_immutable_count` | Number of unified-memtable immutables queued for flush |
1256+
| `Tidesdb_unified_is_flushing` | 1 while a unified-memtable flush is in flight, 0 otherwise |
1257+
| `Tidesdb_unified_wal_generation` | Current unified-WAL generation number; advances each time the active WAL segment rolls over |
1258+
| `Tidesdb_object_store_enabled` | 1 when an object-store backend is configured (S3 or remote), 0 for purely local storage |
1259+
| `Tidesdb_replica_mode_active` | 1 when this instance is running as a read-only object-store replica |
1260+
| `Tidesdb_local_cache_bytes` | Bytes currently held in the local object-store cache directory |
1261+
| `Tidesdb_local_cache_files` | Number of files in the local object-store cache |
1262+
| `Tidesdb_upload_queue_depth` | Pending uploads waiting on the object-store uploader threads |
1263+
| `Tidesdb_total_uploads` | Lifetime count of successful uploads to the object store |
1264+
| `Tidesdb_upload_failures` | Lifetime count of upload attempts that failed and were retried |
1265+
| `Tidesdb_last_uploaded_generation` | Highest unified-WAL generation acknowledged by the object store (lags `Tidesdb_unified_wal_generation` by the upload queue) |
1266+
| `Tidesdb_uwal_bytes_written` | Bytes written by the plugin to the unified WAL over the process lifetime |
1267+
| `Tidesdb_wal_bytes_written` | Bytes written to per-CF WALs (sum across column families when unified memtable is OFF) |
1268+
| `Tidesdb_flush_bytes_written` | Bytes emitted to L0 SSTables by flush jobs across all column families |
1269+
| `Tidesdb_compaction_bytes_written` | Bytes written by compaction jobs across all column families |
1270+
| `Tidesdb_compaction_bytes_read` | Bytes read by compaction jobs as input across all column families |
1271+
| `Tidesdb_user_bytes_written` | Logical user bytes ingested via the plugin's write path (key + value, before any WAL/SSTable framing) |
1272+
| `Tidesdb_flush_count` | Number of flush operations completed across all column families |
1273+
| `Tidesdb_compaction_count` | Number of compaction operations completed across all column families |
12391274
| `Tidesdb_lock_waits` | Number of times any thread blocked in the pessimistic lock-wait loop |
12401275
| `Tidesdb_lock_wait_us` | Total microseconds spent in the pessimistic lock-wait loop |
12411276
| `Tidesdb_lock_deadlocks` | Number of wait-for-graph cycle detections that returned `ER_LOCK_DEADLOCK` |

0 commit comments

Comments
 (0)