diff --git a/README.md b/README.md index f29613e..5966a74 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,9 @@ [![Dev](https://img.shields.io/badge/docs-dev-blue.svg)](https://JuliaDatabases.github.io/LMDB.jl/dev) Julia bindings for [LMDB](http://www.lmdb.tech/doc/), the Lightning -Memory-Mapped Database: An embedded, memory-mapped, ACID key-value store -developed by Symas for OpenLDAP. Small, fast, persisted to disk, and reads at -near in-memory speeds. +Memory-Mapped Database. LMDB is an embedded, memory-mapped, ACID key-value +store developed by Symas for OpenLDAP. It persists to disk while reading +at near in-memory speeds. ```julia using Pkg; Pkg.add("LMDB") @@ -18,15 +18,15 @@ using Pkg; Pkg.add("LMDB") LMDB.jl exposes the same database through three surfaces: -- **High-level interface**: `LMDBDict <: AbstractDict`, an +- High-level interface: `LMDBDict <: AbstractDict`, an `AbstractDict{K,V}` over a single LMDB file. Standard library machinery (`merge!`, `filter!`, `pairs`, iteration, …) works out of the box. Reach for this when you want a persistent `Dict`. -- **Julia wrappers**: `Environment`, `Transaction`, `DBI`, `Cursor`. +- Julia wrappers: `Environment`, `Transaction`, `DBI`, `Cursor`. Julia-shaped wrappers around handles, transactions, and cursors, - with finalizers, `do`-block forms, etc. Use this when you want - explicit transactions. -- **C API**: `LMDB.mdb_*` and `LMDB.MDB_*`. Raw `ccall` bindings and + with finalizers, `do`-block forms, and so on. Use these when you + want explicit transactions. +- C API: `LMDB.mdb_*` and `LMDB.MDB_*`. Raw `ccall` bindings and status-code constants. Use this when the Julia wrappers don't expose a particular API or you want to inspect status codes directly. @@ -86,8 +86,8 @@ end The package decodes `String`, `Vector{T}` for any bitstype `T`, and the primitive numeric types out of the box. To plug in a custom representation, -define a `Base.read(io::IO, ::Type{T})` method; it will be picked up by `tryget` -/ `get` / `walk(f, cur, K, V)` and the cursor accessors `key`/`value`/`item`. +define a `Base.read(io::IO, ::Type{T})` method; it will be picked up by `tryget`, +`get`, `walk(f, cur, K, V)`, and the cursor accessors `key`/`value`/`item`. Status-code matchers live on `LMDBError`: ```julia diff --git a/docs/src/index.md b/docs/src/index.md index 5556271..c7a905d 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -4,9 +4,8 @@ Memory-Mapped Database.* LMDB is an embedded, memory-mapped, ACID key-value store developed by -Symas for OpenLDAP. It is small, fast, and persists to disk while reading -at near in-memory speeds — limited only by the size of the virtual address -space. +Symas for OpenLDAP. It persists to disk while reading at near in-memory +speeds, limited only by the size of the virtual address space. ```julia using Pkg; Pkg.add("LMDB") @@ -18,16 +17,16 @@ LMDB.jl exposes the same database through three surfaces: | Surface | What it offers | When to use | |---------|----------------|-------------| -| **High-level interface** | `LMDBDict <: AbstractDict{K,V}` | When you want a persistent `Dict`. | -| **Julia wrappers** | `Environment`, `Transaction`, `DBI`, `Cursor` | When you want explicit transactions and cursors with Julia-shaped wrappers. | -| **C API** | `LMDB.mdb_*`, `LMDB.MDB_*` | Raw `ccall` bindings and status-code constants, for custom data layouts or when you want to skip allocations on hot paths. | +| High-level interface | `LMDBDict <: AbstractDict{K,V}` | When you want a persistent `Dict`. | +| Julia wrappers | `Environment`, `Transaction`, `DBI`, `Cursor` | When you want explicit transactions and cursors with Julia-shaped wrappers. | +| C API | `LMDB.mdb_*`, `LMDB.MDB_*` | Raw `ccall` bindings and status-code constants, for custom data layouts or when you want to skip allocations on hot paths. | `MDBValue`, `MDBArg`, and the [`MDBValueIO`](@ref LMDB.MDBValueIO) type sit between the C API and the Julia wrappers. `MDBValueIO` is an `IO` view over `MDB_val`; defining `Base.read(io, T)` on it is how you teach the typed reads about a custom value type. -The Usage section starts simple and gets more involved: [Essentials](@ref) +The Usage section starts simple and gets more involved. [Essentials](@ref) has a working example, [Dictionary interface](@ref) covers `LMDBDict`, and [Environments](@ref), [Transactions](@ref), [Databases](@ref), [Cursors](@ref), and [Duplicate-sort databases](@ref) cover the wrappers. diff --git a/docs/src/lib/cursors.md b/docs/src/lib/cursors.md index 575a2d9..9bb34ee 100644 --- a/docs/src/lib/cursors.md +++ b/docs/src/lib/cursors.md @@ -5,7 +5,7 @@ CurrentModule = LMDB ``` A `Cursor` is a positioned iterator over the entries in a `DBI`. Cursors -are bound to a transaction; closing the txn invalidates the cursor. +are bound to a transaction. Closing the txn invalidates the cursor. ## Construction diff --git a/docs/src/lib/databases.md b/docs/src/lib/databases.md index a5d9832..d405c20 100644 --- a/docs/src/lib/databases.md +++ b/docs/src/lib/databases.md @@ -30,7 +30,7 @@ tryget ``` `get(txn, dbi, key, T, default)` falls back to `default` if `key` is -missing — same shape as `Base.get(dict, key, default)`. +missing, matching `Base.get(dict, key, default)`. ## Writes diff --git a/docs/src/lib/dict.md b/docs/src/lib/dict.md index 8ad44af..d501dc3 100644 --- a/docs/src/lib/dict.md +++ b/docs/src/lib/dict.md @@ -23,17 +23,17 @@ Writes (`setindex!`, `delete!`, `pop!`, `empty!`) likewise. `delete!` silently no-ops on a missing key, matching `Base.delete!`'s "if any" contract. -Everything `AbstractDict` derives — `merge!`, `merge`, `mergewith!`, +Everything `AbstractDict` derives (`merge!`, `merge`, `mergewith!`, `filter!`, `filter`, `==`, `isequal`, `hash`, `in(::Pair, d)`, -`copy(d)` — comes along for free. +`copy(d)`) comes along for free. -`LMDBDict` iterates in lexicographic key order, which is stronger than +`LMDBDict` iterates in lexicographic key order, which is stricter than `Base.Dict`'s no-order promise. ## Lifecycle `close(::LMDBDict)` closes the underlying env (and the default DBI). -Idempotent — also called from the finalizer. +Idempotent, and also called from the finalizer. ## Prefix-scan helpers diff --git a/docs/src/lib/errors.md b/docs/src/lib/errors.md index fa80507..1b7de2d 100644 --- a/docs/src/lib/errors.md +++ b/docs/src/lib/errors.md @@ -6,8 +6,8 @@ CurrentModule = LMDB Every LMDB-internal error surfaces as an `LMDBError` whose `code` field is the raw status `Cint` returned by the underlying binding. Status-code -matchers cover the three most common branches; less common codes can be -matched against `LMDB.MDB_*` constants directly. +matchers cover the most common branches; less common codes can be matched +against `LMDB.MDB_*` constants directly. ## Exception type @@ -54,7 +54,7 @@ branch on `MDB_NOTFOUND`/`MDB_KEYEXIST` and friends. At the Julia wrappers, handle methods that wrap status-returning bindings let `LMDBError` propagate. `tryget` and `get(..., default)` -swallow `MDB_NOTFOUND` and return `nothing`/`default`; +swallow `MDB_NOTFOUND` and return `nothing` or `default`. `delete!(txn, dbi, key)` likewise swallows `MDB_NOTFOUND` and returns `false`. diff --git a/docs/src/lib/lowlevel.md b/docs/src/lib/lowlevel.md index 112c47d..b8dc802 100644 --- a/docs/src/lib/lowlevel.md +++ b/docs/src/lib/lowlevel.md @@ -19,7 +19,7 @@ for callers that need to inspect the raw status code. Every status-returning binding in `liblmdb.jl` is paired with an `unchecked_*` companion at definition time. Use the bare name when any -error should propagate (the common case); use `unchecked_*` when you +error should propagate (the common case). Use `unchecked_*` when you need to inspect the raw `Cint` yourself, for example to distinguish `MDB_NOTFOUND` from a real error: @@ -34,15 +34,15 @@ Bindings that return non-status data (`mdb_strerror`, `mdb_version`, `mdb_txn_id`, `mdb_cmp`, `mdb_dcmp`, `mdb_env_get_maxkeysize`, `mdb_env_get_userctx`, `mdb_cursor_txn`, `mdb_cursor_dbi`) and `Cvoid`-returning ones (`mdb_env_close`, `mdb_dbi_close`, -`mdb_txn_abort`, `mdb_txn_reset`, `mdb_cursor_close`) are left bare; -there is nothing to check. +`mdb_txn_abort`, `mdb_txn_reset`, `mdb_cursor_close`) are left bare, +since there is nothing to check. ## Customisation point: `MDBValueIO` -`tryget` / `get` / `key` / `value` / `item` / typed `walk` / `pop!` / +`tryget`, `get`, `key`, `value`, `item`, typed `walk`, `pop!`, and `replace!` all go through `read(::MDBValueIO, T)` to decode an `MDB_val` into a Julia value. Define a `Base.read` method on -`MDBValueIO` to plug in a custom representation; see [Cursors](@ref) +`MDBValueIO` to plug in a custom representation. See [Cursors](@ref) for a worked example. ```@docs diff --git a/docs/src/man/cursors.md b/docs/src/man/cursors.md index df95220..abb7a1c 100644 --- a/docs/src/man/cursors.md +++ b/docs/src/man/cursors.md @@ -5,8 +5,8 @@ CurrentModule = LMDB ``` A `Cursor` is a positioned iterator over a `DBI`. Use it for ordered -scans, range queries, or when you want to amortise the per-lookup -overhead of `mdb_get` across many keys. +scans, range queries, or to amortise the per-lookup overhead of +`mdb_get` across many keys. ## Opening a cursor @@ -21,8 +21,8 @@ end ``` A cursor is bound to its transaction; closing the txn invalidates the -cursor. The cursor's finalizer is idempotent, so a still-open cursor -is reclaimed when GC visits it. +cursor. The cursor's finalizer is idempotent, so a still-open cursor is +reclaimed when GC visits it. ## Navigation @@ -30,10 +30,10 @@ Each navigation function repositions the cursor and returns the new key, or `nothing` if the move would step past the end: ```julia -seek!(cur) # MDB_FIRST — first entry -seek_last!(cur) # MDB_LAST — last entry -seek!(cur, key) # MDB_SET_KEY — exact key match -seek_range!(cur, key) # MDB_SET_RANGE — smallest key ≥ `key` +seek!(cur) # MDB_FIRST first entry +seek_last!(cur) # MDB_LAST last entry +seek!(cur, key) # MDB_SET_KEY exact key match +seek_range!(cur, key) # MDB_SET_RANGE smallest key ≥ `key` next!(cur) # MDB_NEXT prev!(cur) # MDB_PREV ``` @@ -80,24 +80,24 @@ start(env; flags = MDB_RDONLY) do txn end ``` -For the same pattern *one level up* (already wrapped, returns a +For the same pattern one level up (already wrapped, returns a `Vector{Pair}`), use [`LMDB.scan(d; prefix)`](@ref LMDB.scan) on an `LMDBDict`. -## [Bulk walk — zero-copy iteration](@id man-cur-walk) +## [Bulk walk: zero-copy iteration](@id man-cur-walk) `walk` runs a callback over every entry the cursor visits. It exists in two shapes: ```julia -# Untyped — receives Ref{MDB_val} pairs (zero-copy, mmap pointers) +# Untyped: receives Ref{MDB_val} pairs (zero-copy, mmap pointers) walk(cur) do k_ref, v_ref kv = k_ref[]; vv = v_ref[] # kv.mv_data / vv.mv_data are mmap pointers, valid in this scope do_something(kv.mv_size, vv.mv_size) end -# Typed — runs each ref through `read(MDBValueIO, K)` / `read(MDBValueIO, V)` +# Typed: runs each ref through `read(MDBValueIO, K)` / `read(MDBValueIO, V)` walk(cur, String, Vector{UInt8}) do k::String, v::Vector{UInt8} println(k, " => ", length(v), " bytes") end @@ -110,12 +110,12 @@ The callback can return `false` to stop iteration; any other return (including `nothing`) continues. Use the untyped form when you want to inspect raw byte sizes, copy -slices, or feed a custom decoder; the data pointers are into LMDB's +slices, or feed a custom decoder. The data pointers are into LMDB's mmap and are valid only inside the callback (and only for the surrounding txn). The typed form is the iteration analogue of `tryget(..., T)` and works for any `T` for which `Base.read(io::IO, ::Type{T})` (or `Base.read(io::LMDB.MDBValueIO, ::Type{T})`) is -defined; see [Custom value decoding](@ref). +defined. See [Custom value decoding](@ref). ## Cursor mutation @@ -134,14 +134,14 @@ key (1 in non-DUPSORT databases). ## Custom value decoding -`tryget` / `get` / `key` / `value` / `item` / typed `walk` all funnel +`tryget`, `get`, `key`, `value`, `item`, and typed `walk` all funnel through `Base.read(io::IO, ::Type{T})` against an [`MDBValueIO`](@ref LMDB.MDBValueIO). The defaults cover Base's primitive numeric types (`Int8`/…/`Float64`, `Bool`, `Char`, `Ptr`), `String`, and (added by this package) `Vector{E}` for any bitstype `E`. For everything else, including `isbitstype` structs and framed -values, define a single `Base.read` method on the abstract `IO`: +values, define a single `Base.read` method on the abstract `IO`. ```julia struct PrefixedBlob end @@ -165,7 +165,7 @@ end decoders end up reading like any other Julia binary parser, and the same decoder works against any byte source. This is the analogue of heed's `BytesDecode<'txn>` trait, expressed through Julia's existing IO -extension point rather than a bespoke trait. +extension point instead of a bespoke trait. ## Reset and renew diff --git a/docs/src/man/databases.md b/docs/src/man/databases.md index 399dd8a..975850d 100644 --- a/docs/src/man/databases.md +++ b/docs/src/man/databases.md @@ -24,7 +24,7 @@ open(txn, "users") do dbi end ``` -In practice you'll rarely *want* to close a DBI handle explicitly: the +In practice you'll rarely *want* to close a DBI handle explicitly. The env owns it, and `mdb_dbi_close` is documented as rarely useful. The env's finalizer cascades through any open DBI handles. @@ -53,8 +53,8 @@ get(txn, dbi, key, T, default) # default on miss ``` `T` is anything `read(::LMDB.MDBValueIO, ::Type{T})` knows how to -decode — `String`, `Vector{E}` for any bitstype `E`, or any bitstype -scalar: +decode: `String`, `Vector{E}` for any bitstype `E`, or any bitstype +scalar. ```julia tryget(txn, dbi, "name", String) # → Union{String, Nothing} @@ -82,7 +82,7 @@ Useful write flags: |------|---------| | `MDB_NOOVERWRITE` | fail with `MDB_KEYEXIST` if `key` is already present | | `MDB_NODUPDATA` | (DUPSORT) fail if the `(key, val)` pair already exists | -| `MDB_APPEND` | append; only valid if the new key sorts after every existing key — *much* faster for sorted bulk loads | +| `MDB_APPEND` | append; only valid if the new key sorts after every existing key. Much faster for sorted bulk loads | ```julia # Bulk import in sorted order: @@ -98,7 +98,7 @@ end `replace!` and `pop!` do the read-modify pair inside the same transaction, so there is no time-of-check / time-of-use gap. -## `put_reserved!` — write directly into the mmap +## `put_reserved!`: write directly into the mmap When the value is large or assembled from multiple sources, you can skip the intermediate `Vector{UInt8}` round-trip and write straight @@ -115,8 +115,8 @@ end inside the callback, and only inside the surrounding write txn; don't escape it. -`put_reserved!` is the equivalent of heed's `Database::put_reserved`, -and is incompatible with DUPSORT. +`put_reserved!` is the equivalent of heed's `Database::put_reserved`. +It is incompatible with DUPSORT. ## Stats @@ -137,4 +137,4 @@ drop(txn, dbi; delete = true) # delete the DB and close the handle For named sub-DBs, `delete = true` removes the entry from the env's main DB. For the main DB itself, `delete = true` is treated as -`delete = false` (LMDB cannot delete its own root). +`delete = false`, since LMDB cannot delete its own root. diff --git a/docs/src/man/dict.md b/docs/src/man/dict.md index 8ba4ce6..2ffaac2 100644 --- a/docs/src/man/dict.md +++ b/docs/src/man/dict.md @@ -36,7 +36,7 @@ tasks. ## Storing and retrieving Anything that round-trips through the package's `MDB_val` glue and -[`MDBValueIO`](@ref LMDB.MDBValueIO) works as a value type — `String`, +[`MDBValueIO`](@ref LMDB.MDBValueIO) works as a value type: `String`, `Vector{T}` for any bitstype `T`, and any bitstype scalar (`Int`, `Float32`, `(Int, UInt32)` `Tuple`, …). @@ -66,7 +66,7 @@ for (k, v) in d end ``` -Iteration is in **lexicographic key order** — strictly stronger than +Iteration is in lexicographic key order, which is stricter than `Base.Dict`'s no-order promise. Each `for` loop opens a fresh read transaction; the txn is committed on normal exit and aborted (via the `Transaction` finalizer) on early break or throw. @@ -97,7 +97,7 @@ filter!(((k, v),) -> v > 0, d) ## Prefix-scoped scans For hierarchical key schemes (e.g. `"users/123/name"`), LMDB's -lexicographic order makes prefix scans cheap — they're a single +lexicographic order makes prefix scans cheap: a single `MDB_SET_RANGE` plus iteration until the prefix stops matching. ```julia @@ -119,7 +119,7 @@ LMDB.scan(d, prefix = "users/2/") # "users/2/name" => "Bob" ``` -For directory-style listings — leaf keys appear as-is, anything with +For directory-style listings, leaf keys appear as-is and anything with the separator after the prefix collapses to its first segment: ```julia @@ -127,7 +127,7 @@ LMDB.list_dirs(d, prefix = "") # ["other", "users/"] LMDB.list_dirs(d, prefix = "users/") # ["users/1/", "users/2/"] ``` -`LMDB.valuesize(d; prefix)` sums byte sizes — useful for quick storage +`LMDB.valuesize(d; prefix)` sums byte sizes. Useful for quick storage audits without `stat`. ## When to drop down diff --git a/docs/src/man/dupsort.md b/docs/src/man/dupsort.md index 2eb5e6e..c463ea0 100644 --- a/docs/src/man/dupsort.md +++ b/docs/src/man/dupsort.md @@ -7,7 +7,7 @@ CurrentModule = LMDB By default each key in an LMDB database has a single value. Opening a DB with `MDB_DUPSORT` instead allows multiple values per key, kept in sorted order. This is what LMDB offers for inverted indexes, -many-to-many edges, time-series buckets, and other "key → set of +many-to-many edges, time-series buckets, and similar "key → set of values" patterns. ```julia @@ -40,21 +40,21 @@ DUPSORT wins when: (`seek_range!` style). It loses if you need fast aggregate reads of every value at a key. For -that case, use `MDB_DUPFIXED` (fixed-size duplicates), which stores the +that case, use `MDB_DUPFIXED` (fixed-size duplicates), which stores values contiguously and returns them in batches. ## Navigation -DUPSORT layers an extra dimension on the cursor: the cursor is +DUPSORT layers an extra dimension on the cursor. The cursor is positioned at a `(key, value)` pair, and you can navigate either across keys or across values within the current key. ```julia -seek!(cur, "a") # position at (a, b) — the first dup +seek!(cur, "a") # position at (a, b), the first dup seek_first_dup!(cur) # value of first dup of current key next_dup!(cur) # next dup of current key → (a, c) next_dup!(cur) # → (a, d) -next_dup!(cur) # nothing — out of dups for "a" +next_dup!(cur) # nothing, out of dups for "a" next_nodup!(cur) # skip to next key, first dup → (b, c) ``` @@ -62,7 +62,7 @@ next_nodup!(cur) # skip to next key, first dup → (b, c) |----------|---------|-----------------|------------------| | `next!` | `MDB_NEXT` | yes (next dup) | yes (next key when dups exhausted) | | `prev!` | `MDB_PREV` | yes | yes | -| `next_dup!` | `MDB_NEXT_DUP` | yes | no — `nothing` past last dup | +| `next_dup!` | `MDB_NEXT_DUP` | yes | no, `nothing` past last dup | | `prev_dup!` | `MDB_PREV_DUP` | yes | no | | `next_nodup!` | `MDB_NEXT_NODUP` | jump out | yes (first dup of next key) | | `prev_nodup!` | `MDB_PREV_NODUP` | jump out | yes (first dup of previous key) | @@ -88,11 +88,11 @@ the rest of the dups intact. comparison. - `MDB_DUPSORT | MDB_DUPFIXED`: every duplicate has the same byte size; LMDB stores them as a packed array per key. Required for the - `MDB_GET_MULTIPLE` / `MDB_NEXT_MULTIPLE` cursor ops (reachable from - the C API). + `MDB_GET_MULTIPLE` / `MDB_NEXT_MULTIPLE` cursor ops, which are + reachable from the C API. - `MDB_DUPSORT | MDB_DUPFIXED | MDB_INTEGERDUP`: values are native-endian integers; sorted numerically. - `MDB_DUPSORT | MDB_REVERSEDUP`: values compared back-to-front. -`MDB_RESERVE` (and therefore `put_reserved!`) is **not** valid against -a DUPSORT database — LMDB rejects it. +`MDB_RESERVE` (and therefore `put_reserved!`) is not valid against a +DUPSORT database; LMDB rejects it. diff --git a/docs/src/man/environments.md b/docs/src/man/environments.md index 9d0e466..8af14e7 100644 --- a/docs/src/man/environments.md +++ b/docs/src/man/environments.md @@ -63,18 +63,18 @@ end | `MDB_NOMETASYNC` | `fsync` data but not metadata | | `MDB_WRITEMAP` | mmap as writable; faster but requires more discipline (no torn writes from other processes) | | `MDB_NOMEMINIT` | skip zero-init of new pages | -| `MDB_NOTLS` | drop thread-local reader slots — needed for multiple read txns on one thread | -| `MDB_NORDAHEAD` | turn off OS-level read-ahead — better for cold-page workloads | +| `MDB_NOTLS` | drop thread-local reader slots, needed for multiple read txns on one thread | +| `MDB_NORDAHEAD` | turn off OS-level read-ahead, better for cold-page workloads | | `MDB_NOLOCK` | the caller takes responsibility for locking | -`MDB_RDONLY` can only be set at `open` time — calling `set!(env, +`MDB_RDONLY` can only be set at `open` time. Calling `set!(env, MDB_RDONLY)` on an open env will return `EINVAL`. ## Sizing the map `mapsize` is a *virtual* limit on the env's address space, not the on-disk size. Pick a generous power of two (say, 1 GiB or 8 GiB) up -front; the on-disk file grows incrementally as data is written. +front. The on-disk file grows incrementally as data is written. If a write txn would exceed `mapsize`, LMDB returns `MDB_MAP_FULL` (catchable via [`is_map_full(::LMDBError)`](@ref is_map_full)). To diff --git a/docs/src/man/essentials.md b/docs/src/man/essentials.md index b662476..0470859 100644 --- a/docs/src/man/essentials.md +++ b/docs/src/man/essentials.md @@ -47,15 +47,15 @@ or any bitstype scalar. LMDB.jl exposes the same database three ways, in increasing order of control: -- The **high-level interface** ([`LMDBDict`](@ref)) is the - `AbstractDict{K,V}` surface — start here unless you need +- The high-level interface ([`LMDBDict`](@ref)) is the + `AbstractDict{K,V}` surface. Start here unless you need transactional grouping or zero-copy reads. -- The **Julia wrappers** (`Environment`, `Transaction`, `DBI`, +- The Julia wrappers (`Environment`, `Transaction`, `DBI`, `Cursor`) give you explicit lifetimes and fine-grained control with finalizers, parent refs, and `do`-block forms. Drop down to these via [Environments](@ref) → [Transactions](@ref) → [Databases](@ref) → [Cursors](@ref). -- The **C API** (`mdb_*`, `MDB_*`, `unchecked_mdb_*`) is the raw +- The C API (`mdb_*`, `MDB_*`, `unchecked_mdb_*`) is the raw `@ccall` surface. `MDBValue`, `MDBArg`, and `MDBValueIO` glue Julia values to `Ptr{MDB_val}` and let custom decoders plug in via `Base.read(io, T)`. Reach for the [Low-level bindings](@ref) only @@ -77,8 +77,8 @@ with a finalizer: Parent references pin the lifetime: a `Cursor` keeps its `Transaction` alive, which keeps its `Environment` alive. `close`, `commit`, and `abort` are idempotent: calling them twice, or on a handle that was -never opened, is a silent no-op. So an abandoned write txn — say, from -a `for … break` over an `LMDBDict`, or any error path — gets reclaimed +never opened, is a silent no-op. An abandoned write txn (say, from a +`for … break` over an `LMDBDict`, or any error path) gets reclaimed when GC runs. The do-block constructors are usually what you want: diff --git a/docs/src/man/lowlevel.md b/docs/src/man/lowlevel.md index cc9315a..c63cd7d 100644 --- a/docs/src/man/lowlevel.md +++ b/docs/src/man/lowlevel.md @@ -11,7 +11,7 @@ integrate with a custom data layout, branch on a status code that the Julia wrappers don't surface, or skip allocations on a hot path. For the full inventory, see [the API reference](@ref API-LowLevel). What -follows is a tour of how the surface is shaped. +follows is an overview of how the surface is shaped. ## The auto-throwing convention @@ -25,7 +25,7 @@ LMDB.unchecked_mdb_env_open(env, path, flags, mode) # returns the raw Cint Use the bare name when any error should propagate (the common case). Use the `unchecked_*` companion when you need to inspect the raw status -yourself — e.g. distinguishing `MDB_NOTFOUND` from a real error: +yourself, for example to distinguish `MDB_NOTFOUND` from a real error: ```julia val_ref = Ref(LMDB.MDB_val(zero(Csize_t), C_NULL)) @@ -41,7 +41,7 @@ Bindings that don't return a status (`mdb_strerror`, `mdb_version`, `mdb_txn_id`, `mdb_cmp`, `mdb_dcmp`, `mdb_env_get_maxkeysize`, `mdb_cursor_txn`, `mdb_cursor_dbi`) and `Cvoid`-returning ones (`mdb_env_close`, `mdb_dbi_close`, `mdb_txn_abort`, `mdb_txn_reset`, -`mdb_cursor_close`) are left bare — there is nothing to check. +`mdb_cursor_close`) are left bare; there is nothing to check. ## ccall glue: passing values to `Ptr{MDB_val}` @@ -52,7 +52,7 @@ ccall to fill in. LMDB.jl ships `Base.cconvert` overloads on type), `Base.RefValue` over a bitstype, any bitstype scalar, and a pre-built `Ref{MDB_val}` (used as an out-param). Each routes through a self-rooted argument that `ccall`'s automatic `GC.@preserve` keeps -alive across the call, so callers don't have to write `Ref(...)` or +alive across the call, so callers do not have to write `Ref(...)` or `GC.@preserve` for input arguments themselves. ```julia @@ -91,7 +91,7 @@ The package ships these defaults: |-----|-----------| | `String` | one `unsafe_string` over the remaining bytes | | `Vector{E}` for bitstype `E` | one alloc + `unsafe_copyto!`; the buffer is Julia-owned | -| any bitstype scalar `T` | one `unsafe_load` of `sizeof(T)` bytes — zero allocations | +| any bitstype scalar `T` | one `unsafe_load` of `sizeof(T)` bytes (zero allocations) | Add custom representations by overloading `Base.read` on the abstract `IO`. This is the idiomatic Julia form and keeps the decoder portable @@ -121,8 +121,7 @@ so one method definition makes a custom representation usable across the package. Because `MDBValueIO <: IO`, the standard `Base` IO primitives (`position`, `seek`, `skip`, `read(io)`, `read(io, n::Integer)`, `read!(io, A)`, `bytesavailable`, `eof`) work out of the -box, so structured framed-value decoders read exactly like any other -Julia parser. +box. Structured framed-value decoders read like any other Julia parser. ## Memory ownership rules @@ -141,11 +140,11 @@ Julia parser. A few LMDB features are reachable only through the C API because the Julia wrappers deliberately don't include them: -- **Custom comparators.** `LMDB.mdb_set_compare` / +- Custom comparators: `LMDB.mdb_set_compare` and `LMDB.mdb_set_dupsort` accept a `MDB_cmp_func` callback. Use `@cfunction` to lift a Julia function into the right C signature. -- **`mdb_set_relfunc` / `mdb_set_relctx`.** Used by +- `mdb_set_relfunc` / `mdb_set_relctx`: used by `MDB_FIXEDMAP`-style relocations; rarely needed. -- **`MDB_GET_MULTIPLE` / `MDB_NEXT_MULTIPLE` cursor ops.** Reachable by +- `MDB_GET_MULTIPLE` / `MDB_NEXT_MULTIPLE` cursor ops: reachable by passing the constant directly to `LMDB.mdb_cursor_get`. Useful with `MDB_DUPFIXED` databases for batched reads. diff --git a/docs/src/man/transactions.md b/docs/src/man/transactions.md index 3c93e5c..29a9bb9 100644 --- a/docs/src/man/transactions.md +++ b/docs/src/man/transactions.md @@ -5,8 +5,8 @@ CurrentModule = LMDB ``` Every LMDB operation runs inside a transaction. Transactions are either -**read-only** (any number can run concurrently) or **read-write** (one -at a time per environment). +read-only (any number can run concurrently) or read-write (one at a +time per environment). ## Starting a transaction @@ -57,13 +57,13 @@ for batch in batches end end reset(txn) # release the reader slot but keep the handle - renew(txn) # acquire a fresh slot — sees newly-committed writes + renew(txn) # acquire a fresh slot; sees newly-committed writes end abort(txn) ``` `reset` is only valid on read-only txns. `renew` fetches a fresh -database snapshot; without it, the parked txn won't see writes that +database snapshot. Without it, the parked txn won't see writes that landed in the meantime. ## Sub-transactions @@ -95,12 +95,12 @@ write txn. ## Reader slots Each open read txn occupies one reader slot. The default `maxreaders` -is small (126); raise it via `Environment(...; maxreaders = N)` for +is small (126). Raise it via `Environment(...; maxreaders = N)` for high-concurrency read workloads, or call [`reader_check(env)`](@ref) to reap slots left behind by crashed processes. Aggressive `for … break` over an `LMDBDict` without GC pressure can -pile up read txns. If that's a concern, use +pile up read txns. If that becomes a problem, use [`walk(f, cur)`](@ref API-Cur-walk) inside an explicit `open(txn) do …` block instead. @@ -109,13 +109,13 @@ pile up read txns. If that's a concern, use The most common patterns: ```julia -# Hot read path — many small lookups, no writes +# Hot read path: many small lookups, no writes start(env; flags = MDB_RDONLY) do txn ... end -# Bulk import — single transaction across many writes (atomic, fast) +# Bulk import: single transaction across many writes (atomic, fast) start(env) do txn ... end -# Long-running reader (e.g. background scrubber) — reset + renew loop +# Long-running reader (e.g. background scrubber): reset + renew loop txn = start(env; flags = MDB_RDONLY) while running ... diff --git a/src/LMDB.jl b/src/LMDB.jl index aede100..6b63a80 100644 --- a/src/LMDB.jl +++ b/src/LMDB.jl @@ -23,18 +23,18 @@ export # commonly-needed write flags MDB_NOOVERWRITE, MDB_NODUPDATA, MDB_APPEND, MDB_RESERVE, - # Julia wrappers — environment + # Julia wrappers: environment Environment, create, environment, sync, set!, unset!, info, stat, path, isopen, isflagset, reader_check, reader_list, - # Julia wrappers — transaction + # Julia wrappers: transaction Transaction, start, abort, commit, reset, renew, - # Julia wrappers — database (DBI) + # Julia wrappers: database (DBI) DBI, drop, get, put!, put_reserved!, delete!, tryget, replace!, - # Julia wrappers — cursor + # Julia wrappers: cursor Cursor, count, transaction, database, seek!, seek_last!, seek_range!, next!, prev!, key, value, item, walk, @@ -100,7 +100,7 @@ done after `close(env)` without rewriting the database). is_map_full # --------------------------------------------------------------------------- -# C API — raw bindings, types, constants. Public-but-unexported. +# C API: raw bindings, types, constants. Public-but-unexported. # # Every status-returning binding has a `@checked` wrapper (auto-throws) and an # `unchecked_*` companion (returns the raw `Cint` for callers that need to @@ -140,7 +140,7 @@ include("dbi.jl") include("cur.jl") # --------------------------------------------------------------------------- -# High-level interface — `LMDBDict`. +# High-level interface: `LMDBDict`. # --------------------------------------------------------------------------- include("dicts.jl") diff --git a/src/checked.jl b/src/checked.jl index 5535a51..2894268 100644 --- a/src/checked.jl +++ b/src/checked.jl @@ -1,10 +1,10 @@ # Applied to a C-API binding that returns an LMDB status code (`Cint`). # Emits two functions: # -# * `(...)` — same name, throws `LMDBError` on a non-zero -# status; returns the status (always 0) otherwise. -# * `unchecked_(...)` — returns the raw status; the caller decides what -# to do (e.g. branch on `MDB_NOTFOUND`). +# * `(...)` same name, throws `LMDBError` on a non-zero +# status; returns the status (always 0) otherwise. +# * `unchecked_(...)` returns the raw status; the caller decides what +# to do (e.g. branch on `MDB_NOTFOUND`). # # Used in `liblmdb.jl` for every binding whose return type is a status. Bindings # that return a value (`mdb_strerror`, `mdb_txn_id`, comparators, …) or are diff --git a/src/common.jl b/src/common.jl index 0bb54d0..0be9c7e 100644 --- a/src/common.jl +++ b/src/common.jl @@ -1,4 +1,4 @@ -# Zero-valued `MDB_val` sentinels — used as out-parameters and for the +# Zero-valued `MDB_val` sentinels, used as out-parameters and for the # "no value" form of `delete!`. Constructing a non-empty `MDB_val` from a # Julia value requires taking a raw pointer into that value, which is only # safe when the value is GC-preserved across the eventual ccall. We keep @@ -11,7 +11,7 @@ MDBValue(::Nothing) = MDBValue() # uninitialized `Ref{MDB_val}`; `data` is the Julia-owned buffer whose # pointer the C call needs to see. `cconvert` returns an `MDBArg`, ccall # preserves it across the call, and `unsafe_convert` (below) is the one -# place pointer extraction happens — that means `data` is provably alive +# place pointer extraction happens, so `data` is provably alive # at the moment its pointer is taken. struct MDBArg{D} box::Base.RefValue{MDB_val} @@ -19,7 +19,7 @@ struct MDBArg{D} MDBArg(data::D) where {D} = new{D}(Ref{MDB_val}(), data) end -# Lazy pointer extraction. Runs while ccall is preserving `m`; therefore +# Lazy pointer extraction. Runs while ccall is preserving `m`, so # `m.data` is alive at the point we ask it for a pointer, satisfying the # Julia GC contract for `Base.unsafe_convert`. @inline function Base.unsafe_convert(::Type{Ptr{MDB_val}}, m::MDBArg{String}) @@ -43,11 +43,11 @@ Base.cconvert(::Type{Ptr{MDB_val}}, x::MDB_val) = Ref(x) # Pre-built `Ref{MDB_val}` (iterator state, `get` out-param): passthrough; # ccall reads/writes the box directly. Base.cconvert(::Type{Ptr{MDB_val}}, x::Base.RefValue{MDB_val}) = x -# User input — package the data, defer pointer extraction. +# User input: package the data, defer pointer extraction. Base.cconvert(::Type{Ptr{MDB_val}}, x::String) = MDBArg(x) Base.cconvert(::Type{Ptr{MDB_val}}, x::Array) = MDBArg(x) -# Other AbstractArrays that support `unsafe_convert(Ptr{T}, x)` flow through — -# contiguous `SubArray`, `ReinterpretArray`, etc. Non-contiguous inputs +# Other AbstractArrays that support `unsafe_convert(Ptr{T}, x)` flow through: +# contiguous `SubArray`, `ReinterpretArray`, and so on. Non-contiguous inputs # surface the standard "cannot take pointer" error from `unsafe_convert`. # The `Array` method above stays as a more-specific overload to break the # ambiguity with `Base.cconvert(::Type{<:Ptr}, ::Array)`. @@ -68,13 +68,13 @@ A read-only `IO` view over an LMDB-owned `MDB_val`. Wraps the package's typed-read path is the standard `Base.read(io, T)`. Any `T` for which `Base.read(io::IO, ::Type{T})` is defined can be -passed to `tryget` / `get` / `key` / `value` / `item` / typed `walk` / -`pop!` / `replace!`. Out of the box this covers everything Base ships: -the primitive numeric types (`Int8`/…/`Int128`, `Float16`/…/`Float64`, -`Bool`, `Char`, `Ptr{T}`) plus `String`, all zero-allocation thanks to -the `@inline` `unsafe_read` override below. The package adds two more -overloads, `Vector{E}` for any bitstype `E` and `UInt8`, that consume -the remaining buffer in a single copy. +passed to `tryget`, `get`, `key`, `value`, `item`, typed `walk`, +`pop!`, and `replace!`. Out of the box this covers everything Base +ships: the primitive numeric types (`Int8`/…/`Int128`, `Float16`/…/ +`Float64`, `Bool`, `Char`, `Ptr{T}`) plus `String`, all zero-allocation +thanks to the `@inline` `unsafe_read` override below. The package adds +two more overloads, `Vector{E}` for any bitstype `E` and `UInt8`, that +consume the remaining buffer in a single copy. To plug in a custom representation (including bitstype structs that Base's primitive reads don't cover), define a single `Base.read` @@ -97,10 +97,10 @@ pattern: Base.read(io::IO, ::Type{T}) = read!(io, Ref{T}())[] This is the analogue of heed's `BytesDecode<'txn>` trait, expressed -through Julia's existing IO interface rather than a bespoke function. +through Julia's existing IO interface instead of a bespoke function. The underlying buffer points into LMDB's mmap and is only valid for -the producing transaction's lifetime; copy out anything you want to +the producing transaction's lifetime. Copy out anything you want to retain past commit/abort. The default `String` and `Vector{E}` reads both copy. """ @@ -165,15 +165,15 @@ end # Note: we deliberately don't define a `Base.read(io::MDBValueIO, ::Type{T}) # where T` catch-all for `isbitstype(T)`. Such a generic conflicts with -# users defining the idiomatic `Base.read(io::IO, ::Type{MyT})` — Julia +# users defining the idiomatic `Base.read(io::IO, ::Type{MyT})`: Julia # treats `(MDBValueIO, Type{T} where T)` and `(IO, Type{MyT})` as # unordered (one is more specific in arg1, the other in arg2), so the # call ambiguates. Instead, we rely on Base's existing # `read(::IO, T::Union{Int8,…,Float64,Bool,Char,Ptr})` specialisations -# for the well-known primitives; our `@inline unsafe_read` above lets +# for the well-known primitives. Our `@inline unsafe_read` above lets # the optimiser elide the `Ref{T}` Base allocates internally, so they -# stay zero-allocation. User-defined types — including `isbitstype` -# structs — just need a one-line `Base.read(io::IO, ::Type{MyT})` method +# stay zero-allocation. User-defined types, including `isbitstype` +# structs, just need a one-line `Base.read(io::IO, ::Type{MyT})` method # defined wherever the user owns the type. # Whole-blob defaults: read everything from the current position to the end. diff --git a/src/cur.jl b/src/cur.jl index 7809ba5..baf4e14 100644 --- a/src/cur.jl +++ b/src/cur.jl @@ -42,7 +42,7 @@ end function close(cur::Cursor) cur.handle == C_NULL && return # Per `lmdb.h`, write-txn cursors are freed by the parent txn's - # commit/abort and `mdb_cursor_close` afterwards is undefined; for + # commit/abort, and `mdb_cursor_close` afterwards is undefined. For # read-txn cursors, the txn handle is required to still be valid. # If the parent txn is already finalised in the wrapper, drop the # handle without calling into LMDB. @@ -357,10 +357,10 @@ end delete!(cur::Cursor; flags=0) Delete the entry the cursor is currently positioned at. Throws -`LMDBError` if the cursor is not on a live entry (LMDB returns `EINVAL`, -not `MDB_NOTFOUND`, so the Bool/idempotent shape used by -`delete!(txn, dbi, key)` doesn't apply here; position the cursor first -with `seek!`/`next!` if you need to recover from a missing entry). +`LMDBError` if the cursor is not on a live entry. LMDB returns `EINVAL` +rather than `MDB_NOTFOUND`, so the Bool/idempotent shape used by +`delete!(txn, dbi, key)` doesn't apply here. Position the cursor first +with `seek!` or `next!` if you need to recover from a missing entry. After a successful delete, LMDB advances the cursor to the next entry. """ diff --git a/src/dbi.jl b/src/dbi.jl index 52cb390..0e674ff 100644 --- a/src/dbi.jl +++ b/src/dbi.jl @@ -77,13 +77,13 @@ value's bytes can be produced straight into a destination buffer (for example, an `unsafe_store!` of a header followed by `copyto!` of a payload). Equivalent to heed's `Database::put_reserved`. -`buf` is an `unsafe_wrap` over the LMDB-allocated page; it is *only +`buf` is an `unsafe_wrap` over the LMDB-allocated page. It is *only valid inside `f`* (and only inside the enclosing write txn). The buffer's length is exactly `size`. Don't escape `buf` past `f`'s return; copy what you want to keep. -Cannot be combined with `MDB_DUPSORT`/`MDB_DUPFIXED` databases (LMDB -forbids `MDB_RESERVE` there). +Cannot be combined with `MDB_DUPSORT` or `MDB_DUPFIXED` databases, +since LMDB forbids `MDB_RESERVE` there. """ function put_reserved!(f, txn::Transaction, dbi::DBI, key, size::Integer; flags::Integer = zero(Cuint)) @@ -102,7 +102,7 @@ Delete `key` (or, in `MDB_DUPSORT`, the specific `(key, val)` pair) from the database. Returns `true` if an entry was removed, `false` if the key was not present. Other LMDB errors propagate as `LMDBError`. -The Bool-return / no-throw-on-miss shape matches `Base.delete!`'s "if +The Bool-return, no-throw-on-miss shape matches `Base.delete!`'s "if any" contract and the LMDB-binding convention shared by heed, py-lmdb, lmdb-js, and lmdbxx. """ diff --git a/src/dicts.jl b/src/dicts.jl index d571f2e..431130b 100644 --- a/src/dicts.jl +++ b/src/dicts.jl @@ -3,11 +3,11 @@ A persistent `AbstractDict{K,V}` backed by a single LMDB environment plus its default DBI. Keys and values are encoded as raw bytes; -`String`, `Vector{T}` (where `T` is bitstype), or any bitstype scalar +`String`, `Vector{T}` (where `T` is bitstype), and any bitstype scalar all work. For prefix-scoped scans (e.g. hierarchical "directory" key schemes), -see `LMDB.scan` / `LMDB.scan_keys` / `LMDB.scan_values` / `LMDB.list_dirs`. +see `LMDB.scan`, `LMDB.scan_keys`, `LMDB.scan_values`, and `LMDB.list_dirs`. """ mutable struct LMDBDict{K,V} <: AbstractDict{K,V} env::LMDB.Environment @@ -205,7 +205,7 @@ function Base.pop!(d::LMDBDict{K,V}, k, default) where {K,V} end end -# `pop!(d)` without a key — pops the first entry, mirroring `Base.pop!(::Dict)`. +# `pop!(d)` without a key: pops the first entry, mirroring `Base.pop!(::Dict)`. function Base.pop!(d::LMDBDict{K,V}) where {K,V} txn_dbi_do(d) do txn, dbi LMDB.open(txn, dbi) do cur diff --git a/src/env.jl b/src/env.jl index dc1a185..f0df12c 100644 --- a/src/env.jl +++ b/src/env.jl @@ -2,7 +2,7 @@ A DB environment supports multiple databases, all residing in the same shared-memory map. Wrapping a raw `Ptr{MDB_env}` in `Environment(h)` takes ownership of the -handle: it will be closed when the wrapper is garbage-collected, unless +handle. The handle is closed when the wrapper is garbage-collected, unless `close` was already called explicitly. Closing is idempotent. """ mutable struct Environment @@ -76,8 +76,8 @@ end Environment(path::AbstractString; mapsize=nothing, maxreaders=nothing, maxdbs=nothing, flags=0, mode=0o755) -> Environment -One-call equivalent of `create()` + (optional) `setindex!` for `MapSize` / -`Readers` / `DBs` + `open(env, path)`. Mirrors py-lmdb's +One-call equivalent of `create()`, optional `setindex!` for `MapSize`, +`Readers`, or `DBs`, and `open(env, path)`. Mirrors py-lmdb's `Environment(path, **kwargs)` and lmdb-rs's `EnvironmentBuilder.open(path)`. If anything fails between `create` and a successful `open`, the partially @@ -104,7 +104,7 @@ end """Close the environment and release the memory map. Idempotent: calling `close` on an already-closed `Environment` is a -silent no-op, matching the convention of `close(::IO)`. This makes +silent no-op, matching the convention of `close(::IO)`. That makes finalizers safe to run after an explicit close. """ function close(env::Environment) @@ -293,7 +293,7 @@ function reader_check(env::Environment) return Int(dead[]) end -# Callback for `mdb_reader_list` — appends the message to the IOBuffer +# Callback for `mdb_reader_list`: appends the message to the IOBuffer # referenced through `ctx`. Returns 0 to continue, non-zero to stop. function _reader_list_cb(msg::Ptr{Cchar}, ctx::Ptr{Cvoid})::Cint io = unsafe_pointer_to_objref(ctx)::IOBuffer diff --git a/src/txn.jl b/src/txn.jl index b5dd51c..d7f796d 100644 --- a/src/txn.jl +++ b/src/txn.jl @@ -4,8 +4,8 @@ Transactions may be read-only or read-write. A `Transaction` keeps a reference to its parent `Environment`, both to expose it via `env(txn)` and to ensure the env outlives the txn under -GC. If a transaction is dropped without an explicit `commit` or `abort`, -its finalizer aborts it. +GC. A transaction dropped without an explicit `commit` or `abort` is +aborted by its finalizer. """ mutable struct Transaction handle::Ptr{MDB_txn} @@ -53,8 +53,8 @@ end """Abandon all the operations of the transaction instead of saving them. -The transaction and its cursors must not be used after, because its handle is freed. -Idempotent: safe to call after a previous `commit`/`abort` or on a never-opened txn. +The transaction and its cursors must not be used afterward, because the handle is freed. +Idempotent: safe to call after a previous `commit` or `abort`, or on a never-opened txn. """ function abort(txn::Transaction) txn.handle == C_NULL && return @@ -68,7 +68,7 @@ end """Commit all the operations of a transaction into the database. -The transaction and its cursors must not be used after, because its handle is freed. +The transaction and its cursors must not be used afterward, because the handle is freed. Idempotent. """ function commit(txn::Transaction)