Skip to content

Improve LMDBDict - #54

Merged
maleadt merged 4 commits into
mainfrom
tb/dict
May 21, 2026
Merged

Improve LMDBDict#54
maleadt merged 4 commits into
mainfrom
tb/dict

Conversation

@maleadt

@maleadt maleadt commented May 21, 2026

Copy link
Copy Markdown
Collaborator
  • Rebuild on top of the new high-level API (Add a high-level API #53)
  • Make more idiomatic (<:AbstractDict, providing the expected behavior, etc)
  • Performance improvements

Iterator behavior inspired by LLVM.jl.

maleadt and others added 4 commits May 21, 2026 22:20
dicts.jl no longer reaches into liblmdb directly: keys/values/collect/
list_dirs/valuesize switch from LMDBIterator to a private _walk_prefix
helper built on `walk` + `seek_range!`/`next!`, and haskey/get/get! use
`tryget` and `get(...,default)` instead of `unchecked_mdb_get`.

`get!`-with-default now writes inside the same write txn that observed
the missing key, instead of opening a separate write txn after a read
txn — a small atomicity improvement; the previous form raced against
concurrent writers.

Also: `walk` now stops if its callback returns `false` (any other
return — including `nothing` — continues), which is what makes the
prefix-scan loop in `_walk_prefix` work without dropping back to
tier-1 cursor ops.

`grep '\bmdb_' src/dicts.jl` is empty after this commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
LMDBDict now subtypes AbstractDict{K,V}, so merge!/filter!/pairs/==/
hash/lazy keys/lazy values come from the standard library. The
hand-rolled keys/values/collect overloads with a `prefix` kwarg are
gone; `keys(d)` and `values(d)` are the lazy `KeySet`/`ValueIterator`
the AbstractDict contract promises.

Iteration follows LLVM.jl's `BasicBlockInstructionSet` pattern: the
state is `(txn, cur)`, opened on the first `iterate` call (analogous
to LLVM's `LLVMGetFirstInstruction` seed) and advanced with
`MDB_NEXT` each step. Normal completion commits the txn and closes
the cursor; early `break`/throw is reclaimed by the Step-9 finalizers.

Other AbstractDict-shaped changes:

  - `getindex(d, k)` throws `KeyError(k)` on miss (was `LMDBError`).
  - `pop!(d, k)` throws `KeyError`; `pop!(d, k, default)` returns the
    default; both are atomic in one write txn.
  - `length(d)` via `mdb_stat`; `isempty`, `empty!` follow.
  - `delete!(d, k)` no longer throws on missing keys (Base contract).

Prefix-scoped scans move to LMDB-namespaced helpers — `LMDB.scan(d)`,
`LMDB.scan_keys(d)`, `LMDB.scan_values(d)` — alongside `list_dirs`
and `valuesize`.

Defaults:

  - `MDB_NOTLS` is now set on the LMDBDict env, matching py-lmdb. Without
    it, a `length(d)` call mid-iteration tripped MDB_BAD_RSLOT.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…sor.

cur.jl loses ~100 lines of legacy iterator scaffolding that nothing in
the new API exercises:

  - LMDBIterator{R} + ReturnKeys/ReturnValues/ReturnBoth/ReturnValueSize
  - DirectoryLister + process_returns + init_values + arcopy
  - keys(cur, T; prefix) / Base.values(cur, T; prefix) / a broken
    iterate(cur, K, V) that referenced an undefined `prefix` variable
  - get(cur, key, T, op) — superseded by seek!+value at the right op

`mbd_unpack` was always a typo for `mdb_unpack`; renamed in src and tests.

Cursor now stores its parent DBI alongside its parent Transaction;
database(cur) returns it directly instead of round-tripping through
`mdb_cursor_dbi` and synthesizing a fresh wrapper with an empty name.

Adds the AbstractDict gap that Base.Dict supplies but
Base.AbstractDict does not — `pop!(d::LMDBDict)` (no key form) pops
the lexicographically-first entry, throws ArgumentError if empty.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
AbstractDict's default merge! / mergewith! / filter! call d[k]=v or
delete!(d, k) in a loop, and each one of those opens its own LMDB write
txn (with its own commit/fsync). For a 1k-entry merge! that's 1k
transactions, which dominates wall time on durable storage. Override
the three to land in one txn instead.
@codecov

codecov Bot commented May 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.63314% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.40%. Comparing base (97af9ac) to head (fe773a4).

Files with missing lines Patch % Lines
src/dicts.jl 97.57% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #54      +/-   ##
==========================================
+ Coverage   80.59%   83.40%   +2.80%     
==========================================
  Files           9        9              
  Lines         701      717      +16     
==========================================
+ Hits          565      598      +33     
+ Misses        136      119      -17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@maleadt
maleadt merged commit 9bec48d into main May 21, 2026
10 checks passed
@maleadt
maleadt deleted the tb/dict branch May 21, 2026 20:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant