Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 61 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,67 @@ The same source lives at [`cmd/demo/main.go`](cmd/demo/main.go). Could clone thi

---

## Architecture

High-level layout: the **`DB`** type ties together **options**, an in-memory **keydir**, **segment files** (active + sealed), optional **hint** sidecars for recovery, and an **advisory lock**. **`entity`** defines the on-disk record layout shared by writes, reads, hints, and merge.

```mermaid
flowchart TB
subgraph Client["Client"]
App["Application / cmd/demo"]
end

subgraph API["Package root"]
DB["DB · Get / Set / Delete / Merge / recovery"]
Opt["Options"]
Lk["Advisory lock · lock_unix / lock_other"]
end

subgraph Idx["index/"]
KD["KeyDir · key → DataPosition"]
end

subgraph St["storage/"]
DF["DataFiles"]
Act["ActiveFile · append-only segment"]
Old["OldFile · sealed segments"]
Hnt["Hint files · read/write"]
end

subgraph Ent["entity/"]
Rec["Entry · Meta · CRC · tombstone"]
end

subgraph Disk["Data directory on disk"]
Dat["*.dat"]
Hin["*.hint"]
Lf[".tiny-bitcask.lock"]
end

App --> DB
DB --> Opt
DB --> KD
DB --> DF
DB --> Lk
Lk --> Lf

DF --> Act
DF --> Old
DF --> Hnt

Act --> Dat
Old --> Dat
Hnt --> Hin

Rec -.->|encode / decode / verify| Act
Rec -.->|encode / decode / verify| Old
Rec -.->|compact index rows| Hnt
```

**Typical paths (mental model).** **Write:** append an `Entry` to the active `.dat`, then update `KeyDir`. **Read:** `KeyDir` lookup → `ReadAt` on the segment identified by `fid`. **Open:** scan segments (or load hints) to rebuild `KeyDir`. **Merge:** scan sealed segments for live rows only, rewrite into the active file, drop old `.dat` / `.hint` pairs.

---

## What is implemented

- **Open / create**: `NewDB` — empty directory creates a new store; existing directory **recovers** the keydir by scanning `*.dat` files in order (or hints for sealed segments). **`Options`**: `VerifyCRC` (default on), `ReadOnly` (open existing store read-only), `ExclusiveLock` (Unix advisory `flock` on `.tiny-bitcask.lock`; shared lock when `ReadOnly`).
Expand Down Expand Up @@ -112,13 +173,3 @@ Some items from [bitcask-intro.pdf](https://riak.com/assets/bitcask-intro.pdf) a
| `options.go` | `Dir`, `SegmentSize`, `VerifyCRC`, `ReadOnly`, `ExclusiveLock` |

---

## Todo (paper-aligned and robustness)

- [x] **Hint files**: format, write on rotation, load hint instead of full scan on open when present.
- [x] **Recovery**: populate old-file ID list from on-disk segments so `Merge` works after restart; timestamps from full scan recovery.
- [x] **Merge**: compare keydir `(fid, offset)` to the **start** offset of each scanned entry; test `TestDB_Merge_LiveKeyOnlyInOldSegment`.
- [x] **Delete / tombstones**: valid meta + key for tombstone records; recovery applies tombstones; merge skips tombstone bodies.
- [x] **CRC verification** on read (`Options.VerifyCRC`, default `true`).
- [x] **`Sync` / `Close`**: flush and fsync active file; close FDs and lock.
- [x] **API**: `ListKeys`, `Fold`, read-only open (`Options.ReadOnly`), Unix advisory lock (`Options.ExclusiveLock`).
Loading