pi

Author: selimslab

_CONTENT/eng/data/acid.md

@@ -1,9 +1,11 @@
 ---
 ---
+## acid
+
 atomicity
-[[consistency]]
+consistency
 [[isolation]]
-durability
+durability: WAL, replicas, fsync, power hardware
 
 tx
 group ops. to atomic units
@@ -13,7 +15,32 @@ fk integrity
 data sync
 
 
-WAL
-replicas
-fsync
-power hardware
+
+## data flow
+db
+api
+msg passing
+
+push: pubsub, ws, sse, webhook
+pull: query, poll
+
+stream: events
+batch: cron
+
+req-resp
+q
+
+MPI: message passing interface
+no central coordinator
+nodes communicate directly
+
+
+## encoding
+backward comp: old data, new code
+
+breaking:
+deleting required fields
+changing field types
+
+keep unknown fields
+tags vs names: compact + rename later

_CONTENT/eng/distdata/batch.md _CONTENT/eng/data/batch.md_CONTENT/eng/distdata/batch.md renamed to _CONTENT/eng/data/batch.md

@@ -44,32 +44,5 @@ better compression and disk life
 less stable response times in higher percentiles
 
 
-## columnar vs wide column 
-
-both lsm 
-
-⏺ The key differences are in data layout and access patterns, not just the underlying LSM storage:
-
-  Data Layout:
-  - Wide-column (Cassandra): Row-oriented within LSM - stores row_key → {col1: val1, col2: val2}
-  - Columnar (ClickHouse): Column-oriented within LSM - stores col1 → [val1, val2, val3...]
-
-  LSM Implementation Details:
-  - Cassandra: SSTables store rows, compaction merges by row key
-  - ClickHouse: MergeTree parts store columns separately, merges entire column chunks
-  - InfluxDB: TSM optimized for time-series compression patterns
-
-  Query Optimization:
-  - Wide-column: Efficient for SELECT * WHERE row_key = X
-  - Columnar: Efficient for SELECT col1, col2 WHERE condition (column pruning)
-
-  Compression:
-  - Wide-column: Compresses mixed data types within rows
-  - Columnar: Much better compression on homogeneous column data
-
-  Example:
-  Wide-column LSM: user123 → {name: "John", age: 30, city: "NYC"}
-  Columnar LSM:    name_col → ["John", "Jane", "Bob"]
-                   age_col  → [30, 25, 40]
-
-  Same LSM merge/compaction strategy, completely different data organization for different workloads.
+## columnar vs wide column
+both lsm

_CONTENT/eng/data/dataflow.md

@@ -0,0 +1,16 @@
+---
+---
+key range
+hash
+hybrid (hashkey, sortkey)
+
+choose partition keys well
+hot keys: random prefix/suffix
+
+## rebalancing
+fixed: many more parts upfront
+dynamic: split large, merge small. good for key-range
+hybrid
+
+secondary indexes: local or global
+keep related data together to prevent scatter/gather

_CONTENT/eng/data/encoding.md

@@ -4,26 +4,18 @@ wal: low-level, version dependent
 logical: version free. CDC
 statement: compact but non-det.
 
-single leader
-eg. psql streaming rep
-
-multi leader
-multi-datacenter
-offline apps
-docs
-
+single leader: eg. psql streaming
+multi leader: multi-datacenter, offline apps, docs
 leaderless
 
 ## conflicts
-avoid
-route to same leader
-crdts
+avoid by routing to same leader or crdts
+
+or resolve by read repair, anti-entropy, or app logic
 
-or resolve by
 read repair: compare replica responses
 background anti-entropy: detect using hashes of data parts and vector clocks
 last-write-wins
-app-logic
 
 ## concerns
 replication lag