a

Author: selimslab

_CONTENT/eng/data/acid.md

@@ -0,0 +1,59 @@
+---
+---
+atomicity: tx 
+consistency: invariants, FK, constraints 
+isolation: concurrent txs do not interfere
+durability: WAL, replicas, fsync, power hardware
+
+## tx 
+group ops. to atomic units
+no partial failure
+consistency
+fk integrity
+data sync
+
+## isolation problems
+lost updates
+
+1. dirty-read: see uncommitted 
+2. non-repeatable read: same query, different result
+3. phantom read: new rows appear or disappear
+4. lost updates: conc. writes lose data
+5. read-skew: existing rows change between reads
+6. write skew: two tx reads same data to decide writes
+may violate invariants, eg. no doctors remaining
+needs serial isolation
+
+# solutions
+
+## read committed
+only see committed rows
+read-skew possible
+
+## repeatable read
+snapshot
+only see data committed before tx began
+
+solves read-skew 
+
+impl. by MVCC
+multi-version conc. control
+each tx gets a snapshot (a set of tx ids)
+
+analytics, backups
+
+## serializable
+prevent all races
+solves phantoms and write-skew 
+
+MVCC + SSI
+20% perf cost
+
+SSI: serializable snapshot isolation
+predicate locks
+dependency cycles
+
+## dist tx
+saga: commit or compensate
+
+or replicate entire transaction log through consensus

_CONTENT/eng/data/dataflow.md

@@ -1,26 +1,44 @@
 ---
 ---
+## encoding
+backward comp: old data, new code
+
+breaking:
+deleting required fields
+changing field types
+
+keep unknown fields
+tags vs names: compact + rename later
+
+schema evolution
+avro 
+protobuf 
+
 ## ipc
 db
 api
 msg passing
 
 push: pubsub, ws, sse, webhook
 pull: query, poll
+q: decouple, buffer 
 
-stream: events
-batch: cron
-
-req-resp
-q
-
-MPI: message passing interface
+MPI
+message passing interface
 no central coordinator
 nodes communicate directly
 
+## delivery guarantees
+at-most-once 
+at-least-once: retries + idempotent receiver
+exactly-once: at-least once + dedup 
 
-## batch 
+producer: add uniq msg ids, retry, track sent msgs on outbox table 
+consumer: store seen ids, dedup, process+ack in tx 
+
+producer retry + consumer dedup 
 
+## batch 
 immutable inputs
 atomic ops
 
@@ -42,7 +60,6 @@ spark df
 Delta lake: parquet + transaction log + metadata
 
 ## stream 
-
 immutable events
 side effects
 
@@ -61,17 +78,12 @@ watermark
 backpressure
 circuit breaker
 
-exactly once
-idempotence + atomic commits
+exactly once: idempotence + tx commits
 
 probabilistic dsa like bloomfilter, hyperloglog
 
-windows
-fixed
-overlapping
-sliding
-session
+windows: fixed, overlapping, sliding, session
 
 stream + stream : window
 stream + table : enrich
-table + table : materialized view of join
+table + table : materialized view of join

_CONTENT/eng/data/dist.md

@@ -0,0 +1,38 @@
+---
+---
+## consistency
+linearizable
+single copy illusion 
+single leader
+election consensus
+sync replication or quorum
+
+causal
+vector clocks + dependency tracking
+
+eventual
+async replication
+conflict resolution (LWW, CRDTs, version vectors)
+
+## consensus
+raft: majority ack, one leader per term
+split brain: lease + fencing token
+paxos
+pbft: o(n2)
+
+## time and order 
+NTP
+GPS 
+
+lamport clock: single counter per process, can only tell if A happens-before B
+vector clocks: list of counters per process, can detect concurrency, detects conflicts 
+version vector: each replica tracks versions of replicated data objects 
+
+## availability
+heartbeat pings with timeout, adapt to network conditions
+lease with ttl
+gossip
+
+## atomic commit
+2PC: ask all, commit if they all ack, like marriage, coordinator spof 
+practical: 2pc + raft for coordinator failover 

_CONTENT/eng/data/isolation.md

@@ -0,0 +1,46 @@
+---
+---
+## replication 
+```
+wal: low-level, version dependent
+logical: version free. CDC
+statement: compact but non-det.
+
+leaderless
+single leader: eg. psql streaming
+multi leader: multi-datacenter, offline apps, docs
+
+replication lag
+    session models
+        read after write: route reads to where you wrote or use timestamps 
+        monotonic reads: read from same server, sticky session
+        consistent prefix reads: read in causal order
+
+    avoid conflicts 
+        same leader
+        crdt 
+
+    resolve conflicts
+        read repair: compare replica responses
+        background anti-entropy: detect using hashes of data parts and version vectors
+        app logic: eg. last-write-wins
+
+version vector: each replica tracks versions of replicated data objects 
+```
+
+## sharding 
+```
+key range
+hash
+hybrid (hashkey, sortkey)
+
+rebalancing
+    fixed: many more parts upfront
+    dynamic: split large, merge small. good for key-range
+    hybrid
+
+secondary indexes: local or global
+keep related data together to prevent scatter/gather
+hot keys: random prefix/suffix
+
+```

_CONTENT/eng/data/rep.md

@@ -1,15 +1,5 @@
 ---
 ---
-## encoding
-backward comp: old data, new code
-
-breaking:
-deleting required fields
-changing field types
-
-keep unknown fields
-tags vs names: compact + rename later
-
 ## log segments
 kafka
 
@@ -18,9 +8,15 @@ parquet
 clickhouse
 influx
 
+compression
+skip ranges
+only needed cols
+
 ## B+ trees
 all data in leaves
+100-500 children per node (fanout)
 leaves are linked for range queries
+write amplification from splitting/rebalancing
 
 one node = one disk page (4kb)
 
@@ -61,6 +57,9 @@ periodic merge
 local data, seq. io, flat files
 better compression and disk life
 
-10x write throughput vs b-tree, 0.5x read/s
 less stable response times in higher percentiles
 
+vs B-tree
+slower point queries  
+better sequential write throughput
+multiple versions of same key until compaction