up

Author: selimslab

_CONTENT/eng/data/acid.md

@@ -1,59 +1,21 @@
 ---
 ---
-atomicity: tx 
-consistency: invariants, FK, constraints 
-isolation: concurrent txs do not interfere
-durability: WAL, replicas, fsync, power hardware
-
-## tx 
-group ops. to atomic units
-no partial failure
-consistency
-fk integrity
-data sync
-
-## isolation problems
-lost updates
+```
+acid: tx, FK, CHECK, .. 
 
+isolation problems
 1. dirty-read: see uncommitted 
 2. non-repeatable read: same query, different result
 3. phantom read: new rows appear or disappear
 4. lost updates: conc. writes lose data
 5. read-skew: existing rows change between reads
 6. write skew: two tx reads same data to decide writes
-may violate invariants, eg. no doctors remaining
-needs serial isolation
-
-# solutions
-
-## read committed
-only see committed rows
-read-skew possible
-
-## repeatable read
-snapshot
-only see data committed before tx began
-
-solves read-skew 
 
-impl. by MVCC
-multi-version conc. control
-each tx gets a snapshot (a set of tx ids)
-
-analytics, backups
-
-## serializable
-prevent all races
-solves phantoms and write-skew 
-
-MVCC + SSI
-20% perf cost
+solutions
+1. read committed
+2. snapshot/repeatable read: MVCC, must for analytics, backups 
+3. serializable MVCC + SSI
 
 SSI: serializable snapshot isolation
-predicate locks
-dependency cycles
-
-## dist tx
-saga: commit or compensate
-
-or replicate entire transaction log through consensus
+predicate locks + dep. cycles
+```

_CONTENT/eng/data/dataflow.md

@@ -1,89 +1,53 @@
 ---
 ---
-## encoding
-backward comp: old data, new code
-
-breaking:
-deleting required fields
-changing field types
-
-keep unknown fields
-tags vs names: compact + rename later
-
-schema evolution
-avro 
-protobuf 
-
-## ipc
-db
-api
-msg passing
-
-push: pubsub, ws, sse, webhook
-pull: query, poll
-q: decouple, buffer 
-
-MPI
-message passing interface
-no central coordinator
-nodes communicate directly
-
-## delivery guarantees
-at-most-once 
-at-least-once: retries + idempotent receiver
-exactly-once: at-least once + dedup 
-
-producer: add uniq msg ids, retry, track sent msgs on outbox table 
-consumer: store seen ids, dedup, process+ack in tx 
-
-producer retry + consumer dedup 
-
-## batch 
-immutable inputs
-atomic ops
-
-batch 
-partition
-compose
-
-data locality
-sequential i/o
-vectorize
-columnar
-
-pre-compute expensive ops
-checkpoints
-
-declerative apis = better optimization
-
-spark df
-Delta lake: parquet + transaction log + metadata
-
-## stream 
-immutable events
-side effects
-
-log compaction
-
-event time
-delivery time
-e2e latency
-
-consumer lag
-checkpoint
-
-grace period
-watermark
-
-backpressure
-circuit breaker
-
-exactly once: idempotence + tx commits
-
-probabilistic dsa like bloomfilter, hyperloglog
-
-windows: fixed, overlapping, sliding, session
-
-stream + stream : window
-stream + table : enrich
-table + table : materialized view of join
+## comm. 
+```
+ipc
+    db
+    services, api, rpc, http 
+    msg passing, q
+
+    push: pubsub, ws, sse, webhook
+    pull: query, poll
+    q: decouple, buffer 
+
+MPI, message passing interface
+    no central coordinator
+    nodes communicate directly
+
+delivery semantics 
+    exactly once
+        producer retry + consumer dedup 
+        producer outbox + consumer ack in tx 
+```
+
+## dataproc
+```
+batch
+    atomic ops on seq. data 
+    delta lake: parquet + transaction log + metadata
+
+stream
+    immutable events
+
+    time 
+        event 
+        delivery 
+        processing  
+
+    flow control
+        backpressure
+        circuit breaker
+
+    consumer lag
+        checkpoint
+        watermark
+        grace period
+        publish correction
+
+    windows: fixed, overlapping, sliding, session
+
+    log compaction
+    joins 
+    probabilistic dsa
+```

_CONTENT/eng/data/dist.md

@@ -1,30 +1,72 @@
 ---
 ---
-https://muratbuffalo.blogspot.com/2023/10/hints-for-distributed-systems-design.html
+## replication
+```
+(part x, rep y)
+replicate WAL or rows 
 
-## consistency
-linearizable
-single copy illusion 
-single leader
-election consensus
-sync replication or quorum
+leader 
+    single 
+    multi leader cause write conflicts
+    none, dynamo, cassandra, quorums are still eventual 
+
+    failover: detect, elect, fence 
+
+lag
+    read your writes 
+    monotonic reads 
+    consistent prefix reads 
+
+    conflicts 
+        avoid: CRDT, OT, same-writer
+        resolve: read-repair, anti-entropy, app logic 
+
+other 
+    order events by version vectors 
+    different sort order per replica
+    topology 
+
+```
+
+## partitioning
+```
+types
+    key range
+    hash of key 
+    (hashkey, sortkey)
+
+hot spots: random prefix suffix
+
+indexes
+    local vs global 
+    scatter-gatter: tail-latency amp. 
+
+rebalancing is expensive 
+    fixed # of parts
+    dynamic, split large, merge small. good for key-range
+    hybrid 
 
-causal
-vector clocks + dependency tracking
+    service discovery, request routing 
+```
 
-eventual$$
-async replication
-conflict resolution (LWW, CRDTs, version vectors)
+
+## consistency
+linearizable: single copy illusion, single leader + election consensus + sync replication
+causal: vector clocks + dependency tracking
+eventual: async replication + conflict resolution
 
 ## consensus
-raft: majority ack, one leader per term
-split brain: lease + fencing token
+raft: majority ack, term number fencing 
+
 paxos
 pbft: o(n2)
 
+## atomic commit
+2PC: ask all, commit if they all ack, like marriage, coordinator spof 
+practical: 2pc + raft for coordinator failover 
+k
 ## time and order 
-NTP
-GPS 
+NTP, GPS 
 
 lamport clock: single counter per process, can only tell if A happens-before B
 vector clocks: list of counters per process, can detect concurrency, detects conflicts 
@@ -35,6 +77,3 @@ heartbeat pings with timeout, adapt to network conditions
 lease with ttl
 gossip
 
-## atomic commit
-2PC: ask all, commit if they all ack, like marriage, coordinator spof 
-practical: 2pc + raft for coordinator failover