up

Author: selimslab

_CONTENT/eng/data/acid.md

@@ -1,2 +1,97 @@
 ---
 ---
+## schema
+```
+encoding
+    text 
+    binary 
+
+schema evolution
+    avro 
+    protobuf 
+
+    keep unknown fields
+    tags vs names: compact + rename later
+
+    breaking
+        deleting required fields
+        changing field types
+
+    old code -> new data 
+    old data <- new code 
+```
+
+## integrity
+```
+acid: TX, FK, CHECK, WAL, .. 
+
+isolation problems
+    reads
+        dirty
+        non-repeatable
+        phantom
+        read-skew
+
+    writes 
+        lost 
+        decision skew
+
+solutions
+    1. read committed
+    2. snapshot/repeatable read: MVCC, must for analytics, backups 
+    3. serializable MVCC + SSI
+
+SSI: serializable snapshot isolation
+    predicate locks + dep. cycles
+```
+
+
+## proc
+```
+batch
+    atomic ops on seq. data 
+    delta lake: parquet + transaction log + metadata
+
+stream
+    immutable events
+
+    time 
+        event 
+        delivery 
+        processing  
+
+    flow control
+        backpressure
+        circuit breaker
+
+    consumer lag
+        checkpoint
+        watermark
+        grace period
+        publish correction
+
+    windows: fixed, overlapping, sliding, session
+
+    log compaction
+    joins 
+    probabilistic dsa
+```
+
+## comm 
+```
+ipc
+    db, services, messages 
+
+    push: pubsub, ws, sse, webhook
+    pull: query, poll
+    q: decouple, buffer 
+
+    MPI: message passing interface
+        no central coordinator
+        nodes communicate directly
+
+delivery semantics 
+    exactly once
+        producer retry + consumer dedup 
+        producer outbox + consumer ack in tx 
+```

_CONTENT/eng/data/data.md

@@ -49,37 +49,47 @@ rebalancing is expensive
     service discovery, request routing 
 ```
 
-## consistency
-linearizable: single copy illusion, single leader + election consensus + sync replication
-
-causal: vector clocks + dependency tracking
-
-eventual: async replication + conflict resolution
-
-## consensus
-raft: majority ack, term number fencing 
-
-## atomic commit
-2PC: ask all, commit if they all ack, like marriage, coordinator spof 
-practical: 2pc + raft for coordinator failover 
 
 ## time and order 
-NTP, GPS 
+```
+lamport clock
+    single counter per process
+    can only tell if A happens-before B
+
+vector clocks
+    list of counters per process
+    can detect concurrency, detects conflicts 
+
+versions
+    each replica tracks versions of replicated data objects 
+
+availability
+    heartbeat pings with timeout
+    adapt to network conditions
+    lease with ttl
+    gossip
+```
 
-lamport clock: single counter per process, can only tell if A happens-before B
 
-vector clocks: list of counters per process, can detect concurrency, detects conflicts 
+## consistency
+```
+linearizable
+    single copy illusion
+    single leader + election consensus + (sync replication or raft quorum)
 
-version vector: each replica tracks versions of replicated data objects 
+causal: vector clocks + dependency tracking
 
+eventual: async replication + conflict resolution
 ```
-lamport int 
-vector []int 
-versions map[object]version
+
+## consensus
 ```
+raft: majority ack, term number fencing 
 
-## availability
-heartbeat pings with timeout, adapt to network conditions
-lease with ttl
-gossip
+atomic commit
+    2PC
+        ask all, commit if they all ack
+        like marriage, coordinator spof 
 
+    practical: 2pc + raft for coordinator failover 
+```

_CONTENT/eng/data/dataflow.md

@@ -0,0 +1,87 @@
+---
+---
+## tradeoffs 
+```
+Bias-variance: simple vs complex
+accurate vs interpretable
+complex vs general: simpler models generalize better 
+```
+
+## precision vs recall
+precise has few false positives but misses some true ones 
+
+high recall: more true positives but also more false ones 
+
+F1-score: harmonic mean of both 
+
+eg. spam detection - balance of catching more spam without marking good emails as spam 
+
+
+## diffusion
+```
+training
+    start with an image
+    add random noise
+    make the model predict how much noise is added and where
+
+generating
+    take a text prompt
+    turn it into a text embedding
+    start with random noise
+    gradually remove noise in the direction of text embedding tensor
+```
+
+## RL
+An **agent** takes **actions** in an **environment** to maximize **reward**
+
+Agent can use 
+    its internal **state**
+    its decision **policy**
+    its **model** of environment etc. 
+
+<https://lilianweng.github.io/posts/2018-02-19-rl-overview/>
+
+
+## tf 
+
+<https://jalammar.github.io/illustrated-transformer/>
+
+## deep 
+
+```
+A network is layers of weights
+[nums] -> network -> num
+
+For each training pass:
+    Multiply input tensor with weights and add bias, layer by layer
+    Compare prediction to correct answer using a loss function
+    Find gradients using backprop
+    Minimize loss via gradient descent
+
+For a 3-layer network
+
+1. Forward pass
+
+    Input → Layer1 → Layer2 → Layer3 → Output → Loss
+
+    x = input data
+    l1 = activation_func(W1.x + b1)
+    l2 = activation_func(W2.x + b2)
+    l3 = ..
+    output = W3.l3 + b4
+
+    loss = loss_func(output, correct_answer)
+
+2. Backprop
+
+    Update weights using chain rule
+
+    Loss → Layer3 → Layer2 → Layer1 → Input
+
+    d_loss/d3
+    d_loss/d2 = d_loss/d3 . d2/d3
+    d_loss/d1 = d_loss/d2 . d2/d1
+    d_loss/input = d_loss/d1 . d1/input
+
+    W = W - learning_rate.d_loss/dW
+```