Merge pull request #18 from cockroachdb/chore/update-cockroachdb-skills

chore: update cockroachdb-skills submodule (1 new commits)

Author: rafiss

skills/application-development/benchmarking-transaction-patterns/SKILL.md

@@ -13,82 +13,43 @@ Guides users through benchmarking, explaining, and comparing two formulations of
 
 **Complement to design skills:** For general transaction design principles, see [designing-application-transactions](../designing-application-transactions/SKILL.md). For SQL syntax and query patterns, see [cockroachdb-sql](../../query-and-schema-design/cockroachdb-sql/SKILL.md).
 
-## When to Use This Skill
-
-- Comparing explicit multi-statement transactions versus CTE-based single-statement transactions
-- Benchmarking CockroachDB workloads under high concurrency or hot-key contention
-- Investigating retry pressure, p95/p99 latency, or throughput differences between transaction formulations
-- Deciding whether to rewrite a multi-step application flow into a single SQL statement
-- Setting up a fair side-by-side benchmark with proper reset discipline
-- Interpreting benchmark results (throughput, retries, tail latency, failures)
-- Explaining why SQL Activity still shows waiting even with CTE transactions
-
-## Prerequisites
-
-- CockroachDB test cluster (do not benchmark on production)
-- SQL client or JDBC driver for benchmark execution
-- Understanding of CockroachDB SERIALIZABLE isolation and retry behavior
-- Familiarity with basic concurrency testing concepts
-
 ## Core Concept
 
-When two implementations perform the same business behavior, the transaction formulation itself can be a primary performance lever under contention.
+Under contention, the transaction formulation itself is a primary performance lever. The **explicit model** (multi-statement `BEGIN`/`COMMIT`) keeps the transaction open across round trips, widening the contention window. The **CTE model** (single-statement) collapses the same logic into one atomic statement, reducing transaction duration and retries.
 
 ### Explicit Transaction Model
 
-The application orchestrates the workflow as separate SQL statements inside a transaction: read state, apply logic, write changes, commit.
-
 ```sql
 BEGIN;
-
 SELECT balance FROM accounts WHERE id = $1;
-
 -- Application decides whether transfer is allowed
-
 UPDATE accounts SET balance = balance - $2 WHERE id = $1;
 UPDATE accounts SET balance = balance + $2 WHERE id = $3;
-
 INSERT INTO transfers (from_acct, to_acct, amount, created_at)
 VALUES ($1, $3, $2, now());
-
 COMMIT;
 ```
 
-This keeps the transaction open across multiple statements and often includes application-side decision logic between steps.
-
 ### CTE Transaction Model
 
-The same read/decision/write logic is expressed as a single SQL statement, so the database evaluates and applies the business operation atomically without intermediate client orchestration.
+CockroachDB rejects multiple mutations of the same table in a single statement by default (`sql.multiple_modifications_of_table.enabled`), so the debit and credit are folded into one `UPDATE` using `CASE`.
 
 ```sql
-WITH debit AS (
+WITH funded AS (
+  SELECT 1 FROM accounts WHERE id = $1 AND balance >= $2
+), upd AS (
   UPDATE accounts
-  SET balance = balance - $2
-  WHERE id = $1
-    AND balance >= $2
-  RETURNING id
-), credit AS (
-  UPDATE accounts
-  SET balance = balance + $2
-  WHERE id = $3
-    AND EXISTS (SELECT 1 FROM debit)
+  SET balance = CASE WHEN id = $1 THEN balance - $2 ELSE balance + $2 END
+  WHERE id IN ($1, $3) AND EXISTS (SELECT 1 FROM funded)
   RETURNING id
 ), ins AS (
   INSERT INTO transfers (from_acct, to_acct, amount, created_at)
-  SELECT $1, $3, $2, now()
-  WHERE EXISTS (SELECT 1 FROM debit)
-    AND EXISTS (SELECT 1 FROM credit)
+  SELECT $1, $3, $2, now() WHERE (SELECT count(*) FROM upd) = 2
   RETURNING id
 )
 SELECT id FROM ins;
 ```
 
-### Why CTE Tends to Win Under Contention
-
-The explicit version keeps the transaction open across multiple statements, increasing the time window for write conflicts, timestamp pushes, and retries. Under high concurrency, each retry repeats the read and write work and continues contending for the same hot data.
-
-The CTE version collapses the same business logic into a single atomic statement, reducing transaction duration and sharply narrowing the contention window.
-
 ## Steps
 
 ### 1. Prepare the Benchmark Environment
@@ -125,7 +86,25 @@ ON CONFLICT (id) DO UPDATE SET balance = 1000.00;
 
 ### 3. Run the Explicit Transaction Benchmark
 
-Execute with realistic concurrency (e.g., 64-128 workers) and a fixed duration or iteration count. Record throughput, retries, p50/p95/p99 latency, max latency, and failures.
+Execute with realistic concurrency. Example using `pgbench` (PostgreSQL-compatible):
+
+```bash
+# Create a pgbench script file: explicit_transfer.sql
+# \set from_id random(1, 10000)
+# \set to_id random(1, 10000)
+# \set amount 10.00
+# BEGIN;
+# SELECT balance FROM accounts WHERE id = :from_id;
+# UPDATE accounts SET balance = balance - :amount WHERE id = :from_id;
+# UPDATE accounts SET balance = balance + :amount WHERE id = :to_id;
+# INSERT INTO transfers (from_acct, to_acct, amount, created_at) VALUES (:from_id, :to_id, :amount, now());
+# COMMIT;
+
+pgbench -n -c 64 -j 8 -T 120 -f explicit_transfer.sql \
+  "postgresql://root@localhost:26257/bankbench?sslmode=disable"
+```
+
+Record throughput (tps), retries, p50/p95/p99 latency, max latency, and failures.
 
 ### 4. Reset Between Runs for Fair Comparison
 
@@ -137,7 +116,13 @@ UPDATE accounts SET balance = 1000.00;
 
 ### 5. Run the CTE Transaction Benchmark
 
-Execute with the same concurrency, duration, and parameters as the explicit run.
+Execute with the same concurrency, duration, and parameters as the explicit run:
+
+```bash
+# Create a pgbench script file: cte_transfer.sql containing the CTE query above
+pgbench -n -c 64 -j 8 -T 120 -f cte_transfer.sql \
+  "postgresql://root@localhost:26257/bankbench?sslmode=disable"
+```
 
 ### 6. Compare Results
 
@@ -153,6 +138,20 @@ Always compare these metrics side by side:
 | Max latency        | Outlier behavior                                                 |
 | Failures           | Non-retryable errors                                             |
 
+### 7. Validate Benchmark Integrity
+
+Before interpreting results, verify the benchmark ran cleanly:
+
+```sql
+-- Confirm expected transfer volume
+SELECT COUNT(*) AS total_transfers FROM transfers;
+```
+
+```bash
+# Check node liveness and start times (no node restarts mid-benchmark)
+cockroach node status --certs-dir=<certs-dir>     # or --insecure for an insecure cluster
+```
+
 ## Benchmark Reference Results
 
 In a reported high-contention run comparing the two models:
@@ -202,11 +201,9 @@ Extended runs preserved the same directional result at higher total volume, with
 
 ## Common Misconceptions
 
-**"CTE always wins in every workload"** — No. The claim is narrower: when the same business workflow can be expressed as a single atomic statement and the workload is contention-sensitive, collapsing the transaction shape can materially improve performance and stability.
-
-**"SQL Activity showing waiting means CTE failed"** — Single-statement CTE execution does not eliminate contention. Statements can still wait on row conflicts, write intents, latches, or scheduling. The right comparison is overall throughput, tail latency, and retry profile.
-
-**"Single-statement means no contention"** — A CTE can still wait under contention. The benefit is a narrower contention window, not the elimination of contention.
+- **"CTE always wins"** — Only when the workflow is contention-sensitive and expressible as a single atomic statement.
+- **"SQL Activity showing waiting means CTE failed"** — CTE reduces but does not eliminate contention. Compare throughput, tail latency, and retry profile.
+- **"Single-statement means no contention"** — CTE narrows the contention window but does not eliminate it.
 
 ## Safety Considerations
 

skills/observability-and-diagnostics/auditing-table-statistics/SKILL.md

@@ -13,92 +13,22 @@ Audits optimizer table statistics for staleness, missing column coverage, and ro
 
 **Complement to profiling-statement-fingerprints:** This skill diagnoses optimizer statistics issues; for identifying historically slow queries, see [profiling-statement-fingerprints](../profiling-statement-fingerprints/SKILL.md).
 
-## When to Use This Skill
-
-- Query performance degrades after bulk INSERT, UPDATE, or DELETE operations
-- EXPLAIN plans show unexpected full table scans or suboptimal join orders
-- Plan instability: same query produces different execution plans over time
-- After schema changes: ADD COLUMN, DROP COLUMN, or CREATE INDEX operations
-- Tables experience >20-30% row count changes without statistics refresh
-- SQL-only diagnostics needed without DB Console access
-- Validating automatic statistics collection is working correctly
-
-**For historical query analysis:** Use [profiling-statement-fingerprints](../profiling-statement-fingerprints/SKILL.md) to identify slow statement patterns.
-**For live query triage:** Use [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md) for immediate incident response.
-
 ## Prerequisites
 
-- SQL connection to CockroachDB cluster
-- **Privilege requirement:** Any privilege on target tables (SELECT, INSERT, UPDATE, DELETE, or admin role)
-  - Much less restrictive than VIEWACTIVITY: any table access grants statistics visibility for that table
+- SQL connection with any privilege on target tables
 - Automatic statistics collection enabled (default): `sql.stats.automatic_collection.enabled = true`
 
-**Check automatic collection status:**
-```sql
-SHOW CLUSTER SETTING sql.stats.automatic_collection.enabled;  -- Should return: true
-```
-
-**Verify table access:**
-```sql
-SHOW GRANTS ON TABLE database_name.table_name;
-```
+**Related skills:** [profiling-statement-fingerprints](../profiling-statement-fingerprints/SKILL.md) for historical query analysis, [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md) for live triage.
 
 ## Core Concepts
 
-### What Are Table Statistics?
-
-**Table statistics** provide the optimizer with data distribution information to estimate query costs:
-
-| Statistic Type | Description | Impact on Optimizer |
-|----------------|-------------|---------------------|
-| **row_count** | Total rows in table | Cardinality estimates for full scans |
-| **distinct_count** | Unique values per column | Selectivity estimates for WHERE/JOIN predicates |
-| **null_count** | NULL values per column | IS NULL / IS NOT NULL predicate costs |
-| **histogram** | Value distribution buckets | Range scan selectivity (e.g., `WHERE age BETWEEN 20 AND 30`) |
-
-**Multi-column statistics** capture correlation between columns (e.g., city + state + zip) for more accurate multi-predicate estimates.
-
-### Statistics Lifecycle
-
-**Automatic collection (default):**
-- Triggered when row count changes by ~20% since last statistics collection
-- Runs as background job (non-blocking, but consumes resources)
-- May be delayed for very large tables (>10M rows)
-
-**Manual collection:**
-- Explicit `CREATE STATISTICS` command for immediate refresh
-- Required for multi-column statistics (automatic collection only creates single-column stats)
-- Recommended after bulk data loads or significant schema changes
-
-### Staleness Indicators
+**CockroachDB-specific defaults:**
+- Automatic collection triggers at ~20% row count change (`sql.stats.automatic_collection.fraction_stale_rows`)
+- Auto-collection covers index column groups (when `sql.stats.multi_column_collection.enabled = true`, the default); ad-hoc multi-column stats on non-indexed columns require manual `CREATE STATISTICS`
+- Large tables (>10M rows) may have delayed auto-collection
+- Staleness thresholds: refresh if >7 days (OLTP) or >30 days (OLAP), or >20-30% row count drift
 
-| Indicator | Definition | Recommended Action |
-|-----------|------------|-------------------|
-| **Age** | Time since last statistics collection | Refresh if >7 days (OLTP) or >30 days (OLAP) |
-| **Row count drift** | Percent difference between current and cached row_count | Refresh if >20-30% drift detected |
-| **Missing columns** | Columns without statistics | CREATE STATISTICS for frequently queried columns |
-| **Missing histograms** | Columns without distribution data | Automatic collection handles; may need manual refresh |
-
-See [references/statistics-thresholds.md](references/statistics-thresholds.md) for workload-specific threshold guidance.
-
-### When Statistics Are Auto-Collected
-
-**Default trigger:** ~20% row count change (controlled by `sql.stats.automatic_collection.fraction_stale_rows`)
-
-**Collection schedule:**
-- Small tables (<10K rows): Immediate
-- Medium tables (10K-10M rows): Within minutes to hours
-- Large tables (>10M rows): May be delayed hours to avoid resource contention
-
-**Check pending jobs:**
-```sql
-SELECT job_id, description, status, fraction_completed
-FROM [SHOW JOBS]
-WHERE job_type = 'AUTO CREATE STATS'
-  AND status IN ('pending', 'running')
-ORDER BY created DESC
-LIMIT 20;
-```
+See [references/statistics-thresholds.md](references/statistics-thresholds.md) for workload-specific guidance.
 
 ## Core Diagnostic Queries
 
@@ -326,9 +256,9 @@ ORDER BY created DESC;
 - `column_count`: Number of columns in composite statistic
 
 **Interpretation:**
-- **Present**: Manual multi-column statistics exist (automatic collection only creates single-column)
-- **Absent**: May need manual creation for correlated columns (e.g., city + state + zip)
-- Common use case: Composite index columns that are queried together
+- **Present**: Multi-column statistics exist — either auto-collected for an index column group or manually created
+- **Absent**: No multi-column stats yet; may need manual creation for correlated non-indexed columns (e.g., city + state + zip)
+- Common use case: Manual stats on correlated columns that aren't covered by an index
 
 See [references/create-statistics-examples.md](references/create-statistics-examples.md) for multi-column creation patterns.
 
@@ -577,110 +507,13 @@ FROM ...;
 
 ## Key Considerations
 
-### Automatic vs Manual Collection
-
-**Automatic (default):**
-- **Pros:** Zero maintenance, adapts to data changes, covers all single columns
-- **Cons:** May lag for very large tables, no multi-column statistics
-- **Recommendation:** Keep enabled for baseline coverage
-
-**Manual:**
-- **Pros:** Immediate refresh, supports multi-column statistics, controlled timing
-- **Cons:** Requires monitoring and operational overhead
-- **Recommendation:** Use for critical tables, correlated columns, post-bulk-load scenarios
-
-### Statistics Retention
+- **Auto vs manual:** Keep automatic collection enabled for baseline; use manual `CREATE STATISTICS` for ad-hoc post-bulk-load refresh and critical tables
+- **Multi-column statistics:** Auto-collection covers index column groups; manual `CREATE STATISTICS` is needed for correlated non-indexed columns queried together (e.g., `CREATE STATISTICS city_state_stats ON city, state FROM addresses;`)
+- **Large tables (>10M rows):** Schedule `CREATE STATISTICS` during maintenance windows; monitor with `SHOW JOBS WHERE job_type = 'CREATE STATS'`
+- **Staleness tuning:** OLTP: 3-7 days, OLAP: 14-30 days, hybrid: critical tables 3-7 days, archive 30+ days
+- **Privilege:** Any table privilege (SELECT, INSERT, etc.) grants statistics visibility
 
-**Default retention:** Controlled by `sql.stats.persisted_rows.max` (default ~10 million rows across cluster)
-- Older statistics are pruned automatically
-- Typically 7-30 days of history retained depending on statistics volume
-
-**Historical analysis:**
-```sql
--- View statistics history for a table
-SELECT column_names, row_count, created
-FROM [SHOW STATISTICS FOR TABLE table_name]
-WHERE column_names = '{}'
-ORDER BY created DESC
-LIMIT 10;  -- Last 10 collections
-```
-
-### Histogram Limitations
-
-**What histograms optimize:**
-- Range queries: `WHERE age BETWEEN 20 AND 30`
-- Inequality predicates: `WHERE price > 100`
-- ORDER BY selectivity: `ORDER BY created_at LIMIT 10`
-
-**What histograms don't optimize:**
-- Exact equality: `WHERE id = 123` (uses distinct_count instead)
-- Multi-column predicates: Requires multi-column statistics
-- Very sparse columns: Histograms may be ineffective for highly skewed distributions
-
-### Multi-Column Statistics
-
-**When to create:**
-- Columns frequently queried together in WHERE clauses
-- Composite index columns with correlated values
-- Geographic columns (city + state + zip)
-- Time-based partitioning columns (year + month)
-
-**Creation:**
-```sql
--- Example: Correlated columns
-CREATE STATISTICS city_state_stats ON city, state FROM addresses;
-```
-
-**Limitation:** Only manual creation (automatic collection does NOT create multi-column statistics)
-
-See [references/create-statistics-examples.md](references/create-statistics-examples.md) for comprehensive patterns.
-
-### Performance Impact Mitigation
-
-**Large table strategies:**
-- Schedule during maintenance windows (nights/weekends)
-- Use database-specific batching (one database at a time)
-- Monitor cluster metrics: CPU, disk I/O, network saturation
-- Consider `AS OF SYSTEM TIME` for historical analysis without impacting live traffic
-
-**Resource monitoring during collection:**
-```sql
--- Check running statistics jobs
-SELECT job_id, description, status, fraction_completed, running_status
-FROM [SHOW JOBS]
-WHERE job_type IN ('CREATE STATS', 'AUTO CREATE STATS')
-  AND status = 'running';
-```
-
-### Staleness Threshold Tuning
-
-**OLTP workloads:**
-- **Recommended refresh:** 3-7 days
-- **Rationale:** Frequent updates, query patterns change rapidly
-
-**OLAP/Analytics workloads:**
-- **Recommended refresh:** 14-30 days
-- **Rationale:** Batch-oriented, stable query patterns
-
-**Hybrid workloads:**
-- Critical tables: 3-7 days
-- Archive/historical tables: 30+ days
-
-See [references/statistics-thresholds.md](references/statistics-thresholds.md) for detailed guidance.
-
-### Privilege Requirements
-
-**Required:** Any privilege on table (SELECT, INSERT, UPDATE, DELETE, or admin role)
-
-**Comparison to other diagnostics:**
-- **Less restrictive than:** VIEWACTIVITY (required for statement_statistics)
-- **More restrictive than:** Public cluster settings
-
-**Grant example:**
-```sql
--- Grant SELECT (least privileged) for statistics visibility
-GRANT SELECT ON TABLE database_name.table_name TO diagnostics_user;
-```
+See [references/create-statistics-examples.md](references/create-statistics-examples.md) and [references/statistics-thresholds.md](references/statistics-thresholds.md) for detailed guidance.
 
 ## References