obs,sql,server,ui: add statement store and V2 SQL Activity endpoints#167502
Draft
kyle-a-wong wants to merge 6 commits intocockroachdb:masterfrom
Draft
obs,sql,server,ui: add statement store and V2 SQL Activity endpoints#167502kyle-a-wong wants to merge 6 commits intocockroachdb:masterfrom
kyle-a-wong wants to merge 6 commits intocockroachdb:masterfrom
Conversation
Introduce `StatementStore`, a write-through cache that deduplicates and asynchronously persists statement fingerprints to `system.statements`. The store uses an in-memory FIFO cache (sized via the `obs.statements.store.cache_size` cluster setting, default 100k entries) to skip fingerprints that have already been seen. Cache misses are enqueued to a buffered channel and flushed in batches by a background goroutine. When the channel is full, items are silently dropped and retried after cache eviction. On startup, the cache is prewarmed with the most recent 1000 fingerprint IDs from `system.statements` to avoid redundant upserts after restarts. The store is disabled/enabled via the `obs.statements.store.enabled` cluster setting (default true). Epic: none Release note: None Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Add a `*statementstore.StatementStore` parameter to `NewSQLStatsIngester` so the ingester can persist statement fingerprints as they flow through the stats pipeline. The ingester calls `storeStatementFingerprint` at two points: - In `processStatement`, when a statement is first observed. - In `flushBuffer`, when the fingerprint ID is recomputed because the transaction's `ImplicitTxn` value changed (explicit vs implicit). When the store is nil (e.g. in unit tests that don't need persistence), the call is a no-op. Epic: none Release note: None Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Create a `StatementStore` in `NewServer`, configure its internal executor after the `InternalDB` is available, pass it to the `SQLStatsIngester`, and start its background writer alongside the other stats subsystems. The store's internal executor and memory monitor are set up via `SetInternalExecutor` after `NewServer` returns (breaking a dependency cycle), and `Start` is called during `Server.Start` alongside the ingester and persisted stats. Epic: none Release note: None Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Add cluster version `V26_3_PopulateSystemStatements` (26.2, Internal 4) and a tenant upgrade that backfills `system.statements` with distinct fingerprints from `system.statement_statistics`. The migration seeds the statement store so that fingerprints observed before the store existed are immediately available for the V2 SQL Activity endpoints without waiting for re-execution. The backfill uses `INSERT ... ON CONFLICT DO NOTHING` so it is idempotent and safe to re-run. The `RestoreActionNotRequired` annotation indicates that restoring a cluster predating this migration can leave the table empty; fingerprints will be populated on next execution. Epic: none Release note: None Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Add `GetStatements` and `GetTransactions` BFF endpoints under `/api/v2/dbconsole/` that return paginated, sorted, aggregated statement and transaction statistics. These endpoints query the denormalized covering indexes on `system.statement_statistics` and `system.transaction_statistics` directly with server-side pagination and sorting, avoiding the expensive full-table scans and client-side aggregation of the existing gRPC endpoints. Key features: - Server-side pagination (pageSize, pageNum) and sorting (sortBy, sortOrder) with parameterized queries to prevent SQL injection. - Time range filtering via Unix epoch seconds (start, end). - Optional filters: search (ILIKE on fingerprint text), appName (exact match), database (exact match), excludeInternal (default true). - Standard deviation computed from sum-of-squares columns. - Transaction endpoint includes a secondary query to fetch statement query summaries via the system.statements lookup table. - Datadriven EXPLAIN (SHAPE) tests verify that queries use the expected covering indexes. Epic: none Release note: None Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Add `StatementsPageV2` and `TransactionsPageV2` React components to the DB Console, accessible via the Advanced Debug page under "SQL Activity (V2)". These pages use the new V2 BFF endpoints with server-side pagination and sorting via SWR hooks. Features: - SWR-based data fetching with automatic cache key management. - Server-side pagination and sorting (no client-side aggregation). - Time scale dropdown with URL state persistence. - Search, app name, database, and internal statement filters. - Bar chart visualizations for latency, CPU, contention, and I/O metrics using the existing barChartFactory. - Transaction page shows constituent statement summaries via the secondary query in the transactions endpoint. The pages are currently only linked from the debug panel to allow iterative development before replacing the existing SQL Activity pages. Also includes the regenerated top-level pkg/BUILD.bazel for the new statementstore package. Epic: none Release note: None Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
Contributor
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
|
Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
Member
Member
⚪ Sysbench [SQL, 3node, oltp_read_write]
Reproducebenchdiff binaries: mkdir -p benchdiff/2944efe/bin/1058449141
gcloud storage cp gs://cockroach-microbench-ci/builds/2944efe978a7b7eaa17826772f9d4ca633b6f75a/bin/pkg_sql_tests benchdiff/2944efe/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
chmod +x benchdiff/2944efe/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
mkdir -p benchdiff/3944370/bin/1058449141
gcloud storage cp gs://cockroach-microbench-ci/builds/3944370be734756fdc54b5aa04934f6c4315464c/bin/pkg_sql_tests benchdiff/3944370/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
chmod +x benchdiff/3944370/bin/1058449141/cockroachdb_cockroach_pkg_sql_testsbenchdiff command: # NB: for best (most stable) results, also add a suitable `--benchtime` that
# results in ~1s to ~5s of benchmark runs. For example, if ops average ~3ms, a
# benchtime of `1000x` is appropriate.
#
# Some benchmarks (in particular BenchmarkSysbench) output additional memory
# profiles covering only the execution (excluding the setup/teardown) - those
# should be preferred for analysis since they more closely correspond to what's
# reported as B/op and alloc/op.
benchdiff --run=^BenchmarkSysbench/SQL/3node/oltp_read_write$ --old=3944370 --new=2944efe --memprofile ./pkg/sql/tests⚪ Sysbench [KV, 3node, oltp_read_only]
Reproducebenchdiff binaries: mkdir -p benchdiff/2944efe/bin/1058449141
gcloud storage cp gs://cockroach-microbench-ci/builds/2944efe978a7b7eaa17826772f9d4ca633b6f75a/bin/pkg_sql_tests benchdiff/2944efe/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
chmod +x benchdiff/2944efe/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
mkdir -p benchdiff/3944370/bin/1058449141
gcloud storage cp gs://cockroach-microbench-ci/builds/3944370be734756fdc54b5aa04934f6c4315464c/bin/pkg_sql_tests benchdiff/3944370/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
chmod +x benchdiff/3944370/bin/1058449141/cockroachdb_cockroach_pkg_sql_testsbenchdiff command: # NB: for best (most stable) results, also add a suitable `--benchtime` that
# results in ~1s to ~5s of benchmark runs. For example, if ops average ~3ms, a
# benchtime of `1000x` is appropriate.
#
# Some benchmarks (in particular BenchmarkSysbench) output additional memory
# profiles covering only the execution (excluding the setup/teardown) - those
# should be preferred for analysis since they more closely correspond to what's
# reported as B/op and alloc/op.
benchdiff --run=^BenchmarkSysbench/KV/3node/oltp_read_only$ --old=3944370 --new=2944efe --memprofile ./pkg/sql/tests🔴 Sysbench [KV, 3node, oltp_write_only]
Reproducebenchdiff binaries: mkdir -p benchdiff/2944efe/bin/1058449141
gcloud storage cp gs://cockroach-microbench-ci/builds/2944efe978a7b7eaa17826772f9d4ca633b6f75a/bin/pkg_sql_tests benchdiff/2944efe/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
chmod +x benchdiff/2944efe/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
mkdir -p benchdiff/3944370/bin/1058449141
gcloud storage cp gs://cockroach-microbench-ci/builds/3944370be734756fdc54b5aa04934f6c4315464c/bin/pkg_sql_tests benchdiff/3944370/bin/1058449141/cockroachdb_cockroach_pkg_sql_tests
chmod +x benchdiff/3944370/bin/1058449141/cockroachdb_cockroach_pkg_sql_testsbenchdiff command: # NB: for best (most stable) results, also add a suitable `--benchtime` that
# results in ~1s to ~5s of benchmark runs. For example, if ops average ~3ms, a
# benchtime of `1000x` is appropriate.
#
# Some benchmarks (in particular BenchmarkSysbench) output additional memory
# profiles covering only the execution (excluding the setup/teardown) - those
# should be preferred for analysis since they more closely correspond to what's
# reported as B/op and alloc/op.
benchdiff --run=^BenchmarkSysbench/KV/3node/oltp_write_only$ --old=3944370 --new=2944efe --memprofile ./pkg/sql/testsArtifactsdownload: mkdir -p new
gcloud storage cp gs://cockroach-microbench-ci/artifacts/2944efe978a7b7eaa17826772f9d4ca633b6f75a/23961324784-1/\* new/
mkdir -p old
gcloud storage cp gs://cockroach-microbench-ci/artifacts/3944370be734756fdc54b5aa04934f6c4315464c/23961324784-1/\* old/built with commit: 2944efe978a7b7eaa17826772f9d4ca633b6f75a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a new statement fingerprint store and V2 SQL Activity API endpoints
with server-side pagination, sorting, and filtering. This replaces the
expensive client-side aggregation in the existing gRPC endpoints with
efficient server-side queries against the denormalized covering indexes
on
system.statement_statisticsandsystem.transaction_statistics.pkg/obs/statementstore): Write-through FIFOcache that deduplicates and asynchronously persists statement
fingerprints to
system.statementsvia a background writer goroutine.each statement observation, including when fingerprint IDs are
recomputed due to implicit/explicit transaction changes.
system.statementsfromsystem.statement_statisticsso fingerprintsobserved before the store existed are immediately available.
GET /api/v2/dbconsole/statements/andGET /api/v2/dbconsole/transactions/with server-side pagination,sorting, time range, search, app name, database, and internal
statement filters. Datadriven EXPLAIN tests verify covering index
usage.
StatementsPageV2andTransactionsPageV2Reactcomponents using SWR hooks, accessible from the Advanced Debug panel.
The individual commits speak for themselves — each is a self-contained
logical unit that builds on the previous one.
Epic: none
Release note: None