perf: sqlite prepare cached

[francesco-stacks]

Description

Profiling a mainnet follower during catch-up (on top of #7300) showed the SQLite SQL parser (resolveExprStep, sqlite3RunParser, yy_reduce) as the largest remaining CPU sink: ~5.3% of chains-coordinator self-time. We re-compile the same SQL strings millions of times (the Clarity side store on every data read, the MARF on every trie-cache miss, and the query_* helpers in util_lib/db.rs for every caller).

rusqlite ships a per-connection prepared-statement cache, prepare_cached, which compiles once and reuses the bytecode. We just don't use it on these paths, and its default capacity (16) would LRU-thrash under the node's breadth of statements.

This PR routes the hot read paths through the cache and raises the per-connection cache capacity to 200 in sqlite_open.

Measured impact - A/B on a mainnet follower (identical chainstate copies, same start height, 10-minute windows):

	baseline	prepare_cached
catch-up throughput	357 blk/min	417 blk/min (+16.8%)
SQL-parser self-time (coordinator)	~5.3%	~1.9%

I also run a first iteration (that wasn't caching everything this one is caching) with stacks-bench in cylewitruk#5 (comment) that measured it ~4% faster during the benchmark.

Applicable issues

• fixes #

Additional info (benefits, drawbacks, caveats)

Checklist

• Test coverage for new or modified code paths
• For new Clarity features or consensus changes, add property tests (see docs/property-testing.md)
• Changelog fragment(s) or "no changelog" label added (see changelog.d/README.md)
• Required documentation changes (e.g., rpc/openapi.yaml for RPC endpoints, event-dispatcher.md for new events)
• New clarity functions have corresponding PR in clarity-benchmarking repo

[Copilot]

Pull request overview

This PR improves SQLite query performance by switching hot-path queries to use rusqlite’s prepared-statement cache (prepare_cached) and increasing per-connection statement-cache capacity to reduce repeated SQL parsing.

Changes:

• Use prepare_cached in shared SQLite query helpers and several frequently-called read paths (Clarity VM headers queries, MARF squashed-block lookups, SortDB tip lookups).
• Increase prepared-statement cache capacity on long-lived connections (stackslib: 200; Clarity side-store: 32).
• Add a changelog fragment documenting the performance improvement.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
stackslib/src/util_lib/db.rs	Switch generic query_* helpers and a MARF lookup to prepare_cached; bump per-connection statement-cache capacity to 200 in sqlite_open().
stackslib/src/clarity_vm/database/mod.rs	Cache dynamically-built header-table SELECT statements via prepare_cached for repeated access patterns.
stackslib/src/chainstate/stacks/index/trie_sql.rs	Use prepare_cached for MARF squashed-block and block-id/hash lookup queries.
stackslib/src/chainstate/burn/db/sortdb.rs	Use prepare_cached for canonical tip lookup queries in SortDB.
clarity/src/vm/database/sqlite.rs	Use prepare_cached for side-store reads/writes; bump statement-cache capacity to 32.
changelog.d/sqlite-prepare-cached.changed	Document statement caching + cache-capacity increase.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coveralls]

Coverage Report for CI Build 29319771434

Coverage increased (+0.9%) to 86.564%

Details

• Coverage increased (+0.9%) from the base build.
• Patch coverage: 1 uncovered change across 1 file (70 of 71 lines covered, 98.59%).
• 11964 coverage regressions across 165 files.

Uncovered Changes

File	Changed	Covered	%
stackslib/src/chainstate/stacks/index/trie_sql.rs	24	23	95.83%
Total (5 files)	71	70	98.59%

Coverage Regressions

11964 previously-covered lines in 165 files lost coverage.

Top 10 Files by Coverage Loss	Lines Losing Coverage	Coverage
stackslib/src/chainstate/burn/db/sortdb.rs	637	90.45%
stackslib/src/chainstate/stacks/db/blocks.rs	528	89.93%
stackslib/src/chainstate/nakamoto/mod.rs	490	84.77%
stackslib/src/config/mod.rs	324	78.53%
stackslib/src/net/relay.rs	322	74.86%
stackslib/src/net/mod.rs	312	78.18%
stackslib/src/chainstate/stacks/index/storage.rs	277	82.41%
clarity/src/vm/database/clarity_db.rs	273	82.02%
stackslib/src/chainstate/stacks/db/transactions.rs	269	97.13%
stackslib/src/chainstate/stacks/miner.rs	265	83.4%

---

Coverage Stats

Relevant Lines:	231315
Covered Lines:	200236
Line Coverage:	86.56%
Coverage Strength:	19552602.25 hits per line

---

💛 - Coveralls

[benjamin-stacks]

🚀

[federico-stacks]

nit: even if used in one place we could define 32 (and 200) as named constants with related docstring.

maybe we could even use just one constants (the bigger), but I'm fine keeping them separated due to the working set.

[federico-stacks]

nit: this comment looks like can easily drift if we change the cache configuration