Big nodes, big loads and lots of (overdue) optimizations#8677

Merged

rustyrussell merged 45 commits into

ElementsProject:masterfrom

rustyrussell:guilt/bkpr-post-migration

Nov 20, 2025

Commits on Nov 20, 2025

bookkeeper: fix printing of bad JSON results.
rustyrussell
committed
pytest: fix changed msg on plugin failure.
rustyrussell
committed
plugins/sql: remove size limit.
rustyrussell
committed
plugins/sql: print times taken to do list comand, populate table, and create index.
rustyrussell
committed
bookkeeper: no longer read listchannelmoves 1000 entries at a time.
rustyrussell
committed
Revert "bookkeeper: don't flood logs if we have many channelmoves all at once."
rustyrussell
committed
pytest: latency and speed test on large coinmoves.
rustyrussell
committed
lightningd: don't copy hooks array into hook request, simply don't shrink it.
rustyrussell
committed
lightningd: avoid race when runtime-added plugins register hooks.
rustyrussell
committed
lightningd: log when we have to defer hook registration.
rustyrussell
committed
lightningd: don't loop through all commands every time one finishes.
rustyrussell
committed
common: remove tracing exponential behaviour from large numbers of requests.
rustyrussell
committed
common: avoid allocations for small numbers of traces.
rustyrussell
committed
pytest: increase test_generate_coinmoves to 2M entries.
rustyrussell
committed
lightningd: handle large numbers of command outputs gracefully.
rustyrussell
committed
JSONRPC: use a bigger default buffer.
rustyrussell
committed
lightningd: support "filters" in plugins manifest to restrict when hooks are called.
rustyrussell
committed
lightningd: add support for filters on "rpc_command" hook.
rustyrussell
committed
libplugin: allow plugins to register optional filters for each hook they want.
rustyrussell
committed
libplugin: don't wait for clean_tmpctx() to free requests as we process them.
rustyrussell
committed
xpay: use filtering on rpc_command so we only get called on "pay".
rustyrussell
committed
pyln-client: support hook filters.
rustyrussell
committed
lightningd: allow filtering on custommsg hook too.
rustyrussell
committed
commando, chanbackup: use custommsg hooks.
rustyrussell
committed
common: increase jsonrpc_io buffer size temporarily to aggrevate perf issues.
rustyrussell
committed
common: optimize json parsing.
rustyrussell
committed
ccan: update to get io_loop fairness.
rustyrussell
committed
lightningd: don't process more than 100 commands from a JSONRPC at once.
rustyrussell
committed
lightningd: don't process more than 100 commands from a plugin at once.
rustyrussell
committed
plugins/sql: use modern data style, not globals.
rustyrussell
committed
sql: if we use `dev-sqlfilename`, don't bother syncing it to disk.
rustyrussell
committed
sql: use wait RPC so we don't have to check listchannelmoves/listchainmoves each time.
rustyrussell
committed
lightningd: optimize find_cmd.
rustyrussell
committed
bookkeeper: restore limit on asking for all channelmoves at once.
rustyrussell
committed
sql: limit how many chainmoves/channelmoves entries we ask for at once.
rustyrussell
committed
pytest: increase test_generate_coinmoves to 5M entries.
rustyrussell
committed
pytest: test for 1M JSONRPC calls which don't need transactions.
rustyrussell
committed
db: don't start transactions unless we really need to.
rustyrussell
committed
Python: update pyproject.toml so we can run tests/benchmarks.py.
rustyrussell
committed
pyln-testing: don't assume we're doing debug logging for fundwallet and line_graph helpers.
rustyrussell
committed
tests/benchmark.py: tune nodes a little.
rustyrussell
committed
pytest: move test_coinmoves.py::test_generate_coinmoves and test_plugin.py::test_spam_commands to benchmark.py
rustyrussell
committed
libplugin: add spamlistcommand
rustyrussell
committed
db: don't actually create an explicit transaction for read commands.
rustyrussell
committed
lightningd: print last method we called if we abort processing loop.
rustyrussell
committed