Access to keys with pagination

[MathieuDutSik]

Motivation

Accessing key-values in MapView is an expensive operation that blocks the system.
It would be better to be able to do pagination.

Proposal

When doing a find_{keys,key_values}_by_prefix in ScyllaDB, RocksDB and others, we are accessing the keys with a start and end with the start and end specifically defined. So, there is a very clear way to generalize the code. We also have a very clear way to limit the size of the request. In ScyllaDB, this is done via the " LIMIT X" appended, and for RocksDB, we can stop in the middle.

Now, if we ask for 200 keys and we get 190 then we know that we do not need to continue. If we get exactly 200, then the number of keys may be 200 or 201. But we do not that. So, we report is_finished=false in that case. The number of keys being returned may in the end be much less than 200 because the value_splitting forces to split some keys in several smaller keys.

This opens the possibility of reading the key-values of a MapView with pagination. This has not been done in this PR.

Test Plan

CI.
The tests have been expanded.

Release Plan

This can be backported to testnet_conway.

Links

None.

[github-actions]

Instruction Count Benchmark Results

Baseline: b835ec6337

Warning

2 instruction-count regression(s) detected (>1% threshold):

• load_all_100_from_storage: +2.37%
• load_all_100_from_storage: +2.46%

Deterministic metrics — reproducible across runs (34 benchmarks)

Benchmark	Instructions	Total R+W
BucketQueueView
delete_500_from_1000	22,383 (-0.01%)	34,338 (-0.02%)
front_100_from_1000	5,680 (-0.37%)	8,394 (-0.31%)
pre_save_1000	42,118 (${\color{green}\textbf{-2.28\%%}}$)	59,007 (${\color{green}\textbf{-2.42\%%}}$)
push_1000	24,199 (-0.69%)	33,096 (-0.68%)
Cold Load
load_1000	693,836 (-0.00%)	1,010,312 (-0.00%)
CollectionView
indices_100	189,004 (${\color{green}\textbf{-1.76\%%}}$)	261,107 (${\color{green}\textbf{-2.45\%%}}$)
load_all_100_from_storage	651,640 (${\color{red}\textbf{+2.37\%%}}$)	919,041 (${\color{red}\textbf{+2.26\%%}}$)
load_all_100_in_memory	337,372 (${\color{green}\textbf{-1.02\%%}}$)	470,163 (${\color{green}\textbf{-1.46\%%}}$)
pre_save_100	265,515 (-0.14%)	367,211 (-0.13%)
try_load_10_from_100	100,175 (-0.39%)	141,887 (-0.36%)
MapView
contains_key_10_from_100	51,975 (${\color{green}\textbf{-1.06\%%}}$)	73,886 (-0.89%)
contains_key_10_from_1000	351,924 (-0.89%)	497,676 (-0.83%)
get_10_from_100	54,637 (-0.72%)	77,681 (-0.66%)
get_10_from_1000	354,651 (-0.83%)	501,556 (-0.79%)
get_100_missing_from_1000	606,027 (-0.66%)	845,974 (-0.62%)
indices_100	98,602 (${\color{green}\textbf{-1.85\%%}}$)	135,657 (${\color{green}\textbf{-1.94\%%}}$)
indices_1000	927,767 (${\color{green}\textbf{-2.16\%%}}$)	1,291,981 (${\color{green}\textbf{-2.28\%%}}$)
insert_100	255,953 (-0.50%)	354,295 (-0.39%)
insert_1000	2,950,644 (-0.43%)	3,999,426 (-0.34%)
post_save_1000	1,024,163 (-0.28%)	1,477,183 (-0.26%)
pre_save_100	330,034 (-0.77%)	459,377 (-0.67%)
pre_save_1000	3,357,857 (-0.71%)	4,729,876 (-0.61%)
remove_500_from_1000	1,192,229 (+0.22%)	1,666,625 (+0.37%)
QueueView
delete_500_from_1000	10,222 (-0.21%)	12,325 (-0.21%)
front_100_from_1000	9,040 (${\color{green}\textbf{-1.06\%%}}$)	13,787 (-0.68%)
pre_save_1000	993,905 (${\color{green}\textbf{-4.70\%%}}$)	1,423,616 (${\color{green}\textbf{-5.00\%%}}$)
push_1000	24,294 (No change)	33,225 (No change)
ReentrantCollectionView
contains_key_10_from_100	141,458 (-0.31%)	201,275 (-0.27%)
indices_100	234,212 (${\color{green}\textbf{-1.22\%%}}$)	326,462 (${\color{green}\textbf{-1.77\%%}}$)
load_all_100_from_storage	821,377 (${\color{red}\textbf{+2.46\%%}}$)	1,158,242 (${\color{red}\textbf{+2.51\%%}}$)
load_all_100_in_memory	412,252 (+0.25%)	567,039 (+0.14%)
pre_save_100	350,425 (-0.11%)	487,977 (-0.10%)
RegisterView
get_set_100	81,292 (No change)	120,126 (No change)
pre_save	5,467 (-0.33%)	8,073 (-0.20%)

Regression threshold: 1% — ${\color{red}\textbf{red}}$ = regression, ${\color{green}\textbf{green}}$ = improvement.

Cache-dependent metrics — expect fluctuations between runs (34 benchmarks)

Benchmark	L1 Hits	LLC Hits	RAM Hits	Est. Cycles
BucketQueueView
delete_500_from_1000	34,145 (-0.02%)	32 (${\color{red}\textbf{+6.67\%%}}$)	161 (${\color{green}\textbf{-1.23\%%}}$)	39,940 (-0.16%)
front_100_from_1000	8,224 (-0.29%)	36 (${\color{red}\textbf{+2.86\%%}}$)	134 (${\color{green}\textbf{-2.19\%%}}$)	13,094 (-0.94%)
pre_save_1000	58,674 (${\color{green}\textbf{-2.42\%%}}$)	59 (${\color{red}\textbf{+5.36\%%}}$)	274 (${\color{green}\textbf{-4.20\%%}}$)	68,559 (${\color{green}\textbf{-2.64\%%}}$)
push_1000	32,900 (-0.65%)	36 (${\color{green}\textbf{-20.00\%%}}$)	160 (-0.62%)	38,680 (-0.76%)
Cold Load
load_1000	1,001,786 (-0.00%)	8,353 (-0.05%)	173 (${\color{green}\textbf{-1.14\%%}}$)	1,049,606 (-0.01%)
CollectionView
indices_100	259,831 (${\color{green}\textbf{-2.47\%%}}$)	855 (No change)	421 (${\color{red}\textbf{+7.40\%%}}$)	278,841 (${\color{green}\textbf{-1.95\%%}}$)
load_all_100_from_storage	914,467 (${\color{red}\textbf{+2.27\%%}}$)	3,890 (+0.21%)	684 (${\color{red}\textbf{+1.18\%%}}$)	957,857 (${\color{red}\textbf{+2.20\%%}}$)
load_all_100_in_memory	468,012 (${\color{green}\textbf{-1.47\%%}}$)	1,389 (-0.14%)	762 (${\color{red}\textbf{+2.42\%%}}$)	501,627 (${\color{green}\textbf{-1.25\%%}}$)
pre_save_100	365,274 (-0.13%)	1,339 (-0.37%)	598 (+0.50%)	392,899 (-0.10%)
try_load_10_from_100	141,022 (-0.37%)	637 (+0.79%)	228 (${\color{red}\textbf{+3.17\%%}}$)	152,187 (-0.17%)
MapView
contains_key_10_from_100	73,588 (-0.89%)	92 (No change)	206 (-0.48%)	81,258 (-0.85%)
contains_key_10_from_1000	494,495 (-0.83%)	2,975 (+0.10%)	206 (No change)	516,580 (-0.79%)
get_10_from_100	77,379 (-0.66%)	86 (${\color{green}\textbf{-5.49\%%}}$)	216 (${\color{red}\textbf{+1.41\%%}}$)	85,369 (-0.51%)
get_10_from_1000	498,360 (-0.79%)	2,980 (-0.13%)	216 (${\color{red}\textbf{+1.89\%%}}$)	520,820 (-0.73%)
get_100_missing_from_1000	842,757 (-0.62%)	2,983 (+0.07%)	234 (+0.43%)	865,862 (-0.60%)
indices_100	134,996 (${\color{green}\textbf{-1.98\%%}}$)	231 (${\color{red}\textbf{+3.12\%%}}$)	430 (${\color{red}\textbf{+6.44\%%}}$)	151,201 (${\color{green}\textbf{-1.16\%%}}$)
indices_1000	1,284,273 (${\color{green}\textbf{-2.30\%%}}$)	6,493 (+0.11%)	1,215 (${\color{red}\textbf{+2.36\%%}}$)	1,359,263 (${\color{green}\textbf{-2.11\%%}}$)
insert_100	353,544 (-0.39%)	95 (${\color{red}\textbf{+7.95\%%}}$)	656 (+0.15%)	376,979 (-0.35%)
insert_1000	3,992,372 (-0.35%)	3,064 (+0.69%)	3,990 (+0.13%)	4,147,342 (-0.33%)
post_save_1000	1,465,792 (-0.26%)	11,206 (-0.01%)	185 (${\color{red}\textbf{+2.21\%%}}$)	1,528,297 (-0.24%)
pre_save_100	457,993 (-0.67%)	767 (-0.26%)	617 (+0.49%)	483,423 (-0.62%)
pre_save_1000	4,715,948 (-0.61%)	10,109 (+0.01%)	3,819 (+0.08%)	4,900,158 (-0.58%)
remove_500_from_1000	1,662,242 (+0.37%)	4,202 (+0.02%)	181 (+0.56%)	1,689,587 (+0.36%)
QueueView
delete_500_from_1000	12,156 (-0.16%)	33 (${\color{green}\textbf{-5.71\%%}}$)	136 (${\color{green}\textbf{-2.86\%%}}$)	17,081 (-0.99%)
front_100_from_1000	13,590 (-0.68%)	35 (${\color{red}\textbf{+2.94\%%}}$)	162 (${\color{green}\textbf{-1.22\%%}}$)	19,435 (-0.81%)
pre_save_1000	1,418,924 (${\color{green}\textbf{-5.02\%%}}$)	2,732 (-0.18%)	1,960 (-0.20%)	1,501,184 (${\color{green}\textbf{-4.77\%%}}$)
push_1000	33,018 (+0.01%)	46 (${\color{green}\textbf{-8.00\%%}}$)	161 (No change)	38,883 (-0.04%)
ReentrantCollectionView
contains_key_10_from_100	200,046 (-0.27%)	1,031 (-0.29%)	198 (-1.00%)	212,131 (-0.29%)
indices_100	324,862 (${\color{green}\textbf{-1.79\%%}}$)	1,203 (+0.25%)	397 (${\color{red}\textbf{+6.15\%%}}$)	344,772 (${\color{green}\textbf{-1.45\%%}}$)
load_all_100_from_storage	1,151,638 (${\color{red}\textbf{+2.52\%%}}$)	6,163 (+0.39%)	441 (${\color{red}\textbf{+6.27\%%}}$)	1,197,888 (${\color{red}\textbf{+2.51\%%}}$)
load_all_100_in_memory	564,622 (+0.13%)	1,839 (-0.70%)	578 (${\color{red}\textbf{+5.09\%%}}$)	594,047 (+0.28%)
pre_save_100	485,056 (-0.10%)	2,236 (-0.04%)	685 (-0.29%)	520,211 (-0.11%)
RegisterView
get_set_100	119,906 (+0.00%)	40 (${\color{green}\textbf{-2.44\%%}}$)	180 (${\color{green}\textbf{-1.10\%%}}$)	126,406 (-0.06%)
pre_save	7,863 (-0.23%)	47 (${\color{red}\textbf{+6.82\%%}}$)	163 (-0.61%)	13,803 (-0.27%)

Cache metrics fluctuate because anything that changes the virtual memory layout
shifts which data lands on which cache lines, changing the L1/LLC/RAM distribution.
Probable causes: ASLR (even across identical binaries), executable binary size changes,
shared library size changes, and even filename length differences.

Cachegrind simulates a two-level cache (L1 + LLC) auto-detected from the host CPU.
Est. Cycles = L1 hits + 5 × LLC hits + 35 × RAM hits.

Runner cache sizes: L1d cache: 64 KiB (2 instances);L1i cache: 64 KiB (2 instances) L2 cache: 2 MiB (2 instances);L3 cache: 32 MiB (1 instance)

[deuszx]

if you get to work on pagination for try_load_all_entries – please reference the corresponding issue #2742

[ma2bd]

This is cool.