Optimize variable-size single row replacements

[timsehn]

Summary

• expands the single-row prolly replacement fast path to handle value-size changes when the rebuilt leaf proves its chunk-boundary shape is unchanged
• keeps the existing same-size path unchanged and falls back to streaming merge when shape validation fails
• adds invariant coverage for variable-size replacements on a BLOB primary-key table

Benchmark Target

Last merged PR checked: #1038 (558c46ffc1, "Optimize exact composite primary key seeks").

Highest sysbench multiplier in the generated comments:

• key type: blobpk
• mode: file-backed
• autocommit: yes
• metric: oltp_delete_insert_ac
• multiplier: 5.03x (88,459us Doltlite / 17,578us SQLite)

Local DoltLite-only comparison on 100K-row BLOB PK targets, 30 runs each:

Test	Baseline median	New median	Baseline trimmed median	New trimmed median
oltp_delete_insert_ac	135,623us	60,493us	96,981us	55,614us
oltp_update_index_ac	167,301us	76,659us	126,243us	51,405us
oltp_insert_ac	41,581us	40,805us	41,180us	39,631us

Local runs are noisy on this machine, but the before/after was run from clean origin/master and this branch using the same harness.

Validation

• BENCH_ROWS=200 BENCH_RUNS=1 BENCH_MAX_MULTIPLIER=1000 BENCH_AVG_MAX_MULTIPLIER=1000 ./test/sysbench_compare_blobpk.sh
• make -j8 invariant_test corruption_test c-tests
• ./invariant_test
• test/run_c_tests.sh .

[github-actions]

Sysbench-Style Benchmark: Doltlite vs SQLite

In-Memory

Reads

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	22,165	35,264	1.59
oltp_range_select	9,465	14,000	1.48
oltp_sum_range	8,864	14,115	1.59
oltp_order_range	2,424	3,172	1.31
oltp_distinct_range	3,509	4,228	1.20
oltp_index_scan	3,658	6,589	1.80
select_random_points	8,892	17,273	1.94
select_random_ranges	2,634	5,115	1.94
covering_index_scan	4,073	4,312	1.06
groupby_scan	30,173	34,514	1.14
index_join	5,797	9,152	1.58
index_join_scan	3,034	5,336	1.76
types_table_scan	1,043,094	1,362,070	1.31
table_scan	1,156,887	1,519,537	1.31
oltp_read_only	98,473	136,042	1.38
Average			1.49

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert	174,608	237,710	1.36
oltp_insert	15,100	26,632	1.76
oltp_update_index	47,169	100,870	2.14
oltp_update_non_index	31,887	66,417	2.08
oltp_delete_insert	42,146	76,382	1.81
oltp_write_only	20,315	47,563	2.34
types_delete_insert	23,096	43,196	1.87
oltp_read_write	62,952	122,556	1.95
Average			1.91

File-Backed

Reads

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	97,426	101,335	1.04
oltp_range_select	18,396	34,583	1.88
oltp_sum_range	18,280	34,751	1.90
oltp_order_range	3,500	6,072	1.73
oltp_distinct_range	4,573	7,097	1.55
oltp_index_scan	11,654	17,073	1.46
select_random_points	22,999	61,888	2.69
select_random_ranges	10,424	14,301	1.37
covering_index_scan	11,279	7,631	0.68
groupby_scan	33,009	49,972	1.51
index_join	10,042	17,155	1.71
index_join_scan	4,319	11,899	2.76
types_table_scan	1,188,041	2,642,471	2.22
table_scan	1,375,423	3,308,844	2.41
oltp_read_only	213,594	281,790	1.32
Average			1.75

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert	182,733	252,451	1.38
oltp_insert	24,257	36,294	1.50
oltp_update_index	120,752	214,904	1.78
oltp_update_non_index	78,826	142,057	1.80
oltp_delete_insert	92,734	152,583	1.65
oltp_write_only	59,871	100,714	1.68
types_delete_insert	48,554	83,122	1.71
oltp_read_write	119,377	257,601	2.16
Average			1.71

File-Backed (autocommit)

Each statement runs as its own transaction — exposes per-commit
fixed costs that the wrapped-in-BEGIN/COMMIT tests amortize away.
SQLite uses WAL mode with synchronous=FULL in this section so
the comparison uses SQLite's durable WAL autocommit path.

Reads

Reads have no commit cost; these are the same SQL files as the
File-Backed Reads section, included here for symmetry and to
catch any per-statement overhead doltlite pays on the read path.

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	52,550	102,358	1.95
oltp_range_select	14,007	34,767	2.48
oltp_sum_range	13,802	34,657	2.51
oltp_order_range	3,006	6,118	2.04
oltp_distinct_range	4,066	7,113	1.75
oltp_index_scan	7,296	16,967	2.33
select_random_points	18,405	61,661	3.35
select_random_ranges	6,012	14,371	2.39
covering_index_scan	6,714	7,631	1.14
groupby_scan	32,611	49,927	1.53
index_join	7,784	17,262	2.22
index_join_scan	3,806	11,918	3.13
types_table_scan	1,196,696	2,645,665	2.21
table_scan	1,376,375	3,304,261	2.40
oltp_read_only	148,806	281,518	1.89
Average			2.22

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert_ac	20,253	73,030	3.61
oltp_insert_ac	21,744	87,209	4.01
oltp_update_index_ac	27,410	99,437	3.63
oltp_update_non_index_ac	21,053	82,871	3.94
oltp_delete_insert_ac	22,360	92,688	4.15
oltp_write_only_ac	22,727	92,677	4.08
types_delete_insert_ac	19,494	81,916	4.20
oltp_read_write_ac	27,295	104,661	3.83
Average			3.93

100000 rows, median of 5 invocations per test, workload-only timing via host monotonic clock when available.

Performance Ceiling Check (6x individual, 5x average)

All tests within ceilings.

[github-actions]

Sysbench-Style Benchmark (composite PK): Doltlite vs SQLite

Companion to the classic Sysbench-Style Benchmark. Every workload here
runs against tables with a 2-column INTEGER PRIMARY KEY(a, b) WITHOUT ROWID.
Individual ratios gated at 6×; section averages gated at 5×.

In-Memory

Reads

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	31,819	45,982	1.45
oltp_range_select	18,200	26,436	1.45
oltp_sum_range	17,405	25,736	1.48
oltp_order_range	3,430	4,438	1.29
oltp_distinct_range	4,581	5,547	1.21
oltp_index_scan	4,274	7,336	1.72
select_random_points	27,031	38,738	1.43
select_random_ranges	7,394	9,912	1.34
covering_index_scan	4,064	4,791	1.18
groupby_scan	36,850	43,676	1.19
index_join	7,853	12,422	1.58
index_join_scan	3,979	6,679	1.68
types_table_scan	1,052,888	1,491,970	1.42
table_scan	1,184,339	1,660,455	1.40
oltp_read_only	145,288	196,261	1.35
Average			1.41

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert	238,670	326,564	1.37
oltp_insert	18,744	33,453	1.78
oltp_update_index	64,329	122,710	1.91
oltp_update_non_index	48,580	82,759	1.70
oltp_delete_insert	47,469	96,102	2.02
oltp_write_only	26,375	56,176	2.13
types_delete_insert	31,433	57,545	1.83
oltp_read_write	97,773	170,968	1.75
Average			1.81

File-Backed

Reads

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	106,797	116,782	1.09
oltp_range_select	27,809	50,729	1.82
oltp_sum_range	27,229	49,316	1.81
oltp_order_range	4,513	7,773	1.72
oltp_distinct_range	5,568	8,881	1.60
oltp_index_scan	12,460	19,386	1.56
select_random_points	43,993	88,643	2.01
select_random_ranges	15,595	20,870	1.34
covering_index_scan	10,905	10,003	0.92
groupby_scan	39,681	61,160	1.54
index_join	12,731	25,619	2.01
index_join_scan	5,113	15,876	3.11
types_table_scan	1,211,580	2,999,263	2.48
table_scan	1,414,268	3,677,152	2.60
oltp_read_only	264,888	361,963	1.37
Average			1.80

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert	251,456	348,527	1.39
oltp_insert	25,779	48,573	1.88
oltp_update_index	150,502	256,435	1.70
oltp_update_non_index	102,449	168,663	1.65
oltp_delete_insert	102,982	184,534	1.79
oltp_write_only	63,034	118,797	1.88
types_delete_insert	60,672	102,711	1.69
oltp_read_write	162,094	324,048	2.00
Average			1.75

File-Backed (autocommit)

Each statement runs as its own transaction — exposes per-commit
fixed costs that the wrapped-in-BEGIN/COMMIT tests amortize away.
SQLite uses WAL mode with synchronous=FULL in this section so
the comparison uses SQLite's durable WAL autocommit path.

Reads

Reads have no commit cost; these are the same SQL files as the
File-Backed Reads section, included here for symmetry and to
catch any per-statement overhead doltlite pays on the read path.

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	62,478	116,619	1.87
oltp_range_select	22,908	50,162	2.19
oltp_sum_range	22,557	49,280	2.18
oltp_order_range	3,995	7,757	1.94
oltp_distinct_range	5,089	8,938	1.76
oltp_index_scan	7,924	19,394	2.45
select_random_points	38,333	89,199	2.33
select_random_ranges	10,703	20,582	1.92
covering_index_scan	6,646	10,012	1.51
groupby_scan	39,730	60,831	1.53
index_join	10,428	25,598	2.45
index_join_scan	4,668	15,923	3.41
types_table_scan	1,203,740	3,001,911	2.49
table_scan	1,420,554	3,674,454	2.59
oltp_read_only	200,329	362,942	1.81
Average			2.16

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert_ac	23,398	78,177	3.34
oltp_insert_ac	24,099	94,882	3.94
oltp_update_index_ac	30,870	108,141	3.50
oltp_update_non_index_ac	23,748	88,947	3.75
oltp_delete_insert_ac	27,314	101,804	3.73
oltp_write_only_ac	25,814	97,696	3.78
types_delete_insert_ac	21,969	96,346	4.39
oltp_read_write_ac	35,214	112,685	3.20
Average			3.70

100000 rows, median of 5 invocations per test, workload-only timing via host monotonic clock when available.

Performance Ceiling Check (6x individual, 5x average)

All tests within ceilings.

[github-actions]

Sysbench-Style Benchmark (BLOB PK): Doltlite vs SQLite

Companion to the classic Sysbench-Style Benchmark. Every workload here
runs against tables with a 16-byte big-endian BLOB PRIMARY KEY.
Individual ratios gated at 6×; section averages gated at 5×.

In-Memory

Reads

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	31,792	46,280	1.46
oltp_range_select	13,470	20,967	1.56
oltp_sum_range	12,401	20,442	1.65
oltp_order_range	2,993	3,961	1.32
oltp_distinct_range	4,007	5,033	1.26
oltp_index_scan	4,744	7,596	1.60
select_random_points	18,152	28,529	1.57
select_random_ranges	4,121	6,512	1.58
covering_index_scan	4,314	5,449	1.26
groupby_scan	32,528	40,128	1.23
index_join	6,871	12,788	1.86
index_join_scan	4,130	8,279	2.00
types_table_scan	1,137,373	1,580,091	1.39
table_scan	1,319,970	1,758,188	1.33
oltp_read_only	127,745	172,826	1.35
Average			1.50

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert	242,884	341,467	1.41
oltp_insert	20,854	39,626	1.90
oltp_update_index	71,566	142,951	2.00
oltp_update_non_index	50,862	90,603	1.78
oltp_delete_insert	51,549	110,221	2.14
oltp_write_only	29,794	66,567	2.23
types_delete_insert	33,727	61,229	1.82
oltp_read_write	88,794	163,151	1.84
Average			1.89

File-Backed

Reads

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	107,654	123,426	1.15
oltp_range_select	22,024	50,414	2.29
oltp_sum_range	21,324	50,048	2.35
oltp_order_range	3,932	8,003	2.04
oltp_distinct_range	4,992	9,027	1.81
oltp_index_scan	13,153	21,828	1.66
select_random_points	34,274	85,237	2.49
select_random_ranges	11,640	17,749	1.52
covering_index_scan	11,765	13,281	1.13
groupby_scan	35,408	62,188	1.76
index_join	13,336	34,618	2.60
index_join_scan	6,646	24,359	3.67
types_table_scan	1,260,159	3,597,636	2.85
table_scan	1,493,146	4,298,971	2.88
oltp_read_only	240,416	366,391	1.52
Average			2.11

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert	254,506	364,872	1.43
oltp_insert	36,665	58,151	1.59
oltp_update_index	174,919	309,065	1.77
oltp_update_non_index	107,085	188,752	1.76
oltp_delete_insert	117,523	213,126	1.81
oltp_write_only	73,220	134,828	1.84
types_delete_insert	63,699	114,781	1.80
oltp_read_write	156,374	343,979	2.20
Average			1.78

File-Backed (autocommit)

Each statement runs as its own transaction — exposes per-commit
fixed costs that the wrapped-in-BEGIN/COMMIT tests amortize away.
SQLite uses WAL mode with synchronous=FULL in this section so
the comparison uses SQLite's durable WAL autocommit path.

Reads

Reads have no commit cost; these are the same SQL files as the
File-Backed Reads section, included here for symmetry and to
catch any per-statement overhead doltlite pays on the read path.

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	62,903	120,595	1.92
oltp_range_select	17,655	49,258	2.79
oltp_sum_range	16,992	49,618	2.92
oltp_order_range	3,490	7,946	2.28
oltp_distinct_range	4,532	8,982	1.98
oltp_index_scan	8,586	21,365	2.49
select_random_points	28,719	83,077	2.89
select_random_ranges	7,155	17,621	2.46
covering_index_scan	7,573	13,119	1.73
groupby_scan	34,778	62,167	1.79
index_join	11,106	32,985	2.97
index_join_scan	6,329	23,123	3.65
types_table_scan	1,251,579	3,574,428	2.86
table_scan	1,495,730	4,299,855	2.87
oltp_read_only	175,248	363,046	2.07
Average			2.51

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert_ac	22,927	83,221	3.63
oltp_insert_ac	27,223	105,859	3.89
oltp_update_index_ac	28,605	116,605	4.08
oltp_update_non_index_ac	24,974	96,092	3.85
oltp_delete_insert_ac	26,765	111,213	4.16
oltp_write_only_ac	28,276	104,970	3.71
types_delete_insert_ac	24,019	95,781	3.99
oltp_read_write_ac	33,606	120,851	3.60
Average			3.86

100000 rows, median of 5 invocations per test, workload-only timing via host monotonic clock when available.

Performance Ceiling Check (6x individual, 5x average)

All tests within ceilings.

[github-actions]

Sysbench-Style Benchmark (TEXT PK): Doltlite vs SQLite

Companion to the classic Sysbench-Style Benchmark. Every workload here
runs against tables with a 32-char hex TEXT PRIMARY KEY (UUID-shaped).
Individual ratios gated at 6×; section averages gated at 5×.

In-Memory

Reads

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	30,248	43,136	1.43
oltp_range_select	14,338	19,417	1.35
oltp_sum_range	12,276	19,142	1.56
oltp_order_range	3,184	3,810	1.20
oltp_distinct_range	4,250	4,943	1.16
oltp_index_scan	4,563	7,309	1.60
select_random_points	17,494	27,346	1.56
select_random_ranges	4,055	6,594	1.63
covering_index_scan	4,851	5,363	1.11
groupby_scan	35,615	40,964	1.15
index_join	6,844	12,214	1.78
index_join_scan	4,534	8,091	1.78
types_table_scan	1,158,426	1,693,454	1.46
table_scan	1,646,576	2,022,660	1.23
oltp_read_only	140,515	175,016	1.25
Average			1.42

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert	232,485	334,207	1.44
oltp_insert	24,836	41,659	1.68
oltp_update_index	86,323	169,195	1.96
oltp_update_non_index	54,012	102,390	1.90
oltp_delete_insert	58,731	121,224	2.06
oltp_write_only	33,762	71,195	2.11
types_delete_insert	35,650	65,133	1.83
oltp_read_write	93,477	163,351	1.75
Average			1.84

File-Backed

Reads

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	126,799	139,075	1.10
oltp_range_select	25,432	58,104	2.28
oltp_sum_range	24,387	58,558	2.40
oltp_order_range	4,318	8,954	2.07
oltp_distinct_range	5,244	9,807	1.87
oltp_index_scan	15,051	23,582	1.57
select_random_points	37,254	95,256	2.56
select_random_ranges	13,607	20,166	1.48
covering_index_scan	14,326	15,027	1.05
groupby_scan	38,974	72,374	1.86
index_join	15,866	38,794	2.45
index_join_scan	8,859	27,924	3.15
types_table_scan	1,413,446	4,394,628	3.11
table_scan	1,703,048	5,236,744	3.07
oltp_read_only	265,021	399,765	1.51
Average			2.10

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert	246,281	357,611	1.45
oltp_insert	47,655	61,581	1.29
oltp_update_index	200,103	341,434	1.71
oltp_update_non_index	126,692	212,244	1.68
oltp_delete_insert	135,208	237,152	1.75
oltp_write_only	119,575	147,578	1.23
types_delete_insert	68,469	128,568	1.88
oltp_read_write	173,041	376,663	2.18
Average			1.65

File-Backed (autocommit)

Each statement runs as its own transaction — exposes per-commit
fixed costs that the wrapped-in-BEGIN/COMMIT tests amortize away.
SQLite uses WAL mode with synchronous=FULL in this section so
the comparison uses SQLite's durable WAL autocommit path.

Reads

Reads have no commit cost; these are the same SQL files as the
File-Backed Reads section, included here for symmetry and to
catch any per-statement overhead doltlite pays on the read path.

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_point_select	68,586	134,063	1.95
oltp_range_select	19,440	57,365	2.95
oltp_sum_range	18,461	57,222	3.10
oltp_order_range	3,745	8,829	2.36
oltp_distinct_range	4,753	9,717	2.04
oltp_index_scan	9,539	23,313	2.44
select_random_points	31,124	93,181	2.99
select_random_ranges	8,096	19,994	2.47
covering_index_scan	9,260	14,623	1.58
groupby_scan	38,202	71,063	1.86
index_join	13,256	38,008	2.87
index_join_scan	8,310	27,898	3.36
types_table_scan	1,398,033	4,397,406	3.15
table_scan	1,693,154	5,232,886	3.09
oltp_read_only	187,615	398,975	2.13
Average			2.56

Writes

Test	SQLite (us)	Doltlite (us)	Multiplier
oltp_bulk_insert_ac	17,653	57,604	3.26
oltp_insert_ac	19,929	77,253	3.88
oltp_update_index_ac	20,657	101,192	4.90
oltp_update_non_index_ac	16,401	76,947	4.69
oltp_delete_insert_ac	18,359	88,015	4.79
oltp_write_only_ac	19,554	94,837	4.85
types_delete_insert_ac	16,368	78,689	4.81
oltp_read_write_ac	27,676	111,515	4.03
Average			4.40

100000 rows, median of 5 invocations per test, workload-only timing via host monotonic clock when available.

Performance Ceiling Check (6x individual, 5x average)

All tests within ceilings.