Skip to content

Fix release benchmark row count title#1037

Merged
timsehn merged 1 commit into
masterfrom
fix/release-benchmark-title
May 22, 2026
Merged

Fix release benchmark row count title#1037
timsehn merged 1 commit into
masterfrom
fix/release-benchmark-title

Conversation

@timsehn
Copy link
Copy Markdown
Collaborator

@timsehn timsehn commented May 22, 2026

Summary

  • make the release benchmark explicitly run with BENCH_ROWS=100000
  • update the release notes benchmark heading to the int-keys naming and 100K rows
  • raise the release benchmark timeout to match the normal benchmark workflow

Tests

  • ruby -e 'require "yaml"; YAML.load_file(".github/workflows/release.yml"); puts "release.yml parsed"'

@github-actions
Copy link
Copy Markdown

Sysbench-Style Benchmark: Doltlite vs SQLite

In-Memory

Reads

Test SQLite (us) Doltlite (us) Multiplier
oltp_point_select 24,283 34,689 1.43
oltp_range_select 10,761 14,146 1.31
oltp_sum_range 9,604 13,467 1.40
oltp_order_range 2,698 3,209 1.19
oltp_distinct_range 3,850 4,253 1.10
oltp_index_scan 3,912 6,293 1.61
select_random_points 10,350 17,249 1.67
select_random_ranges 3,095 5,299 1.71
covering_index_scan 4,260 4,441 1.04
groupby_scan 33,040 36,646 1.11
index_join 5,912 8,754 1.48
index_join_scan 3,356 5,463 1.63
types_table_scan 1,105,099 1,393,594 1.26
table_scan 1,264,050 1,604,864 1.27
oltp_read_only 106,087 134,519 1.27
Average 1.37

Writes

Test SQLite (us) Doltlite (us) Multiplier
oltp_bulk_insert 178,825 238,968 1.34
oltp_insert 15,915 27,873 1.75
oltp_update_index 51,463 103,315 2.01
oltp_update_non_index 35,567 68,486 1.93
oltp_delete_insert 44,947 78,225 1.74
oltp_write_only 22,554 50,175 2.22
types_delete_insert 24,948 45,156 1.81
oltp_read_write 64,281 119,682 1.86
Average 1.83

File-Backed

Reads

Test SQLite (us) Doltlite (us) Multiplier
oltp_point_select 115,364 113,173 0.98
oltp_range_select 21,048 39,084 1.86
oltp_sum_range 20,541 38,419 1.87
oltp_order_range 3,742 6,616 1.77
oltp_distinct_range 4,758 7,632 1.60
oltp_index_scan 13,622 18,715 1.37
select_random_points 27,308 71,272 2.61
select_random_ranges 12,197 16,590 1.36
covering_index_scan 12,511 8,544 0.68
groupby_scan 36,092 55,415 1.54
index_join 10,737 18,163 1.69
index_join_scan 4,607 13,403 2.91
types_table_scan 1,279,651 2,966,822 2.32
table_scan 1,529,187 3,831,590 2.51
oltp_read_only 240,976 305,221 1.27
Average 1.76

Writes

Test SQLite (us) Doltlite (us) Multiplier
oltp_bulk_insert 193,271 257,481 1.33
oltp_insert 21,545 39,022 1.81
oltp_update_index 130,310 243,720 1.87
oltp_update_non_index 81,141 160,921 1.98
oltp_delete_insert 92,273 170,378 1.85
oltp_write_only 56,408 113,165 2.01
types_delete_insert 49,796 90,852 1.82
oltp_read_write 119,829 282,678 2.36
Average 1.88

File-Backed (autocommit)

Each statement runs as its own transaction — exposes per-commit
fixed costs that the wrapped-in-BEGIN/COMMIT tests amortize away.
SQLite uses WAL mode with synchronous=FULL in this section so
the comparison uses SQLite's durable WAL autocommit path.

Reads

Reads have no commit cost; these are the same SQL files as the
File-Backed Reads section, included here for symmetry and to
catch any per-statement overhead doltlite pays on the read path.

Test SQLite (us) Doltlite (us) Multiplier
oltp_point_select 60,888 114,327 1.88
oltp_range_select 15,463 39,015 2.52
oltp_sum_range 15,126 38,438 2.54
oltp_order_range 3,304 6,683 2.02
oltp_distinct_range 4,312 7,633 1.77
oltp_index_scan 8,326 18,997 2.28
select_random_points 21,484 71,809 3.34
select_random_ranges 6,864 16,572 2.41
covering_index_scan 7,363 8,438 1.15
groupby_scan 35,283 55,600 1.58
index_join 8,038 18,094 2.25
index_join_scan 4,093 13,420 3.28
types_table_scan 1,282,380 2,980,586 2.32
table_scan 1,502,170 3,778,739 2.52
oltp_read_only 162,813 300,211 1.84
Average 2.25

Writes

Test SQLite (us) Doltlite (us) Multiplier
oltp_bulk_insert_ac 14,962 57,465 3.84
oltp_insert_ac 17,635 75,165 4.26
oltp_update_index_ac 19,467 88,712 4.56
oltp_update_non_index_ac 17,369 71,472 4.11
oltp_delete_insert_ac 18,369 83,023 4.52
oltp_write_only_ac 18,595 81,673 4.39
types_delete_insert_ac 16,331 70,762 4.33
oltp_read_write_ac 23,349 94,514 4.05
Average 4.26

100000 rows, median of 5 invocations per test, workload-only timing via host monotonic clock when available.

Performance Ceiling Check (6x individual, 5x average)

All tests within ceilings.

@github-actions
Copy link
Copy Markdown

Sysbench-Style Benchmark (composite PK): Doltlite vs SQLite

Companion to the classic Sysbench-Style Benchmark. Every workload here
runs against tables with a 2-column INTEGER PRIMARY KEY(a, b) WITHOUT ROWID.

Individual ratios gated at 6×; section averages gated at 5×.

In-Memory

Reads

Test SQLite (us) Doltlite (us) Multiplier
oltp_point_select 31,794 46,969 1.48
oltp_range_select 18,823 26,260 1.40
oltp_sum_range 17,703 25,714 1.45
oltp_order_range 3,517 4,459 1.27
oltp_distinct_range 4,550 5,530 1.22
oltp_index_scan 4,402 7,566 1.72
select_random_points 28,459 40,244 1.41
select_random_ranges 7,533 9,994 1.33
covering_index_scan 4,114 4,778 1.16
groupby_scan 36,790 43,104 1.17
index_join 7,949 12,528 1.58
index_join_scan 4,024 6,985 1.74
types_table_scan 1,055,511 1,496,839 1.42
table_scan 1,182,682 1,655,471 1.40
oltp_read_only 145,724 197,027 1.35
Average 1.41

Writes

Test SQLite (us) Doltlite (us) Multiplier
oltp_bulk_insert 239,675 325,088 1.36
oltp_insert 18,701 33,342 1.78
oltp_update_index 65,399 124,205 1.90
oltp_update_non_index 49,583 85,229 1.72
oltp_delete_insert 48,405 96,453 1.99
oltp_write_only 25,663 56,885 2.22
types_delete_insert 31,536 58,664 1.86
oltp_read_write 99,709 171,443 1.72
Average 1.82

File-Backed

Reads

Test SQLite (us) Doltlite (us) Multiplier
oltp_point_select 107,689 116,904 1.09
oltp_range_select 28,110 49,597 1.76
oltp_sum_range 27,683 49,027 1.77
oltp_order_range 4,495 7,641 1.70
oltp_distinct_range 5,631 8,739 1.55
oltp_index_scan 12,772 19,310 1.51
select_random_points 44,047 89,194 2.02
select_random_ranges 15,832 20,537 1.30
covering_index_scan 11,240 9,954 0.89
groupby_scan 40,032 60,516 1.51
index_join 13,106 25,828 1.97
index_join_scan 5,196 16,192 3.12
types_table_scan 1,219,395 3,007,331 2.47
table_scan 1,436,305 3,687,464 2.57
oltp_read_only 267,390 359,466 1.34
Average 1.77

Writes

Test SQLite (us) Doltlite (us) Multiplier
oltp_bulk_insert 251,340 350,406 1.39
oltp_insert 29,575 48,507 1.64
oltp_update_index 153,751 255,589 1.66
oltp_update_non_index 107,567 169,591 1.58
oltp_delete_insert 110,364 188,502 1.71
oltp_write_only 75,517 120,649 1.60
types_delete_insert 61,665 104,011 1.69
oltp_read_write 171,674 326,875 1.90
Average 1.65

File-Backed (autocommit)

Each statement runs as its own transaction — exposes per-commit
fixed costs that the wrapped-in-BEGIN/COMMIT tests amortize away.
SQLite uses WAL mode with synchronous=FULL in this section so
the comparison uses SQLite's durable WAL autocommit path.

Reads

Reads have no commit cost; these are the same SQL files as the
File-Backed Reads section, included here for symmetry and to
catch any per-statement overhead doltlite pays on the read path.

Test SQLite (us) Doltlite (us) Multiplier
oltp_point_select 62,466 116,769 1.87
oltp_range_select 23,184 49,762 2.15
oltp_sum_range 23,076 49,329 2.14
oltp_order_range 4,038 7,596 1.88
oltp_distinct_range 5,157 8,813 1.71
oltp_index_scan 8,108 19,049 2.35
select_random_points 38,038 89,426 2.35
select_random_ranges 10,884 20,280 1.86
covering_index_scan 6,679 9,889 1.48
groupby_scan 39,454 60,619 1.54
index_join 10,530 25,738 2.44
index_join_scan 4,645 15,894 3.42
types_table_scan 1,206,711 3,009,726 2.49
table_scan 1,420,059 3,680,414 2.59
oltp_read_only 202,404 361,322 1.79
Average 2.14

Writes

Test SQLite (us) Doltlite (us) Multiplier
oltp_bulk_insert_ac 31,162 118,733 3.81
oltp_insert_ac 38,335 133,010 3.47
oltp_update_index_ac 42,714 131,125 3.07
oltp_update_non_index_ac 30,984 125,333 4.05
oltp_delete_insert_ac 34,306 134,518 3.92
oltp_write_only_ac 37,171 138,802 3.73
types_delete_insert_ac 32,507 126,005 3.88
oltp_read_write_ac 45,251 153,610 3.39
Average 3.67

100000 rows, median of 5 invocations per test, workload-only timing via host monotonic clock when available.

Performance Ceiling Check (6x individual, 5x average)

All tests within ceilings.

@github-actions
Copy link
Copy Markdown

Sysbench-Style Benchmark (TEXT PK): Doltlite vs SQLite

Companion to the classic Sysbench-Style Benchmark. Every workload here
runs against tables with a 32-char hex TEXT PRIMARY KEY (UUID-shaped).

Individual ratios gated at 6×; section averages gated at 5×.

In-Memory

Reads

Test SQLite (us) Doltlite (us) Multiplier
oltp_point_select 29,328 45,920 1.57
oltp_range_select 13,202 20,450 1.55
oltp_sum_range 11,720 19,743 1.68
oltp_order_range 2,948 3,872 1.31
oltp_distinct_range 3,979 4,956 1.25
oltp_index_scan 4,406 7,679 1.74
select_random_points 17,350 29,011 1.67
select_random_ranges 3,979 6,531 1.64
covering_index_scan 4,385 5,282 1.20
groupby_scan 32,194 39,034 1.21
index_join 6,761 12,540 1.85
index_join_scan 4,406 7,705 1.75
types_table_scan 1,070,844 1,610,100 1.50
table_scan 1,257,806 1,767,894 1.41
oltp_read_only 118,959 172,239 1.45
Average 1.52

Writes

Test SQLite (us) Doltlite (us) Multiplier
oltp_bulk_insert 226,332 338,150 1.49
oltp_insert 21,491 38,683 1.80
oltp_update_index 71,657 144,778 2.02
oltp_update_non_index 47,993 92,939 1.94
oltp_delete_insert 50,826 108,078 2.13
oltp_write_only 27,907 63,611 2.28
types_delete_insert 32,305 60,455 1.87
oltp_read_write 84,328 164,894 1.96
Average 1.94

File-Backed

Reads

Test SQLite (us) Doltlite (us) Multiplier
oltp_point_select 104,774 122,281 1.17
oltp_range_select 22,478 52,155 2.32
oltp_sum_range 21,720 51,860 2.39
oltp_order_range 3,923 8,295 2.11
oltp_distinct_range 4,976 9,216 1.85
oltp_index_scan 13,026 21,310 1.64
select_random_points 33,968 84,692 2.49
select_random_ranges 11,478 17,683 1.54
covering_index_scan 12,242 13,162 1.08
groupby_scan 35,612 64,624 1.81
index_join 14,320 34,409 2.40
index_join_scan 7,846 24,424 3.11
types_table_scan 1,292,951 3,859,445 2.98
table_scan 1,569,095 4,583,803 2.92
oltp_read_only 239,931 373,817 1.56
Average 2.09

Writes

Test SQLite (us) Doltlite (us) Multiplier
oltp_bulk_insert 244,959 368,240 1.50
oltp_insert 48,063 58,758 1.22
oltp_update_index 188,451 326,586 1.73
oltp_update_non_index 133,650 195,439 1.46
oltp_delete_insert 129,472 216,974 1.68
oltp_write_only 90,882 137,933 1.52
types_delete_insert 67,780 120,735 1.78
oltp_read_write 189,998 354,570 1.87
Average 1.60

File-Backed (autocommit)

Each statement runs as its own transaction — exposes per-commit
fixed costs that the wrapped-in-BEGIN/COMMIT tests amortize away.
SQLite uses WAL mode with synchronous=FULL in this section so
the comparison uses SQLite's durable WAL autocommit path.

Reads

Reads have no commit cost; these are the same SQL files as the
File-Backed Reads section, included here for symmetry and to
catch any per-statement overhead doltlite pays on the read path.

Test SQLite (us) Doltlite (us) Multiplier
oltp_point_select 61,654 124,731 2.02
oltp_range_select 18,403 52,146 2.83
oltp_sum_range 17,379 52,313 3.01
oltp_order_range 3,603 8,279 2.30
oltp_distinct_range 4,690 9,338 1.99
oltp_index_scan 8,634 21,287 2.47
select_random_points 28,844 85,733 2.97
select_random_ranges 7,121 17,914 2.52
covering_index_scan 8,477 13,404 1.58
groupby_scan 35,136 64,921 1.85
index_join 12,415 34,609 2.79
index_join_scan 7,754 24,785 3.20
types_table_scan 1,309,076 3,895,964 2.98
table_scan 1,531,668 4,562,216 2.98
oltp_read_only 175,967 375,787 2.14
Average 2.51

Writes

Test SQLite (us) Doltlite (us) Multiplier
oltp_bulk_insert_ac 22,876 81,344 3.56
oltp_insert_ac 25,396 100,517 3.96
oltp_update_index_ac 27,227 122,073 4.48
oltp_update_non_index_ac 24,436 104,029 4.26
oltp_delete_insert_ac 24,787 109,627 4.42
oltp_write_only_ac 25,685 105,059 4.09
types_delete_insert_ac 22,085 92,354 4.18
oltp_read_write_ac 32,104 119,003 3.71
Average 4.08

100000 rows, median of 5 invocations per test, workload-only timing via host monotonic clock when available.

Performance Ceiling Check (6x individual, 5x average)

All tests within ceilings.

@github-actions
Copy link
Copy Markdown

Sysbench-Style Benchmark (BLOB PK): Doltlite vs SQLite

Companion to the classic Sysbench-Style Benchmark. Every workload here
runs against tables with a 16-byte big-endian BLOB PRIMARY KEY.

Individual ratios gated at 6×; section averages gated at 5×.

In-Memory

Reads

Test SQLite (us) Doltlite (us) Multiplier
oltp_point_select 30,591 42,070 1.38
oltp_range_select 13,276 18,896 1.42
oltp_sum_range 12,164 18,348 1.51
oltp_order_range 3,137 3,871 1.23
oltp_distinct_range 4,222 4,840 1.15
oltp_index_scan 4,541 7,511 1.65
select_random_points 18,796 26,490 1.41
select_random_ranges 4,272 6,731 1.58
covering_index_scan 4,481 5,289 1.18
groupby_scan 34,722 41,019 1.18
index_join 6,808 11,977 1.76
index_join_scan 4,158 7,685 1.85
types_table_scan 1,124,821 1,597,580 1.42
table_scan 1,423,275 1,870,291 1.31
oltp_read_only 121,040 159,632 1.32
Average 1.42

Writes

Test SQLite (us) Doltlite (us) Multiplier
oltp_bulk_insert 243,122 321,948 1.32
oltp_insert 20,552 38,395 1.87
oltp_update_index 69,973 142,536 2.04
oltp_update_non_index 50,191 90,426 1.80
oltp_delete_insert 50,213 107,760 2.15
oltp_write_only 28,614 65,787 2.30
types_delete_insert 33,789 59,419 1.76
oltp_read_write 90,343 159,943 1.77
Average 1.88

File-Backed

Reads

Test SQLite (us) Doltlite (us) Multiplier
oltp_point_select 123,606 130,605 1.06
oltp_range_select 24,587 53,527 2.18
oltp_sum_range 23,421 52,895 2.26
oltp_order_range 4,179 8,375 2.00
oltp_distinct_range 5,224 9,309 1.78
oltp_index_scan 14,646 22,707 1.55
select_random_points 35,335 91,076 2.58
select_random_ranges 13,422 19,720 1.47
covering_index_scan 13,974 14,374 1.03
groupby_scan 37,979 67,585 1.78
index_join 14,447 35,966 2.49
index_join_scan 7,150 26,201 3.66
types_table_scan 1,344,762 4,004,316 2.98
table_scan 1,662,774 4,828,380 2.90
oltp_read_only 261,590 384,510 1.47
Average 2.08

Writes

Test SQLite (us) Doltlite (us) Multiplier
oltp_bulk_insert 256,557 348,965 1.36
oltp_insert 36,684 60,353 1.65
oltp_update_index 180,910 349,081 1.93
oltp_update_non_index 112,159 200,788 1.79
oltp_delete_insert 119,246 227,362 1.91
oltp_write_only 71,005 144,273 2.03
types_delete_insert 63,497 120,749 1.90
oltp_read_write 158,086 367,324 2.32
Average 1.86

File-Backed (autocommit)

Each statement runs as its own transaction — exposes per-commit
fixed costs that the wrapped-in-BEGIN/COMMIT tests amortize away.
SQLite uses WAL mode with synchronous=FULL in this section so
the comparison uses SQLite's durable WAL autocommit path.

Reads

Reads have no commit cost; these are the same SQL files as the
File-Backed Reads section, included here for symmetry and to
catch any per-statement overhead doltlite pays on the read path.

Test SQLite (us) Doltlite (us) Multiplier
oltp_point_select 68,622 129,829 1.89
oltp_range_select 18,840 53,895 2.86
oltp_sum_range 18,126 53,386 2.95
oltp_order_range 3,749 8,529 2.28
oltp_distinct_range 4,779 9,444 1.98
oltp_index_scan 9,467 23,450 2.48
select_random_points 30,741 90,280 2.94
select_random_ranges 7,984 19,495 2.44
covering_index_scan 8,230 14,247 1.73
groupby_scan 37,577 68,221 1.82
index_join 11,863 36,092 3.04
index_join_scan 6,739 26,204 3.89
types_table_scan 1,385,925 3,997,396 2.88
table_scan 1,646,225 4,913,912 2.98
oltp_read_only 184,616 379,736 2.06
Average 2.55

Writes

Test SQLite (us) Doltlite (us) Multiplier
oltp_bulk_insert_ac 15,421 62,044 4.02
oltp_insert_ac 17,911 87,363 4.88
oltp_update_index_ac 21,908 102,127 4.66
oltp_update_non_index_ac 18,273 82,143 4.50
oltp_delete_insert_ac 18,122 91,673 5.06
oltp_write_only_ac 18,840 88,964 4.72
types_delete_insert_ac 15,214 77,935 5.12
oltp_read_write_ac 25,312 104,899 4.14
Average 4.64

100000 rows, median of 5 invocations per test, workload-only timing via host monotonic clock when available.

Performance Ceiling Check (6x individual, 5x average)

All tests within ceilings.

@timsehn timsehn merged commit 81a679b into master May 22, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant