Skip to content

Commit b4527c7

Browse files
SockworkOrangeSockworkOrange
committed
fix: have benchmark in CI
1 parent 3132fab commit b4527c7

42 files changed

Lines changed: 603 additions & 468 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/benchmark.yml‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
name: Performance Benchmark
2+
defaults:
3+
run:
4+
shell: bash
5+
6+
env:
7+
DOTNET_VERSION: '8.0.x'
8+
9+
on:
10+
push:
11+
branches: [ "add-performance-benchmark-with-efcore" ]
12+
release:
13+
types: [ published ]
14+
workflow_dispatch:
15+
16+
jobs:
17+
benchmark-mysql:
18+
name: MySQL Benchmark
19+
runs-on: ubuntu-latest
20+
21+
steps:
22+
- uses: actions/checkout@v4
23+
24+
- name: Set up .NET
25+
uses: actions/setup-dotnet@v4
26+
with:
27+
dotnet-version: ${{ env.DOTNET_VERSION }}
28+
29+
- name: Load .env file
30+
uses: xom9ikk/dotenv@v2.3.0
31+
with:
32+
load-mode: strict
33+
34+
- name: Docker compose
35+
uses: hoverkraft-tech/compose-action@v1.5.1
36+
with:
37+
services: mysql
38+
39+
- name: Run Benchmark
40+
run: ./benchmark/scripts/run_benchmark.sh mysql
41+
42+
benchmark-postgresql:
43+
name: PostgreSQL Benchmark
44+
runs-on: ubuntu-latest
45+
46+
steps:
47+
- uses: actions/checkout@v4
48+
49+
- name: Set up .NET
50+
uses: actions/setup-dotnet@v4
51+
with:
52+
dotnet-version: ${{ env.DOTNET_VERSION }}
53+
54+
- name: Load .env file
55+
uses: xom9ikk/dotenv@v2.3.0
56+
with:
57+
load-mode: strict
58+
59+
- name: Docker compose
60+
uses: hoverkraft-tech/compose-action@v1.5.1
61+
with:
62+
services: postgresql
63+
64+
- name: Run Benchmark
65+
run: ./benchmark/scripts/run_benchmark.sh postgresql
66+
67+
benchmark-sqlite:
68+
name: SQLite Benchmark
69+
runs-on: ubuntu-latest
70+
71+
steps:
72+
- uses: actions/checkout@v4
73+
74+
- name: Set up .NET
75+
uses: actions/setup-dotnet@v4
76+
with:
77+
dotnet-version: ${{ env.DOTNET_VERSION }}
78+
79+
- name: Load .env file
80+
uses: xom9ikk/dotenv@v2.3.0
81+
with:
82+
load-mode: strict
83+
84+
- name: Run Benchmark
85+
run: ./benchmark/scripts/run_benchmark.sh sqlite

‎Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ $(WASM_PLUGIN_OUTPUT): $(WASM_SOURCES)
4747
./scripts/wasm/update_sha.sh sqlc.ci.yaml
4848

4949
run-benchmark: sqlc-generate
50-
./benchmark/scripts/run_benchmark.sh
50+
./benchmark/scripts/run_benchmark_all_dbs.sh
5151

5252
run-end2end-tests:
5353
./end2end/scripts/run_tests.sh

‎benchmark/BenchmarkDotNet.Artifacts/results/BenchmarkRunner.Benchmarks.MysqlReadBenchmark-report-github.md‎

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -4,22 +4,22 @@ BenchmarkDotNet v0.13.12, macOS 15.6.1 (24G90) [Darwin 24.6.0]
44
Apple M2, 1 CPU, 8 logical and 8 physical cores
55
.NET SDK 8.0.416
66
[Host] : .NET 8.0.22 (8.0.2225.52707), Arm64 RyuJIT AdvSIMD
7-
Job-KOIWQR : .NET 8.0.22 (8.0.2225.52707), Arm64 RyuJIT AdvSIMD
7+
Job-BMUQQB : .NET 8.0.22 (8.0.2225.52707), Arm64 RyuJIT AdvSIMD
88
9-
Runtime=.NET 8.0 InvocationCount=1 IterationCount=8
10-
UnrollFactor=1 WarmupCount=2 Categories=Read
9+
Runtime=.NET 8.0 InvocationCount=1 IterationCount=10
10+
UnrollFactor=1 WarmupCount=3 Categories=Read
1111
1212
```
13-
| Method | Limit | Mean | Error | StdDev | Median | Ratio | RatioSD | Gen0 | Allocated | Alloc Ratio |
14-
|-------------------------------------------- |------ |----------:|-----------:|-----------:|---------:|------:|--------:|----------:|----------:|------------:|
15-
| **'SQLC - GetCustomerOrders'** | **5000** | **10.467 ms** | **0.6702 ms** | **0.3505 ms** | **10.42 ms** | **1.00** | **0.00** | **-** | **2.73 MB** | **1.00** |
16-
| 'EFCore (NoTracking) - GetCustomerOrders' | 5000 | 9.912 ms | 0.5659 ms | 0.2960 ms | 10.01 ms | 0.95 | 0.02 | - | 3.01 MB | 1.11 |
17-
| 'EFCore (WithTracking) - GetCustomerOrders' | 5000 | 10.363 ms | 2.3849 ms | 1.2473 ms | 10.67 ms | 0.99 | 0.15 | - | 3 MB | 1.10 |
18-
| | | | | | | | | | | |
19-
| **'SQLC - GetCustomerOrders'** | **10000** | **19.610 ms** | **1.8452 ms** | **0.9651 ms** | **19.53 ms** | **1.00** | **0.00** | **-** | **5.41 MB** | **1.00** |
20-
| 'EFCore (NoTracking) - GetCustomerOrders' | 10000 | 15.459 ms | 1.2221 ms | 0.6392 ms | 15.53 ms | 0.79 | 0.06 | - | 5.87 MB | 1.08 |
21-
| 'EFCore (WithTracking) - GetCustomerOrders' | 10000 | 21.082 ms | 3.8873 ms | 2.0332 ms | 20.69 ms | 1.08 | 0.13 | - | 5.87 MB | 1.09 |
22-
| | | | | | | | | | | |
23-
| **'SQLC - GetCustomerOrders'** | **20000** | **38.896 ms** | **23.7991 ms** | **12.4474 ms** | **30.64 ms** | **1.00** | **0.00** | **-** | **10.95 MB** | **1.00** |
24-
| 'EFCore (NoTracking) - GetCustomerOrders' | 20000 | 33.703 ms | 2.6060 ms | 1.1571 ms | 33.35 ms | 0.97 | 0.24 | 1000.0000 | 11.59 MB | 1.06 |
25-
| 'EFCore (WithTracking) - GetCustomerOrders' | 20000 | 51.777 ms | 3.1317 ms | 1.3905 ms | 51.31 ms | 1.49 | 0.39 | 1000.0000 | 11.49 MB | 1.05 |
13+
| Method | Limit | Mean | Error | StdDev | Median | Ratio | RatioSD | Allocated | Alloc Ratio |
14+
|-------------------------------------------- |------ |---------:|----------:|----------:|---------:|------:|--------:|----------:|------------:|
15+
| **'SQLC - GetCustomerOrders'** | **5000** | **2.074 μs** | **1.3828 μs** | **0.8229 μs** | **1.792 μs** | **1.00** | **0.00** | **1872 B** | **1.00** |
16+
| 'EFCore (NoTracking) - GetCustomerOrders' | 5000 | 1.714 μs | 0.6764 μs | 0.3538 μs | 1.667 μs | 0.97 | 0.45 | 1792 B | 0.96 |
17+
| 'EFCore (WithTracking) - GetCustomerOrders' | 5000 | 1.891 μs | 1.3561 μs | 0.7092 μs | 1.688 μs | 1.01 | 0.43 | 1792 B | 0.96 |
18+
| | | | | | | | | | |
19+
| **'SQLC - GetCustomerOrders'** | **10000** | **1.895 μs** | **1.1294 μs** | **0.7470 μs** | **1.916 μs** | **1.00** | **0.00** | **1872 B** | **1.00** |
20+
| 'EFCore (NoTracking) - GetCustomerOrders' | 10000 | 2.720 μs | 2.3335 μs | 1.5435 μs | 2.042 μs | 1.76 | 1.36 | 1792 B | 0.96 |
21+
| 'EFCore (WithTracking) - GetCustomerOrders' | 10000 | 3.066 μs | 2.0087 μs | 1.3286 μs | 3.271 μs | 1.95 | 1.27 | 1792 B | 0.96 |
22+
| | | | | | | | | | |
23+
| **'SQLC - GetCustomerOrders'** | **20000** | **2.005 μs** | **1.1535 μs** | **0.7630 μs** | **1.771 μs** | **1.00** | **0.00** | **576 B** | **1.00** |
24+
| 'EFCore (NoTracking) - GetCustomerOrders' | 20000 | 1.719 μs | 0.8812 μs | 0.4609 μs | 1.708 μs | 0.95 | 0.43 | 1792 B | 3.11 |
25+
| 'EFCore (WithTracking) - GetCustomerOrders' | 20000 | 3.074 μs | 2.9087 μs | 1.7309 μs | 2.208 μs | 1.91 | 1.42 | 1792 B | 3.11 |

‎benchmark/BenchmarkDotNet.Artifacts/results/BenchmarkRunner.Benchmarks.MysqlReadBenchmark-report.csv‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
1-
Method,Job,AnalyzeLaunchVariance,EvaluateOverhead,MaxAbsoluteError,MaxRelativeError,MinInvokeCount,MinIterationTime,OutlierMode,Affinity,EnvironmentVariables,Jit,LargeAddressAware,Platform,PowerPlanMode,Runtime,AllowVeryLargeObjects,Concurrent,CpuGroups,Force,HeapAffinitizeMask,HeapCount,NoAffinitize,RetainVm,Server,Arguments,BuildConfiguration,Clock,EngineFactory,NuGetReferences,Toolchain,IsMutator,InvocationCount,IterationCount,IterationTime,LaunchCount,MaxIterationCount,MaxWarmupIterationCount,MemoryRandomization,MinIterationCount,MinWarmupIterationCount,RunStrategy,UnrollFactor,WarmupCount,Categories,Limit,Mean,Error,StdDev,Median,Ratio,RatioSD,Gen0,Allocated,Alloc Ratio
2-
'SQLC - GetCustomerOrders',Job-KOIWQR,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,8,Default,Default,Default,Default,Default,Default,Default,Default,1,2,Read,5000,10.467 ms,0.6702 ms,0.3505 ms,10.42 ms,1.00,0.00,0.0000,2.73 MB,1.00
3-
'EFCore (NoTracking) - GetCustomerOrders',Job-KOIWQR,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,8,Default,Default,Default,Default,Default,Default,Default,Default,1,2,Read,5000,9.912 ms,0.5659 ms,0.2960 ms,10.01 ms,0.95,0.02,0.0000,3.01 MB,1.11
4-
'EFCore (WithTracking) - GetCustomerOrders',Job-KOIWQR,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,8,Default,Default,Default,Default,Default,Default,Default,Default,1,2,Read,5000,10.363 ms,2.3849 ms,1.2473 ms,10.67 ms,0.99,0.15,0.0000,3 MB,1.10
5-
'SQLC - GetCustomerOrders',Job-KOIWQR,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,8,Default,Default,Default,Default,Default,Default,Default,Default,1,2,Read,10000,19.610 ms,1.8452 ms,0.9651 ms,19.53 ms,1.00,0.00,0.0000,5.41 MB,1.00
6-
'EFCore (NoTracking) - GetCustomerOrders',Job-KOIWQR,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,8,Default,Default,Default,Default,Default,Default,Default,Default,1,2,Read,10000,15.459 ms,1.2221 ms,0.6392 ms,15.53 ms,0.79,0.06,0.0000,5.87 MB,1.08
7-
'EFCore (WithTracking) - GetCustomerOrders',Job-KOIWQR,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,8,Default,Default,Default,Default,Default,Default,Default,Default,1,2,Read,10000,21.082 ms,3.8873 ms,2.0332 ms,20.69 ms,1.08,0.13,0.0000,5.87 MB,1.09
8-
'SQLC - GetCustomerOrders',Job-KOIWQR,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,8,Default,Default,Default,Default,Default,Default,Default,Default,1,2,Read,20000,38.896 ms,23.7991 ms,12.4474 ms,30.64 ms,1.00,0.00,0.0000,10.95 MB,1.00
9-
'EFCore (NoTracking) - GetCustomerOrders',Job-KOIWQR,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,8,Default,Default,Default,Default,Default,Default,Default,Default,1,2,Read,20000,33.703 ms,2.6060 ms,1.1571 ms,33.35 ms,0.97,0.24,1000.0000,11.59 MB,1.06
10-
'EFCore (WithTracking) - GetCustomerOrders',Job-KOIWQR,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,8,Default,Default,Default,Default,Default,Default,Default,Default,1,2,Read,20000,51.777 ms,3.1317 ms,1.3905 ms,51.31 ms,1.49,0.39,1000.0000,11.49 MB,1.05
1+
Method,Job,AnalyzeLaunchVariance,EvaluateOverhead,MaxAbsoluteError,MaxRelativeError,MinInvokeCount,MinIterationTime,OutlierMode,Affinity,EnvironmentVariables,Jit,LargeAddressAware,Platform,PowerPlanMode,Runtime,AllowVeryLargeObjects,Concurrent,CpuGroups,Force,HeapAffinitizeMask,HeapCount,NoAffinitize,RetainVm,Server,Arguments,BuildConfiguration,Clock,EngineFactory,NuGetReferences,Toolchain,IsMutator,InvocationCount,IterationCount,IterationTime,LaunchCount,MaxIterationCount,MaxWarmupIterationCount,MemoryRandomization,MinIterationCount,MinWarmupIterationCount,RunStrategy,UnrollFactor,WarmupCount,Categories,Limit,Mean,Error,StdDev,Median,Ratio,RatioSD,Allocated,Alloc Ratio
2+
'SQLC - GetCustomerOrders',Job-BMUQQB,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,10,Default,Default,Default,Default,Default,Default,Default,Default,1,3,Read,5000,2.074 μs,1.3828 μs,0.8229 μs,1.792 μs,1.00,0.00,1872 B,1.00
3+
'EFCore (NoTracking) - GetCustomerOrders',Job-BMUQQB,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,10,Default,Default,Default,Default,Default,Default,Default,Default,1,3,Read,5000,1.714 μs,0.6764 μs,0.3538 μs,1.667 μs,0.97,0.45,1792 B,0.96
4+
'EFCore (WithTracking) - GetCustomerOrders',Job-BMUQQB,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,10,Default,Default,Default,Default,Default,Default,Default,Default,1,3,Read,5000,1.891 μs,1.3561 μs,0.7092 μs,1.688 μs,1.01,0.43,1792 B,0.96
5+
'SQLC - GetCustomerOrders',Job-BMUQQB,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,10,Default,Default,Default,Default,Default,Default,Default,Default,1,3,Read,10000,1.895 μs,1.1294 μs,0.7470 μs,1.916 μs,1.00,0.00,1872 B,1.00
6+
'EFCore (NoTracking) - GetCustomerOrders',Job-BMUQQB,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,10,Default,Default,Default,Default,Default,Default,Default,Default,1,3,Read,10000,2.720 μs,2.3335 μs,1.5435 μs,2.042 μs,1.76,1.36,1792 B,0.96
7+
'EFCore (WithTracking) - GetCustomerOrders',Job-BMUQQB,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,10,Default,Default,Default,Default,Default,Default,Default,Default,1,3,Read,10000,3.066 μs,2.0087 μs,1.3286 μs,3.271 μs,1.95,1.27,1792 B,0.96
8+
'SQLC - GetCustomerOrders',Job-BMUQQB,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,10,Default,Default,Default,Default,Default,Default,Default,Default,1,3,Read,20000,2.005 μs,1.1535 μs,0.7630 μs,1.771 μs,1.00,0.00,576 B,1.00
9+
'EFCore (NoTracking) - GetCustomerOrders',Job-BMUQQB,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,10,Default,Default,Default,Default,Default,Default,Default,Default,1,3,Read,20000,1.719 μs,0.8812 μs,0.4609 μs,1.708 μs,0.95,0.43,1792 B,3.11
10+
'EFCore (WithTracking) - GetCustomerOrders',Job-BMUQQB,False,Default,Default,Default,Default,Default,Default,00000000,Empty,RyuJit,Default,Arm64,8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c,.NET 8.0,False,True,False,True,Default,Default,False,False,False,Default,Default,Default,Default,Default,Default,Default,1,10,Default,Default,Default,Default,Default,Default,Default,Default,1,3,Read,20000,3.074 μs,2.9087 μs,1.7309 μs,2.208 μs,1.91,1.42,1792 B,3.11

0 commit comments

Comments
 (0)