Skip to content

Commit 323961b

Browse files
twccleggtwcclegg
authored
Merge pull request #351 from twcclegg/pertTestSpeed
reduce performance tests scope for speed
2 parents d8f91e7 + 5b8c653 commit 323961b

9 files changed

Lines changed: 86 additions & 90 deletions

‎.github/workflows/run_performance_tests_windows.yml‎

Lines changed: 59 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,16 @@ name: run_performance_tests_windows
33
on:
44
pull_request:
55
branches: [ "main" ]
6+
# Pushes to main refresh the cached benchmark baseline that PRs compare against,
7+
# so we never recompute main's numbers on every PR run.
8+
push:
9+
branches: [ "main" ]
10+
# GitHub evicts any cache not accessed within 7 days (not configurable). This repo
11+
# can idle longer than that, so re-measure and re-save the main baseline twice a week
12+
# to keep it warm — and to self-heal if it was ever evicted. Sun + Wed keeps the
13+
# largest gap to ~4 days, comfortably under the 7-day window.
14+
schedule:
15+
- cron: "0 7 * * 0,3"
616

717
concurrency:
818
group: ${{ github.workflow }}-${{ github.ref }}
@@ -13,32 +23,58 @@ jobs:
1323
runs-on: windows-latest
1424
timeout-minutes: 30
1525
steps:
16-
- name: Checkout PR / push branch
17-
uses: actions/checkout@v6
18-
with:
19-
path: branch-code
20-
21-
- name: Checkout main branch (for comparison)
22-
if: github.event_name == 'pull_request'
26+
- name: Checkout
2327
uses: actions/checkout@v6
24-
with:
25-
ref: main
26-
path: main-code
2728

2829
- name: Setup .NET
2930
uses: actions/setup-dotnet@v5
3031
with:
3132
dotnet-version: 10.x
3233

33-
- name: Run benchmarks (PR branch)
34-
run: dotnet run -c Release --framework net10.0 -- --filter "*"
35-
working-directory: ./branch-code/csharp/PhoneNumbers.PerformanceTest
36-
37-
- name: Run benchmarks (main branch)
34+
# On a PR, restore the most recent main baseline produced by a push-to-main run
35+
# instead of re-benchmarking main here. The prefix restore-key returns the latest
36+
# available baseline regardless of the exact main commit it was measured on.
37+
- name: Restore main benchmark baseline
3838
if: github.event_name == 'pull_request'
39+
uses: actions/cache/restore@v4
40+
with:
41+
path: main-baseline
42+
# actions/cache/restore requires `key`, but we never expect an exact hit:
43+
# save keys carry a unique -<run_id> suffix, so this is a deliberate near-miss
44+
# and the prefix restore-key below always supplies the most recently created baseline.
45+
key: benchmark-main-
46+
restore-keys: |
47+
benchmark-main-
48+
49+
- name: Run benchmarks
3950
run: dotnet run -c Release --framework net10.0 -- --filter "*"
40-
working-directory: ./main-code/csharp/PhoneNumbers.PerformanceTest
51+
working-directory: ./csharp/PhoneNumbers.PerformanceTest
52+
53+
# On push to main (or the keep-warm schedule), publish this run's results as the
54+
# new baseline for future PRs.
55+
- name: Stage main baseline
56+
if: github.event_name != 'pull_request'
57+
shell: pwsh
58+
run: |
59+
New-Item -ItemType Directory -Force -Path "main-baseline" | Out-Null
60+
$resultsDir = "csharp/PhoneNumbers.PerformanceTest/BenchmarkDotNet.Artifacts/results"
61+
if (Test-Path $resultsDir) {
62+
Copy-Item "$resultsDir/*-report-github.md" "main-baseline/" -ErrorAction SilentlyContinue
63+
}
64+
65+
- name: Save main benchmark baseline
66+
if: github.event_name != 'pull_request'
67+
uses: actions/cache/save@v4
68+
with:
69+
path: main-baseline
70+
# Unique per run so every push/cron writes a *fresh* cache entry. cache/save
71+
# cannot overwrite an existing key, and the keep-warm cron never restores the
72+
# baseline, so reusing benchmark-main-<sha> would let the timer lapse when main
73+
# is unchanged. The PR prefix restore-key picks the most recently created entry.
74+
key: benchmark-main-${{ github.sha }}-${{ github.run_id }}
4175

76+
# On a PR, bundle the freshly-measured branch results with the restored main
77+
# baseline so the follow-up workflow can post a side-by-side comparison.
4278
- name: Stage benchmark artifact
4379
if: github.event_name == 'pull_request'
4480
shell: pwsh
@@ -47,11 +83,13 @@ jobs:
4783
New-Item -ItemType Directory -Force -Path "$stage/branch-results" | Out-Null
4884
New-Item -ItemType Directory -Force -Path "$stage/main-results" | Out-Null
4985
50-
$prDir = "branch-code/csharp/PhoneNumbers.PerformanceTest/BenchmarkDotNet.Artifacts/results"
51-
$mainDir = "main-code/csharp/PhoneNumbers.PerformanceTest/BenchmarkDotNet.Artifacts/results"
52-
53-
if (Test-Path $prDir) { Copy-Item "$prDir/*-report-github.md" "$stage/branch-results/" -ErrorAction SilentlyContinue }
54-
if (Test-Path $mainDir) { Copy-Item "$mainDir/*-report-github.md" "$stage/main-results/" -ErrorAction SilentlyContinue }
86+
$prDir = "csharp/PhoneNumbers.PerformanceTest/BenchmarkDotNet.Artifacts/results"
87+
if (Test-Path $prDir) {
88+
Copy-Item "$prDir/*-report-github.md" "$stage/branch-results/" -ErrorAction SilentlyContinue
89+
}
90+
if (Test-Path "main-baseline") {
91+
Copy-Item "main-baseline/*-report-github.md" "$stage/main-results/" -ErrorAction SilentlyContinue
92+
}
5593
5694
# PR metadata for the follow-up workflow (use head SHA, not the merge SHA in GITHUB_SHA)
5795
@{

‎CLAUDE.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,8 @@ The library tracks upstream metadata releases (~every two weeks) via the `create
1010

1111
## Repository layout
1212

13-
- `csharp/PhoneNumbers/` — main library (NuGet `libphonenumber-csharp`). Multi-targets `netstandard2.0;net8.0;net9.0`. `<TreatWarningsAsErrors>true</TreatWarningsAsErrors>` is set, so warnings break the build.
14-
- `csharp/PhoneNumbers.Test/` — xUnit tests, ported from the Java tests. Multi-targets `netframework4.8;net8.0;net9.0`.
13+
- `csharp/PhoneNumbers/` — main library (NuGet `libphonenumber-csharp`). Multi-targets `netstandard2.0;net8.0;net9.0;net10.0`. `<TreatWarningsAsErrors>true</TreatWarningsAsErrors>` is set, so warnings break the build.
14+
- `csharp/PhoneNumbers.Test/` — xUnit tests, ported from the Java tests. Multi-targets `netframework4.8;net8.0;net9.0;net10.0`.
1515
- `csharp/PhoneNumbers.Extensions/` — separate NuGet (`libphonenumber-csharp.extensions`) with C#-idiomatic helpers that don't exist in the Java library.
1616
- `csharp/PhoneNumbers.PerformanceTest/` — BenchmarkDotNet harness.
1717
- `csharp/PhoneNumbers.MetadataBuilder/` — build-time tool that converts XML metadata + geocoding/timezone text files into per-region binary files. Source-links a small set of files from `PhoneNumbers/` so it doesn't depend on (and can't cycle with) the main library at build time.
@@ -58,8 +58,8 @@ Benchmarks:
5858

5959
```bash
6060
cd csharp/PhoneNumbers.PerformanceTest
61-
dotnet run -c Release --framework net9.0 -- --filter "*"
62-
dotnet run -c Release --framework net9.0 -- --filter "*PhoneNumberWorkflowBenchmark*"
61+
dotnet run -c Release --framework net10.0 -- --filter "*"
62+
dotnet run -c Release --framework net10.0 -- --filter "*PhoneNumberWorkflowBenchmark*"
6363
```
6464

6565
## Architecture notes that span files

‎csharp/PhoneNumbers.PerformanceTest/Benchmarks/AsYouTypeFormatterBenchmark.cs‎

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -4,20 +4,13 @@
44
namespace PhoneNumbers.PerformanceTest.Benchmarks
55
{
66
[MemoryDiagnoser]
7-
[SimpleJob(RuntimeMoniker.Net48)]
8-
[SimpleJob(RuntimeMoniker.Net80)]
9-
[SimpleJob(RuntimeMoniker.Net90)]
7+
[SimpleJob(RuntimeMoniker.Net10_0)]
108
public class AsYouTypeFormatterBenchmark
119
{
12-
#if NETFRAMEWORK
13-
private PhoneNumberUtil _phoneNumberUtil = null;
14-
private PhoneNumberBenchmarkCase[] _phoneNumbers = null;
15-
#else
1610
private PhoneNumberUtil _phoneNumberUtil = null!;
1711
private PhoneNumberBenchmarkCase[] _phoneNumbers = null!;
18-
#endif
1912

20-
[Params(1000, 10000)]
13+
[Params(1000)]
2114
public int PhoneNumberCount { get; set; }
2215

2316
[GlobalSetup]

‎csharp/PhoneNumbers.PerformanceTest/Benchmarks/ColdStartBenchmark.cs‎

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -10,21 +10,14 @@ namespace PhoneNumbers.PerformanceTest.Benchmarks
1010
/// use of the library, before any region metadata has been loaded.
1111
/// </summary>
1212
[MemoryDiagnoser]
13-
[SimpleJob(RunStrategy.ColdStart, RuntimeMoniker.Net48, launchCount: 1, warmupCount: 1, iterationCount: 20, invocationCount: 1)]
14-
[SimpleJob(RunStrategy.ColdStart, RuntimeMoniker.Net80, launchCount: 1, warmupCount: 1, iterationCount: 20, invocationCount: 1)]
15-
[SimpleJob(RunStrategy.ColdStart, RuntimeMoniker.Net90, launchCount: 1, warmupCount: 1, iterationCount: 20, invocationCount: 1)]
13+
[SimpleJob(RunStrategy.ColdStart, RuntimeMoniker.Net10_0, launchCount: 1, warmupCount: 1, iterationCount: 20, invocationCount: 1)]
1614
public class ColdStartBenchmark
1715
{
1816
// The country-code-to-region map and one fresh PhoneNumberUtil are kept around so the
1917
// FirstRegionLookup benchmark has a pre-constructed util whose region cache has NOT been
2018
// touched for the target region (we pick a region we never look up during setup).
21-
#if NETFRAMEWORK
22-
private PhoneNumberUtil _warmInstance = null;
23-
private string[] _supportedRegions = null;
24-
#else
2519
private PhoneNumberUtil _warmInstance = null!;
2620
private string[] _supportedRegions = null!;
27-
#endif
2821

2922
// Region selected for FirstRegionLookup. Chosen as a small-but-real region so its metadata
3023
// payload size is representative of the average region rather than an outlier like US/CN.

‎csharp/PhoneNumbers.PerformanceTest/Benchmarks/ParsingHelpersBenchmark.cs‎

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -4,20 +4,13 @@
44
namespace PhoneNumbers.PerformanceTest.Benchmarks
55
{
66
[MemoryDiagnoser]
7-
[SimpleJob(RuntimeMoniker.Net48)]
8-
[SimpleJob(RuntimeMoniker.Net80)]
9-
[SimpleJob(RuntimeMoniker.Net90)]
7+
[SimpleJob(RuntimeMoniker.Net10_0)]
108
public class ParsingHelpersBenchmark
119
{
12-
#if NETFRAMEWORK
13-
private string[] _inputs = null;
14-
private string[] _inputsWithLeadingJunk = null;
15-
#else
1610
private string[] _inputs = null!;
1711
private string[] _inputsWithLeadingJunk = null!;
18-
#endif
1912

20-
[Params(1000, 10000)]
13+
[Params(1000)]
2114
public int PhoneNumberCount { get; set; }
2215

2316
[GlobalSetup]

‎csharp/PhoneNumbers.PerformanceTest/Benchmarks/PhoneNumberMatcherBenchmark.cs‎

Lines changed: 2 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -5,26 +5,18 @@
55
namespace PhoneNumbers.PerformanceTest.Benchmarks
66
{
77
[MemoryDiagnoser]
8-
[SimpleJob(RuntimeMoniker.Net48)]
9-
[SimpleJob(RuntimeMoniker.Net80)]
10-
[SimpleJob(RuntimeMoniker.Net90)]
8+
[SimpleJob(RuntimeMoniker.Net10_0)]
119
public class PhoneNumberMatcherBenchmark
1210
{
1311
// Filler text interleaved between embedded numbers so the matcher has to skip non-number
1412
// content. Kept short to keep total input length proportional to PhoneNumberCount.
1513
private const string Filler = " Lorem ipsum dolor sit amet, consectetur adipiscing elit. Call ";
1614

17-
#if NETFRAMEWORK
18-
private PhoneNumberUtil _phoneNumberUtil = null;
19-
private string _defaultRegion = null;
20-
private string _text = null;
21-
#else
2215
private PhoneNumberUtil _phoneNumberUtil = null!;
2316
private string _defaultRegion = null!;
2417
private string _text = null!;
25-
#endif
2618

27-
[Params(100, 1000)]
19+
[Params(100)]
2820
public int PhoneNumberCount { get; set; }
2921

3022
[GlobalSetup]

‎csharp/PhoneNumbers.PerformanceTest/Benchmarks/PhoneNumberWorkflowBenchmark.cs‎

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -4,20 +4,13 @@
44
namespace PhoneNumbers.PerformanceTest.Benchmarks
55
{
66
[MemoryDiagnoser]
7-
[SimpleJob(RuntimeMoniker.Net48)]
8-
[SimpleJob(RuntimeMoniker.Net80)]
9-
[SimpleJob(RuntimeMoniker.Net90)]
7+
[SimpleJob(RuntimeMoniker.Net10_0)]
108
public class PhoneNumberWorkflowBenchmark
119
{
12-
#if NETFRAMEWORK
13-
private PhoneNumberUtil _phoneNumberUtil = null;
14-
private PhoneNumberBenchmarkCase[] _phoneNumbers = null;
15-
#else
1610
private PhoneNumberUtil _phoneNumberUtil = null!;
1711
private PhoneNumberBenchmarkCase[] _phoneNumbers = null!;
18-
#endif
1912

20-
[Params(1000, 10000, 100000)]
13+
[Params(1000, 10000)]
2114
public int PhoneNumberCount { get; set; }
2215

2316
[GlobalSetup]

‎csharp/PhoneNumbers.PerformanceTest/PhoneNumbers.PerformanceTest.csproj‎

Lines changed: 4 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,18 @@
1-
<Project Sdk="Microsoft.NET.Sdk">
1+
<Project Sdk="Microsoft.NET.Sdk">
22

33
<PropertyGroup>
44
<OutputType>Exe</OutputType>
5-
<TargetFrameworks>netframework4.8;net8.0;net9.0;net10.0</TargetFrameworks>
5+
<!-- Host TFM only. The benchmark runtime matrix is driven by the [SimpleJob]
6+
attributes on each benchmark class, which target net10.0. -->
7+
<TargetFramework>net10.0</TargetFramework>
68
<ImplicitUsings>disable</ImplicitUsings>
7-
</PropertyGroup>
8-
9-
<PropertyGroup Condition="'$(TargetFramework)' == 'net8.0' Or '$(TargetFramework)' == 'net9.0' Or '$(TargetFramework)' == 'net10.0'">
109
<Nullable>enable</Nullable>
1110
</PropertyGroup>
1211

1312
<ItemGroup>
1413
<PackageReference Include="BenchmarkDotNet" Version="0.15.8" />
1514
</ItemGroup>
1615

17-
<ItemGroup Condition=" '$(TargetFramework)' == 'net48' ">
18-
<PackageReference Include="System.Memory" Version="4.5.4" />
19-
</ItemGroup>
20-
2116
<ItemGroup>
2217
<ProjectReference Include="..\PhoneNumbers\PhoneNumbers.csproj" />
2318
</ItemGroup>

‎csharp/PhoneNumbers.PerformanceTest/README.md‎

Lines changed: 10 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -4,25 +4,25 @@ See [Github Actions](https://github.com/twcclegg/libphonenumber-csharp/actions/w
44

55
## Running locally
66

7-
Install the .NET SDKs or runtimes needed by the benchmark jobs you want to run. The benchmark
8-
project is launched with .NET 9, while BenchmarkDotNet can execute the configured .NET Framework
9-
4.8, .NET 6, and .NET 8 jobs when those runtimes are available locally.
7+
Install the .NET 10 SDK. The benchmark project targets `net10.0` only, and every benchmark
8+
is configured with a single `net10.0` job (`[SimpleJob(RuntimeMoniker.Net10_0)]`), so .NET 10
9+
is the only runtime required.
1010

1111
```powershell
1212
cd csharp/PhoneNumbers.PerformanceTest
13-
dotnet run -c Release --framework net9.0 -- --filter "*"
13+
dotnet run -c Release --framework net10.0 -- --filter "*"
1414
```
1515

1616
BenchmarkDotNet writes detailed reports to `BenchmarkDotNet.Artifacts/results`.
1717

1818
To run only the phone number workflow benchmark, pass a filter after `--`:
1919

2020
```powershell
21-
dotnet run -c Release --framework net9.0 -- --filter "*PhoneNumberWorkflowBenchmark*"
21+
dotnet run -c Release --framework net10.0 -- --filter "*PhoneNumberWorkflowBenchmark*"
2222
```
2323

24-
The full benchmark includes the `100000` phone-number data set and may take several minutes,
25-
especially when multiple runtime jobs are available on the machine.
24+
The `PhoneNumberWorkflowBenchmark` runs the largest data sets (up to `10000` numbers) and is
25+
the slowest; the full suite still completes in a few minutes on a single runtime.
2626

2727
Other available benchmarks:
2828

@@ -38,14 +38,13 @@ Other available benchmarks:
3838
`invocationCount: 1` so each measurement is a genuine first-use, not a steady-state loop.
3939

4040
The benchmark data is generated from valid example numbers in the bundled metadata and expanded
41-
deterministically to the configured `PhoneNumberCount` values, up to 100,000 inputs. Each benchmark
41+
deterministically to the configured `PhoneNumberCount` values, up to 10,000 inputs. Each benchmark
4242
iteration parses, validates, and formats every number in that data set.
4343

4444
Below you can see a sample of what the results might look like
4545

4646
| Method | PhoneNumberCount | Job | Runtime | Mean | Error | StdDev | Gen0 | Allocated |
4747
|------------------------------------ |-----------------:|------------------- |------------------- |---------:|----------:|----------:|--------:|----------:|
48-
| ParseValidateAndFormatPhoneNumbers | 1000 | .NET 8.0 | .NET 8.0 | 1.25 ms | 0.018 ms | 0.017 ms | 31.2500 | 512 KB |
49-
| ParseValidateAndFormatPhoneNumbers | 10000 | .NET 8.0 | .NET 8.0 | 12.46 ms | 0.231 ms | 0.216 ms | 312.500 | 5120 KB |
50-
| ParseValidateAndFormatPhoneNumbers | 100000 | .NET 8.0 | .NET 8.0 | 125.1 ms | 2.48 ms | 2.32 ms | 3125.00 | 51200 KB |
48+
| ParseValidateAndFormatPhoneNumbers | 1000 | .NET 10.0 | .NET 10.0 | 1.25 ms | 0.018 ms | 0.017 ms | 31.2500 | 512 KB |
49+
| ParseValidateAndFormatPhoneNumbers | 10000 | .NET 10.0 | .NET 10.0 | 12.46 ms | 0.231 ms | 0.216 ms | 312.500 | 5120 KB |
5150

0 commit comments

Comments
 (0)