Skip to content

Commit 5f5f669

Browse files
KSemenenkoKSemenenko
committed
Refine benchmark workloads and graph search performance
1 parent 9f474c2 commit 5f5f669

19 files changed

Lines changed: 886 additions & 147 deletions

‎README.md‎

Lines changed: 89 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1312,45 +1312,102 @@ dotnet run --project benchmarks/MarkdownLd.Kb.Benchmarks -c Release -- --filter
13121312
dotnet run --project benchmarks/MarkdownLd.Kb.Benchmarks -c Release -- --filter "*GraphBuildBenchmarks*"
13131313
dotnet run --project benchmarks/MarkdownLd.Kb.Benchmarks -c Release -- --filter "*GraphSearchBenchmarks*"
13141314
dotnet run --project benchmarks/MarkdownLd.Kb.Benchmarks -c Release -- --filter "*TiktokenSearchBenchmarks*"
1315+
dotnet run --project benchmarks/MarkdownLd.Kb.Benchmarks -c Release -- --filter "*GraphPersistenceBenchmarks*"
1316+
dotnet run --project benchmarks/MarkdownLd.Kb.Benchmarks -c Release -- --filter "*GraphLifecycleSmokeBenchmarks*" --job Dry
13151317
MARKDOWN_LD_KB_BENCHMARK_PROFILE=cpu dotnet run --project benchmarks/MarkdownLd.Kb.Benchmarks -c Release -- --filter "*FuzzyEditDistanceBenchmarks*"
13161318
```
13171319

1318-
Benchmark reports are written to `artifacts/benchmarks/results` as Markdown, CSV, and JSON. The reports are intentionally ignored by git because they depend on the local machine and current system load. PR validation runs `FuzzyEditDistanceBenchmarks` as a mandatory smoke benchmark and uploads the reports as the `benchmark-smoke` artifact. The full benchmark workflow in `.github/workflows/benchmarks.yml` runs manually or on the weekly schedule and uploads the complete `benchmarkdotnet-results` artifact.
1320+
Benchmark reports are written to `artifacts/benchmarks/results` as Markdown, CSV, and full JSON. The reports are intentionally ignored by git because they depend on the local machine and current system load. PR validation runs `FuzzyEditDistanceBenchmarks` as a mandatory smoke benchmark and uploads the reports as the `benchmark-smoke` artifact. The full benchmark workflow in `.github/workflows/benchmarks.yml` runs manually or on the weekly schedule and uploads the complete `benchmarkdotnet-results` artifact. The benchmark config adds one default `ShortRun` job only when the command does not already pass `--job`, `--job=...`, or `-j`.
13191321

1320-
Latest local benchmark run, executed on May 3, 2026 with BenchmarkDotNet 0.15.8, .NET 10.0.5, ShortRun, Apple M2 Pro, exported these reports:
1322+
The exported BenchmarkDotNet reports include the diagnostic columns that matter for this library:
13211323

1322-
| Suite | Benchmarks executed | Export prefix |
1323-
| --- | ---: | --- |
1324-
| Fuzzy edit distance | 8 | `ManagedCode.MarkdownLd.Kb.Benchmarks.FuzzyEditDistanceBenchmarks-report` |
1325-
| Graph build | 3 | `ManagedCode.MarkdownLd.Kb.Benchmarks.GraphBuildBenchmarks-report` |
1326-
| Graph search | 54 | `ManagedCode.MarkdownLd.Kb.Benchmarks.GraphSearchBenchmarks-report` |
1327-
| Tiktoken search | 18 | `ManagedCode.MarkdownLd.Kb.Benchmarks.TiktokenSearchBenchmarks-report` |
1328-
1329-
Graph build scales over generated Markdown corpora like this:
1330-
1331-
| Documents | Mean | Allocated |
1332-
| ---: | ---: | ---: |
1333-
| 25 | 1.169 ms | 1.81 MB |
1334-
| 250 | 9.873 ms | 14.65 MB |
1335-
| 1000 | 70.672 ms | 57.94 MB |
1324+
| Area | Report data | Used for |
1325+
| --- | --- | --- |
1326+
| Latency | `Mean`, `Error`, `StdDev`, `Ratio`, `RatioSD`; full JSON also keeps min, quartiles, max, percentiles, and raw measurements | compare retrieval paths under the same generated workload |
1327+
| Allocation and GC | `Allocated`, `Alloc Ratio`, `Gen0`, `Gen1`, `Gen2` | find APIs that allocate enough to hurt repeated search calls |
1328+
| Threading | `Completed Work Items`, `Lock Contentions` | identify SPARQL and federation paths that schedule work or contend on locks |
1329+
| Repro metadata | runtime, JIT, platform, job, iteration counts, corpus profile, query scenario | keep local runs comparable without pretending they are machine-independent |
1330+
| Optional profiles | EventPipe `cpu`, `gc`, or `jit` artifacts when `MARKDOWN_LD_KB_BENCHMARK_PROFILE` is set | inspect hot methods after a suspicious benchmark result |
1331+
1332+
Benchmark workload profiles are named by shape instead of using unexplained document-count params:
1333+
1334+
| Profile | Shape |
1335+
| --- | --- |
1336+
| `ShortDocuments` | 250 compact runbook-like Markdown documents |
1337+
| `LongDocuments` | 80 long recovery playbooks with repeated sections |
1338+
| `LargeCorpus` | 1000 compact documents for scale, persistence, and build pressure |
1339+
| `TokenizedMultilingual` | 250 multilingual/CJK/token-heavy documents |
1340+
| `FederatedRunbooks` | 250 SPARQL/service/runbook documents for local federation paths |
1341+
1342+
Latest local benchmark run, executed on May 3, 2026 with BenchmarkDotNet 0.15.8, .NET 10.0.5, Apple M2 Pro, exported these reports:
1343+
1344+
| Suite | Job | Benchmarks executed | Export prefix |
1345+
| --- | --- | ---: | --- |
1346+
| Fuzzy edit distance | ShortRun | 8 | `ManagedCode.MarkdownLd.Kb.Benchmarks.FuzzyEditDistanceBenchmarks-report` |
1347+
| Graph build | ShortRun | 4 | `ManagedCode.MarkdownLd.Kb.Benchmarks.GraphBuildBenchmarks-report` |
1348+
| Graph search | ShortRun | 54 | `ManagedCode.MarkdownLd.Kb.Benchmarks.GraphSearchBenchmarks-report` |
1349+
| Tiktoken search | ShortRun | 12 | `ManagedCode.MarkdownLd.Kb.Benchmarks.TiktokenSearchBenchmarks-report` |
1350+
| Graph persistence | ShortRun | 39 | `ManagedCode.MarkdownLd.Kb.Benchmarks.GraphPersistenceBenchmarks-report` |
1351+
| Graph lifecycle smoke | Dry | 1 | `ManagedCode.MarkdownLd.Kb.Benchmarks.GraphLifecycleSmokeBenchmarks-report` |
1352+
1353+
Graph build:
1354+
1355+
| Profile | Mean | StdDev | Allocated |
1356+
| --- | ---: | ---: | ---: |
1357+
| `ShortDocuments` | 9.548 ms | 0.0298 ms | 14.70 MB |
1358+
| `LongDocuments` | 7.544 ms | 0.0149 ms | 14.35 MB |
1359+
| `LargeCorpus` | 59.453 ms | 12.7272 ms | 58.08 MB |
1360+
| `TokenizedMultilingual` | 12.433 ms | 0.0508 ms | 17.77 MB |
13361361

13371362
Graph search exact-query mean time:
13381363

1339-
| Documents | Ranked graph | BM25 | BM25 fuzzy | Focused | Schema SPARQL | Local federated |
1340-
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
1341-
| 25 | 0.111 ms | 0.197 ms | 0.234 ms | 0.191 ms | 5.799 ms | 8.769 ms |
1342-
| 250 | 1.225 ms | 2.251 ms | 2.663 ms | 2.103 ms | 59.994 ms | 65.124 ms |
1343-
| 1000 | 8.907 ms | 16.187 ms | 17.939 ms | 13.258 ms | 282.885 ms | 293.240 ms |
1344-
1345-
Typo-query search at 1000 generated documents measured 7.551 ms for ranked graph search, 12.044 ms for BM25, 20.340 ms for BM25 fuzzy, 12.700 ms for focused search, 258.953 ms for schema SPARQL, and 306.332 ms for local federated schema search. The fuzzy paths are opt-in and are expected to spend extra time to recover typo-heavy queries; they are not intended to beat exact lexical matching on raw speed.
1346-
1347-
Tiktoken token-distance search mean time:
1348-
1349-
| Documents | Exact query | Fuzzy-corrected exact | Typo query | Fuzzy-corrected typo | No-match query | Fuzzy-corrected no-match |
1350-
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
1351-
| 25 | 15.04 us | 15.94 us | 17.15 us | 24.99 us | 15.39 us | 18.60 us |
1352-
| 100 | 61.65 us | 57.72 us | 64.87 us | 75.74 us | 57.71 us | 64.71 us |
1353-
| 250 | 146.74 us | 146.27 us | 161.66 us | 184.64 us | 145.79 us | 153.14 us |
1364+
| Profile | Ranked graph | BM25 | BM25 fuzzy | Focused | Schema SPARQL | Local federated |
1365+
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
1366+
| `ShortDocuments` | 1.200 ms | 2.018 ms | 2.627 ms | 2.053 ms | 46.034 ms | 49.615 ms |
1367+
| `LongDocuments` | 0.480 ms | 3.577 ms | 3.574 ms | 0.642 ms | 12.819 ms | 14.561 ms |
1368+
| `FederatedRunbooks` | 1.334 ms | 2.723 ms | 2.720 ms | 2.271 ms | 45.981 ms | 55.269 ms |
1369+
1370+
Graph search exact-query allocated memory per operation:
1371+
1372+
| Profile | Ranked graph | BM25 | BM25 fuzzy | Focused | Schema SPARQL | Local federated |
1373+
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
1374+
| `ShortDocuments` | 2.37 MB | 4.83 MB | 7.22 MB | 3.27 MB | 60.34 MB | 62.33 MB |
1375+
| `LongDocuments` | 1.91 MB | 10.67 MB | 10.67 MB | 1.21 MB | 20.22 MB | 22.21 MB |
1376+
| `FederatedRunbooks` | 2.54 MB | 6.80 MB | 6.80 MB | 3.48 MB | 60.75 MB | 62.61 MB |
1377+
1378+
The `ShortDocuments` exact-query diagnostic slice shows the current hot paths:
1379+
1380+
| Method | Mean | Allocated | Alloc ratio | Gen0 | Gen1 | Gen2 | Work items | Lock contentions |
1381+
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
1382+
| Ranked graph | 1.200 ms | 2.37 MB | 1.00x | 296.8750 | 107.4219 | 0 | 0 | 0 |
1383+
| BM25 | 2.018 ms | 4.83 MB | 2.04x | 601.5625 | 210.9375 | 0 | 0 | 0 |
1384+
| BM25 fuzzy | 2.627 ms | 7.22 MB | 3.04x | 902.3438 | 230.4688 | 0 | 0 | 0 |
1385+
| Focused | 2.053 ms | 3.27 MB | 1.38x | 406.2500 | 179.6875 | 0 | 0 | 0 |
1386+
| Schema SPARQL | 46.034 ms | 60.34 MB | 25.44x | 8500.0000 | 1833.3333 | 500.0000 | 551 | 325 |
1387+
| Local federated | 49.615 ms | 62.33 MB | 26.27x | 8666.6667 | 2166.6667 | 500.0000 | 552 | 315.1667 |
1388+
1389+
Allocation and GC columns come directly from BenchmarkDotNet diagnosers. Treat the ratios and relative pressure inside the same run as the useful signal; ShortRun is a fast diagnostic pass, not a release-grade SLA measurement.
1390+
1391+
Persistence and export on the `LargeCorpus` profile:
1392+
1393+
| Method | Mean | StdDev | Allocated |
1394+
| --- | ---: | ---: | ---: |
1395+
| `CreateSnapshot` | 4.527 ms | 0.008 ms | 5.31 MB |
1396+
| `SerializeTurtle` | 9.203 ms | 0.088 ms | 18.07 MB |
1397+
| `SerializeJsonLd` | 13.157 ms | 0.086 ms | 20.31 MB |
1398+
| `SaveTurtleToFile` | 29.853 ms | 0.122 ms | 34.74 MB |
1399+
| `SaveJsonLdToFile` | 38.144 ms | 1.436 ms | 37.02 MB |
1400+
| `LoadTurtleFromFile` | 35.983 ms | 0.373 ms | 28.10 MB |
1401+
| `LoadJsonLdFromFile` | 99.980 ms | 2.262 ms | 75.32 MB |
1402+
1403+
Tiktoken token-distance search:
1404+
1405+
| Profile | Query | Exact | Fuzzy-corrected | Exact allocated | Fuzzy allocated |
1406+
| --- | --- | ---: | ---: | ---: | ---: |
1407+
| `LongDocuments` | Exact | 955.1 us | 952.7 us | 2.38 MB | 2.38 MB |
1408+
| `LongDocuments` | Typo | 1.112 ms | 1.291 ms | 2.78 MB | 3.73 MB |
1409+
| `TokenizedMultilingual` | Exact | 680.8 us | 690.5 us | 1.81 MB | 1.81 MB |
1410+
| `TokenizedMultilingual` | Typo | 811.3 us | 861.2 us | 1.81 MB | 1.82 MB |
13541411

13551412
Fuzzy edit-distance mean time:
13561413

@@ -1361,4 +1418,4 @@ Fuzzy edit-distance mean time:
13611418
| Long insertion | 21.980 ns | 7,990.146 ns | 363.53x | 0 B | 640 B |
13621419
| Long no-match | 70.283 ns | 8,990.700 ns | 127.92x | 328 B | 672 B |
13631420

1364-
These numbers are local measurements, not a cross-machine performance contract. The full Markdown, CSV, and JSON BenchmarkDotNet reports remain the source for detailed diagnostics.
1421+
These numbers are local measurements, not a cross-machine performance contract. The README keeps compact slices only; [Performance Benchmarks](docs/Features/PerformanceBenchmarks.md) and the full Markdown, CSV, and JSON BenchmarkDotNet reports remain the source for detailed diagnostics.

‎benchmarks/MarkdownLd.Kb.Benchmarks/AGENTS.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,9 @@ Purpose: BenchmarkDotNet performance suite for Markdown-LD Knowledge Bank.
2121
## Commands
2222

2323
- build: `(cd ../.. && dotnet build MarkdownLd.Kb.slnx --no-restore)`
24-
- smoke: `(cd ../.. && dotnet run --project benchmarks/MarkdownLd.Kb.Benchmarks -c Release -- --filter "*FuzzyEditDistanceBenchmarks*" --dry)`
24+
- smoke: `(cd ../.. && dotnet run --project benchmarks/MarkdownLd.Kb.Benchmarks -c Release -- --filter "*GraphLifecycleSmokeBenchmarks*" --job Dry)`
2525
- quick: `(cd ../.. && dotnet run --project benchmarks/MarkdownLd.Kb.Benchmarks -c Release -- --filter "*GraphSearchBenchmarks*")`
26+
- persistence: `(cd ../.. && dotnet run --project benchmarks/MarkdownLd.Kb.Benchmarks -c Release -- --filter "*GraphPersistenceBenchmarks*")`
2627
- profile: `(cd ../.. && MARKDOWN_LD_KB_BENCHMARK_PROFILE=cpu dotnet run --project benchmarks/MarkdownLd.Kb.Benchmarks -c Release -- --filter "*FuzzyEditDistanceBenchmarks*")`
2728

2829
## Applicable Skills

‎benchmarks/MarkdownLd.Kb.Benchmarks/BenchmarkCategories.cs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,15 @@ internal static class BenchmarkCategories
44
{
55
internal const string Algorithm = "algorithm";
66
internal const string Build = "build";
7+
internal const string Export = "export";
78
internal const string Federation = "federation";
89
internal const string Fuzzy = "fuzzy";
910
internal const string Graph = "graph";
11+
internal const string Load = "load";
12+
internal const string Persistence = "persistence";
13+
internal const string Save = "save";
1014
internal const string Search = "search";
15+
internal const string Serialization = "serialization";
16+
internal const string Smoke = "smoke";
1117
internal const string Tiktoken = "tiktoken";
1218
}

‎benchmarks/MarkdownLd.Kb.Benchmarks/BenchmarkCorpusFactory.cs‎

Lines changed: 23 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -6,22 +6,21 @@ internal static class BenchmarkCorpusFactory
66
{
77
private const string BaseUriText = "https://bench.example/";
88
private const string LocalFederatedEndpointText = "https://bench.example/sparql/local";
9-
private const string CacheTitlePrefix = "Cache restore runbook";
10-
private const string BillingTitlePrefix = "Billing export guide";
11-
private const string ReleaseTitlePrefix = "Release evidence checklist";
129
private const string CacheQuery = "cache restore manifest evidence";
10+
private const string LongQuery = "incident escalation recovery dependency timeline checkpoint";
11+
private const string TokenizedQuery = "cache restore manifest token 実行 evidence";
12+
private const string FederatedQuery = "federated sparql service binding runbook evidence";
1313
private const string TypoQuery = "cach restre manifst evidnce";
14+
private const string LongTypoQuery = "incidnt escalaton recovry depndency chekpoint";
15+
private const string TokenizedTypoQuery = "cach restore manifst tokne 実行 evidnce";
16+
private const string FederatedTypoQuery = "federatd sparq servce bindng evidnce";
1417
private const string NoMatchQuery = "satellite coffee roasting";
1518
private static readonly Uri BaseUri = new(BaseUriText);
1619
private static readonly Uri LocalFederatedEndpoint = new(LocalFederatedEndpointText);
1720

18-
public static MarkdownSourceDocument[] CreateSources(int documentCount)
21+
public static MarkdownSourceDocument[] CreateSources(BenchmarkCorpusProfile profile)
1922
{
20-
return Enumerable.Range(0, documentCount)
21-
.Select(index => new MarkdownSourceDocument(
22-
$"content/bench/doc-{index:D5}.md",
23-
CreateMarkdown(index)))
24-
.ToArray();
23+
return BenchmarkMarkdownCorpus.CreateSources(profile);
2524
}
2625

2726
public static MarkdownKnowledgeBuildResult BuildNone(
@@ -46,13 +45,13 @@ public static MarkdownKnowledgeBuildResult BuildTiktoken(
4645
return pipeline.BuildAsync(sources).GetAwaiter().GetResult();
4746
}
4847

49-
public static string GetQuery(BenchmarkQueryScenario scenario)
48+
public static string GetQuery(BenchmarkCorpusProfile profile, BenchmarkQueryScenario scenario)
5049
{
5150
return scenario switch
5251
{
53-
BenchmarkQueryScenario.Typo => TypoQuery,
52+
BenchmarkQueryScenario.Typo => GetTypoQuery(profile),
5453
BenchmarkQueryScenario.NoMatch => NoMatchQuery,
55-
_ => CacheQuery,
54+
_ => GetExactQuery(profile),
5655
};
5756
}
5857

@@ -95,54 +94,25 @@ public static FederatedSparqlExecutionOptions CreateFederatedOptions(KnowledgeGr
9594
};
9695
}
9796

98-
private static string CreateMarkdown(int index)
99-
{
100-
var family = index % 3;
101-
var title = CreateTitle(family, index);
102-
var topic = CreateTopic(family);
103-
var body = CreateBody(family, index);
104-
return $$"""
105-
---
106-
title: {{title}}
107-
summary: {{topic}} summary for benchmark document {{index}}.
108-
tags:
109-
- benchmark
110-
- {{topic}}
111-
---
112-
# {{title}}
113-
114-
{{body}}
115-
""";
116-
}
117-
118-
private static string CreateTitle(int family, int index)
97+
private static string GetExactQuery(BenchmarkCorpusProfile profile)
11998
{
120-
return family switch
99+
return profile switch
121100
{
122-
1 => $"{BillingTitlePrefix} {index:D5}",
123-
2 => $"{ReleaseTitlePrefix} {index:D5}",
124-
_ => $"{CacheTitlePrefix} {index:D5}",
125-
};
126-
}
127-
128-
private static string CreateTopic(int family)
129-
{
130-
return family switch
131-
{
132-
1 => "billing invoice export payment checkpoint",
133-
2 => "release gate approval evidence checklist",
134-
_ => "cache restore manifest rollback evidence",
101+
BenchmarkCorpusProfile.LongDocuments => LongQuery,
102+
BenchmarkCorpusProfile.TokenizedMultilingual => TokenizedQuery,
103+
BenchmarkCorpusProfile.FederatedRunbooks => FederatedQuery,
104+
_ => CacheQuery,
135105
};
136106
}
137107

138-
private static string CreateBody(int family, int index)
108+
private static string GetTypoQuery(BenchmarkCorpusProfile profile)
139109
{
140-
var identifier = $"validationfingerprintcheckpointtoken{index:D5}manifestwindowrollbackevidence";
141-
return family switch
110+
return profile switch
142111
{
143-
1 => $"Billing export verifies invoice payment checkpoint evidence with marker {identifier}.",
144-
2 => $"Release evidence checklist confirms approval gates and deployment notes with marker {identifier}.",
145-
_ => $"Cache restore validates manifest rollback evidence and runbook recovery with marker {identifier}.",
112+
BenchmarkCorpusProfile.LongDocuments => LongTypoQuery,
113+
BenchmarkCorpusProfile.TokenizedMultilingual => TokenizedTypoQuery,
114+
BenchmarkCorpusProfile.FederatedRunbooks => FederatedTypoQuery,
115+
_ => TypoQuery,
146116
};
147117
}
148118
}

‎benchmarks/MarkdownLd.Kb.Benchmarks/BenchmarkCorpusProfile.cs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
namespace ManagedCode.MarkdownLd.Kb.Benchmarks;
2+
3+
public enum BenchmarkCorpusProfile
4+
{
5+
ShortDocuments,
6+
LongDocuments,
7+
LargeCorpus,
8+
TokenizedMultilingual,
9+
FederatedRunbooks,
10+
}

0 commit comments

Comments
 (0)