Skip to content

Commit 03d39a6

Browse files
committed
Document the medium benchmark input
I've removed the comment header so the file is the exact output of `helm template`.
1 parent 7b5177e commit 03d39a6

2 files changed

Lines changed: 46 additions & 12 deletions

File tree

benches/README.md

Lines changed: 46 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ cargo bench json
4040
The argument to `cargo bench` is a substring match against the full benchmark
4141
names of the form `{size}_{format}/{source}`.
4242

43-
- **size**: `small` (see below)
43+
- **size**: `small` or `medium` (see below)
4444
- **format**: A full format name as given to xt's `-f` or `-t` (e.g. `json`)
4545
- **source**: `buffer` (non-streaming) or `reader` (streaming)
4646

@@ -50,22 +50,58 @@ benchmark run, including charts and comparisons with any previous run.
5050

5151
## Test Inputs
5252

53+
Each benchmark loads test data into an in-memory buffer by translating a
54+
"default" version of the input with xt. This approach limits the size of the xt
55+
repository and ensures that disk I/O performance doesn't influence the results.
56+
However, it allows changes to xt's output formatting (e.g. whitespace, quoting)
57+
to influence the results. I expect such changes to be rare, at least compared
58+
to other changes whose impact is worth benchmarking.
59+
60+
### Small
61+
5362
The small input, `k8s-job.yaml`, is a simple Kubernetes `Job` that runs the
5463
Docker `hello-world` image. Translation time is usually a few microseconds for
5564
even the slowest input formats, so each benchmark runs in just a few seconds.
5665
This provides relatively fast feedback as you work.
5766

67+
### Medium
68+
69+
The medium input, `k8s-kyverno.yaml`, is a full set of Kubernetes manifests for
70+
deploying [Kyverno][kyverno] v1.16.2, generated from version 3.6.2 of the
71+
official chart using Helm v4.1.0 on `darwin/arm64`:
72+
73+
```sh
74+
helm template kyverno kyverno/kyverno \
75+
--version 3.6.2 \
76+
--set admissionController.replicas=1 \
77+
--set backgroundController.replicas=1 \
78+
--set reportsController.replicas=1 \
79+
--set cleanupController.replicas=1 \
80+
--set webhooksCleanup.image.pullPolicy=IfNotPresent
81+
```
82+
83+
To ensure TOML compatibility:
84+
85+
1. The above `--set` options were chosen to eliminate all `null` values.
86+
2. The benchmark harness processes the raw Helm output by turning the stream of
87+
YAML documents into a single object, with a single `manifests` field
88+
containing an array of the documents. It does this by creating a small
89+
MessagePack "header" to set up the object structure and type-length marker
90+
for an array, then translating the YAML documents with xt. It then
91+
translates the complete object to the final format for benchmarking.
92+
93+
The strategy for generating the medium input is intended to be reproducible and
94+
auditable. The size of the input was chosen to balance space requirements for
95+
an xt repository checkout with the desire to avoid non-human-readable encodings.
96+
97+
### Large (removed)
98+
5899
The benchmarks previously included a 20 - 30 MB large input based on a sample of
59100
GitHub events, which was included in the xt repository (and remains in its
60101
history) as a Zstandard compressed archive of MessagePack data. Based on the
61102
reveal of the xz-utils backdoor that was obfuscated in part as compressed test
62-
data, **I have chosen to temporarily eliminate the large benchmarks** until they
63-
are reimplemented to rely exclusively on human-readable inputs, ideally without
64-
bloating the size of xt repository checkouts.
103+
data, **I have chosen to eliminate the large benchmarks** until they are
104+
reimplemented to rely exclusively on human-readable inputs, ideally without
105+
bloating the size of xt repository checkouts too much.
65106

66-
Each benchmark loads test data into an in-memory buffer by translating a
67-
"default" version of the input with xt. This approach reduces the size of the xt
68-
repository and ensures that disk I/O performance does not influence the
69-
benchmark results. However, it allows changes to xt's output formatting
70-
(whitespace, quoting, etc.) to influence the results. I expect such changes to
71-
be rare, at least compared to other changes whose impact is worth benchmarking.
107+
[kyverno]: https://kyverno.io/

benches/k8s-kyverno.yaml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
1-
# helm template kyverno kyverno/kyverno --set admissionController.replicas=1 --set backgroundController.replicas=1 --set reportsController.replicas=1 --set cleanupController.replicas=1 --set webhooksCleanup.image.pullPolicy=IfNotPresent
2-
# Chart version 3.6.2; Kyverno v1.16.2
31
---
42
# Source: kyverno/templates/admission-controller/serviceaccount.yaml
53
apiVersion: v1

0 commit comments

Comments
 (0)