REQUEST: Increase the number of bare metal machine for benchmark testing

[XSAM]

The number of bare-metal machines (only one) seems insufficient to run the benchmark task on time. Most of the benchmark tasks are waiting for the machine in this PR for more than 8 hours.

---

Regarding the machine size, we don't need 64 CPUs to run the benchmark. A 4 CPU machine is good enough for the benchmark of https://github.com/open-telemetry/opentelemetry-go.

If reducing the machine size (if available) could enable more bare-metal machines, it would be a good trade-off.

[trask]

Unfortunately, this is the smallest bare metal runner size already.

We can probably add a second one.

First though it would be good to check that we're using the one we have efficiently:

cc @open-telemetry/go-approvers @open-telemetry/collector-contrib-approvers @open-telemetry/arrow-approvers @open-telemetry/java-approvers

Last 30 days

Repository	Workflow	Runs	Total	Avg/run	Max run	Max queue wait
opentelemetry-go	benchmark.yml	102	301h 42m	2h 57m	4h 18m	23h 58m
opentelemetry-collector-contrib	load-tests.yml	68	67h 37m	59m 39s	1h 24m	33h 56m
otel-arrow	pipeline-perf-test-nightly.yml	60	54h 13m	54m 13s	1h 22m	2h 28m
otel-arrow	pipeline-perf-test-continuous.yml	160	25h 9m	9m 25s	17m 50s	6h 20m
opentelemetry-java	benchmark.yml	79	17h 53m	13m 35s	20m 13s	4h 27m
opentelemetry-js	benchmark.yml	61	3h 28m	3m 24s	10m 5s	6h 31m
opentelemetry-rust	benchmark.yml	30	2h 14m	4m 29s	14m 13s	11h 29m
opentelemetry-python	benchmarks.yml	53	1h 12m	1m 21s	1m 39s	7h 55m
otel-arrow	pipeline-perf-test-manual-pr.yaml	1	14m 8s	14m 8s	14m 8s	4h 40m
otel-arrow	pipeline-perf-on-label.yaml	2	14s	7s	7s	35s

[cijothomas]

OTel-Go's avg run duration is ~3hrs. If we can run a small subset on every commit, and move the rest to nightly only, that'd free up some bandwidth I think. (I have probably mentioned this to Tyler, and was asked to raise in OTel-Go channel, which I haven't done yet)

[cijothomas]

otel-arrow is finishing it's phase 2, and we are actively adding more perf tests, so we'll be increasingly using up the perf machine. (And the 64 core, 2 socket machine is precisely we need, esp. for testing NUMA awareness etc.)

[XSAM]

@cijothomas

If we can run a small subset on every commit, and move the rest to nightly only, that'd free up some bandwidth I think.

I am verifying the reliability with a new method (codspeed) to run benchmarks (thus will run full benchmarks) in OTel-Go. If it can pass, I can change it to partial benchmarks. This will drop our usage significantly.

[cijothomas]

Thanks @XSAM I believe we can wait for that before deciding to add second runner (I assume they are very expensive boxes!)

[jack-berg]

The java benchmarks will take longer than reported. I recently merged a PR to increase the sample size and sample time parameters because the variance was unacceptably high. Just realized today looking at this that they haven't finished since merging that because there's a timeout of 20 minutes. Upping that here: open-telemetry/opentelemetry-java#8358

Some things to think about for other maintainers:

• Which benchmarks need to be run? In java we have many benchmarks but only run a few as part of CI.
• What is the smallest set of test cases and sample size / sample time parameters you can get away with to get the signal you need within tolerable variance?
• Do we need to run on every commit to main or can we can away with running daily? What about filtering out commits like dependency updates?

[martincostello]

Just adding a comment here to say I'm looking into adding benchmarks for opentelemetry-dotnet and opentelemetry-dotnet-contrib, so we'd be adding to the contention if we onboarded to the same runner.

I don't know the total runtime for the every possible benchmark we have in both repos, but from experience trying to run them locally I think it's Quite A While™️ so (at least initially) it would likely be a subset of "golden scenarios" (yet to be identified) to not go beyond an hour for either repo (so I guess 2 hours max total, which is then 1/12th of a day thinking about it...). Of course a 64 core machine vs. my local laptop, they'd probably run much faster 🚀

[jack-berg]

Of course a 64 core machine vs. my local laptop, they'd probably run much faster

Not if the benchmark harness is time based rather than operation based!

[cijothomas]

(at least initially) it would likely be a subset of "golden scenarios" (yet to be identified) to not go beyond an hour for either repo (so I guess 2 hours max total,

#3402 (comment)
Avg duration taken by benches in language implementation is ~10 mins only (Collector/Arrow takes a lot naturally!).
Lets try to find a golden set in .NET and keep it ~10 min max.

Machine spec:

CPU(s):                                  128
On-line CPU(s) list:                     0-127
Model name:                              Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz
Thread(s) per core:                      2
Core(s) per socket:                      32
Socket(s):                               2
CPU(s) scaling MHz:                      24%
NUMA node(s):                            2
NUMA node0 CPU(s):                       0-31,64-95
NUMA node1 CPU(s):                       32-63,96-127

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
node 0 size: 515623 MB
node 0 free: 194469 MB
node 1 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127
node 1 size: 516067 MB
node 1 free: 161933 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 

[trask]

just wanted to check in here, how are things going with queue time for bare metal benchmark runs? I'm thinking to go ahead and create a second one, though still great to optimize what we have so far since I'm sure the project's benchmarking needs will only grow

[martincostello]

I haven't started using it for .NET yet - will be looking to do an initial pass week of the 1st June.