Skip to content

Commit c71f148

Browse files
dougqhdevflow.devflow-routing-intake
authored and
TophrC-dd
committed
Overhaul set benchmarks: split Immutable / SingleThreaded, add Set.copyOf (#11721)
Overhaul set benchmarks: split Immutable / SingleThreaded, add Set.copyOf Mirror the map-benchmark overhaul for sets. Replace the single SetBenchmark (shared mutable counter under @threads(8); contains_treeSet bug that queried HASH_SET) with two classes that each pick the right threading model: - ImmutableSetBenchmark: fixed read-only membership shared across threads (@State(Scope.Benchmark)); array / sortedArray / HashSet / TreeSet / Set.copyOf (the JDK compact SetN the agent actually uses for config sets, via CollectionUtils.tryMakeImmutableSet). hit/miss split, per-thread cursor. - SingleThreadedSetBenchmark: per-thread mutable lifecycle (@State(Scope.Thread)); create/clone + contains/iterate, plus a Collections.synchronizedSet case for the uncontended synchronization tax (per-thread => bias never revoked; biased-locking story across JVMs). StringIndex rows fold in later. Result blocks empty pending a fresh multi-JVM run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Add Java 17 results to set benchmark Javadocs ImmutableSetBenchmark: HashSet fastest; Set.copyOf (SetN) ~10% behind on hit, the compact form the agent uses for fixed config sets. SingleThreadedSetBenchmark: uncontended synchronizedSet tax ~37% on contains (biased locking off, Java 17), near-zero on iterate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Rename copyOf -> tracerImmutableSet in ImmutableSetBenchmark Per bric3 review: copyOf named the mechanism; tracerImmutableSet names the role (the agent's fixed-config-set representation, i.e. Set.copyOf / SetN). Prose keeps the Set#copyOf reference for the mechanism. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Fix clone_synchronizedSet to clone the synchronized set (not the plain one) Per Codex review: it copied `hashSet`, unlike the other clone_* methods which copy their own structure. Copy `synchronizedSet` so it faithfully measures cloning the synchronized variant. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Merge branch 'master' into dougqh/set-benchmark Co-authored-by: devflow.devflow-routing-intake <devflow.devflow-routing-intake@kubernetes.us1.ddbuild.io>
1 parent 958b528 commit c71f148

3 files changed

Lines changed: 385 additions & 128 deletions

File tree

‎internal-api/src/jmh/java/datadog/trace/util/ImmutableSetBenchmark.java‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
package datadog.trace.util;
2+
3+
import java.util.Arrays;
4+
import java.util.HashSet;
5+
import java.util.Set;
6+
import java.util.TreeSet;
7+
import org.openjdk.jmh.annotations.Benchmark;
8+
import org.openjdk.jmh.annotations.Fork;
9+
import org.openjdk.jmh.annotations.Level;
10+
import org.openjdk.jmh.annotations.Measurement;
11+
import org.openjdk.jmh.annotations.Scope;
12+
import org.openjdk.jmh.annotations.Setup;
13+
import org.openjdk.jmh.annotations.State;
14+
import org.openjdk.jmh.annotations.Threads;
15+
import org.openjdk.jmh.annotations.Warmup;
16+
17+
/**
18+
* Membership over a small, fixed, read-only string set shared across threads — split into hit and
19+
* miss lookups (different cost shapes per structure).
20+
*
21+
* <p>The set is built once and only read, so a single shared instance ({@link Scope#Benchmark})
22+
* read by all {@code @Threads} is realistic and contention-free. This is the read-mostly
23+
* counterpart to the per-thread mutable {@link SingleThreadedSetBenchmark}, and mirrors {@link
24+
* ImmutableMapBenchmark} on the set side. Sets in the tracer skew strongly toward this fixed,
25+
* read-only shape.
26+
*
27+
* <p>Strategies compared:
28+
*
29+
* <ul>
30+
* <li>{@code array} / {@code sortedArray} — linear scan / binary search; slow on miss.
31+
* <li>{@link HashSet} — idiomatic, fast; node-based, allocates per element.
32+
* <li>{@link TreeSet} — comparator-ordered; worth it only for a custom comparator, not speed.
33+
* <li>{@code tracerImmutableSet} — {@link java.util.Set#copyOf} (via {@link
34+
* CollectionUtils#tryMakeImmutableSet}), the JDK's compact, array-backed immutable set
35+
* ({@code ImmutableCollections.SetN}), which is what the agent actually uses for fixed config
36+
* sets. Java 10+; falls back to {@code HashSet} pre-10. The realistic baseline for any
37+
* flat/immutable set comparison.
38+
* </ul>
39+
*
40+
* <p>Lookups are interned (the {@code ==} fast path where a structure has one); misses are short
41+
* and never present.
42+
*
43+
* <p>Java 17 results (Apple M1, {@code @Fork(2)}, {@code @Threads(8)}; M ops/s = millions):
44+
*
45+
* <pre>{@code
46+
* Structure hit miss
47+
* hashSet 2159 1751 (fastest)
48+
* tracerImmutableSet 1946 1633 (Set.copyOf / SetN)
49+
* array 926 584
50+
* sortedArray 664 588
51+
* treeSet 642 593
52+
* }</pre>
53+
*
54+
* <p>Key findings:
55+
*
56+
* <ul>
57+
* <li>{@code HashSet} is fastest; {@link java.util.Set#copyOf} ({@code SetN}) trails by only ~10%
58+
* on hit and ~7% on miss — and it's the compact, array-backed form the agent already uses for
59+
* fixed config sets, so it's a strong default when the set is immutable.
60+
* <li>{@code array} / {@code sortedArray} / {@code treeSet} cluster at ~0.6–0.9B — they scan,
61+
* binary-search, or tree-walk per lookup, so they trail the hashed structures, most visibly
62+
* on the miss path.
63+
* </ul>
64+
*/
65+
@Fork(2)
66+
@Warmup(iterations = 2)
67+
@Measurement(iterations = 3)
68+
@Threads(8)
69+
@State(Scope.Benchmark)
70+
public class ImmutableSetBenchmark {
71+
static final String[] STRINGS = {
72+
"foo", "bar", "baz", "quux", "hello", "world",
73+
"service", "queryString", "lorem", "ipsum", "dolem", "sit"
74+
};
75+
76+
/** Distinct String instances that are never present, for the miss path. */
77+
static final String[] MISSES = newMisses();
78+
79+
static String[] newMisses() {
80+
String[] misses = new String[STRINGS.length * 4];
81+
for (int i = 0; i < misses.length; ++i) {
82+
misses[i] = "dne-" + i;
83+
}
84+
return misses;
85+
}
86+
87+
// Built once, never mutated -- safe to share across the reader threads.
88+
String[] array;
89+
String[] sortedArray;
90+
HashSet<String> hashSet;
91+
TreeSet<String> treeSet;
92+
Set<String> tracerImmutableSet;
93+
94+
@Setup(Level.Trial)
95+
public void setUp() {
96+
array = STRINGS;
97+
sortedArray = Arrays.copyOf(STRINGS, STRINGS.length);
98+
Arrays.sort(sortedArray);
99+
hashSet = new HashSet<>(Arrays.asList(STRINGS));
100+
treeSet = new TreeSet<>(Arrays.asList(STRINGS));
101+
tracerImmutableSet = CollectionUtils.tryMakeImmutableSet(Arrays.asList(STRINGS));
102+
}
103+
104+
/** Per-thread lookup cursor so each reader thread cycles keys independently. */
105+
@State(Scope.Thread)
106+
public static class Cursor {
107+
int hitIndex = 0;
108+
int missIndex = 0;
109+
110+
String nextHit() {
111+
int i = hitIndex + 1;
112+
if (i >= STRINGS.length) {
113+
i = 0;
114+
}
115+
hitIndex = i;
116+
return STRINGS[i];
117+
}
118+
119+
String nextMiss() {
120+
int i = missIndex + 1;
121+
if (i >= MISSES.length) {
122+
i = 0;
123+
}
124+
missIndex = i;
125+
return MISSES[i];
126+
}
127+
}
128+
129+
static boolean arrayContains(String[] array, String needle) {
130+
for (String s : array) {
131+
if (needle.equals(s)) {
132+
return true;
133+
}
134+
}
135+
return false;
136+
}
137+
138+
@Benchmark
139+
public boolean array_hit(Cursor cursor) {
140+
return arrayContains(array, cursor.nextHit());
141+
}
142+
143+
@Benchmark
144+
public boolean array_miss(Cursor cursor) {
145+
return arrayContains(array, cursor.nextMiss());
146+
}
147+
148+
@Benchmark
149+
public boolean sortedArray_hit(Cursor cursor) {
150+
return Arrays.binarySearch(sortedArray, cursor.nextHit()) >= 0;
151+
}
152+
153+
@Benchmark
154+
public boolean sortedArray_miss(Cursor cursor) {
155+
return Arrays.binarySearch(sortedArray, cursor.nextMiss()) >= 0;
156+
}
157+
158+
@Benchmark
159+
public boolean hashSet_hit(Cursor cursor) {
160+
return hashSet.contains(cursor.nextHit());
161+
}
162+
163+
@Benchmark
164+
public boolean hashSet_miss(Cursor cursor) {
165+
return hashSet.contains(cursor.nextMiss());
166+
}
167+
168+
@Benchmark
169+
public boolean treeSet_hit(Cursor cursor) {
170+
return treeSet.contains(cursor.nextHit());
171+
}
172+
173+
@Benchmark
174+
public boolean treeSet_miss(Cursor cursor) {
175+
return treeSet.contains(cursor.nextMiss());
176+
}
177+
178+
@Benchmark
179+
public boolean tracerImmutableSet_hit(Cursor cursor) {
180+
return tracerImmutableSet.contains(cursor.nextHit());
181+
}
182+
183+
@Benchmark
184+
public boolean tracerImmutableSet_miss(Cursor cursor) {
185+
return tracerImmutableSet.contains(cursor.nextMiss());
186+
}
187+
}

‎internal-api/src/jmh/java/datadog/trace/util/SetBenchmark.java‎

Lines changed: 0 additions & 128 deletions
This file was deleted.

0 commit comments

Comments
 (0)