Skip to content

Commit 873b40b

Browse files
LoCoBench Botclaude
LoCoBench Bot
and
claude
committed
feat: US-002 - Create docgen-arch-002 Istio Pilot discovery architecture doc
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
1 parent 8d6a4af commit 873b40b

7 files changed

Lines changed: 483 additions & 1 deletion

File tree

‎benchmarks/ccb_docgen/docgen-arch-002/environment/Dockerfile‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
FROM python:3.11-slim
2+
3+
WORKDIR /workspace
4+
5+
# Install dependencies
6+
RUN apt-get update && apt-get install -y \
7+
git \
8+
curl \
9+
npm \
10+
&& rm -rf /var/lib/apt/lists/*
11+
12+
# Install Claude Code CLI
13+
RUN npm install -g @anthropic-ai/claude-code
14+
15+
# Clone Istio at pinned commit (blobless clone — repo is large)
16+
RUN git clone --filter=blob:none --no-checkout https://github.com/istio/istio.git . && \
17+
git checkout f8af3caeb1960d4cb1db5c4c0d4913c366625c71 && \
18+
git config user.email "agent@example.com" && \
19+
git config user.name "Agent"
20+
21+
# Create output directories
22+
RUN mkdir -p /logs/agent /logs/verifier

‎benchmarks/ccb_docgen/docgen-arch-002/instruction.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# Architecture Document: Istio Pilot Discovery Service
2+
3+
**Repository:** istio/istio
4+
**Output:** Write your document to `/workspace/documentation.md`
5+
6+
## Task
7+
8+
Produce an architecture document for **Istio Pilot's Discovery Service** subsystem. The document must explain how Pilot serves xDS (Envoy Discovery Service) configuration to proxies, translates Istio's high-level configuration into Envoy-specific resources, integrates with service registries, and supports multi-cluster deployments. Do not simply list APIs — explain the **design**, **data flow**, and **extension points**.
9+
10+
## Scope
11+
12+
Your document must cover these four components and how they work together:
13+
14+
### 1. DiscoveryServer — The xDS gRPC Server
15+
The central component implementing Pilot's gRPC interface for Envoy's xDS APIs. Explain:
16+
- How it implements Envoy's xDS v3 protocol (ADS, incremental/state-of-the-world modes)
17+
- The role of `adsClients` in tracking active proxy connections
18+
- How configuration pushes are coordinated: debouncing, rate limiting, concurrent push limits
19+
- Cache management for xDS resources (CDS, LDS, RDS, EDS)
20+
- The lifecycle: initialization, handling new connections, shutdown
21+
22+
### 2. ADS and the Connection Model
23+
The Aggregated Discovery Service stream handler. Explain:
24+
- How `Stream()` establishes a bidirectional gRPC stream with an Envoy proxy
25+
- The `Connection` type: wrapping the stream with proxy metadata and state
26+
- Request processing: how `processRequest()` determines what configuration to send
27+
- Push coordination: how `pushConnection()` transmits updates to a single proxy
28+
- The `StartPush()` broadcast mechanism for fleet-wide updates
29+
30+
### 3. PushContext — Configuration Snapshot and Translation
31+
The immutable configuration snapshot used during a push. Explain:
32+
- What `PushContext` holds: services, destination rules, virtual services, sidecars, gateways
33+
- How `InitContext()` rebuilds the push context when Istio config changes (CRD watches)
34+
- Config translation: how Pilot converts Istio's `VirtualService` and `DestinationRule` into Envoy's `RouteConfiguration` and `Cluster` resources
35+
- The relationship between push triggers (config changes, service registry updates) and push context regeneration
36+
- Metrics and error tracking during push generation
37+
38+
### 4. Service Registry and Multi-Cluster Support
39+
The service discovery layer feeding the DiscoveryServer. Explain:
40+
- How `ServiceRegistry` (Kubernetes controller) watches K8s Services, Endpoints, and Pods
41+
- The aggregate controller pattern for multi-registry support (K8s services + `ServiceEntry` resources)
42+
- Multi-cluster support: how Pilot aggregates services from multiple clusters
43+
- How endpoint updates (pod scale-up, health changes) trigger EDS pushes
44+
- The relationship between the service registry and xDS: when does a service change trigger CDS vs. EDS?
45+
46+
## Document Requirements
47+
48+
1. **Component Responsibilities** — what each component owns
49+
2. **Data Flow** — the path from Istio config change → DiscoveryServer → xDS push to Envoy proxies
50+
3. **Extension Points** — where users extend Istio (custom config translation, external service registries, telemetry plugins)
51+
4. **Error Handling** — how errors at each stage (config validation, push failures, proxy disconnects) are handled
52+
5. **Key Source Files** — reference the actual source files in the istio/istio repository (e.g., `pilot/pkg/xds/discovery.go`, `pilot/pkg/model/push_context.go`)
53+
54+
## Anti-Requirements
55+
56+
- Do NOT generate a simple API listing or struct dump
57+
- Do NOT fabricate type names or file paths that don't exist in the repository
58+
- Do NOT cover Istio components outside the Pilot discovery path (e.g., Citadel, Galley, sidecar injection)

‎benchmarks/ccb_docgen/docgen-arch-002/task.toml‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
version = "1.0"
2+
3+
[metadata]
4+
name = "docgen-arch-002"
5+
description = "Generate architecture documentation for Istio Pilot's discovery service spanning xDS serving, config translation, service registry, and multi-cluster support"
6+
license = "Apache-2.0"
7+
8+
[task]
9+
id = "docgen-arch-002"
10+
repo = "istio/istio"
11+
category = "architecture_doc"
12+
language = "go"
13+
pre_fix_rev = "f8af3caeb1960d4cb1db5c4c0d4913c366625c71"
14+
difficulty = "hard"
15+
time_limit_sec = 1200
16+
17+
[task.metadata]
18+
complexity_dimension = "architecture_documentation"
19+
doc_type = "architecture"
20+
target_subsystem = "Pilot discovery service"
21+
key_components = ["DiscoveryServer", "ADS", "PushContext", "ServiceRegistry"]
22+
output_path = "/workspace/documentation.md"
23+
24+
[verification]
25+
type = "test"
26+
command = "bash /tests/test.sh"
27+
reward_type = "checklist"
28+
description = "Weighted checklist: required topics (0.40), file references (0.25), data flow (0.20), extension points (0.15)"
29+
30+
[environment]
31+
build_timeout_sec = 900.0
32+
33+
[environment.setup_scripts]
34+
mcp_config = """#!/bin/bash
35+
if [ -n "$SOURCEGRAPH_ACCESS_TOKEN" ] && [ -n "$SOURCEGRAPH_URL" ]; then
36+
echo "Setting up Sourcegraph MCP configuration..."
37+
mkdir -p /root/.config/claude
38+
cat > /root/.config/claude/mcp.json << 'EOF'
39+
{
40+
"mcpServers": {
41+
"sourcegraph": {
42+
"command": "npx",
43+
"args": ["-y", "@sourcegraph/mcp-server"],
44+
"env": {
45+
"SRC_ACCESS_TOKEN": "$SOURCEGRAPH_ACCESS_TOKEN",
46+
"SOURCEGRAPH_URL": "$SOURCEGRAPH_URL"
47+
}
48+
}
49+
}
50+
}
51+
EOF
52+
echo "OK MCP configuration created"
53+
else
54+
echo "No Sourcegraph credentials provided, MCP disabled"
55+
fi
56+
exit 0
57+
"""

‎benchmarks/ccb_docgen/docgen-arch-002/tests/ground_truth.json‎

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
{
2+
"task_id": "docgen-arch-002",
3+
"description": "Istio Pilot Discovery Service Architecture Documentation",
4+
"weights": {
5+
"required_topics": 0.40,
6+
"file_references": 0.25,
7+
"data_flow": 0.20,
8+
"extension_points": 0.15
9+
},
10+
"required_topics": [
11+
{
12+
"id": "t1",
13+
"description": "Explains DiscoveryServer as the xDS gRPC server",
14+
"patterns": ["DiscoveryServer", "xDS.*server|gRPC.*server|xDS.*gRPC"],
15+
"weight": 0.10
16+
},
17+
{
18+
"id": "t2",
19+
"description": "Explains ADS (Aggregated Discovery Service)",
20+
"patterns": ["ADS|Aggregated.*Discovery.*Service|StreamAggregatedResources"],
21+
"weight": 0.10
22+
},
23+
{
24+
"id": "t3",
25+
"description": "Explains adsClients or connection tracking",
26+
"patterns": ["adsClients|connection.*tracking|active.*connections|proxy.*connections"],
27+
"weight": 0.07
28+
},
29+
{
30+
"id": "t4",
31+
"description": "Explains debouncing or push coordination",
32+
"patterns": ["debounce|debouncing|DebounceOptions|push.*coordination|batch.*updates"],
33+
"weight": 0.07
34+
},
35+
{
36+
"id": "t5",
37+
"description": "Explains xDS resource types (CDS, LDS, RDS, EDS)",
38+
"patterns": ["CDS|LDS|RDS|EDS|ClusterDiscoveryService|ListenerDiscoveryService|RouteDiscoveryService|EndpointDiscoveryService"],
39+
"weight": 0.08
40+
},
41+
{
42+
"id": "t6",
43+
"description": "Explains Connection type wrapping xDS stream",
44+
"patterns": ["Connection.*type|Connection.*struct|stream.*wrapper|proxy.*metadata"],
45+
"weight": 0.08
46+
},
47+
{
48+
"id": "t7",
49+
"description": "Explains PushContext as configuration snapshot",
50+
"patterns": ["PushContext", "configuration.*snapshot|config.*snapshot|immutable.*context"],
51+
"weight": 0.10
52+
},
53+
{
54+
"id": "t8",
55+
"description": "Explains InitContext or push context regeneration",
56+
"patterns": ["InitContext|push.*context.*regen|rebuild.*context|config.*change.*trigger"],
57+
"weight": 0.07
58+
},
59+
{
60+
"id": "t9",
61+
"description": "Explains config translation (VirtualService, DestinationRule to Envoy resources)",
62+
"patterns": ["VirtualService|DestinationRule|RouteConfiguration|Cluster.*resource|config.*translation|Istio.*CRD"],
63+
"weight": 0.10
64+
},
65+
{
66+
"id": "t10",
67+
"description": "Explains ServiceRegistry or Kubernetes service discovery",
68+
"patterns": ["ServiceRegistry|Kubernetes.*controller|K8s.*services|service.*discovery"],
69+
"weight": 0.08
70+
},
71+
{
72+
"id": "t11",
73+
"description": "Explains aggregate controller or multi-registry support",
74+
"patterns": ["aggregate.*controller|multi.*registry|ServiceEntry|external.*services"],
75+
"weight": 0.05
76+
},
77+
{
78+
"id": "t12",
79+
"description": "Explains multi-cluster support",
80+
"patterns": ["multi.*cluster|multicluster|cross.*cluster|remote.*cluster"],
81+
"weight": 0.05
82+
},
83+
{
84+
"id": "t13",
85+
"description": "Explains endpoint updates triggering EDS pushes",
86+
"patterns": ["endpoint.*update|EDS.*push|pod.*scale|health.*change|endpoint.*controller"],
87+
"weight": 0.05
88+
}
89+
],
90+
"file_references": [
91+
{
92+
"id": "r1",
93+
"description": "References discovery.go",
94+
"patterns": ["pilot/pkg/xds/discovery\\.go|xds/discovery\\.go"],
95+
"weight": 0.25
96+
},
97+
{
98+
"id": "r2",
99+
"description": "References ads.go",
100+
"patterns": ["pilot/pkg/xds/ads\\.go|xds/ads\\.go"],
101+
"weight": 0.20
102+
},
103+
{
104+
"id": "r3",
105+
"description": "References push_context.go",
106+
"patterns": ["pilot/pkg/model/push_context\\.go|model/push_context\\.go"],
107+
"weight": 0.20
108+
},
109+
{
110+
"id": "r4",
111+
"description": "References service registry controller files",
112+
"patterns": ["pilot/pkg/serviceregistry/kube/controller|serviceregistry.*controller|kube/controller/controller\\.go"],
113+
"weight": 0.20
114+
},
115+
{
116+
"id": "r5",
117+
"description": "References multicluster files or networking/core config translation",
118+
"patterns": ["pilot/pkg/serviceregistry.*multicluster|pilot/pkg/networking/core|networking/core"],
119+
"weight": 0.15
120+
}
121+
],
122+
"data_flow": [
123+
{
124+
"id": "d1",
125+
"description": "Istio config change (CRD watch) -> PushContext rebuild -> DiscoveryServer push queue",
126+
"patterns": ["config.*change|CRD.*watch|Kubernetes.*watch", "PushContext|InitContext|rebuild", "push.*queue|push.*trigger|DiscoveryServer"],
127+
"ordered": true,
128+
"weight": 0.40
129+
},
130+
{
131+
"id": "d2",
132+
"description": "DiscoveryServer push -> Connection.pushConnection -> xDS response to Envoy proxy",
133+
"patterns": ["DiscoveryServer|push.*coordination", "Connection|pushConnection|stream", "xDS.*response|DiscoveryResponse|Envoy"],
134+
"ordered": true,
135+
"weight": 0.35
136+
},
137+
{
138+
"id": "d3",
139+
"description": "Service registry update (pod scale, health) -> EDS push",
140+
"patterns": ["service.*registry|endpoint.*update|pod.*scale|health", "EDS|EndpointDiscoveryService|endpoint.*push"],
141+
"weight": 0.25
142+
}
143+
],
144+
"extension_points": [
145+
{
146+
"id": "e1",
147+
"description": "Custom config translation or CRD extensions",
148+
"patterns": ["custom.*CRD|CRD.*extension|config.*translation|VirtualService|DestinationRule|Gateway"],
149+
"weight": 0.30
150+
},
151+
{
152+
"id": "e2",
153+
"description": "External service registries (ServiceEntry)",
154+
"patterns": ["ServiceEntry|external.*service|external.*registry|non.*Kubernetes.*service"],
155+
"weight": 0.25
156+
},
157+
{
158+
"id": "e3",
159+
"description": "Telemetry plugins or Wasm extensions",
160+
"patterns": ["telemetry|Wasm|WebAssembly|EnvoyFilter|extension.*plugin"],
161+
"weight": 0.25
162+
},
163+
{
164+
"id": "e4",
165+
"description": "Custom generators or xDS resource customization",
166+
"patterns": ["[Gg]enerator|custom.*generator|xDS.*customization|XdsResourceGenerator"],
167+
"weight": 0.20
168+
}
169+
]
170+
}

0 commit comments

Comments
 (0)