Skip to content

Commit 25af894

Browse files
LoCoBench Botclaude
LoCoBench Bot
and
claude
committed
feat: US-003 - Create docgen-arch-003: Terraform plan/apply pipeline architecture doc
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
1 parent c74bfb8 commit 25af894

7 files changed

Lines changed: 506 additions & 0 deletions

File tree

‎benchmarks/ccb_docgen/docgen-arch-003/environment/Dockerfile‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
FROM ubuntu:22.04
2+
3+
RUN apt-get update && apt-get install -y \
4+
git \
5+
python3 \
6+
python3-pip \
7+
&& rm -rf /var/lib/apt/lists/*
8+
9+
WORKDIR /repo
10+
11+
# Clone Terraform repository (blobless clone for efficiency)
12+
# Using commit a3dc571150a7651a1a4a8b302342d26089c97795 from main branch (2026-02-16)
13+
RUN git clone --filter=blob:none --no-checkout https://github.com/hashicorp/terraform.git . \
14+
&& git checkout a3dc571150a7651a1a4a8b302342d26089c97795
15+
16+
# Checkout only the internal/ directory to save space and time
17+
RUN git sparse-checkout init --cone \
18+
&& git sparse-checkout set internal docs \
19+
&& git checkout
20+
21+
WORKDIR /workspace

‎benchmarks/ccb_docgen/docgen-arch-003/instruction.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# Task: Terraform Plan/Apply Pipeline Architecture Documentation
2+
3+
## Objective
4+
5+
Generate comprehensive architecture documentation for Terraform's plan/apply pipeline. Your documentation should explain how Terraform executes the plan and apply operations, covering the internal components and their interactions.
6+
7+
## Scope
8+
9+
Your documentation should cover the following architectural components:
10+
11+
1. **Graph Builder System**
12+
- How the dependency graph is constructed
13+
- The role of graph transformers in building the execution graph
14+
- How resources, providers, and modules are represented as graph nodes
15+
- How dependencies between resources are discovered and encoded as edges
16+
17+
2. **Provider Interface and Lifecycle**
18+
- How providers are initialized and managed during execution
19+
- The provider plugin architecture
20+
- How resource operations are delegated to providers
21+
- Provider configuration and instance management
22+
23+
3. **State Management**
24+
- How state is read, modified, and persisted during plan/apply
25+
- The role of state managers and state synchronization
26+
- How state snapshots enable concurrent graph evaluation
27+
- State locking and remote state backends
28+
29+
4. **Execution Flow and Hook System**
30+
- The overall execution flow from command invocation to completion
31+
- How graph nodes are evaluated (plan vs apply execution)
32+
- The walker pattern for graph traversal
33+
- Hook points for extending Terraform's behavior
34+
- Dynamic expansion for count/for_each resources
35+
36+
## Requirements
37+
38+
- **Component Responsibilities**: Clearly explain what each major component does
39+
- **Data Flow**: Describe how data flows through the system during plan and apply operations
40+
- **Extension Points**: Identify where the architecture allows for customization or extension
41+
- **Error Handling**: Explain how errors are propagated and handled during execution
42+
43+
## Output Format
44+
45+
Write your documentation to `/workspace/documentation.md` in Markdown format.
46+
47+
Your documentation should be technical and precise, aimed at developers who want to understand Terraform's internal architecture. Include specific details about component interactions, not just high-level descriptions.
48+
49+
## Success Criteria
50+
51+
Your documentation will be evaluated on:
52+
- Coverage of all required architectural topics
53+
- Accurate description of component responsibilities and interactions
54+
- Clear explanation of data flow through the pipeline
55+
- Identification of key extension points in the architecture
56+
- Technical depth appropriate for internal architecture documentation

‎benchmarks/ccb_docgen/docgen-arch-003/task.toml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
[task]
2+
name = "docgen-arch-003"
3+
category = "architecture_doc"
4+
description = "Generate architecture documentation for Terraform's plan/apply pipeline"
5+
6+
[runtime]
7+
language = "go"
8+
difficulty = "hard"
9+
time_limit_sec = 1200
10+
11+
[scoring]
12+
method = "weighted_checklist"
13+
max_reward = 1.0

‎benchmarks/ccb_docgen/docgen-arch-003/tests/ground_truth.json‎

Lines changed: 234 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,234 @@
1+
{
2+
"scoring_categories": {
3+
"required_topics": {
4+
"weight": 0.40,
5+
"topics": [
6+
{
7+
"name": "GraphTransformer interface and transform sequence",
8+
"weight": 0.08,
9+
"check_any_pattern": [
10+
"GraphTransformer",
11+
"graph transform",
12+
"transform.*sequence",
13+
"mutate.*graph"
14+
]
15+
},
16+
{
17+
"name": "ConfigTransformer for resource vertices",
18+
"weight": 0.07,
19+
"check_any_pattern": [
20+
"ConfigTransformer",
21+
"resource.*vertex.*configuration",
22+
"resource block.*vertex"
23+
]
24+
},
25+
{
26+
"name": "StateTransformer for existing instances",
27+
"weight": 0.07,
28+
"check_any_pattern": [
29+
"StateTransformer",
30+
"existing.*instance.*state",
31+
"state.*vertex"
32+
]
33+
},
34+
{
35+
"name": "ReferenceTransformer for dependency analysis",
36+
"weight": 0.08,
37+
"check_any_pattern": [
38+
"ReferenceTransformer",
39+
"dependency.*analysis",
40+
"reference.*dependency"
41+
]
42+
},
43+
{
44+
"name": "ProviderTransformer for initialization ordering",
45+
"weight": 0.07,
46+
"check_any_pattern": [
47+
"ProviderTransformer",
48+
"provider.*initialization.*order",
49+
"provider.*before.*resource"
50+
]
51+
},
52+
{
53+
"name": "Provider plugin architecture and EvalContext",
54+
"weight": 0.08,
55+
"check_any_pattern": [
56+
"EvalContext",
57+
"provider.*plugin",
58+
"provider.*retrieve.*context"
59+
]
60+
},
61+
{
62+
"name": "Provider instance diff calculation",
63+
"weight": 0.07,
64+
"check_any_pattern": [
65+
"instance diff",
66+
"provider.*diff",
67+
"state.*configuration.*difference"
68+
]
69+
},
70+
{
71+
"name": "State manager interfaces (statemgr)",
72+
"weight": 0.08,
73+
"check_any_pattern": [
74+
"statemgr",
75+
"state manager",
76+
"Filesystem.*manager",
77+
"remote.*state.*backend"
78+
]
79+
},
80+
{
81+
"name": "SyncState for concurrent access",
82+
"weight": 0.08,
83+
"check_any_pattern": [
84+
"SyncState",
85+
"synchronized.*state.*access",
86+
"concurrent.*state"
87+
]
88+
},
89+
{
90+
"name": "GraphNodeExecutable interface",
91+
"weight": 0.08,
92+
"check_any_pattern": [
93+
"GraphNodeExecutable",
94+
"Execute.*method",
95+
"vertex.*executable"
96+
]
97+
},
98+
{
99+
"name": "Node types (NodePlannableResourceInstance, NodeApplyableResourceInstance)",
100+
"weight": 0.08,
101+
"check_any_pattern": [
102+
"NodePlannableResourceInstance",
103+
"NodeApplyableResourceInstance",
104+
"NodeDestroyResourceInstance"
105+
]
106+
},
107+
{
108+
"name": "ContextGraphWalker and graph traversal",
109+
"weight": 0.08,
110+
"check_any_pattern": [
111+
"ContextGraphWalker",
112+
"graph.*walk",
113+
"graph.*traversal",
114+
"walker.*traverse"
115+
]
116+
},
117+
{
118+
"name": "Dynamic expansion (GraphNodeDynamicExpandable)",
119+
"weight": 0.08,
120+
"check_any_pattern": [
121+
"GraphNodeDynamicExpandable",
122+
"dynamic.*expansion",
123+
"count.*for_each.*expansion"
124+
]
125+
}
126+
]
127+
},
128+
"file_references": {
129+
"weight": 0.25,
130+
"files": [
131+
{
132+
"path": "internal/terraform/context.go",
133+
"weight": 0.25,
134+
"description": "Core context and execution orchestration"
135+
},
136+
{
137+
"path": "internal/terraform/graph_builder",
138+
"weight": 0.20,
139+
"description": "Graph builder implementations"
140+
},
141+
{
142+
"path": "internal/terraform/node_resource",
143+
"weight": 0.20,
144+
"description": "Resource node implementations for plan/apply"
145+
},
146+
{
147+
"path": "internal/providers",
148+
"weight": 0.20,
149+
"description": "Provider interface and plugin coordination"
150+
},
151+
{
152+
"path": "internal/states",
153+
"weight": 0.15,
154+
"description": "State management and synchronization"
155+
}
156+
]
157+
},
158+
"data_flow": {
159+
"weight": 0.20,
160+
"flows": [
161+
{
162+
"name": "Plan execution flow",
163+
"weight": 0.35,
164+
"check_all_patterns": [
165+
"graph.*build",
166+
"provider.*initialize",
167+
"resource.*plan",
168+
"diff.*calculate"
169+
]
170+
},
171+
{
172+
"name": "Apply execution flow",
173+
"weight": 0.35,
174+
"check_all_patterns": [
175+
"graph.*walk",
176+
"node.*execute",
177+
"provider.*apply",
178+
"state.*update"
179+
]
180+
},
181+
{
182+
"name": "State read-modify-persist cycle",
183+
"weight": 0.30,
184+
"check_all_patterns": [
185+
"state.*read",
186+
"state.*modify",
187+
"state.*persist"
188+
]
189+
}
190+
]
191+
},
192+
"extension_points": {
193+
"weight": 0.15,
194+
"points": [
195+
{
196+
"name": "GraphTransformer for custom graph mutations",
197+
"weight": 0.25,
198+
"check_any_pattern": [
199+
"implement.*GraphTransformer",
200+
"custom.*transform",
201+
"extend.*graph.*build"
202+
]
203+
},
204+
{
205+
"name": "Provider plugin interface",
206+
"weight": 0.30,
207+
"check_any_pattern": [
208+
"provider.*plugin",
209+
"custom.*provider",
210+
"provider.*interface.*implement"
211+
]
212+
},
213+
{
214+
"name": "State backend implementations",
215+
"weight": 0.25,
216+
"check_any_pattern": [
217+
"state.*backend",
218+
"remote.*state",
219+
"statemgr.*implement"
220+
]
221+
},
222+
{
223+
"name": "Hook system for execution observability",
224+
"weight": 0.20,
225+
"check_any_pattern": [
226+
"hook",
227+
"callback",
228+
"execution.*observer"
229+
]
230+
}
231+
]
232+
}
233+
}
234+
}

0 commit comments

Comments
 (0)