Skip to content

Commit f2ca2ac

Browse files
LoCoBench Botclaude
LoCoBench Bot
and
claude
committed
feat: US-007 - Create docgen-migration-001: Migration guide from Terraform code changes
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
1 parent 7589619 commit f2ca2ac

7 files changed

Lines changed: 468 additions & 2 deletions

File tree

‎benchmarks/ccb_docgen/docgen-migration-001/environment/Dockerfile‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
FROM ubuntu:22.04
2+
3+
# Install minimal dependencies
4+
RUN apt-get update && \
5+
apt-get install -y --no-install-recommends \
6+
git \
7+
ca-certificates \
8+
&& rm -rf /var/lib/apt/lists/*
9+
10+
WORKDIR /workspace
11+
12+
# Clone Terraform repo with blobless clone for speed
13+
# Then create two worktrees for the old and new versions
14+
RUN git clone --filter=blob:none --no-checkout https://github.com/hashicorp/terraform.git terraform-repo && \
15+
cd terraform-repo && \
16+
# Create worktree for v1.9.0 (commit 7637a92)
17+
git worktree add ../terraform-v1.9.0 7637a9216e68f2812887eaf0722f9d3f9804c45e && \
18+
# Create worktree for v1.10.0 (commit 24236f4)
19+
git worktree add ../terraform-v1.10.0 24236f4f0bd10ada71d70868abe15f9d88099747 && \
20+
# Remove the main clone to save space
21+
cd /workspace && \
22+
rm -rf terraform-repo
23+
24+
# Verify worktrees exist
25+
RUN test -d /workspace/terraform-v1.9.0 && \
26+
test -d /workspace/terraform-v1.10.0 && \
27+
echo "Both Terraform worktrees created successfully"
28+
29+
# Set working directory
30+
WORKDIR /workspace

‎benchmarks/ccb_docgen/docgen-migration-001/instruction.md‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Task: Terraform v1.9.0 to v1.10.0 Migration Guide
2+
3+
## Objective
4+
5+
Analyze the changes between Terraform v1.9.0 and v1.10.0 and produce a comprehensive migration guide for users upgrading from v1.9.0 to v1.10.0.
6+
7+
## Context
8+
9+
The workspace contains two versions of the Terraform codebase:
10+
- **v1.9.0**: Located in `/workspace/terraform-v1.9.0/` (commit 7637a92)
11+
- **v1.10.0**: Located in `/workspace/terraform-v1.10.0/` (commit 24236f4)
12+
13+
Your task is to identify breaking changes and behavioral changes that affect users, then document migration steps.
14+
15+
## Required Analysis
16+
17+
Your migration guide must cover:
18+
19+
1. **S3 Backend Changes**
20+
- Removal of deprecated IAM role assumption attributes
21+
- Migration to `assume_role` block syntax
22+
- New S3 native state locking support
23+
24+
2. **Moved Blocks Syntax Changes**
25+
- New requirement to prepend `resource.` identifier when referencing resources with type names matching top-level blocks/keywords
26+
- Impact on existing `moved` blocks in configurations
27+
28+
3. **Sensitive Value Handling**
29+
- Changes to mark propagation in conditional expressions
30+
- When sensitive marks are now preserved (previously lost)
31+
- How to use `nonsensitive()` to override when needed
32+
33+
4. **Ephemeral Resources and Values** (new feature with migration implications)
34+
- Introduction of ephemeral input variables and outputs
35+
- New ephemeral resource mode
36+
- Impact on secret handling in state files
37+
38+
## Expected Output
39+
40+
Write your migration guide to `/workspace/documentation.md` with the following structure:
41+
42+
1. **Overview** - Summary of the upgrade and major themes
43+
2. **Breaking Changes** - Each breaking change with:
44+
- What changed and why
45+
- Before/after code examples
46+
- Step-by-step migration instructions
47+
- Common pitfalls to avoid
48+
3. **New Features with Migration Impact** - Features that change how users should write Terraform
49+
4. **Testing Your Migration** - How to validate the upgrade was successful
50+
5. **Rollback Guidance** - How to safely rollback if needed
51+
52+
## Scoring Criteria
53+
54+
Your documentation will be evaluated on:
55+
- **Completeness**: All required breaking changes are documented
56+
- **Accuracy**: Migration steps correctly reflect the actual code changes
57+
- **Clarity**: Instructions are clear with concrete examples
58+
- **Practicality**: Includes testing steps and rollback guidance
59+
60+
## Tips
61+
62+
- Use MCP tools like `compare_revisions` to identify changes in specific files
63+
- Look for UPGRADE notes in release documentation
64+
- Examine test files to understand behavioral changes
65+
- Check for deprecation warnings and removed features
66+
- Pay attention to changes in internal/backend/remote-state/s3/, internal/terraform/node_resource_abstract.go, and lang/marks/ directories
67+
68+
Good luck!

‎benchmarks/ccb_docgen/docgen-migration-001/task.toml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
[task]
2+
id = "docgen-migration-001"
3+
category = "migration_guide"
4+
language = "go"
5+
difficulty = "hard"
6+
time_limit_sec = 1200
7+
8+
[metadata]
9+
source_repo = "hashicorp/terraform"
10+
old_version = "v1.9.0"
11+
new_version = "v1.10.0"
12+
old_commit = "7637a9216e68f2812887eaf0722f9d3f9804c45e"
13+
new_commit = "24236f4f0bd10ada71d70868abe15f9d88099747"
14+
description = "Migration guide for upgrading from Terraform v1.9.0 to v1.10.0"

‎benchmarks/ccb_docgen/docgen-migration-001/tests/ground_truth.json‎

Lines changed: 217 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
{
2+
"scoring_categories": {
3+
"breaking_changes": {
4+
"weight": 0.50,
5+
"items": [
6+
{
7+
"name": "S3 Backend: Removal of deprecated IAM role attributes",
8+
"patterns": [
9+
"(?i)s3.*backend",
10+
"(?i)assume.*role",
11+
"(?i)deprecated.*attribute",
12+
"role_arn|session_name|external_id|policy"
13+
],
14+
"weight": 0.15
15+
},
16+
{
17+
"name": "S3 Backend: Migration to assume_role block",
18+
"patterns": [
19+
"assume_role\\s*\\{",
20+
"(?i)before.*after|migration.*example",
21+
"(?i)role_arn.*moved|session_name.*moved"
22+
],
23+
"weight": 0.10
24+
},
25+
{
26+
"name": "Moved Blocks: resource. prefix requirement",
27+
"patterns": [
28+
"(?i)moved.*block",
29+
"resource\\..*identifier",
30+
"(?i)prepend.*resource\\.",
31+
"(?i)type.*name.*match.*top.*level"
32+
],
33+
"weight": 0.15
34+
},
35+
{
36+
"name": "Moved Blocks: Migration examples",
37+
"patterns": [
38+
"moved\\s*\\{",
39+
"from\\s*=.*to\\s*=",
40+
"resource\\.data|resource\\.locals|resource\\.module"
41+
],
42+
"weight": 0.10
43+
},
44+
{
45+
"name": "Sensitive Values: Conditional expression behavior change",
46+
"patterns": [
47+
"(?i)sensitive.*conditional|conditional.*sensitive",
48+
"(?i)mark.*preserved|mark.*propagat",
49+
"\\?.*:.*sensitive|sensitive.*\\?.*:"
50+
],
51+
"weight": 0.15
52+
},
53+
{
54+
"name": "Sensitive Values: nonsensitive() override usage",
55+
"patterns": [
56+
"nonsensitive\\(",
57+
"(?i)override.*sensitive|explicit.*non.?sensitive",
58+
"(?i)false.*\\?.*sensitive.*:.*value"
59+
],
60+
"weight": 0.10
61+
},
62+
{
63+
"name": "Ephemeral Resources: Introduction and impact",
64+
"patterns": [
65+
"(?i)ephemeral.*resource|ephemeral.*value",
66+
"(?i)not.*stored.*state|secret.*handling",
67+
"ephemeral_resource|ephemeral.*input|ephemeral.*output"
68+
],
69+
"weight": 0.15
70+
},
71+
{
72+
"name": "S3 Backend: Native state locking",
73+
"patterns": [
74+
"(?i)s3.*native.*lock|native.*state.*lock",
75+
"(?i)dynamodb.*deprecat",
76+
"(?i)state.*lock.*s3"
77+
],
78+
"weight": 0.10
79+
}
80+
]
81+
},
82+
"migration_steps": {
83+
"weight": 0.30,
84+
"items": [
85+
{
86+
"name": "S3 Backend: Step-by-step assume_role migration",
87+
"patterns": [
88+
"(?i)step.*1|first.*step|1\\.",
89+
"(?i)update.*backend.*configuration|modify.*s3.*backend",
90+
"assume_role\\s*\\{"
91+
],
92+
"weight": 0.25
93+
},
94+
{
95+
"name": "Moved Blocks: Step-by-step resource. prefix migration",
96+
"patterns": [
97+
"(?i)step.*\\d|\\d+\\.",
98+
"(?i)search.*moved.*block|find.*moved",
99+
"(?i)add.*resource\\.|prepend.*resource\\."
100+
],
101+
"weight": 0.25
102+
},
103+
{
104+
"name": "Sensitive Values: Identifying affected conditionals",
105+
"patterns": [
106+
"(?i)identify.*sensitive.*conditional|find.*sensitive",
107+
"(?i)test.*plan|terraform.*plan",
108+
"(?i)warning.*sensitive"
109+
],
110+
"weight": 0.20
111+
},
112+
{
113+
"name": "Sensitive Values: Applying nonsensitive() fixes",
114+
"patterns": [
115+
"(?i)wrap.*nonsensitive|add.*nonsensitive",
116+
"nonsensitive\\(",
117+
"(?i)example.*fix|how.*to.*fix"
118+
],
119+
"weight": 0.15
120+
},
121+
{
122+
"name": "Testing migration",
123+
"patterns": [
124+
"(?i)test.*migration|validat.*upgrade",
125+
"terraform.*init|terraform.*plan",
126+
"(?i)verify.*no.*error|confirm.*success"
127+
],
128+
"weight": 0.15
129+
}
130+
]
131+
},
132+
"code_examples": {
133+
"weight": 0.15,
134+
"items": [
135+
{
136+
"name": "S3 Backend: Before/after assume_role example",
137+
"patterns": [
138+
"```.*hcl|```.*terraform",
139+
"(?i)before:|# before",
140+
"(?i)after:|# after",
141+
"assume_role\\s*\\{"
142+
],
143+
"weight": 0.30
144+
},
145+
{
146+
"name": "Moved Blocks: Before/after resource. prefix example",
147+
"patterns": [
148+
"```",
149+
"(?i)before:|# before",
150+
"(?i)after:|# after",
151+
"moved\\s*\\{.*from.*=.*to.*="
152+
],
153+
"weight": 0.30
154+
},
155+
{
156+
"name": "Sensitive Values: nonsensitive() usage example",
157+
"patterns": [
158+
"```",
159+
"nonsensitive\\(",
160+
"\\?.*:.*sensitive|sensitive.*\\?.*:"
161+
],
162+
"weight": 0.25
163+
},
164+
{
165+
"name": "Ephemeral Resources: Basic usage example",
166+
"patterns": [
167+
"```",
168+
"ephemeral_resource|ephemeral.*{",
169+
"(?i)ephemeral.*input|ephemeral.*output"
170+
],
171+
"weight": 0.15
172+
}
173+
]
174+
},
175+
"documentation_structure": {
176+
"weight": 0.05,
177+
"items": [
178+
{
179+
"name": "Overview section",
180+
"patterns": [
181+
"(?i)^#+.*overview|^#+.*introduction|^#+.*summary",
182+
"(?i)terraform.*1\\.9.*1\\.10|v1\\.9.*v1\\.10"
183+
],
184+
"weight": 0.20
185+
},
186+
{
187+
"name": "Breaking Changes section",
188+
"patterns": [
189+
"(?i)^#+.*breaking.*change"
190+
],
191+
"weight": 0.30
192+
},
193+
{
194+
"name": "Migration steps section",
195+
"patterns": [
196+
"(?i)^#+.*migration|^#+.*upgrad.*step|^#+.*how.*to"
197+
],
198+
"weight": 0.20
199+
},
200+
{
201+
"name": "Testing section",
202+
"patterns": [
203+
"(?i)^#+.*test|^#+.*validat|^#+.*verif"
204+
],
205+
"weight": 0.15
206+
},
207+
{
208+
"name": "Rollback guidance section",
209+
"patterns": [
210+
"(?i)^#+.*rollback|^#+.*revert|^#+.*downgrad"
211+
],
212+
"weight": 0.15
213+
}
214+
]
215+
}
216+
}
217+
}

0 commit comments

Comments
 (0)