Skip to content

Commit 80b927e

Browse files
sjarmakclaude
sjarmak
and
claude
committed
Fix scaling-gap oracle curation and baseline Dockerfiles
- Re-curate 12 Org task oracles via Daytona curator (fixes 4 copy-paste shifted repos, replaces all manually fabricated file paths with real curator-generated ground truth) - Fix 8 baseline Dockerfiles: clone as root with sg-evals mirrors (Daytona builds fail when cloning as USER claude due to /workspace permissions) - Add smoke_test_tasks.py for parallel Daytona build+verifier validation - All 20 task/config pairs pass smoke tests (10 sg_only + 10 baseline) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 2b88375 commit 80b927e

File tree

60 files changed

+7386
-89
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

60 files changed

+7386
-89
lines changed

‎benchmarks/csb_org_compliance/ccx-compliance-286/tests/ground_truth.json‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
{
2+
"files": [
3+
"tidb::pkg/privilege/privilege.go",
4+
"tidb::pkg/privilege/privileges/privileges.go",
5+
"tidb::pkg/privilege/privileges/cache.go",
6+
"tidb::pkg/privilege/privileges/errors.go",
7+
"tidb::pkg/privilege/conn/conn.go",
8+
"tidb::pkg/planner/core/optimizer.go",
9+
"tidb::pkg/planner/optimize.go",
10+
"tidb::pkg/planner/core/planbuilder.go",
11+
"tidb::pkg/planner/core/logical_plan_builder.go",
12+
"tidb::pkg/planner/core/preprocess.go",
13+
"tidb::pkg/planner/core/point_get_plan.go",
14+
"tidb::pkg/planner/core/expression_codec_fn.go",
15+
"tidb::pkg/planner/core/plan_cache.go",
16+
"tidb::pkg/planner/core/plan_cache_utils.go",
17+
"tidb::pkg/planner/core/expression_rewriter.go"
18+
],
19+
"symbols": [
20+
{
21+
"file": "pkg/privilege/privilege.go",
22+
"symbol": "Manager",
23+
"repo": "sg-evals/tidb--v8.5.0"
24+
},
25+
{
26+
"file": "pkg/privilege/privileges/cache.go",
27+
"symbol": "MySQLPrivilege",
28+
"repo": "sg-evals/tidb--v8.5.0"
29+
},
30+
{
31+
"file": "pkg/privilege/privileges/cache.go",
32+
"symbol": "Handle",
33+
"repo": "sg-evals/tidb--v8.5.0"
34+
},
35+
{
36+
"file": "pkg/privilege/privileges/cache.go",
37+
"symbol": "immutable",
38+
"repo": "sg-evals/tidb--v8.5.0"
39+
},
40+
{
41+
"file": "pkg/privilege/privileges/cache.go",
42+
"symbol": "roleGraphEdgesTable",
43+
"repo": "sg-evals/tidb--v8.5.0"
44+
},
45+
{
46+
"file": "pkg/privilege/privileges/privileges.go",
47+
"symbol": "UserPrivileges",
48+
"repo": "sg-evals/tidb--v8.5.0"
49+
},
50+
{
51+
"file": "pkg/planner/core/planbuilder.go",
52+
"symbol": "visitInfo",
53+
"repo": "sg-evals/tidb--v8.5.0"
54+
},
55+
{
56+
"file": "pkg/planner/core/optimizer.go",
57+
"symbol": "CheckPrivilege",
58+
"repo": "sg-evals/tidb--v8.5.0"
59+
},
60+
{
61+
"file": "pkg/planner/core/optimizer.go",
62+
"symbol": "VisitInfo4PrivCheck",
63+
"repo": "sg-evals/tidb--v8.5.0"
64+
},
65+
{
66+
"file": "pkg/planner/core/optimizer.go",
67+
"symbol": "CheckTableLock",
68+
"repo": "sg-evals/tidb--v8.5.0"
69+
}
70+
]
71+
}

‎benchmarks/csb_org_compliance/ccx-compliance-286/tests/ground_truth_meta.json‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{
2+
"has_ground_truth": true,
3+
"has_chunk_ground_truth": false,
4+
"ground_truth_source": "curator_agent",
5+
"ground_truth_confidence": "medium",
6+
"task_name": "ccx-compliance-286",
7+
"curator_agent_version": "2.0",
8+
"model": "claude-opus-4-6",
9+
"backend": "hybrid",
10+
"timestamp": "2026-03-06T21:15:25Z",
11+
"files_count": 15,
12+
"edit_files_count": 0,
13+
"chunks_count": 0,
14+
"symbols_count": 10,
15+
"cost_usd": 0.8287657500000001,
16+
"elapsed_sec": 134.3,
17+
"exploration_notes": "The privilege/RBAC enforcement in TiDB spans two main packages:\n\n**pkg/privilege/ (privilege subsystem)**:\n- `privilege.go`: Defines the `Manager` interface \u2014 the contract for all privilege operations including `RequestVerification`, `RequestDynamicVerification`, `ConnectionVerification`, role methods (`ActiveRoles`, `FindEdge`, `GetAllRoles`, `GetDefaultRoles`), and `DBIsVisible`.\n- `privileges/privileges.go`: `UserPrivileges` struct implements `Manager`. Its `RequestVerification` method is the"
18+
}

‎benchmarks/csb_org_compliance/ccx-compliance-286/tests/oracle_answer.json‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
{
2+
"files": [
3+
{
4+
"repo": "sg-evals/tidb--v8.5.0",
5+
"path": "pkg/privilege/privilege.go"
6+
},
7+
{
8+
"repo": "sg-evals/tidb--v8.5.0",
9+
"path": "pkg/privilege/privileges/privileges.go"
10+
},
11+
{
12+
"repo": "sg-evals/tidb--v8.5.0",
13+
"path": "pkg/privilege/privileges/cache.go"
14+
},
15+
{
16+
"repo": "sg-evals/tidb--v8.5.0",
17+
"path": "pkg/privilege/privileges/errors.go"
18+
},
19+
{
20+
"repo": "sg-evals/tidb--v8.5.0",
21+
"path": "pkg/privilege/conn/conn.go"
22+
},
23+
{
24+
"repo": "sg-evals/tidb--v8.5.0",
25+
"path": "pkg/planner/core/optimizer.go"
26+
},
27+
{
28+
"repo": "sg-evals/tidb--v8.5.0",
29+
"path": "pkg/planner/optimize.go"
30+
},
31+
{
32+
"repo": "sg-evals/tidb--v8.5.0",
33+
"path": "pkg/planner/core/planbuilder.go"
34+
},
35+
{
36+
"repo": "sg-evals/tidb--v8.5.0",
37+
"path": "pkg/planner/core/logical_plan_builder.go"
38+
},
39+
{
40+
"repo": "sg-evals/tidb--v8.5.0",
41+
"path": "pkg/planner/core/preprocess.go"
42+
},
43+
{
44+
"repo": "sg-evals/tidb--v8.5.0",
45+
"path": "pkg/planner/core/point_get_plan.go"
46+
},
47+
{
48+
"repo": "sg-evals/tidb--v8.5.0",
49+
"path": "pkg/planner/core/expression_codec_fn.go"
50+
},
51+
{
52+
"repo": "sg-evals/tidb--v8.5.0",
53+
"path": "pkg/planner/core/plan_cache.go"
54+
},
55+
{
56+
"repo": "sg-evals/tidb--v8.5.0",
57+
"path": "pkg/planner/core/plan_cache_utils.go"
58+
},
59+
{
60+
"repo": "sg-evals/tidb--v8.5.0",
61+
"path": "pkg/planner/core/expression_rewriter.go"
62+
}
63+
],
64+
"symbols": [
65+
{
66+
"repo": "sg-evals/tidb--v8.5.0",
67+
"path": "pkg/privilege/privilege.go",
68+
"symbol": "Manager"
69+
},
70+
{
71+
"repo": "sg-evals/tidb--v8.5.0",
72+
"path": "pkg/privilege/privileges/cache.go",
73+
"symbol": "MySQLPrivilege"
74+
},
75+
{
76+
"repo": "sg-evals/tidb--v8.5.0",
77+
"path": "pkg/privilege/privileges/cache.go",
78+
"symbol": "Handle"
79+
},
80+
{
81+
"repo": "sg-evals/tidb--v8.5.0",
82+
"path": "pkg/privilege/privileges/cache.go",
83+
"symbol": "immutable"
84+
},
85+
{
86+
"repo": "sg-evals/tidb--v8.5.0",
87+
"path": "pkg/privilege/privileges/cache.go",
88+
"symbol": "roleGraphEdgesTable"
89+
},
90+
{
91+
"repo": "sg-evals/tidb--v8.5.0",
92+
"path": "pkg/privilege/privileges/privileges.go",
93+
"symbol": "UserPrivileges"
94+
},
95+
{
96+
"repo": "sg-evals/tidb--v8.5.0",
97+
"path": "pkg/planner/core/planbuilder.go",
98+
"symbol": "visitInfo"
99+
},
100+
{
101+
"repo": "sg-evals/tidb--v8.5.0",
102+
"path": "pkg/planner/core/optimizer.go",
103+
"symbol": "CheckPrivilege"
104+
},
105+
{
106+
"repo": "sg-evals/tidb--v8.5.0",
107+
"path": "pkg/planner/core/optimizer.go",
108+
"symbol": "VisitInfo4PrivCheck"
109+
},
110+
{
111+
"repo": "sg-evals/tidb--v8.5.0",
112+
"path": "pkg/planner/core/optimizer.go",
113+
"symbol": "CheckTableLock"
114+
}
115+
],
116+
"chain": [
117+
{
118+
"repo": "sg-evals/tidb--v8.5.0",
119+
"path": "pkg/planner/optimize.go",
120+
"symbol": "optimize"
121+
},
122+
{
123+
"repo": "sg-evals/tidb--v8.5.0",
124+
"path": "pkg/planner/core/optimizer.go",
125+
"symbol": "CheckPrivilege"
126+
},
127+
{
128+
"repo": "sg-evals/tidb--v8.5.0",
129+
"path": "pkg/privilege/privilege.go",
130+
"symbol": "Manager.RequestVerification"
131+
},
132+
{
133+
"repo": "sg-evals/tidb--v8.5.0",
134+
"path": "pkg/privilege/privileges/privileges.go",
135+
"symbol": "UserPrivileges.RequestVerification"
136+
},
137+
{
138+
"repo": "sg-evals/tidb--v8.5.0",
139+
"path": "pkg/privilege/privileges/cache.go",
140+
"symbol": "MySQLPrivilege.RequestVerification"
141+
},
142+
{
143+
"repo": "sg-evals/tidb--v8.5.0",
144+
"path": "pkg/privilege/privileges/cache.go",
145+
"symbol": "MySQLPrivilege.FindAllUserEffectiveRoles"
146+
},
147+
{
148+
"repo": "sg-evals/tidb--v8.5.0",
149+
"path": "pkg/privilege/privileges/cache.go",
150+
"symbol": "MySQLPrivilege.FindAllRole"
151+
},
152+
{
153+
"repo": "sg-evals/tidb--v8.5.0",
154+
"path": "pkg/planner/core/planbuilder.go",
155+
"symbol": "PlanBuilder.GetVisitInfo"
156+
},
157+
{
158+
"repo": "sg-evals/tidb--v8.5.0",
159+
"path": "pkg/planner/core/logical_plan_builder.go",
160+
"symbol": "appendVisitInfo"
161+
},
162+
{
163+
"repo": "sg-evals/tidb--v8.5.0",
164+
"path": "pkg/planner/core/logical_plan_builder.go",
165+
"symbol": "appendDynamicVisitInfo"
166+
},
167+
{
168+
"repo": "sg-evals/tidb--v8.5.0",
169+
"path": "pkg/planner/core/optimizer.go",
170+
"symbol": "VisitInfo4PrivCheck"
171+
}
172+
],
173+
"text": "The privilege/RBAC enforcement in TiDB spans two main packages:\n\n**pkg/privilege/ (privilege subsystem)**:\n- `privilege.go`: Defines the `Manager` interface \u2014 the contract for all privilege operations including `RequestVerification`, `RequestDynamicVerification`, `ConnectionVerification`, role methods (`ActiveRoles`, `FindEdge`, `GetAllRoles`, `GetDefaultRoles`), and `DBIsVisible`.\n- `privileges/privileges.go`: `UserPrivileges` struct implements `Manager`. Its `RequestVerification` method is the entry point: it handles SEM (Security Enhanced Mode) restrictions, in-memory DB checks, extension access checks, then delegates to `MySQLPrivilege.RequestVerification` on the cache.\n- `privileges/cache.go`: `MySQLPrivilege` is the in-memory cache of all mysql.* privilege tables. It contains the `immutable` struct (user records, db records, tablesPriv, columnsPriv, defaultRoles, globalPriv, dynamicPriv, and `roleGraph`). Key methods: `RequestVerification` (checks user\u2192db\u2192table\u2192column privilege cascade with role expansion), `FindAllRole` (BFS traversal of `roleGraph` map), `FindAllUserEffectiveRoles`, `FindRole`, `LoadRoleGraph`, and `LoadAll`. The `Handle` struct wraps an atomic pointer to `MySQLPrivilege` for concurrent access.\n- `privileges/errors.go`: Error definitions for privilege failures (ErrAccessDenied, etc.).\n- `conn/conn.go`: `AuthConn` interface for auth plugin communication.\n\n**pkg/planner/ (privilege enforcement in query planning)**:\n- `optimize.go`: The `optimize` function is the main planner entry point. After building the logical plan, it calls `CheckPrivilege(activeRoles, pm, visitInfo)` and `CheckTableLock`.\n- `core/optimizer.go`: `CheckPrivilege` iterates over `[]visitInfo`, calling `pm.RequestVerification` for static privileges and `pm.RequestDynamicVerification` for dynamic ones. `VisitInfo4PrivCheck` adjusts privilege requirements for temp tables. `CheckTableLock` verifies table-level locks.\n- `core/planbuilder.go`: Defines the `visitInfo` struct (privilege, db, table, column, dynamicPrivs) and `PlanBuilder.visitInfo` slice. `GetVisitInfo()` returns it. The `buildXxx` methods throughout this file append `visitInfo` entries for each SQL statement type (e.g., `mysql.SuperPriv`, `mysql.ConfigPriv`, `mysql.ShowViewPriv`).\n- `core/logical_plan_builder.go`: `appendVisitInfo` and `appendDynamicVisitInfo` helper functions. Extensive per-statement privilege mapping: SELECT\u2192SelectPriv, DELETE\u2192DeletePriv, UPDATE\u2192UpdatePriv, INSERT\u2192InsertPriv, etc., appended during logical plan building.\n- `core/preprocess.go`: Early-stage privilege check \u2014 calls `pm.RequestVerification` to verify the user can access referenced tables/databases before plan building proceeds.\n- `core/point_get_plan.go`: Direct `pm.RequestVerification` call for point-get optimized paths, plus `CheckTableLock` for lock verification.\n- `core/expression_codec_fn.go`: `RequestVerification` call for codec helper functions that access table data.\n- `core/plan_cache.go`: `checkPreparedPriv` re-checks privileges for cached prepared statements using stored `VisitInfos`.\n- `core/plan_cache_utils.go`: `PlanCacheStmt` stores `VisitInfos []visitInfo` for cached privilege checks.\n- `core/expression_rewriter.go`: Appends `RESTRICTED_VARIABLES_ADMIN` dynamic privilege for restricted system variable access.",
174+
"_metadata": {
175+
"model": "claude-opus-4-6",
176+
"backend": "hybrid",
177+
"prompt_version": "phase1",
178+
"cost_usd": 0.8287657500000001,
179+
"elapsed_sec": 134.3,
180+
"timestamp": "2026-03-06T21:15:25Z",
181+
"tool_calls": 42,
182+
"generator": "daytona_curator_runner"
183+
}
184+
}

0 commit comments

Comments
 (0)