fix: re-mirror sg-benchmarks at pinned commits + fix 5 oracles

sjarmak · claude · sjarmak · commit 94b13197e9eb · 2026-02-21T17:56:18.000Z
Re-mirrored 3 sg-benchmarks repos at Dockerfile-pinned versions to
eliminate version drift between MCP and baseline configs:
- expressjs-express: v5.2.1 → 4.21.1
- grafana-loki: v3.6.5 → v3.3.4
- kubernetes-client-go: v12.0.0 → v0.32.0

Fixed oracles for 5 tasks:
- ccx-incident-031: server/storage/mvcc/ → server/mvcc/ (etcd v3.5.17)
- ccx-onboard-050-ds: same etcd path rename
- ccx-vuln-remed-011: cookie ^0.7.1 → 0.7.1 (express 4.21.1)
- ccx-vuln-remed-014: pkg/dataobj/explorer/service.go → pkg/loki/modules.go
  (dataobj/ dir DNE at loki v3.3.4; modules.go has unauth ingester endpoints)
- ccx-dep-trace-001: 8→9 files (remove lister.go false positive, add
  informer.go + golden_test.go at client-go v0.32.0)

Co-Authored-By: Claude Opus 4.6 &lt;noreply@anthropic.com&gt;
diff --git a/benchmarks/ccb_mcp_crossrepo_tracing/ccx-dep-trace-001/tests/oracle_answer.json b/benchmarks/ccb_mcp_crossrepo_tracing/ccx-dep-trace-001/tests/oracle_answer.json
@@ -1,18 +1,21 @@
 {
   "files": [
     {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/client_test.go"},
+    {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/dynamicinformer/informer.go"},
     {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/dynamicinformer/informer_test.go"},
-    {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/dynamiclister/lister.go"},
     {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/dynamiclister/lister_test.go"},
     {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/dynamiclister/shim.go"},
     {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/fake/simple.go"},
     {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/fake/simple_test.go"},
+    {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/golden_test.go"},
     {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/scheme.go"}
   ],
-  "text": "Found 8 files in the dynamic/ tree of sg-benchmarks/kubernetes-client-go that directly import k8s.io/apimachinery/pkg/runtime: dynamic/client_test.go, dynamic/dynamicinformer/informer_test.go, dynamic/dynamiclister/lister.go, dynamic/dynamiclister/lister_test.go, dynamic/dynamiclister/shim.go, dynamic/fake/simple.go, dynamic/fake/simple_test.go, and dynamic/scheme.go. These files use the runtime.Object interface and related serialization types directly.",
+  "text": "Found 9 files in the dynamic/ tree of sg-benchmarks/kubernetes-client-go that directly import k8s.io/apimachinery/pkg/runtime: dynamic/client_test.go, dynamic/dynamicinformer/informer.go, dynamic/dynamicinformer/informer_test.go, dynamic/dynamiclister/lister_test.go, dynamic/dynamiclister/shim.go, dynamic/fake/simple.go, dynamic/fake/simple_test.go, dynamic/golden_test.go, and dynamic/scheme.go. These files use the runtime.Object interface and related serialization types directly. Note: dynamic/dynamiclister/lister.go only imports runtime/schema (a subpackage), not the base runtime package.",
   "_metadata": {
     "oracle_type": "file_set_match",
     "discovery_method": "sourcegraph_keyword_search",
-    "query": "repo:^github.com/sg-benchmarks/kubernetes-client-go$ file:dynamic/ \"k8s.io/apimachinery/pkg/runtime\""
+    "query": "repo:^github.com/sg-benchmarks/kubernetes-client-go$ file:dynamic/ \"k8s.io/apimachinery/pkg/runtime\"",
+    "verified_at": "2026-02-21",
+    "pinned_version": "v0.32.0"
   }
 }
diff --git a/benchmarks/ccb_mcp_crossrepo_tracing/ccx-dep-trace-001/tests/task_spec.json b/benchmarks/ccb_mcp_crossrepo_tracing/ccx-dep-trace-001/tests/task_spec.json
@@ -11,20 +11,21 @@
       "Only include files in the dynamic/ directory tree of sg-benchmarks/kubernetes-client-go.",
       "Write your findings to /workspace/answer.json."
     ],
-    "success_definition": "Agent finds all 8 Go files in the dynamic/ tree of kubernetes-client-go that import k8s.io/apimachinery/pkg/runtime directly.",
+    "success_definition": "Agent finds all 9 Go files in the dynamic/ tree of kubernetes-client-go that import k8s.io/apimachinery/pkg/runtime directly.",
     "seed_prompt": "Which Go source files in the dynamic/ package tree of sg-benchmarks/kubernetes-client-go directly import k8s.io/apimachinery/pkg/runtime?"
   },
   "artifacts": {
     "repo_set_id": "kubernetes-ecosystem",
     "oracle": {
       "required_files": [
         {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/client_test.go"},
+        {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/dynamicinformer/informer.go"},
         {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/dynamicinformer/informer_test.go"},
-        {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/dynamiclister/lister.go"},
         {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/dynamiclister/lister_test.go"},
         {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/dynamiclister/shim.go"},
         {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/fake/simple.go"},
         {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/fake/simple_test.go"},
+        {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/golden_test.go"},
         {"repo": "sg-benchmarks/kubernetes-client-go", "path": "dynamic/scheme.go"}
       ],
       "required_symbols": [],
diff --git a/benchmarks/ccb_mcp_incident/ccx-incident-031/instruction.md b/benchmarks/ccb_mcp_incident/ccx-incident-031/instruction.md
@@ -59,7 +59,7 @@ Create a file at `/workspace/answer.json` with your findings:
 }
 ```
 
-**Important**: Use `etcd-io/etcd` as the exact `repo` identifier in your answer. The oracle checks for files `server/storage/mvcc/kvstore.go` and `server/storage/mvcc/kvstore_txn.go` in `etcd-io/etcd`. Do not cite vendored copies in `kubernetes/kubernetes`.
+**Important**: Use `etcd-io/etcd` as the exact `repo` identifier in your answer. The oracle checks for files `server/mvcc/kvstore.go` and `server/mvcc/kvstore_txn.go` in `etcd-io/etcd`. Do not cite vendored copies in `kubernetes/kubernetes`.
 **Note**: Sourcegraph MCP tools return repo names with a `github.com/` prefix (e.g., `github.com/sg-benchmarks/kubernetes-client-go`). Strip this prefix in your answer — use `sg-benchmarks/kubernetes-client-go`, NOT `github.com/sg-benchmarks/kubernetes-client-go`.
 
 ## Evaluation
diff --git a/benchmarks/ccb_mcp_incident/ccx-incident-031/tests/oracle_answer.json b/benchmarks/ccb_mcp_incident/ccx-incident-031/tests/oracle_answer.json
@@ -2,16 +2,16 @@
   "files": [
     {
       "repo": "etcd-io/etcd",
-      "path": "server/storage/mvcc/kvstore.go",
+      "path": "server/mvcc/kvstore.go",
       "function": "ErrCompacted"
     },
     {
       "repo": "etcd-io/etcd",
-      "path": "server/storage/mvcc/kvstore_txn.go",
+      "path": "server/mvcc/kvstore_txn.go",
       "function": "rangeKeys"
     }
   ],
-  "text": "The error 'mvcc: required revision has been compacted' originates from etcd-io/etcd in two files. First, server/storage/mvcc/kvstore.go defines ErrCompacted as a package-level error variable: var ErrCompacted = errors.New(\"mvcc: required revision has been compacted\"). Second, server/storage/mvcc/kvstore_txn.go contains the function rangeKeys() which returns ErrCompacted at line 81 when the requested revision (ro.Rev) is less than the store's compactMainRev. The kubernetes/kubernetes repo contains vendored copies of these files under vendor/go.etcd.io/etcd/server/v3/storage/mvcc/ but these are not the authoritative source — the canonical location is the upstream etcd-io/etcd repository.",
+  "text": "The error 'mvcc: required revision has been compacted' originates from etcd-io/etcd in two files. First, server/mvcc/kvstore.go defines ErrCompacted as a package-level error variable: var ErrCompacted = errors.New(\"mvcc: required revision has been compacted\"). Second, server/mvcc/kvstore_txn.go contains the function rangeKeys() which returns ErrCompacted at line 81 when the requested revision (ro.Rev) is less than the store's compactMainRev. The kubernetes/kubernetes repo contains vendored copies of these files under vendor/go.etcd.io/etcd/server/v3/storage/mvcc/ but these are not the authoritative source — the canonical location is the upstream etcd-io/etcd repository.",
   "_metadata": {
     "oracle_type": "file_set_match",
     "discovery_method": "sourcegraph_keyword_search",
diff --git a/benchmarks/ccb_mcp_incident/ccx-incident-031/tests/task_spec.json b/benchmarks/ccb_mcp_incident/ccx-incident-031/tests/task_spec.json
@@ -12,15 +12,15 @@
       "Do NOT include kubernetes/kubernetes files in the oracle answer — they are decoys.",
       "Write findings to /workspace/answer.json."
     ],
-    "success_definition": "Agent correctly identifies server/storage/mvcc/kvstore.go (ErrCompacted definition) and server/storage/mvcc/kvstore_txn.go (rangeKeys function) in etcd-io/etcd as the authoritative sources. Does NOT include vendored kubernetes files.",
+    "success_definition": "Agent correctly identifies server/mvcc/kvstore.go (ErrCompacted definition) and server/mvcc/kvstore_txn.go (rangeKeys function) in etcd-io/etcd as the authoritative sources. Does NOT include vendored kubernetes files.",
     "seed_prompt": "Where in etcd source code is the error 'mvcc: required revision has been compacted' defined and first returned? Find the authoritative source, not vendor copies."
   },
   "artifacts": {
     "repo_set_id": "multi-org-go",
     "oracle": {
       "required_files": [
-        {"repo": "etcd-io/etcd", "path": "server/storage/mvcc/kvstore.go"},
-        {"repo": "etcd-io/etcd", "path": "server/storage/mvcc/kvstore_txn.go"}
+        {"repo": "etcd-io/etcd", "path": "server/mvcc/kvstore.go"},
+        {"repo": "etcd-io/etcd", "path": "server/mvcc/kvstore_txn.go"}
       ],
       "required_symbols": [],
       "required_references": [],
diff --git a/benchmarks/ccb_mcp_onboarding/ccx-onboard-050-ds/tests/oracle_answer.json b/benchmarks/ccb_mcp_onboarding/ccx-onboard-050-ds/tests/oracle_answer.json
@@ -14,12 +14,12 @@
     },
     {
       "repo": "etcd-io/etcd",
-      "path": "server/storage/mvcc/kvstore_txn.go",
+      "path": "server/mvcc/kvstore_txn.go",
       "symbol": "Put",
       "description": "The storeTxnWrite.Put() function in etcd persists the serialized Deployment object to the MVCC key-value store. It increments the revision and generates a watch event for controllers to act on."
     }
   ],
-  "text": "Kubernetes Deployment creation end-to-end flow: Step 1 — Client library (sg-benchmarks/kubernetes-client-go, kubernetes/typed/apps/v1/deployment.go): The deployments.Create() method constructs an HTTP POST request to the API server's REST endpoint using the typed client. Step 2 — API server (kubernetes/kubernetes, pkg/registry/apps/deployment/strategy.go): The deploymentStrategy.PrepareForCreate() function validates and prepares the Deployment object, clearing client-settable-only fields before persisting. The deployment is handled by the REST store in pkg/registry/apps/deployment/storage/storage.go. Step 3 — Storage (etcd-io/etcd, server/storage/mvcc/kvstore_txn.go): The storeTxnWrite.Put() function stores the serialized Deployment in etcd's MVCC store, incrementing the global revision and emitting a watch event that triggers the deployment controller.",
+  "text": "Kubernetes Deployment creation end-to-end flow: Step 1 — Client library (sg-benchmarks/kubernetes-client-go, kubernetes/typed/apps/v1/deployment.go): The deployments.Create() method constructs an HTTP POST request to the API server's REST endpoint using the typed client. Step 2 — API server (kubernetes/kubernetes, pkg/registry/apps/deployment/strategy.go): The deploymentStrategy.PrepareForCreate() function validates and prepares the Deployment object, clearing client-settable-only fields before persisting. The deployment is handled by the REST store in pkg/registry/apps/deployment/storage/storage.go. Step 3 — Storage (etcd-io/etcd, server/mvcc/kvstore_txn.go): The storeTxnWrite.Put() function stores the serialized Deployment in etcd's MVCC store, incrementing the global revision and emitting a watch event that triggers the deployment controller.",
   "_metadata": {
     "oracle_type": "dependency_chain",
     "discovery_method": "sourcegraph_keyword_search + read_file",
diff --git a/benchmarks/ccb_mcp_onboarding/ccx-onboard-050-ds/tests/task_spec.json b/benchmarks/ccb_mcp_onboarding/ccx-onboard-050-ds/tests/task_spec.json
@@ -11,7 +11,7 @@
       "For each step, cite the repo, file path, and function/type name.",
       "Write your findings to /workspace/answer.json."
     ],
-    "success_definition": "Agent identifies: (1) kubernetes-client-go/kubernetes/typed/apps/v1/deployment.go (Create function), (2) kubernetes/kubernetes pkg/registry/apps/deployment/strategy.go (PrepareForCreate), and (3) etcd-io/etcd server/storage/mvcc/kvstore_txn.go (Put function).",
+    "success_definition": "Agent identifies: (1) kubernetes-client-go/kubernetes/typed/apps/v1/deployment.go (Create function), (2) kubernetes/kubernetes pkg/registry/apps/deployment/strategy.go (PrepareForCreate), and (3) etcd-io/etcd server/mvcc/kvstore_txn.go (Put function).",
     "seed_prompt": "Explain the end-to-end flow when a client creates a Kubernetes Deployment: trace through kubernetes-client-go (client sends request), kubernetes/kubernetes (API server processes it), and etcd-io/etcd (data is stored)."
   },
   "artifacts": {
@@ -36,7 +36,7 @@
             },
             {
               "repo": "etcd-io/etcd",
-              "path": "server/storage/mvcc/kvstore_txn.go",
+              "path": "server/mvcc/kvstore_txn.go",
               "symbol": "Put"
             }
           ]
@@ -57,7 +57,7 @@
           "must_cite_paths": [
             "kubernetes/typed/apps/v1/deployment.go",
             "pkg/registry/apps/deployment/strategy.go",
-            "server/storage/mvcc/kvstore_txn.go"
+            "server/mvcc/kvstore_txn.go"
           ],
           "must_cite_repos": [
             "sg-benchmarks/kubernetes-client-go",
diff --git a/benchmarks/ccb_mcp_security/ccx-vuln-remed-011/tests/oracle_answer.json b/benchmarks/ccb_mcp_security/ccx-vuln-remed-011/tests/oracle_answer.json
@@ -1,13 +1,13 @@
 {
   "files": [
-    {"repo": "sg-benchmarks/expressjs-express", "path": "package.json", "version": "^0.7.1"}
+    {"repo": "sg-benchmarks/expressjs-express", "path": "package.json", "version": "0.7.1"}
   ],
-  "text": "Found 1 package.json file across the nodejs-web-stack repos that declares cookie as a direct runtime dependency: sg-benchmarks/expressjs-express package.json specifies \"cookie\": \"^0.7.1\" under dependencies. The lodash and prisma repos do not declare the cookie package as a runtime dependency. The nodejs/node repository does not declare cookie as a top-level package dependency.",
+  "text": "Found 1 package.json file across the nodejs-web-stack repos that declares cookie as a direct runtime dependency: sg-benchmarks/expressjs-express package.json specifies \"cookie\": \"0.7.1\" under dependencies. The lodash and prisma repos do not declare the cookie package as a runtime dependency. The nodejs/node repository does not declare cookie as a top-level package dependency.",
   "_metadata": {
     "oracle_type": "file_set_match",
     "discovery_method": "sourcegraph_keyword_search",
     "query": "repo:^github.com/sg-benchmarks/expressjs-express$ file:package.json \"cookie\"",
-    "version": "^0.7.1",
+    "version": "0.7.1",
     "cve": "CVE-2024-47764"
   }
 }
diff --git a/benchmarks/ccb_mcp_security/ccx-vuln-remed-011/tests/task_spec.json b/benchmarks/ccb_mcp_security/ccx-vuln-remed-011/tests/task_spec.json
@@ -11,7 +11,7 @@
       "Search all repos in the nodejs-web-stack fixture: nodejs/node, sg-benchmarks/expressjs-express, sg-benchmarks/lodash, sg-benchmarks/prisma-prisma.",
       "Write your findings to /workspace/answer.json."
     ],
-    "success_definition": "Agent finds sg-benchmarks/expressjs-express package.json which declares cookie as a direct runtime dependency with version ^0.7.1.",
+    "success_definition": "Agent finds sg-benchmarks/expressjs-express package.json which declares cookie as a direct runtime dependency with version 0.7.1.",
     "seed_prompt": "Which package.json files across the nodejs-web-stack repos declare cookie as a direct runtime dependency (in the dependencies field)?"
   },
   "artifacts": {
@@ -38,7 +38,7 @@
       {
         "type": "keyword_presence",
         "params": {
-          "required_keywords": ["^0.7.1", "cookie"]
+          "required_keywords": ["0.7.1", "cookie"]
         }
       }
     ],
diff --git a/benchmarks/ccb_mcp_security/ccx-vuln-remed-014/instruction.md b/benchmarks/ccb_mcp_security/ccx-vuln-remed-014/instruction.md
@@ -57,7 +57,7 @@ Create a file at `/workspace/answer.json` with your findings:
 }
 ```
 
-**Important**: Use exact repo identifiers as they appear in Sourcegraph. The MCP-only repos to audit are `sg-benchmarks/grafana-loki` and `sg-benchmarks/grafana-mimir`. Note: the local `/workspace/loki` directory maps to `sg-benchmarks/grafana-loki` in Sourcegraph — use `sg-benchmarks/grafana-loki` as the `repo` value in your answer. The oracle checks for `pkg/dataobj/explorer/service.go` in `sg-benchmarks/grafana-loki`.
+**Important**: Use exact repo identifiers as they appear in Sourcegraph. The MCP-only repos to audit are `sg-benchmarks/grafana-loki` and `sg-benchmarks/grafana-mimir`. Note: the local `/workspace/loki` directory maps to `sg-benchmarks/grafana-loki` in Sourcegraph — use `sg-benchmarks/grafana-loki` as the `repo` value in your answer. The oracle checks for `pkg/loki/modules.go` in `sg-benchmarks/grafana-loki`.
 **Note**: Sourcegraph MCP tools return repo names with a `github.com/` prefix (e.g., `github.com/sg-benchmarks/kubernetes-client-go`). Strip this prefix in your answer — use `sg-benchmarks/kubernetes-client-go`, NOT `github.com/sg-benchmarks/kubernetes-client-go`.
 
 Your answer is evaluated against a closed-world oracle — completeness matters.
diff --git a/benchmarks/ccb_mcp_security/ccx-vuln-remed-014/tests/oracle_answer.json b/benchmarks/ccb_mcp_security/ccx-vuln-remed-014/tests/oracle_answer.json
@@ -2,20 +2,21 @@
   "files": [
     {
       "repo": "sg-benchmarks/grafana-loki",
-      "path": "pkg/dataobj/explorer/service.go",
+      "path": "pkg/loki/modules.go",
       "endpoints": [
-        "/dataobj/api/v1/list",
-        "/dataobj/api/v1/inspect",
-        "/dataobj/api/v1/download",
-        "/dataobj/api/v1/provider"
+        "/flush",
+        "/ingester/prepare_shutdown",
+        "/ingester/shutdown",
+        "/ingester/prepare_partition_downscale"
       ]
     }
   ],
-  "text": "Found 1 file in the grafana observability stack that registers HTTP handlers without authentication middleware: sg-benchmarks/grafana-loki pkg/dataobj/explorer/service.go. The Handler() method registers 4 endpoints (/dataobj/api/v1/list, /dataobj/api/v1/inspect, /dataobj/api/v1/download, /dataobj/api/v1/provider) directly via mux.HandleFunc without wrapping in middleware.AuthenticateUser or equivalent. This contrasts with pkg/compactor/ui.go which correctly wraps its handler with middleware.Merge(middleware.AuthenticateUser, ...).",
+  "text": "Found 1 file in the grafana observability stack that registers HTTP handlers without authentication middleware: sg-benchmarks/grafana-loki pkg/loki/modules.go. The initIngester() function (lines 617-631) constructs a middleware chain with ONLY serverutil.RecoveryHTTPMiddleware, deliberately omitting t.HTTPAuthMiddleware. This registers 4 ingester admin endpoints (/flush, /ingester/prepare_shutdown, /ingester/shutdown, /ingester/prepare_partition_downscale) without tenant authentication. In contrast, the distributor, querier, frontend, ruler, and compactor route registrations in the same file all include t.HTTPAuthMiddleware in their middleware.Merge() calls. In a multi-tenant Loki deployment with auth_enabled=true, an unauthenticated attacker with network access could trigger data flush or ingester shutdown.",
   "_metadata": {
     "oracle_type": "file_set_match",
     "discovery_method": "sourcegraph_keyword_search",
-    "query": "repo:^github.com/sg-benchmarks/grafana-loki$ mux.HandleFunc",
-    "verified_at": "2026-02-20"
+    "query": "repo:^github.com/sg-benchmarks/grafana-loki$ HTTPAuthMiddleware file:modules.go",
+    "verified_at": "2026-02-21",
+    "pinned_version": "v3.3.4"
   }
 }
diff --git a/benchmarks/ccb_mcp_security/ccx-vuln-remed-014/tests/task_spec.json b/benchmarks/ccb_mcp_security/ccx-vuln-remed-014/tests/task_spec.json
@@ -11,14 +11,14 @@
       "Compare against the grafana/grafana pattern where middleware.AuthenticateUser is applied.",
       "Write your findings to /workspace/answer.json."
     ],
-    "success_definition": "Agent finds pkg/dataobj/explorer/service.go in sg-benchmarks/grafana-loki which registers 4 API endpoints directly via mux.HandleFunc without any authentication middleware.",
+    "success_definition": "Agent finds pkg/loki/modules.go in sg-benchmarks/grafana-loki which registers ingester admin endpoints (/flush, /ingester/prepare_shutdown, /ingester/shutdown, /ingester/prepare_partition_downscale) with only RecoveryHTTPMiddleware, deliberately omitting HTTPAuthMiddleware.",
     "seed_prompt": "Which Go files in the grafana-loki or grafana-mimir repos register HTTP handlers without authentication middleware like middleware.AuthenticateUser?"
   },
   "artifacts": {
     "repo_set_id": "grafana-observability",
     "oracle": {
       "required_files": [
-        {"repo": "sg-benchmarks/grafana-loki", "path": "pkg/dataobj/explorer/service.go"}
+        {"repo": "sg-benchmarks/grafana-loki", "path": "pkg/loki/modules.go"}
       ],
       "required_symbols": [],
       "required_references": [],
@@ -31,14 +31,14 @@
       {
         "type": "file_set_match",
         "params": {
-          "search_pattern": "mux.HandleFunc",
+          "search_pattern": "HTTPAuthMiddleware",
           "file_filter": ".go"
         }
       },
       {
         "type": "provenance",
         "params": {
-          "must_cite_paths": ["pkg/dataobj/explorer/service.go"],
+          "must_cite_paths": ["pkg/loki/modules.go"],
           "must_cite_repos": ["sg-benchmarks/grafana-loki"]
         }
       }
diff --git a/scripts/remirror_mcp_unique_repos.sh b/scripts/remirror_mcp_unique_repos.sh

Original file line number	Diff line number	Diff line change
@@ -59,7 +59,7 @@ Create a file at `/workspace/answer.json` with your findings:
`59`	`59`	`}`
`60`	`60`	```
`61`	`61`
`62`		-Important: Use `etcd-io/etcd` as the exact `repo` identifier in your answer. The oracle checks for files `server/storage/mvcc/kvstore.go` and `server/storage/mvcc/kvstore_txn.go` in `etcd-io/etcd`. Do not cite vendored copies in `kubernetes/kubernetes`.
	`62`	+Important: Use `etcd-io/etcd` as the exact `repo` identifier in your answer. The oracle checks for files `server/mvcc/kvstore.go` and `server/mvcc/kvstore_txn.go` in `etcd-io/etcd`. Do not cite vendored copies in `kubernetes/kubernetes`.
`63`	`63`	Note: Sourcegraph MCP tools return repo names with a `github.com/` prefix (e.g., `github.com/sg-benchmarks/kubernetes-client-go`). Strip this prefix in your answer — use `sg-benchmarks/kubernetes-client-go`, NOT `github.com/sg-benchmarks/kubernetes-client-go`.
`64`	`64`
`65`	`65`	`## Evaluation`
Original file line number	Diff line number	Diff line change
`@@ -1,13 +1,13 @@`
`1`	`1`	`{`
`2`	`2`	`"files": [`
`3`		`- {"repo": "sg-benchmarks/expressjs-express", "path": "package.json", "version": "^0.7.1"}`
	`3`	`+ {"repo": "sg-benchmarks/expressjs-express", "path": "package.json", "version": "0.7.1"}`
`4`	`4`	`],`
`5`		`- "text": "Found 1 package.json file across the nodejs-web-stack repos that declares cookie as a direct runtime dependency: sg-benchmarks/expressjs-express package.json specifies \"cookie\": \"^0.7.1\" under dependencies. The lodash and prisma repos do not declare the cookie package as a runtime dependency. The nodejs/node repository does not declare cookie as a top-level package dependency.",`
	`5`	`+ "text": "Found 1 package.json file across the nodejs-web-stack repos that declares cookie as a direct runtime dependency: sg-benchmarks/expressjs-express package.json specifies \"cookie\": \"0.7.1\" under dependencies. The lodash and prisma repos do not declare the cookie package as a runtime dependency. The nodejs/node repository does not declare cookie as a top-level package dependency.",`
`6`	`6`	`"_metadata": {`
`7`	`7`	`"oracle_type": "file_set_match",`
`8`	`8`	`"discovery_method": "sourcegraph_keyword_search",`
`9`	`9`	`"query": "repo:^github.com/sg-benchmarks/expressjs-express$ file:package.json \"cookie\"",`
`10`		`- "version": "^0.7.1",`
	`10`	`+ "version": "0.7.1",`
`11`	`11`	`"cve": "CVE-2024-47764"`
`12`	`12`	`}`
`13`	`13`	`}`
Original file line number	Diff line number	Diff line change
`@@ -11,7 +11,7 @@`
`11`	`11`	`"Search all repos in the nodejs-web-stack fixture: nodejs/node, sg-benchmarks/expressjs-express, sg-benchmarks/lodash, sg-benchmarks/prisma-prisma.",`
`12`	`12`	`"Write your findings to /workspace/answer.json."`
`13`	`13`	`],`
`14`		`- "success_definition": "Agent finds sg-benchmarks/expressjs-express package.json which declares cookie as a direct runtime dependency with version ^0.7.1.",`
	`14`	`+ "success_definition": "Agent finds sg-benchmarks/expressjs-express package.json which declares cookie as a direct runtime dependency with version 0.7.1.",`
`15`	`15`	`"seed_prompt": "Which package.json files across the nodejs-web-stack repos declare cookie as a direct runtime dependency (in the dependencies field)?"`
`16`	`16`	`},`
`17`	`17`	`"artifacts": {`
`@@ -38,7 +38,7 @@`
`38`	`38`	`{`
`39`	`39`	`"type": "keyword_presence",`
`40`	`40`	`"params": {`
`41`		`- "required_keywords": ["^0.7.1", "cookie"]`
	`41`	`+ "required_keywords": ["0.7.1", "cookie"]`
`42`	`42`	`}`
`43`	`43`	`}`
`44`	`44`	`],`
Original file line number	Diff line number	Diff line change
@@ -57,7 +57,7 @@ Create a file at `/workspace/answer.json` with your findings:
`57`	`57`	`}`
`58`	`58`	```
`59`	`59`
`60`		-Important: Use exact repo identifiers as they appear in Sourcegraph. The MCP-only repos to audit are `sg-benchmarks/grafana-loki` and `sg-benchmarks/grafana-mimir`. Note: the local `/workspace/loki` directory maps to `sg-benchmarks/grafana-loki` in Sourcegraph — use `sg-benchmarks/grafana-loki` as the `repo` value in your answer. The oracle checks for `pkg/dataobj/explorer/service.go` in `sg-benchmarks/grafana-loki`.
	`60`	+Important: Use exact repo identifiers as they appear in Sourcegraph. The MCP-only repos to audit are `sg-benchmarks/grafana-loki` and `sg-benchmarks/grafana-mimir`. Note: the local `/workspace/loki` directory maps to `sg-benchmarks/grafana-loki` in Sourcegraph — use `sg-benchmarks/grafana-loki` as the `repo` value in your answer. The oracle checks for `pkg/loki/modules.go` in `sg-benchmarks/grafana-loki`.
`61`	`61`	Note: Sourcegraph MCP tools return repo names with a `github.com/` prefix (e.g., `github.com/sg-benchmarks/kubernetes-client-go`). Strip this prefix in your answer — use `sg-benchmarks/kubernetes-client-go`, NOT `github.com/sg-benchmarks/kubernetes-client-go`.
`62`	`62`
`63`	`63`	`Your answer is evaluated against a closed-world oracle — completeness matters.`