Skip to content

Commit 32b7c26

Browse files
sjarmakclaude
andcommitted
feat: 5-task pilot cross-validation — agent oracle vs SG oracle
Pilot results (hybrid backend, Sonnet 4.6): - ccx-dep-trace-116: F1=1.000 (3/3 files, k8s TypeMeta chain) - ccx-dep-trace-176: F1=0.160 (4/12 matched, Envoy over-retrieved 38) - ccx-domain-071: F1=0.875 (7/9 matched, Kafka acks chain) - ccx-migration-022: F1=0.933 (7/7+1 extra, Kafka deprecated API) - ccx-vuln-remed-011: F1=0.000 (fixture gap — express not in repo set) Mean F1 (excl fixture-gap): 0.742 Also adds path-suffix fallback to cross_validate_oracles.py for monorepo staging→published repo path matching. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent d9309fb commit 32b7c26

File tree

6 files changed

+1056
-3
lines changed

6 files changed

+1056
-3
lines changed
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
{
2+
"files": [
3+
{
4+
"repo": "kubernetes/kubernetes",
5+
"path": "staging/src/k8s.io/api/core/v1/types.go"
6+
},
7+
{
8+
"repo": "kubernetes/api",
9+
"path": "core/v1/types.go"
10+
},
11+
{
12+
"repo": "kubernetes/kubernetes",
13+
"path": "staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go"
14+
}
15+
],
16+
"symbols": [
17+
{
18+
"repo": "kubernetes/kubernetes",
19+
"path": "staging/src/k8s.io/api/core/v1/types.go",
20+
"symbol": "Pod"
21+
},
22+
{
23+
"repo": "kubernetes/kubernetes",
24+
"path": "staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go",
25+
"symbol": "TypeMeta"
26+
},
27+
{
28+
"repo": "kubernetes/api",
29+
"path": "core/v1/types.go",
30+
"symbol": "Pod"
31+
}
32+
],
33+
"chain": [
34+
{
35+
"repo": "kubernetes/kubernetes",
36+
"path": "staging/src/k8s.io/api/core/v1/types.go",
37+
"symbol": "Pod"
38+
},
39+
{
40+
"repo": "kubernetes/api",
41+
"path": "core/v1/types.go",
42+
"symbol": "Pod"
43+
},
44+
{
45+
"repo": "kubernetes/kubernetes",
46+
"path": "staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go",
47+
"symbol": "TypeMeta"
48+
}
49+
],
50+
"text": "Dependency chain traced across three locations:\n\n1. **kubernetes/kubernetes** \u2014 `staging/src/k8s.io/api/core/v1/types.go`:\n - Line 21: `metav1 \"k8s.io/apimachinery/pkg/apis/meta/v1\"` \u2014 import alias is `metav1`, path is `k8s.io/apimachinery/pkg/apis/meta/v1`.\n - Line 5012: `type Pod struct {`\n - Line 5013: `metav1.TypeMeta \\`json:\",inline\"\\`` \u2014 Pod embeds TypeMeta via the `metav1` alias.\n\n2. **kubernetes/api** (sg-evals__api--v0.32.0) \u2014 `core/v1/types.go`:\n - This is the published `k8s.io/api` module (a mirror of the staging area).\n - Line 21: `metav1 \"k8s.io/apimachinery/pkg/apis/meta/v1\"` \u2014 same import alias and path.\n - Line 5012\u20135013: `type Pod struct { metav1.TypeMeta ... }` \u2014 re-exposes TypeMeta through embedding.\n - The `kubernetes/api` module does NOT have its own `meta/v1` package; it re-exports TypeMeta by depending on `k8s.io/apimachinery v0.32.0` (see go.mod line 13).\n\n3. **kubernetes/kubernetes** staging for apimachinery \u2014 `staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go`:\n - Line 42: `type TypeMeta struct {` \u2014 authoritative definition.\n - Line 49: `Kind string \\`json:\"kind,omitempty\" protobuf:\"bytes,1,opt,name=kind\"\\``\n - Line 56: `APIVersion string \\`json:\"apiVersion,omitempty\" protobuf:\"bytes,2,opt,name=apiVersion\"\\``\n - Package comment (line 20) explicitly notes: 'external (serialized) types that lack their own version (e.g TypeMeta)'.",
51+
"_metadata": {
52+
"generator": "context_retrieval_agent",
53+
"model": "claude-sonnet-4-6",
54+
"backend": "hybrid",
55+
"input_tokens": 151568,
56+
"output_tokens": 5341,
57+
"tool_calls": 35,
58+
"elapsed_sec": 68.2,
59+
"cost_usd": 0.5348,
60+
"timestamp": "2026-03-01T23:34:20.785209+00:00"
61+
}
62+
}

0 commit comments

Comments
 (0)