Skip to content

Commit b081a17

Browse files
dfa1claude
andcommitted
raincloud: add missing_auth status
Replace the ad-hoc "#-comment above an untriaged line" convention with a proper matrix status: missing_auth marks a slug that can't even be hydrated because its source (Kaggle, gated HuggingFace, ...) needs credentials this environment doesn't have. Distinguishes "blocked on a credential" from plain untriaged ("just hasn't been attempted") so future triagers don't waste time rediscovering the same blocker. Harness treats it identically to untriaged (reportUntriaged, not assertStillFails — that path means "known reader gap", which this isn't). Applied to the 3 Kaggle-blocked slugs from the last round. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 parent b1397b5 commit b081a17

2 files changed

Lines changed: 22 additions & 16 deletions

File tree

integration/src/test/java/io/github/dfa1/vortex/integration/RaincloudConformanceIntegrationTest.java

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,11 @@ Stream<DynamicTest> conformancePerSlug() throws IOException {
8787
String status = expected.getOrDefault(slug, "untriaged");
8888
switch (status) {
8989
case "ok" -> assertMatchesParquetOracle(vortex, parquet);
90-
case "untriaged" -> reportUntriaged(vortex, parquet);
90+
// missing_auth slugs never reach here in practice — hydration fails
91+
// before a manifest entry exists — but report the same as untriaged
92+
// rather than falling through to assertStillFails (wrong: that path
93+
// means "known reader gap", not "couldn't even attempt this").
94+
case "untriaged", "missing_auth" -> reportUntriaged(vortex, parquet);
9195
default -> assertStillFails(vortex, parquet, status);
9296
}
9397
}));

integration/src/test/resources/raincloud/expected-status.csv

Lines changed: 17 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,30 +1,32 @@
11
# Raincloud conformance matrix — one line per vortex-enabled corpus slug: <slug>,<status>
2-
# status: ok — vortex-java must read the file and match the parquet oracle exactly
3-
# gap:<n> — known reader gap tracked by issue #<n>; the test asserts the failure
4-
# still occurs, so a fix forces flipping the entry to ok in the same change
5-
# untriaged — not yet run; the test executes it and reports the outcome as an
6-
# aborted (skipped) test instead of failing the build — flip to ok/gap
7-
# as hydration coverage grows
8-
# A `#`-comment directly above an `untriaged` line notes WHY it hasn't been hydrated yet
9-
# (e.g. missing credentials for a gated source) — status stays literal "untriaged" (the
10-
# harness string-matches it exactly), the comment is for triagers only.
2+
# status: ok — vortex-java must read the file and match the parquet oracle exactly
3+
# gap:<n> — known reader gap tracked by issue #<n>; the test asserts the failure
4+
# still occurs, so a fix forces flipping the entry to ok in the same change
5+
# untriaged — not yet run; the test executes it and reports the outcome as an
6+
# aborted (skipped) test instead of failing the build — flip to ok/gap
7+
# as hydration coverage grows
8+
# missing_auth — can't even hydrate: the source (Kaggle, gated HuggingFace, …) needs
9+
# credentials this environment doesn't have. Distinguishes "blocked on
10+
# a credential" from plain untriaged ("just hasn't been attempted yet")
11+
# so a future triager doesn't waste time rediscovering the same blocker.
12+
# An optional `#`-comment above the line can name the credential.
1113
# Corpus: all slugs with a Vortex artifact in the Raincloud catalog (spiraldb/raincloud),
1214
# written by vortex-data 0.69.0. Hydrate any subset with scripts/hydrate-raincloud-corpus.sh.
1315
120-years-of-olympic-history-athletes-and-results,ok
1416
ai2-arc,ok
15-
# needs Kaggle credentials to hydrate (kaggle auth login / KAGGLE_API_TOKEN) — not attempted
16-
airbnb-prices-in-european-cities,untriaged
17-
# needs Kaggle credentials to hydrate (kaggle auth login / KAGGLE_API_TOKEN) — not attempted
18-
airbnbopendata,untriaged
17+
# kaggle auth login / KAGGLE_API_TOKEN
18+
airbnb-prices-in-european-cities,missing_auth
19+
# kaggle auth login / KAGGLE_API_TOKEN
20+
airbnbopendata,missing_auth
1921
amazon-reviews-2023-subscription-boxes,ok
2022
ampds-whe,ok
2123
anthropic-economic-index,ok
2224
anthropic-hh-rlhf-helpful-base,ok
2325
anthropic-interviewer,ok
2426
aya-collection-templated,ok
2527
aya-dataset,ok
26-
# needs Kaggle credentials to hydrate (kaggle auth login / KAGGLE_API_TOKEN) — not attempted
27-
bank-account-fraud-dataset-neurips-2022,untriaged
28+
# kaggle auth login / KAGGLE_API_TOKEN
29+
bank-account-fraud-dataset-neurips-2022,missing_auth
2830
basketball,untriaged
2931
behavioral-risk-factor-surveillance-system,ok
3032
beir-msmarco,untriaged

0 commit comments

Comments
 (0)