Skip to content

Commit 937eb49

Browse files
dfa1claude
andcommitted
raincloud: triage 34 slugs, fix VARIANT oracle crash, matrix at 158 ok
- Oracle guard: hardwood's NestedRowReader.getStruct() mis-casts a VARIANT-shredded group (metadata/value/typed_value) to its plain Struct FieldDesc and throws ClassCastException — an oracle bug, not a vortex-java gap. RaincloudConformanceIntegrationTest now aborts on group.isVariant() before hitting it, same as the existing isMap() guard (unblocks countries-of-the-world, which regressed to a hard failure once the corpus was rebuilt with a VARIANT column). - Triage: 32 untriaged slugs verified ok against the parquet oracle (13 large BI slugs + 19 smaller HF/benchmark slugs). squad-v2 and mnist flip from gap:268 to ok now that the previous commit's fix lands. oasst1 hits a different, new bug — a null Struct row renders as {"field":null,...} instead of an empty cell (#270, filed) — so it moves from gap:268 to gap:270 rather than to ok. Matrix: 124 ok / 0 gaps / 123 untriaged -> 158 ok / 1 gap / 85 untriaged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 parent d320d42 commit 937eb49

2 files changed

Lines changed: 41 additions & 35 deletions

File tree

integration/src/test/java/io/github/dfa1/vortex/integration/RaincloudConformanceIntegrationTest.java

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -366,6 +366,12 @@ private static String oracleCell(SchemaNode node, RowReader rows, int fieldIndex
366366
if (group.isMap()) {
367367
throw new TestAbortedException("oracle cannot format MAP column: " + group.name());
368368
}
369+
// hardwood's own NestedRowReader.getStruct() mis-casts a VARIANT-shredded group
370+
// (metadata/value/typed_value) to its plain-Struct FieldDesc and throws
371+
// ClassCastException — an oracle bug, not a vortex-java gap. Abort before hitting it.
372+
if (group.isVariant()) {
373+
throw new TestAbortedException("oracle cannot format VARIANT column: " + group.name());
374+
}
369375
return oracleJsonObject(group, rows.getStruct(fieldIndex));
370376
}
371377
SchemaNode.PrimitiveNode prim = (SchemaNode.PrimitiveNode) node;

integration/src/test/resources/raincloud/expected-status.csv

Lines changed: 35 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -29,11 +29,11 @@ aya-dataset,ok
2929
bank-account-fraud-dataset-neurips-2022,missing_auth
3030
basketball,untriaged
3131
behavioral-risk-factor-surveillance-system,ok
32-
beir-msmarco,untriaged
32+
beir-msmarco,ok
3333
bi-arade,ok
3434
bi-bimbo,ok
3535
bi-citymaxcapita,ok
36-
bi-cmsprovider,untriaged
36+
bi-cmsprovider,ok
3737
bi-commongovernment,untriaged
3838
bi-corporations,ok
3939
bi-eixo,ok
@@ -43,28 +43,28 @@ bi-generico,untriaged
4343
bi-hashtags,ok
4444
bi-hatred,ok
4545
bi-iglocations1,ok
46-
bi-iglocations2,untriaged
46+
bi-iglocations2,ok
4747
bi-iublibrary,ok
48-
bi-medicare1,untriaged
49-
bi-medicare2,untriaged
48+
bi-medicare1,ok
49+
bi-medicare2,ok
5050
bi-medicare3,ok
5151
bi-medpayment1,ok
5252
bi-medpayment2,ok
53-
bi-mlb,untriaged
54-
bi-motos,untriaged
53+
bi-mlb,ok
54+
bi-motos,ok
5555
bi-mulheresmil,ok
56-
bi-nyc,untriaged
56+
bi-nyc,ok
5757
bi-pancreactomy1,ok
58-
bi-pancreactomy2,untriaged
58+
bi-pancreactomy2,ok
5959
bi-physicians,ok
6060
bi-provider,untriaged
6161
bi-realestate1,untriaged
6262
bi-realestate2,untriaged
63-
bi-redfin1,untriaged
64-
bi-redfin2,untriaged
65-
bi-redfin3,untriaged
63+
bi-redfin1,ok
64+
bi-redfin2,ok
65+
bi-redfin3,ok
6666
bi-redfin4,ok
67-
bi-rentabilidad,untriaged
67+
bi-rentabilidad,ok
6868
bi-romance,ok
6969
bi-salariesfrance,untriaged
7070
bi-tablerosistemapenal,ok
@@ -76,7 +76,7 @@ bi-uberlandia,ok
7676
bi-uscensus,untriaged
7777
bi-wins,untriaged
7878
bi-yalelanguages,ok
79-
bigscience-p3-eval-subset,untriaged
79+
bigscience-p3-eval-subset,ok
8080
building-permit-applications-data,untriaged
8181
c4-en-validation,untriaged
8282
california-housing-prices,ok
@@ -86,14 +86,14 @@ clickbench-hits,untriaged
8686
cnn-dailymail,untriaged
8787
codeparrot-clean-valid,untriaged
8888
cohere-wikipedia-simple-embed,untriaged
89-
coig,untriaged
89+
coig,ok
9090
coronahack-chest-xraydataset,untriaged
9191
cosmopedia-stanford,untriaged
9292
countries-of-the-world,ok
9393
covid-world-vaccination-progress,ok
9494
covid19-data-from-john-hopkins-university,untriaged
9595
crimes-in-boston,untriaged
96-
databricks-dolly-15k,untriaged
96+
databricks-dolly-15k,ok
9797
dbpedia-embeddings,untriaged
9898
diabetes-health-indicators-dataset,ok
9999
disease-symptom-description-dataset,untriaged
@@ -103,13 +103,13 @@ emotions-dataset-for-nlp,ok
103103
fhv_tripdata_2025,ok
104104
fhvhv_tripdata_2025,untriaged
105105
finemath-4plus,untriaged
106-
finepdfs-en-test,untriaged
106+
finepdfs-en-test,ok
107107
finetranslations-swedish,untriaged
108108
fineweb-2-swedish,untriaged
109109
fineweb-sample-10bt,untriaged
110110
fitbit,untriaged
111111
formula-1-world-championship-1950-2020,untriaged
112-
frames-benchmark,untriaged
112+
frames-benchmark,ok
113113
ghcn-daily,untriaged
114114
glass,ok
115115
global-fossil-co2-emissions-by-country-2002-2022,untriaged
@@ -119,16 +119,16 @@ glove-6b-50d,ok
119119
goodbooks-10k,ok
120120
google-cluster-trace-2011-machine-events,ok
121121
green_tripdata_2025,ok
122-
gsm8k,untriaged
122+
gsm8k,ok
123123
gtsrb-german-traffic-sign,ok
124124
hacker-news,untriaged
125125
heart-disease-health-indicators-dataset,ok
126-
hellaswag,untriaged
127-
helpsteer2,untriaged
126+
hellaswag,ok
127+
helpsteer2,ok
128128
hotel-booking-demand,ok
129129
hotpotqa-fullwiki,untriaged
130130
housing-prices-dataset,untriaged
131-
humaneval,untriaged
131+
humaneval,ok
132132
insurance,ok
133133
international-football-results-from-1872-to-2017,ok
134134
iowa-liquor-sales,untriaged
@@ -139,24 +139,24 @@ librispeech-test-clean,untriaged
139139
loan-default-dataset,untriaged
140140
lung-cancer,ok
141141
marketing-data,ok
142-
mbpp,untriaged
143-
medmcqa,untriaged
142+
mbpp,ok
143+
medmcqa,ok
144144
mlcourse,untriaged
145-
mmlu,untriaged
146-
mmlu-pro,untriaged
147-
mmmlu,untriaged
145+
mmlu,ok
146+
mmlu-pro,ok
147+
mmmlu,ok
148148
mmmu,untriaged
149-
mnist,untriaged
149+
mnist,ok
150150
movies,ok
151151
new-york-city-airbnb-open-data,untriaged
152-
news-headlines-dataset-for-sarcasm-detection,untriaged
152+
news-headlines-dataset-for-sarcasm-detection,ok
153153
nips-papers,ok
154-
no-robots,untriaged
154+
no-robots,ok
155155
nyc-311,untriaged
156156
nyc-parking-tickets,untriaged
157157
nyc-property-sales,ok
158158
nypd-complaints,untriaged
159-
oasst1,untriaged
159+
oasst1,gap:270
160160
open-food-facts,untriaged
161161
openlibrary-authors,untriaged
162162
openlibrary-editions,untriaged
@@ -178,21 +178,21 @@ palmer-archipelago-antarctica-penguin-data,ok
178178
peoples-speech-clean-validation,untriaged
179179
personal-key-indicators-of-heart-disease,ok
180180
pleias-synth,untriaged
181-
pubmedqa-labeled,untriaged
181+
pubmedqa-labeled,ok
182182
sf-salaries,ok
183183
slimpajama-6b,untriaged
184-
squad-v2,untriaged
184+
squad-v2,ok
185185
stack-overflow-2018-developer-survey,ok
186186
stackoverflow-badges,untriaged
187187
stackoverflow-postlinks,ok
188188
stackoverflow-posts,untriaged
189189
stackoverflow-tags,ok
190190
stackoverflow-users,untriaged
191-
synthetic-text-to-sql,untriaged
191+
synthetic-text-to-sql,ok
192192
temperature-change,ok
193193
tinystories,untriaged
194194
titanic-dataset,untriaged
195-
truthfulqa-mc,untriaged
195+
truthfulqa-mc,ok
196196
uber-pickups-in-new-york-city,ok
197197
uci-abalone,ok
198198
uci-adult,ok

0 commit comments

Comments
 (0)