Commit ab4e7c6
committed
[analytics-engine] Address review: path-aware strip, bulk-error fail-loud, stronger verifier
Resolves the correctness holes raised in review.
Blocking 1 — path-aware mapping+source strip. stripUnsupportedMappingFields
now returns exact dropped PATHS (e.g. [location, point]) instead of only
top-level names; stripBulkFields removes those paths recursively from _source,
descending through objects AND arrays of objects, preserving siblings
(location.city etc.). Adds unit coverage for a complex_geo shape and arrays of
objects.
Blocking 2 — fail loud on bulk item failures. loadDataByRestClient now parses
the _bulk response and throws on errors=true (naming the index + first item
errors), for ALL test bulk loads. Silent partial ingestion no longer slips
through a 200 status.
Blocking 3 — verifier proves the invariant. AnalyticsUnsupportedFieldStripVerifyIT
deletes each index first so the real load path always runs (not short-circuited
by isIndexExist), then per dataset checks: clean live mapping, no stripped path
left in sampled _source, and doc-count == source-doc count (tolerating a
cluster-side count error on system indices with a transparent logged skip).
NB1 — GeoPointFormatsIT exclude moved under the parsed analyticsEnabled gate
(was gated on mere property presence, so =false wrongly excluded it).
NB2 — out-of-scope skip (join) now refuses to skip if the mapping also declares
any unsupported type, so a mixed failure can't be hidden.
binary is now conditional: kept when store:true (engine reads VARBINARY), but
stripped when store:false because the parquet store can't create it
("Unable to derive source for [X] with store disabled", verified on the AE
cluster). Confirmed end-to-end: the verifier passes over all Index datasets on a
force-routed AE cluster.
Signed-off-by: Eric Wei <mengwei.eric@gmail.com>1 parent efdcc7e commit ab4e7c6
4 files changed
Lines changed: 530 additions & 117 deletions
File tree
- integ-test
- src/test/java/org/opensearch/sql
- calcite/remote
- legacy
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1033 | 1033 | | |
1034 | 1034 | | |
1035 | 1035 | | |
1036 | | - | |
1037 | | - | |
1038 | | - | |
1039 | | - | |
1040 | | - | |
1041 | | - | |
1042 | | - | |
1043 | | - | |
1044 | | - | |
1045 | | - | |
1046 | 1036 | | |
1047 | 1037 | | |
1048 | 1038 | | |
| |||
1075 | 1065 | | |
1076 | 1066 | | |
1077 | 1067 | | |
1078 | | - | |
1079 | | - | |
1080 | | - | |
1081 | | - | |
1082 | | - | |
| 1068 | + | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
1083 | 1074 | | |
| 1075 | + | |
| 1076 | + | |
1084 | 1077 | | |
1085 | 1078 | | |
1086 | 1079 | | |
| |||
0 commit comments