Parquet does prevent records with unknown union fields to be written as it would create a TProtocol violation. But it also prevents records with unions having one their field itself having an unknown field (which is acceptable if it is a struct).
The recursive check should be removed, and only when a union field is unknown should the record be deemed invalid.
Reporter: Laurent Goujon / @laurentgo
Note: This issue was originally created as PARQUET-362. Please see the migration documentation for further details.
Parquet does prevent records with unknown union fields to be written as it would create a TProtocol violation. But it also prevents records with unions having one their field itself having an unknown field (which is acceptable if it is a struct).
The recursive check should be removed, and only when a union field is unknown should the record be deemed invalid.
Reporter: Laurent Goujon / @laurentgo
Note: This issue was originally created as PARQUET-362. Please see the migration documentation for further details.