Commit 8560cca
authored
fix(operator): guard against NoSuchElementException in ParallelCSVScanSourceOpDesc.getPhysicalOp (#4375)
### What changes were proposed in this PR?
`ParallelCSVScanSourceOpDesc.getPhysicalOp` called `customDelimiter.get`
without
first checking whether the `Option` is defined. When `customDelimiter`
is `None`
(the field's default), this throws a `NoSuchElementException` before the
fallback
comma delimiter can be applied.
**Before:**
```scala
if (customDelimiter.get.isEmpty) { // throws NoSuchElementException when None
customDelimiter = Option(",")
}
```
**After:**
```scala
if (customDelimiter.isEmpty || customDelimiter.get.isEmpty) {
customDelimiter = Option(",")
}
```
This brings the parallel variant in line with `CSVScanSourceOpDesc`,
which has
always used the correct two-part guard.
### Any related issues, documentation, discussions?
Closes #4374
### How was this PR tested?
Two new test cases were added to `CSVScanSourceOpDescSpec`:
1. `"use comma as the default delimiter when customDelimiter is not set
for parallel CSV"` — verifies that `getPhysicalOp` does not throw when
`customDelimiter` is `None` and that the default `,` is applied.
2. `"use comma as the default delimiter when customDelimiter is empty
string for parallel CSV"` — same verification for `Some("")`.
The existing parallel-CSV schema-inference tests continue to pass
unchanged.
### Was this PR authored or co-authored using generative AI tooling?
No.
---------
Signed-off-by: Asish Kumar <officialasishkumar@gmail.com>1 parent ec02509 commit 8560cca
3 files changed
Lines changed: 34 additions & 2 deletions
File tree
- common/workflow-operator/src
- main/scala/org/apache/texera/amber/operator/source/scan/csv
- test/scala/org/apache/texera/amber/operator/source/scan/csv
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
58 | | - | |
| 58 | + | |
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
59 | | - | |
| 59 | + | |
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
| |||
Lines changed: 32 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
128 | 128 | | |
129 | 129 | | |
130 | 130 | | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
131 | 163 | | |
0 commit comments