fix(dataset): raise a clear error on an empty dataset entry#9756
fix(dataset): raise a clear error on an empty dataset entry#9756he-yufeng wants to merge 2 commits into
Conversation
DatasetSyntax.parse('') fell through os.path.exists('') to _safe_split, which
returns (None, None) for an empty string, so dataset became None and the next
os.path.exists(dataset) crashed with a cryptic "stat: path should be string ...
not NoneType" TypeError. An empty entry is easy to hit from a shell invocation
like `--dataset "$DS1" "$DS2"` where a variable is unset but quoted, and it
crashes before any --strict handling. Reject an empty entry up front with a
clear, actionable ValueError instead.
There was a problem hiding this comment.
Code Review
This pull request adds validation to reject empty dataset entries in DatasetSyntax.parse with a clear ValueError and includes a corresponding unit test. The review feedback suggests enhancing this validation by stripping leading and trailing whitespace to also catch whitespace-only inputs, and expanding the unit test to cover these cases.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if not dataset: | ||
| raise ValueError('Received an empty dataset entry. Check `--dataset`/`--val_dataset` for a stray empty ' | ||
| 'string (e.g. an unset shell variable wrapped in quotes).') |
There was a problem hiding this comment.
Checking if not dataset: only handles empty strings or None, but does not handle strings that contain only whitespace (e.g., " "). A whitespace-only string will bypass this check and cause a failure later.
Additionally, stripping the dataset string at the very beginning of parse instead of only at the end (line 82) prevents potential bugs where leading or trailing whitespaces interfere with os.path.exists() checks or prefix matching (e.g., " ms::dataset" would fail to match "ms" because of the leading space).
| if not dataset: | |
| raise ValueError('Received an empty dataset entry. Check `--dataset`/`--val_dataset` for a stray empty ' | |
| 'string (e.g. an unset shell variable wrapped in quotes).') | |
| if isinstance(dataset, str): | |
| dataset = dataset.strip() | |
| if not dataset: | |
| raise ValueError('Received an empty dataset entry. Check `--dataset`/`--val_dataset` for a stray empty ' | |
| 'string (e.g. an unset shell variable wrapped in quotes).') |
There was a problem hiding this comment.
Fixed in 65ffb8a — strip the entry at the top of parse so a whitespace-only string collapses to empty and hits the same ValueError, and updated the message to say "empty or whitespace-only".
| def test_dataset_syntax_rejects_empty_entry(): | ||
| import pytest | ||
|
|
||
| from swift.dataset.dataset_syntax import DatasetSyntax | ||
|
|
||
| # An empty dataset entry (e.g. an unset shell variable wrapped in quotes in a | ||
| # `--dataset "$DS1" "$DS2"` invocation) must raise a clear ValueError instead | ||
| # of crashing deep inside with a cryptic `os.path.exists(None)` TypeError. | ||
| with pytest.raises(ValueError, match='empty dataset entry'): | ||
| DatasetSyntax.parse('') |
There was a problem hiding this comment.
Expand the test to cover whitespace-only inputs (such as spaces, tabs, and newlines) to ensure they are also correctly rejected with a clear ValueError instead of causing issues down the line.
| def test_dataset_syntax_rejects_empty_entry(): | |
| import pytest | |
| from swift.dataset.dataset_syntax import DatasetSyntax | |
| # An empty dataset entry (e.g. an unset shell variable wrapped in quotes in a | |
| # `--dataset "$DS1" "$DS2"` invocation) must raise a clear ValueError instead | |
| # of crashing deep inside with a cryptic `os.path.exists(None)` TypeError. | |
| with pytest.raises(ValueError, match='empty dataset entry'): | |
| DatasetSyntax.parse('') | |
| def test_dataset_syntax_rejects_empty_entry(): | |
| import pytest | |
| from swift.dataset.dataset_syntax import DatasetSyntax | |
| # An empty dataset entry (e.g. an unset shell variable wrapped in quotes in a | |
| # `--dataset "$DS1" "$DS2"` invocation) must raise a clear ValueError instead | |
| # of crashing deep inside with a cryptic `os.path.exists(None)` TypeError. | |
| for empty_input in ('', ' ', ' \n\t '): | |
| with pytest.raises(ValueError, match='empty dataset entry'): | |
| DatasetSyntax.parse(empty_input) |
There was a problem hiding this comment.
Done in 65ffb8a — the test now loops over ('', ' ', '\t', '\n') and asserts each raises the clear ValueError.
Address review: strip the entry at the start of DatasetSyntax.parse so a whitespace-only string (e.g. " " from an unset shell variable) is rejected with the same clear ValueError as an empty one instead of slipping through to a failure downstream. Expand the test to cover spaces, tabs, and newlines.
What
DatasetSyntax.parse('')crashes with a crypticTypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType. An empty string falls throughos.path.exists('')(False) into_safe_split('', '::', ...), which returns(None, None)for an empty string, sodatasetbecomesNoneand the nextos.path.exists(dataset)getsNone.An empty entry is easy to hit:
swift sft --dataset "$DS1" "$DS2"where a variable is unset but quoted parses$DS2into a stray empty string in the--datasetlist.BaseArguments.load_datasetonly guardsif self.dataset:(the list is non-empty), not each element, andload_datasetcallsDatasetSyntax.parseper entry with no filter. The crash also happens before any--stricthandling, so it hard-fails regardless.Fix
Reject an empty entry at the top of
parsewith a clear, actionableValueError. Non-empty entries are unaffected.Test
test_dataset_syntax_rejects_empty_entryassertsDatasetSyntax.parse('')raisesValueError(previously aTypeError). I confirmed the crash and the fix by extracting theparse/_safe_splitlogic standalone (the local checkout cannot importswiftwithoutdatasets/torch); the added test runs in CI.