fix: use data_dir for directory paths in ShardedDataset (#1301)

## Summary
- `datasets`' `resolve_pattern` only matches entries with
`type=="file"`, so passing a bare directory path as `data_files` to
`load_dataset` results in `FileNotFoundError` even when the directory
exists on disk
- Detect directory paths in `ShardedDataset._load_dataset()` and pass
them via `data_dir` instead of `data_files`

## Reproduction
```python
from datasets import load_dataset
# This fails with FileNotFoundError:
load_dataset("json", data_files="/path/to/data_directory")
# This works:
load_dataset("json", data_dir="/path/to/data_directory")
```

## Test plan
- [ ] Verify existing EAGLE3/DFlash training pipelines that pass
directory paths work
- [ ] Verify file path and glob patterns still work (falls through to
`data_files`)
- [ ] Verify `data_files=None` (no data_files arg) still works

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Bug Fixes

* Fixed an issue with dataset loading that prevented proper handling of
directory-based data sources. Directories are now correctly detected and
processed during dataset initialization.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Ye Yu <yeyu@nvidia.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: yeyu-nvidia

modelopt/torch/utils/plugins/transformers_dataset.py

@@ -73,10 +73,20 @@ def __getitem__(self, index):
         return self._raw_samples[index]
 
     def _load_dataset(self):
+        # datasets' resolve_pattern only matches entries with type=="file", so passing
+        # a bare directory path as data_files results in FileNotFoundError.
+        # Use data_dir for directory paths instead.
+        data_dir = None
+        data_files = self.data_files
+        if data_files and os.path.isdir(data_files):
+            data_dir = data_files
+            data_files = None
+
         dataset = load_dataset(
             self.name,
             self.subset,
-            data_files=self.data_files,
+            data_files=data_files,
+            data_dir=data_dir,
             split=self.split,
             # num_proc=4,  # TODO: Make this configurable
             streaming=self.num_streaming_samples is not None,