Skip to content

fix: honor index_path when input_dir is a local directory#839

Open
javierdejesusda wants to merge 1 commit into
Lightning-AI:mainfrom
javierdejesusda:fix/800-local-index-path
Open

fix: honor index_path when input_dir is a local directory#839
javierdejesusda wants to merge 1 commit into
Lightning-AI:mainfrom
javierdejesusda:fix/800-local-index-path

Conversation

@javierdejesusda

Copy link
Copy Markdown
Before submitting
  • Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

What does this PR do?

Fixes #800.

When input_dir points to a local directory, a user-provided index_path was ignored and loading failed. In dataset_utilities.py the branch that copies index.json from index_path into the cache directory was nested inside isinstance(input_dir.url, str), so it only ran for remote inputs. For a local input_dir the url is None, the branch was skipped, the index was never materialized into the cache path, and the dataset could not resolve.

The fix checks index_path first, independent of whether the input is local or remote: if index.json is not already cached and an explicit index_path was provided, copy it into the cache path. The existing remote-download branch is preserved for the case where no index_path is given.

Added test_subsample_streaming_dataset_with_local_index_path, which uses a local input_dir with no index.json plus an index_path in a separate directory, and asserts that subsampling resolves correctly and the index is materialized into the local dataset directory. It fails on main and passes with this change.

The index_path copy was gated behind isinstance(input_dir.url, str), so a
user-provided index_path was silently ignored when input_dir is a local
directory (url is None), raising a misleading "doesn't contain any
index.json file" error. Copy the provided index into the local cache
regardless of whether input_dir.url is a string.
@codecov-commenter

codecov-commenter commented Jul 8, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 87.50000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 81%. Comparing base (5201360) to head (ed371c0).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@         Coverage Diff         @@
##           main   #839   +/-   ##
===================================
  Coverage    81%    81%           
===================================
  Files        54     54           
  Lines      7671   7671           
===================================
+ Hits       6186   6191    +5     
+ Misses     1485   1480    -5     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

index_path is ignored in StreamingDataset when input_dir is a local directory

2 participants