refactor(worker): use revision_exists from huggingface_hub to check branch existence

[ArjunJagdale]

This PR refactors the create_branch function in services/worker/src/worker/utils.py to replace the manual revision existence check using hf_api.list_repo_refs(...) with the more robust and concise revision_exists(...) utility introduced in huggingface_hub>=0.21.0.

Changes

• Removed list_repo_refs call and related retry logic.
• Replaced revision existence check with revision_exists(dataset, target_revision).
• Preserved all existing behavior and error handling.

Benefits

• Cleaner and more readable logic.
• Leverages upstream improvements in huggingface_hub.
• Removes the need for iterating over refs.converts.

Requirements

• huggingface_hub version must be >= 0.21.0.

Tests

No behavioral change; existing tests covering create_branch(...) should continue to pass. If needed, additional tests can mock revision_exists() to verify branching behavior.

---

Closes #2562

[ArjunJagdale]

Noting that the end-to-end tests are currently failing with a 502 Bad Gateway error during setup:

AssertionError: assert 502 == 404

This happens in the poll("/") check from tests/conftest.py, which suggests that the test backend server is temporarily unreachable. The error occurs before any actual test logic related to this PR is exercised.

Since the failure is unrelated to the change (which only touches the revision existence check in create_branch), I believe this is a temporary infra issue.

@lhoestq @severo

[severo]

maybe due to the fact that the PR branch is in another repository. cc @AndreaFrancis @lhoestq

[ArjunJagdale]

maybe due to the fact that the PR branch is in another repository. cc @AndreaFrancis @lhoestq

the original link from #2562 was pointing to a frozen commit, which I couldn't push to, and i guess where all this went wrong. So i tried to apply changes to main. What can be done now?

[severo]

I meant that possibly e2e tests are not running correctly on external PRs like yours. Nothing to do on your side for this point.

I think the two other failing checks (on services/worker) are not affected by this CI bug, and can be fixed so that we can review the PR.

[ArjunJagdale]

I meant that possibly e2e tests are not running correctly on external PRs like yours. Nothing to do on your side for this point.

I think the two other failing checks (on services/worker) are not affected by this CI bug, and can be fixed so that we can review the PR.

I’ve fixed the code-quality check by removing the unused requests import in utils.py.

Regarding the unit test failure in services/worker/tests/test_executor.py -

I looked into test_executor_raises_on_bad_worker, which runs start_worker_loop_that_crashes() and expects a ProcessExitedWithError or TimeoutExpired.

From what I can see, my patch doesn’t interact with subprocess handling or the worker loop — it only affects the revision checking logic in create_branch(). So I suspect the failure may not be directly caused by this PR, but I’m happy to investigate further.
WDYT? or there can be any different reason?

[ArjunJagdale]

Hi! any update about this issue/PR?

[ArjunJagdale]

Formatted util.py, now

Also i can see that

=========================== short test summary info ============================
FAILED tests/job_runners/config/test_parquet_and_info.py::test_parquet_file_path_in_repo[config-split-0-100001-False-None] - Failed: DID NOT RAISE <class 'ValueError'>
= 1 failed, 434 passed, 1 skipped, 14 deselected, 134 warnings in 448.25s (0:07:28) =
                                                                                                         
Error: Process completed with exit code 1.

It looks like the test expected a ValueError, but the current logic no longer raises it for the (config, split, 0, 100001, False, None) case.

[severo]

do we still need LIST_REPO_REFS_RETRY_SLEEPS in the code base? If not, we should remove it

[ArjunJagdale]

do we still need LIST_REPO_REFS_RETRY_SLEEPS in the code base? If not, we should remove it

I guess we do not need it, since the list_repo_refs retry logic is no longer used after switching to revision_exists, I’ve removed LIST_REPO_REFS_RETRY_SLEEPS.

[severo]

I'll let @lhoestq review the PR. I think it's good but I'm just wondering if we should keep the retry loop? I think not