Avoid reading self source code via file open()

[rasmi]

What does this PR do?

Avoid open()-reading transformers' own .py source in create_import_structure_from_path (alongside #46207). This is helpful for hermetic Python builds (e.g. with Bazel) and other non-standard environments (as opposed to open() which can vary based on the precise environment and filesystem).

Changes

• Refactors create_import_structure_from_path into a path-based wrapper plus _create_import_structure_from_traversable, which walks via Traversable.iterdir() / read_text(). _resolve_traversable obtains the Traversable once from the already-imported root transformers package and recurses through it (avoiding files(subpackage), which would force-import the subpackage and defeat the _LazyModule bootstrap). Paths outside the transformers package (e.g. test fixtures) fall back to pathlib.Path, which exposes the same interface. (I left the API surface as-is so as to not have to refactor callers to here from __init__.py -- but can clean up the method and update callers if preferred).

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

• I confirm that this is not a pure code agent PR.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@zucchini-nlp @Rocketknight1

[zucchini-nlp]

Hmm, tbh the "open file to infer if impl can be set dynamically" is a dirty hack made in the past. I wonder if we can instead rely on self._supports_attention_backend to check if dynamic impl are supported, since checking the file content is not bullet proof anyway cc @Cyrilvallez

[vasqu]

_supports_attention_backend sadly alone is not viable because of some unique circumstances here and there (i.e. encoder-decoder attentions where we do not have proper kwargs handling atm) or just some models being legacy and having missing flags or similar.

So in the end, we are still kind of forced to check this imo. But maybe I'm overlooking something

[rasmi]

Thanks @zucchini-nlp @vasqu. I updated _can_set_attn_implementation and _can_set_experts_implementation to inspect the classes directly instead of reading the source (I didn't realize it was a hack to begin with). This tests the same properties as before, just by reading the class attributes instead of the file text.

I checked the behavior of the new class-based checks against the old source-reading approach for all 2150 PreTrainedModel classes / 457 modeling files:

• _can_set_experts_implementation: agrees on every class.
• _can_set_attn_implementation: agrees on every class except 12 in legacy models -- FSMT (3), OpenAIGPT (5), SLANet (4) -- where the old check returned True as a false positive because its regex (class \w+Attention(nn.Module)) didn't match either bare class Attention(nn.Module): (FSMT, GPT) or attention-named non-transformer modules (SLANet's GRU cell).

For those 12 mismatches, the new approach yielding False seems more correct to me. None of the FSMT / OpenAIGPT / SLANet models actually use _attn_implementation, so set_attn_implementation would silently appear to succeed without doing anything. Now it logs a warning instead, aligning with other models that don't use the modern interface.

LMK if this is okay, or if a different approach is preferred!

[Cyrilvallez]

Hey @rasmi. No matter what we decide here, we really need to break this PR in two different PRs, one for can_set_xxx, and the other for create_import_structure_from_path. This is because one is much much more sensitive than the other, and we need to be able to properly review and backtrack it later

[Cyrilvallez]

Would be much better/easier with inspect.getsource or something

[rasmi]

@Cyrilvallez -- split this out into #46207 as requested, but we can keep the discussion here if easier. WDYT about the false positives in the source-reading approach vs. class inspection? Any preference in one approach over the other?

[rasmi]

Done in #46207.

[rasmi]

@Cyrilvallez -- moved the _can_set_*_implementation to #46207 and left this one for create_import_structure_from_path!

[rasmi]

(@Cyrilvallez I requested a review here instead of #46207 by accident but it would be appreciated here too!)