refactor: replace os.path usage with pathlib#1753
Open
wowi42 wants to merge 8 commits into
Open
Conversation
Migrate os.path and `from os import path` call sites to pathlib across the library and CLI. Remote unix path handling uses PurePosixPath / posixpath rather than platform-dependent pathlib so it stays correct when the controller runs on Windows. os.path.relpath is kept in two spots since pathlib has no non-strict equivalent on Python 3.10.
test_api_operations patched os.path.isfile, which the migrated files.put no longer calls, so the local-file check ran for real and failed on CI. Patch the Path.is_file the code now uses instead.
pathlib.Path normalises separators to backslash and treats /foo as non-absolute on Windows, whereas the previous `from os import path` (ntpath) preserved forward slashes for the POSIX-style paths pyinfra uses. Switch the cwd-relative joins, expanduser and isabs sites to posixpath so behaviour is identical across platforms; pathlib.Path is kept only for local filesystem predicates.
get_file_path resolves real local controller paths (state.cwd from getcwd(), current_exec_filename is an actual file). posixpath.dirname returns "" for a backslash Windows path, breaking nested deploy/task include resolution on Windows. Use the platform-aware pathlib.Path so Windows paths parse correctly while POSIX behaviour is unchanged.
…pathlib # Conflicts: # src/pyinfra/connectors/ssh_util.py # src/pyinfra_cli/cli.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replace
os.pathandfrom os import pathcall sites withpathlibacross the library (api,connectors,facts,operations) and the CLI.The migration is not a blanket
os.pathtopathlib.Pathswap. Path strings in pyinfra serve three different roles, and each needs a different replacement:PurePosixPath/posixpath:files.move/copy,_raise_or_remove_invalid_path, thefiles.syncwalk loop, the scp client,LinuxDistributionrelease-file parsing, and the deploy-dir relative joins infiles.put/get/templateandserverauthorized-keys handling. The remote target is always unix, so these must not depend on the controller platform. This also fixes a latent bug whereos.path.joinon a Windows controller would emit backslashes into remote unix paths.pathlib.Path:get_file_pathresolvesstate.cwd(fromgetcwd()) andcurrent_exec_filename(an actual local file). On Windows these are native backslash paths.pathlib.Pathis itself the platform branch (WindowsPathon Windows,PosixPathelsewhere), so it parses them correctly, whereas forcing posix semantics produced empty dirnames and broke nested deploy/task includes.Pathmethods:exists,is_file,is_dir,is_symlink,mkdir,samefile,readlink.Intentional non-changes
os.path.relpathis retained inapi/util.pyandpyinfra_cli/cli.py:pathlibhas no non-strict relpath on Python 3.10 (Path.relative_to(walk_up=True)is 3.12+), and swapping it would silently change behaviour for paths without a common prefix.os.path.normcaseinvirtualenv.pywas dropped (nopathlibequivalent); venv detection is now case-sensitive, a minor behaviour change on Windows only.Tests
tests/util.pyaddsPathmethod patches alongside the existingos.path.*patches; the latter are kept because some modules still reach paths via legacy callers during a run.posixpathmodule (orPathmethods) instead of the oldpath.Xaliases.from os import pathpreviously madeclient.pathandconfig.paththe same module object, so one patch covered both; the equivalent is restored by patchingposixpath.Key lesson
The original
from os import pathwas doing double duty: on Windowsos.pathisntpath, which transparently accepts both/and\. A singleos.pathcall site could therefore be handling a real local Windows path and a pyinfra POSIX-style path interchangeably. Neitherpathlib.Path(normalises separators to\on Windows) norposixpath(cannot parse native Windows paths) reproduces that on its own. The migration is only correct once each site is classified as remote-POSIX, real-local, or a filesystem predicate, and given the matching tool.Requirements
3.xat this time)scripts/dev-test.sh)scripts/dev-lint.sh)