Skip to content

refactor: replace os.path usage with pathlib#1753

Open
wowi42 wants to merge 8 commits into
pyinfra-dev:3.xfrom
KalvadTech:refactor/os-path-to-pathlib
Open

refactor: replace os.path usage with pathlib#1753
wowi42 wants to merge 8 commits into
pyinfra-dev:3.xfrom
KalvadTech:refactor/os-path-to-pathlib

Conversation

@wowi42

@wowi42 wowi42 commented May 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

Replace os.path and from os import path call sites with pathlib across the library (api, connectors, facts, operations) and the CLI.

The migration is not a blanket os.path to pathlib.Path swap. Path strings in pyinfra serve three different roles, and each needs a different replacement:

  • Genuinely remote / POSIX paths use PurePosixPath / posixpath: files.move/copy, _raise_or_remove_invalid_path, the files.sync walk loop, the scp client, LinuxDistribution release-file parsing, and the deploy-dir relative joins in files.put/get/template and server authorized-keys handling. The remote target is always unix, so these must not depend on the controller platform. This also fixes a latent bug where os.path.join on a Windows controller would emit backslashes into remote unix paths.
  • Real local controller paths use the platform-aware pathlib.Path: get_file_path resolves state.cwd (from getcwd()) and current_exec_filename (an actual local file). On Windows these are native backslash paths. pathlib.Path is itself the platform branch (WindowsPath on Windows, PosixPath elsewhere), so it parses them correctly, whereas forcing posix semantics produced empty dirnames and broke nested deploy/task includes.
  • Local filesystem predicates use Path methods: exists, is_file, is_dir, is_symlink, mkdir, samefile, readlink.

Intentional non-changes

  • os.path.relpath is retained in api/util.py and pyinfra_cli/cli.py: pathlib has no non-strict relpath on Python 3.10 (Path.relative_to(walk_up=True) is 3.12+), and swapping it would silently change behaviour for paths without a common prefix.
  • os.path.normcase in virtualenv.py was dropped (no pathlib equivalent); venv detection is now case-sensitive, a minor behaviour change on Windows only.

Tests

  • tests/util.py adds Path method patches alongside the existing os.path.* patches; the latter are kept because some modules still reach paths via legacy callers during a run.
  • Connector tests patch the shared posixpath module (or Path methods) instead of the old path.X aliases. from os import path previously made client.path and config.path the same module object, so one patch covered both; the equivalent is restored by patching posixpath.

Key lesson

The original from os import path was doing double duty: on Windows os.path is ntpath, which transparently accepts both / and \. A single os.path call site could therefore be handling a real local Windows path and a pyinfra POSIX-style path interchangeably. Neither pathlib.Path (normalises separators to \ on Windows) nor posixpath (cannot parse native Windows paths) reproduces that on its own. The migration is only correct once each site is classified as remote-POSIX, real-local, or a filesystem predicate, and given the matching tool.

Requirements

  • Pull request is based on the default branch (3.x at this time)
  • Pull request includes tests for any new/updated operations/facts
  • Pull request includes documentation for any new/updated operations/facts (behavioural change only, no surface change)
  • Tests pass (see scripts/dev-test.sh)
  • Type checking & code style passes (see scripts/dev-lint.sh)

wowi42 added 5 commits May 15, 2026 10:42
Migrate os.path and `from os import path` call sites to pathlib across
the library and CLI. Remote unix path handling uses PurePosixPath /
posixpath rather than platform-dependent pathlib so it stays correct
when the controller runs on Windows. os.path.relpath is kept in two
spots since pathlib has no non-strict equivalent on Python 3.10.
test_api_operations patched os.path.isfile, which the migrated
files.put no longer calls, so the local-file check ran for real and
failed on CI. Patch the Path.is_file the code now uses instead.
pathlib.Path normalises separators to backslash and treats /foo as
non-absolute on Windows, whereas the previous `from os import path`
(ntpath) preserved forward slashes for the POSIX-style paths pyinfra
uses. Switch the cwd-relative joins, expanduser and isabs sites to
posixpath so behaviour is identical across platforms; pathlib.Path is
kept only for local filesystem predicates.
get_file_path resolves real local controller paths (state.cwd from
getcwd(), current_exec_filename is an actual file). posixpath.dirname
returns "" for a backslash Windows path, breaking nested deploy/task
include resolution on Windows. Use the platform-aware pathlib.Path so
Windows paths parse correctly while POSIX behaviour is unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant