Rationale behind files.copy() and files.move() failing for non-existent files

[pascal-cm]

Both files.copy() (source) and files.move() (source) fail when the source file is not present.

To reproduce:

pyinfra -vvv @local files.copy src=/foo dest=/bar
--> Loading config...
--> Loading inventory...
--> Connecting to hosts...
    [@local] Connected

--> Preparing operation func...
[@local] >>> sh -c '! (test -e /foo || test -L /foo ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /foo 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' /foo || ls -ld /foo )'
    [@local] Loaded fact files.Directory
[@local] >>> sh -c '! (test -e /foo || test -L /foo ) || ( stat -c '"'"'user=%U group=%G mode=%A atime=%X mtime=%Y ctime=%Z size=%s %N'"'"' /foo 2> /dev/null || stat -f '"'"'user=%Su group=%Sg mode=%Sp atime=%a mtime=%m ctime=%c size=%z %N%SY'"'"' /foo || ls -ld /foo )'
    [@local] Loaded fact files.File

--> Disconnecting from hosts...
--> Operation error: src /foo does not exist

I ran into this issue with a real life deploy like below:

from pyinfra.api import deploy
from pyinfra.operations import files, git
import os

git_repo_url = "https://github.com/pyinfra-dev/pyinfra.git"
git_repo_dir_path = "/tmp/repo"
copy_file_src=os.path.join(git_repo_dir_path, "pyproject.toml")
copy_file_dest=os.path.join(git_repo_dir_path, "pyproject.toml.backup")

@deploy()
def run():
    git.repo(name="Check out repo", src=git_repo_url, dest=git_repo_dir_path)
    files.copy(name="Copy file", src = copy_file_src, dest=copy_file_dest)

run()

Since the repository is not cloned before its first run, the deploy fails.

What's the rationale for those operations to fail with missing src?

With what I understand now, I suggest this to be changed, so those operations do not fail, unless a flag is given, e.g. assume_src_exists.

Will do a PR (for 4.x?) if a change is desired.

[EricDriussi]

Hi there! Both the cp and mv command fail when there is nothing to copy/move:

[mv/cp]: cannot stat '[file/dir]': No such file or directory

I for one would find it very strange if this wasn't the default behavior in pyinfra.

TBH it's not clear to me why you encounter this error in your run() function: the repo should be cloned by the time pyproject.toml is copied, right?

[pascal-cm]

Hi there! Both the cp and mv command fail when there is nothing to copy/move:

[mv/cp]: cannot stat '[file/dir]': No such file or directory

I for one would find it very strange if this wasn't the default behavior in pyinfra.

Agreed, with the exception that it should ...

TBH it's not clear to me why you encounter this error in your run() function: the repo should be cloned by the time pyproject.toml is copied, right?

... not fail during the prepare phase, as it does now, rather it should fail in the execute phase.

Pyinfra's operations are generator functions, i.e. the actual cp / mv invocation happens during the execution phase from what yield returns.

To drive my point home: it should not fail before the actual cp / mv command is issued, as operations executed prior to files.copy() or files.move() weren't called yet. (Unless someone intentionally designed it like that – in which case please document it).

[Dexmachi]

I agree with this

for instance, I currently have ran into this specific race condition when making a dotfile manager using pyinfra

I have to clone a specific repo and then link files inside of it to other places, however, the files.File or files.Directory operation fails due to... well, the git repo not being cloned when it should

[EricDriussi]

Oooh my bad, I didn't understand the issue you were bringing up. Agreed, it would make much more sense as you describe it.