Add --no-overwrite option to aws s3 cp/mv

[alexjurkiewicz]

It would be nice to have a convenience function --no-overwrite for aws s3 cp/mv commands, which would check the target destination doesn't already exist before putting a file into an s3 bucket.

Of course this logic couldn't be guaranteed by the AWS API (afaik...) and is vulnerable to race conditions, etc. But it would be helpful to prevent unintentional mistakes!

[kyleknap]

Marking as a feature request. The tricky part if we did it in cp or mv is that the CLI may have to query S3 to see if the file exists before trying to upload it. So it may make more sense to add it to sync as it already does that.

[sgrimm-sg]

I'd like to see this in cp and/or mv as well.

The reason I don't use sync for this right now is that sync has major performance problems if the destination bucket has a lot of existing files under the target directory.

When you run aws s3 cp --recursive newdir s3://bucket/parentdir/, it only visits each of the files it's actually copying.

When you run aws s3 sync newdir s3://bucket/parentdir/, it visits the files it's copying, but also walks the entire list of files in s3://bucket/parentdir (which may already contain thousands or millions of files) and gets metadata for each existing file.

On a sufficiently large destination bucket, aws s3 cp --recursive can take seconds and aws s3 sync can take hours to copy the same data.

Obviously fixing sync would be nice, but if adding a "check to see if the file already exists" query to cp is a more tractable problem than revamping the sync code to make it fast, it might make sense to do that instead.

[slavaaaaaaaaaa]

I'm also very interested in this feature. An optional interactive prompt for overwriting files would also be nice to have.

[shabeebk]

yes @sgrimm-sg, it makes sense. I am also interested to see CLI cp command which can actually handle these conditions.

[jhoblitt]

It would be extremely useful for this to be an option on aws s3 sync. rsync has this functionality available as --ignore-existing. My preference would be try to use the same option names as rsync as I suspect there are a lot of folks already familiar with rsync.

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

This entry can specifically be found on UserVoice at: https://aws.uservoice.com/forums/598381-aws-command-line-interface/suggestions/33168406-add-no-overwrite-option-to-aws-s3-cp-mv

[kenorb]

Related:

• Add rsync-like --update to s3 sync. #404: Add rsync-like --update to s3 sync
• aws s3 sync --existing #1449: aws s3 sync --existing
• Suggestion: Add --no-overwrite option to aws s3 cp/mv
• Suggestion: aws s3 sync --existing
• Suggestion: Add rsync-like --update to s3 sync

[jamesls]

Based on community feedback, we have decided to return feature requests to GitHub issues.

[guyisra]

@jamesls that's great! can you please respond to the suggest at hand? --no-overwrite would be a great addition and it will avoid wrapping the calls with scripts