Ongoing upload issues with large files.

[brianmcmichael]

I'm getting reports that large file uploads are failing with InvalidSignatureException errors. There appear to be different file limit thresholds for each user, some people can upload files of several gigabytes in size, while others can't upload more than a few hundred megabytes.

I am suspecting that there is a problem inside of the AWS ArchiveTransferManager, but I need to review all of my client creation methods to make sure nothing is getting overlapped.

[brianmcmichael]

This may in part be due to incomplete uploads. If an upload is cancelled and then started again, the hash signatures will match and AWS is expecting that the second attempt is a continuation of the file. Upon completion, the file on Amazon's end doesn't match the signature. In progress multipart uploads need to be deleted without interfering with in progress uploads on another thread.

[mNantern]

Hi !

I think that I've hit that bug on Windows with SAGU v0.74.5.

I got "unable to finish upload" when trying to upload a file of 1.6G and GlacierError was empty.

[phretor]

I also have experienced this bug on Mac OS X. What is the maximum file size which SAGU has been tested with?

[brianmcmichael]

I'm getting differing reports, and I'm beginning to think that it may have to do with a user's system memory, basically each file gets pumped into the ArchiveTransferManager and I think it may be that java is putting the whole file into memory. Could you guys report what size file you're trying to upload and what your system ram is? I've successfully uploaded 9 gb, but I've got 24 gb on my system.

[phretor]

@brianmcmichael I've tried with up to ~10GB with no issues, then I tried a 89GB DMG image, but it got stuck after a while and in the end it crashed with an exception. Unfortunately, it's been about a month ago and I do not precisely remember what the stack trace was about, nor I have a copy of the logfile.

According to Glacier's API, shouldn't the upload of a file happen in chunks of fixed size (which size determines the maximum size of that file)? So there is a tradeoff between large chunks and larger files vs. smaller chunks and more "resumable" uploads.

[mNantern]

I failed to upload a 1.7 GB file and I have 4GB ram. I think that the best solution in this case is to resume the upload automatically if it's possible, no ?

[brianmcmichael]

The ArchiveTransferManager breaks things into 100mb chunks for upload and does 6 retries for each, so resuming the file is built in.

ArchiveTransferManager is an AWS method included in the developer kit, and I think that java may be trying to place the uploaded portion of the entire file into memory, so when the Java VM runs out of memory it throws the exception and kills the upload. I've run into the memory error on the Java VM when calculating primes, and 1/3 of system ram sounds about right.

There's not an easy way to fix this that I can think of, since it seems to be a memory management issue inside of Amazon's method. I don't have a method to resume an upload since the ArchiveTransferManager only returns the archive ID once the file is complete.

There are a couple of options I see:

1. Reverse engineer the AWS methods and rebuild them for better control. (Which is time consuming for me and probably a violation of the AWS TOS)
2. Submit a request to the AWS dev team to check how memory is managed inside of the ArchiveTransferManager.
3. Implement a check inside of SAGU to warn users that downloads may crash if the file size is over a certain percentage of their system ram. I would advise to try splitting the file into multipart zip files using zZip (Win) or Keka (OSX).

I don't really like any of these options, but hopefully the opensourciness of this project can get some bigger brains than me looking at it.

[mNantern]

For option 1 maybe you can use the low level API for upload (http://docs.aws.amazon.com/amazonglacier/latest/dev/uploading-archive-mpu.html ).

It provides an uploadID so you can resume your upload and even upload chunk in parallel.

And how much memory is assigned to the jar executable? Maybe by increasing that size it will run smoothly

[KenSharp]

I've had no issues uploading 100GB files with SAGU.