Rather than download the redwood urls we should generate signed URLs and allow the pipeline to download them. This is necessary for autoscaling dockers.
To do this, I plan on modifying DockstoreRunner.download_and_transform_json to look at an argument along the lines of --generate_signed_urls. If that argument is passed, we can map the redwood url to an s3 url and use boto's generate_url function to create our signed url.
Rather than download the redwood urls we should generate signed URLs and allow the pipeline to download them. This is necessary for autoscaling dockers.
To do this, I plan on modifying
DockstoreRunner.download_and_transform_jsonto look at an argument along the lines of--generate_signed_urls. If that argument is passed, we can map the redwood url to an s3 url and use boto'sgenerate_urlfunction to create our signed url.