Skip to content

Commit e154db1

Browse files
authored
[develop][cli] Improve curl retry for GitHub package downloads (#7180)
- Increase retry count from 3 to 9 - Add `--retry-all-errors` to retry on SSL and HTTP/2 errors If we retry 9 times, it will wait: 1 + 2 + 4 + 8 + 16 + 32 + 64 + 128 + 256 = 511 seconds = 8.51 minutes It can handle the China region network issue. This fixes intermittent download failures in China regions caused by network instability when accessing GitHub.
1 parent 4c5760c commit e154db1

4 files changed

Lines changed: 4 additions & 4 deletions

File tree

cli/src/pcluster/resources/compute_node/user_data.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -137,7 +137,7 @@ write_files:
137137
error_exit "This AMI was not baked by ParallelCluster. Please use pcluster build-image command to create an AMI by providing your AMI as parent image."
138138
fi
139139
if [ "${!custom_cookbook}" != "NONE" ]; then
140-
curl --retry 3 -v -L -o /etc/chef/aws-parallelcluster-cookbook.tgz ${!cookbook_url}
140+
curl --retry 9 --retry-all-errors -v -L -o /etc/chef/aws-parallelcluster-cookbook.tgz ${!cookbook_url}
141141
vendor_cookbook
142142
fi
143143
cd /tmp

cli/src/pcluster/resources/head_node/user_data.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,7 @@ else
123123
error_exit "This AMI was not baked by ParallelCluster. Please use pcluster build-image command to create an AMI by providing your AMI as parent image."
124124
fi
125125
if [ "${!custom_cookbook}" != "NONE" ]; then
126-
curl --retry 3 -v -L -o /etc/chef/aws-parallelcluster-cookbook.tgz ${!cookbook_url}
126+
curl --retry 9 --retry-all-errors -v -L -o /etc/chef/aws-parallelcluster-cookbook.tgz ${!cookbook_url}
127127
vendor_cookbook
128128
fi
129129

cli/src/pcluster/resources/imagebuilder/parallelcluster.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -219,7 +219,7 @@ phases:
219219
220220
# Download cookbook
221221
mkdir -p /etc/chef && sudo chown -R root:root /etc/chef
222-
curl --retry 3 -L -o /etc/chef/aws-parallelcluster-cookbook.tgz "$COOKBOOK_URL"
222+
curl --retry 9 --retry-all-errors -L -o /etc/chef/aws-parallelcluster-cookbook.tgz "$COOKBOOK_URL"
223223
224224
cd /etc/chef && tar -xzf /etc/chef/aws-parallelcluster-cookbook.tgz --strip-components 1 && rm -f aws-parallelcluster-cookbook.tgz
225225

cli/src/pcluster/resources/login_node/user_data.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,7 @@ write_files:
120120
error_exit "This AMI was not baked by ParallelCluster. Please use pcluster build-image command to create an AMI by providing your AMI as parent image."
121121
fi
122122
if [ "${!custom_cookbook}" != "NONE" ]; then
123-
curl --retry 3 -v -L -o /etc/chef/aws-parallelcluster-cookbook.tgz ${!cookbook_url}
123+
curl --retry 9 --retry-all-errors -v -L -o /etc/chef/aws-parallelcluster-cookbook.tgz ${!cookbook_url}
124124
vendor_cookbook
125125
fi
126126
cd /tmp

0 commit comments

Comments
 (0)