Skip to content

Commit bc4f3d1

Browse files
jochenehretjochenehret
committed
Update AWS S3 blobstore configuration to Storage CLI
1 parent 88da943 commit bc4f3d1

File tree

1 file changed

+174
-115
lines changed

1 file changed

+174
-115
lines changed

‎common/cc-blobstore-config.html.md.erb‎

Lines changed: 174 additions & 115 deletions
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,9 @@ WebDAV is also supported for an internal blobstore, and blobstores on NFS-mounte
2121

2222
This document describes the following common blobstore configurations:
2323

24-
* [Fog with AWS Credentials](#fog-aws-creds)
25-
* [Fog with AWS Server Side Encryption](#fog-aws-sse)
26-
* [Fog with AWS IAM Instance Profiles](#fog-aws-iam)
24+
* [Storage CLI with AWS Credentials](#storage-cli-aws)
25+
* [Storage CLI with AWS Server Side Encryption](#storage-cli-aws-sse)
26+
* [Storage CLI with AWS IAM Instance Profiles](#storage-cli-aws-iam)
2727
* [Fog with Google Cloud Storage](#fog-gcs)
2828
* [Fog with Google Cloud Storage Service Accounts](#fog-gcs-service-account)
2929
* [Storage CLI with AliCloud Storage](#storage-cli-alicloud)
@@ -32,139 +32,182 @@ This document describes the following common blobstore configurations:
3232
* [Fog with NFS](#fog-local-nfs)
3333
* [WebDAV](#webdav) internal blobstore
3434

35-
##<a id="fog-aws-creds"></a> Fog with AWS Credentials
35+
##<a id="storage-cli-aws"></a> Storage CLI with AWS Credentials
3636

37-
To use Fog blobstores with AWS credentials, do the following:
37+
To use the Storage CLI blobstore interface with AWS credentials, do the following:
3838

3939
1. Insert the following configuration into your manifest under `properties.cc`:
4040

4141
```
4242
cc:
4343
buildpacks:
44-
blobstore_type: fog
44+
blobstore_provider: AWS
45+
blobstore_type: storage-cli
4546
buildpack_directory_key: YOUR-AWS-BUILDPACK-BUCKET
46-
fog_connection: &fog_connection
47+
connection_config:
4748
aws_access_key_id: AWS-ACCESS-KEY
4849
aws_secret_access_key: AWS-SECRET-ACCESS-KEY
49-
provider: AWS
50-
region: us-east-1
50+
region: AWS-REGION
51+
bucket_name: YOUR-AWS-BUILDPACK-BUCKET
5152
droplets:
52-
blobstore_type: fog
53+
blobstore_provider: AWS
54+
blobstore_type: storage-cli
5355
droplet_directory_key: YOUR-AWS-DROPLET-BUCKET
54-
fog_connection: *fog_connection
56+
connection_config:
57+
aws_access_key_id: AWS-ACCESS-KEY
58+
aws_secret_access_key: AWS-SECRET-ACCESS-KEY
59+
region: AWS-REGION
60+
bucket_name: YOUR-AWS-DROPLET-BUCKET
5561
packages:
56-
blobstore_type: fog
62+
blobstore_provider: AWS
63+
blobstore_type: storage-cli
5764
app_package_directory_key: YOUR-AWS-PACKAGE-BUCKET
58-
fog_connection: *fog_connection
65+
connection_config:
66+
aws_access_key_id: AWS-ACCESS-KEY
67+
aws_secret_access_key: AWS-SECRET-ACCESS-KEY
68+
region: AWS-REGION
69+
bucket_name: YOUR-AWS-PACKAGE-BUCKET
5970
resource_pool:
60-
blobstore_type: fog
71+
blobstore_provider: AWS
72+
blobstore_type: storage-cli
6173
resource_directory_key: YOUR-AWS-RESOURCE-BUCKET
62-
fog_connection: *fog_connection
74+
connection_config:
75+
aws_access_key_id: AWS-ACCESS-KEY
76+
aws_secret_access_key: AWS-SECRET-ACCESS-KEY
77+
region: AWS-REGION
78+
bucket_name: YOUR-AWS-RESOURCE-BUCKET
6379
```
64-
1. Replace `AWS-ACCESS-KEY` and `AWS-SECRET-ACCESS-KEY` with your AWS credentials.
80+
1. Replace `AWS-ACCESS-KEY` and `AWS-SECRET-ACCESS-KEY` with your AWS credentials. Replace `AWS-REGION` with the region of your AWS buckets.
6581

66-
1. Replace `YOUR-AWS-BUILDPACK-BUCKET`, `YOUR-AWS-DROPLET-BUCKET`, `YOUR-AWS-PACKAGE-BUCKET`, and `YOUR-AWS-RESOURCE-BUCKET` with the names of your AWS buckets. Do not use periods (`.`) in your AWS bucket names. In the AWS console, you must assign your credentials an IAM policy that allows all S3 actions on all of these buckets.
82+
1. Replace `YOUR-AWS-BUILDPACK-BUCKET`, `YOUR-AWS-DROPLET-BUCKET`, `YOUR-AWS-PACKAGE-BUCKET`, and `YOUR-AWS-RESOURCE-BUCKET` with the names of your AWS buckets. Do not use periods (`.`) in your AWS bucket names. In the AWS console, you must assign your credentials an IAM policy that allows all S3 actions on all of these buckets.
6783

68-
1. (Optional) Provide additional configuration through the `fog_connection` hash, which is passed through to the Fog gem.
84+
1. (Optional) Provide additional configuration through the `connection_config` hash, which is passed through to the Storage CLI. For more information about configuration options, see the [S3-Specific Configuration][storage-cli-s3-options].
6985

70-
##<a id="fog-aws-sse"></a> Fog with AWS Server-Side Encryption
86+
##<a id="storage-cli-aws-sse"></a> Storage CLI with AWS Server-Side Encryption
7187

7288
AWS S3 offers Server-Side Encryption at rest. For more information, see <a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html">Protecting Data Using Server-Side Encryption</a>.
7389

7490
<strong>AWS SSE-S3 blobstore encryption</strong>
7591

7692
1. Insert the following configuration into your manifest under `properties.cc`:
7793

78-
```
79-
cc:
80-
buildpacks:
81-
blobstore_type: fog
82-
buildpack_directory_key: YOUR-AWS-BUILDPACK-BUCKET
83-
fog_connection: &fog_connection
84-
aws_access_key_id: AWS-ACCESS-KEY
85-
aws_secret_access_key: AWS-SECRET-ACCESS-KEY
86-
provider: AWS
87-
region: us-east-1
88-
fog_aws_storage_options: &fog_aws_storage_options
89-
encryption: 'AES256'
90-
droplets:
91-
blobstore_type: fog
92-
droplet_directory_key: YOUR-AWS-DROPLET-BUCKET
93-
fog_connection: *fog_connection
94-
fog_aws_storage_options: *fog_aws_storage_options
95-
packages:
96-
blobstore_type: fog
97-
app_package_directory_key: YOUR-AWS-PACKAGE-BUCKET
98-
fog_connection: *fog_connection
99-
fog_aws_storage_options: *fog_aws_storage_options
100-
resource_pool:
101-
blobstore_type: fog
102-
resource_directory_key: YOUR-AWS-RESOURCE-BUCKET
103-
fog_connection: *fog_connection
104-
fog_aws_storage_options: *fog_aws_storage_options
105-
```
106-
107-
1. Replace `AWS_ACCESS_KEY` and `AWS_SECRET_ACCESS_KEY` with your AWS credentials.
108-
109-
1. Replace `YOUR-AWS-BUILDPACK-BUCKET`, `YOUR-AWS-DROPLET-BUCKET`, `YOUR-AWS-PACKAGE-BUCKET`, and `YOUR-AWS-RESOURCE-BUCKET` with the names of your AWS buckets. Do not use periods (`.`) in your AWS bucket names. In the AWS console, you must assign your credentials an IAM policy that allows all S3 actions on all of these buckets.
110-
111-
1. You can provide further configuration through the `fog_connection` hash, which is passed through to the Fog gem.
112-
113-
1. `fog_aws_storage_options` takes a hash with the key `encryption`. Operators can set its value to a type of encryption algorithm. In the configuration information above, `encryption` is set to `AES256` to enable AWS SSE-S3 encryption.
114-
115-
1. You can provide further configuration through the `fog_aws_storage_options` hash, which is passed through to the Fog gem.
94+
```
95+
cc:
96+
buildpacks:
97+
blobstore_provider: AWS
98+
blobstore_type: storage-cli
99+
buildpack_directory_key: YOUR-AWS-BUILDPACK-BUCKET
100+
connection_config:
101+
aws_access_key_id: AWS-ACCESS-KEY
102+
aws_secret_access_key: AWS-SECRET-ACCESS-KEY
103+
region: AWS-REGION
104+
bucket_name: YOUR-AWS-BUILDPACK-BUCKET
105+
server_side_encryption: AES256
106+
droplets:
107+
blobstore_provider: AWS
108+
blobstore_type: storage-cli
109+
droplet_directory_key: YOUR-AWS-DROPLET-BUCKET
110+
connection_config:
111+
aws_access_key_id: AWS-ACCESS-KEY
112+
aws_secret_access_key: AWS-SECRET-ACCESS-KEY
113+
region: AWS-REGION
114+
bucket_name: YOUR-AWS-DROPLET-BUCKET
115+
server_side_encryption: AES256
116+
packages:
117+
blobstore_provider: AWS
118+
blobstore_type: storage-cli
119+
app_package_directory_key: YOUR-AWS-PACKAGE-BUCKET
120+
connection_config:
121+
aws_access_key_id: AWS-ACCESS-KEY
122+
aws_secret_access_key: AWS-SECRET-ACCESS-KEY
123+
region: AWS-REGION
124+
bucket_name: YOUR-AWS-PACKAGE-BUCKET
125+
server_side_encryption: AES256
126+
resource_pool:
127+
blobstore_provider: AWS
128+
blobstore_type: storage-cli
129+
resource_directory_key: YOUR-AWS-RESOURCE-BUCKET
130+
connection_config:
131+
aws_access_key_id: AWS-ACCESS-KEY
132+
aws_secret_access_key: AWS-SECRET-ACCESS-KEY
133+
region: AWS-REGION
134+
bucket_name: YOUR-AWS-RESOURCE-BUCKET
135+
server_side_encryption: AES256
136+
```
137+
138+
1. Replace `AWS-ACCESS-KEY` and `AWS-SECRET-ACCESS-KEY` with your AWS credentials. Replace `AWS-REGION` with the region of your AWS buckets.
139+
140+
1. Replace `YOUR-AWS-BUILDPACK-BUCKET`, `YOUR-AWS-DROPLET-BUCKET`, `YOUR-AWS-PACKAGE-BUCKET`, and `YOUR-AWS-RESOURCE-BUCKET` with the names of your AWS buckets. Do not use periods (`.`) in your AWS bucket names. In the AWS console, you must assign your credentials an IAM policy that allows all S3 actions on all of these buckets.
141+
142+
1. Set the value of the `server_side_encryption` key to a type of encryption algorithm. In the configuration information above, `server_side_encryption` is set to `AES256` to enable AWS SSE-S3 encryption.
143+
144+
1. (Optional) Provide additional configuration through the `connection_config` hash, which is passed through to the Storage CLI. For more information about configuration options, see the [S3-Specific Configuration][storage-cli-s3-options].
116145

117146
<strong>AWS SSE-KMS blobstore encryption</strong>
118147

119148
1. Obtain your KMS Key ID. For information about managing KMS keys, see the <a href='http://docs.aws.amazon.com/kms/latest/developerguide/getting-started.html'>AWS Key Management Service Getting Started guide.</a>
120149

121150
1. Insert the following configuration into your manifest under `properties.cc`:
122151

123-
```
124-
cc:
125-
buildpacks:
126-
blobstore_type: fog
127-
buildpack_directory_key: YOUR-AWS-BUILDPACK-BUCKET
128-
fog_connection: &fog_connection
129-
aws_access_key_id: AWS-ACCESS-KEY
130-
aws_secret_access_key: AWS-SECRET-ACCESS-KEY
131-
provider: AWS
132-
region: us-east-1
133-
fog_aws_storage_options: &fog_aws_storage_options
134-
encryption: 'aws:kms'
135-
x-amz-server-side-encryption-aws-kms-key-id: "YOUR-AWS-KMS-KEY-ID"
136-
droplets:
137-
blobstore_type: fog
138-
droplet_directory_key: YOUR-AWS-DROPLET-BUCKET
139-
fog_connection: *fog_connection
140-
fog_aws_storage_options: *fog_aws_storage_options
141-
packages:
142-
blobstore_type: fog
143-
app_package_directory_key: YOUR-AWS-PACKAGE-BUCKET
144-
fog_connection: *fog_connection
145-
fog_aws_storage_options: *fog_aws_storage_options
146-
resource_pool:
147-
blobstore_type: fog
148-
resource_directory_key: YOUR-AWS-RESOURCE-BUCKET
149-
fog_connection: *fog_connection
150-
fog_aws_storage_options: *fog_aws_storage_options
151-
```
152-
153-
1. Replace `AWS-ACCESS-KEY` and `AWS-SECRET-ACCESS-KEY` with your AWS credentials.
154-
155-
1. Replace `YOUR-AWS-BUILDPACK-BUCKET`, `YOUR-AWS-DROPLET-BUCKET`, `YOUR-AWS-PACKAGE-BUCKET`, and `YOUR-AWS-RESOURCE-BUCKET` with the names of your AWS buckets. Do not use periods (`.`) in your AWS bucket names. In the AWS console, you must assign your credentials an IAM policy that allows all S3 actions on all of these buckets.
156-
157-
1. You can provide further configuration through the `fog_connection` hash, which is passed through to the Fog gem.
158-
159-
1. Replace `YOUR-AWS-KMS-KEY-ID` with your KMS Key ID.
160-
161-
1. `fog_aws_storage_options` takes a hash with the key `encryption`. Operators can set its value to a type of encryption algorithm. In the configuration information above, `encryption` is set to `aws:kms` to enable AWS SSE-KMS encryption.
162-
163-
1. You can provide further configuration through the `fog_aws_storage_options` hash, which is passed through to the Fog gem.
164-
165-
##<a id="fog-aws-iam"></a> Fog with AWS IAM Instance Profiles
166-
167-
To configure Fog blobstores to use <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html">AWS IAM Instance Profiles</a>, do the following:
152+
```
153+
cc:
154+
buildpacks:
155+
blobstore_provider: AWS
156+
blobstore_type: storage-cli
157+
buildpack_directory_key: YOUR-AWS-BUILDPACK-BUCKET
158+
connection_config:
159+
aws_access_key_id: AWS-ACCESS-KEY
160+
aws_secret_access_key: AWS-SECRET-ACCESS-KEY
161+
region: AWS-REGION
162+
bucket_name: YOUR-AWS-BUILDPACK-BUCKET
163+
server_side_encryption: "aws:kms"
164+
x-amz-server-side-encryption-aws-kms-key-id: YOUR-AWS-KMS-KEY-ID
165+
droplets:
166+
blobstore_provider: AWS
167+
blobstore_type: storage-cli
168+
droplet_directory_key: YOUR-AWS-DROPLET-BUCKET
169+
connection_config:
170+
aws_access_key_id: AWS-ACCESS-KEY
171+
aws_secret_access_key: AWS-SECRET-ACCESS-KEY
172+
region: AWS-REGION
173+
bucket_name: YOUR-AWS-DROPLET-BUCKET
174+
server_side_encryption: "aws:kms"
175+
x-amz-server-side-encryption-aws-kms-key-id: YOUR-AWS-KMS-KEY-ID
176+
packages:
177+
blobstore_provider: AWS
178+
blobstore_type: storage-cli
179+
app_package_directory_key: YOUR-AWS-PACKAGE-BUCKET
180+
connection_config:
181+
aws_access_key_id: AWS-ACCESS-KEY
182+
aws_secret_access_key: AWS-SECRET-ACCESS-KEY
183+
region: AWS-REGION
184+
bucket_name: YOUR-AWS-PACKAGE-BUCKET
185+
server_side_encryption: "aws:kms"
186+
x-amz-server-side-encryption-aws-kms-key-id: YOUR-AWS-KMS-KEY-ID
187+
resource_pool:
188+
blobstore_provider: AWS
189+
blobstore_type: storage-cli
190+
resource_directory_key: YOUR-AWS-RESOURCE-BUCKET
191+
connection_config:
192+
aws_access_key_id: AWS-ACCESS-KEY
193+
aws_secret_access_key: AWS-SECRET-ACCESS-KEY
194+
region: AWS-REGION
195+
bucket_name: YOUR-AWS-RESOURCE-BUCKET
196+
server_side_encryption: "aws:kms"
197+
x-amz-server-side-encryption-aws-kms-key-id: YOUR-AWS-KMS-KEY-ID
198+
```
199+
200+
1. Replace `AWS-ACCESS-KEY` and `AWS-SECRET-ACCESS-KEY` with your AWS credentials. Replace `AWS-REGION` with the region of your AWS buckets.
201+
202+
1. Replace `YOUR-AWS-BUILDPACK-BUCKET`, `YOUR-AWS-DROPLET-BUCKET`, `YOUR-AWS-PACKAGE-BUCKET`, and `YOUR-AWS-RESOURCE-BUCKET` with the names of your AWS buckets. Do not use periods (`.`) in your AWS bucket names. In the AWS console, you must assign your credentials an IAM policy that allows all S3 actions on all of these buckets.
203+
204+
1. Set the `server_side_encryption` key to the value `aws:kms`. Replace `YOUR-AWS-KMS-KEY-ID` with your KMS Key ID.
205+
206+
1. (Optional) Provide additional configuration through the `connection_config` hash, which is passed through to the Storage CLI. For more information about configuration options, see the [S3-Specific Configuration][storage-cli-s3-options].
207+
208+
##<a id="storage-cli-aws-iam"></a> Storage CLI with AWS IAM Instance Profiles
209+
210+
To configure the Storage CLI to use <a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html">AWS IAM Instance Profiles</a>, do the following:
168211

169212
1. Configure an additional <code>cloud-controller</code> IAM role with the following policy to give access to the S3 buckets you plan to use:
170213

@@ -227,29 +270,44 @@ To configure Fog blobstores to use <a href="http://docs.aws.amazon.com/IAM/lates
227270
```
228271
cc:
229272
buildpacks:
230-
blobstore_type: fog
273+
blobstore_provider: AWS
274+
blobstore_type: storage-cli
231275
buildpack_directory_key: YOUR-AWS-BUILDPACK-BUCKET
232-
fog_connection: &fog_connection
233-
provider: AWS
234-
region: us-east-1
235-
use_iam_profile: true
276+
connection_config:
277+
region: AWS-REGION
278+
bucket_name: YOUR-AWS-BUILDPACK-BUCKET
279+
credentials_source: cloud-controller
236280
droplets:
237-
blobstore_type: fog
281+
blobstore_provider: AWS
282+
blobstore_type: storage-cli
238283
droplet_directory_key: YOUR-AWS-DROPLET-BUCKET
239-
fog_connection: *fog_connection
284+
connection_config:
285+
region: AWS-REGION
286+
bucket_name: YOUR-AWS-DROPLET-BUCKET
287+
credentials_source: cloud-controller
240288
packages:
241-
blobstore_type: fog
289+
blobstore_provider: AWS
290+
blobstore_type: storage-cli
242291
app_package_directory_key: YOUR-AWS-PACKAGE-BUCKET
243-
fog_connection: *fog_connection
292+
connection_config:
293+
region: AWS-REGION
294+
bucket_name: YOUR-AWS-PACKAGE-BUCKET
295+
credentials_source: cloud-controller
244296
resource_pool:
245-
blobstore_type: fog
297+
blobstore_provider: AWS
298+
blobstore_type: storage-cli
246299
resource_directory_key: YOUR-AWS-RESOURCE-BUCKET
247-
fog_connection: *fog_connection
300+
connection_config:
301+
region: AWS-REGION
302+
bucket_name: YOUR-AWS-RESOURCE-BUCKET
303+
credentials_source: cloud-controller
248304
```
249305

250-
Replace `YOUR-AWS-BUILDPACK-BUCKET`, `YOUR-AWS-DROPLET-BUCKET`, `YOUR-AWS-PACKAGE-BUCKET`, and `YOUR-AWS-RESOURCE-BUCKET` with the names of your AWS buckets. Do not use periods (`.`) in your AWS bucket names.
306+
1. Replace `AWS-REGION` with the region of your AWS buckets.
251307

252-
1. (Optional) Provide other configuration with the `fog_connection` hash, which is passed through to the Fog gem.
308+
1. Replace `YOUR-AWS-BUILDPACK-BUCKET`, `YOUR-AWS-DROPLET-BUCKET`, `YOUR-AWS-PACKAGE-BUCKET`, and `YOUR-AWS-RESOURCE-BUCKET` with the names of your AWS buckets. Do not use periods (`.`) in your AWS bucket names.
309+
310+
1. (Optional) Provide additional configuration through the `connection_config` hash, which is passed through to the Storage CLI. For more information about configuration options, see the [S3-Specific Configuration][storage-cli-s3-options].
253311

254312
##<a id="fog-gcs"></a>Fog with Google Cloud Storage
255313

@@ -622,3 +680,4 @@ To configure your blobstores to use the WebDAV protocol, perform the steps below
622680
[azure-name-restrictions]: https://docs.microsoft.com/en-us/azure/architecture/best-practices/naming-conventions#storage
623681
[alicloud-name-restrictions]: https://www.alibabacloud.com/help/en/oss/user-guide/bucket-naming-conventions
624682
[alicloud-regions-and-endpoints]: https://www.alibabacloud.com/help/en/oss/user-guide/regions-and-endpoints
683+
[storage-cli-s3-options]: https://github.com/cloudfoundry/storage-cli/tree/main/s3

0 commit comments

Comments
 (0)