Skip to content

Commit 9c338b3

Browse files
kathapkathap
authored
Use storage-cli types instead of legacy fog provider names (#4908)
* Use storage-cli types instead of legacy fog provider names Accept storage-cli native types in config and temporarily map legacy fog provider names for backward compatibility during some time. * add reference to storage-cli in decision md
1 parent e216043 commit 9c338b3

File tree

3 files changed

+144
-19
lines changed

3 files changed

+144
-19
lines changed

‎decisions/0014-storage-clis-for-blobstore-operations.md‎

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,21 @@
22

33
## Status
44

5-
📝 **Accepted** - This ADR defines a shared direction for replacing fog-based blobstore implementations.
5+
✅ **Implemented** - Storage-CLI support is fully implemented for following Iaas Providers with native type names.
66

77
| Provider | Status | Notes |
88
|--------------|---------------------------|---------------------------------------------------------------------------------------------------------|
9-
| Azure | 🚧 PoC in Progress | [PoC](https://github.com/cloudfoundry/cloud_controller_ng/pull/4397) done with `bosh-azure-storage-cli` |
10-
| AWS | 🧭 Open for Contribution | |
11-
| GCP | 🧭 Open for Contribution | |
12-
| Alibaba Cloud| 🧭 Open for Contribution | |
9+
| Azure | ✅ Implemented | Uses [`storage-cli`](https://github.com/cloudfoundry/storage-cli/tree/main/azurebs) with native type `azurebs` |
10+
| AWS | ✅ Implemented | Uses [`storage-cli`](https://github.com/cloudfoundry/storage-cli/tree/main/s3) with native type `s3` |
11+
| GCP | ✅ Implemented | Uses [`storage-cli`](https://github.com/cloudfoundry/storage-cli/tree/main/gcs) with native type `gcs` |
12+
| Alibaba Cloud| ✅ Implemented | Uses [`storage-cli`](https://github.com/cloudfoundry/storage-cli/tree/main/alioss) with native type `alioss` |
13+
14+
**Configuration Migration Status:**
15+
- The `blobstore_provider` field accepts both native storage-cli type names AND legacy fog names
16+
- **Recommended:** Use native storage-cli type names (azurebs, s3, gcs, alioss)
17+
- **Legacy fog names** (AzureRM, AWS, Google, aliyun) still supported for backwards compatibility
18+
- **WebDAV/dav intentionally excluded** until fully supported
19+
- **Timeline:** Legacy fog name support to be removed May 2026
1320

1421

1522
## Context
@@ -46,15 +53,16 @@ Specifically, we will:
4653
packages:
4754
app_package_directory_key: app-packages
4855
blobstore_type: storage-cli
56+
blobstore_provider: azurebs # Native storage-cli type (RECOMMENDED)
57+
# OR: blobstore_provider: AzureRM # Legacy fog name (DEPRECATED)
4958
connection_config:
5059
azure_storage_access_key: <access_key>
5160
azure_storage_account_name: <account_name>
5261
container_name: app-packages
5362
environment: AzureCloud
54-
provider: AzureRM
5563
max_package_size: 1610612736
5664
```
57-
* Field `provider` will be used to determine the corresponding storage CLI blobstore client class (same approach is used for fog)
65+
* Field `blobstore_provider` will be used to determine the corresponding storage CLI blobstore client class (same approach is used for fog)
5866
* The `fog_connection` field will be renamed to `connection_config` to make it independent
5967
* Values from `connection_config` are used to generate the corresponding config file for the Bosh storage CLIs
6068
* Config generation could be moved away from ccng into capi-release to avoid duplication

‎lib/cloud_controller/blobstore/storage_cli/storage_cli_client.rb‎

Lines changed: 28 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -17,24 +17,45 @@ class StorageCliClient < BaseClient
1717
'resource_pool' => :storage_cli_config_file_resource_pool
1818
}.freeze
1919

20-
PROVIDER_TO_STORAGE_CLI_STORAGETYPE = {
20+
# Native storage-cli type names supported by CC (dav intentionally excluded for now)
21+
STORAGE_CLI_TYPES = %w[azurebs alioss s3 gcs].freeze
22+
23+
# DEPRECATED: Legacy fog provider names (remove after migration window)
24+
LEGACY_PROVIDER_TO_STORAGE_CLI_TYPE = {
2125
'AzureRM' => 'azurebs',
2226
'aliyun' => 'alioss',
2327
'AWS' => 's3',
24-
'webdav' => 'dav',
2528
'Google' => 'gcs'
29+
# 'webdav' => 'dav', # intentionally not enabled yet
2630
}.freeze
2731

28-
IMPLEMENTED_PROVIDERS = %w[AzureRM aliyun Google AWS].freeze
29-
3032
def initialize(directory_key:, resource_type:, root_dir:, min_size: nil, max_size: nil)
3133
raise 'Missing resource_type' if resource_type.nil?
3234

3335
config_file_path = config_path_for(resource_type)
3436
cfg = fetch_config(resource_type)
35-
@provider = cfg['provider'].to_s
36-
raise BlobstoreError.new("No provider specified in config file: #{File.basename(config_file_path)}") if @provider.empty?
37-
raise "Unimplemented provider: #{@provider}, implemented ones are: #{IMPLEMENTED_PROVIDERS.join(', ')}" unless IMPLEMENTED_PROVIDERS.include?(@provider)
37+
38+
# Get provider field (can contain either fog name or storage-cli type)
39+
provider = cfg['provider']&.to_s
40+
raise BlobstoreError.new("No provider specified in config file: #{File.basename(config_file_path)}") if provider.nil? || provider.empty?
41+
42+
# Explicitly block unfinished webdav storage-cli support to avoid confusion and wasted effort on debugging
43+
# unsupported providers. Remove this check when webdav support is added.
44+
raise "provider '#{provider}' is not supported yet" if %w[webdav dav].include?(provider)
45+
46+
@storage_type =
47+
if STORAGE_CLI_TYPES.include?(provider)
48+
provider
49+
else
50+
# START LEGACY FOG SUPPORT (delete this whole else-branch after migration)
51+
LEGACY_PROVIDER_TO_STORAGE_CLI_TYPE[provider]
52+
# END LEGACY FOG SUPPORT
53+
end
54+
55+
unless @storage_type
56+
raise "Unknown provider: #{provider}. Supported storage-cli types: #{STORAGE_CLI_TYPES.join(', ')} " \
57+
"(legacy fog names accepted temporarily: #{LEGACY_PROVIDER_TO_STORAGE_CLI_TYPE.keys.join(', ')})"
58+
end
3859

3960
@cli_path = cli_path
4061
@config_file = config_file_path
@@ -43,7 +64,6 @@ def initialize(directory_key:, resource_type:, root_dir:, min_size: nil, max_siz
4364
@root_dir = root_dir
4465
@min_size = min_size || 0
4566
@max_size = max_size
46-
@storage_type = PROVIDER_TO_STORAGE_CLI_STORAGETYPE[@provider]
4767
end
4868

4969
def fetch_config(resource_type)

‎spec/unit/lib/cloud_controller/blobstore/storage_cli/storage_cli_client_spec.rb‎

Lines changed: 101 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,9 @@ module CloudController
55
module Blobstore
66
RSpec.describe StorageCliClient do
77
describe 'client init' do
8-
it 'init the correct client when JSON has provider AzureRM' do
8+
# DEPRECATED: Legacy fog provider tests - remove after migration window
9+
# START LEGACY FOG SUPPORT TESTS
10+
it 'init the correct client when JSON has provider AzureRM (legacy fog name)' do
911
droplets_cfg = Tempfile.new(['droplets', '.json'])
1012
droplets_cfg.write({ provider: 'AzureRM',
1113
account_key: 'bommelkey',
@@ -23,16 +25,44 @@ module Blobstore
2325
root_dir: 'dummy-root',
2426
resource_type: 'droplets'
2527
)
26-
expect(client.instance_variable_get(:@provider)).to eq('AzureRM')
2728
expect(client.instance_variable_get(:@storage_type)).to eq('azurebs')
2829
expect(client.instance_variable_get(:@resource_type)).to eq('droplets')
2930
expect(client.instance_variable_get(:@root_dir)).to eq('dummy-root')
3031
expect(client.instance_variable_get(:@directory_key)).to eq('dummy-key')
3132

3233
droplets_cfg.close!
3334
end
35+
# END LEGACY FOG SUPPORT TESTS
3436

35-
it 'raises an error for an unimplemented provider' do
37+
it 'init the correct client when JSON has provider azurebs (native storage-cli type)' do
38+
droplets_cfg = Tempfile.new(['droplets', '.json'])
39+
droplets_cfg.write({ provider: 'azurebs',
40+
account_key: 'bommelkey',
41+
account_name: 'bommel',
42+
container_name: 'bommelcontainer',
43+
environment: 'BommelCloud' }.to_json)
44+
droplets_cfg.flush
45+
46+
config_double = instance_double(VCAP::CloudController::Config)
47+
allow(VCAP::CloudController::Config).to receive(:config).and_return(config_double)
48+
allow(config_double).to receive(:get).with(:storage_cli_config_file_droplets).and_return(droplets_cfg.path)
49+
50+
client = StorageCliClient.new(
51+
directory_key: 'dummy-key',
52+
root_dir: 'dummy-root',
53+
resource_type: 'droplets'
54+
)
55+
expect(client.instance_variable_get(:@storage_type)).to eq('azurebs')
56+
expect(client.instance_variable_get(:@resource_type)).to eq('droplets')
57+
expect(client.instance_variable_get(:@root_dir)).to eq('dummy-root')
58+
expect(client.instance_variable_get(:@directory_key)).to eq('dummy-key')
59+
60+
droplets_cfg.close!
61+
end
62+
63+
# DEPRECATED: Legacy fog provider tests - remove after migration window
64+
# START LEGACY FOG SUPPORT TESTS
65+
it 'raises an error for an unknown legacy provider' do
3666
droplets_cfg = Tempfile.new(['droplets', '.json'])
3767
droplets_cfg.write(
3868
{ provider: 'UnknownProvider',
@@ -49,10 +79,77 @@ module Blobstore
4979

5080
expect do
5181
StorageCliClient.new(directory_key: 'dummy-key', root_dir: 'dummy-root', resource_type: 'droplets')
52-
end.to raise_error(RuntimeError, 'Unimplemented provider: UnknownProvider, implemented ones are: AzureRM, aliyun, Google, AWS')
82+
end.to raise_error(RuntimeError, /Unknown provider: UnknownProvider/)
83+
84+
droplets_cfg.close!
85+
end
86+
87+
it 'blocks webdav/dav provider explicitly' do
88+
droplets_cfg = Tempfile.new(['droplets', '.json'])
89+
droplets_cfg.write(
90+
{ provider: 'webdav',
91+
account_key: 'bommelkey' }.to_json
92+
)
93+
droplets_cfg.flush
94+
95+
config_double = instance_double(VCAP::CloudController::Config)
96+
allow(VCAP::CloudController::Config).to receive(:config).and_return(config_double)
97+
allow(config_double).to receive(:get).with(:storage_cli_config_file_droplets).and_return(droplets_cfg.path)
98+
99+
expect do
100+
StorageCliClient.new(directory_key: 'dummy-key', root_dir: 'dummy-root', resource_type: 'droplets')
101+
end.to raise_error(RuntimeError, /is not supported yet/)
102+
103+
droplets_cfg.close!
104+
end
105+
# END LEGACY FOG SUPPORT TESTS
106+
107+
it 'raises an error for an unknown storage-cli type' do
108+
droplets_cfg = Tempfile.new(['droplets', '.json'])
109+
droplets_cfg.write(
110+
{ provider: 'unknown_type',
111+
account_key: 'bommelkey',
112+
account_name: 'bommel',
113+
container_name: 'bommelcontainer',
114+
environment: 'BommelCloud' }.to_json
115+
)
116+
droplets_cfg.flush
117+
118+
config_double = instance_double(VCAP::CloudController::Config)
119+
allow(VCAP::CloudController::Config).to receive(:config).and_return(config_double)
120+
allow(config_double).to receive(:get).with(:storage_cli_config_file_droplets).and_return(droplets_cfg.path)
121+
122+
expect do
123+
StorageCliClient.new(directory_key: 'dummy-key', root_dir: 'dummy-root', resource_type: 'droplets')
124+
end.to raise_error(RuntimeError, /Unknown provider: unknown_type/)
125+
126+
droplets_cfg.close!
127+
end
128+
129+
# DEPRECATED: Legacy fog provider test - remove after migration window
130+
# START LEGACY FOG SUPPORT TESTS
131+
it 'raises an error when provider is missing' do
132+
droplets_cfg = Tempfile.new(['droplets', '.json'])
133+
droplets_cfg.write(
134+
{ account_key: 'bommelkey',
135+
account_name: 'bommel',
136+
container_name: 'bommelcontainer',
137+
environment: 'BommelCloud' }.to_json
138+
)
139+
droplets_cfg.flush
140+
141+
config_double = instance_double(VCAP::CloudController::Config)
142+
allow(VCAP::CloudController::Config).to receive(:config).and_return(config_double)
143+
allow(config_double).to receive(:get).with(:storage_cli_config_file_droplets).and_return(droplets_cfg.path)
144+
145+
expect do
146+
StorageCliClient.new(directory_key: 'dummy-key', root_dir: 'dummy-root', resource_type: 'droplets')
147+
end.to raise_error(BlobstoreError, /No provider specified/)
53148

54149
droplets_cfg.close!
55150
end
151+
# END LEGACY FOG SUPPORT TESTS
152+
# After removal, change error message expectation to /No storage_type specified/
56153

57154
it 'raise when no resource type' do
58155
expect do

0 commit comments

Comments
 (0)