Skip to content

chore: add WITH_AWS_CLOUD support for S3 snapshot storage#6992

Merged
romange merged 2 commits into
mainfrom
AWS_CLOUD
Apr 6, 2026
Merged

chore: add WITH_AWS_CLOUD support for S3 snapshot storage#6992
romange merged 2 commits into
mainfrom
AWS_CLOUD

Conversation

@romange
Copy link
Copy Markdown
Collaborator

@romange romange commented Mar 26, 2026

Summary

  • Add alternative S3 snapshot storage implementation using helio's native S3Storage and AwsCredsProvider when WITH_AWS_CLOUD=ON, removing the dependency on the AWS C++ SDK
  • Implement AwsS3SnapshotStorage::Init(), ListObjects, and destructor following the GCS/Azure pattern
  • Update helio submodule with S3Storage virtual-hosted-style URLs and SigV4 signing fixes

🤖 Generated with Claude Code

Copilot AI review requested due to automatic review settings March 26, 2026 21:28
@augmentcode
Copy link
Copy Markdown

augmentcode Bot commented Mar 26, 2026

🤖 Augment PR Summary

Summary: Adds a WITH_AWS_CLOUD build option to support S3 snapshot storage via Helio’s native SigV4 S3 client, reducing the dependency on the AWS C++ SDK.

Changes:

  • Wire WITH_AWS_CLOUD into src/server/CMakeLists.txt and link aws_cloud_lib
  • Extend AwsS3SnapshotStorage with Init() and cleanup logic, plus a Helio-based ListObjects() path
  • Initialize S3 snapshot storage from CreateCloudSnapshotStorage() similarly to the GCS path
  • Update the helio submodule for S3 URL/signing fixes

🤖 Was this summary useful? React with 👍 or 👎

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a build-time alternative for S3 snapshot storage via WITH_AWS_CLOUD, intending to use helio’s native AWS/S3 helpers instead of the AWS C++ SDK, and wires S3 initialization into the snapshot storage factory.

Changes:

  • Extend S3 snapshot storage selection to allow WITH_AWS_CLOUD builds.
  • Add AwsS3SnapshotStorage::Init(), destructor, and a WITH_AWS_CLOUD implementation of ListObjects().
  • Update server linking/defines to include WITH_AWS_CLOUD support.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

File Description
src/server/server_family.cc Enables S3 snapshot storage creation for WITH_AWS_CLOUD and calls AwsS3SnapshotStorage::Init().
src/server/detail/snapshot_storage.h Adds helio AWS cloud includes and new members/API (Init, dtor, creds provider, SSL ctx).
src/server/detail/snapshot_storage.cc Implements Init, dtor, and ListObjects for WITH_AWS_CLOUD; adjusts S3 read/write compilation guards.
src/server/CMakeLists.txt Adds WITH_AWS_CLOUD define and links an additional AWS cloud library target.
Comments suppressed due to low confidence (2)

src/server/detail/snapshot_storage.cc:618

  • When building with WITH_AWS_CLOUD (and without WITH_AWS), this method always returns "AWS support not compiled in", so loading snapshots from s3:// will fail even though ListObjects is implemented. Add a WITH_AWS_CLOUD implementation (helio S3Storage-backed read file) or gate S3 snapshot storage on WITH_AWS until read support exists.
  if (!bucket_path) {
    return nonstd::make_unexpected(GenericError("Invalid S3 path"));
  }
  auto [bucket, key] = *bucket_path;
  return new aws::S3ReadFile(bucket, key, s3_);
#else
  return nonstd::make_unexpected(GenericError("AWS support not compiled in"));
#endif
}

io::Result<std::string, GenericError> AwsS3SnapshotStorage::LoadPath(std::string_view dir,
                                                                     std::string_view dbfilename) {
  if (dbfilename.empty())

src/server/detail/snapshot_storage.cc:536

  • In WITH_AWS_CLOUD builds, the ec2_metadata and sign_payload constructor arguments are effectively ignored because the only code that uses them is under WITH_AWS. Either plumb these flags into AwsCredsProvider/S3Storage configuration (or env var handling) or remove the flags from this constructor in the WITH_AWS_CLOUD path to avoid silently changing behavior.
      setenv("AWS_EC2_METADATA_DISABLED", "true", 0);
    }
    // S3ClientConfiguration may request configuration and credentials from
    // EC2 metadata so must be run in a proactor thread.
    Aws::S3::S3ClientConfiguration s3_conf;
    s3_conf.checksumConfig.responseChecksumValidation =
        Aws::Client::ResponseChecksumValidation::WHEN_REQUIRED;

Comment thread src/server/detail/snapshot_storage.cc Outdated
Comment thread src/server/detail/snapshot_storage.cc
Copy link
Copy Markdown

@augmentcode augmentcode Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. 3 suggestions posted.

Fix All in Augment

Comment augment review to trigger a new review at any time.

Comment thread src/server/detail/snapshot_storage.cc
Comment thread src/server/server_family.cc
Comment thread src/server/detail/snapshot_storage.cc
@romange
Copy link
Copy Markdown
Collaborator Author

romange commented Mar 27, 2026

augment review

Copy link
Copy Markdown

@augmentcode augmentcode Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. 1 suggestion posted.

Fix All in Augment

Comment augment review to trigger a new review at any time.

Comment thread src/server/detail/snapshot_storage.cc
@romange romange force-pushed the AWS_CLOUD branch 5 times, most recently from bc9877e to c3867a8 Compare April 2, 2026 20:50
@romange romange requested a review from BorysTheDev April 2, 2026 20:51
Comment thread src/server/detail/snapshot_storage.cc
Comment thread src/server/detail/snapshot_storage.cc Outdated
Comment thread src/server/detail/snapshot_storage.cc Outdated
BorysTheDev
BorysTheDev previously approved these changes Apr 3, 2026
romange added 2 commits April 4, 2026 11:32
Add alternative S3 implementation using helio's native
S3Storage and AwsCredsProvider when WITH_AWS_CLOUD is ON,
removing the dependency on the AWS C++ SDK.

Changes:
- Add AwsS3SnapshotStorage::Init() to initialize credentials
  and SSL context (matching GCS/Azure pattern)
- Implement ListObjects using cloud::aws::S3Storage::List
- Add destructor to free SSL context
- Wire up Init call in CreateCloudSnapshotStorage
- Update CMakeLists.txt to link aws_cloud_lib
- Update helio submodule with S3Storage virtual-hosted-style
  URLs and SigV4 signing fixes

This PR substitutes ListObjects implementation ONLY if WITH_AWS is disabled and
WITH_AWS_CLOUD enabled.

Signed-off-by: Roman Gershman <roman@dragonflydb.io>
@romange romange merged commit 40f9507 into main Apr 6, 2026
12 checks passed
@romange romange deleted the AWS_CLOUD branch April 6, 2026 07:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants