Support Git-style searchPaths with wildcards in AWS S3 buckets (#2812)

[tomy8964]

Summary

Add full Git-style searchPaths support (placeholders + wildcards) to the AWS S3 backend so that users can migrate existing Git-based configurations without renaming to application.*.

• Implements literal, dot-wildcard (.*), single (?) and double (**) wildcard matching
• Retains the lookup order .properties → .json → .yml → .yaml when expanding literals or .* patterns
• Scans “directory” patterns for nested files
• Deduplicates identical keys across multiple patterns

This issue (#2812)
resolves #2812

---

Changes

1. Core Logic

   • Extended getS3ConfigFileWithSearchPaths(...) to treat {application} and {profile} placeholders identically to the Git backend

   • Added support for:

     • Literal + auto-ext: stops at first existing extension
     • Dot-wildcard (.*): expands against supported extensions in priority order
     • Single-character wildcard (?)
     • Multi-level wildcard (**)
     • Directory scan for literal prefixes ending with /

   • Ensured seenKeys set to avoid duplicate property sources

2. Tests
   Added new JUnit 5 tests in AwsS3EnvironmentRepositoryTests to cover all scenarios:

   • searchPaths_placeholderOnly_shouldResolveExactFile
   • searchPaths_wildcardOnly_shouldResolveAllProperties
   • searchPaths_placeholderAndWildcard_shouldResolveMatchingKeys
   • searchPaths_orderMatters_forPropertySourceOrder
   • searchPath_extensionPreserved (dynamic tests for each extension)
   • searchPaths_applicationAsDirectory_shouldStillHonorSearchPaths
   • multiDocumentYaml_withSearchPaths_shouldNotSplitDocuments
   • getLocations_returnsCorrect
   • searchPaths_deduplication_shouldOnlyAddOnce
   • searchPaths_singleCharacterWildcard_shouldMatchExactlyOneChar
   • searchPaths_withEmptyLabel_shouldUseDefaultLabel
   • searchPaths_multipleLabels_shouldApplyForEachLabelInReverseOrder
   • searchPaths_literalNotFound_shouldReturnEmpty

   …plus the additional edge cases for literal-stop, dot-wildcard priority, and nested directory patterns.

3. Example Usage

spring:
  cloud:
    config:
      server:
        awss3:
          bucket: my-config-bucket
          search-paths:
            - "{label}/{application}"          # literal + auto-ext
            - "{label}/{application}.*"        # dot-wildcard expansion
            - "{label}/common/*.json"          # JSON only in common/
            - "{label}/{application}.yml"      # explicit YML

---

Additional Notes

Backward Compatibility

This update does not break any existing AWS S3 config setups. The default lookup behavior is unchanged if no placeholders or wildcards are used in searchPaths.

Migration

Existing users migrating from Git-backed config servers can now use their current searchPaths (including wildcards and placeholders) on S3 with no renaming or convention change required.

Documentation

Documentation and usage examples for the enhanced searchPaths will be updated in the relevant documentation files after the merge.
If there are specific locations that require documentation updates, please let me know—I will be happy to update them.

Performance and Cost

Using wildcards (such as * or **) in searchPaths may result in additional AWS S3 API calls (e.g., ListObjects), especially for large buckets or deeply nested directory patterns.
This could lead to increased latency and higher AWS costs.
Users should consider the structure and size of their buckets when designing searchPaths and monitor AWS S3 usage accordingly.

---

[tomy8964]

Hi @ryanjbaxter!
I have updated the branch to resolve the merge conflicts with the latest main. Could you please take a look when you have some time? Thank you!

[Copilot]

Pull request overview

This PR extends the AWS S3 environment repository to support Git-style searchPaths (placeholders + wildcard matching), enabling S3-backed config layouts to match existing Git-backed repository conventions.

Changes:

• Add searchPaths configuration to the AWS S3 backend and wire it through the factory.
• Implement placeholder expansion and wildcard-based S3 key discovery (including directory scans) with deduplication.
• Add an extensive JUnit 5 test suite covering placeholder/wildcard resolution scenarios.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 10 comments.

File	Description
spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/environment/AwsS3EnvironmentRepository.java	Adds searchPaths support, wildcard matching via AntPathMatcher, S3 list/head probing, and key→property source wrapping.
spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/environment/AwsS3EnvironmentRepositoryFactory.java	Passes searchPaths from properties into the repository constructor.
spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/environment/AwsS3EnvironmentProperties.java	Introduces searchPaths as a bindable configuration property.
spring-cloud-config-server/src/test/java/org/springframework/cloud/config/server/environment/AwsS3EnvironmentRepositoryTests.java	Adds coverage for placeholder/wildcard behavior, ordering, and special cases.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[tomy8964]

Hi @ryanjbaxter!

I have successfully addressed all the feedback from the Copilot review:

1. Fixed property source precedence and default application fallback in findOne().
2. Restored the native Spring Cloud Config profile activation lifecycle (including negated profiles) by refactoring addPropertySources().
3. Added path normalization (trimming leading slashes and duplicate //) and optimized literal extension probing to avoid redundant S3 calls.
4. Isolated the test state in AwsS3EnvironmentRepositoryTests by using a fresh ConfigServerProperties instance.

The checks are passing, and it's ready for your review. Could you please take another look when you have some time? Thank you!

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

[tomy8964]

Hi @ryanjbaxter!

I have successfully addressed the second round of feedback from the Copilot review:

• Generalized S3 Search Logic: Refactored getS3ConfigFileWithSearchPaths by introducing a functional Function (keyWrapper) parameter, eliminating duplicate S3 object crawling and pagination logic between profile-specific and negated-profile cases.
• Restored Negated Profiles: Implemented wrapKeyWithNegatedConfigFiles to fully support multi-document YAML files with negated profile expressions under the searchPaths architecture.
• Formatting & Factory Cleanup: Streamlined the method chains, wrapped arguments to comply with project formatting conventions, and removed the redundant setOrder call in AwsS3EnvironmentRepositoryFactory.
• Test Syntax Fixes: Corrected minor YAML formatting issues (missing spaces after colons) within AwsS3EnvironmentRepositoryTests.

The checks are passing again, and the codebase is now much cleaner and more reusable. Could you please take another look when you have some time? Thank you!

[ryanjbaxter]

Could you also add documentation?

[tomy8964]

Hi @ryanjbaxter!
We have successfully addressed all of your feedback and suggestions. Here is a summary of the changes:

Summary of Changes

1. searchPaths Optimization & Safety
   • Initialized searchPaths using Collections.emptyList() by default in both AwsS3EnvironmentProperties and AwsS3EnvironmentRepository.
   • Updated the repository constructor to guard against null and pass Collections.emptyList() instead.
   • Cleaned up the factory builder by introducing a constructor that accepts AwsS3EnvironmentProperties directly, removing the redundant setOrder call.
2. Cleanups & Constant Extraction
   • Extracted S3 file extension arrays into class-level private static final constants (SUPPORTED_EXTENSIONS and EMPTY_EXTENSION) and reused them.
   • Refactored isSimpleProfileName to be a package-private static helper method and reused it in wrapKeyWithNegatedConfigFiles.
   • Removed numbered comments in the test cases (AwsS3EnvironmentRepositoryTests).
3. Subclass Refactoring
   • Consolidated the redundant key-based S3ConfigFile subclasses (PropertyConfigFileFromKey, YamlConfigFileFromKey, and JsonConfigFileFromKey) into a single S3ConfigFileFromKey class, completely removing the callReadImmediately flag.
4. S3 File Probe & Scanning Improvements
   • Corrected a bug by removing the break statement in the literal file probing loop, ensuring all matching files with different extensions (e.g., both .properties and .yml at the same path) are properly loaded.
   • Added an INFO level log statement when checking S3 object keys before rethrowing unexpected S3Exceptions to assist troubleshooting.
   • Refactored getS3ConfigFileWithSearchPaths into smaller, self-documenting helper methods (probeLiteralPattern, scanDirectoryPattern, probeDotWildcardPattern, scanWildcardPattern) to improve readability and maintainability.
5. Documentation
   • Added comprehensive documentation for the S3 search-paths feature in aws-s3-backend.adoc, explaining literal/directory paths, wildcards, placeholders, and providing a YAML configuration example.
     All tests have been run and verified. Please let us know if there is anything else that needs to be addressed. Thank you! 🙏

[ryanjbaxter]

The return for these 3 conditions looks identical to me

[ryanjbaxter]

We cannot remove public methods

[ryanjbaxter]

What about validating the 2nd property source

[ryanjbaxter]

I think there is a bug here if the pattern begins with *.

If q == -1 and idx == 0 (wildcard at the very start), the else if is never entered because the outer if was false (q == -1), so idx stays 0 and the method falls through to the lastIndexOf logic rather than returning ""