feat: S3 integrator#107
Open
SimonRichardson wants to merge 11 commits into
Open
Conversation
This was referenced Mar 10, 2026
jujubot
added a commit
to juju/juju
that referenced
this pull request
Mar 25, 2026
#22002 This set of changes is snapped off from #21938, where it introduces a way to relate to an s3 relation and get the correct credentials instead of using the ones on controller config. By utilising the [juju-controller charm](juju/juju-controller#107) to drive the changes, we'll be able to ensure that we benefit from the latest charm infrastructure. The changes only target the state code at the moment, with services coming in a later patch. ## QA steps This breaks s3 charm integration, as we require all service changes to be in place. ## Links <!-- Link to all relevant specification, documentation, bug, issue or JIRA card. --> <!-- Replace #19267 with an issue reference or link, otherwise remove this line. --> **Issue:** Fixes #19267. <!-- Place JIRA number in both places below. --> **Jira card:** [JUJU-](https://warthogs.atlassian.net/browse/JUJU-)
SimonRichardson
added a commit
to SimonRichardson/juju
that referenced
this pull request
Apr 17, 2026
Adds s3 endpoints to update the objectstore backend to s3. This is broken off from[1], as that was rather large, it's better to break the work down. This follows both the metrics and tracing endpoint. It will integrate with the controller charm[2]. Ultimately this will allow the charm integrations drive the controller like a real application. Adds endpoints and calls service methods with the JSON input. Currently the removal of a relation will not be permitted, this will change in the future. 1. juju#21938 2. juju/juju-controller#107
SimonRichardson
added a commit
to SimonRichardson/juju
that referenced
this pull request
Apr 20, 2026
Adds s3 endpoints to update the objectstore backend to s3. This is broken off from[1], as that was rather large, it's better to break the work down. This follows both the metrics and tracing endpoint. It will integrate with the controller charm[2]. Ultimately this will allow the charm integrations drive the controller like a real application. Adds endpoints and calls service methods with the JSON input. Currently the removal of a relation will not be permitted, this will change in the future. 1. juju#21938 2. juju/juju-controller#107
jujubot
added a commit
to juju/juju
that referenced
this pull request
Apr 20, 2026
…ration #22267 Adds s3 endpoints to update the objectstore backend to s3. This is broken off from[1], as that was rather large, it's better to break the work down. This follows both the metrics and tracing endpoint. It will integrate with the controller charm[2]. Ultimately this will allow the charm integrations drive the controller like a real application. Adds endpoints and calls service methods with the JSON input. Currently the removal of a relation will not be permitted, this will change in the future. 1. #21938 2. juju/juju-controller#107 ## QA steps Regression tests should be enough for this: #22146 Otherwise a simple bootstrap should suffice as nothing is wired up: ```sh $ juju bootstrap lxd test ```
This was referenced Apr 27, 2026
jujubot
added a commit
to juju/juju
that referenced
this pull request
May 5, 2026
…ig-s3-integration #22304 Removes controller config for s3 integration, instead the s3 credentials will be driven by the controller charm[1]. This is broken off from[2], as that was rather large, it's better to break the work down. This starts to lay the ground work for implementing all the methods required for getting a file backed controller to an s3 backed controller. This should be the penultimate PR for completing the tasks. Further changes to allow s3 to s3 bucket transfer will be done in the future, but not now. In order to make life easier when implementing the drainer, the controller config is just removed. Everything should be driven off from the controller object store service. > [!NOTE] > As this is controller only config, and we don't allow controller migrations, we can deprecated the old config gracefully. 1. juju/juju-controller#107 2. #21938 ## QA steps Regression tests, though the file to s3 backed storage draining is broken in this PR, until the subsequent PR lands. ```sh $ juju bootstrap lxd test ```
This will ensure that the relation can integrate with other charms.
Ensure we call adding and removing s3 credentials when a relation is added or removed.
Using the event reveals all the information required, no need to ask for the information from somewhere else.
SimonRichardson
force-pushed
the
s3-integrator
branch
from
e66687b to
8a2ac87
Compare
jujubot
added a commit
to juju/juju
that referenced
this pull request
May 21, 2026
#22380 ~~Requires #22304 This is one of the last change sets that allows the draining from file object store to s3 backed object stores. We don't currently allow s3 to another s3, as there are some concerns around this. For now it's restricted until we iron out those issues. In order to make life easier when implementing the drainer, the controller config is just removed. Everything should be driven off from the controller object store service. ---- A lot has changed in this PR because we can reduce a lot of complexity from the prior object-store values in controller-config. This includes trying to push as much of the calls into singular transactions, which should prevent any potential races. ---- As we'll need to follow-up with s3-to-s3 draining, but this effectively removes a potential issue with exposing keys to the user. 1. juju/juju-controller#107 2. #21938 ## QA steps ### S3 setup I'm using minio here, but that's only because of familiarity and we should probably use something that isn't no longer maintained. ```sh docker run -d \n -p 9000:9000 \n -p 9001:9001 \n --user $(id -u):$(id -g) \n --name minio1 \n -e "MINIO_ROOT_USER=ROOTUSER" \n -e "MINIO_ROOT_PASSWORD=CHANGEME123" \n -v /home/jenkins/.local/minio/data:/data \n quay.io/minio/minio server /data --console-address ":9001" ``` ```sh $ docker exec -it <container> bash $ mc alias set foo http://localhost:9000 ROOTUSER CHANGEME123 $ mc admin info foo $ mc admin accesskey create foo # Note down the access keys ``` ### Build controller charm Checkout the following juju/juju-controller#107 and then pack it for use. ```sh $ charmcraft pack $ mv juju-controller_ubuntu@24.04-amd64.charm <juju location> ``` ### Bootstrap ```sh $ juju bootstrap lxd src --bootstrap-base=ubuntu@24.04 --controller-charm-path=./juju-controller_ubuntu@24.04-amd64.charm --build-agent ``` Add another model so we can test that: ```sh $ juju add-model foo $ juju deploy juju-qa-test ``` Add the s3-integrator so it can drive the draining: ```sh $ juju switch controller $ juju deploy s3-integrator $ juju integrate controller:s3-backend s3-integrator $ juju config s3-integrator endpoint=<url to minio or S3 compatible> $ juju run s3-integrator/leader sync-s3-credentials access-key=<access key> secret-key=<secret key> ``` This should start the draining - check `juju debug-log -m controller` for errors. Alternatively check the db: ```sh $ juju ssh -m controller 0 $ juju_db_repl repl (controller)> SELECT prdesc FROM object_store_backend uuid life_id type_id updated_at 653813f9-2896-5332-8cbe-629a337a56a3 2 0 2026-05-07 10:40:57.846945418 +0000 UTC f080e651-7c56-4242-8773-260110069191 0 1 2026-05-07 10:40:57.846945418 +0000 UTC ``` We're looking for the life of the file (type_id: 0) to be 2, ensuring it's completed and it's dead. ### Verification Check the buckets: ```sh $ mc ls foo [2026-05-07 10:40:59 UTC] 0B juju-7f618674-29f6-4781-88fb-6e61c6339c42/ ``` Check the controller file based object stores are empty: ```sh $ juju ssh -m controller 0 $ ls -la /var/lib/juju/objectstore ``` Check that we can move into HA (there is a [bug moving into HA](juju/juju-controller#117)): ```sh $ juju add-unit -m controller controller -n 2 ``` Check you can deploy other things: ```sh $ juju switch foo $ juju deploy ubuntu ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Use S3 integrator charm to wire up S3 credentials. This will be the source of draining from file backed storage to s3 storage.