Story 2269: Enable Conditional S3 Storage for development by jlchilders11 · Pull Request #2287 · boostorg/website-v2

jlchilders11 · 2026-04-09T18:03:09Z

Issue: #2269

Summary & Context

Adds a custom tag for handling large static content on S3, as well as a just script for syncing this value back down to local for development.

Link to components/page: http://localhost:8000/v3/demo/components/#image-handling-example

Changes

Adds custom_static template tags to core project, with a large_static custom tag for handling large static content.
Adds down_sync_images just command to handle syncing content from S3 to local for development.
Adds upload_images just command to wrap upload script.

‼️ Risks & Considerations ‼️

Please list any potential risks or areas that need extra attention during review/testing

This may add additional set up and tribal knowledge for beginning development for new devs.

Self-review Checklist

Tag at least one team member from each team to review this PR
Link this PR to the related GitHub Project ticket

gregjkal

Nice work - a few minor quibbles.

gregjkal · 2026-04-09T18:14:53Z

+    <div class="v3-examples-section__block" id="{{ section_title|slugify }}">
+      <h3>{{ section_title }}</h3>
+      <div class="v3-examples-section__example-box">
+        {% large_static '/img/v3/learn-page/Learn_Octopus.png' as img_url %}


Don't include a leading slash - I know we lstrip it, but let's follow best practices in our examples 😊.

Also, why are we storing this as a variable instead of using it in-line?

Good calls, I was storing it as a variable just out of habit, it makes for cleaner image tags. Unnecessary in this case.

gregjkal · 2026-04-09T18:16:32Z

    docker compose run --rm web python manage.py {{ args }}
+
+# Static File Management
+@down_sync_images BUCKET='stage.boost.org.v2': ## syncs all items from specified bucket to static/static-large for local development. See scripts/upload-images.sh for required vars.


syncs all items from specified bucket

Is that really what we want to do? Local installs should already have all static files other than the static-large assets, right?

Currently, the only content found in the static folder of the S3 Bucket are our V3 Image assets. Based on that, I assumed that the intent is for that folder to be our static-large equivalent in S3, but named in a way that is more standard for URLs

gregjkal · 2026-04-09T19:14:30Z

 extract_links.py
+
+# Large static images
+static/static-large/*


I don't think you want the trailing * - are you seeing the directory itself as untracked in git status? I would expect you to with the *.

So, this is intentional on my part. The intent and current behavior is that the static-large folder will be created on checkout, but the content of said folder is ignored. This lets the sync process work without having to check the existence of the folder.

Gotcha - ok, this means we'll deploy this empty directory, but that seems harmless.

gregjkal · 2026-04-10T13:58:14Z

+    if settings.LOCAL_DEVELOPMENT:
+        return f"{settings.STATIC_URL}static-large/{file_path}"
+    else:
+        return f"{settings.STATIC_CONTENT_AWS_S3_ENDPOINT_URL}/{settings.STATIC_CONTENT_BUCKET_NAME}/static/{file_path}"


It looks like this code assumes parentheses correctly and parentheses that STATIC_URL has a trailing slash, but STATIC_CONTENT_AWS_S3_ENDPOINT_URL has no trailing slash... that seems a little fragile. I wonder if we should rstrip those two settings constants.

Suggested change

if settings.LOCAL_DEVELOPMENT:

return f"{settings.STATIC_URL}static-large/{file_path}"

else:

return f"{settings.STATIC_CONTENT_AWS_S3_ENDPOINT_URL}/{settings.STATIC_CONTENT_BUCKET_NAME}/static/{file_path}"

static_url = settings.STATIC_URL.rstrip("/")

static_content_aws_s3_endpoint_url = settings.STATIC_CONTENT_AWS_S3_ENDPOINT_URL.rstrip("/")

if settings.LOCAL_DEVELOPMENT:

return f"{static_url}/static-large/{file_path}"

else:

return f"{static_content_aws_s3_endpoint_url}/{settings.STATIC_CONTENT_BUCKET_NAME}/static/{file_path}"

Good call. Avoidable footguns should be avoided

gregjkal

LGTM!

julhoang

I think we might be missing a --profile in the just command, otherwise everything works well!

julhoang · 2026-04-10T20:03:52Z

+    fi
+
+    if echo {{VALID_BUCKETS}} | grep -q -w '{{BUCKET}}'; then \
+        aws s3 sync s3://{{BUCKET}}/static/  static/static-large/; \


Suggested change

aws s3 sync s3://{{BUCKET}}/static/ static/static-large/; \

aws s3 sync --profile upload-images s3://{{BUCKET}}/static/ static/static-large/; \

I think the upload-images.sh script specifies [upload-images] profile for the credentials. Meanwhile this command doesn't specify one (fallbacks to [default]) and failed on my end. Would you mind double-checking this?

The profile --profile upload-images is needed to upload images. Downloading is slightly less restrictive, if you are already using keys which have read access to the buckets, they will work. Or set --profile as Julia suggests.

@julhoang Good call, I had logged in more thoroughly so my default profile had access. Implemented that change.

Thanks for the update @jlchilders11 and the clarification too @sdarwin ! I have a few more questions:

1/ We don't currently provide commands for deleting or replacing images – should we add them? At minimum, a replace command seems useful to me. For deletion, if we decide to support it, we should probably look into safeguards like S3 versioning / a soft-delete strategy first.

2/ This may be a rare case depending on whether we support deletion, but: the current down_sync_images command pulls new or updated files but won't remove local files that no longer exist in S3. It's only a local DX issue, but I imagine it could potentially cause confusion if an image renders fine locally but doesn't load in production. Maybe it's worth considering adding the --delete flag to the aws s3 sync command? 🤔

a replace command seems useful to me.

Both "sync" and "cp" already include "replace", so adding a "replace" command is not needed.

won't remove local files

How often will that be an issue? Mostly files are added, rather than removed. From time to time you could delete your entire /static-large/ folder. That said, it's optional to add a --delete flag here.

look into safeguards like S3 versioning

S3 versioning is enabled. Even safer though, during uploads only "cp" is used, not "sync", and deletes are not being done with the current script. If needed (rarely) someone could log into S3 and remove files manually.

julhoang · 2026-04-10T21:12:56Z

+    <div class="v3-examples-section__block" id="{{ section_title|slugify }}">
+      <h3>{{ section_title }}</h3>
+      <div class="v3-examples-section__example-box">
+        <img src="{% large_static 'img/v3/learn-page/Learn_Octopus.png' %}" alt="Large Media Example" />


Just an idea: Can we consider standardizing on using lowercases for file names and adding a step in upload-images.sh script to enforce the normalization before upload?

That's an interesting idea, a good idea, so then upload-images.sh should be modified with a preflight check early in the script to notice capital letters or blank spaces, and fix them, and alert the user that has happened. You are welcome to add it in another pull request.
For the current pull request maybe Jeremy could just manually adjust capitalization, be sure the images being used in 2287 are lower case, and then proceed. It doesn't need to block this one.

For now I went ahead and made that name update.

herzog0

Hey @jlchilders11, thanks for this, it works perfectly and the usage in the codebase is pretty simple.

I just had some thoughts about the way we're dealing with these image sync scripts and thought I should open a PR on top of yours so you can see the suggestions more clearly: #2291.

The PR description outlines all the things I changed/added, so it's a better starting point than looking at the script directly, which may need some more time to fully review.

@sdarwin also worth tagging you, since I'm modifying a few things from your original script.

Let me know your thoughts!

sdarwin · 2026-04-13T16:32:25Z

@sdarwin also worth tagging you, since I'm modifying a few things from your original script.

the updated script looks pretty good!

… development` (#2291)

julhoang

Hi @jlchilders11 ! I'm mostly requesting change to address the bug with the upload to all buckets command, others are just some outdated copies and small typo.

julhoang · 2026-04-15T17:14:23Z

+@up_sync_images_all_buckets:
+    scripts/sync-large-static-images.sh --up-sync --all-buckets;


I've just noticed this command doesn't work correctly since it only uploads to staging.

When I tried to adjust the args order to --all-bucket --up-sync it works though:

I'm curious if the argument parsing be a bit more flexible so that order of args doesn't really matter, just in case we add more flags to this command in the future. That being said, it's just a thought, this current implementation is fine as-is too.

Sorry, bit of an oversight from me here.
It works in this reversed order because the argument parser reads the input flags in a loop, and when it finds a flag to run the upload, it jumps straight into it instead of keep reading for the next flags.

I'm ok with leaving it like this (just updating the flag order), but if we wanna move into a more flexible solution I think I'd change the flag parser block to turn on flags like IS_UP_SYNC=true and then call the functions conditionally based on these flags, after the flags have been parsed.

I've changed how the script parses commands to be more flexible on ordering, take a look and let me know what you think!

…xible

julhoang

I added 2 tiny nits, but otherwise this looks great to me! Thanks for all the updates @jlchilders11 🙌

herzog0

LGTM, thanks for addressing all the items!

kattyode

QA Approved

jlchilders11 added 3 commits April 9, 2026 13:45

Story 2269: Enable Conditional S3 Storage for development

c644d84

Add more comments

874b70c

Add upload_images wrapper command

a8be871

jlchilders11 requested review from gregjkal and herzog0 April 9, 2026 18:03

jlchilders11 linked an issue Apr 9, 2026 that may be closed by this pull request

Enable Conditional S3 Storage for large static content #2269

Closed

gregjkal requested changes Apr 10, 2026

View reviewed changes

gregjkal reviewed Apr 10, 2026

View reviewed changes

Comment thread templates/v3/examples/_v3_example_section.html Outdated

PR feedback: trailing slash footgun avoidance, clean up example

bedd36f

jlchilders11 requested a review from gregjkal April 10, 2026 14:30

gregjkal approved these changes Apr 10, 2026

View reviewed changes

julhoang requested changes Apr 10, 2026

View reviewed changes

PR feedback: ensure proper profile is used for syncing files

8051a36

jlchilders11 requested a review from julhoang April 10, 2026 20:30

julhoang reviewed Apr 10, 2026

View reviewed changes

herzog0 mentioned this pull request Apr 13, 2026

Feedback PR for #2287: Story 2269: Enable Conditional S3 Storage for development #2291

Merged

herzog0 requested changes Apr 13, 2026

View reviewed changes

PR feedback, rename example image

dde67fc

jlchilders11 requested a review from julhoang April 13, 2026 16:22

Feedback PR for #2287: `Story 2269: Enable Conditional S3 Storage for…

4cf4ef4

… development` (#2291)

jlchilders11 requested a review from herzog0 April 14, 2026 18:50

julhoang requested changes Apr 15, 2026

View reviewed changes

PR: Update documentation, make script file argument ordering more fle…

7613ee6

…xible

jlchilders11 requested a review from julhoang April 15, 2026 19:45

julhoang approved these changes Apr 15, 2026

View reviewed changes

Comment thread scripts/sync-large-static-images.sh Outdated

Comment thread scripts/sync-large-static-images.sh Outdated

julhoang reviewed Apr 15, 2026

View reviewed changes

Comment thread scripts/sync-large-static-images.sh Outdated

Clean up script

c45477b

herzog0 approved these changes Apr 16, 2026

View reviewed changes

kattyode approved these changes Apr 20, 2026

View reviewed changes

jlchilders11 merged commit cc1019d into develop Apr 21, 2026
4 checks passed

jlchilders11 deleted the jc/conditional-image-storage branch April 21, 2026 17:46

	aws s3 sync s3://{{BUCKET}}/static/ static/static-large/; \
	aws s3 sync --profile upload-images s3://{{BUCKET}}/static/ static/static-large/; \

		@up_sync_images_all_buckets:
		scripts/sync-large-static-images.sh --up-sync --all-buckets;

Conversation

jlchilders11 commented Apr 9, 2026

Issue: #2269

Summary & Context

Changes

‼️ Risks & Considerations ‼️

Self-review Checklist

Uh oh!

gregjkal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gregjkal Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gregjkal left a comment

Choose a reason for hiding this comment

Uh oh!

julhoang left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julhoang Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdarwin Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julhoang Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdarwin Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

herzog0 left a comment

Choose a reason for hiding this comment

Uh oh!

sdarwin commented Apr 13, 2026

Uh oh!

julhoang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gregjkal Apr 10, 2026 •

edited

Loading

julhoang Apr 10, 2026 •

edited

Loading

sdarwin Apr 10, 2026 •

edited

Loading

julhoang Apr 10, 2026 •

edited

Loading

sdarwin Apr 10, 2026 •

edited

Loading

herzog0 Apr 15, 2026 •

edited

Loading