Skip to content

Commit 37f47a2

Browse files
Brandon-RozekBrandon-Rozek
committed
New Post
1 parent 3aa031f commit 37f47a2

2 files changed

Lines changed: 191 additions & 0 deletions

File tree

‎content/blog/synchronizing-static-website-object-storage.md‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
---
2+
title: "Synchronizing my Static Website with Object Storage"
3+
date: 2026-06-13T15:51:08-04:00
4+
draft: true
5+
tags: []
6+
math: false
7+
medium_enabled: false
8+
---
9+
10+
I recently updated all my [geo-distributed](/blog/implementing-cdn-geodns/) web servers to run on Fedora CoreOS ([yes, I still love it](/blog/fedora-coreos-first-impressions/)). This gave me an opportunity to revisit how I handle synchronization. Before, I used [Syncthing](https://syncthing.net/) which while awesome is a pain to configure. I don't update my website or certs too frequently so having an always online setup seemed overkill.
11+
12+
So this time I went with an object storage setup.
13+
14+
![](/files/images/blog/website-object-store.svg)
15+
16+
I created a bucket (e.g `my-website`) and within it I have the following directories
17+
18+
```
19+
my-website
20+
├── etc
21+
└── letsencrypt
22+
└── live
23+
└── example.com
24+
├── cert.pem
25+
├── chain.pem
26+
├── fullchain.pem
27+
└── privkey.pem
28+
└── var
29+
└── www
30+
├── website1
31+
├── website2
32+
└── websiten
33+
```
34+
35+
The core idea is that the webservers will read from this object store to stay up to date with my SSL certificates and my static website files. The rest of the post will go over how I 1) modified my deployment pipeline to push to the object store, 2) push the SSL certificates which are renewed, and 3) pull both the SSL certificates and the website files.
36+
37+
## Deploying my website files
38+
39+
Currently, I use [Github Actions](https://brandonrozek.com/blog/deploying-hugo-website-through-gh-actions/) (labeled as `CI/CD` in the diagram) to build and deploy my website. Beforehand, I had to create a special SSH key-pair and lock it down in case it leaked. For this new setup, we can instead use application keys to authenticate with our object store.
40+
41+
To lower the threat surface further, we can limit the buckets the application key has access to, whether it has read/write permissions, and what file prefixes the application can access.
42+
43+
For my Github action, I created an application key which has read/write permissions to the prefix `var/www/website`. We need both permissions if we want to delete files that don't exist in the build anymore. Here's the script that I use within the GitHub action to synchronize with the object store after I built the website.
44+
45+
```bash
46+
#!/usr/bin/env sh
47+
set -e
48+
49+
# Environmental variables we need to set within the runner
50+
: "${AWS_ACCESS_KEY_ID:?AWS_ACCESS_KEY_ID is not set}"
51+
: "${AWS_SECRET_ACCESS_KEY:?AWS_SECRET_ACCESS_KEY is not set}"
52+
: "${AWS_ENDPOINT_URL:?AWS_ENDPOINT_URL is not set}"
53+
: "${S3_BUCKET:?S3_BUCKET is not set}"
54+
: "${S3_PATH:?S3_PATH is not set}"
55+
56+
# Safety check so we don't wipe our website!
57+
if [ ! -d "public" ] || [ -z "$(ls -A public)" ]; then
58+
echo "public/ is empty or missing"
59+
exit 1
60+
fi
61+
62+
aws s3 sync public/ "s3://${S3_BUCKET}/${S3_PATH}/" \
63+
--delete \
64+
--exclude "*.bak"
65+
```
66+
67+
Summarizing how the script works:
68+
69+
- The environmental variables at the beginning are used to authenticate with the object store.
70+
- We sanity check that the `public` folder exists after running `hugo` and that it's non-empty so that we don't accidentally wipe the object store's files.
71+
- We use the `aws` command to perform the sync. Note that this command is [baked into](https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md) the default Ubuntu image used by GitHub's runners, so no installation step is required.
72+
73+
## Pushing SSL Certificates
74+
75+
I use [Certbot](https://certbot.eff.org/) to request SSL certificates from Let's Encrypt. After a renewal certificate is issued, the client will run any scripts located within `/etc/letsencrypt/renewal-hooks/deploy` ([documentation](https://eff-certbot.readthedocs.io/en/stable/using.html#renewing-certificates)). In that case, we want to add a `push-certs-to-object-store.sh` file which does just that.
76+
77+
```bash
78+
#!/bin/bash
79+
set -u
80+
81+
S3_BUCKET="INSERT_BUCKET_NAME"
82+
ENDPOINT="INSERT_ENDPOINT_URL"
83+
AWS_ACCESS_KEY_ID="INSERT_KEY_ID_HERE"
84+
AWS_SECRET_ACCESS_KEY="INSERT_SECRET_KEY_HERE"
85+
86+
podman run --rm \
87+
-e AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
88+
-e AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
89+
-v "${RENEWED_LINEAGE}:${RENEWED_LINEAGE}:ro,z" \
90+
-v "/etc/letsencrypt/archive:/etc/letsencrypt/archive:ro,z" \
91+
docker.io/amazon/aws-cli s3 cp "${RENEWED_LINEAGE}/" \
92+
"s3://${S3_BUCKET}/${RENEWED_LINEAGE#/}/" \
93+
--recursive \
94+
--endpoint-url "${ENDPOINT}"
95+
96+
if [ $? -ne 0 ]; then
97+
# Insert failure notification technique here
98+
exit 1
99+
fi
100+
101+
```
102+
103+
Make sure that this script is executable after saving it. This script is different from the last in that it uses `podman` to run `aws-cli` as opposed to executing it directly. This is because I'm using Fedora CoreOS an immutable distribution which [highly discourages overlaying packages](https://docs.fedoraproject.org/en-US/fedora-coreos/faq/#_how_do_i_run_custom_applications_on_fedora_coreos).
104+
105+
Going over the script:
106+
107+
- `${RENEWED_LINEAGE}` is the folder path which contains the renewed certs (e.g `/etc/letsencrypt/live/example.com`)
108+
- We need to mount the `${RENEWED_LINEAGE}` path as well as `/etc/letsencrypt/archive` since the live folder only contains symbolic links to the files which are actually stored in the archive.
109+
- Since we're not modifying these files, we can treat them as read-only (the `ro` flag). Since I have SELinux enabled I threw in the `z` flag so that Podman can automatically handle the contexts for me.
110+
- We're using `cp` instead of `sync` since the renewal procedure will overwrite all the existing files with new ones. Meaning that we don't need read permissions for this application key to update the certificates.
111+
- Personally for alerting, I send a curl request to a webhook on failure.
112+
113+
## Pulling the files
114+
115+
We can create an application key with read-only permissions in order to pull the SSL certificates and the website files.
116+
117+
### Website Files
118+
119+
Here's the script that I use to synchronize the local copy with that of the object store:
120+
121+
```bash
122+
#!/bin/bash
123+
set -e
124+
125+
: "${AWS_ACCESS_KEY_ID:?AWS_ACCESS_KEY_ID is not set}"
126+
: "${AWS_SECRET_ACCESS_KEY:?AWS_SECRET_ACCESS_KEY is not set}"
127+
: "${AWS_ENDPOINT_URL:?AWS_ENDPOINT_URL is not set}"
128+
: "${S3_BUCKET:?S3_BUCKET is not set}"
129+
130+
sync_site() {
131+
local path="$1"
132+
133+
mkdir -p "/${path}"
134+
135+
podman run --rm \
136+
-e AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
137+
-e AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
138+
-v "/${path}:/${path}:z" \
139+
docker.io/amazon/aws-cli s3 sync \
140+
"s3://${S3_BUCKET}/${path}/" "/${path}/" \
141+
--delete \
142+
--endpoint-url "${AWS_ENDPOINT_URL}"
143+
}
144+
145+
sync_site "var/www/website1"
146+
sync_site "var/www/website2"
147+
sync_site "var/www/websiten"
148+
```
149+
150+
I have a corresponding systemd unit file and timer which runs every 15 minutes to check for changes.
151+
152+
### SSL Certificates
153+
154+
For my certificates, I only check for new ones daily. The script is very similar...
155+
156+
```bash
157+
#!/bin/bash
158+
set -e
159+
160+
: "${AWS_ACCESS_KEY_ID:?AWS_ACCESS_KEY_ID is not set}"
161+
: "${AWS_SECRET_ACCESS_KEY:?AWS_SECRET_ACCESS_KEY is not set}"
162+
: "${AWS_ENDPOINT_URL:?AWS_ENDPOINT_URL is not set}"
163+
: "${S3_BUCKET:?S3_BUCKET is not set}"
164+
: "${S3_PATH:?S3_PATH is not set}"
165+
166+
mkdir -p -m 700 "/${S3_PATH}"
167+
168+
podman run --rm \
169+
-e AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
170+
-e AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
171+
-v "/${S3_PATH}/":"/${S3_PATH}/":z \
172+
docker.io/amazon/aws-cli s3 cp \
173+
"s3://${S3_BUCKET}/${S3_PATH}/" \
174+
"/${S3_PATH}/" \
175+
--recursive \
176+
--endpoint-url "${AWS_ENDPOINT_URL}"
177+
178+
```
179+
180+
## Conclusion
181+
182+
That's at least how I have it set up at the time of writing. I've only been running this setup for two days, so we'll see how I feel ultimately. Right now, I'm happy that it simplifies my Ansible setup for these servers. Before this, I had to manually setup Syncthing using their webui.
183+
184+
This updated method allows me to copy a few scripts and systemd unit files over and call it a day. I'm also happy that I don't have to deal with creating a special `build` user and making sure that's locked down. Now we'll see if I can be patient for 15 minutes to see my website changes ;D
185+
186+
187+

‎static/files/images/blog/website-object-store.svg‎

Lines changed: 4 additions & 0 deletions
Loading

0 commit comments

Comments
 (0)