Skip to content

Separate S3 buckets for raw data and HTML exports #41

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
marcarl merged 2 commits into from
Jan 7, 2026
Merged

Separate S3 buckets for raw data and HTML exports #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3971ba3
Separate R2 buckets for raw data and HTML exports
marcarl Jan 7, 2026
8fcbbb6
Fix workflow name in GitHub Pages deployment
marcarl Jan 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/fetch-sfs-workflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ jobs:
aws configure set output json

# Upload all JSON files to R2 for backup
aws s3 sync data/sfs_json/ s3://${{ secrets.CLOUDFLARE_R2_BUCKET_NAME }}/sfs_json/ \
aws s3 sync data/sfs_json/ s3://${{ vars.CLOUDFLARE_R2_RAWDATA_BUCKET_NAME }}/ \
--endpoint-url https://${{ secrets.CLOUDFLARE_R2_ACCOUNT_ID }}.r2.cloudflarestorage.com \
--content-type "application/json" \
--exclude "*.md" \
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/github-pages-workflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Exportera till HTML-format och publicera till GitHub Pages
name: Exportera till HTML och publicera till GitHub Pages

on:
workflow_dispatch: # Tillåter manuell körning
Expand Down Expand Up @@ -70,7 +70,7 @@ jobs:

# Download all JSON files from R2
mkdir -p data/sfs_json
aws s3 sync s3://${{ secrets.CLOUDFLARE_R2_BUCKET_NAME }}/sfs_json/ data/sfs_json/ \
aws s3 sync s3://${{ vars.CLOUDFLARE_R2_RAWDATA_BUCKET_NAME }}/ data/sfs_json/ \
--endpoint-url https://${{ secrets.CLOUDFLARE_R2_ACCOUNT_ID }}.r2.cloudflarestorage.com \
--exclude "*" \
--include "*.json"
Expand Down
16 changes: 6 additions & 10 deletions .github/workflows/html-export-workflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ jobs:

# Download all JSON files from R2
mkdir -p data/sfs_json
aws s3 sync s3://${{ secrets.CLOUDFLARE_R2_BUCKET_NAME }}/sfs_json/ data/sfs_json/ \
aws s3 sync s3://${{ vars.CLOUDFLARE_R2_RAWDATA_BUCKET_NAME }}/ data/sfs_json/ \
--endpoint-url https://${{ secrets.CLOUDFLARE_R2_ACCOUNT_ID }}.r2.cloudflarestorage.com \
--exclude "*" \
--include "*.json"
Expand Down Expand Up @@ -104,10 +104,6 @@ jobs:
echo "Error: CLOUDFLARE_R2_SECRET_ACCESS_KEY secret is not set"
exit 1
fi
if [ -z "${{ secrets.CLOUDFLARE_R2_BUCKET_NAME }}" ]; then
echo "Error: CLOUDFLARE_R2_BUCKET_NAME secret is not set"
exit 1
fi
if [ -z "${{ secrets.CLOUDFLARE_R2_ACCOUNT_ID }}" ]; then
echo "Error: CLOUDFLARE_R2_ACCOUNT_ID secret is not set"
exit 1
Expand All @@ -123,7 +119,7 @@ jobs:

- name: Upload HTML folder to Cloudflare R2
run: |
aws s3 sync output/html/ s3://${{ secrets.CLOUDFLARE_R2_BUCKET_NAME }}/sfs/ \
aws s3 sync output/html/ s3://${{ vars.CLOUDFLARE_R2_HTMLEXPORT_BUCKET_NAME }}/sfs/ \
--endpoint-url https://${{ secrets.CLOUDFLARE_R2_ACCOUNT_ID }}.r2.cloudflarestorage.com \
--delete \
--cache-control "public, max-age=3600" \
Expand All @@ -135,11 +131,11 @@ jobs:

- name: Upload index pages to Cloudflare R2
run: |
aws s3 cp index.html s3://${{ secrets.CLOUDFLARE_R2_BUCKET_NAME }}/index.html \
aws s3 cp index.html s3://${{ vars.CLOUDFLARE_R2_HTMLEXPORT_BUCKET_NAME }}/index.html \
--endpoint-url https://${{ secrets.CLOUDFLARE_R2_ACCOUNT_ID }}.r2.cloudflarestorage.com \
--cache-control "public, max-age=1800" \
--content-type "text/html"
aws s3 cp latest.html s3://${{ secrets.CLOUDFLARE_R2_BUCKET_NAME }}/latest.html \
aws s3 cp latest.html s3://${{ vars.CLOUDFLARE_R2_HTMLEXPORT_BUCKET_NAME }}/latest.html \
--endpoint-url https://${{ secrets.CLOUDFLARE_R2_ACCOUNT_ID }}.r2.cloudflarestorage.com \
--cache-control "public, max-age=1800" \
--content-type "text/html"
Expand All @@ -149,7 +145,7 @@ jobs:
- name: Upload summary
run: |
echo "HTML export completed at $(date)" > upload-summary.txt
echo "Files uploaded to Cloudflare R2 bucket: ${{ secrets.CLOUDFLARE_R2_BUCKET_NAME }}/sfs/" >> upload-summary.txt
echo "Files uploaded to Cloudflare R2 bucket: ${{ vars.CLOUDFLARE_R2_HTMLEXPORT_BUCKET_NAME }}/sfs/" >> upload-summary.txt
echo "Index pages uploaded: index.html (30 senaste), latest.html (10 senaste)" >> upload-summary.txt
aws s3 cp upload-summary.txt s3://${{ secrets.CLOUDFLARE_R2_BUCKET_NAME }}/sfs/last-update.txt \
aws s3 cp upload-summary.txt s3://${{ vars.CLOUDFLARE_R2_HTMLEXPORT_BUCKET_NAME }}/sfs/last-update.txt \
--endpoint-url https://${{ secrets.CLOUDFLARE_R2_ACCOUNT_ID }}.r2.cloudflarestorage.com