Skip to content

Commit 97abb7f

Browse files
fix(site): close audit findings with pipeline parity, safer imports, and nav cleanup
- make CI run import + linked-stub generation before Jekyll build - install jq/rsync explicitly in CI and Pages workflows - add retry/backoff for sparse clone step in import_sources.sh - generate canonical linked stubs under projects/linked with legacy docs redirects - enforce LF line endings via .gitattributes - remove unused _data/navigation.yml - fix archive nav visibility, stale site-architecture details, and dynamic footer year - ignore generated artifacts/cache paths in .gitignore
1 parent fbfda73 commit 97abb7f

10 files changed

Lines changed: 98 additions & 39 deletions

File tree

.gitattributes

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
* text=auto
2+
3+
*.sh text eol=lf
4+
*.yml text eol=lf
5+
*.yaml text eol=lf
6+
*.py text eol=lf
7+
*.md text eol=lf
8+
*.html text eol=lf
9+
*.scss text eol=lf
10+
Gemfile text eol=lf
11+
Gemfile.lock text eol=lf
12+
13+
*.png binary
14+
*.ico binary

.github/workflows/ci.yml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,5 +17,13 @@ jobs:
1717
ruby-version: "3.3" # Not needed with a .ruby-version file
1818
bundler-cache: true # runs 'bundle install' and caches installed gems automatically
1919
cache-version: 0 # Increment this number if you need to re-download cached gems
20+
- name: Install import tooling
21+
run: sudo apt-get update && sudo apt-get install -y jq rsync
22+
- name: Import remote repository docs
23+
run: bash scripts/import_sources.sh
24+
- name: Generate linked project stubs
25+
run: python3 scripts/generate_linked_redirects.py
2026
- name: Build with Jekyll
2127
run: bundle exec jekyll build
28+
env:
29+
JEKYLL_ENV: production

.github/workflows/pages.yml

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ on:
44
push: { branches: ["main"] }
55
workflow_dispatch:
66
schedule:
7-
# 23:11 EST (UTC-5) daily -> 04:11 UTC
7+
# 04:11 UTC daily (~23:11 ET winter, ~00:11 ET summer)
88
- cron: "11 4 * * *"
99

1010
permissions:
@@ -29,6 +29,9 @@ jobs:
2929
ruby-version: "3.3"
3030
bundler-cache: true
3131

32+
- name: Install import tooling
33+
run: sudo apt-get update && sudo apt-get install -y jq rsync
34+
3235
- name: Setup Pages
3336
id: pages
3437
uses: actions/configure-pages@v6

.gitignore

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,9 @@ _site/
77
# Ignore folders generated by Bundler
88
.bundle/
99
vendor/
10+
__pycache__/
11+
12+
# Ignore generated content from local import/link scripts
13+
_ext/
14+
docs/projects/linked/
15+
projects/linked/

_data/navigation.yml

Lines changed: 0 additions & 21 deletions
This file was deleted.

_includes/footer_custom.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
<p class="text-small text-grey-dk-100 mb-0" style="text-align: center;">
2-
© 2025 <a href="https://feastorg.github.io" class="text-blue-00">feastorg</a>.
2+
© {{ "now" | date: "%Y" }} <a href="https://feastorg.github.io" class="text-blue-00">feastorg</a>.
33
See the corresponding <a href="https://github.com/feastorg" class="text-grey-dk-100">GitHub repository</a> for license and source details.
44
</p>

_pages/site-architecture.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ feastorg uses a hybrid Jekyll architecture that combines automated content impor
1111

1212
## Technical Stack
1313

14-
**Core**: Jekyll 4.4+ with Just the Docs v0.10.1, Ruby 3.3, GitHub Pages
14+
**Core**: Jekyll 4.4+ with Just the Docs v0.12.0, Ruby 3.3, GitHub Pages
1515
**Plugins**: SEO-tag, feed, relative-links, redirect-from, include-cache
1616

1717
## Content Architecture
@@ -36,7 +36,7 @@ feastorg uses a hybrid Jekyll architecture that combines automated content impor
3636

3737
Some projects maintain standalone GitHub Pages sites but are linked through the hub due to complex CI/CD requirements. For example, hardware projects have their own build pipeline using [bread-infra](https://github.com/feastorg/bread-infra), or the Freeboard Project, which has its own CI/CD setup using auto-generated developer API reference and component documentation from monorepo packages, published via VuePress and GitHub Actions.
3838

39-
These are managed via `_data/linked_projects.json` and automatically generate redirect stub pages.
39+
These are managed via `_data/linked_projects.json` and automatically generate redirect stub pages under `projects/linked/` (with legacy compatibility redirects under `docs/projects/linked/`).
4040

4141
## Navigation & Styling
4242

@@ -51,7 +51,7 @@ These are managed via `_data/linked_projects.json` and automatically generate re
5151

5252
## Build Pipeline
5353

54-
**Triggers**: Push to main, daily at 23:11 EST, manual dispatch
54+
**Triggers**: Push to main, daily at 04:11 UTC (~23:11 ET in winter / ~00:11 ET in summer), manual dispatch
5555
**Process**: Environment setup → Content import → Stub generation → Jekyll build → Deploy
5656

5757
**Features**:

archive/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ title: Archive
44
permalink: /archive/
55
nav_order: 100
66
has_children: true
7+
nav_exclude: false
78
---
89

910
## Archive

scripts/generate_linked_redirects.py

Lines changed: 20 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,41 +1,54 @@
11
#!/usr/bin/env python3
22
"""
33
Generate redirect stub pages for linked projects.
4-
Each page redirects to a GitHub Pages site and is nested under 'Projects > Linked Projects'.
4+
Canonical pages are generated under projects/linked, with legacy redirects
5+
preserved under docs/projects/linked for backward compatibility.
56
"""
67

78
import json
89
from pathlib import Path
910

1011
CONFIG = Path("_data/linked_projects.json")
11-
OUT_DIR = Path("docs/projects/linked")
12+
CANONICAL_OUT_DIR = Path("projects/linked")
13+
LEGACY_OUT_DIR = Path("docs/projects/linked")
1214

1315

1416
def main():
1517
with CONFIG.open("r", encoding="utf-8") as f:
1618
data = json.load(f)
1719

1820
projects = data.get("linked_projects", [])
19-
OUT_DIR.mkdir(parents=True, exist_ok=True)
21+
CANONICAL_OUT_DIR.mkdir(parents=True, exist_ok=True)
22+
LEGACY_OUT_DIR.mkdir(parents=True, exist_ok=True)
2023

2124
for i, proj in enumerate(projects, 1):
2225
name = proj["name"]
2326
title = proj.get("title", name)
2427
url = proj["url"]
25-
path = OUT_DIR / f"{name}.md"
28+
canonical_path = CANONICAL_OUT_DIR / f"{name}.md"
29+
legacy_path = LEGACY_OUT_DIR / f"{name}.md"
2630

27-
content = f"""---
31+
canonical_content = f"""---
2832
layout: redirect
2933
title: {title}
3034
parent: "Linked Projects 🔗"
3135
grand_parent: Projects
3236
nav_order: {i}
3337
redirect_to: {url}
3438
---
39+
"""
40+
legacy_content = f"""---
41+
layout: redirect
42+
title: {title}
43+
nav_exclude: true
44+
redirect_to: {url}
45+
---
3546
"""
3647

37-
path.write_text(content.strip() + "\n", encoding="utf-8")
38-
print(f"Generated redirect: {path}")
48+
canonical_path.write_text(canonical_content.strip() + "\n", encoding="utf-8")
49+
legacy_path.write_text(legacy_content.strip() + "\n", encoding="utf-8")
50+
print(f"Generated redirect: {canonical_path}")
51+
print(f"Generated legacy redirect: {legacy_path}")
3952

4053

4154
if __name__ == "__main__":

scripts/import_sources.sh

Lines changed: 41 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,44 @@ require_cmd() {
4242
command -v "$1" >/dev/null 2>&1 || die "$1 not found"
4343
}
4444

45+
# Retry helper with exponential backoff.
46+
retry_cmd() {
47+
local label="$1"
48+
shift
49+
50+
local max_retries="${IMPORT_MAX_RETRIES:-3}"
51+
local base_delay="${IMPORT_RETRY_BASE_SECONDS:-2}"
52+
local attempt=1
53+
local delay="$base_delay"
54+
55+
while true; do
56+
if "$@"; then
57+
return 0
58+
fi
59+
if (( attempt >= max_retries )); then
60+
echo "error: ${label} failed after ${attempt} attempts" >&2
61+
return 1
62+
fi
63+
echo "warning: ${label} failed on attempt ${attempt}/${max_retries}; retrying in ${delay}s..." >&2
64+
sleep "$delay"
65+
attempt=$((attempt + 1))
66+
delay=$((delay * 2))
67+
done
68+
}
69+
70+
clone_sparse_repo() {
71+
local url="$1"
72+
local ref="$2"
73+
local tmp_abs="$3"
74+
local sub="$4"
75+
76+
rm -rf -- "${tmp_abs:?}"
77+
git clone --quiet --filter=blob:none --no-checkout --depth 1 --branch "$ref" "$url" "$tmp_abs"
78+
git -C "$tmp_abs" sparse-checkout init --cone
79+
git -C "$tmp_abs" sparse-checkout set "$sub"
80+
git -C "$tmp_abs" checkout --quiet
81+
}
82+
4583
sanitize_relpath() {
4684
# Reject empty, absolute, or parent-traversal paths.
4785
local p="${1:-}"
@@ -80,11 +118,8 @@ jq -c '.sources[]' "$MANIFEST" | while IFS= read -r row; do
80118
url="https://github.com/${repo}.git"
81119

82120
echo "::group::Clone ${repo}@${ref} -> ${tmp} (sparse: ${sub})"
83-
rm -rf -- "${TMP_ABS:?}"
84-
git clone --quiet --filter=blob:none --no-checkout --depth 1 --branch "$ref" "$url" "$TMP_ABS"
85-
git -C "$TMP_ABS" sparse-checkout init --cone
86-
git -C "$TMP_ABS" sparse-checkout set "$sub"
87-
git -C "$TMP_ABS" checkout --quiet
121+
retry_cmd "clone ${repo}@${ref}" clone_sparse_repo "$url" "$ref" "$TMP_ABS" "$sub" \
122+
|| die "unable to import ${repo}@${ref}"
88123
echo "::endgroup::"
89124

90125
echo "::group::Sync ${sub} -> ${mount}"
@@ -115,4 +150,4 @@ jq -c '.sources[]' "$MANIFEST" | while IFS= read -r row; do
115150
python3 "${ROOT}/scripts/ensure_front_matter.py" "$MOUNT_ABS"
116151
echo "::endgroup::"
117152
done
118-
# --- EOF -----------------------------------------------------------------------
153+
# --- EOF -----------------------------------------------------------------------

0 commit comments

Comments
 (0)