Skip to content

Commit d5b5297

Browse files
vaquarkhanvaquarkhan
committed
Merge issue-664-cloud-native-aws into bedrock-integration-backup
2 parents f593ee3 + 36a9b60 commit d5b5297

109 files changed

Lines changed: 16682 additions & 1100 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.asf.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,9 @@ github:
6969
require_code_owner_reviews: false
7070
required_approving_review_count: 1
7171

72+
collaborators:
73+
- andreahlert
74+
7275
notifications:
7376
commits: commits@burr.apache.org
7477
issues: notifications@burr.apache.org

‎.github/workflows/build-site.yml‎

Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
#<!--
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.
18+
#-->
19+
name: Build and Deploy Site
20+
21+
on:
22+
push:
23+
branches: [main]
24+
paths:
25+
- 'docs/**'
26+
- 'website/**'
27+
- '.github/workflows/build-site.yml'
28+
workflow_dispatch:
29+
30+
concurrency:
31+
group: "site-deploy"
32+
cancel-in-progress: true
33+
34+
jobs:
35+
build-and-deploy:
36+
runs-on: ubuntu-latest
37+
38+
steps:
39+
- uses: actions/checkout@v4
40+
with:
41+
fetch-depth: 0
42+
43+
# ── Build Next.js landing page ──
44+
- name: Set up Node.js
45+
uses: actions/setup-node@v4
46+
with:
47+
node-version: 20
48+
cache: npm
49+
cache-dependency-path: website/package-lock.json
50+
51+
- name: Install Node dependencies
52+
working-directory: ./website
53+
run: npm ci
54+
55+
- name: Build landing page
56+
working-directory: ./website
57+
run: npm run build
58+
59+
# ── Build Sphinx docs ──
60+
- name: Set up Python 3.12
61+
uses: actions/setup-python@v5
62+
with:
63+
python-version: '3.12'
64+
cache: 'pip'
65+
66+
- name: Install system dependencies
67+
run: |
68+
sudo apt-get update
69+
sudo apt-get install -y graphviz
70+
71+
- name: Install Sphinx and dependencies
72+
run: |
73+
python -m pip install --upgrade --no-cache-dir pip setuptools
74+
python -m pip install --upgrade --no-cache-dir sphinx sphinx-rtd-theme sphinx-simplepdf
75+
pip install -e ".[documentation]"
76+
77+
- name: Build Sphinx documentation
78+
working-directory: ./docs
79+
run: |
80+
python -m sphinx -T -W --keep-going -b dirhtml -d _build/doctrees -D language=en . _build/html
81+
82+
# ── Merge outputs and deploy ──
83+
- name: Assemble site
84+
run: |
85+
mkdir -p /tmp/site-output
86+
87+
# Landing page at root
88+
cp -r website/out/* /tmp/site-output/
89+
90+
# Sphinx docs at /docs/
91+
mkdir -p /tmp/site-output/docs
92+
cp -r docs/_build/html/* /tmp/site-output/docs/
93+
94+
# Redirects for old Sphinx paths
95+
cp .htaccess /tmp/site-output/
96+
97+
echo "Site structure:"
98+
ls -la /tmp/site-output/
99+
echo "Docs:"
100+
ls -la /tmp/site-output/docs/ | head -20
101+
102+
# TODO: Remove this step once we confirm ASF infra honors .htaccess (AllowOverride).
103+
# This was added as a fallback for PR #679 (landing page migration) because:
104+
# The .htaccess was not being deployed (dotglob fix above solves this) and
105+
# ASF infra may not honor .htaccess for the publish serving mode in .asf.yaml.
106+
# Once confirmed, the .htaccess 301s are sufficient and these static HTML
107+
# redirects can be removed to avoid bloating the asf-site branch.
108+
- name: Generate HTML redirect fallbacks
109+
run: |
110+
# Scan Sphinx build output and generate static HTML redirects for every page.
111+
# These act as fallback if the server does not process .htaccess.
112+
BASE_URL="https://burr.apache.org"
113+
OUTPUT="/tmp/site-output"
114+
DOCS_BUILD="docs/_build/html"
115+
COUNT=0
116+
117+
# Find every index.html in the Sphinx output (dirhtml builder creates dir/index.html per page)
118+
while IFS= read -r file; do
119+
# Get path relative to build root, e.g. "getting_started/install/index.html"
120+
rel="${file#$DOCS_BUILD/}"
121+
dir="$(dirname "$rel")"
122+
123+
# Skip the docs root (landing page lives there) and paths already in site output
124+
if [ "$dir" = "." ] || [ -e "$OUTPUT/$dir/index.html" ]; then
125+
continue
126+
fi
127+
128+
target="$BASE_URL/docs/$dir/"
129+
mkdir -p "$OUTPUT/$dir"
130+
printf '<!DOCTYPE html><html><head>\n<meta charset="utf-8">\n<meta http-equiv="refresh" content="0; url=%s">\n<link rel="canonical" href="%s">\n<script>window.location.replace("%s")</script>\n</head><body>Redirecting to <a href="%s">%s</a>...</body></html>\n' \
131+
"$target" "$target" "$target" "$target" "$target" \
132+
> "$OUTPUT/$dir/index.html"
133+
COUNT=$((COUNT + 1))
134+
done < <(find "$DOCS_BUILD" -name "index.html" -type f)
135+
136+
echo "Generated $COUNT HTML redirect fallbacks."
137+
138+
- name: Deploy to asf-site / asf-staging
139+
if: github.event_name != 'pull_request'
140+
run: |
141+
if [ "${{ github.ref }}" = "refs/heads/main" ]; then
142+
TARGET_BRANCH="asf-site"
143+
else
144+
TARGET_BRANCH="asf-staging"
145+
fi
146+
147+
git config --global user.name "GitHub Actions"
148+
git config --global user.email "actions@github.com"
149+
150+
mkdir -p /tmp/gh-pages
151+
152+
if ! git clone --branch $TARGET_BRANCH --single-branch \
153+
https://github.com/${{ github.repository }}.git /tmp/gh-pages 2>/dev/null; then
154+
rm -rf /tmp/gh-pages
155+
mkdir -p /tmp/gh-pages
156+
cd /tmp/gh-pages
157+
git init
158+
git checkout -b $TARGET_BRANCH
159+
git remote add origin https://github.com/${{ github.repository }}.git
160+
cd -
161+
fi
162+
163+
rm -rf /tmp/gh-pages/content
164+
mkdir -p /tmp/gh-pages/content
165+
shopt -s dotglob
166+
cp -r /tmp/site-output/* /tmp/gh-pages/content/
167+
168+
cd /tmp/gh-pages
169+
170+
if [ ! -f README.md ]; then
171+
echo "# Apache Burr Website" > README.md
172+
echo "This branch contains the built site (landing page + docs)." >> README.md
173+
fi
174+
175+
git add -A
176+
177+
if [ -n "$(git status --porcelain)" ]; then
178+
git commit -m "Deploy site from ${{ github.sha }}"
179+
git push https://x-access-token:${{ github.token }}@github.com/${{ github.repository }}.git $TARGET_BRANCH
180+
echo "Deployed to $TARGET_BRANCH"
181+
else
182+
echo "No changes to deploy"
183+
fi

‎.github/workflows/docs.yml‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,6 @@ on:
2828

2929
permissions:
3030
contents: write
31-
pull-requests: write
3231

3332
concurrency: ci-${{ github.ref }}
3433

@@ -37,7 +36,7 @@ jobs:
3736
runs-on: ubuntu-latest
3837
env:
3938
PR_PATH: pull/${{github.event.number}}
40-
BASE_URL: https://burr.dagworks.io/pull/${{github.event.number}}
39+
BASE_URL: https://burr.apache.org/pull/${{github.event.number}}
4140
steps:
4241
- uses: actions/checkout@v4
4342
- uses: actions/setup-python@v4
@@ -49,7 +48,7 @@ jobs:
4948
- name: Sphinx build
5049
run: |
5150
sphinx-build docs -b dirhtml _build
52-
echo "burr.dagworks.io" > _build/CNAME # keep the cname file which this clobbers -- todo, unhardcode
51+
echo "burr.apache.org" > _build/CNAME # keep the cname file which this clobbers -- todo, unhardcode
5352
- name: Comment on PR
5453
uses: hasura/comment-progress@v2.2.0
5554
if: github.ref != 'refs/heads/main'

‎.github/workflows/python-package.yml‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,12 +69,35 @@ jobs:
6969
7070
- name: Install dependencies
7171
run: |
72-
python -m pip install -e ".[tests,tracking-client,graphviz,tracking-server-s3]"
72+
python -m pip install -e ".[tests,tracking-client,graphviz]"
7373
7474
- name: Run tests
7575
run: |
7676
python -m pytest tests --ignore=tests/integrations/persisters
7777
78+
test-tracking-server-s3:
79+
runs-on: ubuntu-latest
80+
strategy:
81+
fail-fast: false
82+
matrix:
83+
python-version: ['3.9', '3.10', '3.11', '3.12']
84+
85+
steps:
86+
- uses: actions/checkout@v4
87+
88+
- name: Set up Python ${{ matrix.python-version }}
89+
uses: actions/setup-python@v4
90+
with:
91+
python-version: ${{ matrix.python-version }}
92+
93+
- name: Install dependencies
94+
run: |
95+
python -m pip install -e ".[tests,tracking-client,tracking-server-s3]"
96+
97+
- name: Run S3 tracking server tests
98+
run: |
99+
python -m pytest tests/tracking/test_bip0042_s3_buffering.py -v
100+
78101
validate-examples:
79102
runs-on: ubuntu-latest
80103
steps:

‎.github/workflows/sphinx-docs.yml‎

Lines changed: 3 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -85,73 +85,6 @@ jobs:
8585
path: docs/_build/pdf/
8686
retention-days: 5
8787

88-
- name: Deploy documentation
89-
working-directory: ./docs
90-
run: |
91-
# Set target branch based on current branch
92-
if [ "${{ github.ref }}" = "refs/heads/main" ]; then
93-
TARGET_BRANCH="asf-site"
94-
echo "Deploying to production (asf-site) branch"
95-
else
96-
TARGET_BRANCH="asf-staging"
97-
echo "Deploying to staging (asf-staging) branch"
98-
fi
99-
100-
# Configure git
101-
git config --global user.name "GitHub Actions"
102-
git config --global user.email "actions@github.com"
103-
104-
# Create a temporary directory
105-
mkdir -p /tmp/gh-pages
106-
107-
# Store current directory
108-
CURRENT_DIR=$(pwd)
109-
ls -lsa $CURRENT_DIR
110-
111-
# Try to clone the repository with the target branch
112-
if ! git clone --branch $TARGET_BRANCH --single-branch \
113-
https://github.com/${{ github.repository }}.git /tmp/gh-pages 2>/dev/null; then
114-
# If branch doesn't exist, initialize a new repository and create the branch
115-
echo "Branch $TARGET_BRANCH doesn't exist. Creating it..."
116-
rm -rf /tmp/gh-pages
117-
mkdir -p /tmp/gh-pages
118-
cd /tmp/gh-pages
119-
git init
120-
git config --local init.defaultBranch $TARGET_BRANCH
121-
git checkout -b $TARGET_BRANCH
122-
git remote add origin https://github.com/${{ github.repository }}.git
123-
cd "$CURRENT_DIR"
124-
else
125-
echo "CD'ing into $CURRENT_DIR"
126-
cd "$CURRENT_DIR"
127-
fi
128-
129-
# Remove existing content directory if it exists
130-
rm -rf /tmp/gh-pages/content
131-
132-
# # Ensure build directories exist
133-
# mkdir -p "$CURRENT_DIR/_build/html"
134-
135-
# Copy the built HTML documentation to the content directory
136-
mkdir -p /tmp/gh-pages/content
137-
cp -r "$CURRENT_DIR/_build/html/"* /tmp/gh-pages/content/ 2>/dev/null || echo "No HTML files to copy"
138-
139-
# Add, commit and push the changes
140-
cd /tmp/gh-pages
141-
git status
142-
ls -lhsa content
143-
# Create a README if it doesn't exist
144-
if [ ! -f README.md ]; then
145-
echo "# Documentation for $TARGET_BRANCH" > README.md
146-
echo "This branch contains the built documentation." >> README.md
147-
fi
148-
git add -A
149-
git status
150-
# Check if there are changes to commit (including untracked files)
151-
if [ -n "$(git status --porcelain)" ]; then
152-
git commit -m "Deploy documentation from ${{ github.sha }}"
153-
git push https://x-access-token:${{ github.token }}@github.com/${{ github.repository }}.git $TARGET_BRANCH
154-
echo "Changes pushed to $TARGET_BRANCH branch"
155-
else
156-
echo "No changes to deploy"
157-
fi
88+
# NOTE: Deployment to asf-site is now handled by build-site.yml
89+
# which builds both the landing page and Sphinx docs together.
90+
# This workflow only builds + uploads artifacts for PR review.

‎.htaccess‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
# Redirect old Sphinx root paths to /docs/ after landing page migration.
19+
# The Sphinx docs moved from / to /docs/ when the Next.js landing page
20+
# was introduced at the site root.
21+
22+
RewriteEngine On
23+
24+
# Sphinx doc sections that moved to /docs/
25+
RewriteRule ^concepts/(.*)$ /docs/concepts/$1 [R=301,L]
26+
RewriteRule ^getting_started/(.*)$ /docs/getting_started/$1 [R=301,L]
27+
RewriteRule ^reference/(.*)$ /docs/reference/$1 [R=301,L]
28+
RewriteRule ^examples/(.*)$ /docs/examples/$1 [R=301,L]
29+
RewriteRule ^contributing/(.*)$ /docs/contributing/$1 [R=301,L]
30+
RewriteRule ^asf/(.*)$ /docs/asf/$1 [R=301,L]
31+
32+
# Sphinx static assets and internals
33+
RewriteRule ^_static/(.*)$ /docs/_static/$1 [R=301,L]
34+
RewriteRule ^_sources/(.*)$ /docs/_sources/$1 [R=301,L]
35+
RewriteRule ^genindex/?$ /docs/genindex/ [R=301,L]
36+
RewriteRule ^search/?$ /docs/search/ [R=301,L]

‎README.md‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -160,9 +160,7 @@ While Apache Burr is attempting something (somewhat) unique, there are a variety
160160
## 🌯 Why the name Burr?
161161

162162
Apache Burr is named after [Aaron Burr](https://en.wikipedia.org/wiki/Aaron_Burr), founding father, third VP of the United States, and murderer/arch-nemesis of [Alexander Hamilton](https://en.wikipedia.org/wiki/Alexander_Hamilton).
163-
What's the connection with Hamilton? This is [DAGWorks](www.dagworks.io)' second open-source library release after the [Apache Hamilton library](https://github.com/apache/hamilton)
164-
We imagine a world in which Burr and Hamilton lived in harmony and saw through their differences to better the union. We originally
165-
built Apache Burr as a _harness_ to handle state between executions of Apache Hamilton DAGs (because DAGs don't have cycles),
163+
What's the connection with (Apache) Hamilton? We imagine a world in which Burr and Hamilton lived in harmony and saw through their differences to better the union. Originally Apache Burr was built as a _harness_ to handle state between executions of Apache Hamilton DAGs (because DAGs don't have cycles),
166164
but realized that it has a wide array of applications and decided to release it more broadly.
167165

168166
# Testimonials

0 commit comments

Comments
 (0)