Skip to content

Commit c2e497a

Browse files
RoelBollens-TomTomRoelBollens-TomTom
committed
Merge branch 'staging'
2 parents f028083 + 5ac1834 commit c2e497a

77 files changed

Lines changed: 810 additions & 15 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/check-python-code.yaml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
name: Check Python package code
2+
3+
on:
4+
pull_request_target:
5+
paths:
6+
- 'packages/**'
7+
- 'pyproject.toml'
8+
- 'uv.lock'
9+
push:
10+
branches: [main, dev]
11+
paths:
12+
- 'packages/**'
13+
- 'pyproject.toml'
14+
- 'uv.lock'
15+
16+
jobs:
17+
check:
18+
runs-on: ubuntu-latest
19+
if: github.event.pull_request.head.repo.full_name == github.repository
20+
21+
steps:
22+
- uses: actions/checkout@v4
23+
with:
24+
ref: ${{ github.event.pull_request.head.sha }}
25+
26+
- name: Install uv
27+
uses: astral-sh/setup-uv@v4
28+
with:
29+
version: "latest"
30+
31+
- name: Set up Python
32+
uses: actions/setup-python@v5
33+
with:
34+
python-version: "3.10"
35+
36+
- name: Run make check
37+
run: make check

‎.github/workflows/check-python-package-versions.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
name: Check Python package version numbers
2+
3+
on:
4+
pull_request_target:
5+
paths:
6+
- '**/pyproject.toml'
7+
- 'packages/**/__about__.py'
8+
9+
permissions:
10+
id-token: write
11+
contents: read
12+
13+
jobs:
14+
check:
15+
if: github.event.pull_request.head.repo.full_name == github.repository
16+
uses: ./.github/workflows/reusable-check-python-package-versions.yaml
17+
with:
18+
before_commit: ${{ github.event.pull_request.base.sha }}
19+
after_commit: ${{ github.event.pull_request.head.sha }}

.github/workflows/github-actions-copy-latest-docs-to-staging.yaml renamed to .github/workflows/copy-latest-docs-to-staging.yaml

File renamed without changes.

.github/workflows/github-actions-copy-pr-docs-to-staging.yaml renamed to .github/workflows/copy-pr-docs-to-staging.yaml

File renamed without changes.

.github/workflows/github-actions-enforce-change-type-label.yaml renamed to .github/workflows/enforce-change-type-label.yaml

File renamed without changes.

.github/workflows/github-actions-publish-docs-gh-pages.yaml renamed to .github/workflows/publish-docs-gh-pages.yaml

File renamed without changes.

‎.github/workflows/reusable-check-python-package-versions.yaml‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
name: "[REUSABLE] Check Python package versions"
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
before_commit:
7+
description: >-
8+
The base Git commit to compare against, i.e., the base of the PR or the previous commit
9+
in a push.
10+
type: string
11+
required: true
12+
after_commit:
13+
description: >-
14+
The Git commit representing the head of the change to be checked, i.e. the head of the
15+
PR or the latest commit in a push.
16+
type: string
17+
required: true
18+
19+
jobs:
20+
get-index-url:
21+
uses: ./.github/workflows/reusable-get-code-artifact-index-url.yaml
22+
23+
check-python-package-versions:
24+
needs: get-index-url
25+
runs-on: ubuntu-latest
26+
steps:
27+
- name: Install jq
28+
run: sudo apt-get update && sudo apt-get install -y jq
29+
30+
- name: Install uv
31+
uses: astral-sh/setup-uv@v4
32+
with:
33+
version: "latest"
34+
35+
- name: Set up Python
36+
uses: actions/setup-python@v5
37+
with:
38+
python-version: "3.10"
39+
40+
- name: Check out code before change
41+
uses: actions/checkout@v4
42+
with:
43+
ref: ${{ inputs.before_commit }}
44+
45+
- name: Sync code before change to make packages visible to Python
46+
run: uv sync --all-packages
47+
48+
- name: Capture package versions before change
49+
run: uv run python ./.github/workflows/scripts/package-versions.py collect > /tmp/package-versions-before.json
50+
51+
- name: Check out code after change
52+
uses: actions/checkout@v6
53+
with:
54+
ref: ${{ inputs.after_commit }}
55+
56+
- name: Sync code after change to make packages visible to Python
57+
run: uv sync --all-packages --refresh
58+
59+
- name: Capture package versions after change
60+
run: uv run python ./.github/workflows/scripts/package-versions.py collect > /tmp/package-versions-after.json
61+
62+
- name: Compare package versions before and after change
63+
run: |
64+
uv run python ./.github/workflows/scripts/package-versions.py compare \
65+
/tmp/package-versions-before.json \
66+
/tmp/package-versions-after.json \
67+
>/tmp/package-version-diff.json
68+
69+
- name: Print changed versions
70+
run: cat /tmp/package-version-diff.json
71+
72+
- name: Fail if any of the new versions already exist in the repo
73+
run: |
74+
jq -c '.[]' /tmp/package-version-diff.json | while read -r entry; do
75+
package=$(echo "$entry" | jq -r '.package')
76+
after=$(echo "$entry" | jq -r '.after')
77+
exit_code=0
78+
output=$(uv run pip download "${package}==${after}" --index-url "${{ needs.get-index-url.outputs.index_url }}simple/" --no-deps -d /tmp --quiet 2>&1) || exit_code=$?
79+
if [[ $exit_code -eq 0 || (
80+
"${output,,}" != *"could not find a version"* &&
81+
"${output,,}" != *"no matching distributions"*
82+
) ]]; then
83+
echo "Package ${package} version ${after} already exists in the repository. Failing the workflow."
84+
echo " pip exit code: ${exit_code}."
85+
echo " pip stderr: ${output}."
86+
exit 1
87+
else
88+
echo "Package ${package} version ${after} is new, as expected. Continuing."
89+
fi
90+
done

‎.github/workflows/reusable-get-code-artifact-index-url.yaml‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
name: "[REUSABLE] Get CodeArtifact Python package index URL"
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
account_id:
7+
description: The AWS account ID that owns the CodeArtifact domain
8+
type: string
9+
required: false
10+
default: 505071440022
11+
aws_region:
12+
description: The AWS region where the CodeArtifact repository is hosted
13+
type: string
14+
required: false
15+
default: us-west-2
16+
role_name:
17+
description: The name of the IAM role to assume for accessing CodeArtifact
18+
type: string
19+
required: false
20+
default: GithubActions_Schema_CodeArtifact_ReadOnly
21+
domain:
22+
description: The CodeArtifact domain name
23+
type: string
24+
required: false
25+
default: overture-pypi
26+
repository:
27+
description: The CodeArtifact repository name
28+
type: string
29+
required: false
30+
default: overture
31+
outputs:
32+
index_url:
33+
description: The CodeArtifact Python index URL
34+
value: ${{ jobs.get-code-artifact-index-url.outputs.index_url }}
35+
36+
jobs:
37+
get-code-artifact-index-url:
38+
runs-on: ubuntu-latest
39+
outputs:
40+
index_url: ${{ steps.get-code-artifact-index-url.outputs.index_url }}
41+
steps:
42+
- name: Configure AWS credentials
43+
uses: aws-actions/configure-aws-credentials@v4
44+
with:
45+
aws-region: ${{ inputs.aws_region }}
46+
role-to-assume: arn:aws:iam::${{ inputs.account_id }}:role/${{ inputs.role_name }}
47+
role-session-name: GitHubActions_${{github.job}}_${{github.run_id}}
48+
49+
- name: Get CodeArtifact authorization token
50+
id: get-code-artifact-auth-token
51+
run: |
52+
AUTH_TOKEN=$(aws codeartifact get-authorization-token \
53+
--domain ${{ inputs.domain }} \
54+
--domain-owner ${{ inputs.account_id }} \
55+
--query authorizationToken \
56+
--output text)
57+
echo "index_url=https://aws:${AUTH_TOKEN}@$${{ inputs.domain }}-${{ inputs.account_id }}.d.codeartifact.${{ inputs.aws_region }}.amazonaws.com/" >> $GITHUB_OUTPUT

‎.github/workflows/scripts/package-versions.py‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
#!/usr/bin/env python3
2+
3+
from importlib import metadata
4+
from pathlib import Path
5+
import json
6+
import sys
7+
8+
9+
def collect():
10+
"""
11+
Collect Python package versions and print them as a JSON array.
12+
13+
Form of the JSON array:
14+
15+
[ {"package": "p1", "version": "v1"}, {"package": "p2", "version": "v2"}, ... ]
16+
"""
17+
packages_dir = Path("packages")
18+
19+
packages = sorted(
20+
d.name
21+
for d in packages_dir.iterdir()
22+
if d.is_dir() and d.name.startswith("overture-schema")
23+
)
24+
25+
package_versions = [
26+
{"package": p, "version": metadata.version(p.replace("-", "."))}
27+
for p in packages
28+
]
29+
30+
print(json.dumps(package_versions, indent=2))
31+
32+
33+
def compare(before_file: str, after_file: str):
34+
"""
35+
Compare two JSON files containing package versions and print the packages that have a version
36+
number change as a JSON array.
37+
38+
Form of the JSON array:
39+
40+
[ {"package": "p1", "before": "v1", "after": "v2"}, ... ]
41+
42+
Note that `before` will be `null` if the package did not exist in the "before" file, and `after`
43+
will be `null` if the package did not exist in the "after" file.
44+
"""
45+
before_array = load(before_file)
46+
after_array = load(after_file)
47+
48+
before_dict = {item["package"]: item["version"] for item in before_array}
49+
after_dict = {item["package"]: item["version"] for item in after_array}
50+
51+
combined_keys = sorted(list(set(before_dict.keys()) | set(after_dict.keys())))
52+
53+
changed_packages = []
54+
for package in combined_keys:
55+
before_version = before_dict.get(package)
56+
after_version = after_dict.get(package)
57+
if before_version != after_version:
58+
changed_packages.append(
59+
{
60+
"package": package,
61+
"before": before_version,
62+
"after": after_version,
63+
}
64+
)
65+
66+
print(json.dumps(changed_packages, indent=2))
67+
68+
69+
def load(file_path: str) -> list[dict[str, str]]:
70+
path = Path(file_path)
71+
if not path.exists():
72+
print(f"File not found: {file_path}")
73+
sys.exit(1)
74+
75+
with path.open() as f:
76+
value = json.load(f)
77+
78+
if not isinstance(value, list):
79+
print(
80+
f"File {file_path} contains unexpected root value: expected a `list` but got value {repr(value)} of type `{type(value).__name__}`"
81+
)
82+
sys.exit(1)
83+
84+
for i, item in enumerate(value):
85+
if not isinstance(item, dict):
86+
print(
87+
f"File {file_path} contains unexpected item at index {i}: expected `dict` but got value {repr(item)} of type `{type(item).__name__}`"
88+
)
89+
sys.exit(1)
90+
elif sorted(item.keys()) != ["package", "version"]:
91+
print(
92+
f"File {file_path} contains unexpected item at index {i}: expected keys `['package', 'version']` but got keys {sorted(item.keys())}"
93+
)
94+
sys.exit(1)
95+
elif not isinstance(item["package"], str):
96+
print(
97+
f"File {file_path} contains unexpected item at index {i}: expected `package` to be of type `str` but got value {repr(item['package'])} of type `{type(item['package']).__name__}`"
98+
)
99+
sys.exit(1)
100+
elif not isinstance(item["version"], str):
101+
print(
102+
f"File {file_path} contains unexpected item at index {i}: expected `version` to be of type `str` but got value {repr(item['version'])} of type `{type(item['version']).__name__}`"
103+
)
104+
sys.exit(1)
105+
106+
return value
107+
108+
109+
def usage():
110+
print("Usage:")
111+
print(f" ./{sys.argv[0]} collect")
112+
print(f" ./{sys.argv[0]} compare BEFORE_FILE AFTER_FILE")
113+
sys.exit(1)
114+
115+
116+
def main():
117+
if len(sys.argv) < 2:
118+
usage()
119+
120+
cmd = sys.argv[1]
121+
122+
if cmd == "collect":
123+
collect()
124+
elif cmd == "compare":
125+
if len(sys.argv) != 4:
126+
usage()
127+
compare(sys.argv[2], sys.argv[3])
128+
else:
129+
print(f"Unknown command: {cmd}")
130+
usage()
131+
132+
133+
if __name__ == "__main__":
134+
main()

.github/workflows/github-actions-test-schema.yaml renamed to .github/workflows/test-schema.yaml

File renamed without changes.

0 commit comments

Comments
 (0)