Skip to content

chore(ci): enhance Python E2E and SonarCloud workflows with unit and and integration tests #26481

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ulixius9 merged 13 commits into from
Mar 23, 2026
Merged

chore(ci): enhance Python E2E and SonarCloud workflows with unit and and integration tests #26481

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
6a136d2
chore(ci): enhance Python E2E and SonarCloud workflows with unit and …
SumanMaharana Mar 13, 2026
fb14dd3
Merge branch 'main' into fix-python-e2e
SumanMaharana Mar 16, 2026
f257e5e
seperate the unit and integration test
SumanMaharana Mar 16, 2026
d14ce72
address commensts
SumanMaharana Mar 19, 2026
bc79220
Potential fix for pull request finding
SumanMaharana Mar 19, 2026
f7253e4
Potential fix for pull request finding
SumanMaharana Mar 19, 2026
66a2d52
Potential fix for pull request finding
SumanMaharana Mar 19, 2026
efa49dd
Potential fix for pull request finding
SumanMaharana Mar 19, 2026
5fa43f1
Potential fix for pull request finding
SumanMaharana Mar 19, 2026
60f9039
Potential fix for pull request finding
SumanMaharana Mar 19, 2026
8fcb728
Potential fix for pull request finding
SumanMaharana Mar 20, 2026
b2cbd1b
Merge branch 'main' into fix-python-e2e
SumanMaharana Mar 20, 2026
38f413e
address comments
ulixius9 Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions .github/workflows/py-cli-e2e-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,11 @@ jobs:
id: python-e2e-test
run: |
source env/bin/activate
make coverage
cd ingestion
nox --no-venv -s unit-tests
nox --no-venv -s integration-tests -- --durations=5
env:
Comment on lines 101 to +110
TESTCONTAINERS_RYUK_DISABLED: true

Comment thread
SumanMaharana marked this conversation as resolved.
- name: Run CLI E2E Python Tests & record coverage
if: matrix.e2e-test != 'python'
Expand Down Expand Up @@ -168,7 +172,8 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: coverage-${{ matrix.e2e-test }}
path: .coverage
path: ingestion/.coverage
include-hidden-files: true
Comment thread
SumanMaharana marked this conversation as resolved.
Comment thread
SumanMaharana marked this conversation as resolved.

- name: Upload coverage artifact for CLI E2E tests
if: matrix.e2e-test != 'python' && steps.e2e-test.outcome == 'success' && env.DEBUG == 'false'
Expand Down
136 changes: 132 additions & 4 deletions .github/workflows/py-sonarcloud-nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,71 @@ env:
PROJECT_BASE_DIR: "ingestion"

jobs:
ingestion-sonarcloud-analysis:
py-unit-tests:
name: Unit Tests
timeout-minutes: 60
runs-on: ubuntu-latest
steps:
- name: Free Disk Space (Ubuntu)
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
android: true
dotnet: true
haskell: true
large-packages: false
swap-storage: true
docker-images: false

- name: Checkout
uses: actions/checkout@v4
with:
ref: ${{ env.BRANCH_NAME }}
fetch-depth: 0

- name: Setup Openmetadata Test Environment
uses: ./.github/actions/setup-openmetadata-test-environment
with:
python-version: ${{ env.PYTHON_VERSION }}
install-server: 'false'

- name: Run Unit Tests
run: |
source env/bin/activate
cd ingestion
nox --no-venv -s unit-tests
shell: bash

- name: Upload coverage artifact
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v4
with:
name: coverage-unit
path: ingestion/.coverage
include-hidden-files: true

Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The unit test job produces JUnit XML (via the nox session) but only uploads the coverage file. Since the Sonar scan runs in a separate job, the JUnit report(s) need to be uploaded as an artifact here (e.g., ingestion/junit/test-results-.xml) and later downloaded in the combine/scan job, otherwise Sonar will not find any xUnit reports (sonar.python.xunit.reportPath=junit/test-results-.xml).

Suggested change
- name: Upload JUnit test results
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v4
with:
name: junit-unit
path: ingestion/junit/test-results-*.xml

Copilot uses AI. Check for mistakes.
py-integration-tests:
name: "Integration Tests (${{ matrix.shard.name }})"
timeout-minutes: 180
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
shard:
- name: "shard-1"
nox-args: >-
tests/integration/ometa
tests/integration/postgres
tests/integration/mysql
tests/integration/profiler
tests/integration/data_quality
- name: "shard-2"
nox-args: >-
--ignore=tests/integration/ometa
--ignore=tests/integration/postgres
--ignore=tests/integration/mysql
--ignore=tests/integration/profiler
--ignore=tests/integration/data_quality
steps:
- name: Free Disk Space (Ubuntu)
uses: jlumbroso/free-disk-space@main
Expand All @@ -57,11 +119,22 @@ jobs:
args: "-m no-ui"
ingestion_dependency: "mysql,elasticsearch,sample-data"

- name: Run Python Tests & Record Coverage
- name: Run Integration Tests
run: |
source env/bin/activate
make coverage
rm pom.xml
cd ingestion
nox --no-venv -s integration-tests -- --standalone --durations=5 ${{ matrix.shard.nox-args }}
env:
TESTCONTAINERS_RYUK_DISABLED: true
shell: bash

- name: Upload coverage artifact
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v4
with:
name: coverage-integration-${{ matrix.shard.name }}
path: ingestion/.coverage
include-hidden-files: true

Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as unit tests: the integration nox session generates JUnit XML under ingestion/junit, but this job only uploads the coverage file. To keep Sonar test reporting intact when scanning in a downstream job, upload the JUnit XML as an artifact from here as well.

Suggested change
- name: Upload JUnit XML artifact
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v4
with:
name: junit-integration-${{ matrix.shard.name }}
path: ingestion/junit

Copilot uses AI. Check for mistakes.
- name: Clean Up
if: ${{ !cancelled() }}
Expand All @@ -70,6 +143,61 @@ jobs:
docker compose down --remove-orphans
sudo rm -rf ${PWD}/docker-volume

py-combine-coverage:
Comment thread
SumanMaharana marked this conversation as resolved.
if: ${{ !cancelled() }}
needs: [py-unit-tests, py-integration-tests]
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout
uses: actions/checkout@v4
with:
ref: ${{ env.BRANCH_NAME }}
fetch-depth: 0

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}

- name: Install uv
run: pip install uv
shell: bash

- name: Install coverage
run: |
python3 -m venv env
source env/bin/activate
uv pip install "coverage[toml]" nox
shell: bash

- name: Download coverage artifacts
uses: actions/download-artifact@v4
with:
pattern: coverage-*
path: ingestion/coverage-data/

Comment thread
SumanMaharana marked this conversation as resolved.
Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The combine/scan job only downloads coverage artifacts (pattern: coverage-). If unit/integration jobs upload JUnit XML artifacts for Sonar, this job also needs to download them and place them under ingestion/junit/ before running the Sonar scan; otherwise sonar.python.xunit.reportPath=junit/test-results-.xml won't resolve.

Suggested change
- name: Download JUnit artifacts
uses: actions/download-artifact@v4
with:
pattern: junit-*
path: ingestion/junit/

Copilot uses AI. Check for mistakes.
- name: Prepare coverage files
run: |
cd ingestion
[ -f coverage-data/coverage-unit/.coverage ] && mv coverage-data/coverage-unit/.coverage .coverage.unit
for dir in coverage-data/coverage-integration-*/; do
shard=$(basename "$dir" | sed 's/coverage-integration-//')
[ -f "$dir/.coverage" ] && mv "$dir/.coverage" ".coverage.integration-$shard"
done
shell: bash

- name: Combine coverage
run: |
source env/bin/activate
cd ingestion
nox --no-venv -s combine-coverage
Comment thread
SumanMaharana marked this conversation as resolved.
shell: bash

- name: Remove pom.xml
run: rm pom.xml
shell: bash

- name: Push Results To Sonar
if: ${{ !cancelled() }}
id: push-to-sonar
Expand Down
Loading