Skip to content

Commit 3f28279

Browse files
hubciohubcio
committed
feat(ci): overhaul CI/CD with smart detection & optimized caching
Major improvements: - Intelligent change detection with dependency graph resolution - Only test affected components + transitive dependencies - Dual caching: sccache for artifacts + rust-cache for deps - Parallel matrix execution for all test suites - Python tests use server-start/stop instead of Docker - Centralized component config in components.yml - Unified test workflow handling all languages - Maven Nexus publishing for Java SDK - Multi-arch Docker builds with buildx - Manual tagging workflow for controlled releases - CI summary with execution times per component - License header & conventional commit validation Fixes: Missing Docker inputs, sccache persistence, order-sensitive pattern matching, empty duration column, Maven credentials. Reduces CI runtime by ~40% while improving reliability.
1 parent f3ff279 commit 3f28279

117 files changed

Lines changed: 6853 additions & 5547 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.dockerignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,5 +13,10 @@
1313
/local_data
1414
/performance_results
1515
/scripts
16+
**/target
1617
/target
18+
!/target/debug/iggy
19+
!/target/debug/iggy-server
20+
!/target/debug/iggy-mcp
21+
!/target/debug/iggy-connectors
1722
/web

‎.github/actions/csharp-dotnet/post-merge/action.yml‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: csharp-dotnet-post-merge
19+
description: .NET post-merge publishing github iggy actions
20+
21+
inputs:
22+
version:
23+
description: "Version for publishing"
24+
required: true
25+
dry_run:
26+
description: "Dry run mode"
27+
required: false
28+
default: "false"
29+
30+
runs:
31+
using: "composite"
32+
steps:
33+
- name: Setup .NET
34+
uses: actions/setup-dotnet@v4
35+
with:
36+
dotnet-version: "8.0.x"
37+
38+
- name: Restore dependencies
39+
run: |
40+
cd foreign/csharp
41+
dotnet restore
42+
shell: bash
43+
44+
- name: Build Release
45+
run: |
46+
cd foreign/csharp
47+
48+
# Build in Release mode
49+
dotnet build -c Release --no-restore
50+
51+
# List build output
52+
echo "Build output:"
53+
find . -name "*.dll" -path "*/bin/Release/*" | head -20 || echo "No DLLs found in Release folders"
54+
shell: bash
55+
56+
- name: Pack NuGet packages
57+
run: |
58+
cd foreign/csharp
59+
60+
# Set version if provided
61+
if [ -n "${{ inputs.version }}" ]; then
62+
dotnet pack -c Release \
63+
-p:PackageVersion=${{ inputs.version }} \
64+
-o ./nupkgs \
65+
--no-build
66+
else
67+
echo "❌ Version is required for packing"
68+
exit 1
69+
fi
70+
71+
# List packages
72+
echo "NuGet packages:"
73+
ls -la ./nupkgs/ || echo "No packages found"
74+
shell: bash
75+
76+
- name: Publish to NuGet
77+
env:
78+
NUGET_API_KEY: ${{ env.NUGET_API_KEY }}
79+
run: |
80+
cd foreign/csharp
81+
82+
if [ "${{ inputs.dry_run }}" = "true" ]; then
83+
echo "🔍 Dry run - would publish these packages:"
84+
ls -la ./nupkgs/*.nupkg
85+
86+
# Validate packages
87+
for package in ./nupkgs/*.nupkg; do
88+
echo "Validating: $package"
89+
dotnet nuget locals all --clear
90+
# Extract package info
91+
unzip -l "$package" | head -20
92+
done
93+
else
94+
if [ -z "$NUGET_API_KEY" ]; then
95+
echo "❌ NUGET_API_KEY is not set"
96+
exit 1
97+
fi
98+
99+
echo "📦 Publishing packages to NuGet..."
100+
# Push to NuGet
101+
for package in ./nupkgs/*.nupkg; do
102+
echo "Publishing: $(basename $package)"
103+
dotnet nuget push "$package" \
104+
--api-key "$NUGET_API_KEY" \
105+
--source https://api.nuget.org/v3/index.json \
106+
--skip-duplicate
107+
done
108+
echo "✅ Publishing completed"
109+
fi
110+
shell: bash

‎.github/actions/csharp-dotnet/pre-merge/action.yml‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
# TODO(hubcio): Currently, C# tests don't need server. They use testcontainers with 'edge' image.
19+
# We should change this to use server-start/stop action, so that code from PR is tested.
20+
21+
name: csharp-dotnet-pre-merge
22+
description: .NET pre-merge testing github iggy actions
23+
24+
inputs:
25+
task:
26+
description: "Task to run (lint, test, build, e2e)"
27+
required: true
28+
29+
runs:
30+
using: "composite"
31+
steps:
32+
- name: Setup .NET
33+
uses: actions/setup-dotnet@v4
34+
with:
35+
dotnet-version: "8.0.x"
36+
37+
- name: Setup Rust with cache
38+
if: inputs.task == 'test' || inputs.task == 'e2e'
39+
uses: ./.github/actions/utils/setup-rust-with-cache
40+
with:
41+
cache-targets: false # Only cache registry and git deps, not target dir (sccache handles that)
42+
43+
- name: Install netcat
44+
if: inputs.task == 'e2e'
45+
run: sudo apt-get update && sudo apt-get install -y netcat-openbsd
46+
shell: bash
47+
48+
- name: Restore dependencies
49+
run: |
50+
cd foreign/csharp
51+
dotnet restore
52+
shell: bash
53+
54+
- name: Build
55+
if: inputs.task == 'test' || inputs.task == 'build' || inputs.task == 'lint'
56+
run: |
57+
cd foreign/csharp
58+
dotnet build --no-restore
59+
shell: bash
60+
61+
- name: Lint (Code Analysis)
62+
if: inputs.task == 'lint'
63+
run: |
64+
cd foreign/csharp
65+
66+
# Run code analysis
67+
dotnet build --no-restore /p:EnforceCodeStyleInBuild=true /p:TreatWarningsAsErrors=false
68+
69+
# TODO: make format check blocking (requires dotnet-format tool)
70+
dotnet format --verify-no-changes --verbosity diagnostic || true
71+
72+
shell: bash
73+
74+
- name: Test
75+
if: inputs.task == 'test'
76+
run: |
77+
cd foreign/csharp
78+
79+
# Run unit tests
80+
dotnet test Iggy_SDK_Tests --no-build --verbosity normal
81+
82+
# Run integration tests
83+
dotnet test Iggy_SDK.Tests.Integration --no-build --verbosity normal
84+
85+
# Run BDD tests if they exist
86+
if [ -d "Iggy_SDK.Tests.BDD" ]; then
87+
dotnet test Iggy_SDK.Tests.BDD --no-build --verbosity normal
88+
fi
89+
shell: bash
90+
91+
- name: Start Iggy server
92+
id: iggy
93+
if: inputs.task == 'e2e'
94+
uses: ./.github/actions/utils/server-start
95+
with:
96+
mode: cargo
97+
cargo-bin: iggy-server
98+
port: 8090
99+
100+
- name: Run integration tests
101+
if: inputs.task == 'e2e'
102+
run: |
103+
cd foreign/csharp
104+
dotnet test Iggy_SDK.Tests.Integration --no-build --verbosity normal
105+
shell: bash
106+
107+
- name: Stop Iggy server
108+
if: inputs.task == 'e2e'
109+
uses: ./.github/actions/utils/server-stop
110+
with:
111+
pid-file: ${{ steps.iggy.outputs.pid_file }}
112+
log-file: ${{ steps.iggy.outputs.log_file }}
113+
114+
- name: Build Release
115+
if: inputs.task == 'build'
116+
run: |
117+
cd foreign/csharp
118+
119+
# Build in Release mode
120+
dotnet build -c Release --no-restore
121+
122+
# List build output
123+
echo "Build output:"
124+
find . -name "*.dll" -path "*/bin/Release/*" | head -20 || echo "No DLLs found in Release folders"
125+
shell: bash

‎.github/actions/go/post-merge/action.yml‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: go-post-merge
19+
description: Go post-merge tag preparation github iggy actions
20+
21+
inputs:
22+
version:
23+
description: "Version for tagging (without 'v' prefix)"
24+
required: true
25+
dry_run:
26+
description: "Dry run mode"
27+
required: false
28+
default: "false"
29+
30+
runs:
31+
using: "composite"
32+
steps:
33+
- name: Validate version format
34+
run: |
35+
VERSION="${{ inputs.version }}"
36+
37+
# Check if version matches semantic versioning
38+
if ! echo "$VERSION" | grep -E '^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.-]+)?(\+[a-zA-Z0-9.-]+)?$' > /dev/null; then
39+
echo "❌ Invalid version format: $VERSION"
40+
echo "Expected format: X.Y.Z or X.Y.Z-prerelease+metadata"
41+
exit 1
42+
fi
43+
44+
echo "✅ Version format valid: $VERSION"
45+
shell: bash
46+
47+
- name: Prepare Go module for tagging
48+
run: |
49+
VERSION="${{ inputs.version }}"
50+
TAG="foreign/go/v${VERSION}"
51+
52+
echo "📦 Go Module Publishing Information"
53+
echo "===================================="
54+
echo "Version: v${VERSION}"
55+
echo "Git tag: ${TAG}"
56+
echo ""
57+
58+
if [ "${{ inputs.dry_run }}" = "true" ]; then
59+
echo "🔍 DRY RUN MODE - No tag will be created"
60+
echo ""
61+
echo "Would create tag: ${TAG}"
62+
echo ""
63+
echo "After tagging, users could import using:"
64+
echo " go get github.com/${{ github.repository }}/foreign/go@v${VERSION}"
65+
echo ""
66+
echo "Or add to go.mod:"
67+
echo " require github.com/${{ github.repository }}/foreign/go v${VERSION}"
68+
else
69+
echo "✅ Go module ready for tagging"
70+
echo ""
71+
echo "Tag will be created: ${TAG}"
72+
echo "This will be handled by the create-tags job in the publish workflow"
73+
echo ""
74+
echo "After the tag is pushed, users can import using:"
75+
echo " go get github.com/${{ github.repository }}/foreign/go@v${VERSION}"
76+
echo ""
77+
echo "Or add to go.mod:"
78+
echo " require github.com/${{ github.repository }}/foreign/go v${VERSION}"
79+
fi
80+
81+
# Verify the go.mod file exists
82+
if [ ! -f "foreign/go/go.mod" ]; then
83+
echo "⚠️ Warning: foreign/go/go.mod not found"
84+
echo "Make sure the Go module is properly initialized"
85+
else
86+
echo ""
87+
echo "Module information:"
88+
grep "^module" foreign/go/go.mod || echo "Module declaration not found"
89+
fi
90+
shell: bash
91+
92+
- name: Output tag information
93+
id: tag-info
94+
run: |
95+
VERSION="${{ inputs.version }}"
96+
TAG="foreign/go/v${VERSION}"
97+
98+
# Set outputs for use in other jobs
99+
echo "tag=${TAG}" >> "$GITHUB_OUTPUT"
100+
echo "version=v${VERSION}" >> "$GITHUB_OUTPUT"
101+
102+
# Summary for GitHub Actions
103+
{
104+
echo "## 🏷️ Go Module Tag Information"
105+
echo ""
106+
echo "| Property | Value |"
107+
echo "|----------|-------|"
108+
echo "| **Version** | \`v${VERSION}\` |"
109+
echo "| **Git Tag** | \`${TAG}\` |"
110+
echo "| **Import Path** | \`github.com/${{ github.repository }}/foreign/go\` |"
111+
echo ""
112+
if [ "${{ inputs.dry_run }}" = "true" ]; then
113+
echo "**Note:** This is a dry run - no actual tag will be created"
114+
else
115+
echo "**Note:** Tag will be created by the publish workflow"
116+
fi
117+
} >> "$GITHUB_STEP_SUMMARY"
118+
shell: bash

0 commit comments

Comments
 (0)