Skip to content

Commit 8766776

Browse files
mohitrajainmohitrajain
authored
Refactor ci (#771)
* WPB-16722: refractor offline ci to have pluggable scripts for artifacts * WPB-16722: fix expected path in each of the task script * WPB-16722: add logic to have expected directories and pass as arguements * WPB-16722: fix offline.yml to run default-profile and comment terraform check * fix output_dir var proc_wire_binaries.sh * WPB-16722: remove unused atch-ingress-controller-images.sh script * WPB-16722: enable terraform test and fix asset path in cd.sh * WPB-16722: remove reference for patch-ingress-controller-images from nix bundling * WPB-16722: fix the linting complaints [skip_ci] * WPB-16722: fix the linting complaints * WPB-16722: fix the relative paths for archieve * WPB-16722: fix the relative paths while archieving in tasks * WPB-16722: fix the syntax issue * WPB-16722: fix directory issue with build_adminhost_containers.sh * WPB-16722: fix directory issues with tasks * WPB-16722: enable testing the refactored asset * WPB-16722: add min build definition * WPB-16722: run min build process * WPB-16722: fix min build process to remove admin container * WPB-16722: change order of offline build to create min build first * WPB-16722: fix build_adminhost_containers.sh to have arguements to chose from, update builds json to q2-2025 and changed sequence for offline.yml * WPB-16722: fix the values path for list-helm-containers and helm tree image file creation * WPB-16722: remove more charts from min-build profile * WPB-16095: add demo build profile depending on ddefault profile * WPB-16722: building all default, demo and min profle * WPB-16722: remove extra comments and fix linting issues * WPB-16722: add debugging statements to identify the issue * WPB-16722: add debugging statements to identify the issue and keep fake-aws-sqs * WPB-16722: fix issues with scripts * WPB-16722: fix issues with scripts * WPB-16722: fix issues with scripts and improve readibility * WPB-16722: fix build containers-helm.tar creation * WPB-16722: optimize the space used by the builds by linking instead of copying and preserving the state of default-profile while using it in other profiles * WPB-16722: add debugging statements to identify the issue * WPB-16722: comment out create-build-entry call * WPB-16722: fix for the directory path in post_chart_process_1.sh * WPB-16722: fix helm dependency tree output chart output location * WPB-16722: fix demo profile processing and enable versions for wire-binaries.json * WPB-16722: fix demo profile processing and enable versions for wire-binaries.json * WPB-16722: fix demo profile processing and enable containers-system.txt dump * WPB-16722: change priority for demo profile * WPB-16722: fix newline issue * WPB-16722: fix linking issue and pass -C while archieving from default profile * WPB-16722: enablong debugging and picking selective tasks to check their output * WPB-16100: fix wire-binaries.json logic in proc_wire_binaries.sh * WPB-16722: fix logic in helm dependency tree to ignore no-image helm charts * WPB-16722: disable debugging logic * WPB-16095: fix demo build archieving tar command to correctly point to default profile * WPB-16722: remove extra helm charts from the min build * WPB-16722: remove the output dirs for default and demo post upload * WPB-16722: uncomment the debugging code * WPB-16095: fix demo build dir issue * WPB-16095: fix offlie.yaml github runner to not remove assets.tgz required for cd.sh * WPB-16722: remove the calling-test helm chart * WPB-16722: deleting old files not required in refactored-ci * WPB-16722: fix dependency tree logic to align with image archieving * WPB-16095: fix demo build to have debian-builds.json and to work dependency tree * WPB-16095: fix demo build to have debian-builds.json and to work dependency tree * WPB-16722: enable output index.txt from create-container-dump to be present in versions to have a 1:1 map of what has been packed * WPB-16095: fix to remove the tmp dir and to inherit all the versions file * WPB-16722: fix linting errors * WPB-16722: fix linting errors * WPB-16722: fix sed experission for docker image * WPB-16722: comment out images in step-certificates charts * Update prod-values.example.yaml [skip_ci] * finish rebase the master branch * WPB-16722: sync values in smallstep-accomp with upstream * WPB-16722: testing with old smallstep-accomp values * WPB-16722: remove databases-ephemeral from min and added redis-ephemeral with new values * WPB-16722: sync databases-ephemeral and redis-ephemeral * WPB-16722: fix for suggested changes in the review request [skip ci]
1 parent 3519e97 commit 8766776

25 files changed

Lines changed: 886 additions & 371 deletions

‎.github/workflows/offline.yml‎

Lines changed: 53 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -24,12 +24,6 @@ jobs:
2424
- name: Install nix environment
2525
run: nix-env -f default.nix -iA env
2626

27-
- name: Run offline build
28-
run: ./offline/ci.sh
29-
env:
30-
GPG_PRIVATE_KEY: '${{ secrets.GPG_PRIVATE_KEY }}'
31-
DOCKER_LOGIN: '${{ secrets.DOCKER_LOGIN }}'
32-
3327
- name: Get upload name
3428
id: upload_name
3529
run: |
@@ -38,10 +32,60 @@ jobs:
3832
echo ::set-output name=UPLOAD_NAME::$GITHUB_SHA
3933
# echo ::set-output name=UPLOAD_NAME::${SOURCE_TAG:-$GITHUB_SHA}
4034
41-
- name: Copy assets tarball to S3
35+
# min profile build
36+
- name: Process the min profile build
37+
run: ./offline/min-build/build.sh
38+
env:
39+
GPG_PRIVATE_KEY: '${{ secrets.GPG_PRIVATE_KEY }}'
40+
DOCKER_LOGIN: '${{ secrets.DOCKER_LOGIN }}'
41+
42+
- name: Copy min build assets tarball to S3
43+
run: |
44+
# Upload tarball for each profile by specifying their OUTPUT_TAR path
45+
aws s3 cp offline/min-build/output/assets.tgz s3://public.wire.com/artifacts/wire-server-deploy-static-min-${{ steps.upload_name.outputs.UPLOAD_NAME }}.tgz
46+
echo "Uploaded to: https://s3-$AWS_REGION.amazonaws.com/public.wire.com/artifacts/wire-server-deploy-static-min-${{ steps.upload_name.outputs.UPLOAD_NAME }}.tgz"
47+
# remove the archives from the build to optimize the space on the server
48+
rm -rf offline/min-build/output/*
49+
env:
50+
AWS_ACCESS_KEY_ID: '${{ secrets.AWS_ACCESS_KEY_ID }}'
51+
AWS_SECRET_ACCESS_KEY: '${{ secrets.AWS_SECRET_ACCESS_KEY }}'
52+
AWS_REGION: "eu-west-1"
53+
54+
# deafult profile build
55+
- name: Process the default profile build
56+
run: ./offline/default-build/build.sh
57+
env:
58+
GPG_PRIVATE_KEY: '${{ secrets.GPG_PRIVATE_KEY }}'
59+
DOCKER_LOGIN: '${{ secrets.DOCKER_LOGIN }}'
60+
61+
- name: Copy default build assets tarball to S3 and clean up
4262
run: |
43-
aws s3 cp assets.tgz s3://public.wire.com/artifacts/wire-server-deploy-static-${{ steps.upload_name.outputs.UPLOAD_NAME }}.tgz
63+
# Upload tarball for each profile by specifying their OUTPUT_TAR path
64+
aws s3 cp offline/default-build/output/assets.tgz s3://public.wire.com/artifacts/wire-server-deploy-static-${{ steps.upload_name.outputs.UPLOAD_NAME }}.tgz
4465
echo "Uploaded to: https://s3-$AWS_REGION.amazonaws.com/public.wire.com/artifacts/wire-server-deploy-static-${{ steps.upload_name.outputs.UPLOAD_NAME }}.tgz"
66+
# remove the archives from the build to optimize the space on the server
67+
rm offline/default-build/output/containers-helm.tar
68+
env:
69+
AWS_ACCESS_KEY_ID: '${{ secrets.AWS_ACCESS_KEY_ID }}'
70+
AWS_SECRET_ACCESS_KEY: '${{ secrets.AWS_SECRET_ACCESS_KEY }}'
71+
AWS_REGION: "eu-west-1"
72+
73+
# demo profile build
74+
- name: Process the demo profile build
75+
run: ./offline/demo-build/build.sh
76+
env:
77+
GPG_PRIVATE_KEY: '${{ secrets.GPG_PRIVATE_KEY }}'
78+
DOCKER_LOGIN: '${{ secrets.DOCKER_LOGIN }}'
79+
80+
- name: Copy demo build assets tarball to S3 and clean up
81+
run: |
82+
# Upload tarball for each profile by specifying their OUTPUT_TAR path
83+
aws s3 cp offline/demo-build/output/assets.tgz s3://public.wire.com/artifacts/wire-server-deploy-static-demo-${{ steps.upload_name.outputs.UPLOAD_NAME }}.tgz
84+
echo "Uploaded to: https://s3-$AWS_REGION.amazonaws.com/public.wire.com/artifacts/wire-server-deploy-static-demo-${{ steps.upload_name.outputs.UPLOAD_NAME }}.tgz"
85+
# remove the assets from the build to optimize the space on the server
86+
rm -rf offline/demo-build/output/*
87+
# removing everything except assets.tgz as it is not required anymore in the further builds
88+
find offline/default-build/output/ -mindepth 1 -maxdepth 1 ! -name 'assets.tgz' -exec rm -r {} +
4589
env:
4690
AWS_ACCESS_KEY_ID: '${{ secrets.AWS_ACCESS_KEY_ID }}'
4791
AWS_SECRET_ACCESS_KEY: '${{ secrets.AWS_SECRET_ACCESS_KEY }}'
@@ -50,7 +94,6 @@ jobs:
5094
- name: Build and upload wire-server-deploy container
5195
run: |
5296
container_image=$(nix-build --no-out-link -A container)
53-
5497
skopeo copy --retry-times 10 --dest-creds "$DOCKER_LOGIN" \
5598
docker-archive:"$container_image" \
5699
"docker://quay.io/wire/wire-server-deploy:${{ steps.upload_name.outputs.UPLOAD_NAME }}"
@@ -74,3 +117,4 @@ jobs:
74117
run: (cd terraform/examples/wire-server-deploy-offline-hetzner ; terraform init && terraform destroy -auto-approve)
75118
env:
76119
HCLOUD_TOKEN: '${{ secrets.HCLOUD_TOKEN }}'
120+

‎default.nix‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,13 +53,17 @@ rec {
5353
# Linting
5454
shellcheck
5555

56+
# general utilities for bash operations
57+
jq
58+
gnused
59+
curl
60+
5661
niv
5762
nix-prefetch-docker
5863
] ++ [
5964
profileEnv
6065
] ++ lib.optionals pkgs.stdenv.isLinux [
6166
pkgs.containerd
62-
patch-ingress-controller-images # depends on containerd, TODO: migrate to skopeo?
6367

6468

6569
# for RTP session debugging

‎nix/overlay.nix‎

Lines changed: 1 addition & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ super: {
5555
}
5656
''
5757
install -Dm755 ${./scripts/create-container-dump.sh} $out/bin/create-container-dump
58-
wrapProgram $out/bin/create-container-dump --prefix PATH : '${super.lib.makeBinPath [ self.skopeo self.create-build-entry ]}'
58+
wrapProgram $out/bin/create-container-dump --prefix PATH : '${super.lib.makeBinPath [ self.skopeo ]}'
5959
'';
6060

6161

@@ -67,22 +67,4 @@ super: {
6767
install -Dm755 ${./scripts/list-helm-containers.sh} $out/bin/list-helm-containers
6868
wrapProgram $out/bin/list-helm-containers --prefix PATH : '${super.lib.makeBinPath [ self.kubernetes-helm ]}'
6969
'';
70-
71-
patch-ingress-controller-images = super.runCommandNoCC "patch-ingress-controller-images"
72-
{
73-
nativeBuildInputs = [ super.makeWrapper ];
74-
}
75-
''
76-
install -Dm755 ${./scripts/patch-ingress-controller-images.sh} $out/bin/patch-ingress-controller-images
77-
wrapProgram $out/bin/patch-ingress-controller-images --prefix PATH : '${super.lib.makeBinPath [ self.containerd ]}'
78-
'';
79-
80-
create-build-entry = super.runCommandNoCC "create-build-entry"
81-
{
82-
nativeBuildInputs = [ super.makeWrapper ];
83-
}
84-
''
85-
install -Dm755 ${./scripts/create-build-entry.sh} $out/bin/create-build-entry
86-
wrapProgram $out/bin/create-build-entry --prefix PATH : '${super.lib.makeBinPath (with self; [ jq gnutar ])}'
87-
'';
8870
}

‎nix/scripts/create-build-entry.sh‎

Lines changed: 0 additions & 59 deletions
This file was deleted.

‎nix/scripts/create-container-dump.sh‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,5 @@ while IFS= read -r image; do
3636
docker://$image_trimmed docker-archive:${image_path} --additional-tag $image
3737
fi
3838
echo "${image_filename}.tar" >> $(realpath "$1")/index.txt
39-
create-build-entry $image_path $1
4039
fi
4140
done

‎nix/scripts/list-helm-containers.sh‎

Lines changed: 41 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,33 @@
77
# The list is sorted and deduplicated, then printed to stdout.
88
set -eou pipefail
99

10+
VALUES_DIR=""
11+
HELM_IMAGE_TREE_FILE=""
12+
13+
# Parse the arguments
14+
for arg in "$@"
15+
do
16+
case $arg in
17+
VALUES_DIR=*)
18+
VALUES_DIR="${arg#*=}"
19+
;;
20+
HELM_IMAGE_TREE_FILE=*)
21+
HELM_IMAGE_TREE_FILE="${arg#*=}"
22+
;;
23+
*)
24+
echo "Unknown argument: $arg" >&2
25+
exit 1
26+
;;
27+
esac
28+
done
29+
30+
if [[ -z "$VALUES_DIR" || -z "$HELM_IMAGE_TREE_FILE" ]]; then
31+
echo "Error: Both VALUES_DIR and HELM_IMAGE_TREE_FILE must be provided." >&2
32+
echo "Usage: $0 VALUES_DIR=<path> HELM_IMAGE_TREE_FILE=<file>" >&2
33+
exit 1
34+
fi
35+
36+
1037
# Some of these images don't contain a "latest" tag. We don't to download /ALL/
1138
# of them, but only :latest in that case - it's bad enough there's no proper
1239
# versioning here.
@@ -25,15 +52,23 @@ function optionally_complain() {
2552
done
2653
}
2754

55+
images=""
2856
# For each helm chart passed in from stdin, use the example values to
2957
# render the charts, and assemble the list of images this would fetch.
3058
while IFS= read -r chart; do
3159
echo "Running helm template on chart ${chart}…" >&2
32-
# The image values are left as-is
33-
helm template --debug "$chart" \
60+
current_images=$(helm template --debug "${chart}" \
3461
--set federate.dtls.tls.key=emptyString \
3562
--set federate.dtls.tls.crt=emptyString \
36-
$( [[ -f ./values/$(basename $chart)/prod-values.example.yaml ]] && echo "-f ./values/$(basename $chart)/prod-values.example.yaml" ) \
37-
$( [[ -f ./values/$(basename $chart)/prod-secrets.example.yaml ]] && echo "-f ./values/$(basename $chart)/prod-secrets.example.yaml" ) \
38-
| yq -r '..|.image? | select(.)' | optionally_complain | sort -u
39-
done | sort -u
63+
$( [[ -f "${VALUES_DIR}"/$(basename "${chart}")/prod-values.example.yaml ]] && echo "-f ${VALUES_DIR}/$(basename "${chart}")/prod-values.example.yaml" ) \
64+
$( [[ -f "${VALUES_DIR}"/$(basename "${chart}")/prod-secrets.example.yaml ]] && echo "-f ${VALUES_DIR}/$(basename "${chart}")/prod-secrets.example.yaml" ) \
65+
| yq -r '..|.image? | select(.)' | optionally_complain | sort -u)
66+
67+
images+="$current_images\n"
68+
if [[ -n "$current_images" ]]; then
69+
basename "${chart}" >> "${HELM_IMAGE_TREE_FILE}"
70+
echo -e "$current_images\n" >> "${HELM_IMAGE_TREE_FILE}"
71+
#echo -e "\n" >> "${HELM_IMAGE_TREE_FILE}"
72+
fi
73+
done
74+
echo -e "$images" | grep . | sort -u

‎nix/scripts/patch-ingress-controller-images.sh‎

Lines changed: 0 additions & 16 deletions
This file was deleted.

‎offline/cd.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ ssh_private_key=$(cd terraform/examples/wire-server-deploy-offline-hetzner ; ter
1515
eval `ssh-agent`
1616
ssh-add - <<< "$ssh_private_key"
1717

18-
ssh -oStrictHostKeyChecking=accept-new -oConnectionAttempts=10 "root@$adminhost" tar xzv < ./assets.tgz
18+
ssh -oStrictHostKeyChecking=accept-new -oConnectionAttempts=10 "root@$adminhost" tar xzv < offline/default-build/output/assets.tgz
1919

2020
(cd terraform/examples/wire-server-deploy-offline-hetzner; terraform output -json static-inventory)| ssh "root@$adminhost" tee ./ansible/inventory/offline/inventory.yml
2121

0 commit comments

Comments
 (0)