Skip to content

Commit c2841b2

Browse files
FestiveKyleFestiveKyle
authored
Refactor CI pipeline to build and use a dedicated local test image (#7257)
* Refactor CI pipeline to build and use a dedicated local test image * Upgrade typing_extensions to 4.15.0 in requirements.txt * Revert "Upgrade typing_extensions to 4.15.0 in requirements.txt" This reverts commit 2d5375c. * Pin CosmosDB emulator image to `vnext-preview` * Remove `tests` directory from .dockerignore in update-selectors * Switch CosmosDB connection string to use HTTP instead of HTTPS in tests * Add missing python packages to CI build step for update-selectors service * Refactor CI pipeline to build and use a dedicated local test image * Revert "Upgrade typing_extensions to 4.15.0 in requirements.txt" This reverts commit 2d5375c. * Pin CosmosDB emulator image to `vnext-preview` * Remove `tests` directory from .dockerignore in update-selectors * Switch CosmosDB connection string to use HTTP instead of HTTPS in tests * Add missing python packages to CI build step for update-selectors service * Bump typing_extensions to version 4.15.0 * Move test cosmos connection back to https * Override test cosmosdb advertised endpoint * Remove extra cloudbuild test-results environment variable * Use "base" image in update-selectors/Dockerfile * Base update selectors Dockerfile "test" step off "production" step * Switch to dockerhub supplied docker images for cloudbuild in update-selectors service * Fix Docker image name in cloudbuild.yaml for update-selectors service test step * Add logging for image name during Docker build in cloudbuild.yaml * Fix string concatenation for image name in cloudbuild.yaml
1 parent 60219b6 commit c2841b2

4 files changed

Lines changed: 58 additions & 38 deletions

File tree

‎services/update-selectors/.dockerignore‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
**/*.env
22
Dockerfile
3-
tests
43
*.yaml
54
*.md
65
.venv

‎services/update-selectors/Dockerfile‎

Lines changed: 22 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,12 @@
1-
FROM python:3.14.3-alpine AS python-builder
1+
FROM python:3.14.3-alpine AS base
2+
ENV PYTHONUNBUFFERED 1
3+
ENV PYTHONWARNINGS ignore
4+
5+
#===============================================================================================
6+
7+
FROM base AS python-builder
28

39
# Copy local code to the container image.
4-
ENV PYTHONWARNINGS ignore
5-
ENV PYTHONUNBUFFERED 1
610
WORKDIR /working/install
711

812
RUN apk add --no-cache \
@@ -13,18 +17,14 @@ RUN apk add --no-cache \
1317
build-base \
1418
python3-dev
1519

16-
COPY requirements.txt /requirements.txt
20+
COPY requirements.txt ./requirements.txt
1721
# Install python requirements to /working/install directory for cleaner copy
18-
RUN pip3 install --prefix=/working/install -r /requirements.txt
22+
RUN pip3 install --prefix=/working/install -r ./requirements.txt
1923

20-
#===============================================================================================
2124
#===============================================================================================
2225

23-
FROM python:3.14.3-alpine
26+
FROM base AS production
2427

25-
# Copy local code to the container image.
26-
ENV PYTHONUNBUFFERED 1
27-
ENV PYTHONWARNINGS ignore
2828
WORKDIR /selectors
2929

3030
# Copy installed python modules
@@ -36,3 +36,15 @@ RUN adduser -D selectors
3636
USER selectors
3737

3838
CMD ["python3", "main.py"]
39+
40+
#===============================================================================================
41+
42+
FROM production AS test
43+
44+
# Copy test files (which were excluded from release)
45+
COPY . .
46+
USER selectors
47+
48+
#===============================================================================================
49+
50+
FROM production AS final

‎services/update-selectors/cloudbuild.yaml‎

Lines changed: 32 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,68 +1,74 @@
11
steps:
2-
- name: "gcr.io/cloud-builders/docker"
2+
- name: 'docker:29'
33
id: start_testdb
4-
entrypoint: /bin/sh
4+
entrypoint: /bin/ash
55
args:
66
[
7-
"-c",
8-
"docker run --net cloudbuild --name arangodb -e ARANGO_NO_AUTH=1 -d -p 127.0.0.1:8529:8529 arangodb/arangodb:3.12.1",
7+
'-c',
8+
'docker run --net cloudbuild --name arangodb -e ARANGO_NO_AUTH=1 -d -p 127.0.0.1:8529:8529 arangodb/arangodb:3.12.1',
99
]
1010

1111
- name: mikewilliamson/wait-for
1212
id: wait_testdb
13-
args: ["arangodb:8529"]
13+
args: ['arangodb:8529']
1414

15-
- name: "gcr.io/cloud-builders/docker"
15+
- name: 'docker:29'
1616
id: start_test_cosmosdb
17-
entrypoint: /bin/sh
17+
entrypoint: /bin/ash
1818
args:
1919
[
20-
"-c",
21-
"docker run --net cloudbuild --name cosmosdb -d -p 8081:8081 -p 10250-10255:10250-10255 mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest",
20+
'-c',
21+
'docker run --net cloudbuild --name cosmosdb -d -p 8081:8081 -p 10250-10255:10250-10255 -e PROTOCOL=https -e GATEWAY_PUBLIC_ENDPOINT=cosmosdb mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview',
2222
]
2323

2424
- name: mikewilliamson/wait-for
2525
id: wait_test_cosmosdb
26-
args: ["cosmosdb:8081", "--timeout=180"]
26+
args: ['cosmosdb:8081', '--timeout=180']
2727

28-
- name: "northamerica-northeast1-docker.pkg.dev/track-compliance/tracker/ci"
29-
id: test-results
28+
- name: 'docker:29'
29+
id: build-test-image
3030
dir: services/update-selectors
31-
entrypoint: /bin/sh
32-
args: ["-c", "pip3 install -r requirements.txt && python3 -m pytest -v"]
31+
args: ['build', '--target=test', '-t', 'test-image', '.']
32+
33+
- name: 'test-image'
34+
id: test-results
35+
dir: /selectors
36+
entrypoint: python3
37+
args: [ '-m', 'pytest', '-v' ]
3338
env:
3439
- ARANGO_DB_URL=http://arangodb:8529
3540
- ARANGO_DB_USER=root
36-
- ARANGO_DB_PASSWORD=test
41+
- ARANGO_DB_PASS=test
3742
- COSMOS_DB_HOST=cosmosdb
3843

39-
- name: "gcr.io/cloud-builders/docker"
44+
- name: 'docker:29'
4045
id: generate-image-name
41-
entrypoint: "bash"
46+
entrypoint: 'ash'
4247
dir: services/update-selectors
4348
args:
44-
- "-c"
49+
- '-c'
4550
- |
46-
echo "northamerica-northeast1-docker.pkg.dev/track-compliance/tracker/services/update-selectors:$(echo $BRANCH_NAME | sed 's/[^a-zA-Z0-9]/-/g')-$SHORT_SHA-$(date +%s)" > /workspace/imagename
51+
echo 'northamerica-northeast1-docker.pkg.dev/track-compliance/tracker/services/update-selectors:'"$(echo $BRANCH_NAME | sed 's/[^a-zA-Z0-9]/-/g')-$SHORT_SHA-$(date +%s)" > /workspace/imagename
4752
48-
- name: "gcr.io/cloud-builders/docker"
53+
- name: 'docker:29'
4954
id: build-results
50-
entrypoint: "bash"
55+
entrypoint: 'ash'
5156
dir: services/update-selectors
5257
args:
53-
- "-c"
58+
- '-c'
5459
- |
5560
image=$(cat /workspace/imagename)
61+
echo "Building image $image"
5662
docker build -t $image .
5763
58-
- name: "gcr.io/cloud-builders/docker"
64+
- name: 'docker:29'
5965
id: push-results-if-master
60-
entrypoint: "bash"
66+
entrypoint: 'ash'
6167
dir: services/update-selectors
6268
args:
63-
- "-c"
69+
- '-c'
6470
- |
65-
if [[ "$BRANCH_NAME" == "master" ]]
71+
if [[ '$BRANCH_NAME' == 'master' ]]
6672
then
6773
image=$(cat /workspace/imagename)
6874
docker push $image

‎services/update-selectors/tests/docker-compose.yaml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,10 @@ services:
77
environment:
88
- ARANGO_ROOT_PASSWORD=test
99
cosmosdb:
10-
image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
10+
image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview
1111
ports:
1212
- '8081:8081'
1313
- '10250-10255:10250-10255'
14+
environment:
15+
- PROTOCOL=https
16+
- GATEWAY_PUBLIC_ENDPOINT=cosmosdb

0 commit comments

Comments
 (0)