Skip to content

Commit d718522

Browse files
committed
Setup E2E: Improve final unhealthy container checks
- Wait 5 minutes for containers to start registering as healthy. Some of them might needs some time. - Add plenty of logging to see which service is unhelthy and what it's internal Dockerfile HEALTHCHECK is registering. - Dump logs of any container that failed to register as healthy after the 5 minute retry period.
1 parent 6aebabe commit d718522

1 file changed

Lines changed: 29 additions & 2 deletions

File tree

github-actions/setup-e2e-environment/action.yml

Lines changed: 29 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -354,8 +354,35 @@ runs:
354354
run: docker ps
355355
shell: bash
356356

357-
- name: Verify that all containers are healthy
358-
run: '[ -z "$(docker ps -q --filter health=unhealthy)" ]'
357+
- name: Wait and make sure no container registers as unhealthy
358+
run: |
359+
unhealhtyContainerIds=""
360+
361+
# 30 tries, 10s wait time -> Max wait of 5 minutes
362+
# Some of the services only report new HEALTH once per minute.
363+
for i in {1..30}; do
364+
unhealhtyContainerIds=$(docker ps -q --filter health=unhealthy)
365+
366+
if [[ -z "$unhealhtyContainerIds" ]]; then
367+
echo "All containers are up and running fine"
368+
exit 0
369+
else
370+
echo "One or more containers report an unhealthy status:"
371+
docker container inspect "$unhealhtyContainerIds" | jq '.[] | { Name: .Name, Health: .State.Health }'
372+
fi
373+
374+
echo "Wait for healthy services. Try $i/30. Waiting 10s for next try."
375+
sleep 10s
376+
done
377+
378+
for id in $unhealhtyContainerIds; do
379+
echo "Container $id | $(docker container inspect $id | jq '.[0].Name') failed to register as healthy!"
380+
echo "Container $id | logs start: ├─────────────────────────────────────────────────────"
381+
docker logs $id
382+
echo "Container $id | logs end: ─────────────────────────────────────────────────────┤"
383+
done
384+
385+
exit 1
359386
shell: bash
360387

361388
- name: Extract metadata

0 commit comments

Comments
 (0)