fix: add Docker healthcheck for Dremio in CI (#941)

* fix: add retry logic for transient Dremio connection errors in CI

Dremio OSS Docker intermittently drops TCP connections under concurrent
load from 8 pytest-xdist workers, producing RemoteDisconnected errors
during dbt seed and dbt run operations.

Changes:
- Add retry logic (3 attempts, 10s delay) to dbt seed in data_seeder.py
- Add retry logic (3 attempts, 15s delay) to dbt run in env.py init()
- Add Docker healthcheck to Dremio container in docker-compose-dremio.yml
- Use 'docker compose up -d --wait' to ensure Dremio is healthy before tests
- Make dremio-setup depend on Dremio health check passing

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

* fix: replace --wait with manual Dremio health check wait

docker compose --wait fails when one-shot containers (minio-setup)
exit, even with exit code 0. Instead, poll the Dremio container's
health status directly with docker inspect.

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

* fix: catch exceptions in retry loops for seed and init operations

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

* fix: preserve exception chain in retry loops for better debugging

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

* refactor: remove redundant retry wrappers from env.py and data_seeder.py

The generic retry logic in elementary PR #2125 (CommandLineDbtRunner)
now handles transient error detection and retries for all dbt commands,
including seed and run. The CI test harness no longer needs its own
retry wrappers since it calls dbt_runner.run() and dbt_runner.seed()
which go through the runner's retry logic.

This simplifies the PR to just the Docker healthcheck improvements
for Dremio.

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Itamar Hartstein <haritamar@gmail.com>

Author: devin-ai-integration[bot]

.github/workflows/test-warehouse.yml

@@ -91,7 +91,13 @@ jobs:
       - name: Start Dremio
         if: inputs.warehouse-type == 'dremio'
         working-directory: ${{ env.TESTS_DIR }}
-        run: docker compose -f docker-compose-dremio.yml up -d
+        run: |
+          docker compose -f docker-compose-dremio.yml up -d
+          # Wait for Dremio to be healthy (one-shot containers like
+          # minio-setup exit immediately, so --wait would fail).
+          echo "Waiting for Dremio to become healthy..."
+          timeout 180 bash -c 'until [ "$(docker inspect -f {{.State.Health.Status}} dremio 2>/dev/null)" = "healthy" ]; do sleep 5; done'
+          echo "Dremio is healthy."
 
       - name: Setup Python
         uses: actions/setup-python@v6

integration_tests/docker-compose-dremio.yml

@@ -65,12 +65,19 @@ services:
       - dremio_data:/opt/dremio/data:rw
     # Workaround for permission issues in podman
     user: "0"
+    healthcheck:
+      test: ["CMD-SHELL", "curl -sf http://localhost:9047 || exit 1"]
+      interval: 5s
+      timeout: 5s
+      retries: 30
+      start_period: 15s
 
   dremio-setup:
     image: alpine:latest
     container_name: dremio-setup
     depends_on:
-      - dremio
+      dremio:
+        condition: service_healthy
     volumes:
       - ./docker/dremio/dremio-setup.sh:/dremio-setup.sh
     command: sh /dremio-setup.sh