Skip to content

Commit 0271b2a

Browse files
LeiyksLeiyks
authored
feat(ci): merge-gate job that checks failures against a flaky allowlist (#4028)
Add a single `merge-gate` job (new final stage, runs on every branch via `when: always`) that, after the whole pipeline runs, uses the GitLab API (short-lived token fetched like `analyze and create pr`) to collect every failed job across this pipeline and its triggered child pipelines, then matches each failure against the glob patterns in .gitlab/flaky-jobs.txt. A failure matching no pattern is a real regression and fails the gate; failures that match are known-flaky and ignored. The gate is green iff every non-flaky job passed, so flaky jobs may fail (pipeline red) without blocking the merge. No `allow_failure` is added to any job. The gate logic lives in .gitlab/merge-gate.sh (keeps the YAML clean). .gitlab/flaky-jobs.txt is glob-reduced ("base:*" — any failing version marks all versions flaky) and regenerated monthly from the "failed on master" export by a script kept outside the repo.
1 parent 4e0c86b commit 0271b2a

3 files changed

Lines changed: 239 additions & 0 deletions

File tree

‎.gitlab-ci.yml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ stages:
33
- tests
44
- deploy
55
- ci-build
6+
- merge-gate
67

78
variables:
89
GIT_SUBMODULE_STRATEGY: recursive
@@ -111,6 +112,17 @@ shared-trigger:
111112
variables:
112113
PARENT_PIPELINE_ID: $CI_PIPELINE_ID
113114

115+
merge-gate:
116+
stage: merge-gate
117+
image: registry.ddbuild.io/images/dd-octo-sts-ci-base:2025.06-1
118+
tags: [ "arch:amd64" ]
119+
when: always
120+
variables:
121+
GIT_SUBMODULE_STRATEGY: none
122+
GIT_DEPTH: "1"
123+
script:
124+
- .gitlab/merge-gate.sh
125+
114126
package-trigger:
115127
stage: build
116128
needs: [ "generate-templates" ]

‎.gitlab/flaky-jobs.txt‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
# flaky-jobs.txt — glob patterns for CI jobs EXCLUDED from the merge gate.
2+
3+
aggregate tested versions
4+
Amazon_Linux_2023_amd64.SI94:*
5+
Amazon_Linux_2023_amd64.SID0:*
6+
Amazon_Linux_2_arm64.SI94:*
7+
Amazon_Linux_2_arm64.SID0:*
8+
API unit tests:*
9+
appsec integration tests (helper-rust):*
10+
appsec integration tests (ssi):*
11+
appsec integration tests:*
12+
ASAN test_c with multiple observers:*
13+
ASAN test_c:*
14+
Cargo test
15+
compile extension windows:*
16+
compile extension: debug-zts-asan:*
17+
compile extension: debug:*
18+
compile tracing extension asan:*
19+
compile tracing extension:*
20+
compile tracing sidecar:*
21+
create-multiarch-lib-injection-image
22+
Debian_11_amd64.SID0:*
23+
Debian_12_amd64.SID0:*
24+
deploy-s3
25+
Disabled test_c run:*
26+
Extension Tea Tests:*
27+
framework test:*
28+
generate-lib-init-pinned-tag-values
29+
helper-rust integration coverage
30+
installer tests
31+
K8S_LIB_INJECTION_UDS:*
32+
linux-php-laravel-realworld-parallel
33+
linux-php-symfony-realworld-parallel
34+
Loader test on amd64 libc:*
35+
macrobenchmarks:*
36+
min install tests
37+
package-oci:*
38+
pecl tests:*
39+
PHP Language Tests:*
40+
PHP language tests:*
41+
php-app.amd64.DOC:*
42+
php-app.arm64.DOC:*
43+
php-laravel-realworld-parallel-check-slo-breaches
44+
php-symfony-realworld-parallel-check-slo-breaches
45+
php-symfony-realworld-parallel-generate-slos
46+
php-symfony-realworld-parallel-upload-to-bp-api
47+
PHP: FrankePHP Demo
48+
PHP: Laravel Release
49+
PHP: Shopware Demo
50+
PHP: Symfony Demo
51+
profiling tests:*
52+
randomized tests:*
53+
RedHat_8_6_arm64.SID0:*
54+
System Tests:*
55+
test appsec extension:*
56+
test early PHP 8.1
57+
test_auto_instrumentation:*
58+
test_composer:*
59+
test_distributed_tracing:*
60+
test_extension_ci:*
61+
test_integrations_amqp2:*
62+
test_integrations_curl:*
63+
test_integrations_deferred_loading:*
64+
test_integrations_elasticsearch8:*
65+
test_integrations_elasticsearch_latest:*
66+
test_integrations_frankenphp:*
67+
test_integrations_googlespanner_latest:*
68+
test_integrations_guzzle_latest:*
69+
test_integrations_kafka:*
70+
test_integrations_memcached:*
71+
test_integrations_monolog2:*
72+
test_integrations_mysqli:*
73+
test_integrations_openai_latest:*
74+
test_integrations_pcntl:*
75+
test_integrations_predis_2:*
76+
test_integrations_sqlsrv:*
77+
test_integrations_swoole_5:*
78+
test_metrics:*
79+
test_opentelemetry_1:*
80+
test_opentelemetry_beta:*
81+
test_opentracing_10:*
82+
test_web_cakephp_28:*
83+
test_web_cakephp_310:*
84+
test_web_cakephp_45:*
85+
test_web_cakephp_latest:*
86+
test_web_codeigniter_22:*
87+
test_web_codeigniter_31:*
88+
test_web_custom:*
89+
test_web_drupal_101:*
90+
test_web_drupal_89:*
91+
test_web_drupal_95:*
92+
test_web_laminas_mvc_33:*
93+
test_web_laminas_mvc_latest:*
94+
test_web_laminas_rest_latest:*
95+
test_web_laravel_10x:*
96+
test_web_laravel_11x:*
97+
test_web_laravel_42:*
98+
test_web_laravel_57:*
99+
test_web_laravel_58:*
100+
test_web_laravel_8x:*
101+
test_web_laravel_9x:*
102+
test_web_laravel_latest:*
103+
test_web_laravel_octane_latest:*
104+
test_web_lumen_100:*
105+
test_web_lumen_52:*
106+
test_web_lumen_56:*
107+
test_web_lumen_58:*
108+
test_web_lumen_81:*
109+
test_web_lumen_90:*
110+
test_web_magento_23:*
111+
test_web_magento_24:*
112+
test_web_nette_24:*
113+
test_web_nette_31:*
114+
test_web_nette_latest:*
115+
test_web_slim_312:*
116+
test_web_slim_48:*
117+
test_web_slim_latest:*
118+
test_web_symfony_23:*
119+
test_web_symfony_28:*
120+
test_web_symfony_30:*
121+
test_web_symfony_33:*
122+
test_web_symfony_34:*
123+
test_web_symfony_40:*
124+
test_web_symfony_42:*
125+
test_web_symfony_44:*
126+
test_web_symfony_50:*
127+
test_web_symfony_51:*
128+
test_web_symfony_52:*
129+
test_web_symfony_62:*
130+
test_web_symfony_73:*
131+
test_web_symfony_latest:*
132+
test_web_wordpress_48:*
133+
test_web_wordpress_55:*
134+
test_web_wordpress_59:*
135+
test_web_wordpress_61:*
136+
test_web_yii_2049:*
137+
test_web_yii_latest:*
138+
test_web_zend_1:*
139+
test_web_zend_1_21:*
140+
Ubuntu_20_amd64.SID0:*
141+
Ubuntu_22_arm64.SI94:*
142+
Ubuntu_22_arm64.SID0:*
143+
Ubuntu_23_10_arm64.SI94:*
144+
Ubuntu_23_10_arm64.SID0:*
145+
Ubuntu_23_10_arm64.SIM:*
146+
Ubuntu_24_10_amd64.SID0:*
147+
Ubuntu_24_amd64.SID0:*
148+
Unit tests:*
149+
update-latest-versions
150+
verify windows
151+
windows test_c:*
152+
Zend Abstract Interface Tests:*

‎.gitlab/merge-gate.sh‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
#!/usr/bin/env bash
2+
# Merge gate: passes iff every non-flaky job in this pipeline (and its triggered
3+
# child pipelines) succeeded. It collects failed jobs via the GitLab API and
4+
# classifies each against the glob patterns in .gitlab/flaky-jobs.txt; a failure
5+
# matching no pattern is a real regression and fails the gate. See the
6+
# `merge-gate` job in .gitlab-ci.yml.
7+
set -uo pipefail
8+
9+
# Short-lived GitLab API token, same path as `analyze and create pr`
10+
# (Vault-issued JWT for the 'sdm' audience -> BTI CI API).
11+
_vault_jwt() {
12+
local audience="$1"
13+
if [ -n "${VAULT_ADDR:-}" ]; then
14+
curl -sf -H "X-Vault-Request: true" \
15+
"${VAULT_ADDR}/v1/identity/oidc/token/${audience}" | jq -r '.data.token' 2>/dev/null && return 0
16+
fi
17+
if [ -n "${DD_DATACENTER:-}" ]; then
18+
curl -sf -H "X-Vault-Request: true" \
19+
"https://vault.${DD_DATACENTER}/v1/identity/oidc/token/${audience}" | jq -r '.data.token' 2>/dev/null && return 0
20+
fi
21+
return 1
22+
}
23+
BTI_JWT=$(_vault_jwt sdm) || { echo "ERROR: could not obtain a BTI JWT" >&2; exit 1; }
24+
GITLAB_TOKEN=$(curl -sf -H "Authorization: Bearer ${BTI_JWT}" \
25+
"https://bti-ci-api.us1.ddbuild.io/internal/ci/gitlab/token?owner=DataDog&repository=dd-trace-php" \
26+
| jq -r '.token')
27+
GITLAB_API="https://gitlab.ddbuild.io/api/v4"
28+
AUTH="PRIVATE-TOKEN: ${GITLAB_TOKEN}"
29+
30+
# Pipelines to inspect: this parent pipeline + every triggered child.
31+
pipelines=("${CI_PIPELINE_ID}")
32+
bridges=$(curl -sf -H "${AUTH}" \
33+
"${GITLAB_API}/projects/${CI_PROJECT_ID}/pipelines/${CI_PIPELINE_ID}/bridges?per_page=100" || echo "[]")
34+
while read -r child; do
35+
[ -n "${child}" ] && pipelines+=("${child}")
36+
done < <(echo "${bridges}" | jq -r '.[] | select(.downstream_pipeline != null) | .downstream_pipeline.id')
37+
38+
# Collect the names of all failed jobs across those pipelines.
39+
: > failed_jobs.txt
40+
for pid in "${pipelines[@]}"; do
41+
for page in 1 2 3 4 5; do
42+
data=$(curl -g -sf -H "${AUTH}" \
43+
"${GITLAB_API}/projects/${CI_PROJECT_ID}/pipelines/${pid}/jobs?scope[]=failed&per_page=100&page=${page}" || echo "[]")
44+
echo "${data}" | jq -r '.[] | select(.status == "failed") | .name' >> failed_jobs.txt
45+
[ "$(echo "${data}" | jq 'length')" -lt 100 ] && break
46+
done
47+
done
48+
sort -u failed_jobs.txt -o failed_jobs.txt
49+
50+
# Load flaky globs and classify each failure.
51+
mapfile -t GLOBS < <(grep -vE '^[[:space:]]*(#|$)' .gitlab/flaky-jobs.txt)
52+
echo "Loaded ${#GLOBS[@]} flaky patterns; $(wc -l < failed_jobs.txt) distinct failed job(s)."
53+
blocking=0
54+
while IFS= read -r job; do
55+
[ -z "${job}" ] && continue
56+
[ "${job}" = "merge-gate" ] && continue
57+
ok=0
58+
for g in "${GLOBS[@]}"; do
59+
if [[ "${job}" == $g ]]; then ok=1; break; fi
60+
done
61+
if [ "${ok}" -eq 0 ]; then
62+
echo " ✗ non-flaky failure: ${job}"
63+
blocking=1
64+
else
65+
echo " ✓ known-flaky: ${job}"
66+
fi
67+
done < failed_jobs.txt
68+
69+
if [ "${blocking}" -ne 0 ]; then
70+
echo ""
71+
echo "Merge gate FAILED — a required (non-flaky) job failed. See ✗ lines above."
72+
exit 1
73+
fi
74+
echo ""
75+
echo "Merge gate PASSED — no failures, or all failures are known-flaky."

0 commit comments

Comments
 (0)