Skip to content

Commit 833b5bd

Browse files
deepssindeepssin
committed
Add auto-ptl-batch pipeline for grouped PR testing.
Adding auto-ptl-batch job config, Jenkins pipeline flow, PR grouping/conflict logic, and overall CI status checks before pushing batch branches and triggering teuthology via teuthology-runner. Signed-off-by: deepssin <deepssin@redhat.com>
1 parent 041db81 commit 833b5bd

7 files changed

Lines changed: 1423 additions & 5 deletions

File tree

‎auto-ptl-batch/build/Jenkinsfile‎

Lines changed: 381 additions & 0 deletions
Large diffs are not rendered by default.

‎auto-ptl-batch/config/definitions/auto-ptl-batch.yml‎

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
- job:
2+
name: auto-ptl-batch
3+
description: |
4+
Automated PR batch testing for ceph/ceph.
5+
6+
The job discovers all open PRs that carry every label in REQUIRED_LABELS
7+
(default: needs-QA) plus a component label, have green CI, and
8+
carry none of the labels in EXCLUDE_LABELS. PRs are grouped by
9+
(component, base_branch), then split into sub-batches using file-path
10+
conflict detection: two PRs that touch overlapping paths at
11+
CONFLICT_PATH_DEPTH directory levels are placed into separate batches so
12+
that merge conflicts and root-cause analysis stay isolated.
13+
14+
For each resulting sub-batch:
15+
1. CI green check (overall commit status on each PR HEAD SHA)
16+
2. Local merge via ptl-tool.py (--branch HEAD; local merge only)
17+
3. Push to ceph-ci as wip-<component>-<branch>-auto-batch<N>
18+
(component label is sanitized for the branch name: allowed chars only,
19+
others become '-')
20+
- commit trailers constrain ceph-dev-pipeline build scope
21+
(DISTROS/ARCHS/FLAVORS/CI-CONTAINER are configurable job parameters)
22+
- teuthology-runner is triggered asynchronously (delay configurable)
23+
and schedules suite from
24+
COMPONENT_SUITE_MAP
25+
- teuthology-runner waits for the suite, then posts
26+
auto-ptl-batch=success or failure on each PR HEAD SHA
27+
(skipped when SKIP_STATUS_POST=true)
28+
29+
Runs once daily via a cron trigger. Can also be triggered manually;
30+
set CEPH_BASE_BRANCH to limit processing to a specific target branch,
31+
or leave empty to process all branches.
32+
project-type: pipeline
33+
quiet-period: 2
34+
concurrent: false
35+
pipeline-scm:
36+
scm:
37+
- git:
38+
url: https://github.com/ceph/ceph-build
39+
branches:
40+
- ${{CEPH_BUILD_BRANCH}}
41+
shallow-clone: true
42+
submodule:
43+
disable: true
44+
wipe-workspace: true
45+
script-path: auto-ptl-batch/build/Jenkinsfile
46+
lightweight-checkout: true
47+
do-not-fetch-tags: true
48+
49+
triggers:
50+
- timed: 'H 0 * * *'
51+
52+
parameters:
53+
- string:
54+
name: CEPH_BUILD_BRANCH
55+
description: "ceph-build branch to use for this job's Jenkinsfile"
56+
default: "main"
57+
- string:
58+
name: CEPH_BASE_BRANCH
59+
description: |
60+
If set, only process PRs targeting this branch (e.g. main, tentacle).
61+
Leave empty to process PRs across all branches.
62+
default: ""
63+
- string:
64+
name: REQUIRED_LABELS
65+
description: |
66+
Comma-separated labels that every PR must carry to be eligible.
67+
A component label is always required in addition to these.
68+
default: "needs-qa"
69+
- string:
70+
name: EXCLUDE_LABELS
71+
description: |
72+
Comma-separated labels that exclude a PR, even if CI is green.
73+
default: "needs-rebase,ready-to-merge,passed-qa"
74+
- string:
75+
name: COMPONENT_SUITE_MAP
76+
description: |
77+
JSON object mapping each component label value to the
78+
teuthology suite to run for that component's batches.
79+
Add or override entries here without changing the Jenkinsfile.
80+
# Double braces so Jenkins Job Builder does not treat JSON as str.format().
81+
default: >-
82+
{{"bluestore":"rados","build/ops":"smoke","cephfs":"fs",
83+
"common":"smoke","core":"rados","crimson":"crimson",
84+
"dashboard":"dashboard","mds":"fs","mgr":"mgr","mon":"rados",
85+
"msgr":"rados","osd":"rados","pybind":"smoke","rados":"rados",
86+
"rbd":"rbd","rgw":"rgw","tools":"smoke"}}
87+
- string:
88+
name: UPDATED_WITHIN_DAYS
89+
description: |
90+
Only consider PRs that have been updated within this many days.
91+
Keeps the scan fast and ignores stale PRs unlikely to need QA.
92+
default: "7"
93+
- string:
94+
name: CONFLICT_PATH_DEPTH
95+
description: |
96+
Directory depth used to detect file-path conflicts between PRs.
97+
3 -> src/rgw/multisite/ (recommended: catches sub-component conflicts)
98+
2 -> src/rgw/ (component-level only)
99+
0 -> exact file match (fewest splits)
100+
default: "3"
101+
- string:
102+
name: MAX_PRS_PER_BATCH
103+
description: "Hard cap on the number of PRs in a single sub-batch."
104+
default: "5"
105+
- string:
106+
name: MAX_PUSHES
107+
description: |
108+
Maximum number of branches to push to ceph-ci in a single run.
109+
Set to 1 to test only the first eligible batch. 0 = no limit.
110+
default: "0"
111+
- string:
112+
name: BUILD_DISTROS
113+
description: |
114+
Value for DISTROS trailer on pushed batch branches (consumed by ceph-trigger-build).
115+
default: "jammy centos9 rocky10"
116+
- string:
117+
name: BUILD_ARCHS
118+
description: |
119+
Value for ARCHS trailer on pushed batch branches.
120+
default: "x86_64"
121+
- string:
122+
name: BUILD_FLAVORS
123+
description: |
124+
Value for FLAVORS trailer on pushed batch branches.
125+
default: "default"
126+
- bool:
127+
name: BUILD_CI_CONTAINER
128+
description: |
129+
Value for CI-CONTAINER trailer on pushed batch branches.
130+
default: false
131+
- string:
132+
name: TEUTHOLOGY_TRIGGER_DELAY_SECONDS
133+
description: |
134+
Jenkins quietPeriod for triggering teuthology-runner after push.
135+
Use 0 for immediate trigger.
136+
default: "5400"
137+
- bool:
138+
name: DRY_RUN
139+
description: |
140+
If true, run all discovery, CI checks, and merges
141+
but skip the final push to ceph-ci.
142+
default: false
143+
- bool:
144+
name: SKIP_STATUS_POST
145+
description: |
146+
If true, push branches to ceph-ci as normal but do NOT post
147+
auto-ptl-batch commit status on PRs (pending, success, or failure)
148+
and do NOT wait for teuthology or post final results.
149+
Useful for testing the push path without touching existing PRs.
150+
default: false
151+
wrappers:
152+
- inject-passwords:
153+
global: true
154+
mask-password-params: true
155+
- ssh-agent-credentials:
156+
user: 'jenkins-build'
157+
- credentials-binding:
158+
- username-password-separated:
159+
credential-id: github-readonly-token
160+
username: GITHUB_USER
161+
password: GITHUB_PASS

0 commit comments

Comments
 (0)