Skip to content

Commit b87a139

Browse files
simonelbazsimonelbaz
authored
Merge branch 'opensearch-project:main' into dlqpathstyle
2 parents c04ece3 + e70237e commit b87a139

960 files changed

Lines changed: 49682 additions & 5877 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/commands/generate-release-notes.md‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
Generate release notes for Data Prepper version $ARGUMENTS.
2+
3+
## Overview
4+
5+
You are generating user-facing release notes for the OpenSearch Data Prepper project.
6+
The milestone to use is: $ARGUMENTS
7+
8+
You will gather data from GitHub, categorize each item, reword it for clarity, and produce
9+
a release notes file following the project's established format.
10+
11+
## Step 1: Gather Data
12+
13+
Run these commands to fetch all closed issues and PRs for the milestone.
14+
Request the fields: number, url, title, labels, body.
15+
16+
```
17+
gh issue list --repo opensearch-project/data-prepper --search 'milestone:$ARGUMENTS' --state closed --json number,url,title,labels,body --limit 200
18+
```
19+
20+
```
21+
gh pr list --repo opensearch-project/data-prepper --search 'milestone:$ARGUMENTS' --state closed --json number,url,title,labels,body --limit 200
22+
```
23+
24+
Collect all results. Note which items are issues vs PRs.
25+
26+
## Step 2: Read the Label-to-Category Mapping
27+
28+
Read the file `release/script/release-notes/label-category-mapping.md` for the full rules
29+
on how labels map to categories, how to distinguish Features from Enhancements, what to
30+
exclude, and what to consolidate.
31+
32+
## Step 3: Read Example Release Notes for Style
33+
34+
Read 2-3 of the most recent release notes files in `release/release-notes/`. Use only
35+
files matching `data-prepper.release-notes-*.md` -- ignore `data-prepper.change-log-*.md`
36+
files, which are a different format. Study the tone, format, and level of detail.
37+
Key style observations:
38+
39+
- Each item is a single bullet point: `* Description ([#N](url))`
40+
- Descriptions are concise, active, and user-facing
41+
- They describe what changed from the user's perspective, not implementation details
42+
- Multiple related links can appear: `([#N1](url1), [#N2](url2))`
43+
- Security items use the format: `* CVE-XXXX-XXXXX - library version ([#N](pr-url))`
44+
- Lead with the CVE identifier(s), then the library name and the version that fixes it
45+
- Link to the PR that performed the upgrade, not the vulnerability issue
46+
- If multiple CVEs are fixed by the same upgrade PR, combine them on one line:
47+
`* CVE-XXXX-XXXXX, CVE-YYYY-YYYYY - library version ([#N](pr-url))`
48+
- Search for the fixing PR by looking up the CVE or library name in closed PRs
49+
- The date format is YYYY-MM-DD
50+
- Empty categories are kept in the output (the heading with no items below it)
51+
52+
## Step 4: Categorize Each Item
53+
54+
For each issue/PR:
55+
56+
1. **Check exclusion rules first.** Skip release prep PRs, changelog PRs, release notes PRs,
57+
individual dependency bumps, license header updates, and bot housekeeping PRs.
58+
59+
2. **Apply label-based categorization.** Use the mapping rules to assign a category.
60+
61+
3. **Confirm by reading the body.** If the labels suggest one category but the body clearly
62+
indicates another, override the label-based assignment. Pay special attention to:
63+
- Distinguishing Features (new capability) from Enhancements (improving existing)
64+
- Items labeled `enhancement` that are actually bug fixes
65+
- Items with no relevant labels -- categorize based on the body content
66+
67+
4. **Consolidate related items.** When an issue and its implementing PR(s) are both in the
68+
milestone, produce one line. Prefer linking to the issue. If a PR has no corresponding
69+
issue, link to the PR.
70+
71+
## Step 5: Reword Each Item
72+
73+
Rewrite each item to be:
74+
75+
- **Direct and concise**: "Support X" not "Add support for X", "Added support for X",
76+
or "Support for X was added". Lead with the verb, skip filler words.
77+
- **User-facing**: Describe the impact, not the implementation. "Support compressed files
78+
in the file source" not "Add GzipInputStream wrapper to FileSource class"
79+
- **Concise**: One sentence, no trailing periods
80+
- **Specific**: Name the plugin or component when relevant. "Support partition keys in the
81+
OTel metrics source with persistent buffers" not "Support partition keys"
82+
83+
Do NOT use the prefix tags like "feat:", "fix:", "refactor:" from commit messages.
84+
85+
## Step 6: Produce the Output
86+
87+
Write the release notes file to:
88+
`release/release-notes/data-prepper.release-notes-VERSION.md`
89+
90+
where VERSION is derived from the milestone (e.g. milestone `v2.15` produces version `2.15.0`
91+
if it's a minor release).
92+
93+
Use today's date in the header.
94+
95+
Format:
96+
97+
```
98+
## YYYY-MM-DD Version M.m.p
99+
100+
---
101+
102+
### Breaking Changes
103+
104+
* Item ([#N](url))
105+
106+
### Features
107+
108+
* Item ([#N](url))
109+
110+
### Enhancements
111+
112+
* Item ([#N](url))
113+
114+
### Bug Fixes
115+
116+
* Item ([#N](url))
117+
118+
### Security
119+
120+
* Item ([#N](url))
121+
122+
### Maintenance
123+
124+
* Item ([#N](url))
125+
126+
```
127+
128+
Keep all six category headings even if a category has no items.
129+
130+
## Step 7: Review
131+
132+
After writing the file, read it back and check:
133+
- No duplicate items
134+
- No excluded items leaked through
135+
- Items are in the correct categories
136+
- Wording is consistent in style
137+
- All links are correct
138+
139+
Present the final file to the user and note any items where categorization was uncertain
140+
so they can review those specifically.

‎.github/scripts/check-license-headers.py‎

Lines changed: 3 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -126,11 +126,9 @@ def main():
126126

127127
violation_text = '\n'.join(violations)
128128

129-
# Set output for GitHub Actions
130-
github_output = os.environ.get('GITHUB_OUTPUT')
131-
if github_output:
132-
with open(github_output, 'a') as f:
133-
f.write(f"violations<<EOF\n{violation_text}\nEOF\n")
129+
# Write violations file for the comment workflow artifact
130+
with open('violations.txt', 'w') as f:
131+
f.write(violation_text + '\n')
134132

135133
print("\nViolations:")
136134
for violation in violations:
@@ -139,11 +137,6 @@ def main():
139137
sys.exit(1)
140138
else:
141139
print("\n✅ All files have proper license headers!")
142-
# Set empty output for GitHub Actions
143-
github_output = os.environ.get('GITHUB_OUTPUT')
144-
if github_output:
145-
with open(github_output, 'a') as f:
146-
f.write("violations=\n")
147140

148141
if __name__ == "__main__":
149142
main()

‎.github/workflows/compatibility-data-prepper-api.yml‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
#
2+
# Copyright OpenSearch Contributors
3+
# SPDX-License-Identifier: Apache-2.0
4+
#
5+
# The OpenSearch Contributors require contributions made to
6+
# this file be licensed under the Apache-2.0 license or a
7+
# compatible open source license.
8+
#
9+
10+
name: Verify data-prepper-api compatibility
11+
12+
on:
13+
push:
14+
paths:
15+
- 'data-prepper-api/**'
16+
- '*gradle*'
17+
pull_request:
18+
paths:
19+
- 'data-prepper-api/**'
20+
- '*gradle*'
21+
22+
workflow_dispatch:
23+
24+
jobs:
25+
verify-api-compatibility:
26+
runs-on: ubuntu-latest
27+
28+
steps:
29+
- name: Set up JDK
30+
uses: actions/setup-java@v4
31+
with:
32+
java-version: 11
33+
distribution: temurin
34+
35+
- name: Checkout Data Prepper
36+
uses: actions/checkout@v6
37+
38+
- name: Validate API compatibility
39+
run: ./gradlew -p data-prepper-api validateCompatibility
40+
41+
- name: Output compatibility report
42+
if: failure()
43+
run: cat data-prepper-api/build/reports/library-compatibility/report.txt
44+
45+
- name: Upload Compatibility Report
46+
if: failure()
47+
uses: actions/upload-artifact@v7
48+
with:
49+
name: library-compatibility-report
50+
path: ${{ github.workspace }}/data-prepper-api/build/reports/library-compatibility/report.html

‎.github/workflows/gradle-build-src.yml‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
#
2+
# Copyright OpenSearch Contributors
3+
# SPDX-License-Identifier: Apache-2.0
4+
#
5+
# The OpenSearch Contributors require contributions made to
6+
# this file be licensed under the Apache-2.0 license or a
7+
# compatible open source license.
8+
#
9+
10+
name: Gradle - Check buildSrc
11+
12+
on:
13+
push:
14+
paths:
15+
- 'buildSrc/**'
16+
- '*gradle*'
17+
pull_request:
18+
paths:
19+
- 'buildSrc/**'
20+
- '*gradle*'
21+
22+
workflow_dispatch:
23+
24+
jobs:
25+
build:
26+
strategy:
27+
matrix:
28+
java: [11, 17, 21]
29+
fail-fast: false
30+
31+
runs-on: ubuntu-latest
32+
33+
steps:
34+
- name: Set up JDK ${{ matrix.java }}
35+
uses: actions/setup-java@v4
36+
with:
37+
java-version: ${{ matrix.java }}
38+
distribution: temurin
39+
40+
- name: Checkout Data Prepper
41+
uses: actions/checkout@v6
42+
- name: Check Gradle buildSrc
43+
run: ./gradlew -p buildSrc check
44+
- name: Upload Unit Test Results
45+
if: always()
46+
uses: actions/upload-artifact@v4
47+
with:
48+
name: data-prepper-test-results-java-${{ matrix.java }}
49+
path: '**/test-results/**/*.xml'
50+
51+
publish-test-results:
52+
name: "Publish Unit Tests Results"
53+
needs: build
54+
runs-on: ubuntu-latest
55+
if: always()
56+
57+
steps:
58+
- name: Download Artifacts
59+
uses: actions/download-artifact@v4
60+
with:
61+
path: test-results
62+
63+
- name: Publish Unit Test Results
64+
uses: EnricoMi/publish-unit-test-result-action@v1
65+
with:
66+
files: "test-results/**/*.xml"

‎.github/workflows/kafka-plugin-integration-tests.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,8 @@ jobs:
6666
-Dtests.kafka.bootstrap_servers=localhost:9092 \
6767
-Dtests.kafka.authconfig.username=admin -Dtests.kafka.authconfig.password=admin \
6868
-Dtests.kafka.kms_key=alias/DataPrepperTesting \
69-
--tests '*kafka.buffer*' --tests KafkaSourceJsonTypeIT --tests KafkaBufferOTelIT
69+
--tests '*kafka.buffer*' --tests KafkaSourceJsonTypeIT --tests KafkaBufferOTelIT \
70+
--tests KafkaSinkJsonTypeIT --tests KafkaSinkPlainTextTypeIT
7071
7172
- name: Upload Unit Test Results
7273
if: always()

0 commit comments

Comments
 (0)