Skip to content

Commit 9630877

Browse files
msyycCopilot
andauthored
[sdk generation pipeline] Auto-trim oversized CHANGELOG.md during SDK generation (#47809)
* Auto-trim oversized CHANGELOG.md during SDK generation CHANGELOG.md is embedded into the PyPI long_description, so it must not grow unbounded. Add trim_changelog_if_needed to sdk_changelog: when CHANGELOG.md exceeds 128 KB, keep the newest half of version entries and append a PyPI pointer note for older history. Called automatically after the changelog edit in main(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Preserve trim note on the <4-entries no-op path Apply the <4 version-entry early return before stripping the trim note so an existing PyPI-pointer note is not silently lost when modify_file writes content back on the no-op path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Trim CHANGELOG down under the size limit instead of cutting a fixed half The half-cut strategy could still leave the file over the limit (e.g. azure-mgmt-sql stayed at ~190 KB). Keep the newest version entries that fit under size_limit and remove all older entries completely, so the trimmed file is always under the limit. Add a regression test using the real azure-mgmt-sql 4.0.0 CHANGELOG fixture. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add expected trimmed azure-mgmt-sql changelog fixture Check in the trimmed output of the azure-mgmt-sql 4.0.0 CHANGELOG so the result is easy to review, and assert the trim produces it exactly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Raise CHANGELOG trim limit from 128 KB to 192 KB Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add azure-mgmt-network 31.0.0 changelog fixture and share trim assertions Add a second real-world fixture (azure-mgmt-network 31.0.0) with its expected trimmed output, and refactor the fixture check into a shared helper. Assert the trimmed size on normalized (LF) bytes so it matches the pipeline regardless of local newline translation. Normalize fixtures to LF. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Trim CHANGELOG down to half the limit for headroom (high-water/low-water) Keeping the file just under the limit meant it re-trimmed on almost every release (headroom was only ~1-3 KB). Trigger trimming at size_limit (192 KB) but cut down to trim_target (half, 96 KB) so there is headroom and trims are infrequent. Regenerate fixtures and update tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Keep at least 4 newest CHANGELOG entries when trimming Balance usefulness against headroom: trimming aims for the 96 KB target but always retains the 4 newest version entries unless they exceed the 192 KB hard limit. Regenerated sql/network fixtures now keep 4 entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add azure-mgmt-datafactory CHANGELOG trim test fixture 210 KB changelog trims to 4 newest entries (10.0.0b1..9.1.0, ~90 KB). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply black formatting to test_sdk_changelog.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Trigger changelog trim on normalized LF byte length Use the normalized (LF) UTF-8 byte length instead of stat().st_size so the trim trigger is consistent across platforms and matches the Linux pipeline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Lower CHANGELOG size limit to 128 KB Trigger at 128 KB (target 64 KB). Regenerated fixtures: sql keeps 3 newest entries (~127 KB, capped by the hard limit below the 4-entry floor), network and datafactory keep 4. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add containerservice and cosmosdb CHANGELOG trim test fixtures Both have many small entries, so the 64 KB target keeps well above the 4-entry floor: containerservice keeps 15 newest (~60 KB), cosmosdb keeps 18 (~63 KB). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 6e44a95 commit 9630877

12 files changed

Lines changed: 16960 additions & 0 deletions

eng/tools/azure-sdk-tools/packaging_tools/sdk_changelog.py

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
import multiprocessing
66
import os
77
from pathlib import Path
8+
import re
89
import sys
910
import time
1011

@@ -39,6 +40,119 @@ def is_arm_sdk(package_name: str) -> bool:
3940
return package_name.startswith("azure-mgmt-")
4041

4142

43+
# CHANGELOG.md is embedded into the package long_description (see setup.py template), so an
44+
# unbounded changelog bloats the PyPI metadata. Trimming uses a high-water/low-water pattern:
45+
# trimming is *triggered* when the file exceeds CHANGELOG_SIZE_LIMIT_BYTES, but when it runs the
46+
# file is cut down toward CHANGELOG_TRIM_TARGET_BYTES (half the limit). Trimming to a lower
47+
# target leaves headroom so the file does not immediately exceed the limit again on the next
48+
# release, avoiding a churny re-trim of the CHANGELOG on almost every generation. At least
49+
# CHANGELOG_MIN_KEEP_ENTRIES newest entries are always retained for usefulness (unless even that
50+
# many entries would exceed the hard size limit, in which case as many as fit are kept).
51+
CHANGELOG_SIZE_LIMIT_BYTES = 128 * 1024
52+
CHANGELOG_TRIM_TARGET_BYTES = CHANGELOG_SIZE_LIMIT_BYTES // 2
53+
CHANGELOG_MIN_KEEP_ENTRIES = 4
54+
_TRIM_NOTE_PREFIX = "> Changelog entries prior to"
55+
_VERSION_HEADER_RE = re.compile(r"^##\s+\d+\.\d+")
56+
57+
58+
def trim_changelog_if_needed(
59+
package_path: Path,
60+
size_limit: int = CHANGELOG_SIZE_LIMIT_BYTES,
61+
trim_target: Optional[int] = None,
62+
) -> bool:
63+
"""Drop the oldest CHANGELOG.md entries when the file grows too large.
64+
65+
The CHANGELOG is concatenated into the package ``long_description`` uploaded to PyPI, so it
66+
must not grow without bound. Trimming is triggered when ``CHANGELOG.md`` exceeds
67+
``size_limit`` bytes; when it runs, the file is cut down toward ``trim_target`` bytes
68+
(defaults to half of ``size_limit``) by keeping the ``# Release History`` header plus the
69+
newest version entries, removing older entries completely, and appending a note pointing to
70+
PyPI for the full history. Cutting toward the lower target (rather than just under the limit)
71+
leaves headroom so the file does not immediately exceed the limit again on the next release.
72+
73+
At least ``CHANGELOG_MIN_KEEP_ENTRIES`` newest entries are always kept for usefulness, even if
74+
that exceeds ``trim_target`` -- unless keeping that many would exceed the hard ``size_limit``,
75+
in which case only as many newest entries as fit under ``size_limit`` are kept (at least one).
76+
77+
Returns True if the file was trimmed, False otherwise.
78+
"""
79+
changelog_path = package_path / "CHANGELOG.md"
80+
if not changelog_path.exists():
81+
return False
82+
# Use the normalized (LF) UTF-8 byte length rather than stat().st_size: on Windows the file
83+
# is stored with CRLF line endings, so st_size would over-count relative to the LF content the
84+
# pipeline (Linux) actually ships, causing inconsistent trigger behavior across platforms.
85+
if len(changelog_path.read_text(encoding="utf-8").encode("utf-8")) <= size_limit:
86+
return False
87+
88+
if trim_target is None:
89+
trim_target = size_limit // 2
90+
91+
package_name = package_path.name
92+
trimmed = False
93+
94+
def byte_len(lines: list[str]) -> int:
95+
return sum(len(line.encode("utf-8")) for line in lines)
96+
97+
def trim_proc(content: list[str]):
98+
nonlocal trimmed
99+
version_indices = [i for i, line in enumerate(content) if _VERSION_HEADER_RE.match(line)]
100+
# Nothing to trim if there is at most one entry. Return before mutating content so an
101+
# existing trim note is preserved on this no-op path (modify_file always writes content
102+
# back). The note is appended after all version headers and never matches the version
103+
# header regex, so its presence does not affect version_indices.
104+
if len(version_indices) < 2:
105+
return
106+
107+
# Remove any previous trim note so repeated runs don't accumulate duplicates. It lives
108+
# after all version headers, so version_indices computed above stay valid.
109+
content[:] = [line for line in content if not line.startswith(_TRIM_NOTE_PREFIX)]
110+
111+
# Boundaries of each version section; the header (before the first entry) is always kept.
112+
n = len(version_indices)
113+
bounds = version_indices + [len(content)]
114+
header_bytes = byte_len(content[: version_indices[0]])
115+
seg_bytes = [byte_len(content[bounds[j] : bounds[j + 1]]) for j in range(n)]
116+
# Reserve room for the note line so the trimmed file stays under the target/limit.
117+
note_reserve = 256
118+
119+
# Keep the newest entries whose cumulative size fits under the target (always keep >= 1).
120+
keep_count = 1
121+
total = header_bytes + seg_bytes[0]
122+
for j in range(1, n):
123+
if total + seg_bytes[j] + note_reserve > trim_target:
124+
break
125+
total += seg_bytes[j]
126+
keep_count = j + 1
127+
128+
# Always keep at least CHANGELOG_MIN_KEEP_ENTRIES for usefulness, even past the target...
129+
keep_count = min(n, max(keep_count, CHANGELOG_MIN_KEEP_ENTRIES))
130+
# ...but never exceed the hard size limit: drop the oldest kept entries until it fits.
131+
while keep_count > 1 and header_bytes + sum(seg_bytes[:keep_count]) + note_reserve > size_limit:
132+
keep_count -= 1
133+
134+
# Everything already fits (should not happen once the file is over the limit, but guard).
135+
if keep_count >= n:
136+
return
137+
138+
oldest_kept = content[version_indices[keep_count - 1]].split()[1]
139+
note = (
140+
f"{_TRIM_NOTE_PREFIX} {oldest_kept} were removed to reduce file size. "
141+
f"See https://pypi.org/project/{package_name}/{oldest_kept}/ for the older history.\n"
142+
)
143+
del content[version_indices[keep_count] :]
144+
# Ensure a blank line separates the last kept entry from the note.
145+
if content and content[-1].strip():
146+
content.append("\n")
147+
content.append(note)
148+
trimmed = True
149+
150+
modify_file(str(changelog_path), trim_proc)
151+
if trimmed:
152+
_LOGGER.info(f"Trimmed CHANGELOG.md for {package_name} down to under {trim_target} bytes.")
153+
return trimmed
154+
155+
42156
def execute_func_with_timeout(func, timeout: int = 900) -> Any:
43157
"""Execute function with timeout.
44158
@@ -202,6 +316,9 @@ def edit_changelog_proc(content: list[str]):
202316

203317
modify_file(str(package_path / "CHANGELOG.md"), edit_changelog_proc)
204318

319+
# Keep CHANGELOG.md from growing unbounded, since it is embedded into the PyPI long_description.
320+
trim_changelog_if_needed(package_path)
321+
205322
except Exception as e:
206323
log_failed_message(f"Fail to generate changelog for {package_name}: {str(e)}", enable_log_error)
207324
else:

0 commit comments

Comments
 (0)