Skip to content

Commit 65d0193

Browse files
LEANDERANTONYclaude
andcommitted
test(exporters): fix flaky DOCX byte-comparison tests — compare content not raw bytes
Two tests in test_exporters.py compared raw .docx bytes. A .docx is a ZIP archive and python-docx stamps the current mtime into every entry's local file header, so two renders a second apart differ byte-wise even when the document content is byte-identical. * test_export_docx_bytes_unknown_theme_falls_back_to_classic_ats asserted `fallback_bytes == classic_bytes` — intermittently FAILED across a DOS-timestamp 2-second boundary (~1-in-5; surfaced as a byte diff at index ~10, the ZIP last-mod-time field). * test_export_docx_bytes_themes_produce_different_outputs asserted `classic_bytes != neutral_bytes` — never failed, but was effectively vacuous: timestamps alone guarantee the bytes differ, so it would have passed even if the theme/palette switch were completely broken. Both rewritten to compare EXTRACTED document content via the existing helpers instead of raw bytes: - fallback test: _docx_paragraph_pairs + _docx_run_color_hexes + _docx_run_font_names must all be EQUAL (unknown theme renders identically to classic_ats). - themes test: _docx_run_color_hexes must DIFFER (classic_ats warm-brown vs professional_neutral black) — a real palette- switch regression guard. Deterministic now — confirmed stable 5/5 consecutive runs. 32/32 exporter tests green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent beb6f20 commit 65d0193

2 files changed

Lines changed: 44 additions & 18 deletions

File tree

docs/DEVLOG.md

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2799,8 +2799,20 @@ font-assertion tests (classic_ats / professional_neutral) inverted
27992799
to expect Arial-only. 32/32 exporter tests + renderer-fidelity
28002800
runner (OVERALL PASS) green.
28012801

2802-
Noted but NOT fixed (pre-existing, unrelated):
2803-
`test_export_docx_bytes_unknown_theme_falls_back_to_classic_ats`
2804-
compares raw timestamped `.docx` ZIP bytes — flaky across a
2805-
DOS-timestamp second boundary. Should compare extracted content,
2806-
not raw bytes; worth a separate fix.
2802+
Follow-up fix — flaky DOCX byte-comparison tests. Two tests in
2803+
`test_exporters.py` compared raw `.docx` bytes. A `.docx` is a ZIP
2804+
and python-docx stamps the current mtime into every entry header,
2805+
so two renders a second apart differ byte-wise even with identical
2806+
content:
2807+
* `test_export_docx_bytes_unknown_theme_falls_back_to_classic_ats`
2808+
asserted `fallback_bytes == classic_bytes` — intermittently
2809+
failed across a DOS-timestamp boundary (~1-in-5).
2810+
* `test_export_docx_bytes_themes_produce_different_outputs`
2811+
asserted `classic_bytes != neutral_bytes` — never failed, but
2812+
vacuous: timestamps alone make the bytes differ, so it would
2813+
pass even if the theme switch were entirely broken.
2814+
Both rewritten to compare EXTRACTED content via the existing
2815+
helpers — `_docx_paragraph_pairs` + `_docx_run_color_hexes` +
2816+
`_docx_run_font_names` (equal for the fallback test, color-set
2817+
differs for the themes test). Deterministic now; confirmed stable
2818+
5/5 consecutive runs. 32/32 exporter tests green.

tests/test_exporters.py

Lines changed: 27 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -866,34 +866,48 @@ def test_export_docx_bytes_professional_neutral_uses_black_palette():
866866

867867

868868
def test_export_docx_bytes_themes_produce_different_outputs():
869-
"""Sanity check: render the same artifact under both themes and
870-
verify the bytes differ. This is a coarse but cheap regression
871-
against accidentally short-circuiting the palette switch."""
869+
"""Sanity check: render the same artifact under two themes and
870+
verify the PALETTE actually differs — a regression guard against
871+
accidentally short-circuiting the theme switch.
872+
873+
Compares the extracted run-color set, NOT raw .docx bytes: a
874+
.docx is a ZIP and python-docx stamps the current mtime into
875+
every entry header, so two renders always have different bytes
876+
regardless of content — a raw-byte `!=` would pass even if the
877+
palette switch were broken. The color set is the real fingerprint."""
872878
artifact = _make_full_resume_artifact()
873879

874880
artifact.theme = "classic_ats"
875-
classic_bytes = export_docx_bytes(artifact)
881+
classic_doc = _parse_docx(export_docx_bytes(artifact))
876882

877883
artifact.theme = "professional_neutral"
878-
neutral_bytes = export_docx_bytes(artifact)
884+
neutral_doc = _parse_docx(export_docx_bytes(artifact))
879885

880-
assert classic_bytes != neutral_bytes
886+
# classic_ats ships warm-brown ink/accent; professional_neutral is
887+
# pure black/gray. Their run-color sets must differ.
888+
assert _docx_run_color_hexes(classic_doc) != _docx_run_color_hexes(neutral_doc)
881889

882890

883891
def test_export_docx_bytes_unknown_theme_falls_back_to_classic_ats():
884892
"""Unknown theme strings should resolve to classic_ats so the
885893
renderer never crashes on an unexpected artifact.theme value."""
886894
artifact = _make_full_resume_artifact()
887895
artifact.theme = "some_made_up_theme_we_have_not_built"
888-
889-
fallback_bytes = export_docx_bytes(artifact)
896+
fallback_doc = _parse_docx(export_docx_bytes(artifact))
890897

891898
artifact.theme = "classic_ats"
892-
classic_bytes = export_docx_bytes(artifact)
893-
894-
# The unknown-theme render should be identical to classic_ats —
895-
# same palette, same fonts, same content.
896-
assert fallback_bytes == classic_bytes
899+
classic_doc = _parse_docx(export_docx_bytes(artifact))
900+
901+
# The unknown-theme render must be EQUIVALENT to classic_ats —
902+
# same content/structure, same palette, same fonts. We compare
903+
# the extracted document, NOT raw .docx bytes: a .docx is a ZIP
904+
# and python-docx stamps the current mtime into every entry
905+
# header, so two renders a second apart differ byte-wise even
906+
# when the content is identical (this test used to be flaky for
907+
# exactly that reason). Compare what actually matters.
908+
assert _docx_paragraph_pairs(fallback_doc) == _docx_paragraph_pairs(classic_doc)
909+
assert _docx_run_color_hexes(fallback_doc) == _docx_run_color_hexes(classic_doc)
910+
assert _docx_run_font_names(fallback_doc) == _docx_run_font_names(classic_doc)
897911

898912

899913
def test_export_docx_bytes_cover_letter_respects_theme():

0 commit comments

Comments
 (0)