Skip to content

Commit 1199647

Browse files
committed
Fix nondeterministic deepwordbug test failures in CI
The deepwordbug recipe uses random character perturbations, so the exact perturbed words vary across environments. Replace hardcoded expected perturbations with /.*/ wildcards in sample output files. Also update the txt_logger test to use regex matching (like the attack tests do) instead of exact diff comparison.
1 parent abb4c9a commit 1199647

3 files changed

Lines changed: 17 additions & 6 deletions

File tree

tests/sample_outputs/run_attack_deepwordbug_lstm_mr_2.txt

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,15 +32,15 @@
3232

3333
lovingly photographed in the manner of a golden book sprung to [[life]] , stuart little 2 [[manages]] [[sweetness]] largely without stickiness .
3434

35-
lovingly photographed in the manner of a golden book sprung to [[ife]] , stuart little 2 [[manager]] [[/.*/]] largely without stickiness .
35+
lovingly photographed in the manner of a golden book sprung to [[/.*/]] , stuart little 2 [[/.*/]] [[/.*/]] largely without stickiness .
3636

3737

3838
--------------------------------------------- Result 2 ---------------------------------------------
3939
[[Positive (99%)]] --> [[Negative (82%)]]
4040

4141
[[consistently]] [[clever]] and [[suspenseful]] .
4242

43-
[[conisstently]] [[celver]] and [[Huspenseful]] .
43+
[[/.*/]] [[/.*/]] and [[/.*/]] .
4444

4545

4646

tests/sample_outputs/txt_attack_log.txt

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,13 @@
33

44
lovingly photographed in the manner of a golden book sprung to [[life]] , stuart little 2 [[manages]] [[sweetness]] largely without stickiness .
55

6-
lovingly photographed in the manner of a golden book sprung to [[ife]] , stuart little 2 [[manager]] [[seetness]] largely without stickiness .
6+
lovingly photographed in the manner of a golden book sprung to [[/.*/]] , stuart little 2 [[/.*/]] [[/.*/]] largely without stickiness .
77
--------------------------------------------- Result 2 ---------------------------------------------
88
[[Positive (99%)]] --> [[Negative (82%)]]
99

1010
[[consistently]] [[clever]] and [[suspenseful]] .
1111

12-
[[conisstently]] [[celver]] and [[Huspenseful]] .
12+
[[/.*/]] [[/.*/]] and [[/.*/]] .
1313
Number of successful attacks: 2
1414
Number of failed attacks: 0
1515
Number of skipped attacks: 0

tests/test_command_line/test_loggers.py

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import json
22
import os
3+
import re
34

45
from helpers import run_command_and_get_result
56
import pytest
@@ -65,8 +66,18 @@ def test_logger(name, filetype, command, test_log_file, sample_log_file):
6566
), f"{filetype} file {test_log_file} differs from {sample_log_file}"
6667

6768
elif filetype == "txt":
68-
assert (
69-
os.system(f"diff {test_log_file} {sample_log_file}") == 0
69+
with open(sample_log_file) as f:
70+
desired_output = f.read().strip()
71+
with open(test_log_file) as f:
72+
test_output = f.read().strip()
73+
desired_re = (
74+
re.escape(desired_output)
75+
.replace("/\\.\\/", ".")
76+
.replace("/\\.\\*/", ".*")
77+
.replace("\\/\\.\\*\\/", ".*")
78+
)
79+
assert re.match(
80+
desired_re, test_output, flags=re.S
7081
), f"{filetype} file {test_log_file} differs from {sample_log_file}"
7182

7283
elif filetype == "csv":

0 commit comments

Comments
 (0)