Skip to content

fix: make -f w compose with other flags and bind to the whole pattern - #348

Open
vlasky wants to merge 1 commit into
chmln:masterfrom
vlasky:fix/word-boundary-flag-order
Open

fix: make -f w compose with other flags and bind to the whole pattern#348
vlasky wants to merge 1 commit into
chmln:masterfrom
vlasky:fix/word-boundary-flag-order

Conversation

@vlasky

@vlasky vlasky commented Jul 30, 2026

Copy link
Copy Markdown

-f w has three independent bugs. All three are silent: wrong result, exit status 0, no diagnostic. There is no existing issue for them, so the report is inline below.

Two mistakes are packed into the one 'w' arm of the flag loop: it replaces the whole RegexBuilder instead of setting an option on it, and it splices \b into the pattern with no grouping.

let mut regex = regex::bytes::RegexBuilder::new(&look_for);
regex.multi_line(true);                      // <-- applied BEFORE the loop
// ...
'w' => {
    // constructs a NEW builder; everything set on the old one is gone
    regex = regex::bytes::RegexBuilder::new(&format!("\\b{}\\b", look_for));
},

1. Flags to the left of w are discarded

Assigning a fresh builder throws away everything set on the previous one, so -f iw and -f wi differ, even though --help documents flags as freely combinable ("May be combined (like -f mc)") and says nothing about order:

$ echo "FOO bar" | sd -f iw 'foo' 'X'
FOO bar          # 'i' discarded, no match, exit 0
$ echo "FOO bar" | sd -f wi 'foo' 'X'
X bar            # works

Same for s, and for an i set before a c.

2. w disables multi-line matching, unrecoverably

multi_line(true) is applied before the loop, so w discards that too, and nothing puts it back: the 'm' arm is a no-op because multi-line is meant to be the default. There is no way to get word-boundary matching and multi-line anchoring at once. On printf 'aa\nbb\n' replacing ^\w+ with M:

Invocation Result
(none) / -f m M\nM
-f e M\nbb (as documented)
-f w / -f wm / -f mw M\nbb (m does not restore it)

Since #328 made line-by-line the default this needs -A to observe, because per-line anchoring holds either way when input is fed one line at a time. Before #328 it was reachable with no flag at all, and it has been present since 87491b4 (2020) introduced the pre-loop multi_line(true).

3. \b binds to the first and last alternative, not the whole pattern

Independent of the above, and the oldest: it dates to 80d92bd (2019), where w was introduced. | binds looser than concatenation, so foo|bar becomes \bfoo|bar\b, which parses as (\bfoo)|(bar\b). Only the first alternative gets a leading boundary and only the last gets a trailing one; with three or more, the middle ones get neither.

$ echo "foox bar" | sd -f w 'foo|bar' 'X'
Xx X             # replaced the "foo" inside "foox"

This is the opposite of the documented "match full words only": the flag broke the guarantee it advertises, and made a replacement that plain sd 'foo|bar' 'X' would also have made. It is also worse than the other two: they silently do nothing, this silently does something unwanted.

Fix

Resolve w into the pattern before the builder is constructed, so the builder is built exactly once and no state can be discarded, and wrap in a non-capturing group so \b binds to the whole pattern:

let word_boundary = flags.as_deref().is_some_and(|f| f.contains('w'));
let pattern = if word_boundary {
    format!("\\b(?:{look_for})\\b")
} else {
    look_for
};

(?:...) rather than (...) matters: a capturing group would renumber the user's own groups and break sd -f w '(f)(o)o' '$2$1'. There is a test for that.

Flag order is now irrelevant, which is what the help text already promises.

Tests

w had one test since 2019 (look_for: "abc", flags: Some("w")). That case cannot catch any of the three, structurally: abc has no top-level |, w is never combined with another flag, and there is no anchor or newline. This adds six example tests and two proptests:

  • -f iw / -f wi both case-insensitive; -f sw / -f ws both dot-matches-newline
  • -f w / -f wm / -f mw keep ^ and $ anchoring
  • foo|bar anchors all alternatives (3 inputs)
  • -f w '(f)(o)o' with $2$1 does not shift capture groups
  • -f w alone still word-bounded and case-sensitive (regression guard)
  • proptest: w's position in the flag string is irrelevant (the rest of the string keeps its order, so genuine last-one-wins conflicts resolve identically and only w moves)
  • proptest: -f <S>w on P equals -f <S> on \b(?:P)\b, i.e. w only adds boundaries

Of the six example tests, three fail against unfixed mod.rs (one per bug) and three pass either way as regression guards. Both proptests also fail against unfixed mod.rs. Full suite passes; cargo clippy --all-targets --all-features and cargo fmt --check -p sd are clean.

Notes

The `w` arm of the flag loop replaced the whole RegexBuilder instead of
setting an option on it, and wrapped the pattern as `\bP\b`. Three
independent bugs follow, all silent: wrong result, exit 0, no diagnostic.

1. Flags to the left of `w` are discarded, because assigning a fresh
   builder throws away everything set on the old one. `-f iw` and `-f wi`
   therefore differ, even though --help documents flags as freely
   combinable and says nothing about order:

       $ echo "FOO bar" | sd -f iw 'foo' 'X'
       FOO bar          # 'i' discarded, no match, exit 0
       $ echo "FOO bar" | sd -f wi 'foo' 'X'
       X bar            # works

   The same applies to `s`, and to an `i` set before a `c`.

2. multi_line(true) is applied before the loop, so `w` discards that too,
   and nothing puts it back: the 'm' arm is a no-op because multi-line is
   meant to be the default. That leaves no way to get word-boundary
   matching and multi-line anchoring at once. `^\w+` on "aa\nbb" gives
   "M\nM" normally but "M\nbb" under `-f w`, `-f wm` and `-f mw`. Since
   4a7b216 made line-by-line the default this needs `-A` to observe, as
   per-line anchoring holds either way when fed one line at a time.

3. `\b` binds to the first and last alternative rather than the whole
   pattern, so `-f w` on `foo|bar` builds `\bfoo|bar\b`, which is
   `(\bfoo)|(bar\b)`. That matches the "foo" in "foox" and the "bar" in
   "xbar" -- the exact opposite of "match full words only". This one dates
   to 80d92bd (2019), where `w` was introduced.

Resolve `w` into the pattern before the builder is constructed, so the
builder is built exactly once and no state can be discarded, and wrap in a
non-capturing group so `\b` binds to the whole pattern. The group must be
non-capturing, or the user's own `$1` would shift to `$2`.

Flag order is now irrelevant, as the help text already promises.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant