fix: make -f w compose with other flags and bind to the whole pattern - #348
Open
vlasky wants to merge 1 commit into
Open
fix: make -f w compose with other flags and bind to the whole pattern#348vlasky wants to merge 1 commit into
-f w compose with other flags and bind to the whole pattern#348vlasky wants to merge 1 commit into
Conversation
The `w` arm of the flag loop replaced the whole RegexBuilder instead of
setting an option on it, and wrapped the pattern as `\bP\b`. Three
independent bugs follow, all silent: wrong result, exit 0, no diagnostic.
1. Flags to the left of `w` are discarded, because assigning a fresh
builder throws away everything set on the old one. `-f iw` and `-f wi`
therefore differ, even though --help documents flags as freely
combinable and says nothing about order:
$ echo "FOO bar" | sd -f iw 'foo' 'X'
FOO bar # 'i' discarded, no match, exit 0
$ echo "FOO bar" | sd -f wi 'foo' 'X'
X bar # works
The same applies to `s`, and to an `i` set before a `c`.
2. multi_line(true) is applied before the loop, so `w` discards that too,
and nothing puts it back: the 'm' arm is a no-op because multi-line is
meant to be the default. That leaves no way to get word-boundary
matching and multi-line anchoring at once. `^\w+` on "aa\nbb" gives
"M\nM" normally but "M\nbb" under `-f w`, `-f wm` and `-f mw`. Since
4a7b216 made line-by-line the default this needs `-A` to observe, as
per-line anchoring holds either way when fed one line at a time.
3. `\b` binds to the first and last alternative rather than the whole
pattern, so `-f w` on `foo|bar` builds `\bfoo|bar\b`, which is
`(\bfoo)|(bar\b)`. That matches the "foo" in "foox" and the "bar" in
"xbar" -- the exact opposite of "match full words only". This one dates
to 80d92bd (2019), where `w` was introduced.
Resolve `w` into the pattern before the builder is constructed, so the
builder is built exactly once and no state can be discarded, and wrap in a
non-capturing group so `\b` binds to the whole pattern. The group must be
non-capturing, or the user's own `$1` would shift to `$2`.
Flag order is now irrelevant, as the help text already promises.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
-f whas three independent bugs. All three are silent: wrong result, exit status 0, no diagnostic. There is no existing issue for them, so the report is inline below.Two mistakes are packed into the one
'w'arm of the flag loop: it replaces the wholeRegexBuilderinstead of setting an option on it, and it splices\binto the pattern with no grouping.1. Flags to the left of
ware discardedAssigning a fresh builder throws away everything set on the previous one, so
-f iwand-f widiffer, even though--helpdocuments flags as freely combinable ("May be combined (like-f mc)") and says nothing about order:Same for
s, and for aniset before ac.2.
wdisables multi-line matching, unrecoverablymulti_line(true)is applied before the loop, sowdiscards that too, and nothing puts it back: the'm'arm is a no-op because multi-line is meant to be the default. There is no way to get word-boundary matching and multi-line anchoring at once. Onprintf 'aa\nbb\n'replacing^\w+withM:-f mM\nM-f eM\nbb(as documented)-f w/-f wm/-f mwM\nbb(mdoes not restore it)Since #328 made line-by-line the default this needs
-Ato observe, because per-line anchoring holds either way when input is fed one line at a time. Before #328 it was reachable with no flag at all, and it has been present since 87491b4 (2020) introduced the pre-loopmulti_line(true).3.
\bbinds to the first and last alternative, not the whole patternIndependent of the above, and the oldest: it dates to 80d92bd (2019), where
wwas introduced.|binds looser than concatenation, sofoo|barbecomes\bfoo|bar\b, which parses as(\bfoo)|(bar\b). Only the first alternative gets a leading boundary and only the last gets a trailing one; with three or more, the middle ones get neither.This is the opposite of the documented "match full words only": the flag broke the guarantee it advertises, and made a replacement that plain
sd 'foo|bar' 'X'would also have made. It is also worse than the other two: they silently do nothing, this silently does something unwanted.Fix
Resolve
winto the pattern before the builder is constructed, so the builder is built exactly once and no state can be discarded, and wrap in a non-capturing group so\bbinds to the whole pattern:(?:...)rather than(...)matters: a capturing group would renumber the user's own groups and breaksd -f w '(f)(o)o' '$2$1'. There is a test for that.Flag order is now irrelevant, which is what the help text already promises.
Tests
whad one test since 2019 (look_for: "abc",flags: Some("w")). That case cannot catch any of the three, structurally:abchas no top-level|,wis never combined with another flag, and there is no anchor or newline. This adds six example tests and two proptests:-f iw/-f wiboth case-insensitive;-f sw/-f wsboth dot-matches-newline-f w/-f wm/-f mwkeep^and$anchoringfoo|baranchors all alternatives (3 inputs)-f w '(f)(o)o'with$2$1does not shift capture groups-f walone still word-bounded and case-sensitive (regression guard)w's position in the flag string is irrelevant (the rest of the string keeps its order, so genuine last-one-wins conflicts resolve identically and onlywmoves)-f <S>wonPequals-f <S>on\b(?:P)\b, i.e.wonly adds boundariesOf the six example tests, three fail against unfixed
mod.rs(one per bug) and three pass either way as regression guards. Both proptests also fail against unfixedmod.rs. Full suite passes;cargo clippy --all-targets --all-featuresandcargo fmt --check -p sdare clean.Notes
-f wdisabling multi-line, but that was undocumented and almost certainly unintended.master, independent of feat: reject replacement references to undefined capture groups #346 and feat: add --must-match to exit non-zero when nothing matched #347. Will need a trivial rebase if feat: reject replacement references to undefined capture groups #346 lands first, since it moves this code intobuild_regex.mcannot re-enable multi-line after ane;sdisables multi-line as an undocumented side effect; unknown flags (-f z) are silently ignored. Happy to open separate issues if useful.