Keep '=' literal in URL paths so they match patterns (fixes #51)

[tienvantranspk-tech]

Fixes #51.

Problem

_quote_path() (used to normalize the URL being checked) percent-encodes =, because its quote() call uses safe="/%". But _quote_pattern() (used to normalize Allow/Disallow patterns) keeps = literal via safe="/*%=".

Because of this asymmetry, a pattern containing = can never match a URL whose path contains =: the pattern keeps page=, while the URL becomes page%3D.

Example from the issue:

User-agent: *
Allow:    /*/filter/page=*/$
Disallow: /

can_fetch("https://example.com/1/filter/page=5/", "mozilla") returned False (the URL had been normalized to /1/filter/page%3D5/, which the .../page=.../ pattern could not match), when it should be True.

Fix

Add = to the safe set in _quote_path() so URL paths keep = literal, matching how patterns are already normalized. = is a valid path character (an RFC 3986 sub-delimiter) and does not need percent-encoding.

Tests

Added test_equals_sign_in_path, covering the wildcard case from the issue, the trailing-$ boundary, and a plain (non-wildcard) = path. Full suite passes locally on Python 3.14 (4351 tests); ruff check / ruff format are clean.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Fixes robots.txt URL matching involving = in paths by preventing = from being percent-encoded during path quoting, and adds regression tests for the reported mismatch.

Changes:

• Keep = unescaped in _quote_path so URL paths match patterns that treat = as a literal character.
• Add parametrized regression tests covering = within wildcard patterns and in non-wildcard rules.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
tests/test_on_google_spec.py	Adds regression coverage for = handling in paths/patterns (issue #51).
src/protego/_utils.py	Adjusts quoting rules to preserve = in the encoded path for matching parity with patterns.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.97%. Comparing base (81f1b35) to head (a508d1c).

Additional details and impacted files

@@           Coverage Diff           @@
##           master      #77   +/-   ##
=======================================
  Coverage   97.97%   97.97%           
=======================================
  Files           5        5           
  Lines         345      345           
  Branches       69       69           
=======================================
  Hits          338      338           
  Misses          4        4           
  Partials        3        3           

Files with missing lines	Coverage Δ
src/protego/_utils.py	100.00% <100.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[tienvantranspk-tech]

Thanks for the review! I pushed a commit addressing the test feedback:

• Added a regression case with a pre-encoded %3D in the path, confirming the _unquote + quote normalisation keeps matching correct.
• Added a non-wildcard prefix case (/path=1 also matching /path=12) and reworded the comment so it no longer reads as exact-match.

On extracting a shared constant for the safe character set used by _quote_path / _quote_pattern: I kept this change minimal and focused on the fix for now, but I'm happy to factor that out if you'd prefer.

[AdrianAtZyte]

I wonder… Is the right fix not encoding = in both cases, or is the right fix encoding it in both cases?

[tienvantranspk-tech]

Both directions give consistent matching; I went with keeping = literal because:

• It's the smaller change: _quote_pattern already keeps = literal (safe="/*%="), so only _quote_path needed to be brought in line.
• = is a valid path character (an RFC 3986 sub-delimiter), so the literal form is the canonical one.
• It normalises all forms consistently: a literal = and a percent-encoded %3D, in both patterns and URLs, all end up as literal = (since _unquote decodes %3D to = and the quote step keeps it), so they all match each other.

Encoding = to %3D in both places would also be consistent, but it requires changing _quote_pattern too and produces the less-canonical %3D form. Happy to switch to that if you'd prefer it for consistency with how other sub-delimiters (;, ,, ...) are handled.

[AdrianAtZyte]

What do you make of the following analysis?

LLM analysis

The key language from RFC 9309 §2.2.2 is (emphasis mine):

"Octets in the URI and robots.txt paths outside the range of the ASCII coded character set, and those in the reserved range defined by [RFC3986], MUST be percent-encoded as defined by [RFC3986] prior to comparison."

"If a percent-encoded ASCII octet is encountered in the URI, it MUST be unencoded prior to comparison, unless it is a reserved character in the URI as defined by [RFC3986]…"

This is significant for the PR's approach. = is an RFC 3986 sub-delimiter — it's in the reserved set. Under RFC 9309:

• A literal = in a path must be encoded to %3D before comparison
• A %3D in a path must NOT be decoded (reserved character exception)

So RFC 9309 says both pattern and URL should end up with %3D, not =. The PR's fix goes the other direction — it keeps = literal in both — which technically contradicts RFC 9309 even though the matching outcomes are identical in practice.

The RFC 9309-compliant fix would actually be to remove = from _quote_pattern's safe set (encoding it as %3D), rather than adding it to _quote_path's safe set (keeping it literal). That would align with how every other reserved character is already handled by both functions.

In summary:

	Google spec	RFC 9309	PR's fix
Explicitly addresses = in paths?	No	Yes	—
How should = be normalized?	—	Encode → %3D	Keep literal
Result of matching page=* vs page=5?	—	Correct ✓	Correct ✓

The PR achieves consistency and fixes the real bug, but it does so in a way that's at odds with RFC 9309 and inconsistent with how all other reserved characters (like !, $, &, etc.) are handled — they all get encoded. The spec-correct approach would encode = in both functions, not keep it literal in both. The PR is LLM-generated and picked the path-of-least-resistance (add one char to one safe set) without considering the spec or the broader normalization design.