Release v5.16.1 (#114)

* Reduce GTM false positives in malware sniffer

* Update bundled scanner wordlists

* Update bundled scanner wordlists

* Add remote HTTP wordlist support

* Improve path-specific transport failure

* Add proxy-list rotation policy control

* Add proxy-list rotation policy control

* Add dotCMS fingerprint detection

* Document retries fail streak scan guard

* Fix ignored skip progress counter

* Preserve proxy CLI overrides for resumed scans

* Normalize progress counter width

* Improve remote wordlist startup and header validation

* Deduplicate fingerprint evidence output

* Propagate entrypoint exit codes

* Preserve fractional delay values

* Validate port overrides consistently

* Validate method overrides consistently

* Validate recursive scan options consistently

* Validate report output options consistently

* Validate thread count options consistently

* Normalize WAF option handling

* Document CLI normalization rules

* Preserve scan workspace during pause

* Align indexof progress marker

* Update early finish warning test

* Simplify cache-control provider state

* Harden rainbow logging formatter

* Deduplicate fingerprint probe requests

* Expand passive WAF vendor markers

* Update dictionary

* Add debug runtime diagnostics summary

* Move debug runtime diagnostics below summary

* Stop ignoring common discovery paths by default

* Add regression coverage for fingerprint and text report branches

* Prepare 5.16.1 release

Author: stanislav-web

AGENTS.md

@@ -141,6 +141,134 @@ If a risk cannot be eliminated with deterministic tests, document it in the resp
 
 ---
 
+## CLI/config/session normalization rules
+
+CLI arguments, wizard configuration values, and session resume snapshots must follow one consistent normalization model.
+
+When adding or changing any CLI option that can also appear in `opendoor.conf`, wizard output, or session snapshots:
+
+- keep `argparse` defaults as `None` when the option can be restored from wizard/session state;
+- place runtime defaults in `src/lib/browser/config.py`, not in `argparse`, when the default must not overwrite restored state;
+- validate direct CLI input in `src/core/options/filter.py`;
+- validate wizard/session values again in `BrowserConfig`, because those values may bypass CLI parsing;
+- preserve explicit CLI overrides in `Controller.scan_action()` after wizard/session params are restored;
+- keep direct launcher and installed entrypoint behavior aligned.
+
+### Explicit override model
+
+Use this precedence for runtime params:
+
+```text
+explicit CLI value > session-loaded value > wizard/config value > BrowserConfig runtime default
+```
+
+Do not let parser defaults overwrite wizard/session values.
+
+For boolean `store_true` style flags, prefer:
+
+```text
+argparse default=None
+BrowserConfig default=False
+```
+
+This allows the controller to distinguish:
+
+```text
+flag omitted         -> do not override restored wizard/session state
+flag explicitly set  -> override restored wizard/session state
+```
+
+Do not add `--no-*` negation flags in a patch release unless the maintainer explicitly approves the new public CLI semantics.
+
+### Boolean normalization
+
+Boolean-like values from wizard/config/session must be normalized consistently.
+
+Accepted true values:
+
+```text
+true, True, 1, yes, on
+```
+
+Accepted false values:
+
+```text
+false, False, 0, no, off
+```
+
+Invalid boolean-like values must fail early with a clear validation error.
+
+### Numeric option normalization
+
+Numeric options must not silently fall back when the user provided an invalid value.
+
+Examples:
+
+```text
+--threads 0
+--port 0
+--delay -1
+--recursive-depth 0
+```
+
+Invalid explicit values must reach validation and fail early instead of being dropped as falsy values.
+
+If an option has a runtime clamp, keep validation and clamping separate:
+
+```text
+invalid range       -> validation error
+valid high value    -> documented runtime clamp, if legacy behavior requires it
+```
+
+### CSV/list option normalization
+
+CSV/list options must be normalized consistently:
+
+- trim whitespace;
+- remove empty tokens unless an empty value has explicit semantics;
+- deduplicate while preserving order when duplicate execution would be harmful;
+- validate every token;
+- keep user-visible order when order affects behavior.
+
+Unknown values must fail early instead of reaching plugin/report/runtime loading.
+
+### Raw request interaction
+
+When a CLI option can also be inferred from `--raw-request`, keep precedence deterministic:
+
+```text
+explicit CLI value > raw-request value > runtime default
+```
+
+Raw-request-derived values must pass the same validation helpers as direct CLI values.
+
+### Required tests for CLI/config/session changes
+
+For every new or changed CLI/config/session option, add or update tests for:
+
+- normal CLI flow;
+- invalid CLI values;
+- wizard/config normalization;
+- session-load normalization;
+- explicit CLI override over wizard state;
+- explicit CLI override over session state;
+- `BrowserConfig` runtime default;
+- `BrowserConfig` validation for values that bypass CLI parsing;
+- direct launcher and installed entrypoint behavior when exit code or runtime execution can be affected.
+
+Prefer focused tests in the existing suites:
+
+```text
+tests.test_core_options
+tests.test_core_filter
+tests.test_controller
+tests.test_lib_browser_config
+```
+
+Add runtime/session-specific tests only when the option affects scan execution, report output, fingerprinting, WAF behavior, transport, or session snapshots.
+
+---
+
 ## Packaging rules
 
 - Keep `pyproject.toml` present and valid.

CHANGELOG.md

@@ -1,6 +1,41 @@
 CHANGELOG
 =======
-v5.16.0 (16.05.2026)
+v5.16.1 (24.05.2026)
+---------------------------
+- (fix) reduced duplicate fingerprint traffic by reusing exact same method+URL probe responses within a single fingerprint pass.
+- (fix) avoided false early-finish warnings when planned wordlist entries are intentionally skipped before HTTP submission, such as internally ignored paths.
+- (fix) aligned `indexof` runtime progress output with other sniffer findings by rendering it as `OK (IndexOf)` without changing detection or report semantics.
+- (fix) preserved managed runtime wordlist workspaces across interactive `Ctrl+C` pause/resume while keeping cleanup for aborts, process termination and normal scan completion.
+- (fix) normalized `--fingerprint` handling across CLI, wizard and session resume flows while keeping fingerprinting opt-in and cached session results reusable.
+- (fix) normalized WAF detection flags across CLI, wizard and session resume flows while preserving `waf_safe_mode` as an opt-in runtime profile that enables passive WAF detection.
+- (fix) validated `--threads` values, preserved explicit thread-count overrides for wizard/session resume flows, and raised the safe runtime thread clamp from 25 to 50.
+- (fix) validated scan report selections, normalized `--reports-dir`, and preserved explicit report output overrides for wizard/session resume flows.
+- (fix) validated recursive scan options, preserved explicit recursive CLI overrides for wizard/session resume flows, and kept recursive defaults stable at runtime.
+- (fix) validated `--method` values, preserved explicit method overrides for wizard/session resume flows, and kept raw-request method handling deterministic.
+- (fix) validated `--port` values as TCP ports, preserved explicit port overrides for wizard/session resume flows, and rejected invalid raw-request Host ports early.
+- (fix) propagated `--fail-on-bucket` exit codes through the installed `opendoor` entrypoint so CI/CD runs behave the same as `python opendoor.py`.
+- (fix) preserved fractional `--delay` values (0.1, 0.25...etc), rejected negative delays, and allowed explicit delay overrides for wizard/session resume flows.
+- (fix) reduced malware sniffer false positives by ignoring legitimate Google Tag Manager noscript hidden iframes while preserving detection for non-GTM hidden iframe injections.
+- (fix) kept scans running when individual paths exhaust configured retries while preserving configurable abort protection for consecutive retry failures.
+- (fix) ignored-path progress output so `skip [...]` shows the current scan position instead of `00000`.
+- (fix) proxy CLI overrides for wizard and session resume flows so explicit `--proxy`, `--proxy-pool`, and `--proxy-list --proxy-rotation` selections replace restored proxy settings correctly.
+- (fix) filtered/calibrated progress counters to use the same dynamic width as regular scan findings.
+- (fix) fingerprint evidence output to avoid repeating identical evidence values when the same marker is confirmed by multiple signal sources.
+- (fix) ResponseError: Unknown response status : `477` (non standart error) so scans no longer abort on unexpected HTTP status codes. 
+- (fix) stopped pre-skipping common error, index, redirect-like, and not-found paths from the bundled ignore list so they are scanned and classified normally.
+- (fix) made active shadow probe requests honor the configured scan delay while preserving existing retry, timeout, proxy and request-stack behavior.
+- (enhancement) improved the active `shadow` sniffer with bounded path-template probes such as `{file}}2.{ext}` while keeping per-scan probe limits.
+- (enhancement) improved the `stacktrace` sniffer to detect exposed database connection identity errors like `Could not make a database connection using user@host` while avoiding generic connection-error false positives.
+- (enhancement) hardened `--header` validation and normalization, including wizard/session CLI override handling.
+- (enhancement) added remote HTTP(S) wordlist support through `--wordlist`, including streaming download progress and a 500 MB safety limit.
+- (enhancement) added `--proxy-rotation random|sequential` to control existing `--proxy-list` rotation, preserving `random` as the default and adding deterministic file-order sequential mode.
+- (enhancement) added DotCMS and DiafanCMS detection to `--fingerprint`.
+- (enhancement) added configurable `--retries-fail-streak` to control scan aborts after consecutive exhausted retry paths. Default: `10`.
+- (enhancement) expanded passive WAF recognition coverage with additional 21 WAF systems. Details : (https://opendoor.readthedocs.io/detection/waf-detection/).
+- (ux) added debug-only runtime diagnostics to the terminal scan summary.
+- (dictionary) cleaned and normalized the internal directories list (+2365 potential interesting paths).
+
+v5.16.0 (17.05.2026)
 ---------------------------
 - (fix) rendered fingerprint progress as a rotating single-line indicator and persisted only the final `done` state to reduce duplicate progress output.
 - (fix) proxy and transport-loss handling: proxy scans now validate the proxy without directly probing the target, filtered proxy timeouts remain visible, and direct scans abort cleanly after repeated exhausted transport failures when a target goes offline mid-scan.

README.md

@@ -56,7 +56,7 @@ It helps security researchers, penetration testers, bug bounty hunters, DevSecOp
 - subdomain enumeration;
 - multi-threading scans for faster lookups;
 - single target, target file, stdin, IPv4 CIDR, and IPv4 range input modes;
-- custom wordlists, prefixes, shuffling to break scan patterns and extension filters;
+- custom wordlists, prefixes, shuffling to break scan patterns and extension filters, remote wordlists supports;
 - custom request headers, cookies forwarding, and raw HTTP request templates;
 - response filters by status, size, text, regex, and body length;
 - response sniffers for detecting directory listings, empty responses, known file exposures, active shadow-copy probes, collation, possible exposed secrets, errors, exposed debug stack traces, and verified open redirect vulnerabilities;
@@ -107,7 +107,7 @@ OpenDoor includes a heuristic fingerprint engine for detecting probable applicat
 
 | Category                   | Examples |
 |----------------------------|---|
-| CMS                        | WordPress, Drupal, Joomla, TYPO3, Open Journal Systems, InstantCMS, CMS.S3 / Megagroup, Discuz!, NetCat |
+| CMS                        | WordPress, Drupal, Joomla, TYPO3, Open Journal Systems, InstantCMS, DiafanCMS, CMS.S3 / Megagroup, Discuz!, NetCat |
 | E-commerce                 | Magento, WooCommerce, Shopify, PrestaShop, OpenCart, Shopware, Webasyst / Shop-Script |
 | Frameworks / app platforms | Laravel, Symfony, Django, Flask, FastAPI, Express, NestJS, Next.js, Nuxt, Rails, Spring |
 | Runtime / language stack   | PHP, Node.js, JavaScript, Python, Ruby, .NET, Java/JVM, Elixir, static-site targets |
@@ -293,7 +293,7 @@ Useful sniffers include:
 | `skipsizes=46:1024` | Skip responses inside a noisy size range.                                                                                           |
 | `stacktrace`        | Detect exposed debug/runtime stack traces and internal error details.                                                               |
 | `secret`            | Detect possible exposed API keys, tokens, private keys and credentials with redacted report metadata.                               |
-| `shadow`            | Actively probe confirmed `200 OK` file-like hits for exposed backup/shadow copies such as `.bak`, `.old` etc variants.              |
+| `shadow`            | Actively probe confirmed `200 OK` file-like hits for bounded backup/shadow variants such as `.bak`, `.old`, and path templates like `index2.php`. |
 | `openredirect`      | Actively verify redirect-like query parameters with controlled marker URLs and report only confirmed open redirect vulnerabilities. |
 | `malware`           | Detect possible malicious content, webshell markers, injected scripts or obfuscated payloads.                                       |
 

VERSION

@@ -1 +1 @@
-5.16.0
+5.16.1