Skip to content

Refactor loadFromFile to use bufio.Scanner#1791

Open
nandaanomi wants to merge 2 commits into
projectdiscovery:devfrom
nandaanomi:patch-1
Open

Refactor loadFromFile to use bufio.Scanner#1791
nandaanomi wants to merge 2 commits into
projectdiscovery:devfrom
nandaanomi:patch-1

Conversation

@nandaanomi
Copy link
Copy Markdown

@nandaanomi nandaanomi commented May 15, 2026

Proposed changes

Fixes #1790

Two bugs fixed in pkg/runner/util.go:

1. Replaced fileutil.ReadFile (channel-based, goroutine errors not propagatable) with bufio.Scanner + os.Open. scanner.Err() is now returned so any I/O error during mid-read is surfaced to the caller instead of silently producing a partial list.

2. preprocessDomain now strips https:// and http:// prefixes and truncates at the first /, ?, or #. This prevents malformed entries like https://8.8.8.8 in resolver/domain list files from silently failing inside dnsx.

Minor: added make([]string, 0, 64) to reduce reallocs on large input files.

Proof

echo "https://8.8.8.8" > resolvers.txt
subfinder -d example.com -rL resolvers.txt -v

Before: resolver passed as-is, silently unused, no error shown.
After: entry normalized to 8.8.8.8, resolver used correctly.

Checklist

  • Pull request is created against the dev branch
  • All checks passed (lint, unit/integration/regression tests etc.) with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Summary by CodeRabbit

  • Improvements
    • Domain processing now strips HTTP/HTTPS prefixes and removes URL path, query, and fragment components for cleaner normalization, resulting in more consistent matching and deduplication.
    • Improved input file reading for more reliable processing of large or line-oriented inputs.

Review Change Stack

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 15, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f97750ce-cb72-4d87-b4c0-d5071afe4a28

📥 Commits

Reviewing files that changed from the base of the PR and between 133c01d and 553f45a.

📒 Files selected for processing (1)
  • pkg/runner/util.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • pkg/runner/util.go

Walkthrough

pkg/runner/util.go replaces file reading via fileutil.ReadFile with direct bufio.Scanner usage to properly surface mid-read I/O errors, and extends preprocessDomain to strip URL schemes (https://, http://) and truncate input at the first /, ?, or # character to normalize resolver domain entries.

Changes

Resolver File Loading and Domain Preprocessing

Layer / File(s) Summary
File I/O and Domain Preprocessing Pipeline
pkg/runner/util.go
Import dependencies updated from fileutil to bufio, os, and strings. loadFromFile reimplemented to open the file and scan line-by-line with bufio.Scanner, applying preprocessDomain to each line, skipping empty results, and returning scanner.Err() on scan failure to avoid silent mid-read I/O errors. preprocessDomain now also strips leading https:///http:// and truncates at the first /, ?, or #, removing paths, queries, and fragments before normalization.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

A rabbit hops through file streams bright,
Scanner hums and fixes each slight fright.
Schemes are peeled, paths cut clean and fair,
Domains emerge, tidy as a hare. 🐰✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The pull request title clearly and concisely summarizes the main change: refactoring loadFromFile to use bufio.Scanner instead of fileutil.ReadFile.
Linked Issues check ✅ Passed The PR addresses both primary objectives from issue #1790: replacing fileutil.ReadFile with os.Open+bufio.Scanner for proper error propagation, and extending preprocessDomain to strip URL schemes and paths.
Out of Scope Changes check ✅ Passed All changes are directly related to fixing the bugs identified in issue #1790: refactoring file reading, error handling, and URL scheme/path normalization in pkg/runner/util.go.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

  • Generate code and open pull requests
  • Plan features and break down work
  • Investigate incidents and troubleshoot customer tickets together
  • Automate recurring tasks and respond to alerts with triggers
  • Summarize progress and report instantly

Built for teams:

  • Shared memory across your entire org—no repeating context
  • Per-thread sandboxes to safely plan and execute work
  • Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
pkg/runner/util.go (1)

11-28: 💤 Low value

Handle the f.Close() error (or explicitly discard it) to satisfy errcheck.

golangci-lint (errcheck) flags the deferred f.Close() on line 16. For a read-only file the actual risk is negligible, but this will likely fail the CI lint gate. The simplest fix is an explicit discard.

♻️ Proposed tweak
-	defer f.Close()
+	defer func() { _ = f.Close() }()
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/runner/util.go` around lines 11 - 28, The deferred file close in
loadFromFile currently ignores the returned error which trips errcheck; update
the defer to explicitly discard or handle the error from f.Close() (for example
use defer func(){ _ = f.Close() }() or capture and log/return it) so that the
close error is not ignored; change the defer f.Close() inside loadFromFile
accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@pkg/runner/util.go`:
- Around line 11-28: The deferred file close in loadFromFile currently ignores
the returned error which trips errcheck; update the defer to explicitly discard
or handle the error from f.Close() (for example use defer func(){ _ = f.Close()
}() or capture and log/return it) so that the close error is not ignored; change
the defer f.Close() inside loadFromFile accordingly.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 454d648c-6b74-4860-afb8-b5e1ce0b1a3c

📥 Commits

Reviewing files that changed from the base of the PR and between 0ed1b0b and 133c01d.

📒 Files selected for processing (1)
  • pkg/runner/util.go

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] loadFromFile silently drops mid-read I/O errors and does not sanitize URL schemes in pkg/runner/util.go

1 participant