adding per-host rate limit#7365
Conversation
WalkthroughThis change implements per-host rate limiting for nuclei. It introduces a new Changes
Sequence Diagram(s)sequenceDiagram
participant CLI as CLI Configuration
participant Runner as Runner
participant Executor as Protocol Executor
participant Pool as HostRateLimiter Pool
participant Global as Global RateLimiter
participant Protocol as HTTP/DNS Handler
CLI->>Runner: Initialize with RateLimitHost + Duration
Runner->>Pool: Create HostRateLimiter Pool (if enabled)
Runner->>Executor: Forward HostRateLimiter in options
loop Per Request
Protocol->>Executor: Request rate-limiting token
Executor->>Pool: RateLimitTakeFor(hostname)
alt Host limiter enabled & host provided
Pool->>Pool: Lookup/create per-host bucket
Pool-->>Executor: Acquire token from host bucket
else Fallback to global
Executor->>Global: RateLimitTake()
Global-->>Executor: Acquire global token
end
Executor-->>Protocol: Token acquired
Protocol->>Protocol: Execute request
end
Runner->>Pool: Stop() on shutdown
Pool->>Pool: Drain all per-host limiters
Pool-->>Runner: Cleanup complete
sequenceDiagram
participant App as Application
participant Sweeper as Sweep Goroutine
participant Pool as Pool State
participant LRU as LRU Tracker
participant Limiter as Per-Host Limiter
App->>Pool: Create Pool(MaxHosts=100, Inactivity=2m)
Pool->>Sweeper: Start background sweep loop
App->>Pool: Take(host1)
Pool->>LRU: Record access time
Pool->>Limiter: Get/create limiter for host1
Limiter-->>Pool: Token acquired
App->>Pool: Take(host2)
Pool->>LRU: Record access time
Pool->>Limiter: Get/create limiter for host2
Limiter-->>Pool: Token acquired
loop Sweep Interval
Sweeper->>Pool: Check idle entries
Sweeper->>LRU: Find entries idle > Inactivity
LRU-->>Sweeper: Idle host list
Sweeper->>Limiter: Stop() idle limiters
Sweeper->>Pool: Remove from cache
end
App->>Pool: Take(host101) [exceeds MaxHosts]
Pool->>LRU: Find LRU entry
LRU-->>Pool: Evict least-recently-used
Pool->>Limiter: Stop() evicted limiter
Pool->>Limiter: Create limiter for host101
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
pkg/protocols/http/request_fuzz.go (1)
60-67:⚠️ Potential issue | 🟠 MajorPer-host rate limiting is bypassed when fuzzing from request files due to malformed host-key input.
In the
ReqRespflow (lines 60-67),input.MetaInput.Inputis overwritten withbaseRequest.String(), which returns a raw HTTP request string (e.g.,GET /path HTTP/1.1\r\nHost: ...). This string is later passed torateLimitHostKey(input)at line 184, which attempts to parse it as a URL viaurlutil.ParseAbsoluteURL(). Since raw HTTP requests are not valid absolute URLs, the parsing fails and the function silently returns an empty string, disabling per-host rate limiting for fuzzing-from-request files (-rlhflag).The host information is available in
baseRequest.URLor the originalrr.URL; extract the host directly before overwritinginput.MetaInput.Input.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/protocols/http/request_fuzz.go` around lines 60 - 67, When building from a ReqResp file in the branch that calls input.MetaInput.ReqResp.BuildRequest() (the baseRequest path), currently input.MetaInput.Input is set to baseRequest.String() which overwrites the original URL and breaks rateLimitHostKey(url) parsing; instead capture the request's absolute URL/host from baseRequest.URL (or the original rr.URL) before overwriting input.MetaInput.Input and set that host/URL into the field rateLimitHostKey expects (e.g., preserve input.MetaInput.Host or set input.MetaInput.InputUrl) so rateLimitHostKey receives a valid absolute URL; update the code around BuildRequest, request.addHeadersToRequest, and where input.MetaInput.Input is assigned to ensure the host is extracted and stored prior to replacing Input with the raw request string.pkg/protocols/http/request.go (1)
554-575:⚠️ Potential issue | 🟠 MajorTake the host-rate-limit token after the final request URL is known.
Here the token is acquired from
rateLimitHostKey(input)beforegenerator.Make()and beforeGetCopyIfHostOutdated(...). Any template that rewrites host/port will be billed against the original input bucket, and plain-host inputs still fall back to the global limiter on the first request.🐛 Proposed fix
executeFunc := func(data string, payloads, dynamicValue map[string]interface{}) (bool, error) { hasInteractMatchers := interactsh.HasMatchers(request.CompiledOperators) - - request.options.RateLimitTakeFor(rateLimitHostKey(input)) ctx := request.newContext(input) ctxWithTimeout, cancel := context.WithTimeoutCause(ctx, request.options.Options.GetTimeouts().HttpTimeout, ErrHttpEngineRequestDeadline) defer cancel() @@ // ideally if http template used a custom port or hostname // we would want to update it in input but currently templateCtx logic // is closely tied to contextargs.Context so we are temporarily creating // a copy and using it to check for host errors etc // but this should be replaced once templateCtx is refactored properly updatedInput := contextargs.GetCopyIfHostOutdated(input, generatedHttpRequest.URL()) + request.options.RateLimitTakeFor(rateLimitHostKey(updatedInput))🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/protocols/http/request.go` around lines 554 - 575, The host rate-limit token is taken too early (before generator.Make and before contextargs.GetCopyIfHostOutdated), causing requests rewritten by templates to be billed against the original input; remove the call to request.options.RateLimitTakeFor(rateLimitHostKey(input)) before generator.Make and instead call request.options.RateLimitTakeFor(rateLimitHostKey(updatedInput)) after you compute updatedInput (i.e., after generator.Make returns and after updatedInput := contextargs.GetCopyIfHostOutdated(...)); reference request.options.RateLimitTakeFor, rateLimitHostKey, generator.Make, contextargs.GetCopyIfHostOutdated, and updatedInput to locate and move the token acquisition.lib/sdk_private.go (1)
1-1:⚠️ Potential issue | 🟡 MinorFormat the code with
go fmtbefore merging.The file
lib/sdk_private.gohas formatting issues that must be fixed per Go coding guidelines. Rungo fmt ./lib/...to apply required formatting changes. Static analysis withgo vetpasses without issues.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@lib/sdk_private.go` at line 1, The file package declaration in lib/sdk_private.go is not formatted per Go standards; run the Go formatter over the lib package (e.g., run `go fmt ./lib/...` or `gofmt -w lib/sdk_private.go`) to reformat this file (package nuclei) before merging so the package declaration and file conform to gofmt style.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@lib/multi.go`:
- Around line 32-45: The HostRateLimiter being assigned into u.executerOpts from
base.hostRateLimiter causes closeEphemeralObjects() to unconditionally Stop()
shared base state; update createEphemeralObjects()/the executor initialization
(where u.executerOpts.HostRateLimiter is set) to ensure the per-call executor
either creates/clones its own HostRateLimiter when inheriting (so it owns the
lifecycle) or marks that it is inherited and then change closeEphemeralObjects()
to only Stop() the limiter if it is owned by this executor (i.e., not equal to
base.hostRateLimiter). Specifically, adjust the logic around
createEphemeralObjects(), closeEphemeralObjects(), HostRateLimiter and
base.hostRateLimiter so inherited pointers are never stopped by per-call cleanup
(or are replaced by a newly allocated limiter owned by u).
In `@lib/sdk_private.go`:
- Around line 105-109: The per-host rate limiter setup uses
e.opts.RateLimitHostDuration but only defaults when it equals zero; change the
guard in the host limiter initialization (the block that references
e.hostRateLimiter and e.opts.RateLimitHostDuration) to normalize any
non-positive value to time.Second (i.e., treat <= 0 as default) before using it
to construct the limiter so negative durations cannot be passed into the limiter
configuration.
In `@pkg/protocols/common/hostratelimit/hostratelimit_perf_test.go`:
- Around line 41-47: The helper drainGet currently calls require.NoError which
may be invoked from spawned worker goroutines (used by the workers around
wg.Wait), causing FailNow to only terminate the goroutine; change drainGet to
return an error instead of calling require.NoError (i.e., func drainGet(... )
error), propagate that error from the goroutine into an errors channel or
buffered slice, close the channel when workers are done, and after wg.Wait()
drain the channel and call require.NoError/require.Empty or assert/fail the test
based on collected errors; update all call sites (the worker function loop and
any tests invoking drainGet) to handle the returned error and send it into the
collector instead of using require inside the goroutine.
In `@pkg/protocols/common/hostratelimit/hostratelimit.go`:
- Around line 121-128: The pool currently refreshes lastAccess before calling
Limiter.Take(), allowing Sweep or the LRU path to Stop() and delete a limiter
while goroutines are waiting; fix by adding an in-flight counter on the limiter
(e.g., increment just before calling Limiter.Take() in Pool.Take and decrement
after the call returns/blocks are done) and make Sweep and the LRU removal logic
check that inflight==0 before calling Stop()/deleting the limiter; apply the
same pattern to the other affected methods referenced around the other blocks
(the code paths in the 175-217 and 241-257 regions) so entries with active
waiters are excluded from eviction.
In `@README_ES.md`:
- Around line 241-242: Update the help text for the rate-limit flags to state
that the per-host flag overrides the global flag: explicitly note that when both
-rlh (or -rate-limit-host) and -rl (global rate-limit) are set, -rlh takes
precedence rather than stacking; modify the existing lines for
-rlh/-rate-limit-host and -rl to include a short parenthetical or sentence
indicating this behavior and mirror the same wording in all other localized
README files updated in this PR.
In `@README_KR.md`:
- Around line 208-209: Update the Korean help text to state that the
-rate-limit-host (-rlh) flag takes precedence over the global -rate-limit (-rl)
flag; edit the entries for -rate-limit-host / -rlh and/or -rate-limit / -rl in
README_KR.md so the precedence rule is explicit (e.g., add a short note like
"주의: -rate-limit-host가 -rate-limit보다 우선합니다") referencing the flags by name so
readers understand -rlh overrides -rl rather than being additive.
In `@README_PT-BR.md`:
- Around line 241-242: Add the same precedence note to the PT-BR help text:
state that the per-host flag (-rlh / -rate-limit-host) takes precedence over the
global flag (-rl / -rate-limit) when both are provided; update the lines that
document -rlh/-rate-limit-host to include a short parenthetical or sentence
noting this precedence and mirror the wording used in the main README for
consistency.
In `@README.md`:
- Around line 278-282: Update the description for the -rld, -rate-limit-duration
flag so it describes a time window/interval rather than a request count; replace
the misleading "maximum number of requests to send per second" with wording like
"rate-limit interval/window for -rate-limit (default 1s)" or "refill interval
for the global rate limit bucket (default 1s)" to match the style of
-rlhd/-rate-limit-host-duration.
---
Outside diff comments:
In `@lib/sdk_private.go`:
- Line 1: The file package declaration in lib/sdk_private.go is not formatted
per Go standards; run the Go formatter over the lib package (e.g., run `go fmt
./lib/...` or `gofmt -w lib/sdk_private.go`) to reformat this file (package
nuclei) before merging so the package declaration and file conform to gofmt
style.
In `@pkg/protocols/http/request_fuzz.go`:
- Around line 60-67: When building from a ReqResp file in the branch that calls
input.MetaInput.ReqResp.BuildRequest() (the baseRequest path), currently
input.MetaInput.Input is set to baseRequest.String() which overwrites the
original URL and breaks rateLimitHostKey(url) parsing; instead capture the
request's absolute URL/host from baseRequest.URL (or the original rr.URL) before
overwriting input.MetaInput.Input and set that host/URL into the field
rateLimitHostKey expects (e.g., preserve input.MetaInput.Host or set
input.MetaInput.InputUrl) so rateLimitHostKey receives a valid absolute URL;
update the code around BuildRequest, request.addHeadersToRequest, and where
input.MetaInput.Input is assigned to ensure the host is extracted and stored
prior to replacing Input with the raw request string.
In `@pkg/protocols/http/request.go`:
- Around line 554-575: The host rate-limit token is taken too early (before
generator.Make and before contextargs.GetCopyIfHostOutdated), causing requests
rewritten by templates to be billed against the original input; remove the call
to request.options.RateLimitTakeFor(rateLimitHostKey(input)) before
generator.Make and instead call
request.options.RateLimitTakeFor(rateLimitHostKey(updatedInput)) after you
compute updatedInput (i.e., after generator.Make returns and after updatedInput
:= contextargs.GetCopyIfHostOutdated(...)); reference
request.options.RateLimitTakeFor, rateLimitHostKey, generator.Make,
contextargs.GetCopyIfHostOutdated, and updatedInput to locate and move the token
acquisition.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: c4f6a3f5-a9f0-4b89-84ff-d8e0d1b32e1d
📒 Files selected for processing (21)
README.mdREADME_CN.mdREADME_ES.mdREADME_ID.mdREADME_KR.mdREADME_PT-BR.mdcmd/nuclei/main.gointernal/runner/runner.gointernal/server/nuclei_sdk.golib/config.golib/multi.golib/sdk.golib/sdk_private.gopkg/protocols/common/hostratelimit/hostratelimit.gopkg/protocols/common/hostratelimit/hostratelimit_perf_test.gopkg/protocols/common/hostratelimit/hostratelimit_test.gopkg/protocols/dns/request.gopkg/protocols/http/request.gopkg/protocols/http/request_fuzz.gopkg/protocols/protocols.gopkg/types/types.go
| u.executerOpts = &protocols.ExecutorOptions{ | ||
| Output: base.customWriter, | ||
| Options: opts, | ||
| Progress: base.customProgress, | ||
| Catalog: base.catalog, | ||
| IssuesClient: base.rc, | ||
| RateLimiter: base.rateLimiter, | ||
| Interactsh: base.interactshClient, | ||
| Colorizer: aurora.NewAurora(true), | ||
| ResumeCfg: types.NewResumeCfg(), | ||
| Parser: base.parser, | ||
| Browser: base.browserInstance, | ||
| Output: base.customWriter, | ||
| Options: opts, | ||
| Progress: base.customProgress, | ||
| Catalog: base.catalog, | ||
| IssuesClient: base.rc, | ||
| RateLimiter: base.rateLimiter, | ||
| HostRateLimiter: base.hostRateLimiter, | ||
| Interactsh: base.interactshClient, | ||
| Colorizer: aurora.NewAurora(true), | ||
| ResumeCfg: types.NewResumeCfg(), | ||
| Parser: base.parser, | ||
| Browser: base.browserInstance, | ||
| } |
There was a problem hiding this comment.
Don't inherit and then stop the base host limiter.
createEphemeralObjects() seeds HostRateLimiter from base.hostRateLimiter, but closeEphemeralObjects() always stops whatever pointer is there. If a call inherits the base pool, that shuts down shared state for later/concurrent ExecuteNucleiWithOpts* calls. The per-call executor should either own its own pool or avoid stopping inherited ones.
♻️ Proposed fix
type unsafeOptions struct {
executerOpts *protocols.ExecutorOptions
engine *core.Engine
+ ownsHostRateLimiter bool
}
@@
u.executerOpts = &protocols.ExecutorOptions{
Output: base.customWriter,
Options: opts,
Progress: base.customProgress,
Catalog: base.catalog,
IssuesClient: base.rc,
RateLimiter: base.rateLimiter,
- HostRateLimiter: base.hostRateLimiter,
+ HostRateLimiter: nil,
Interactsh: base.interactshClient,
Colorizer: aurora.NewAurora(true),
ResumeCfg: types.NewResumeCfg(),
Parser: base.parser,
Browser: base.browserInstance,
@@
if opts.RateLimitHost > 0 {
hostDuration := opts.RateLimitHostDuration
if hostDuration == 0 {
hostDuration = time.Second
}
u.executerOpts.HostRateLimiter = hostratelimit.NewPool(ctx, hostratelimit.Options{
MaxCount: uint(opts.RateLimitHost),
Duration: hostDuration,
})
+ u.ownsHostRateLimiter = true
}
@@
- u.executerOpts.HostRateLimiter.Stop()
+ if u.ownsHostRateLimiter && u.executerOpts.HostRateLimiter != nil {
+ u.executerOpts.HostRateLimiter.Stop()
+ }Also applies to: 58-70, 78-82
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@lib/multi.go` around lines 32 - 45, The HostRateLimiter being assigned into
u.executerOpts from base.hostRateLimiter causes closeEphemeralObjects() to
unconditionally Stop() shared base state; update createEphemeralObjects()/the
executor initialization (where u.executerOpts.HostRateLimiter is set) to ensure
the per-call executor either creates/clones its own HostRateLimiter when
inheriting (so it owns the lifecycle) or marks that it is inherited and then
change closeEphemeralObjects() to only Stop() the limiter if it is owned by this
executor (i.e., not equal to base.hostRateLimiter). Specifically, adjust the
logic around createEphemeralObjects(), closeEphemeralObjects(), HostRateLimiter
and base.hostRateLimiter so inherited pointers are never stopped by per-call
cleanup (or are replaced by a newly allocated limiter owned by u).
| if e.hostRateLimiter == nil && e.opts.RateLimitHost > 0 { | ||
| hostDuration := e.opts.RateLimitHostDuration | ||
| if hostDuration == 0 { | ||
| hostDuration = time.Second | ||
| } |
There was a problem hiding this comment.
Guard against non-positive per-host duration values.
Line 107 only defaults when duration is 0; negative durations pass through. Normalize <= 0 to avoid invalid limiter configuration.
Suggested patch
- if hostDuration == 0 {
+ if hostDuration <= 0 {
hostDuration = time.Second
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if e.hostRateLimiter == nil && e.opts.RateLimitHost > 0 { | |
| hostDuration := e.opts.RateLimitHostDuration | |
| if hostDuration == 0 { | |
| hostDuration = time.Second | |
| } | |
| if e.hostRateLimiter == nil && e.opts.RateLimitHost > 0 { | |
| hostDuration := e.opts.RateLimitHostDuration | |
| if hostDuration <= 0 { | |
| hostDuration = time.Second | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@lib/sdk_private.go` around lines 105 - 109, The per-host rate limiter setup
uses e.opts.RateLimitHostDuration but only defaults when it equals zero; change
the guard in the host limiter initialization (the block that references
e.hostRateLimiter and e.opts.RateLimitHostDuration) to normalize any
non-positive value to time.Second (i.e., treat <= 0 as default) before using it
to construct the limiter so negative durations cannot be passed into the limiter
configuration.
| func drainGet(t testing.TB, client *http.Client, u string) { | ||
| t.Helper() | ||
| resp, err := client.Get(u) | ||
| require.NoError(t, err) | ||
| _, _ = io.Copy(io.Discard, resp.Body) | ||
| _ = resp.Body.Close() | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
wc -l pkg/protocols/common/hostratelimit/hostratelimit_perf_test.goRepository: projectdiscovery/nuclei
Length of output: 133
🏁 Script executed:
cat -n pkg/protocols/common/hostratelimit/hostratelimit_perf_test.go | head -120Repository: projectdiscovery/nuclei
Length of output: 3908
🌐 Web query:
Go testing FailNow safe goroutine spawned worker
💡 Result:
No, calling t.FailNow from a goroutine spawned by a worker in a Go test is not safe. The official Go testing documentation explicitly states that FailNow must be called from the goroutine running the test or benchmark function, not from other goroutines created during the test. Calling it from a spawned goroutine triggers a data race (detectable with -race), only stops that specific goroutine via runtime.Goexit, and leaves other goroutines (including workers) running, potentially causing leaks, hangs, or missed cleanups. To safely signal failure from a worker goroutine: - Use channels to communicate errors back to the test goroutine. - Call t.Error or t.Fatal from the test goroutine after receiving the signal. - Use sync.WaitGroup to wait for all workers to complete before t.FailNow in the test goroutine. - For leak detection, use tools like go.uber.org/goleak.VerifyNone(t). Example safe pattern for worker goroutines: func TestWithWorkers(t *testing.T) { var wg sync.WaitGroup errCh := make(chan error, 10) wg.Add(10) for i := 0; i < 10; i++ { go func(id int) { defer wg.Done if problem := doWork(id); problem != nil { select { case errCh <- problem: default: } } }(i) } go func { wg.Wait close(errCh) } for err := range errCh { t.Errorf("worker error: %v", err) } } This ensures proper synchronization and failure reporting without violating FailNow rules.
Citations:
- 1: https://pkg.go.dev/testing@go1.26.0
- 2: https://pkg.go.dev/testing
- 3: testing: detect t.FailNow() called from wrong goroutine golang/go#24678
- 4: testing: is the package thread-safe? golang/go#20940
- 5: Calling Called() in a goroutine can result in a call to t.FailNow() stretchr/testify#1701
- 6: https://pkg.go.dev/go.uber.org/goleak
Avoid require.* inside worker goroutines.
drainGet calls require.NoError, but this helper is invoked from spawned worker goroutines (line 109). require fails via FailNow, which terminates only the calling goroutine, not the test. A request failure in any worker will silently kill that goroutine without reliably failing the test, creating potential data races and missed assertions. Return the error from drainGet and collect errors into a channel, then assert after wg.Wait() completes.
Also applies to: 95-109
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@pkg/protocols/common/hostratelimit/hostratelimit_perf_test.go` around lines
41 - 47, The helper drainGet currently calls require.NoError which may be
invoked from spawned worker goroutines (used by the workers around wg.Wait),
causing FailNow to only terminate the goroutine; change drainGet to return an
error instead of calling require.NoError (i.e., func drainGet(... ) error),
propagate that error from the goroutine into an errors channel or buffered
slice, close the channel when workers are done, and after wg.Wait() drain the
channel and call require.NoError/require.Empty or assert/fail the test based on
collected errors; update all call sites (the worker function loop and any tests
invoking drainGet) to handle the returned error and send it into the collector
instead of using require inside the goroutine.
| func (p *Pool) Take(host string) { | ||
| if p == nil || host == "" { | ||
| return | ||
| } | ||
| l := p.getOrCreate(host) | ||
| if l != nil { | ||
| l.Take() | ||
| } |
There was a problem hiding this comment.
Don't evict a limiter while callers can still be blocked on it.
lastAccess is refreshed before Limiter.Take(), so a host with a long refill window can look idle even while goroutines are actively waiting on that bucket. Either the sweep or the LRU path can then Stop() and delete the limiter, and the next access recreates a fresh bucket with full tokens, which breaks the per-host limit.
Please track in-flight users (or otherwise exclude actively waited-on entries from eviction) before reclaiming a limiter.
Also applies to: 175-217, 241-257
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@pkg/protocols/common/hostratelimit/hostratelimit.go` around lines 121 - 128,
The pool currently refreshes lastAccess before calling Limiter.Take(), allowing
Sweep or the LRU path to Stop() and delete a limiter while goroutines are
waiting; fix by adding an in-flight counter on the limiter (e.g., increment just
before calling Limiter.Take() in Pool.Take and decrement after the call
returns/blocks are done) and make Sweep and the LRU removal logic check that
inflight==0 before calling Stop()/deleting the limiter; apply the same pattern
to the other affected methods referenced around the other blocks (the code paths
in the 175-217 and 241-257 regions) so entries with active waiters are excluded
from eviction.
| -rlh, -rate-limit-host int número máximo de peticiones por host por rate-limit-host-duration (0 = desactivado) | ||
| -rlhd, -rate-limit-host-duration value intervalo de recarga del bucket de rate-limit por host (por defecto 1s) |
There was a problem hiding this comment.
Document that -rlh overrides -rl.
The implementation gives the per-host limiter priority when both flags are set. Please say that explicitly here too; otherwise this help text reads as if the two limits stack. It would be worth mirroring the same wording in the other localized READMEs updated in this PR.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@README_ES.md` around lines 241 - 242, Update the help text for the rate-limit
flags to state that the per-host flag overrides the global flag: explicitly note
that when both -rlh (or -rate-limit-host) and -rl (global rate-limit) are set,
-rlh takes precedence rather than stacking; modify the existing lines for
-rlh/-rate-limit-host and -rl to include a short parenthetical or sentence
indicating this behavior and mirror the same wording in all other localized
README files updated in this PR.
| -rlh, -rate-limit-host int 호스트당 rate-limit-host-duration 동안 보낼 최대 요청 수 (0 = 비활성) | ||
| -rlhd, -rate-limit-host-duration value 호스트별 rate-limit 버킷의 리필 간격 (기본값 1s) |
There was a problem hiding this comment.
Mention that -rate-limit-host overrides -rate-limit.
The Korean help text adds the new flags but omits the precedence rule now documented in README.md. That can make -rl and -rlh look additive instead of mutually prioritized.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@README_KR.md` around lines 208 - 209, Update the Korean help text to state
that the -rate-limit-host (-rlh) flag takes precedence over the global
-rate-limit (-rl) flag; edit the entries for -rate-limit-host / -rlh and/or
-rate-limit / -rl in README_KR.md so the precedence rule is explicit (e.g., add
a short note like "주의: -rate-limit-host가 -rate-limit보다 우선합니다") referencing the
flags by name so readers understand -rlh overrides -rl rather than being
additive.
| -rlh, -rate-limit-host int número máximo de solicitações por host por rate-limit-host-duration (0 = desativado) | ||
| -rlhd, -rate-limit-host-duration value intervalo de recarga do bucket de rate-limit por host (padrão 1s) |
There was a problem hiding this comment.
Add the -rlh vs -rl precedence note here too.
The PT-BR help text documents the new flags but not that -rate-limit-host takes priority over the global -rate-limit, which is now called out in the main README.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@README_PT-BR.md` around lines 241 - 242, Add the same precedence note to the
PT-BR help text: state that the per-host flag (-rlh / -rate-limit-host) takes
precedence over the global flag (-rl / -rate-limit) when both are provided;
update the lines that document -rlh/-rate-limit-host to include a short
parenthetical or sentence noting this precedence and mirror the wording used in
the main README for consistency.
| -rl, -rate-limit int maximum number of requests to send per second (ignored when -rate-limit-host is set) (default 150) | ||
| -rld, -rate-limit-duration value maximum number of requests to send per second (default 1s) | ||
| -rlm, -rate-limit-minute int maximum number of requests to send per minute (DEPRECATED) | ||
| -rlh, -rate-limit-host int maximum number of requests to send per host per rate-limit-host-duration (0 = disabled, takes priority over -rate-limit) | ||
| -rlhd, -rate-limit-host-duration value refill interval for the per-host rate limit bucket (default 1s) |
There was a problem hiding this comment.
-rate-limit-duration is described like a count, not a window.
-rld is a duration-valued flag, so “maximum number of requests to send per second” is misleading here. The wording should describe the rate-limit interval/window, similar to the new per-host duration flag on Line 282.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@README.md` around lines 278 - 282, Update the description for the -rld,
-rate-limit-duration flag so it describes a time window/interval rather than a
request count; replace the misleading "maximum number of requests to send per
second" with wording like "rate-limit interval/window for -rate-limit (default
1s)" or "refill interval for the global rate limit bucket (default 1s)" to match
the style of -rlhd/-rate-limit-host-duration.
1 participant
Proposed changes
Adds a per-host request rate limiter so users can cap requests per individual target without throttling the entire scan to that rate.
Same workload (8 hosts × 20 reqs = 160 reqs) at the same effective per-host rate (10 tokens / 100ms): the base version with -rl 10 takes 1.501s (107 rps), this PR with -rlh 10 takes 0.100s (1597 rps), a ~15x speedup at the same per-host safety guarantee. The gain scales with the number of distinct hosts in the scan; single-host scans are unaffected.
Closes #3547.
Checklist
Summary by CodeRabbit
Release Notes
-rate-limit-hostto cap requests per host and-rate-limit-host-durationto configure the time window (defaults to 1 second). When enabled, per-host limits take precedence over global rate limits.