-
Notifications
You must be signed in to change notification settings - Fork 3.4k
adding per-host rate limit #7365
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: dev
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -238,6 +238,8 @@ UNCOVER: | |
| RATE-LIMIT: | ||
| -rl, -rate-limit int número máximo de peticiones a enviar por segundo (por defecto 150) | ||
| -rlm, -rate-limit-minute int número máximo de peticiones a enviar por minuto | ||
| -rlh, -rate-limit-host int número máximo de peticiones por host por rate-limit-host-duration (0 = desactivado) | ||
| -rlhd, -rate-limit-host-duration value intervalo de recarga del bucket de rate-limit por host (por defecto 1s) | ||
|
Comment on lines
+241
to
+242
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Document that The implementation gives the per-host limiter priority when both flags are set. Please say that explicitly here too; otherwise this help text reads as if the two limits stack. It would be worth mirroring the same wording in the other localized READMEs updated in this PR. 🤖 Prompt for AI Agents |
||
| -bs, -bulk-size int número máximo de hosts a ser analizados en paralelo por plantilla (por defecto 25) | ||
| -c, -concurrency int número máximo de plantillas a ejecutar en paralelo (por defecto 25) | ||
| -hbs, -headless-bulk-size int número máximo de hosts headless a ser analizados en paralelo por plantilla (por defecto 10) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -205,6 +205,8 @@ UNCOVER: | |
| RATE-LIMIT: | ||
| -rl, -rate-limit int 초당 보낼 최대 요청 수 (기본값 150) | ||
| -rlm, -rate-limit-minute int 분당 보낼 최대 요청 수 | ||
| -rlh, -rate-limit-host int 호스트당 rate-limit-host-duration 동안 보낼 최대 요청 수 (0 = 비활성) | ||
| -rlhd, -rate-limit-host-duration value 호스트별 rate-limit 버킷의 리필 간격 (기본값 1s) | ||
|
Comment on lines
+208
to
+209
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Mention that The Korean help text adds the new flags but omits the precedence rule now documented in 🤖 Prompt for AI Agents |
||
| -bs, -bulk-size int 템플릿당 병렬로 분석할 최대 호스트 수 (기본값 25) | ||
| -c, -concurrency int 병렬로 실행할 최대 템플릿 수 (기본값 25) | ||
| -hbs, -headless-bulk-size int 템플릿당 병렬로 분석할 최대 headless 호스트 수 (기본값 10) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -238,6 +238,8 @@ UNCOVER: | |
| RATE-LIMIT: | ||
| -rl, -rate-limit int número máximo de solicitações a serem enviadas por segundo (padrão 150) | ||
| -rlm, -rate-limit-minute int número máximo de solicitações a serem enviadas por minuto | ||
| -rlh, -rate-limit-host int número máximo de solicitações por host por rate-limit-host-duration (0 = desativado) | ||
| -rlhd, -rate-limit-host-duration value intervalo de recarga do bucket de rate-limit por host (padrão 1s) | ||
|
Comment on lines
+241
to
+242
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Add the The PT-BR help text documents the new flags but not that 🤖 Prompt for AI Agents |
||
| -bs, -bulk-size int número máximo de hosts a serem analisados em paralelo por template (padrão 25) | ||
| -c, -concurrency int número máximo de templates a serem executados em paralelo (padrão 25) | ||
| -hbs, -headless-bulk-size int número máximo de hosts headless a serem analisados em paralelo por template (padrão 10) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,6 +11,7 @@ import ( | |
| "github.com/projectdiscovery/nuclei/v3/pkg/loader/workflow" | ||
| "github.com/projectdiscovery/nuclei/v3/pkg/output" | ||
| "github.com/projectdiscovery/nuclei/v3/pkg/protocols" | ||
| "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/hostratelimit" | ||
| "github.com/projectdiscovery/nuclei/v3/pkg/types" | ||
| "github.com/projectdiscovery/nuclei/v3/pkg/utils" | ||
| "github.com/projectdiscovery/utils/errkit" | ||
|
|
@@ -29,17 +30,18 @@ type unsafeOptions struct { | |
| func createEphemeralObjects(ctx context.Context, base *NucleiEngine, opts *types.Options) (*unsafeOptions, error) { | ||
| u := &unsafeOptions{} | ||
| u.executerOpts = &protocols.ExecutorOptions{ | ||
| Output: base.customWriter, | ||
| Options: opts, | ||
| Progress: base.customProgress, | ||
| Catalog: base.catalog, | ||
| IssuesClient: base.rc, | ||
| RateLimiter: base.rateLimiter, | ||
| Interactsh: base.interactshClient, | ||
| Colorizer: aurora.NewAurora(true), | ||
| ResumeCfg: types.NewResumeCfg(), | ||
| Parser: base.parser, | ||
| Browser: base.browserInstance, | ||
| Output: base.customWriter, | ||
| Options: opts, | ||
| Progress: base.customProgress, | ||
| Catalog: base.catalog, | ||
| IssuesClient: base.rc, | ||
| RateLimiter: base.rateLimiter, | ||
| HostRateLimiter: base.hostRateLimiter, | ||
| Interactsh: base.interactshClient, | ||
| Colorizer: aurora.NewAurora(true), | ||
| ResumeCfg: types.NewResumeCfg(), | ||
| Parser: base.parser, | ||
| Browser: base.browserInstance, | ||
| } | ||
|
Comment on lines
32
to
45
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Don't inherit and then stop the base host limiter.
♻️ Proposed fix type unsafeOptions struct {
executerOpts *protocols.ExecutorOptions
engine *core.Engine
+ ownsHostRateLimiter bool
}
@@
u.executerOpts = &protocols.ExecutorOptions{
Output: base.customWriter,
Options: opts,
Progress: base.customProgress,
Catalog: base.catalog,
IssuesClient: base.rc,
RateLimiter: base.rateLimiter,
- HostRateLimiter: base.hostRateLimiter,
+ HostRateLimiter: nil,
Interactsh: base.interactshClient,
Colorizer: aurora.NewAurora(true),
ResumeCfg: types.NewResumeCfg(),
Parser: base.parser,
Browser: base.browserInstance,
@@
if opts.RateLimitHost > 0 {
hostDuration := opts.RateLimitHostDuration
if hostDuration == 0 {
hostDuration = time.Second
}
u.executerOpts.HostRateLimiter = hostratelimit.NewPool(ctx, hostratelimit.Options{
MaxCount: uint(opts.RateLimitHost),
Duration: hostDuration,
})
+ u.ownsHostRateLimiter = true
}
@@
- u.executerOpts.HostRateLimiter.Stop()
+ if u.ownsHostRateLimiter && u.executerOpts.HostRateLimiter != nil {
+ u.executerOpts.HostRateLimiter.Stop()
+ }Also applies to: 58-70, 78-82 🤖 Prompt for AI Agents |
||
| if opts.ShouldUseHostError() && base.hostErrCache != nil { | ||
| u.executerOpts.HostErrorsCache = base.hostErrCache | ||
|
|
@@ -52,6 +54,21 @@ func createEphemeralObjects(ctx context.Context, base *NucleiEngine, opts *types | |
| opts.RateLimitDuration = time.Second | ||
| } | ||
| u.executerOpts.RateLimiter = utils.GetRateLimiter(ctx, opts.RateLimit, opts.RateLimitDuration) | ||
|
|
||
| // Per-call ephemeral host rate limiter; the goroutine cost is paid once | ||
| // per ExecuteNucleiWithOpts invocation and Stop()-ed in | ||
| // closeEphemeralObjects so we do not leak limiters across calls. | ||
| if opts.RateLimitHost > 0 { | ||
| hostDuration := opts.RateLimitHostDuration | ||
| if hostDuration == 0 { | ||
| hostDuration = time.Second | ||
| } | ||
| u.executerOpts.HostRateLimiter = hostratelimit.NewPool(ctx, hostratelimit.Options{ | ||
| MaxCount: uint(opts.RateLimitHost), | ||
| Duration: hostDuration, | ||
| }) | ||
| } | ||
|
|
||
| u.engine = core.New(opts) | ||
| u.engine.SetExecuterOptions(u.executerOpts) | ||
| return u, nil | ||
|
|
@@ -62,6 +79,7 @@ func closeEphemeralObjects(u *unsafeOptions) { | |
| if u.executerOpts.RateLimiter != nil { | ||
| u.executerOpts.RateLimiter.Stop() | ||
| } | ||
| u.executerOpts.HostRateLimiter.Stop() | ||
| // dereference all objects that were inherited from base nuclei engine | ||
| // since these are meant to be closed globally by base nuclei engine | ||
| u.executerOpts.Output = nil | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -26,6 +26,7 @@ import ( | |||||||||||||||||||||
| "github.com/projectdiscovery/nuclei/v3/pkg/progress" | ||||||||||||||||||||||
| "github.com/projectdiscovery/nuclei/v3/pkg/protocols" | ||||||||||||||||||||||
| "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/hosterrorscache" | ||||||||||||||||||||||
| "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/hostratelimit" | ||||||||||||||||||||||
| "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh" | ||||||||||||||||||||||
| "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolinit" | ||||||||||||||||||||||
| "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate" | ||||||||||||||||||||||
|
|
@@ -101,6 +102,17 @@ func (e *NucleiEngine) applyRequiredDefaults(ctx context.Context) { | |||||||||||||||||||||
| e.rateLimiter = nucleiUtils.GetRateLimiter(ctx, e.opts.RateLimit, e.opts.RateLimitDuration) | ||||||||||||||||||||||
| } | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| if e.hostRateLimiter == nil && e.opts.RateLimitHost > 0 { | ||||||||||||||||||||||
| hostDuration := e.opts.RateLimitHostDuration | ||||||||||||||||||||||
| if hostDuration == 0 { | ||||||||||||||||||||||
| hostDuration = time.Second | ||||||||||||||||||||||
| } | ||||||||||||||||||||||
|
Comment on lines
+105
to
+109
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Guard against non-positive per-host duration values. Line 107 only defaults when duration is Suggested patch- if hostDuration == 0 {
+ if hostDuration <= 0 {
hostDuration = time.Second
}📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||||||
| e.hostRateLimiter = hostratelimit.NewPool(ctx, hostratelimit.Options{ | ||||||||||||||||||||||
| MaxCount: uint(e.opts.RateLimitHost), | ||||||||||||||||||||||
| Duration: hostDuration, | ||||||||||||||||||||||
| }) | ||||||||||||||||||||||
| } | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| if e.opts.ExcludeTags == nil { | ||||||||||||||||||||||
| e.opts.ExcludeTags = []string{} | ||||||||||||||||||||||
| } | ||||||||||||||||||||||
|
|
@@ -212,6 +224,7 @@ func (e *NucleiEngine) init(ctx context.Context) error { | |||||||||||||||||||||
| Catalog: e.catalog, | ||||||||||||||||||||||
| IssuesClient: e.rc, | ||||||||||||||||||||||
| RateLimiter: e.rateLimiter, | ||||||||||||||||||||||
| HostRateLimiter: e.hostRateLimiter, | ||||||||||||||||||||||
| Interactsh: e.interactshClient, | ||||||||||||||||||||||
| Colorizer: aurora.NewAurora(true), | ||||||||||||||||||||||
| ResumeCfg: types.NewResumeCfg(), | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
-rate-limit-durationis described like a count, not a window.-rldis a duration-valued flag, so “maximum number of requests to send per second” is misleading here. The wording should describe the rate-limit interval/window, similar to the new per-host duration flag on Line 282.🤖 Prompt for AI Agents