fix(aws_s3 source, aws_sqs source): cap poll_secs at AWS limit of 20#25270
Open
st-omarkhalid wants to merge 1 commit intovectordotdev:masterfrom
Open
fix(aws_s3 source, aws_sqs source): cap poll_secs at AWS limit of 20#25270st-omarkhalid wants to merge 1 commit intovectordotdev:masterfrom
st-omarkhalid wants to merge 1 commit intovectordotdev:masterfrom
Conversation
Closes vectordotdev#22934. Both the `aws_s3` source's `sqs.poll_secs` and the `aws_sqs` source's `poll_secs` configuration fields are forwarded to AWS SQS's `ReceiveMessage` API as the `WaitTimeSeconds` parameter, which AWS hard- caps at 20 seconds. Setting a higher value caused AWS to reject the request and surfaced to users as silent ingestion failure with no error message and no hint in the documentation. Add a `validation(range(max = 20))` constraint to both fields so that: - The auto-generated component reference docs surface the 0-20 range in prose (matching the established pattern for other validated numeric fields, e.g. `decrease_ratio` in adaptive_concurrency). - `vector validate` rejects out-of-range configs before startup. Update the field doc comments to state the AWS-imposed limit explicitly, with a link to the AWS API reference. Regenerate the corresponding component CUE docs via `make generate-docs`.
github-actions
Bot
added
work in progress
domain: sources
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #22934. Both the
aws_s3source'ssqs.poll_secsand theaws_sqssource'spoll_secsare forwarded to AWS SQS'sReceiveMessageAPI as theWaitTimeSecondsparameter, which is hard-capped at 20 seconds by AWS. Setting a higher value silently breaks ingestion (AWS rejects the call; Vector logs nothing user-visible) and the field documentation gave no hint of the limit.This PR:
#[configurable(validation(range(max = 20)))]to both fields sovector validaterejects out-of-range configs before startup.website/cue/reference/components/sources/generated/aws_s3.cueandaws_sqs.cue) viamake generate-docsso the user-facing docs surface the new prose. (Per the established pattern, e.g.decrease_ratioinadaptive_concurrency,validation(range(...))is reflected in user docs as prose; the numeric constraint itself is enforced via the JSON schema.)The
aws_sqssource is included alongsideaws_s3because it has the identical bug atsrc/sources/aws_sqs/config.rs:50→src/sources/aws_sqs/source.rs:111. Happy to split into two PRs if maintainers prefer.Vector configuration
Existing aws_s3/aws_sqs configurations are unchanged in behavior for any value 0..=20; values >20 will now fail config validation with a clear schema error instead of silently breaking at runtime.
How did you test this PR?
All checks run inside the official
timberio/vector-dev:latestDocker image (linux/amd64 under emulation on Apple Silicon, withOPENSSL_NO_VENDOR=1to use the image's system OpenSSL):cargo check -p vector --no-default-features --features sources-aws_s3,sources-aws_sqs— clean (1m 40s)cargo clippy -p vector --no-default-features --features sources-aws_s3,sources-aws_sqs -- -D warnings— clean (1m 12s, no warnings)cargo nextest run -p vector --no-default-features --features sources-aws_s3 -E "test(/sources::aws_s3/)"— 6 passed, 0 failedcargo nextest run -p vector --no-default-features --features sources-aws_sqs -E "test(/sources::aws_sqs/)"— 3 passed, 0 failedmake check-generated-docs— clean post-make generate-docsand commit of the regenerated CUE filesChange Type
Is this a breaking change?
A
validation(range(max = 20))only rejects configurations that were already broken at runtime against AWS's API; valid configurations are unaffected.Does this PR include user facing changes?
no-changeloglabel to this PR.Changelog fragment added:
changelog.d/22934_sqs_poll_secs_range_validation.fix.md.References