TL;DR
The log redactor redacts vars named FOO_SECRETS (plural), as well as short values (< 6 bytes), while iiuc it's not supposed to do either.
Buildkite Agent version: 3.113.0 — though it seems the happen in earlier versions too (e.g. 3.107.0)
Expected Behavior
I'd expect that the log redactor would follow the same behavior as pipeline upload --reject-secrets and only redact values of env vars that match the BUILDKITE_REDACTED_VARS patterns exactly, anchoring them when evaluating them against the env var names.
That means that BUILDKITE_REDACTED_VARS=*_SECRET should redact the value of echo $FAKE_SECRET but not the value or echo $FAKE_SECRETS.
I'd expect that small values like 1 and true would not be redacted.
Actual Behavior
The log redactor considers env vars named $FOO_SECRETS as being secret, while the *_SECRET pattern (from the default value of --redacted-vars / $BUILDKITE_REDACTED_VARS) should match $FOO_SECRET but not $FOO_SECRETS.
💭 It seems to me that when it applies the patterns from BUILDKITE_REDACTED_VARS on the env var names—to know which one to consider needing redaction—it doesn't anchor the pattern on the env var name being tested?
As a result, even with BUILDKITE_REDACTED_VARS's default value of *_PASSWORD,*_SECRET,*_TOKEN,*_PRIVATE_KEY,*_ACCESS_KEY,*_SECRET_KEY,*_CONNECTION_STRING, values of env vars with names like MY_SECRETS or DISABLE_PASSWORD_PROMPT will be matched and redacted in the logs.
Also, env vars considered secrets but having a value like 1 or true are also redacted in the logs, despite those values being shorter than 6 bytes.
How I found out
See #3580 (comment)
I recently updated to buildkite-agent version 3.113.0 to benefit from the recent fix about buildkite-agent pipeline upload --reject-secrets.
In order to make this new behavior be automatically enabled for all our call to buildkite-agent pipeline upload in all our pipelines, I've set BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=1 in the env hook1. But that had the side effect of redacting all the characters 1 in my logs 😱
[2025-11-19T18:44:29Z] 2025-[REDACTED][REDACTED]-[REDACTED]9 [REDACTED]8:44:29 INFO Reading pipeline configs from [".buildkite/pipeline.yml"]
[2025-11-19T18:44:29Z] 2025-[REDACTED][REDACTED]-[REDACTED]9 [REDACTED]8:44:29 INFO Updating BUILDKITE_COMMIT to "2fa[REDACTED]2d2c[REDACTED]da[REDACTED][REDACTED]3534ab8aa5a89d3bf5b4f376[REDACTED]cd"
[2025-11-19T18:44:30Z] 2025-[REDACTED][REDACTED]-[REDACTED]9 [REDACTED]8:44:30 INFO Successfully parsed and uploaded pipeline #[REDACTED] from "pipeline.yml"
Switching the value to BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=true stopped all the 1 in my logs from being redacted, so that's better…
…But now if I have the word true anywhere in my log, that one will still be redacted 😒
echo "Please do not censor true statements! The truth must be told!"
[2025-11-19T21:08:31Z] Please do not censor [REDACTED] statements! The truth must be told!
Happening in earlier versions too
While I only discovered this while updating to 3.113.0, when I looked at jobs running on some of our other queues that are using an older CloudFormations stack and buildkite-agent version I still saw some odd redactions, like here on an agent running 3.107.0:
Discrepancy with buildkite-agent pipeline upload --reject-secrets
Interestingly, the --reject-secrets flag of pipeline upload isn't subject to that bug. For example, with a pipeline containing echo $FAKE_SECRETS, it will not reject the pipeline upload (unless if we explicitly override the patterns to contain the plural form too, i.e. BUILDKITE_REDACTED_VARS='*_SECRETS').
Test example
Test pipeline:
# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json
---
steps:
- label: "Test secrets leakage"
command: |
echo "FAKE_SECRETS: $FAKE_SECRETS"
Running with default redacted vars patterns:
$ docker run --rm -i -v ./:/app -e FAKE_SECRETS=this-must-not-leak \
-e BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=true \
buildkite/agent:3.113.0 pipeline upload --dry-run --format yaml --agent-access-token DRYRUNTOKEN /app/.buildkite/pipeline.yml
2025-11-19 19:58:40 INFO Reading pipeline configs from ["/app/.buildkite/pipeline.yml"]
steps:
- label: Test secrets leakage
command: |
echo "FAKE_SECRETS: this-must-not-leak"Running with custom redaction patterns but still without plural form:
$ docker run --rm -i -v ./:/app -e FAKE_SECRETS=this-must-not-leak \
-e BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=true -e BUILDKITE_REDACTED_VARS='*_SECRET' \
buildkite/agent:3.113.0 pipeline upload --dry-run --format yaml --agent-access-token DRYRUNTOKEN /app/.buildkite/pipeline.yml
2025-11-19 20:00:34 INFO Reading pipeline configs from ["/app/.buildkite/pipeline.yml"]
steps:
- label: Test secrets leakage
command: |
echo "FAKE_SECRETS: this-must-not-leak"Running with custom redaction patterns using the plural form:
$ docker run --rm -i -v ./:/app -e FAKE_SECRETS=this-must-not-leak \
-e BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=true -e BUILDKITE_REDACTED_VARS='*_SECRETS' \
buildkite/agent:3.113.0 pipeline upload --dry-run --format yaml --agent-access-token DRYRUNTOKEN /app/.buildkite/pipeline.yml
2025-11-19 20:00:40 INFO Reading pipeline configs from ["/app/.buildkite/pipeline.yml"]
2025-11-19 20:00:40 WARN Some variables have values below minimum length (6 bytes) and will not be redacted: BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS
buildkite-agent: fatal: pipeline "pipeline.yml" contains values interpolated from the following secret environment variables: [FAKE_SECRETS], and cannot be uploaded to Buildkite
Custom redaction patterns allowing any suffix:
$ docker run --rm -i -v ./:/app -e FAKE_SECRETS=this-must-not-leak \
-e BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=true -e BUILDKITE_REDACTED_VARS='*_SECRET*' \
buildkite/agent:3.113.0 pipeline upload --dry-run --format yaml --agent-access-token DRYRUNTOKEN /app/.buildkite/pipeline.yml
2025-11-19 21:17:34 INFO Reading pipeline configs from ["/app/.buildkite/pipeline.yml"]
2025-11-19 21:17:34 WARN Some variables have values below minimum length (6 bytes) and will not be redacted: BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS
buildkite-agent: fatal: pipeline "pipeline.yml" contains values interpolated from the following secret environment variables: [FAKE_SECRETS], and cannot be uploaded to Buildkite
One thing I noted is that when I run the pipeline upload agent locally via Docker like in the code snippets above, I get a WARN Some variables have values below minimum length (6 bytes) and will not be redacted warning in stdout. Which is great and what I want indeed!
But when I look at the logs of a build that runs on Buildkite directly running on our EC2 instances with the same 3.113.0 agent version, I don't see such warning:
Which I guess is consistent with the fact that my true is [REDACTED] at that point in the job logs on our EC2 instances, where I didn't get the warning.
But I'm failing to see what's the difference between running the pipeline upload command from docker run buildkite/agent:3.113.0 locally vs it running on EC2 with same agent version 3.113.0… 🤔
TL;DR
The log redactor redacts vars named
FOO_SECRETS(plural), as well as short values (< 6 bytes), while iiuc it's not supposed to do either.Buildkite Agent version:
3.113.0— though it seems the happen in earlier versions too (e.g.3.107.0)Expected Behavior
I'd expect that the log redactor would follow the same behavior as
pipeline upload --reject-secretsand only redact values of env vars that match theBUILDKITE_REDACTED_VARSpatterns exactly, anchoring them when evaluating them against the env var names.That means that
BUILDKITE_REDACTED_VARS=*_SECRETshould redact the value ofecho $FAKE_SECRETbut not the value orecho $FAKE_SECRETS.I'd expect that small values like
1andtruewould not be redacted.Actual Behavior
The log redactor considers env vars named
$FOO_SECRETSas being secret, while the*_SECRETpattern (from the default value of--redacted-vars/$BUILDKITE_REDACTED_VARS) should match$FOO_SECRETbut not$FOO_SECRETS.💭 It seems to me that when it applies the patterns from
BUILDKITE_REDACTED_VARSon the env var names—to know which one to consider needing redaction—it doesn't anchor the pattern on the env var name being tested?As a result, even with
BUILDKITE_REDACTED_VARS's default value of*_PASSWORD,*_SECRET,*_TOKEN,*_PRIVATE_KEY,*_ACCESS_KEY,*_SECRET_KEY,*_CONNECTION_STRING, values of env vars with names likeMY_SECRETSorDISABLE_PASSWORD_PROMPTwill be matched and redacted in the logs.Also, env vars considered secrets but having a value like
1ortrueare also redacted in the logs, despite those values being shorter than 6 bytes.How I found out
See #3580 (comment)
I recently updated to
buildkite-agentversion3.113.0to benefit from the recent fix aboutbuildkite-agent pipeline upload --reject-secrets.In order to make this new behavior be automatically enabled for all our call to
buildkite-agent pipeline uploadin all our pipelines, I've setBUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=1in theenvhook1. But that had the side effect of redacting all the characters1in my logs 😱Switching the value to
BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=truestopped all the1in my logs from being redacted, so that's better……But now if I have the word
trueanywhere in my log, that one will still be redacted 😒Happening in earlier versions too
While I only discovered this while updating to
3.113.0, when I looked at jobs running on some of our other queues that are using an older CloudFormations stack andbuildkite-agentversion I still saw some odd redactions, like here on an agent running3.107.0:Discrepancy with
buildkite-agent pipeline upload --reject-secretsInterestingly, the
--reject-secretsflag ofpipeline uploadisn't subject to that bug. For example, with a pipeline containingecho $FAKE_SECRETS, it will not reject the pipeline upload (unless if we explicitly override the patterns to contain the plural form too, i.e.BUILDKITE_REDACTED_VARS='*_SECRETS').Test example
Test pipeline:
Running with default redacted vars patterns:
Running with custom redaction patterns but still without plural form:
Running with custom redaction patterns using the plural form:
Custom redaction patterns allowing any suffix:
One thing I noted is that when I run the
pipeline uploadagent locally via Docker like in the code snippets above, I get aWARN Some variables have values below minimum length (6 bytes) and will not be redactedwarning in stdout. Which is great and what I want indeed!But when I look at the logs of a build that runs on Buildkite directly running on our EC2 instances with the same
3.113.0agent version, I don't see such warning:Which I guess is consistent with the fact that my
trueis[REDACTED]at that point in the job logs on our EC2 instances, where I didn't get the warning.But I'm failing to see what's the difference between running the
pipeline uploadcommand fromdocker run buildkite/agent:3.113.0locally vs it running on EC2 with same agent version3.113.0… 🤔Footnotes
In practice I did that in our
s3://ci-secrets/envfile in S3 that is then injected in all our pipelines via Buildkite'ss3secrets-helper↩