chore(deps): update dependency promptfoo to v0.121.17

[renovate]

ℹ️ Note

This PR body was truncated due to platform limits.

This PR contains the following updates:

Package	Change	Age	Confidence
promptfoo (source)	0.121.2 → 0.121.17

---

Warning

Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

---

Release Notes

promptfoo/promptfoo (promptfoo)

v0.121.17

Compare Source

Bug Fixes

• docker: bump pinned Python to 3.14 to match the Alpine base image (#​9769) (24fc5be)
• docker: unpin Python minor by default so Alpine drift can't break the build (#​9771) (c69ef40)
• matchers: drop stranded list markers in RAGAS context-relevance segmentation (#​9767) (b06eb88)
• sanitizer: stop over-redacting benign url values that mention a credential keyword (#​9764) (77abc5e)

v0.121.16

Compare Source

Features

• providers: add Claude Fable and Mythos support (#​9671) (09435af)
• redteam: publish all four promptfoo skills to the Claude Code marketplace (#​9665) (53a7266)

Bug Fixes

• address AI quality findings (#​9637) (c54a306)
• assertions: correct BLEU brevity penalty formula (#​9717) (2a97af6)
• assertions: honor a test/assert-set threshold of 0 (#​9736) (2547025)
• assertions: honor inverse (not-) for cost, latency, and perplexity (#​9725) (1c9daa0)
• assertions: honor inverse (not-) for levenshtein assertion (#​9722) (560ea2d)
• assertions: honor inverse for not-moderation (#​9738) (c56c712)
• assertions: make BLEU closest-reference tie-break order-independent (#​9718) (051fbf9)
• assertions: make not-similar with an array require dissimilarity to all values (#​9737) (6c3e7bb)
• assertions: report best similarity score when no array value passes (#​9721) (94eaaf3)
• assertions: respect inverse when an object value can't parse output as JSON (#​9626) (052a4e1)
• assertions: score ROUGE case-insensitively (#​9739) (386c012)
• assertions: stop BLEU collapsing scores for short candidates (#​9740) (ec0e225)
• build: make biome Style Check run deterministically (fix stack-overflow no-op) + fix unmasked lint errors (#​9631) (836733f)
• ci: update architecture edge baseline for evalTableUtils CSV imports (#​9633) (ea3348e)
• cli: accept falsy provider outputs in validation (#​9644) (c5e4b80)
• code-scan: keep SARIF skip errors off stdout (#​9690) (e1e062d)
• code-scan: restore log level after structured scans (#​9196) (ca4821c)
• codex: propagate trace identity to CLI telemetry (#​9703) (29db0a8)
• csv: unescape all escaped commas in array metadata values (#​9757) (9ef75a2)
• deps: constrain undici to <7.27.1 to fix Node 26 "terminated" error (#​9668) (2774371)
• deps: gate Content-Encoding strip after decompress and require undici >=7.27.1 <8 (#​9683) (d6031dc)
• deps: pin undici to 7.27.1 (#​9711) (67ae571)
• deps: update anthropic packages (#​9634) (d932250)
• deps: update anthropic packages (#​9651) (4fcaa87)
• deps: update anthropic packages (#​9733) (b320a7b)
• deps: update dependency @​opentelemetry/core to v2.8.0 [security] (#​9754) (fd11498)
• eval: escape CSV formula injection in result exports (#​9609) (c039bff)
• eval: handle config directory inputs without promptfooconfig (#​7825) (30dc14b)
• eval: restore CSV export architecture boundary (#​9630) (7aaa3ed)
• honor remote data-sharing controls (#​9664) (49a175a)
• http: redact transformed request debug logs (#​9425) (4190a3c)
• matchers: respect filtered remote target context (#​9704) (86ec3ab)
• matchers: segment prose context into sentences for RAGAS context relevance (#​9734) (ca75779)
• output: wait for JSONL file release (#​9618) (9d46d14)
• preserve prompt function across beforeAll extension hooks (#​9661) (d984059)
• providers: preserve Anthropic thinking token usage (#​9680) (6b41b43)
• providers: respect originator header overrides (#​9585) (c9fe77f)
• redteam: allow token plugins with multi-input (#​9657) (03f888f)
• redteam: make skill commands and script paths work for marketplace users (#​9670) (aba48e4)
• redteam: preserve team context in cloud tasks (#​9677) (a7bac57)
• redteam: publish marketplace skill fixes (#​9676) (3c761c5)
• redteam: surface remote generation body read failures (#​9751) (8a1bec2)
• share: preserve blob refs per result row (#​9749) (e029151)
• webui: redact test filter provider credentials (#​9529) (5d809ef)

v0.121.15

Compare Source

Features

• assertions: grade multimodal outputs (#​9617) (ec4d3b4)
• build: add DAG-aware architecture cycle checks (#​9557) (9d7d810)
• providers: add Azure MAI image provider and MAI model recognition (#​9595) (e091527)
• providers: upgrade MiniMax default model to M3 (#​9598) (43b30c5)

Bug Fixes

• auth: enable auto-share for on-prem Report Server after login (#​9612) (8ece593)
• auth: preserve cloud request headers (#​9610) (3a2c5e6)
• code-scan: process findings in mixed skip responses (#​9525) (aeae790)
• code-scan: reject fractional inline comment lines (#​9556) (0a2b329)
• eval: compose metadata and result filters (#​9604) (49539ee)
• eval: preserve --config ordering when expanding a directory entry (#​9606) (fc081a0)
• harden validation and target discovery (#​9614) (465d610)
• providers: require deployment name for no-default Azure provider types (#​9603) (da48abd)
• providers: suppress optional GCP metadata warning (#​9554) (8dcc44b)
• redteam: restrict remote auth to cloud host (#​9597) (1b2c6d9)

v0.121.14

Compare Source

Features

• add A2A provider (#​9586) (963b264)
• assertions: add agent-rubric grader (#​9453) (cadb3c5)
• build: publish a lightweight promptfoo/contracts subpath (#​9535) (6e89fd4)
• cli: add seeded random sampling (#​9522) (6e4ba60)
• eval: extract eval-creator readiness and validation modules (#​9397) (d3af118)
• eval: surface trace linkage on result rows (#​9027) (9468d66)
• providers: add OpenAI GPT-5.5/5.4 frontier models + Codex on Amazon Bedrock (#​9587) (1e35267)
• providers: add OpenAI originator header (#​9474) (6141070)
• providers: expose traceable agent turn markers (#​9475) (4018837)
• providers: promote Fireworks AI from registry stub to dedicated provider (#​9542) (2bae148)
• tracing: add runtime receiver controls (#​9028) (90df6a1)

Bug Fixes

• assertions: don't classify gen_ai.tool.definitions chat spans as tool calls (#​9524) (b59f397)
• cli: preflight force imports before collision lookup (#​9570) (0b93733)
• cloud: on-prem API host for guardrails and http-generator, with host-resolution tests (#​9580) (b9a014a)
• cloud: use on-prem API host in checkEmailStatus (#​9576) (063c62b)
• db: avoid SQLITE_LOCKED flakiness in shared-cache test database (#​9567) (1fdb59b)
• db: serialize libsql test database cleanup (#​9540) (f4380c2)
• deps: keep ModelAudit pydantic-core pinned to compatible 2.46.4 (b2b35b0)
• deps: update dependency ai to ^6.0.190 (#​9577) (9ec614a)
• eval: canonicalize retry JSONL output with atomic rewrites (#​9547) (8d7c920)
• eval: redact credentials from the persisted browser store (#​9396) (4d5bed5)
• evaluator: preserve and harden programmatic JSONL output (#​9538) (8ddd906)
• output: redact api-key and legacy transport headers in JSONL/DB (#​9546) (e194c85)
• providers: handle Codex SDK rate limits (#​9473) (76d3db4)
• providers: inject n8n sessions into custom bodies (#​9527) (9cc0542)
• providers: preserve n8n array body templates (#​9544) (6cdf63d)
• providers: preserve streamed Anthropic refusal guardrails (#​9560) (ff8eafd)
• providers: serialize persistent browser sessions (#​9414) (097ff9b)
• redteam: authenticate remote-generation requests against on-prem cloud (#​9584) (7df8fae)
• util: restore nested SAS tokens after array reorder (#​9528) (9759e5a)
• webui: render negative-only metric charts (#​9526) (7ccafa4)

v0.121.13

Compare Source

Features

• add MiniMax provider (#​8207) (ebd78f6)
• anthropic: add support for Claude Opus 4.8 (#​9500) (e090730)
• assertions: add ignore option to trajectory:tool-args-match (#​9466) (6423f2a)
• assertions: add metadata shortcut to assertion context (#​6863) (5264680)
• assertions: support glob patterns in trajectory:tool-args-match ignore (#​9484) (18b3eef)
• assertions: tolerate optional defaults in trajectory:tool-args-match (#​9471) (a74894a)
• cli: add prompt optimization (#​9294) (27d6dbe)
• cli: warn on deprecated Node.js 20 runtime (#​9494) (8de3408)
• eval: add eval runtime vars (#​9331) (3c76045)
• eval: add output detail deep links (#​9170) (1f5845b)
• eval: load test sets from azure blob storage (#​9282) (b5ea8ab)
• google: add Gemini 3.5 Flash and refresh Google/Vertex models (#​9376) (6c81c8d)
• output: add html report detail drawer (#​9291) (8c54598)
• output: add named metric columns to CSV exports (#​9480) (4531750)
• providers: add MLflow AI Gateway provider (#​8860) (d0e7e2d)
• providers: add n8n webhook provider (#​6770) (3dc6e32)
• providers: add Novita provider (#​8190) (42930e8)
• providers: add NVIDIA NIM provider (#​9491) (fa3e4a9)
• providers: add OrcaRouter provider (#​9493) (fe115ab)
• redteam: add runtime tags to redteam run (#​9499) (72d4cb4)

Bug Fixes

• app: clarify metric pill filtering (#​9252) (05e5a30)
• app: reorder eval output actions (#​9261) (91dc814)
• cache: make the fetch cache key stable across processes (#​9509) (7be3ef5)
• ci: avoid shell interpolation in telemetry check (#​9280) (4160a90)
• ci: disable deploy dependency cache (#​9281) (f3a2502)
• ci: pin code-scan Node to 24.15.0 to stop install timeout (#​9374) (e200bb5)
• ci: scope code-scan-action mirror sync to release artifacts (#​9363) (76c3a29)
• classify script stderr by severity (#​9459) (1eeef19)
• cli: validate optimization split input (#​9514) (e7e29d1)
• code-scan: emit structured fork PR skip output (#​9426) (61c624c)
• code-scan: honor minimum-severity alias when min-severity is unset (#​9433) (ea5ea9e)
• code-scan: suppress SARIF on skipped fork scans (#​9451) (8236e65)
• config: dedupe repeated executable providers (#​9430) (6325653)
• config: preserve all function providers in combineConfigs (#​9408) (ed4e29e)
• db: isolate libsql test databases (#​9504) (b01b257)
• db: parameterize JSON path filters (#​9345) (dc5e2ed)
• db: replace better-sqlite3 with libsql (#​9486) (ca3dea4)
• db: restore SQLite busy_timeout for the libsql driver (#​9508) (a74b2fe)
• deps: avoid Socket-blocked package releases (#​9456) (89d6e61)
• deps: bump brace-expansion and webpack-dev-server [security] (#​9339) (bb1d57e)
• deps: lazy-load optional integration packages (#​9284) (db37941)
• deps: patch qs denial of service advisory (#​9384) (b392051)
• deps: update anthropic packages (#​9461) (87277b9)
• deps: update dependency engine.io to v6.6.7 (#​9457) (5f36395)
• deps: update dependency engine.io to v6.6.8 (#​9477) (3366b67)
• deps: update dependency engine.io-client to v6.6.5 (#​9478) (32610e3)
• eval: accept defaultTest-only configs in prompt optimizer (#​9365) (f8ecebd)
• eval: error when optimizer pair has no runnable tests (#​9372) (45046ac)
• eval: handle Google Sheets export classification (#​9385) (5ffeb33)
• eval: invalidate count cache after retry cleanup (#​9431) (3ea3599)
• eval: invalidate history cache after result update (#​9487) (2e48494)
• eval: invalidate result-count cache on insert (#​9348) (#​9421) (4651ceb)
• eval: keep eval runtime vars out of persisted results (#​9370) (734db7c)
• eval: preserve all distinct CallApiFunction providers in combineConfigs (#​9402) (b7f2a93)
• eval: preserve optimized prompt routing (#​9391) (fff7ca4)
• eval: preserve prompt optimizer progress when a later round fails (#​9505) (211fadf)
• eval: prevent optimize from polluting jsonl output (#​9364) (1205a2a)
• eval: redact Azure Blob SAS tokens in outputs (#​9386) (23a5376)
• eval: refresh history cache after inserts (#​9496) (ed17310)
• eval: restore result header borders and live metrics (#​9273) (4ceaeb5)
• eval: reuse configured grading provider references (#​9460) (28b26d9)
• eval: show test descriptions in HTML output (#​9390) (206f1dd)
• handle chart and Google response edge cases (#​9409) (09701a8)
• http: isolate auth token cache (#​8636) (8d745cd)
• http: redact debug request metadata (#​8635) (6594491)
• mcp: use cryptographic session identifiers (#​9392) (c6f5144)
• providers: align Gemini safety block handling (#​9387) (45fed35)
• providers: harden callback file URL parsing (#​9479) (d6bfc1f)
• providers: honor Vertex default host overrides (#​9389) (11a8d9b)
• providers: improve invalid provider diagnostics (#​9405) (04c5aa4)
• providers: normalize Claude Opus compatibility (#​9510) (14e9dc1)
• providers: preserve AI Studio grounding metadata (#​9436) (79b6b63)
• providers: preserve raw Google stream chunks (#​9437) (2875f4e)
• providers: price unknown Grok Imagine slugs at the quality tier (#​9337) (efa689e)
• providers: redact MLflow Gateway api key and restore Novita routing test (#​9511) (9e2eaf5)
• providers: reject MiniMax function_call (#​9512) (b9a849b)
• providers: return refreshed HTTP auth tokens safely (#​9394) (9a258a5)
• providers: show labels in missing API key diagnostics (#​9404) (b43625b)
• providers: stop double-counting xAI reasoning tokens in cost (#​9326) (26f6d13)
• providers: stop double-counting xAI reasoning tokens in cost (#​9507) (7f2932c)
• secure script temporary artifacts (#​9393) (c320916)
• server: publish eval job completion only after results are saved (#​9395) (a9f5bcf)
• server: restore Azure Blob SAS tokens by value, not array position (#​9516) (b7f4a04)
• util: don't misreport a missing transitive dep as the optional package (#​9513) (e93c817)
• web-viewer: preserve config variable order instead of alphabetizing (#​9435) (a53d6f7)
• webui: avoid NaN in named metric charts (#​9352) (a6b860f)
• webui: clear row hint on eval selection (#​9380) (4586da4)
• webui: clear stale eval detail row hint (#​9377) (c9a3f61)
• webui: clear stale rowId and details hash on filter clear (#​9369) (e9b8e98)
• webui: harden eval table deep links and header borders (#​9285) (3514f6d)
• webui: make full metric chip area clickable (#​9368) (8eed102)
• webui: page eval deep links by rowId when filtered (#​9371) (16ede34)

Performance Improvements

• webui: speed up evaluation selection (#​9462) (dda3db3)

v0.121.12

Compare Source

Features

• cache: support repeat-aware fetch cache options (#​6844) (c119832)
• cli: add searchable team selector (#​9243) (247eef5)
• cli: import OpenAI Evals dashboard exports (#​9305) (70cebab)
• code-scan: add SARIF output support (#​9161) (4da26e9)
• code-scan: refine SARIF output ergonomics (#​9159) (ea3a655)
• eval: add metrics pills display toggle (#​9253) (eb7ecad)
• eval: support runtime tags (#​9322) (5a09980)
• examples: expand google adk integration (#​9047) (f190033)
• mcp: add response transforms (#​8943) (5d90bb7)
• providers: normalize OpenCode skill usage (#​9250) (8930dad)
• providers: support Agents SDK 0.9 workflows (#​9128) (0505ea1)
• redteam: support context purpose overrides (#​9260) (ca62ec9)
• telemetry: capture node runtime metadata (#​9206) (ec9d308)

Bug Fixes

• app: align table settings info buttons (#​9249) (a3252e9)
• app: improve custom metrics dialog readability (#​9248) (a3eafa4)
• app: sort custom metrics before truncation (#​9246) (74d2f9d)
• cache: include status in JSON parse errors (#​9317) (bb64b2e)
• cli: clearer errors for malformed eval imports (#​9333) (869fd54)
• code-scan: isolate action OIDC token env (#​9309) (30c99e9)
• code-scan: route logs to stderr for structured output (#​9329) (06cce72)
• code-scan: scope OIDC token to scan subprocess (#​9308) (178b57a)
• code-scan: skip unrelated default config startup (#​9230) (335d809)
• deps: avoid incompatible npm release-age config (#​9244) (b8bfb73)
• deps: bump nested ws to a non-vulnerable version (#​9330) (1fe31a4)
• deps: clear protobuf and langsmith alerts (#​9219) (d0995b8)
• deps: remove stray smithy lockfile root dependency (#​9271) (8f34641)
• deps: resolve active advisories (#​9153) (47cd609)
• deps: update anthropic packages (#​9187) (a68ed8b)
• deps: update dependency @​anthropic-ai/sdk to ^0.97.1 (#​9316) (88bc094)
• deps: update dependency @​inquirer/search to v4.1.9 (#​9277) (7a7ae06)
• deps: update dependency @​opentelemetry/exporter-trace-otlp-http to ^0.217.0 (#​9193) (082cb22)
• deps: update dependency ws to v8.20.1 [security] (#​9275) (bac18de)
• deps: update openai packages (#​9189) (f2ceff3)
• dev: use tsx watch for server (#​9301) (f0280e6)
• eval: allow header horizontal scrolling (#​9245) (4e3fb27)
• eval: preserve eval export import parity (#​9310) (ef7f98e)
• fetch: decompress fallback for compressed responses on Node 26 (#​9218) ([93f5f1e](https://redirect.gith

✂ Note

PR body was truncated to here.

---

Configuration

📅 Schedule: (in timezone Europe/London)

• Branch creation
  • "before 10am on friday"
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.

[github-actions]

Caution

[High Risk] New API server is being introduced as a directly reachable endpoint in a public subnet

The change creates a new production-api-server EC2 instance in subnet-07b5b1fb2ba02f964, which is a public subnet in the api-207c90ee VPC, and attaches it to security groups that permit external customer ingress on 443 and broad internal ingress on 9090. Its user data starts a service bound to 0.0.0.0:9090, and the plan also includes an API-server Elastic IP resource change, indicating this instance is intended to become directly reachable rather than staying behind a private load-balancing boundary.

This weakens network segmentation and creates a new directly reachable backend endpoint in the production VPC, violating the organization’s requirement that EC2 instances remain private. Internal callers will also be able to hit the new service on 9090 across 10.0.0.0/8, so traffic that should stay constrained to load balancer paths can bypass those boundaries and reach the instance directly.
_{View reasoning tree here.}

Warning

[Medium Risk] New API server is being deployed as a directly addressable EC2 endpoint with broad internal reachability

The new production API server 540044833068.eu-west-2.ec2-instance.i-025efedc46bef3be1 is being created in subnet-07b5b1fb2ba02f964, which the current environment shows is a public subnet used by internet-routable workloads, and the change also updates EIP 540044833068.eu-west-2.ec2-address.13.134.236.98 to attach to an instance. This puts the workload into a directly addressable EC2 pattern instead of keeping it behind a managed edge tier, which violates the org requirement that EC2 instances must not be directly reachable from the internet and matches SEC05-BP01/REL02-BP01 concerns.

The instance user data starts an HTTP server bound to 0.0.0.0:9090, and attached security group sg-089e5107637083db5 allows 9090/tcp from 10.0.0.0/8. Because the workload VPC participates in peering, that health endpoint becomes reachable broadly across internal and peered address space, while the instance itself is likely to have a stable public IP via the EIP attachment. This expands the attack surface and weakens network isolation even though port 9090 is not open to 0.0.0.0/0 directly.
_{View reasoning tree here.}

Warning

[Medium Risk] New production EC2 API server is created without an IAM instance profile and continues the unencrypted-EBS compute pattern

The change creates a new production EC2 API server on t4g.nano and attaches it to shared production access paths, but it does not show any explicit IAM instance profile on the instance. Current state for the same pattern already exists on 540044833068.eu-west-2.ec2-instance.i-025efedc46bef3be1: it is arm64, publicly reachable through an attached Elastic IP, and has IamInstanceProfile: null. That means this change is perpetuating a production compute pattern with no machine identity, which violates the organization’s compute hardening standard and forces any future AWS access from the instance toward ad-hoc credentials or manual access workarounds.

The data-protection issue is also concrete rather than hypothetical. The current root volumes attached to 540044833068.eu-west-2.ec2-instance.i-025efedc46bef3be1 and 540044833068.eu-west-2.ec2-instance.i-05276660d3a48893d are both unencrypted, and the organization’s policy requires all EBS data at rest to be encrypted. Because the new instance is another EBS-backed production server being attached to public and internal access infrastructure, this change continues a non-compliant compute pattern instead of correcting it. The immediate failure mode is policy and security-control failure rather than service outage, but it affects production API infrastructure and should be treated as a real medium-severity risk under SEC06-BP03 and the internal encryption requirements.
_{View reasoning tree here.}

Signals

Routine → Multiple compute and networking resources showing unusual infrequent updates at 1 event/month for the last 3 months and 1-2 events/week for the last 3-5 months, which is infrequent compared to typical patterns.
Policies → Multiple infrastructure resources showing unusual policy violations that may need review: an S3 bucket is missing required tags and does not have server-side encryption configured, while a security group allows SSH on port 22 from anywhere 0.0.0.0/0.

_{Additional Change Details:} Items 373 Edges 501 model|risks_v6 ✨Encryption Key State Risk ✨KMS Key Creation

[github-actions]

⛔ Auto-Blocked

---

🔴 Decision

Auto-blocked: Routine score (-5) is below minimum (-1)

---

📊 Signals Summary

Routine 🔴 -5

---

🔥 Risks Summary

High 0 · Medium 0 · Low 0

---

💥 Blast Radius

Items 23 · Edges 75

---

View full analysis in Overmind ↗

[github-actions]

⛔ Auto-Blocked

---

🔴 Decision

Auto-blocked: Policy signal (-3) is below threshold (-2); Routine score (-5) is below minimum (-1)

---

📊 Signals Summary

Routine 🔴 -5

Policies 🔴 -3

---

🔥 Risks Summary

High 0 · Medium 0 · Low 0

---

💥 Blast Radius

Items 5 · Edges 20

---

View full analysis in Overmind ↗

[github-actions]

⛔ Auto-Blocked

---

🔴 Decision

Found 2 high risks requiring review

---

📊 Signals Summary

Routine 🔴 -5

Policies 🔴 -3

---

🔥 Risks Summary

High 2 · Medium 0 · Low 0

---

💥 Blast Radius

Items 107 · Edges 219

---

View full analysis in Overmind ↗

[github-actions]

⛔ Auto-Blocked

---

🔴 Decision

Found 1 high risk requiring review

---

📊 Signals Summary

Routine 🔴 -5

Policies 🔴 -3

---

🔥 Risks Summary

High 1 · Medium 1 · Low 0

---

💥 Blast Radius

Items 63 · Edges 135

---

View full analysis in Overmind ↗

[github-actions]

⛔ Auto-Blocked

---

🔴 Decision

Found 2 high risks requiring review

---

📊 Signals Summary

Routine 🔴 -5

Policies 🔴 -3

---

🔥 Risks Summary

High 2 · Medium 0 · Low 0

---

💥 Blast Radius

Items 93 · Edges 217

---

View full analysis in Overmind ↗

[github-actions]

⛔ Auto-Blocked

---

🔴 Decision

Auto-blocked: Policy signal (-3) is below threshold (-2); Routine score (-5) is below minimum (-1)

---

📊 Signals Summary

Routine 🔴 -5

Policies 🔴 -3

---

🔥 Risks Summary

High 0 · Medium 0 · Low 0

---

💥 Blast Radius

Items 79 · Edges 192

---

View full analysis in Overmind ↗