Add a credential-file template and document how to run benchmarks concurrently

Concurrency has two independent constraints and separate keys only address one of
them.

Provider capacity is the one that bites, since the evaluation scope is nearly all
the traffic (~2.9M metered TPM per run against the optimizer's ~4k). Three
benchmarks share fireworks_ai/deepseek-v4-flash, swe-atlas-qna is Fireworks on a
different model, and gaia is the only non-Fireworks target -- so gaia pairs with
anything at near-zero contention while the deepseek trio stacks on one provider.
The single 502 upstream_error observed so far (1 in 4750 requests at 24 slots) was
Fireworks.

Per-key limits are the other: a distinct --env-file per concurrent run keeps a
per-key TPM/RPM throttle on one from starving another and keeps spend
attributable. Separate keys against the same provider account create no new
provider capacity. Modal sandbox capacity is a third ceiling again.

secrets.env.example documents the naming trap: the root .gitignore covers
secrets.env and *.secrets.env, so `fireworks-a.secrets.env` is protected but
`secrets.2.env` is not -- verified both ways.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: varunursekar

harness-engineering-bench/CONFIGURATION.md

@@ -281,7 +281,27 @@ share a pool. `max_requests` is 200 000 on both everywhere (a full officeqa
 finalize needs ~12 000, so this is not the binding constraint either). Sizing is
 per benchmark from its own case counts; see each `build.yaml` for the arithmetic.
 
-§ `max_concurrency` is cases in flight per evaluation, raised 8 → 24 across the
+§ **Running benchmarks concurrently: pair them by target provider, and give each
+run its own credential file.** These are two different constraints and each
+solves a different problem.
+
+*Provider capacity* is the one that actually bites, because the evaluation scope
+is the overwhelming majority of traffic (~2.9 M metered TPM per run vs the
+optimizer's ~4 k). Three benchmarks share `fireworks_ai/deepseek-v4-flash`
+(officeqa, tau3, browsecomp-plus), swe-atlas-qna is Fireworks-but-a-different-model
+(`gpt-oss-120b`), and gaia is the only non-Fireworks target (`gpt-5.4-mini`). So
+**gaia pairs with any of the others at near-zero contention**, while running the
+deepseek trio together stacks demand on one provider. The one `502 upstream_error`
+we have observed (1 in 4 750 requests at 24 slots) came from Fireworks.
+
+*Per-key limits* are separate: a distinct `--env-file` per concurrent run means a
+per-key TPM/RPM throttle on one cannot starve another, and per-workstream spend
+stays attributable. Copy `secrets.env.example` to a `*.secrets.env` name (that
+glob is gitignored; `secrets.2.env` is **not**). Separate keys against the same
+provider account do **not** create new provider capacity — only the pairing above
+does that. Modal sandbox capacity is a third, independent ceiling.
+
+`max_concurrency` is cases in flight per evaluation, raised 8 → 24 across the
 board. It is the throughput lever: finalize wall time is
 `ceil(trials / max_concurrency) × mean case wall`, so officeqa's ~4.1 h finalize
 becomes ~1.4 h. Headroom is measured, not assumed — officeqa run #2 sustained

harness-engineering-bench/secrets.env.example

@@ -0,0 +1,32 @@
+# Template for a run's credential file. Copy to a name matching *.secrets.env
+# (root .gitignore covers `secrets.env` and `*.secrets.env`; a name like
+# `secrets.2.env` would NOT be ignored) and fill in real values:
+#
+#     cp harness-engineering-bench/secrets.env.example vero/fireworks-a.secrets.env
+#     vero harbor run ... --env-file fireworks-a.secrets.env
+#
+# Why more than one: a distinct file per concurrent run gives each its own
+# upstream key, so a per-key TPM/RPM limit on one run cannot throttle another,
+# and per-workstream spend stays separately attributable. Note what it does NOT
+# buy: separate keys against the same provider account do not create new provider
+# capacity. Use the target-model split for that — see the concurrency note in
+# CONFIGURATION.md.
+#
+# Never commit a filled-in copy. Every value here is a live credential.
+
+# Upstream inference the gateway proxies to. The optimizer and the target agent
+# never see this key; the gateway mints per-scope tokens instead.
+OPENAI_API_KEY=
+OPENAI_BASE_URL=
+
+# Modal runs the inner evaluation sandboxes. Parallel runs share one Modal
+# workspace unless these differ, and sandbox capacity is its own ceiling
+# separate from any inference quota.
+MODAL_TOKEN_ID=
+MODAL_TOKEN_SECRET=
+
+# Telemetry. One key is fine across concurrent runs — they are distinguished by
+# `--param wandb_run=`, all landing in the shared harness-engineering-bench
+# project.
+WANDB_API_KEY=
+WANDB_BASE_URL=