feat: async + multi-source create_benchmark

[luke-e-schaefer]

Stacked on #467.

What & why

Brings the SDK's create_benchmark() in line with two server changes to POST /nucleus/benchmarks:

1. Async — the server creates the benchmark in a "building" state and streams members in via a background job, responding 202 { benchmark_id, job_id } (removes the old ~50k item cap on slice/dataset-sourced benchmarks).
2. Multi-source — members can come from any combination of item ids, slices, and datasets, unioned + de-duped server-side.

Changes

Async

• create_benchmark() POSTs, reads { benchmark_id, job_id }, and by default blocks on the build job (reusing the existing AsyncJob poller) then returns the completed "ready" benchmark. Return type unchanged, so existing blocking callers are unaffected. A failed build raises JobError (the poller already treats Errored_Server/Errored_User/etc. as terminal).
• wait_for_completion=False returns the "building" benchmark immediately for callers who poll themselves; verbose controls poll logging.
• Benchmark gains a status field ("building" / "ready" / "failed"), parsed in from_json.

Multi-source

• Adds plural slice_ids and dataset_ids params alongside the singular slice_id/dataset_id/item_ids/items. All sources combine (unioned + de-duped server-side).
• Requirement relaxed from "exactly one source" to "at least one source".

Meta

• Version → 0.20.0; CHANGELOG entry (Added: multi-source; Changed: async).
• Tests (mock-based, no live API): async poll-then-fetch flow, wait_for_completion=False, status parsing, multi-source payload, and at-least-one-source validation.

Notes

• black/pre-commit/pytest weren't run locally (not installed in this env); CI's format/lint/test jobs are the authority.

🤖 Generated with Claude Code

Greptile Summary

This PR updates create_benchmark() to match two server-side changes: the endpoint now returns 202 with {benchmark_id, job_id} while a background job builds the benchmark asynchronously, and members can come from any combination of item_ids, items, slice_id/slice_ids, and dataset_id/dataset_ids.

• Async flow: create_benchmark() now POSTs, extracts the job_id, polls via AsyncJob.sleep_until_complete (default wait_for_completion=True), then re-fetches the completed benchmark; wait_for_completion=False skips polling. Benchmark gains a status field.
• Multi-source: All six source parameters are combined server-side; the "exactly one source" restriction is replaced with "at least one source."
• Critical bug: job_id = response.get(\"job_id\") is missing from the method body — job_id is referenced but never assigned, so every default (wait_for_completion=True) call raises NameError at runtime. The wait_for_completion=False path is unaffected.

Confidence Score: 2/5

Not safe to merge: the default code path (wait_for_completion=True) raises NameError on every call because job_id is never assigned from the server response.

The method body references job_id inside the if wait_for_completion: block but never extracts it from response. Every call with the default arguments — the vast majority of production usage — will crash with NameError: name 'job_id' is not defined before any polling or benchmark fetch occurs. The wait_for_completion=False path is unaffected, but that is the minority case. The fix is a single missing line.

Files Needing Attention: nucleus/init.py — the create_benchmark method body is missing job_id = response.get("job_id"); tests/test_benchmarks.py — test_create_benchmark_from_slice_polls_then_returns_ready will also fail for the same reason.

Important Files Changed

Filename	Overview
nucleus/init.py	Adds async + multi-source support to create_benchmark(), but omits job_id = response.get("job_id") — every blocking call (wait_for_completion=True, the default) raises NameError at runtime.
nucleus/benchmark.py	Adds optional status field to the Benchmark dataclass; correctly parsed in from_json via payload.get("status").
tests/test_benchmarks.py	New async tests correctly describe the intended behavior but will fail alongside the production code due to the missing job_id assignment; wait_for_completion=False and multi-source tests would pass.
CHANGELOG.md	Adds 0.20.0 entry describing multi-source and async create_benchmark changes accurately.
pyproject.toml	Version bumped from 0.19.0 to 0.20.0.

Sequence Diagram

sequenceDiagram
    participant Caller
    participant NucleusClient
    participant Server
    participant AsyncJob

    Caller->>NucleusClient: "create_benchmark(..., wait_for_completion=True)"
    NucleusClient->>Server: POST /nucleus/benchmarks
    Server-->>NucleusClient: "202 {benchmark_id, job_id}"
    Note over NucleusClient: job_id = response.get("job_id") ← MISSING LINE
    alt "wait_for_completion=True (default)"
        NucleusClient->>AsyncJob: get_job(job_id).sleep_until_complete()
        AsyncJob-->>NucleusClient: "job complete (status=ready)"
        NucleusClient->>Server: "GET /nucleus/benchmarks/{benchmark_id}"
        Server-->>NucleusClient: "Benchmark (status="ready")"
        NucleusClient-->>Caller: "Benchmark (status="ready")"
    else "wait_for_completion=False"
        NucleusClient->>Server: "GET /nucleus/benchmarks/{benchmark_id}"
        Server-->>NucleusClient: "Benchmark (status="building")"
        NucleusClient-->>Caller: "Benchmark (status="building")"
    end

Loading

Prompt To Fix All With AI

### Issue 1
nucleus/__init__.py:1177-1185
**`job_id` is never extracted from the response — NameError on every blocking call**

`job_id` is referenced inside the `if wait_for_completion:` block but is never assigned from `response`. Any call with the default `wait_for_completion=True` raises `NameError: name 'job_id' is not defined` at line 1179 before the `ValueError` guard is ever reached. The fix is to add `job_id = response.get("job_id")` immediately after extracting `benchmark_id`. The `wait_for_completion=False` path is unaffected because the entire `if` block is skipped. The test `test_create_benchmark_from_slice_polls_then_returns_ready` would also fail for the same reason — it would hit the NameError before any mock assertions fire.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (3): Last reviewed commit: "Update nucleus/__init__.py" | Re-trigger Greptile}