feat(benchmarks): benchmarks, scenarios and scorers added to oo-sdk

[james-rl]

User description

Description

• This adds support for benchmarks, scenarios and scorers to the OO-SDK interface that we all love

• Note that there are some asymmetries between the python and ts clients that reflect differences in how typescript handles numbers. In particular, the scorers section works around some issues with 0, NaN and undefined that don't show up in the python client

Motivation

With this change we should have parity between the ts and python clients.

Changes

• Many additions; all following the same pattern.

Testing

• Unit tests added
• Integration tests added
• Smoke Tests added/updated
• Tested locally

Breaking Changes

N/A

Checklist

• PR title follows Conventional Commits format (feat: or feat(scope):)
• Documentation updated (if needed)
• Breaking changes documented (if applicable)

---

CodeAnt-AI Description

Add object-oriented APIs for scenarios, scorers, benchmarks and runs

What Changed

• New Scenario builder and ScenarioOps: create scenarios with a fluent builder, preview configs, push to the platform, list and fetch scenarios.
• Scenario run management: start runs (async or waiting for env), access the devbox, wait for environment readiness, submit for scoring, wait for scoring, score+complete, cancel, and download logs.
• Benchmark and BenchmarkRun APIs: create/list benchmarks, start benchmark runs, list their scenario runs, cancel and complete benchmark runs.
• Scorer APIs and objects: create, list, retrieve, update and validate custom scorers; support multiple scorer types (test command, shell, bash, python, AST grep, custom) with weight normalization and preview behavior.
• SDK surface updates and exports: sdk.benchmark, sdk.scenario, sdk.scorer added and corresponding SDK classes and types exported; comprehensive unit and smoke tests added for the new features.

Impact

✅ Shorter scenario creation using a fluent builder
✅ Clearer scoring and run lifecycle (start, await env, score, complete, logs)
✅ Easier benchmark orchestration and scenario-run listing

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

[codeant-ai]

CodeAnt AI is reviewing your PR.

---

Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

[codeant-ai]

Nitpicks 🔍

🔒 No security issues identified

⚡ Recommended areas for review Invalid export syntax The file contains lines like export type * from '...'; which is not valid TypeScript syntax and will cause tsc compilation failures. These new exports must be changed to valid re-export forms (for example export * from '...'; or explicit type-only re-exports). List iteration assumption Several newly added .list() implementations (BenchmarkOps.list, ScenarioOps.list, ScorerOps.list) assume the client list response is an async iterable and use "for await (... of result)". Some API list endpoints return a page wrapper object (e.g., { items: [...] } or { devboxes: [...] }) instead of an async iterable. If the response shape differs at runtime this will throw. Verify the exact shape returned by each endpoint and handle both page-object and async-iterable shapes. Null/undefined spread risk The code spreads ...params into a new object literal. While spreading undefined is typically fine, spreading null will throw at runtime. Ensure callers cannot pass null, or guard by using a safe default (e.g. ...(params ?? {})) to avoid runtime TypeErrors. Barrel export maintenance The file adds many new named exports. Ensure the target modules ('./scenario', './scenario-run', './scenario-builder', './scorer', './benchmark', './benchmark-run') exist, are side-effect free, and the exported names match their module's exports. Missing or renamed exports will cause compile-time errors. Memory / streaming behaviour listScenarioRuns materializes all scenario runs into an array. For large benchmark runs this can use excessive memory. Consider yielding results (async generator) or adding pagination/limit parameters to avoid loading everything at once. API surface mismatch The test now asserts sdk.benchmark exists. The PR adds benchmarks/scenarios/scorers — verify the client factory (makeClientSDK) actually exposes benchmark (singular) and/or benchmarks (plural) consistently with the rest of the codebase. Inconsistencies between singular/plural names will cause brittle tests and runtime failures. Fragile undefined-arg assertions Tests assert that client methods are called with an explicit undefined for optional options. If the implementation changes to omit the parameter (call with one arg) these assertions will fail even though behavior is correct. Consider asserting calledWith/lastCalledWith more flexibly. Async iterable mocking The helper createAsyncIterable returns an object with a Symbol.asyncIterator method. If the production implementation yields paginated pages or expects a different async-iterator shape (e.g. yields arrays/pages), these tests may not accurately reflect real responses and could be brittle. Verify the mocked shape matches the API client's iterator behavior. Assumes caching behavior The test asserts that run.devbox returns the same Devbox instance on subsequent accesses. Verify that the SDK implementation actually caches the instance (not recreated each access). If implementation changes, this test will become brittle. Loose typing The mocked client is declared as any and its methods are plain jest.fn() functions. This reduces type-safety and can hide API shape mismatches between the real client and the mock. Strongly-typed mocks (e.g. jest.Mocked<Runloop> or Partial<Runloop>) would catch broken expectations earlier.

[codeant-ai]

CodeAnt AI finished reviewing your PR.

[github-actions]

⚠️ Object Smoke Tests & Coverage Report

Test Results

✅ All smoke tests passed

Coverage Results

Metric	Coverage	Required	Status
Functions	75.54%	100%	❌
Lines	71.7%	-	ℹ️
Branches	38.07%	-	ℹ️
Statements	70.03%	-	ℹ️

Coverage Requirement: 100% function coverage (all public methods must be called in smoke tests)

⚠️ Some object methods are not covered in smoke tests. Please add tests that call all public methods.

View detailed coverage report

Coverage reports are available in the workflow artifacts. Lines/branches/statements coverage is tracked but not required to be 100%.

📋 View workflow run

[james-rl]

going to split this up into smaller PRs rather than have war & peace in typescript form