feat(sdk): add ScenarioBuilder class to OO-SDK#748
Conversation
|
CodeAnt AI is reviewing your PR. Thanks for using CodeAnt! 🎉We're free for open-source projects. if you're enjoying it, help us grow by sharing. Share on X · |
sid-rl
changed the title
ScenarioBuilder class to OO-SDKScenarioBuilder class to OO-SDK
Sequence DiagramThis PR adds a new fluent builder flow for creating scenarios from the OO SDK. Users can now start from sequenceDiagram
participant User
participant ScenarioOps
participant ScenarioBuilder
participant ScenariosAPI
participant Scenario
User->>ScenarioOps: builder scenario name
ScenarioOps-->>User: ScenarioBuilder instance
User->>ScenarioBuilder: Configure problem environment and scorers
User->>ScenarioBuilder: push
ScenarioBuilder->>ScenariosAPI: Create scenario with built params
ScenariosAPI-->>ScenarioBuilder: Created scenario id
ScenarioBuilder-->>User: Scenario object from id
Generated by CodeAnt AI |
codeant-ai
Bot
added
the
size:XL
Nitpicks 🔍
|
![codeant-ai[bot]](https://avatars.githubusercontent.com/in/646884?s=60&v=4){kind=small}
| if (weight <= 0) { | ||
| throw new Error(`Scorer weight must be positive, got ${weight}`); |
There was a problem hiding this comment.
Suggestion: addScorer only checks weight <= 0, which allows non-finite values like NaN and Infinity. Those values pass the check but break normalization in build() (s.weight / totalWeight), producing invalid weights (NaN) that will cause malformed scoring payloads and API failures. Validate that weight is finite before accepting it. [logic error]
Severity Level: Major ⚠️
- ❌ Scenario creation payload can contain null scorer weights.
- ⚠️ `runloop.scenario.builder(...).push()` may fail API validation.| if (weight <= 0) { | |
| throw new Error(`Scorer weight must be positive, got ${weight}`); | |
| if (!Number.isFinite(weight) || weight <= 0) { | |
| throw new Error(`Scorer weight must be a finite positive number, got ${weight}`); |
Steps of Reproduction ✅
1. Use the public OO-SDK entrypoint `ScenarioOps.builder()` at `src/sdk.ts:1988-1990` to
create a `ScenarioBuilder` instance (normal scenario creation flow exposed to users).
2. Add a scorer with a non-finite computed weight (for example `1/0` or `0/0`) through
`addTestCommandScorer()` at `src/sdk/scenario-builder.ts:156-170`, which forwards to
`addScorer()` at `src/sdk/scenario-builder.ts:437-442`.
3. In `addScorer()` (`src/sdk/scenario-builder.ts:438`), `if (weight <= 0)` does not
reject `NaN`/`Infinity`, so invalid weight is stored in `_scorers`.
4. Call `push()` (`src/sdk/scenario-builder.ts:431-434`) or `build()` (`:366-408`):
normalization computes `weight: s.weight / totalWeight` (`:378-382`), then request
serialization in `src/core.ts:335` uses `JSON.stringify`, which serializes non-finite
numbers to `null`, producing invalid scorer weights in `/v1/scenarios` create payload
(`src/resources/scenarios/scenarios.ts:37-39`).Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** src/sdk/scenario-builder.ts
**Line:** 438:439
**Comment:**
*Logic Error: `addScorer` only checks `weight <= 0`, which allows non-finite values like `NaN` and `Infinity`. Those values pass the check but break normalization in `build()` (`s.weight / totalWeight`), producing invalid weights (`NaN`) that will cause malformed scoring payloads and API failures. Validate that weight is finite before accepting it.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
|
CodeAnt AI finished reviewing your PR. |
✅ Object Smoke Tests & Coverage ReportTest Results✅ All smoke tests passed Coverage Results
Coverage Requirement: 100% function coverage (all public methods must be called in smoke tests) ✅ All tests passed and all object methods are covered! View detailed coverage report
|
a4e4e2f to
e6202b8
Compare
james-rl
left a comment
james-rl
left a comment
There was a problem hiding this comment.
This looks good to me -- I see there are some smoke tests failing but I don't think they're caused by changes in this code. Still, you should make sure they pass before merging this.
e6202b8 to
7017e8e
Compare
|
CodeAnt AI is running Incremental review Thanks for using CodeAnt! 🎉We're free for open-source projects. if you're enjoying it, help us grow by sharing. Share on X · |
codeant-ai
Bot
added
size:XL
| if (!Number.isFinite(weight) || weight <= 0) { | ||
| throw new Error(`Scorer weight must be a finite positive number, got ${weight}`); | ||
| } | ||
| this._scorers.push({ name, weight, scorer }); |
There was a problem hiding this comment.
Suggestion: Scorer names are pushed without enforcing the API contract ([a-zA-Z0-9_-] only), so invalid names pass build() and fail later during push() with a server-side validation error. Add local validation in addScorer to fail fast with a clear message. [logic error]
Severity Level: Major ⚠️
- ⚠️ push() may fail with server validation error.
- ⚠️ Scenario creation flow gets late, unclear failure.| this._scorers.push({ name, weight, scorer }); | |
| if (!/^[a-zA-Z0-9_-]+$/.test(name)) { | |
| throw new Error('Scorer name must only contain [a-zA-Z0-9_-]'); | |
| } |
Steps of Reproduction ✅
1. Call SDK entrypoint `ScenarioOps.builder(name)` in `src/sdk.ts:1988`, which returns
`new ScenarioBuilder(name, this.client)`.
2. Add a scorer with an invalid name (for example `bad name`) through
`addShellCommandScorer()` in `src/sdk/scenario-builder.ts:182`, which forwards to
`addScorer()` at `src/sdk/scenario-builder.ts:178`.
3. Observe `addScorer()` only validates weight at `src/sdk/scenario-builder.ts:179-181`
and pushes name unchanged at `src/sdk/scenario-builder.ts:182`.
4. Call `build()` (`src/sdk/scenario-builder.ts:107`) and see invalid `name` included in
`scoring_function_parameters`.
5. This violates the documented API contract in `src/resources/scenarios/scenarios.ts:16`
("Names must only contain [a-zA-Z0-9_-]"), so failure is deferred to `push()` API call
path (`src/sdk/scenario-builder.ts:172-175` -> `client.scenarios.create`).Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** src/sdk/scenario-builder.ts
**Line:** 441:441
**Comment:**
*Logic Error: Scorer names are pushed without enforcing the API contract (`[a-zA-Z0-9_-]` only), so invalid names pass `build()` and fail later during `push()` with a server-side validation error. Add local validation in `addScorer` to fail fast with a clear message.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
|
CodeAnt AI Incremental review completed. |
2 participants
CodeAnt-AI Description
Add a fluent builder for creating scenarios in the SDK
What Changed
Impact
✅ Shorter scenario setup✅ Fewer failed scenario creations✅ Clearer scenario configuration errors💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.