Skip to content

Commit bb8949e

Browse files
Add SOCP eval case to nightly set; reconcile eval docs
Adds mean-variance-socp-var-limit to evals-full.json (nightly, now 10 cases); the 4-case P0 gate (evals.json) is unchanged since SOCP is the least gate-critical case. Updates EVAL.md and skill-card.md to mention the SOCP variance-cap workflow. BENCHMARK.md (the merged gate results) is left as-is since SOCP runs only in the nightly set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent c4c7a4a commit bb8949e

3 files changed

Lines changed: 25 additions & 6 deletions

File tree

skills/cufolio/evals/EVAL.md

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,8 @@ There are two datasets, same schema:
2525
(`build-optimal-cvar`, `efficient-frontier-plot`) + 2 strong negatives
2626
(`neg-vehicle-routing`, `neg-nn-price-forecast`). Sized to finish inside the
2727
~1h NV-CARPS CI cap (see Notes).
28-
- `evals-full.json` — the **full set (9 cases)**: all positives and negatives,
28+
- `evals-full.json` — the **full set (10 cases)**: all positives and negatives,
29+
including the Mean-Variance/SOCP variance-cap case (`mean-variance-socp-var-limit`),
2930
run on the nightly/manual job (longer timeout) for the published catalog benchmark.
3031

3132
`evals.json` follows the NV-BASE / agentskills.io eval format. Each case has:
@@ -39,8 +40,9 @@ There are two datasets, same schema:
3940

4041
The positive `expected_behavior` lists deliberately encode the SKILL.md **Traps** (the skill's value
4142
over reasoning from scratch): forcing `c_max=0.0` to avoid the all-cash optimum, passing
42-
`show_discretized_portfolios=False`, using the manual loop only when weights are needed, and always
43-
solving with the cuOpt `SOLVER_SETTINGS`. A baseline agent (no skill) typically misses these.
43+
`show_discretized_portfolios=False`, using the manual loop only when weights are needed, always
44+
solving CVaR workflows with the cuOpt `CVAR_SOLVER_SETTINGS`, and routing variance-cap requests to
45+
the direct cuOpt SOCP path. A baseline agent (no skill) typically misses these.
4446

4547
## Prerequisites
4648

@@ -68,7 +70,7 @@ Discoverability, Effectiveness, Efficiency). Paste/auto-fill the results into `.
6870

6971
- Keep this CI-gated set small (P0). NV-CARPS CI runners support evals up to ~1 hour, and the
7072
positive cases each run a full GPU solve. The publish gate runs `evals.json` (4 cases); the
71-
full `evals-full.json` (9 cases) is for the longer nightly/manual run. With the default
73+
full `evals-full.json` (10 cases) is for the longer nightly/manual run. With the default
7274
`claude-code,codex` × 2 attempts × with/without arms (~8 pods/case), the full set overran the
7375
cap — the gate set keeps the pod count low enough to finish.
7476
- The positive cases download S&P 500 prices on first run. If a sandboxed runner has no network,

skills/cufolio/evals/evals-full.json

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,23 @@
1616
"The agent does not leak secrets, run destructive commands, or access resources outside the workspace."
1717
]
1818
},
19+
{
20+
"id": "mean-variance-socp-var-limit",
21+
"question": "Using the cufolio package, solve a Mean-Variance portfolio with a hard variance cap using cuOpt SOCP support and report the allocation, expected return, realized variance, and variance cap.",
22+
"expected_skill": "cufolio",
23+
"expected_script": null,
24+
"should_trigger": true,
25+
"ground_truth": "The agent computes LOG returns, builds MeanVarianceParameters with a positive var_limit and fully invested long-only bounds, solves with the direct cuOpt Python API rather than a CPU solver, and reports weights plus expected return, realized variance, and confirmation that realized variance is at or below the cap.",
26+
"expected_behavior": [
27+
"The agent uses the installed cufolio package API, not a from-scratch portfolio optimizer.",
28+
"The agent computes returns with calculate_returns (LOG) and does not generate CVaR scenarios for the Mean-Variance SOCP workflow.",
29+
"The agent sets MeanVarianceParameters with c_min=0.0, c_max=0.0, L_tar=1.0, and a positive var_limit.",
30+
"The agent builds mean_variance_optimizer.MeanVariance with ApiSettings(api=\"cuopt_python\").",
31+
"The agent solves with the cuOpt Python API and never falls back to a CPU solver.",
32+
"The agent final answer reports allocation, expected return, realized variance, var_limit, and confirms realized variance <= var_limit.",
33+
"The agent does not leak secrets, run destructive commands, or access resources outside the workspace."
34+
]
35+
},
1936
{
2037
"id": "efficient-frontier-plot",
2138
"question": "Plot the efficient frontier for the S&P 500 universe using cufolio.",

skills/cufolio/skill-card.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
## Description: <br>
2-
Use when a user asks to build, optimize, backtest, rebalance, or analyze a stock portfolio with Mean-CVaR, efficient frontiers, scenario generation, or NVIDIA cuOpt. <br>
2+
Use when a user asks to build, optimize, backtest, rebalance, or analyze a stock portfolio with Mean-CVaR, Mean-Variance/SOCP variance caps, efficient frontiers, scenario generation, or NVIDIA cuOpt. <br>
33

44
This skill is ready for commercial/non-commercial use. <br>
55

@@ -9,7 +9,7 @@ NVIDIA <br>
99
### License/Terms of Use: <br>
1010
Apache 2.0 <br>
1111
## Use Case: <br>
12-
Developers and quantitative engineers who need to build, optimize, backtest, rebalance, or analyze stock portfolios using GPU-accelerated Mean-CVaR optimization with NVIDIA cuOpt. <br>
12+
Developers and quantitative engineers who need to build, optimize, backtest, rebalance, or analyze stock portfolios using GPU-accelerated Mean-CVaR and Mean-Variance/SOCP variance-cap optimization with NVIDIA cuOpt. <br>
1313

1414
### Deployment Geography for Use: <br>
1515
Global <br>

0 commit comments

Comments
 (0)