fix(eval): pass simulation instructions to trajectory evaluator #5815
Chibionosintegration_tests.yml
on: pull_request
detect-changed-packages
7s
Matrix: integration-tests
summarize-results
4s
Annotations
49 errors and 46 warnings
|
simulation-testcase / alpha
Process completed with exit code 1.
|
|
eval-level-expected-output / staging
Process completed with exit code 1.
|
|
eval-level-expected-output / alpha
Process completed with exit code 1.
|
|
eval-telemetry-testcase / staging
Process completed with exit code 1.
|
|
calculator-evals / staging
Process completed with exit code 1.
|
|
calculator-crash-evals / alpha
Process completed with exit code 1.
|
|
calculator-evals / alpha
Process completed with exit code 1.
|
|
list-target-output-key-evals / alpha
Process completed with exit code 1.
|
|
tools-evals / staging
Process completed with exit code 1.
|
|
langchain-cross / alpha
Process completed with exit code 1.
|
|
langchain-cross / staging
Process completed with exit code 1.
|
|
target-output-key-evals / alpha
Process completed with exit code 1.
|
|
eval-spans-testcase / staging
Process completed with exit code 1.
|
|
apicalls-testcase / staging
Process completed with exit code 1.
|
|
target-output-key-evals / staging
Process completed with exit code 1.
|
|
basic-testcase / alpha
Process completed with exit code 1.
|
|
basic-testcase / staging
Process completed with exit code 1.
|
|
list-target-output-key-evals / staging
Process completed with exit code 1.
|
|
eval-telemetry-testcase / alpha
Process completed with exit code 1.
|
|
apicalls-testcase / alpha
Process completed with exit code 1.
|
|
simulation-testcase / staging
Process completed with exit code 1.
|
|
tools-evals / alpha
Process completed with exit code 1.
|
|
eval-spans-testcase / alpha
Process completed with exit code 1.
|
|
calculator-crash-evals / staging
Process completed with exit code 1.
|
|
calculator-crash-evals / cloud
The job has exceeded the maximum execution time of 6h0m0s
|
|
simulation-testcase / cloud
The job has exceeded the maximum execution time of 6h0m0s
|
|
apicalls-testcase / cloud
The job has exceeded the maximum execution time of 6h0m0s
|
|
calculator-crash-evals / cloud
The operation was canceled.
|
|
simulation-testcase / cloud
The operation was canceled.
|
|
apicalls-testcase / cloud
The operation was canceled.
|
|
eval-telemetry-testcase / cloud
The job has exceeded the maximum execution time of 6h0m0s
|
|
eval-telemetry-testcase / cloud
The operation was canceled.
|
|
eval-level-expected-output / cloud
The job has exceeded the maximum execution time of 6h0m0s
|
|
eval-level-expected-output / cloud
The operation was canceled.
|
|
list-target-output-key-evals / cloud
The job has exceeded the maximum execution time of 6h0m0s
|
|
list-target-output-key-evals / cloud
The operation was canceled.
|
|
basic-testcase / cloud
The job has exceeded the maximum execution time of 6h0m0s
|
|
basic-testcase / cloud
The operation was canceled.
|
|
tools-evals / cloud
The job has exceeded the maximum execution time of 6h0m0s
|
|
tools-evals / cloud
The operation was canceled.
|
|
target-output-key-evals / cloud
The job has exceeded the maximum execution time of 6h0m0s
|
|
target-output-key-evals / cloud
The operation was canceled.
|
|
langchain-cross / cloud
The job has exceeded the maximum execution time of 6h0m0s
|
|
langchain-cross / cloud
The operation was canceled.
|
|
calculator-evals / cloud
The job has exceeded the maximum execution time of 6h0m0s
|
|
calculator-evals / cloud
The operation was canceled.
|
|
eval-spans-testcase / cloud
The job has exceeded the maximum execution time of 6h0m0s
|
|
eval-spans-testcase / cloud
The operation was canceled.
|
|
summarize-results
Process completed with exit code 1.
|
|
detect-changed-packages
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, actions/setup-python@v5. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
discover-testcases
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
simulation-testcase / alpha
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
eval-level-expected-output / staging
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
eval-level-expected-output / alpha
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
calculator-evals / staging
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
eval-telemetry-testcase / staging
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
calculator-crash-evals / alpha
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
calculator-evals / alpha
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
list-target-output-key-evals / alpha
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
tools-evals / staging
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
langchain-cross / alpha
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
langchain-cross / staging
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
csv-processor-evals / alpha
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
eval-input-overrides / alpha
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
csv-processor-evals / staging
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
target-output-key-evals / alpha
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
eval-spans-testcase / staging
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
apicalls-testcase / staging
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
csv-processor-evals / cloud
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
target-output-key-evals / staging
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
eval-input-overrides / cloud
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
basic-testcase / alpha
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
basic-testcase / staging
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
list-target-output-key-evals / staging
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
eval-input-overrides / staging
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
eval-telemetry-testcase / alpha
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
apicalls-testcase / alpha
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
simulation-testcase / staging
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
tools-evals / alpha
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
eval-spans-testcase / alpha
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
eval-spans-testcase / alpha
Docker login for 'ghcr.io' failed with exit code 1, back off 3.926 seconds before retry.
|
|
calculator-crash-evals / staging
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
calculator-crash-evals / staging
Docker login for 'ghcr.io' failed with exit code 1, back off 4.333 seconds before retry.
|
|
simulation-testcase / cloud
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
calculator-crash-evals / cloud
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
apicalls-testcase / cloud
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
eval-telemetry-testcase / cloud
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
eval-level-expected-output / cloud
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
list-target-output-key-evals / cloud
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
basic-testcase / cloud
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
tools-evals / cloud
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
target-output-key-evals / cloud
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
langchain-cross / cloud
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
calculator-evals / cloud
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
eval-spans-testcase / cloud
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|