Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
4ead78c
vibed CI benchmark bot
witherrrr May 15, 2026
bae0239
increase sccache timeout skull
witherrrr May 15, 2026
0c17574
bump to GCC14
witherrrr May 15, 2026
80a977b
clarify CI name
witherrrr May 15, 2026
b62e1ed
remove hyperfine warmup
witherrrr May 15, 2026
9160338
parallelize, revamp bench CI
witherrrr May 15, 2026
912d451
oops, fix CI
witherrrr May 15, 2026
9b5198f
fix SVG rendering
witherrrr May 15, 2026
715da34
put CI on arm for speed?
witherrrr May 15, 2026
0b9579a
sort names
witherrrr May 15, 2026
84b6b48
clean up report
witherrrr May 15, 2026
586821e
svg too big go back
witherrrr May 16, 2026
2dddf8b
add wpt rendering
witherrrr May 16, 2026
0dae51d
move CI renders to hidden ref
witherrrr May 16, 2026
29e6575
clean, speed up CI
witherrrr May 16, 2026
72ab3c5
bench timeout
witherrrr May 16, 2026
260873f
Revert "bench timeout"
witherrrr May 16, 2026
e112704
new bench timeouts
witherrrr May 16, 2026
81b1302
update CI report to clarify run cnt
witherrrr May 16, 2026
702edbd
way more test benches
witherrrr May 16, 2026
6db5b16
global variant/dt tables
witherrrr May 16, 2026
937a5b4
CI bench header improvement
witherrrr May 16, 2026
4207b8c
rework CI bench parallelism
witherrrr May 16, 2026
da44cd0
MORE CI PARALLELISM
witherrrr May 18, 2026
6af9ab5
[trajoptlib] Rough draft of more accurate motor model
spacey-sooty Mar 28, 2026
0612e52
tmp
spacey-sooty Apr 5, 2026
95dc4f8
new formulation from discord, need to integrate into choreo
spacey-sooty Apr 14, 2026
1711672
some ui progress
spacey-sooty Apr 21, 2026
ae61d88
latest changes
spacey-sooty Apr 21, 2026
48a53a9
rumours of a functional ui, nonfinite initial guess errors tho
spacey-sooty Apr 22, 2026
d5aaef0
oops
spacey-sooty Apr 22, 2026
207966a
Merge pull request #1 from witherrrr/motor-modelling
witherrrr Apr 26, 2026
19f9694
proper units pass through
spacey-sooty Apr 26, 2026
120aa1b
wheel scrub penalization
witherrrr Apr 27, 2026
70c0be8
try linear scrub
witherrrr Apr 27, 2026
97376c4
Revert "try linear scrub"
witherrrr Apr 27, 2026
a1a50e3
better c_scrub value
witherrrr Apr 27, 2026
64da101
formulation cleanup faster gen
witherrrr Apr 27, 2026
9886a03
use tire slip angle formulation
witherrrr Apr 28, 2026
a90a28a
add brake force constraint
witherrrr May 8, 2026
bb186d9
re-formulate for solver robustness
witherrrr May 8, 2026
3b86942
fix: init guess inflated samp_cnt
witherrrr May 9, 2026
6e80303
min-width anti-tunneling fix
witherrrr May 9, 2026
094eff2
feat: add kS to model (works!)
witherrrr May 9, 2026
9577e68
lint, add comments
witherrrr May 11, 2026
574ffbb
lint, add comments
witherrrr May 11, 2026
2c66a48
add comments
witherrrr May 11, 2026
1efccb3
update test chors
witherrrr May 16, 2026
bf83b3c
try initial force guess
witherrrr May 16, 2026
55bda4c
try exponential profile
witherrrr May 16, 2026
8f799da
perf: factor heading trig out of net-torque sum
witherrrr May 16, 2026
0953d21
change default dt to 0.02
spacey-sooty May 31, 2026
2a5c78e
fmt
spacey-sooty May 31, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
279 changes: 279 additions & 0 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,279 @@
name: Solver Time Benchmark

# Comment-triggered: a maintainer (OWNER / MEMBER / COLLABORATOR) types
# `/bench` on a PR. The work is split across parallel jobs to cut wall-clock:
#
# resolve ─ gate + ack + resolve PR head/base SHAs
# │
# ├─ build (matrix: pr | base) ── two self-contained choreo-cli binaries
# │ built concurrently, uploaded as artifacts
# │
# └─ bench (matrix: one job per project variant) ── each downloads both
# binaries, benches ONLY its variant, renders that variant's
# SVGs, uploads a per-variant artifact
# │
# collate ─ merges every per-variant artifact, pushes the rendered SVGs to a
# per-run CUSTOM ref `refs/bench-renders/<run_id>` (a hidden ref,
# not a branch/tag, so it never shows in the Branches/Tags UI) and
# links them by commit SHA so they're viewable in the browser via
# GitHub's blob viewer (not just a ZIP), builds the comparison
# report, uploads the consolidated artifact, posts a fresh PR comment

on:
issue_comment:
types: [created]

permissions:
pull-requests: write
contents: write # collate pushes trajectory SVGs to a per-run refs/bench-renders/* custom ref
actions: read

jobs:
resolve:
# Only run when:
# - the comment is on a PR (not a plain issue)
# - the body starts with "/bench"
# - the commenter has write access (gates external drive-by triggers)
if: >-
github.event.issue.pull_request &&
startsWith(github.event.comment.body, '/bench') &&
(github.event.comment.author_association == 'OWNER' ||
github.event.comment.author_association == 'MEMBER' ||
github.event.comment.author_association == 'COLLABORATOR')
runs-on: ubuntu-slim
outputs:
head_sha: ${{ steps.pr.outputs.head_sha }}
base_sha: ${{ steps.pr.outputs.base_sha }}
head_repo: ${{ steps.pr.outputs.head_repo }}
variants: ${{ steps.variants.outputs.variants }}
steps:
- name: Acknowledge the trigger comment
uses: peter-evans/create-or-update-comment@v4
with:
comment-id: ${{ github.event.comment.id }}
reactions: rocket

- name: Resolve PR head + base
id: pr
uses: actions/github-script@v7
with:
script: |
const pr = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.issue.number,
});
core.setOutput('head_sha', pr.data.head.sha);
core.setOutput('base_sha', pr.data.base.sha);
core.setOutput('head_repo', pr.data.head.repo.full_name);

- name: Checkout test-projects + enumerate script
uses: actions/checkout@v6
with:
repository: ${{ steps.pr.outputs.head_repo }}
ref: ${{ steps.pr.outputs.head_sha }}
persist-credentials: false
sparse-checkout: |
test-projects
scripts/ci/enumerate-variants.sh
sparse-checkout-cone-mode: false

- name: Enumerate project variants
id: variants
run: bash scripts/ci/enumerate-variants.sh

# --------------------------------------------------------------------------
# build choreo-cli for both PR and base concurrently
build:
needs: resolve
runs-on: ubuntu-24.04-arm
strategy:
fail-fast: false
matrix:
side: [pr, base]
steps:
- name: Checkout PR head
uses: actions/checkout@v6
with:
repository: ${{ needs.resolve.outputs.head_repo }}
ref: ${{ needs.resolve.outputs.head_sha }}
fetch-depth: 0
persist-credentials: false

# For the base binary, overlay the base branch's Rust crates on top of
# the PR checkout (everything else — scripts, test-projects — is unused
# by the build itself).
- name: Overlay base sources
if: matrix.side == 'base'
run: |
git fetch origin ${{ needs.resolve.outputs.base_sha }}
git checkout ${{ needs.resolve.outputs.base_sha }} -- \
src-core src-cli trajoptlib Cargo.lock Cargo.toml

- name: Make GCC 14 the default toolchain
run: |
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-14 200
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-14 200

- name: Set up sccache
uses: mozilla-actions/sccache-action@v0.0.10

- name: Build choreo-cli (${{ matrix.side }})
# if base is schema-incompatible continue anyways
continue-on-error: ${{ matrix.side == 'base' }}
run: cargo build --release -p choreo-cli
env:
RUSTC_WRAPPER: sccache
SCCACHE_GHA_ENABLED: "true"
SCCACHE_STARTUP_TIMEOUT: "600"

- name: Upload binary
if: always()
uses: actions/upload-artifact@v7
with:
name: cli-${{ matrix.side }}
path: target/release/choreo-cli
if-no-files-found: warn

# --------------------------------------------------------------------------
# parallelize bench for each project
# not parallelized further to avoid variance in HW causing noisy comparisons
bench:
needs: [resolve, build]
runs-on: ubuntu-24.04-arm
strategy:
fail-fast: false
matrix:
variant: ${{ fromJSON(needs.resolve.outputs.variants) }}
env:
VARIANT: ${{ matrix.variant }}
steps:
- name: Checkout PR head
uses: actions/checkout@v6
with:
repository: ${{ needs.resolve.outputs.head_repo }}
ref: ${{ needs.resolve.outputs.head_sha }}
persist-credentials: false

- name: Download PR binary
uses: actions/download-artifact@v7
with:
name: cli-pr
path: bin-pr

- name: Download base binary
id: dl_base
continue-on-error: true
uses: actions/download-artifact@v7
with:
name: cli-base
path: bin-base

- name: Make binaries executable
run: |
chmod +x bin-pr/choreo-cli
[ -f bin-base/choreo-cli ] && chmod +x bin-base/choreo-cli || true

- name: Stage variant
run: |
mkdir -p tp-pr tp-base
cp -r "test-projects/$VARIANT" "tp-pr/$VARIANT"
cp -r "test-projects/$VARIANT" "tp-base/$VARIANT"

- name: Run PR benchmark
continue-on-error: true
timeout-minutes: 60
run: bash scripts/run-bench.sh "$PWD/bin-pr/choreo-cli" tp-pr out-pr "$VARIANT"


- name: Run base benchmark
continue-on-error: true # tolerate missing / schema-incompatible base
timeout-minutes: 60
run: |
if [ -x bin-base/choreo-cli ]; then
bash scripts/run-bench.sh "$PWD/bin-base/choreo-cli" tp-base out-base "$VARIANT"
else
echo "no base binary available — skipping base run for $VARIANT"
fi

- name: Render trajectory SVGs with linear acceleration coloring
run: bash scripts/ci/render-variant.sh

- name: Upload per-variant artifact
if: always()
uses: actions/upload-artifact@v7
with:
name: bench-${{ matrix.variant }}
path: |
out-pr/**
out-base/**
tp-pr/**
tp-base/**
if-no-files-found: warn

# --------------------------------------------------------------------------
collate:
needs: [resolve, bench]
runs-on: ubuntu-slim
steps:
- name: Checkout PR head
uses: actions/checkout@v6
with:
repository: ${{ needs.resolve.outputs.head_repo }}
ref: ${{ needs.resolve.outputs.head_sha }}
persist-credentials: false
sparse-checkout: scripts
sparse-checkout-cone-mode: false

- name: Merge per-variant artifacts
continue-on-error: true # if every bench job was skipped, still comment
uses: actions/download-artifact@v7
with:
pattern: bench-*
path: merged
merge-multiple: true

- name: Publish renders to a ref
id: renders
continue-on-error: true # a missing renders link must not block the comment
uses: actions/github-script@v7
env:
HEAD_SHA: ${{ needs.resolve.outputs.head_sha }}
with:
script: |
const publish = require(`${process.env.GITHUB_WORKSPACE}/scripts/ci/publish-renders.cjs`)
await publish({ github, context, core })

- name: Build markdown report
env:
ARTIFACT_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
HEAD_SHA: ${{ needs.resolve.outputs.head_sha }}
RENDERS_URL: ${{ steps.renders.outputs.url }}
run: |
node scripts/bench-report.mjs \
--pr merged/tp-pr --base merged/tp-base \
--pr-reports merged/out-pr --base-reports merged/out-base \
--artifact-url "$ARTIFACT_URL" \
${RENDERS_URL:+--renders-url "$RENDERS_URL"} \
--commit "${HEAD_SHA:0:7}" \
--out report.md
cat report.md

- name: Upload consolidated bench artifact
if: always()
uses: actions/upload-artifact@v7
with:
name: bench-output-${{ github.run_id }}
path: |
merged/out-pr/**
merged/out-base/**
merged/tp-pr/**/*.svg
merged/tp-base/**/*.svg
report.md
if-no-files-found: warn

- name: Post benchmark comment on PR
uses: peter-evans/create-or-update-comment@v4
with:
issue-number: ${{ github.event.issue.number }}
body-path: report.md
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -54,3 +54,6 @@ CMakeUserPresets.json
build/
cli/
test-tmp*/

# Local PR-benchmark output (scripts/bench-local.sh)
bench-local/
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading