Skip to content

Commit 3ae358a

Browse files
JuhaoLiang1997JuhaoLiang1997
committed
Initial commit
1 parent 542ccf9 commit 3ae358a

1,053 files changed

Lines changed: 248846 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/ISSUE_TEMPLATE/bug_report.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
---
2+
name: Bug report
3+
about: Report a problem with the benchmark runner, validator, leaderboard, or tooling
4+
title: "[Bug] <short description>"
5+
labels: bug
6+
assignees: ''
7+
---
8+
9+
## What happened?
10+
11+
<!-- A clear, concise description of the bug -->
12+
13+
## What did you expect to happen?
14+
15+
## Steps to reproduce
16+
17+
```bash
18+
# Exact commands you ran
19+
```
20+
21+
## Environment
22+
23+
| Field | Value |
24+
|---|---|
25+
| Platform | <!-- NVIDIA / AMD / Apple / Ascend --> |
26+
| OS | |
27+
| Python | |
28+
| Framework + version | |
29+
| AccelMark commit | <!-- git rev-parse --short HEAD --> |
30+
31+
## Error output
32+
33+
```
34+
# Paste the full error message or traceback here
35+
```
36+
37+
## Additional context
38+
39+
<!-- Attach relevant files (result.json, run.log, etc.) if helpful -->

‎.github/ISSUE_TEMPLATE/community_submission.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
---
2+
name: Community submission
3+
about: Submit a benchmark result to the AccelMark leaderboard
4+
title: "[Submission] <chip> — <suite> — <runner_id>"
5+
labels: community-submission
6+
---
7+
8+
<!-- Title format: [Submission] NVIDIA A100-SXM4-80GB — suite_A — nvidia_vllm_47f5d58e
9+
The CI bot reads your result.json below, validates it, and opens a PR automatically. -->
10+
11+
```json
12+
(paste the full contents of results/community/<your_dir>/result.json here)
13+
```

‎.github/ISSUE_TEMPLATE/config.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
blank_issues_enabled: false
2+
contact_links:
3+
- name: Live leaderboard
4+
url: https://juhaoliang1997.github.io/AccelMark
5+
about: View all submitted benchmark results
6+
- name: Contributing guide
7+
url: https://github.com/JuhaoLiang1997/AccelMark/blob/main/CONTRIBUTING.md
8+
about: How to run and submit a benchmark

‎.github/ISSUE_TEMPLATE/feature_request.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
---
2+
name: Feature request
3+
about: Suggest a new suite, platform support, leaderboard feature, or tooling improvement
4+
title: "[Feature] <short description>"
5+
labels: enhancement
6+
assignees: ''
7+
---
8+
9+
## What problem does this solve?
10+
11+
<!-- Describe the gap or limitation you're hitting -->
12+
13+
## Proposed solution
14+
15+
<!-- What would you like to see added or changed? -->
16+
17+
## Alternatives considered
18+
19+
<!-- Any other approaches you've thought about? -->
20+
21+
## Additional context
22+
23+
<!-- Mockups, references, related issues, etc. -->

‎.github/pull_request_template.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
## Summary
2+
3+
<!-- What does this PR do? One paragraph. -->
4+
5+
## Type of change
6+
7+
- [ ] New platform support
8+
- [ ] Bug fix (runner, validator, leaderboard, or tooling)
9+
- [ ] Suite definition change
10+
- [ ] Schema change
11+
- [ ] Leaderboard / UI improvement
12+
- [ ] Documentation
13+
- [ ] Other: <!-- describe -->
14+
15+
## Testing
16+
17+
<!-- How did you test this? What did you run? -->
18+
19+
```bash
20+
# Commands used to verify
21+
```
22+
23+
## Checklist
24+
25+
- [ ] I have read [CONTRIBUTING.md](../CONTRIBUTING.md)
26+
- [ ] My change does not break existing `result.json` files (or I have explained the migration path)
27+
- [ ] If adding a new platform: runner inherits from `BenchmarkRunner`, produces valid `result.json`, includes a reference result
28+
- [ ] If changing the schema: `validate_submission.py` updated and all existing results still validate
29+
- [ ] If changing the leaderboard generator: `leaderboard/generate.py` produces correct output on existing results
30+
- [ ] I have updated relevant documentation
31+
32+
## Related issues
33+
34+
<!-- Closes #<number> -->

‎.github/workflows/generate_leaderboard.yml‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
name: Generate Leaderboard
2+
3+
# Triggers only when results/ changes land on main.
4+
# This covers both direct merges and squash merges.
5+
on:
6+
push:
7+
branches:
8+
- main
9+
paths:
10+
- 'results/**'
11+
12+
# Allow manual trigger from Actions tab (useful for first deploy)
13+
workflow_dispatch:
14+
15+
jobs:
16+
validate-runners:
17+
name: Validate runner folders
18+
runs-on: ubuntu-latest
19+
steps:
20+
- uses: actions/checkout@v4
21+
22+
- uses: actions/setup-python@v5
23+
with:
24+
python-version: "3.11"
25+
cache: pip
26+
27+
- name: Install dependencies
28+
run: pip install jsonschema
29+
30+
- name: Validate all runner meta.json files and hashes
31+
run: python runners/validate_runners.py
32+
33+
generate:
34+
name: Generate and deploy leaderboard
35+
runs-on: ubuntu-latest
36+
needs: [validate-runners]
37+
permissions:
38+
contents: write # needed to push to gh-pages branch
39+
pages: write
40+
id-token: write
41+
42+
steps:
43+
- name: Checkout main
44+
uses: actions/checkout@v4
45+
46+
- name: Set up Python 3.11
47+
uses: actions/setup-python@v5
48+
with:
49+
python-version: '3.11'
50+
51+
- name: Install dependencies
52+
run: |
53+
pip install jsonschema --quiet
54+
55+
- name: Generate leaderboard
56+
run: |
57+
python leaderboard/generate.py
58+
59+
- name: Verify output files exist
60+
run: |
61+
echo "Checking generated files..."
62+
ls -la leaderboard/site/
63+
ls -la leaderboard/site/api/ || echo "Warning: api/ directory not generated"
64+
65+
# Fail if leaderboard.js is missing
66+
if [ ! -f "leaderboard/site/leaderboard.js" ]; then
67+
echo "ERROR: leaderboard.js was not generated"
68+
exit 1
69+
fi
70+
71+
echo "Generated files:"
72+
wc -l leaderboard/site/leaderboard.js
73+
cat leaderboard/site/api/chips.json 2>/dev/null || echo "(no chips yet)"
74+
75+
- name: Deploy to GitHub Pages
76+
uses: peaceiris/actions-gh-pages@v3
77+
with:
78+
github_token: ${{ secrets.GITHUB_TOKEN }}
79+
publish_dir: ./leaderboard/site
80+
publish_branch: gh-pages
81+
# Keep existing files on gh-pages (don't wipe non-generated files)
82+
keep_files: false
83+
# Commit message
84+
commit_message: "leaderboard: auto-update from ${{ github.sha }}"
85+
# Add a .nojekyll file so GitHub Pages serves files as-is
86+
enable_jekyll: false
87+
88+
- name: Print leaderboard URL
89+
run: |
90+
OWNER="${{ github.repository_owner }}"
91+
REPO="${{ github.event.repository.name }}"
92+
echo "Leaderboard deployed to: https://${OWNER}.github.io/${REPO}/"
93+
echo "API endpoint: https://${OWNER}.github.io/${REPO}/api/index.json"

0 commit comments

Comments
 (0)