-
Notifications
You must be signed in to change notification settings - Fork 0
Add remote fuzzing documentation and AI agent skill #12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
F-WRunTime
wants to merge
7
commits into
main
Choose a base branch
from
feature/fuzzing-docs-and-skill
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
04cf604
Add remote fuzzing documentation and AI agent skill
F-WRunTime d734051
Merge branch 'main' into feature/fuzzing-docs-and-skill
F-WRunTime ea99719
Add cross-platform AI agent knowledge files and setup guide
F-WRunTime 0baef58
Add Jekyll + GitHub Pages workflow; refresh docs; remove Cursor skills
F-WRunTime 0afafa5
docs: address PR #12 review (typos, modes, fuzz flags, AGENTS example)
F-WRunTime 547d44d
documenting how to use kaas-cli and foundry/forge to run tests
F-WRunTime 079bb0d
docs: address PR review (headings, images, multi-agent guide)
F-WRunTime File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| # Build and deploy this repository to GitHub Pages (Jekyll). | ||
| # Repository name can be anything: set Pages source to "GitHub Actions" in repo Settings → Pages. | ||
| name: Deploy Jekyll to GitHub Pages | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - main | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
| pages: write | ||
| id-token: write | ||
|
|
||
| concurrency: | ||
| group: pages | ||
| cancel-in-progress: false | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup Ruby | ||
| uses: ruby/setup-ruby@v1 | ||
| with: | ||
| ruby-version: "3.3.6" | ||
| bundler-cache: true | ||
|
|
||
| - name: Setup Pages | ||
| id: pages | ||
| uses: actions/configure-pages@v5 | ||
|
|
||
| - name: Build with Jekyll | ||
| run: bundle exec jekyll build --baseurl "${{ steps.pages.outputs.base_path }}" | ||
| env: | ||
| JEKYLL_ENV: production | ||
|
|
||
| - name: Upload artifact | ||
| uses: actions/upload-pages-artifact@v3 | ||
| with: | ||
| path: _site | ||
|
|
||
| deploy: | ||
| runs-on: ubuntu-latest | ||
| needs: build | ||
| environment: | ||
| name: github-pages | ||
| url: ${{ steps.deployment.outputs.page_url }} | ||
| steps: | ||
| - name: Deploy to GitHub Pages | ||
| id: deployment | ||
| uses: actions/deploy-pages@v4 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| _site/ | ||
| .sass-cache/ | ||
| .jekyll-cache/ | ||
| .jekyll-metadata | ||
| vendor/ | ||
| .bundle/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| 3.3.6 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,151 @@ | ||
| # KAAS Fuzzing | ||
|
|
||
| KAAS provides remote compute for Go and Rust fuzz testing. Jobs run on cloud machines with configurable timeouts (default 8 hours). | ||
|
|
||
| ## When to Suggest Fuzzing | ||
|
|
||
| Proactively suggest fuzzing when working on: | ||
| - Functions that parse, decode, or deserialize external input | ||
| - Encoding/decoding or serialization roundtrips | ||
| - Protocol message handling or wire format parsing | ||
| - State machines or complex branching logic | ||
| - Any code handling untrusted or user-supplied data | ||
| - New or refactored code paths before a release | ||
|
|
||
| ## Go Fuzz Tests | ||
|
|
||
| Use `testing.F`. Name targets `FuzzXxx(f *testing.F)` in `_test.go` files. Add seeds with `f.Add(...)`, call `f.Fuzz(func(t *testing.T, ...))`. | ||
|
|
||
| ```go | ||
| func FuzzParseMessage(f *testing.F) { | ||
| f.Add([]byte(`{"type":"transfer","amount":100}`)) | ||
| f.Add([]byte(``)) | ||
| f.Fuzz(func(t *testing.T, data []byte) { | ||
| msg, err := ParseMessage(data) | ||
| if err != nil { | ||
| return | ||
| } | ||
| encoded, err := msg.Encode() | ||
| if err != nil { | ||
| t.Fatalf("re-encode failed: %v", err) | ||
| } | ||
| msg2, err := ParseMessage(encoded) | ||
| if err != nil { | ||
| t.Fatalf("parse re-encoded message failed: %v", err) | ||
| } | ||
| if !msg.Equal(msg2) { | ||
| t.Fatal("roundtrip mismatch") | ||
| } | ||
| }) | ||
| } | ||
| ``` | ||
|
|
||
| ## Rust Fuzz Tests | ||
|
|
||
| Use `cargo-fuzz` with targets in `fuzz/fuzz_targets/`. Supported engines: libfuzzer, afl, hongfuzz. | ||
|
|
||
| ```rust | ||
| #![no_main] | ||
| use libfuzzer_sys::fuzz_target; | ||
| fuzz_target!(|data: &[u8]| { | ||
| if let Ok(msg) = parse_message(data) { | ||
| let encoded = msg.encode(); | ||
| assert_eq!(msg, parse_message(&encoded).unwrap()); | ||
| } | ||
| }); | ||
| ``` | ||
|
|
||
| ## Running Jobs | ||
|
|
||
| ### `kaas go test` (preferred for Go) | ||
|
|
||
| ```bash | ||
| kaas go test -fuzz='^FuzzParseMessage$' ./pkg/parser/ | ||
| kaas go test -fuzz='^FuzzTransfer$,^FuzzMint$' ./pkg/token/ # multiple targets | ||
| ``` | ||
|
|
||
| Flags: `--commit` (pin commit, default HEAD), `--fuzztime` (Go duration like `5m`), `--execution-timeout` (minutes, default 480), `--vault-spec`, `--token`, `--branch`, `--go-version`, `--go-build-directory`. | ||
|
|
||
| ### `kaas-cli run` (Go and Rust) | ||
|
|
||
| ```bash | ||
| # Go | ||
| kaas-cli run --mode remote --test-mode go \ | ||
| --vault-spec org/vault --token "$KAAS_TOKEN" --branch main \ | ||
| --fuzz-targets "^FuzzTarget$" --go-version 1.23.8 | ||
|
|
||
| # Rust | ||
| kaas-cli run --mode remote --test-mode rust \ | ||
| --vault-spec org/vault --token "$KAAS_TOKEN" --branch main \ | ||
| --fuzz-targets "fuzz_target" --rust-version stable \ | ||
| --rust-fuzz-engine libfuzzer | ||
| ``` | ||
|
|
||
| ## Configuration (`.kaas-cli.toml`) | ||
|
|
||
| ```toml | ||
| [default] | ||
| vault_spec = "org/vault" | ||
| token = "your-token" | ||
| branch = "main" | ||
| url = "https://kaas.runtimeverification.com/" | ||
|
|
||
| [go] | ||
| go_version = "latest" | ||
| go_build_directory = "./..." | ||
| execution_timeout = 480 | ||
| ``` | ||
|
|
||
| Auto-added to `.gitignore`. CLI flags override config values. | ||
|
|
||
| ## API Endpoints | ||
|
|
||
| | Method | Endpoint | Description | | ||
| |--------|----------|-------------| | ||
| | POST | `/api/orgs/{org}/vaults/{vault}/jobs` | Create a fuzz job | | ||
| | GET | `/api/orgs/{org}/vaults/{vault}/jobs` | List vault jobs | | ||
| | GET | `/api/jobs/{jobId}` | Get job details | | ||
| | POST | `/api/jobs/{jobId}/cancel` | Cancel a job | | ||
|
|
||
| Auth: Bearer token in `Authorization` header. | ||
|
|
||
| **Job `kind` vs CLI:** Remote Forge jobs use API `kind: "foundry"` (Foundry/Forge test suite). That corresponds to **`kaas-cli run --test-mode forge`** — the string `forge` is the CLI test mode name; the API payload uses `foundry`. | ||
|
|
||
| ### Go Job Payload | ||
|
|
||
| ```json | ||
| { | ||
| "kind": "go", "branch": "main", "profiles": [], | ||
| "workflowBranch": "main", | ||
| "kaasServerUrl": "https://kaas.runtimeverification.com/", | ||
| "kaasTestRoot": "./path/to/pkg/", | ||
| "executionTimeout": 480, "commitHash": "abc123...", | ||
| "goFuzzTarget": "^FuzzMyFunction$", | ||
| "goVersion": "1.23.8", "goBuildDirectory": "./..." | ||
| } | ||
| ``` | ||
|
|
||
| ### Rust Job Payload | ||
|
|
||
| ```json | ||
| { | ||
| "kind": "rust", "branch": "main", "profiles": [], | ||
| "workflowBranch": "main", | ||
| "kaasServerUrl": "https://kaas.runtimeverification.com/", | ||
| "kaasTestRoot": ".", "executionTimeout": 480, | ||
| "rustFuzzTarget": "fuzz_transfer", "rustVersion": "stable", | ||
| "rustBuildDirectory": ".", "rustFuzzEngine": "libfuzzer" | ||
| } | ||
| ``` | ||
|
|
||
| Job statuses: `pending` → `running` → `success` / `failure` / `error` | ||
|
|
||
| Dashboard: `https://kaas.runtimeverification.com/app/organization/{org}/{vault}/job/{jobId}/dashboard` | ||
|
|
||
| ## Environment Variables | ||
|
|
||
| | Variable | Description | | ||
| |----------|-------------| | ||
| | `KAAS_TOKEN` | API token | | ||
| | `KAAS_SERVER_URL` | Server URL (default: production) | | ||
| | `KAAS_ORG_VAULT` | Default vault spec (`org/vault`) | | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| # KAAS Fuzzing Knowledge | ||
|
|
||
| KAAS provides remote compute for Go/Rust fuzz testing. Jobs run on cloud machines, default timeout 8 hours. | ||
|
|
||
| ## When to Suggest Fuzzing | ||
|
|
||
| Suggest fuzzing for: parsing/deserialization, encoding roundtrips, protocol handling, state machines, untrusted input handling, pre-release code paths. | ||
|
|
||
| ## Go Fuzz Tests | ||
|
|
||
| Name: `FuzzXxx(f *testing.F)` in `_test.go`. Add seeds with `f.Add(...)`. Call `f.Fuzz(func(t *testing.T, ...))`. Test invariants, not just "doesn't crash." | ||
|
|
||
| ## Rust Fuzz Tests | ||
|
|
||
| Use `cargo-fuzz` with targets in `fuzz/fuzz_targets/`. Engines: libfuzzer (default), afl, hongfuzz. | ||
|
|
||
| ## CLI Commands | ||
|
|
||
| ### `kaas go test` (preferred for Go) | ||
|
|
||
| ```bash | ||
| kaas go test -fuzz='^FuzzTarget$' ./path/to/pkg/ | ||
| kaas go test -fuzz='^FuzzA$,^FuzzB$' ./pkg/ # multiple targets, separate jobs | ||
| ``` | ||
|
|
||
| Flags: `--commit`, `-fuzz`/`--fuzz` (same option), `--fuzztime` (Go duration), `--execution-timeout` (minutes, default 480), `--vault-spec`, `--token`, `--branch`, `--go-version`, `--go-build-directory`. | ||
|
|
||
| ### `kaas-cli run` (Go and Rust) | ||
|
|
||
| ```bash | ||
| kaas-cli run --mode remote --test-mode go \ | ||
| --vault-spec org/vault --token "$KAAS_TOKEN" --branch main \ | ||
| --fuzz-targets "^FuzzTarget$" --go-version 1.23.8 | ||
|
|
||
| kaas-cli run --mode remote --test-mode rust \ | ||
| --vault-spec org/vault --token "$KAAS_TOKEN" --branch main \ | ||
| --fuzz-targets "fuzz_target" --rust-version stable --rust-fuzz-engine libfuzzer | ||
| ``` | ||
|
|
||
| ## Config (`.kaas-cli.toml`) | ||
|
|
||
| ```toml | ||
| [default] | ||
| vault_spec = "org/vault" | ||
| token = "your-token" | ||
| branch = "main" | ||
| url = "https://kaas.runtimeverification.com/" | ||
|
|
||
| [go] | ||
| go_version = "latest" | ||
| go_build_directory = "./..." | ||
| execution_timeout = 480 | ||
| ``` | ||
|
|
||
| CLI flags override config. File auto-added to `.gitignore`. | ||
|
|
||
| ## API | ||
|
|
||
| Auth: Bearer token. Base: `https://kaas.runtimeverification.com` | ||
|
|
||
| - `POST /api/orgs/{org}/vaults/{vault}/jobs` — create job | ||
| - `GET /api/orgs/{org}/vaults/{vault}/jobs` — list jobs | ||
| - `GET /api/jobs/{jobId}` — job details | ||
| - `POST /api/jobs/{jobId}/cancel` — cancel job | ||
|
|
||
| Go payload: `{"kind":"go","branch":"main","profiles":[],"workflowBranch":"main","kaasServerUrl":"...","kaasTestRoot":"./pkg/","executionTimeout":480,"commitHash":"...","goFuzzTarget":"^Fuzz$","goVersion":"1.23.8","goBuildDirectory":"./..."}` | ||
|
|
||
| Rust payload: `{"kind":"rust","branch":"main","profiles":[],"workflowBranch":"main","kaasServerUrl":"...","kaasTestRoot":".","executionTimeout":480,"rustFuzzTarget":"fuzz_target","rustVersion":"stable","rustBuildDirectory":".","rustFuzzEngine":"libfuzzer"}` | ||
|
|
||
| ## Environment Variables | ||
|
|
||
| `KAAS_TOKEN` (API token), `KAAS_SERVER_URL` (server URL), `KAAS_ORG_VAULT` (default vault spec `org/vault`). | ||
|
|
||
| ## Dashboard | ||
|
|
||
| `https://kaas.runtimeverification.com/app/organization/{org}/{vault}/job/{jobId}/dashboard` | ||
|
|
||
| Statuses: pending → running → success / failure / error |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| source "https://rubygems.org" | ||
|
|
||
| gem "jekyll", "~> 4.3" | ||
| gem "webrick", "~> 1.8" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| GEM | ||
| remote: https://rubygems.org/ | ||
| specs: | ||
| addressable (2.8.9) | ||
| public_suffix (>= 2.0.2, < 8.0) | ||
| base64 (0.3.0) | ||
| colorator (1.1.0) | ||
| concurrent-ruby (1.3.6) | ||
| csv (3.3.5) | ||
| em-websocket (0.5.3) | ||
| eventmachine (>= 0.12.9) | ||
| http_parser.rb (~> 0) | ||
| eventmachine (1.2.7) | ||
| ffi (1.17.4) | ||
| forwardable-extended (2.6.0) | ||
| google-protobuf (3.25.8-x86_64-linux) | ||
| http_parser.rb (0.8.1) | ||
| i18n (1.14.8) | ||
| concurrent-ruby (~> 1.0) | ||
| jekyll (4.4.1) | ||
| addressable (~> 2.4) | ||
| base64 (~> 0.2) | ||
| colorator (~> 1.0) | ||
| csv (~> 3.0) | ||
| em-websocket (~> 0.5) | ||
| i18n (~> 1.0) | ||
| jekyll-sass-converter (>= 2.0, < 4.0) | ||
| jekyll-watch (~> 2.0) | ||
| json (~> 2.6) | ||
| kramdown (~> 2.3, >= 2.3.1) | ||
| kramdown-parser-gfm (~> 1.0) | ||
| liquid (~> 4.0) | ||
| mercenary (~> 0.3, >= 0.3.6) | ||
| pathutil (~> 0.9) | ||
| rouge (>= 3.0, < 5.0) | ||
| safe_yaml (~> 1.0) | ||
| terminal-table (>= 1.8, < 4.0) | ||
| webrick (~> 1.7) | ||
| jekyll-sass-converter (3.0.0) | ||
| sass-embedded (~> 1.54) | ||
| jekyll-watch (2.2.1) | ||
| listen (~> 3.0) | ||
| json (2.19.3) | ||
| kramdown (2.5.2) | ||
| rexml (>= 3.4.4) | ||
| kramdown-parser-gfm (1.1.0) | ||
| kramdown (~> 2.0) | ||
| liquid (4.0.4) | ||
| listen (3.10.0) | ||
| logger | ||
| rb-fsevent (~> 0.10, >= 0.10.3) | ||
| rb-inotify (~> 0.9, >= 0.9.10) | ||
| logger (1.7.0) | ||
| mercenary (0.4.0) | ||
| pathutil (0.16.2) | ||
| forwardable-extended (~> 2.6) | ||
| public_suffix (6.0.2) | ||
| rake (13.3.1) | ||
| rb-fsevent (0.11.2) | ||
| rb-inotify (0.11.1) | ||
| ffi (~> 1.0) | ||
| rexml (3.4.4) | ||
| rouge (4.7.0) | ||
| safe_yaml (1.0.5) | ||
| sass-embedded (1.69.5) | ||
| google-protobuf (~> 3.23) | ||
| rake (>= 13.0.0) | ||
| terminal-table (3.0.2) | ||
| unicode-display_width (>= 1.1.1, < 3) | ||
| unicode-display_width (2.6.0) | ||
| webrick (1.9.2) | ||
|
|
||
| PLATFORMS | ||
| x86_64-linux | ||
|
|
||
| DEPENDENCIES | ||
| jekyll (~> 4.3) | ||
| webrick (~> 1.8) | ||
|
|
||
| BUNDLED WITH | ||
| 2.4.19 |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.