-
Notifications
You must be signed in to change notification settings - Fork 0
Add remote fuzzing documentation and AI agent skill #12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
F-WRunTime
wants to merge
7
commits into
main
Choose a base branch
from
feature/fuzzing-docs-and-skill
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
04cf604
Add remote fuzzing documentation and AI agent skill
F-WRunTime d734051
Merge branch 'main' into feature/fuzzing-docs-and-skill
F-WRunTime ea99719
Add cross-platform AI agent knowledge files and setup guide
F-WRunTime 0baef58
Add Jekyll + GitHub Pages workflow; refresh docs; remove Cursor skills
F-WRunTime 0afafa5
docs: address PR #12 review (typos, modes, fuzz flags, AGENTS example)
F-WRunTime 547d44d
documenting how to use kaas-cli and foundry/forge to run tests
F-WRunTime 079bb0d
docs: address PR review (headings, images, multi-agent guide)
F-WRunTime File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,194 @@ | ||
| --- | ||
| name: kaas-fuzzing | ||
| description: >- | ||
| Run and manage Go and Rust fuzz testing jobs on KAAS (K as a Service). | ||
| Use when the user wants to fuzz test code, submit remote fuzz jobs, | ||
| write fuzz tests, configure kaas-cli, or interact with the KAAS API | ||
| for fuzzing workflows. Also use when reviewing code that handles | ||
| untrusted input, parsing, serialization, or protocol logic to | ||
| proactively suggest fuzzing. | ||
| --- | ||
|
|
||
| # KAAS Fuzzing | ||
|
|
||
| KAAS provides remote compute infrastructure for running Go and Rust fuzz tests. Jobs execute on cloud machines with configurable timeouts up to 8 hours (default), allowing deeper input-space exploration than local fuzzing. | ||
|
|
||
| ## Why Fuzz on KAAS | ||
|
|
||
| Fuzz testing finds bugs that unit tests miss: panics, crashes, infinite loops, edge cases in parsing/encoding, and unexpected state transitions. | ||
|
|
||
| KAAS provides long-running cloud compute so fuzzing can explore far more inputs than local runs. Each fuzz target gets its own isolated job. Multiple targets run in parallel without blocking the developer's machine. | ||
|
|
||
| **Proactively suggest fuzzing** when working on: | ||
|
|
||
| - Functions that parse, decode, or deserialize external input | ||
| - Encoding/decoding or serialization roundtrips | ||
| - Protocol message handling or wire format parsing | ||
| - Cryptographic operations or hashing | ||
| - State machines or complex branching logic | ||
| - Any code handling untrusted or user-supplied data | ||
| - New or refactored code paths before a release | ||
|
|
||
| ## Writing Go Fuzz Tests | ||
|
|
||
| Go fuzz tests use the `testing.F` API. They must: | ||
| - Be named `FuzzXxx(f *testing.F)` in a `_test.go` file | ||
| - Add seed corpus entries with `f.Add(...)` | ||
| - Call `f.Fuzz(func(t *testing.T, ...))` with the fuzz function | ||
|
|
||
| ```go | ||
| func FuzzParseMessage(f *testing.F) { | ||
| f.Add([]byte(`{"type":"transfer","amount":100}`)) | ||
| f.Add([]byte(`{}`)) | ||
| f.Add([]byte(``)) | ||
|
|
||
| f.Fuzz(func(t *testing.T, data []byte) { | ||
| msg, err := ParseMessage(data) | ||
| if err != nil { | ||
| return | ||
| } | ||
| // Test invariants on successfully parsed messages | ||
| encoded, err := msg.Encode() | ||
| if err != nil { | ||
| t.Fatalf("parsed message failed to re-encode: %v", err) | ||
| } | ||
| msg2, err := ParseMessage(encoded) | ||
| if err != nil { | ||
| t.Fatalf("roundtrip failed: %v", err) | ||
| } | ||
| if !msg.Equal(msg2) { | ||
| t.Fatal("roundtrip produced different message") | ||
| } | ||
| }) | ||
| } | ||
| ``` | ||
|
|
||
| **Tips for effective fuzz targets:** | ||
| - One function per target — keep the scope focused | ||
| - Add meaningful seeds that exercise real code paths | ||
| - Test invariants, not just "doesn't panic" (roundtrips, idempotency, bounds) | ||
| - Use `t.Skip()` for inputs you intentionally don't handle | ||
|
|
||
| ## Writing Rust Fuzz Tests | ||
|
|
||
| Rust fuzz tests use `cargo-fuzz` with targets in `fuzz/fuzz_targets/`. | ||
|
|
||
| ```rust | ||
| // fuzz/fuzz_targets/fuzz_parse.rs | ||
| #![no_main] | ||
| use libfuzzer_sys::fuzz_target; | ||
| use my_crate::parse_message; | ||
|
|
||
| fuzz_target!(|data: &[u8]| { | ||
| if let Ok(msg) = parse_message(data) { | ||
| let encoded = msg.encode(); | ||
| let msg2 = parse_message(&encoded).expect("roundtrip failed"); | ||
| assert_eq!(msg, msg2); | ||
| } | ||
| }); | ||
| ``` | ||
|
|
||
| Supported engines: `libfuzzer` (default), `afl`, `hongfuzz`. | ||
|
|
||
| ## Running Fuzz Jobs | ||
|
|
||
| ### `kaas go test` (Preferred for Go) | ||
|
|
||
| Mirrors `go test -fuzz` syntax: | ||
|
|
||
| ```bash | ||
| kaas go test -fuzz='^FuzzParseMessage$' ./pkg/parser/ | ||
| ``` | ||
|
|
||
| Multiple targets (each becomes a separate parallel job): | ||
|
|
||
| ```bash | ||
| kaas go test -fuzz='^FuzzTransfer$,^FuzzMint$' ./pkg/token/ | ||
| ``` | ||
|
|
||
| Key flags: | ||
|
|
||
| | Flag | Description | Default | | ||
| |------|-------------|---------| | ||
| | `--fuzz` | Comma-separated fuzz target patterns (required) | — | | ||
|
F-WRunTime marked this conversation as resolved.
Outdated
|
||
| | `--commit` | Pin a specific git commit | HEAD | | ||
| | `--fuzztime` | Go-style duration (`30s`, `5m`, `1h`) | — | | ||
| | `--execution-timeout` | Timeout in minutes | 480 | | ||
| | `--vault-spec` | `org/vault` format | from `.kaas-cli.toml` | | ||
| | `--token` | API token | from `.kaas-cli.toml` | | ||
| | `--branch` | Git branch | `main` | | ||
| | `--go-version` | Go version | `latest` | | ||
| | `--go-build-directory` | Go build directory | `./...` | | ||
|
|
||
| ### `kaas-cli run` (Full form, Go and Rust) | ||
|
|
||
| Go: | ||
|
|
||
| ```bash | ||
| kaas-cli run \ | ||
| --mode remote --test-mode go \ | ||
| --vault-spec org/vault --token "$KAAS_TOKEN" --branch main \ | ||
| --fuzz-targets "^FuzzTransfer$" \ | ||
| --go-version 1.23.8 --go-build-directory "./..." | ||
| ``` | ||
|
|
||
| Rust: | ||
|
|
||
| ```bash | ||
| kaas-cli run \ | ||
| --mode remote --test-mode rust \ | ||
| --vault-spec org/vault --token "$KAAS_TOKEN" --branch main \ | ||
| --fuzz-targets "fuzz_transfer" \ | ||
| --rust-version stable --rust-build-directory "." \ | ||
| --rust-fuzz-engine libfuzzer | ||
| ``` | ||
|
|
||
| ## `.kaas-cli.toml` Configuration | ||
|
|
||
| Persistent config for `kaas go test`. Created interactively on first run. CLI flags override config values. | ||
|
|
||
| ```toml | ||
| [default] | ||
| vault_spec = "org/vault" | ||
| token = "your-token" | ||
| branch = "main" | ||
| url = "https://kaas.runtimeverification.com/" | ||
|
|
||
| [go] | ||
| go_version = "latest" | ||
| go_build_directory = "./..." | ||
| execution_timeout = 480 | ||
| ``` | ||
|
|
||
| The file is auto-added to `.gitignore` (contains token). | ||
|
|
||
| ## API Reference | ||
|
|
||
| For programmatic access, see [api-reference.md](api-reference.md). | ||
|
|
||
| Key endpoints: | ||
|
|
||
| | Method | Endpoint | Description | | ||
| |--------|----------|-------------| | ||
| | POST | `/api/orgs/{org}/vaults/{vault}/jobs` | Create a fuzz job | | ||
| | GET | `/api/orgs/{org}/vaults/{vault}/jobs` | List vault jobs | | ||
| | GET | `/api/jobs/{jobId}` | Get job details | | ||
| | POST | `/api/jobs/{jobId}/cancel` | Cancel a job | | ||
|
|
||
| ## Environment Variables | ||
|
|
||
| | Variable | Description | | ||
| |----------|-------------| | ||
| | `KAAS_TOKEN` | API token (alternative to `--token`) | | ||
| | `KAAS_SERVER_URL` | Server URL (default: `https://kaas.runtimeverification.com/`) | | ||
| | `KAAS_ORG_VAULT` | Default vault spec in `org/vault` format | | ||
|
|
||
| ## Dashboard | ||
|
|
||
| Job results are viewable at: | ||
|
|
||
| ``` | ||
| https://kaas.runtimeverification.com/app/organization/{org}/{vault}/job/{jobId}/dashboard | ||
| ``` | ||
|
|
||
| Job statuses: `pending` → `running` → `success` / `failure` / `error` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| # KAAS Fuzzing API Reference | ||
|
|
||
| Base URL: `https://kaas.runtimeverification.com` | ||
|
|
||
| All endpoints require a Bearer token in the `Authorization` header. | ||
|
|
||
| ## Create a Fuzz Job | ||
|
|
||
| ``` | ||
| POST /api/orgs/{organizationName}/vaults/{vaultName}/jobs | ||
| ``` | ||
|
|
||
| ### Go Fuzzing Payload | ||
|
|
||
| ```json | ||
| { | ||
| "kind": "go", | ||
| "branch": "main", | ||
| "profiles": [], | ||
| "workflowBranch": "main", | ||
| "kaasServerUrl": "https://kaas.runtimeverification.com/", | ||
| "kaasTestRoot": "./path/to/pkg/", | ||
| "executionTimeout": 480, | ||
| "commitHash": "abc123def456...", | ||
| "goFuzzTarget": "^FuzzMyFunction$", | ||
| "goVersion": "1.23.8", | ||
| "goBuildDirectory": "./..." | ||
| } | ||
| ``` | ||
|
|
||
| ### Rust Fuzzing Payload | ||
|
|
||
| ```json | ||
| { | ||
| "kind": "rust", | ||
| "branch": "main", | ||
| "profiles": [], | ||
| "workflowBranch": "main", | ||
| "kaasServerUrl": "https://kaas.runtimeverification.com/", | ||
| "kaasTestRoot": ".", | ||
| "executionTimeout": 480, | ||
| "rustFuzzTarget": "fuzz_transfer", | ||
| "rustVersion": "stable", | ||
| "rustBuildDirectory": ".", | ||
| "rustFuzzEngine": "libfuzzer", | ||
| "rustFuzzArgs": "" | ||
| } | ||
| ``` | ||
|
|
||
| ### Response | ||
|
|
||
| ```json | ||
| { | ||
| "jobId": "unique-job-id", | ||
| "status": "pending" | ||
| } | ||
| ``` | ||
|
|
||
| ## List Vault Jobs | ||
|
|
||
| ``` | ||
| GET /api/orgs/{organizationName}/vaults/{vaultName}/jobs | ||
| ``` | ||
|
|
||
| Query parameters: | ||
| - `page` (int): Page number, default 1 | ||
| - `per_page` (int): Results per page, default 20 | ||
| - `status` (string): Filter by status (`pending`, `running`, `success`, `failure`, `cancelled`, `error`) | ||
| - `commit` (string): Filter by commit SHA | ||
| - `sort` (string): Sort field | ||
| - `direction` (string): Sort direction (`asc`, `desc`) | ||
|
|
||
| ## Get Job Details | ||
|
|
||
| ``` | ||
| GET /api/jobs/{jobId} | ||
| ``` | ||
|
|
||
| Returns full job object including status, configuration, timing, and results. | ||
|
|
||
| ## Cancel a Job | ||
|
|
||
| ``` | ||
| POST /api/jobs/{jobId}/cancel | ||
| ``` | ||
|
|
||
| Cancels a pending or running job. | ||
|
|
||
| ## Get Job Reports | ||
|
|
||
| ``` | ||
| GET /api/jobs/{jobId}/json-report | ||
| POST /api/jobs/{jobId}/pdf-report | ||
| ``` | ||
|
|
||
| ## Job Kind Values | ||
|
|
||
| | Kind | Description | | ||
| |------|-------------| | ||
| | `kontrol` | Kontrol formal verification | | ||
| | `foundry` | Forge test suite | | ||
|
F-WRunTime marked this conversation as resolved.
Outdated
F-WRunTime marked this conversation as resolved.
Outdated
|
||
| | `go` | Go fuzz testing | | ||
| | `rust` | Rust fuzz testing | | ||
|
|
||
| ## Job Status Values | ||
|
|
||
| | Status | Description | | ||
| |--------|-------------| | ||
| | `pending` | Job created, waiting for runner | | ||
| | `running` | Job is executing | | ||
| | `success` | Job completed successfully | | ||
| | `failure` | Job completed with test failures | | ||
| | `cancelled` | Job was cancelled | | ||
| | `error` | Job encountered an infrastructure error | | ||
| | `processed` | Results have been processed | | ||
| | `processing_failed` | Post-processing of results failed | | ||
|
|
||
| ## Dashboard URL Pattern | ||
|
|
||
| ``` | ||
| https://kaas.runtimeverification.com/app/organization/{org}/{vault}/job/{jobId}/dashboard | ||
| ``` | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.