Skip to content

Commit d207734

Browse files
DumbrisDumbris
authored
Merge branch 'main' into upstream-pr/discover-quarantined-tools
2 parents 3690598 + b3a965b commit d207734

203 files changed

Lines changed: 14950 additions & 444 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/bench.yml‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
name: Benchmark Dashboard
2+
3+
# Triggered on stable release tags — runs the live benchmark and publishes the
4+
# dashboard to Cloudflare Pages (mcpproxy-bench project).
5+
#
6+
# Non-blocking: bench failure never gates the release pipeline.
7+
#
8+
# Why host binary instead of bench/docker-compose.yml:
9+
# The Dockerfile uses a distroless runtime image that lacks npx/uvx. The 7
10+
# snapshot-server configs spawn stdio servers via npx/uvx, which need to run
11+
# in the same environment as mcpproxy. The eval.yml retrieval-d1 job solves
12+
# this by building the binary and running it on the host runner (where Node.js
13+
# and uv are installed). We follow the same pattern here.
14+
# The docker-compose.yml is kept for local development; a future PR can add a
15+
# bench-specific image that includes the runtime tools.
16+
#
17+
# Reports are never committed (Spec 065 CN-003) — published as CI artifacts and
18+
# Cloudflare Pages deployments only.
19+
20+
on:
21+
push:
22+
tags: ["v*"]
23+
workflow_dispatch:
24+
25+
permissions:
26+
contents: read
27+
28+
jobs:
29+
bench-dashboard:
30+
name: Run benchmark and publish dashboard
31+
runs-on: ubuntu-latest
32+
environment: production
33+
# Stable releases only — RC/prerelease tags (v*-rc.*, v*-next.*) are handled
34+
# by prerelease.yml. workflow_dispatch allows manual runs from any ref.
35+
if: "github.event_name == 'workflow_dispatch' || (startsWith(github.ref, 'refs/tags/v') && !contains(github.ref_name, '-') && github.repository == 'smart-mcp-proxy/mcpproxy-go')"
36+
# Non-blocking: bench failure never blocks the release.
37+
continue-on-error: true
38+
39+
steps:
40+
- name: Checkout
41+
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
42+
43+
- name: Set up Go
44+
uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0
45+
with:
46+
go-version: "1.25"
47+
cache: true
48+
49+
- name: Set up Node.js (npx-launched MCP reference servers)
50+
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
51+
with:
52+
node-version: "22"
53+
54+
- name: Set up uv (uvx-launched MCP reference servers)
55+
uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
56+
57+
- name: Build mcpproxy (personal edition)
58+
run: go build -o mcpproxy ./cmd/mcpproxy
59+
60+
# Offline benchmark: deterministic token-reduction scores, no live servers.
61+
# Writes bench/results/report.json and bench/results/dashboard.html.
62+
- name: Run offline benchmark
63+
run: go run ./bench/cmd/bench -out bench/results
64+
65+
# Live benchmark: boot mcpproxy with the 7 no-auth reference servers, wait
66+
# for the full tool catalog, then score accuracy + latency + full-schema tokens.
67+
# Writes bench/results/live_report.json.
68+
- name: Boot mcpproxy with reference servers and run live benchmark
69+
env:
70+
DS: ${{ github.workspace }}/specs/065-evaluation-foundation/datasets
71+
BASE: http://127.0.0.1:8092
72+
KEY: eval-corpus-snapshot
73+
run: |
74+
set -uo pipefail
75+
76+
mkdir -p "$RUNNER_TEMP/bench"
77+
./mcpproxy serve \
78+
--config "$DS/snapshot-servers.config.json" \
79+
--data-dir "$RUNNER_TEMP/bench" \
80+
--listen 127.0.0.1:8092 \
81+
--log-level info > "$RUNNER_TEMP/mcpproxy-bench.log" 2>&1 &
82+
server_pid=$!
83+
trap 'kill "$server_pid" 2>/dev/null || true' EXIT
84+
85+
# Wait for the full tool catalog before scoring: the retrieval index is
86+
# built after all servers connect (~45 tools across 7 reference servers).
87+
ready=0
88+
expected=44
89+
for i in $(seq 1 60); do
90+
if ! kill -0 "$server_pid" 2>/dev/null; then
91+
echo "::error::mcpproxy exited during startup"
92+
tail -40 "$RUNNER_TEMP/mcpproxy-bench.log" || true
93+
exit 1
94+
fi
95+
t="$(curl -fsS -H "X-API-Key: $KEY" "$BASE/api/v1/tools" \
96+
| python3 -c 'import sys,json;d=json.load(sys.stdin);print(len((d.get("data") or {}).get("tools", [])))' 2>/dev/null || echo 0)"
97+
echo "attempt $i: catalog has $t tool(s)"
98+
if [ "$t" -ge "$expected" ]; then
99+
echo "Catalog full ($t tools); settling 8s for index build."
100+
sleep 8
101+
ready=1
102+
break
103+
fi
104+
sleep 5
105+
done
106+
if [ "$ready" != "1" ]; then
107+
echo "::error::mcpproxy catalog did not reach ${expected} tools in 5 minutes"
108+
tail -80 "$RUNNER_TEMP/mcpproxy-bench.log" || true
109+
exit 1
110+
fi
111+
112+
go run ./bench/cmd/bench \
113+
-live \
114+
-proxy "$BASE" \
115+
-api-key "$KEY" \
116+
-out bench/results
117+
118+
kill "$server_pid" 2>/dev/null || true
119+
120+
# Serve dashboard.html at the root URL for Cloudflare Pages.
121+
- name: Prepare dashboard index
122+
run: cp bench/results/dashboard.html bench/results/index.html
123+
124+
# Always upload results as a CI artifact — available even if Pages deploy fails.
125+
- name: Upload dashboard artifact
126+
if: always()
127+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
128+
with:
129+
name: bench-dashboard-${{ github.ref_name }}
130+
path: bench/results/
131+
retention-days: 90
132+
if-no-files-found: warn
133+
134+
# Bootstrap: create the Cloudflare Pages project if it does not yet exist.
135+
# Idempotent — fails silently (continue-on-error) when the project already
136+
# exists (Cloudflare error 8000007). Requires Pages:Edit scope on the token.
137+
# If the token is deploy-only, create the project once manually in the
138+
# Cloudflare dashboard (name: mcpproxy-bench, production branch: main) and
139+
# this step will harmlessly fail every run thereafter.
140+
- name: Create Cloudflare Pages project (if missing)
141+
continue-on-error: true
142+
uses: cloudflare/wrangler-action@ebbaa1584979971c8614a24965b4405ff95890e0 # v4.0.0
143+
with:
144+
apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
145+
accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
146+
command: pages project create mcpproxy-bench --production-branch=main
147+
148+
# Publish to Cloudflare Pages (mcpproxy-bench project).
149+
# --commit-dirty=true: bench results are written into the working tree during
150+
# the run and never committed to git, so wrangler would otherwise reject the
151+
# dirty-tree check. The Pages URL will be mcpproxy-bench.pages.dev (or a
152+
# custom domain such as bench.mcpproxy.app once configured in Cloudflare).
153+
- name: Deploy benchmark dashboard to Cloudflare Pages
154+
uses: cloudflare/wrangler-action@ebbaa1584979971c8614a24965b4405ff95890e0 # v4.0.0
155+
with:
156+
apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
157+
accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
158+
command: pages deploy bench/results --project-name=mcpproxy-bench --commit-dirty=true

‎.github/workflows/retry-sign-release.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -399,7 +399,7 @@ jobs:
399399
needs: release
400400
runs-on: ubuntu-latest
401401
environment: production
402-
if: github.repository == 'smart-mcp-proxy/mcpproxy-go'
402+
if: github.repository == 'smart-mcp-proxy/mcpproxy-go' && !contains(inputs.tag, '-')
403403

404404
steps:
405405
- name: Checkout tap repository
@@ -585,7 +585,7 @@ jobs:
585585
# Publish signed apt/yum repositories to Cloudflare R2 (mirrors release.yml)
586586
publish-linux-repos:
587587
needs: [release]
588-
if: github.repository == 'smart-mcp-proxy/mcpproxy-go'
588+
if: github.repository == 'smart-mcp-proxy/mcpproxy-go' && !contains(inputs.tag, '-')
589589
runs-on: ubuntu-latest
590590
environment: production
591591

‎.github/workflows/scanner-images.yml‎

Lines changed: 97 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,13 @@
11
name: scanner-images
22

33
# Build and publish the custom security scanner Docker images to
4-
# ghcr.io/smart-mcp-proxy/scanner-*. Vendor images (trivy, semgrep, the
5-
# AI scanner which lives in a separate repo) are NOT built here.
4+
# ghcr.io/smart-mcp-proxy/scanner-*. Vendor images (semgrep, the AI scanner
5+
# which lives in a separate repo) are NOT built here. trivy IS wrapped in our
6+
# own image (docker/scanners/trivy) to pre-cache the vuln DB (MCP-2150).
67
#
78
# Triggers:
89
# - push to main that touches docker/scanners/** or this workflow file
10+
# - weekly schedule: rebuilds trivy to pick up a fresh vulnerability DB
911
# - manual dispatch (for ad-hoc rebuilds)
1012
# - pull requests: builds without pushing (to catch Dockerfile drift)
1113

@@ -19,6 +21,9 @@ on:
1921
paths:
2022
- 'docker/scanners/**'
2123
- '.github/workflows/scanner-images.yml'
24+
schedule:
25+
# Weekly Monday 03:00 UTC — keeps the trivy vuln DB reasonably current.
26+
- cron: '0 3 * * 1'
2227
workflow_dispatch:
2328
inputs:
2429
scanner:
@@ -33,40 +38,62 @@ permissions:
3338

3439
jobs:
3540
build:
36-
runs-on: ubuntu-latest
41+
# Most scanners use ubuntu-latest (amd64) with QEMU for multi-arch builds.
42+
# ramparts uses per-arch native runners to avoid QEMU timeout on the Rust
43+
# `cargo install` step: `matrix.runner` selects the runner per entry.
44+
runs-on: ${{ matrix.runner }}
3745
strategy:
3846
fail-fast: false
3947
matrix:
40-
# `platforms` defaults to multi-arch. ramparts overrides to linux/amd64
41-
# only: it is the lone Rust-from-source wrapper, and `cargo install
42-
# ramparts` for an emulated linux/arm64 leg runs under QEMU (~5-10x
43-
# slower). A cold build of both arches blows past the runner budget and
44-
# the job gets cancelled mid-compile (MCP-2395). amd64-only finishes in
45-
# time; the binary is consumed on the host arch by Docker isolation.
48+
# `arch` is set only for ramparts per-arch entries. Entries without
49+
# `arch` push a combined multi-arch :latest directly; per-arch entries
50+
# push :<sha>-<arch> / :latest-<arch> tags and the merge-ramparts job
51+
# assembles the final multi-arch :latest manifest.
4652
include:
4753
- id: snyk
54+
runner: ubuntu-latest
4855
image: ghcr.io/smart-mcp-proxy/scanner-snyk
4956
context: docker/scanners/snyk
5057
platforms: linux/amd64,linux/arm64
5158
- id: cisco
59+
runner: ubuntu-latest
5260
image: ghcr.io/smart-mcp-proxy/scanner-cisco
5361
context: docker/scanners/cisco
5462
platforms: linux/amd64,linux/arm64
55-
- id: ramparts
56-
image: ghcr.io/smart-mcp-proxy/scanner-ramparts
57-
context: docker/scanners/ramparts
58-
platforms: linux/amd64
63+
- id: trivy
64+
runner: ubuntu-latest
65+
image: ghcr.io/smart-mcp-proxy/scanner-trivy
66+
context: docker/scanners/trivy
67+
platforms: linux/amd64,linux/arm64
5968
- id: proximity
69+
runner: ubuntu-latest
6070
image: ghcr.io/smart-mcp-proxy/scanner-proximity
6171
context: docker/scanners/proximity
6272
platforms: linux/amd64,linux/arm64
73+
# ramparts: native per-arch builds (MCP-2150).
74+
# Using QEMU to emulate arm64 on an amd64 runner blows past the job
75+
# budget (~5-10x slower for `cargo install`; MCP-2395). Instead each
76+
# arch builds on its native runner and the merge-ramparts job combines
77+
# them into a multi-arch :latest manifest.
78+
- id: ramparts
79+
arch: amd64
80+
runner: ubuntu-latest
81+
image: ghcr.io/smart-mcp-proxy/scanner-ramparts
82+
context: docker/scanners/ramparts
83+
platforms: linux/amd64
84+
- id: ramparts
85+
arch: arm64
86+
runner: ubuntu-24.04-arm
87+
image: ghcr.io/smart-mcp-proxy/scanner-ramparts
88+
context: docker/scanners/ramparts
89+
platforms: linux/arm64
6390
steps:
6491
- name: Checkout
6592
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
6693

6794
- name: Skip if not selected
6895
if: ${{ github.event_name == 'workflow_dispatch' && inputs.scanner != '' && inputs.scanner != matrix.id }}
69-
run: echo "Skipping ${{ matrix.id }} (workflow_dispatch selected ${{ inputs.scanner }})" && exit 0
96+
run: echo "Skipping ${{ matrix.id }}${{ matrix.arch && format('-{0}', matrix.arch) || '' }} (workflow_dispatch selected ${{ inputs.scanner }})" && exit 0
7097

7198
- name: Set up QEMU
7299
uses: docker/setup-qemu-action@06116385d9baf250c9f4dcb4858b16962ea869c3 # v4.1.0
@@ -88,10 +115,22 @@ jobs:
88115
set -euo pipefail
89116
short_sha="${GITHUB_SHA:0:7}"
90117
echo "short_sha=$short_sha" >> "$GITHUB_OUTPUT"
91-
if [ "${GITHUB_REF}" = "refs/heads/main" ]; then
92-
echo "tags=${{ matrix.image }}:latest,${{ matrix.image }}:$short_sha" >> "$GITHUB_OUTPUT"
118+
arch="${{ matrix.arch }}"
119+
if [ -n "$arch" ]; then
120+
# Per-arch build (ramparts): push arch-suffixed tags only.
121+
# The merge-ramparts job assembles :latest and :<sha> from these.
122+
if [ "${GITHUB_REF}" = "refs/heads/main" ]; then
123+
echo "tags=${{ matrix.image }}:latest-${arch},${{ matrix.image }}:${short_sha}-${arch}" >> "$GITHUB_OUTPUT"
124+
else
125+
echo "tags=${{ matrix.image }}:pr-${{ github.event.pull_request.number || 'dev' }}-${arch}" >> "$GITHUB_OUTPUT"
126+
fi
93127
else
94-
echo "tags=${{ matrix.image }}:pr-${{ github.event.pull_request.number || 'dev' }}" >> "$GITHUB_OUTPUT"
128+
# Multi-arch build: push :latest and :<sha> directly.
129+
if [ "${GITHUB_REF}" = "refs/heads/main" ]; then
130+
echo "tags=${{ matrix.image }}:latest,${{ matrix.image }}:$short_sha" >> "$GITHUB_OUTPUT"
131+
else
132+
echo "tags=${{ matrix.image }}:pr-${{ github.event.pull_request.number || 'dev' }}" >> "$GITHUB_OUTPUT"
133+
fi
95134
fi
96135
97136
- name: Build and push
@@ -101,8 +140,8 @@ jobs:
101140
platforms: ${{ matrix.platforms }}
102141
push: ${{ github.event_name != 'pull_request' }}
103142
tags: ${{ steps.tags.outputs.tags }}
104-
cache-from: type=gha,scope=scanner-${{ matrix.id }}
105-
cache-to: type=gha,scope=scanner-${{ matrix.id }},mode=max
143+
cache-from: type=gha,scope=scanner-${{ matrix.id }}${{ matrix.arch && format('-{0}', matrix.arch) || '' }}
144+
cache-to: type=gha,scope=scanner-${{ matrix.id }}${{ matrix.arch && format('-{0}', matrix.arch) || '' }},mode=max
106145
labels: |
107146
org.opencontainers.image.source=https://github.com/smart-mcp-proxy/mcpproxy-go
108147
org.opencontainers.image.revision=${{ github.sha }}
@@ -111,12 +150,12 @@ jobs:
111150
# multi-platform builds (linux/amd64,linux/arm64) cannot be loaded to the runner with
112151
# `load: true`, so there is no local image to scan on PRs. On push the image is
113152
# already in GHCR and Trivy pulls it directly.
114-
# exit-code: '0' → report-only, never fails the build. Scanner base images routinely
115-
# carry unfixable CVEs (upstream OS packages); blocking builds on those would create
116-
# constant noise with no actionable remediation. Visibility via the Security tab and
117-
# the workflow log is the goal.
153+
# Skipped for per-arch ramparts builds (matrix.arch set): those entries push
154+
# arch-suffixed tags that get merged by merge-ramparts; the merged :latest is
155+
# what should be scanned, not the intermediate arch-specific tags.
156+
# exit-code: '0' → report-only, never fails the build.
118157
- name: Scan image with Trivy
119-
if: github.event_name != 'pull_request'
158+
if: github.event_name != 'pull_request' && !matrix.arch
120159
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
121160
with:
122161
image-ref: ${{ matrix.image }}:${{ steps.tags.outputs.short_sha }}
@@ -127,7 +166,7 @@ jobs:
127166
format: table
128167

129168
- name: Upload Trivy SARIF to Security tab
130-
if: github.event_name != 'pull_request'
169+
if: github.event_name != 'pull_request' && !matrix.arch
131170
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
132171
with:
133172
image-ref: ${{ matrix.image }}:${{ steps.tags.outputs.short_sha }}
@@ -139,8 +178,40 @@ jobs:
139178
output: trivy-results-${{ matrix.id }}.sarif
140179

141180
- name: Upload SARIF
142-
if: github.event_name != 'pull_request'
181+
if: github.event_name != 'pull_request' && !matrix.arch
143182
uses: github/codeql-action/upload-sarif@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4.36.2
144183
with:
145184
sarif_file: trivy-results-${{ matrix.id }}.sarif
146185
category: trivy-${{ matrix.id }}
186+
187+
# Assemble the multi-arch ramparts manifest from the two per-arch images
188+
# pushed by the build job. Runs only on push to main and workflow_dispatch
189+
# (not pull_request, since per-arch images aren't pushed on PRs).
190+
merge-ramparts:
191+
needs: build
192+
if: github.event_name != 'pull_request'
193+
runs-on: ubuntu-latest
194+
steps:
195+
- name: Log in to GHCR
196+
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
197+
with:
198+
registry: ghcr.io
199+
username: ${{ github.actor }}
200+
password: ${{ secrets.GITHUB_TOKEN }}
201+
202+
- name: Set up Buildx
203+
uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
204+
205+
- name: Create multi-arch :latest manifest
206+
run: |
207+
docker buildx imagetools create \
208+
-t ghcr.io/smart-mcp-proxy/scanner-ramparts:latest \
209+
ghcr.io/smart-mcp-proxy/scanner-ramparts:latest-amd64 \
210+
ghcr.io/smart-mcp-proxy/scanner-ramparts:latest-arm64
211+
212+
- name: Create multi-arch SHA-tagged manifest
213+
run: |
214+
docker buildx imagetools create \
215+
-t ghcr.io/smart-mcp-proxy/scanner-ramparts:${GITHUB_SHA:0:7} \
216+
ghcr.io/smart-mcp-proxy/scanner-ramparts:${GITHUB_SHA:0:7}-amd64 \
217+
ghcr.io/smart-mcp-proxy/scanner-ramparts:${GITHUB_SHA:0:7}-arm64

‎CLAUDE.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,3 +155,6 @@ tail -f ~/Library/Logs/mcpproxy/main.log # main log (macOS; Linux: ~/.mcpproxy/
155155
- Config changes: update both storage and file system; the file watcher hot-reloads.
156156
- **macOS tray dev** (build / replace / verify with `mcpproxy-ui-test`): [docs/development/macos-tray.md](docs/development/macos-tray.md).
157157
- **Windows installer**: [docs/github-actions-windows-wix-research.md](docs/github-actions-windows-wix-research.md). **Prerelease** (`next` branch + `v*-rc.*` tags, opt-in, off stable channels): [docs/prerelease-builds.md](docs/prerelease-builds.md).
158+
159+
## Recent Changes
160+
- 076-deterministic-tool-scanner: Added Go 1.24 + stdlib only for detection (`unicode`, `unicode/utf8`, `encoding/base64`, `encoding/hex`, `regexp`); `golang.org/x/text/unicode/norm` (already an indirect dep via x/text) for NFKC; existing `internal/security/patterns/`, `internal/security/scanner/`, `internal/runtime/tool_quarantine.go`. No new third-party dependency.

‎bench/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Benchmark run artifacts are never committed (Spec 065 CN-003).
2+
results/

0 commit comments

Comments
 (0)