refactor(plugins): migrate in-tree plugins to PyPI packages (cpex-*) (#3965)

* refactor(plugins): replace in-tree rate_limiter with cpex-rate-limiter package

Remove the in-tree rate_limiter plugin and replace it with the
cpex-rate-limiter PyPI package, a compiled Rust extension providing
the same RateLimiterPlugin class with additional algorithms
(sliding-window, token-bucket) alongside the original fixed-window.

- Add cpex-rate-limiter>=0.0.2 as a [plugins] optional dependency
- Update Containerfile.lite to install the plugins extra
- Remove plugins/rate_limiter/ source directory
- Remove unit and integration tests that imported plugin internals
- Update all config files to use cpex_rate_limiter.RateLimiterPlugin
- Disable RateLimiterPlugin in test fixture config (package not
  available in unit test environment)
- Update documentation to reflect the external package

Signed-off-by: Jonathan Springer <jps@s390x.com>

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* feat(rate-limiter): pluggable algorithms with Rust-backed execution engine, benchmarks, and validation (#3809)

* feat(rate-limiter): pluggable algorithms, tenant isolation fix, and scale load test

- Add pluggable algorithm strategy: fixed_window, sliding_window, token_bucket
- Add Redis backend for shared cross-instance rate limiting
- Fix tenant isolation: skip by_tenant when tenant_id is None
- Fix sliding window: sweep expired timestamps before counting
- Fix backend validation: restore _validate_config check
- Fix token bucket memory path: apply max(1,...) guard to reset timestamp
- Add Redis integration tests for all three algorithms
- Add direct regression tests for get_current_user tenant_id fallback
- Add scale load test with Redis memory timeline and live algorithm detection
- Add RL_PACE_MULTIPLIER for near-limit pace testing and boundary burst detection
- Remove redundant algorithm locustfile; scale file is canonical
- Correct stale comments and README limitations

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* feat(rate-limiter): add Rust-backed engine, check() API, benchmarks, and validation

- Rust-backed sliding window engine with pyo3-log integration
- check() API with tenant propagation, sweep/retry-after support
- Eliminate redundant ZRANGE in sliding window Lua script
- Fix detect-secrets baseline for rate limiter load tests
- Clarify memory backend is single-instance only in docs

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* chore: regenerate detect-secrets baseline after rebase

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* refactor(rate-limiter): review fixes, Redis hardening, key-format parity tests

- Extract _dispatch_hook() shared by prompt_pre_fetch and tool_pre_invoke,
  reducing each hook to a single-line wrapper
- Elevate Redis val_i64/val_f64 parse-error logging from warn to error so
  silent fail-open degradation surfaces in operator dashboards
- Clamp sliding-window reset_timestamp with .max(1) so it is always strictly
  in the future even when the oldest entry expires in < 1 s
- Add 5 s tokio::time::timeout around Redis connection establishment to
  prevent indefinite blocking on network partition
- Replace silent except-pass in EVALSHA SHA tracking with logger.debug
- Document dual Lua-script invariant (rolling-upgrade key-format parity)
  in both Python RedisBackend docstring and Rust redis_backend.rs header
- Add 7 parametrized test_redis_key_format_parity_* tests validating that
  Python and Rust produce identical Redis keys for the same inputs
- Revert unrelated .pyi stub changes for encoded_exfil_detection, pii_filter,
  retry_with_backoff, and secrets_detection

Signed-off-by: Jonathan Springer <jps@s390x.com>

* fix: strip trailing whitespace in pyi stubs, remove accidental .claude/ralph-loop.local.md

- Remove plugins_rust/rate_limiter/.claude/ralph-loop.local.md which
  was accidentally committed — this is a local Claude Code loop state
  file and should never have been checked in.
- Fix trailing whitespace in plugins_rust/rate_limiter/python/
  rate_limiter_rust/__init__.pyi docstrings to pass pre-commit hooks.

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* chore: regenerate detect-secrets baseline for new exfil test strings

Update .secrets.baseline after adding test_extra_sensitive_keywords
in plugins_rust/encoded_exfil_detection/src/lib.rs:969 which contains
a fake credential string that triggers the Secret Keyword detector.
All new entries are false positives (test data).

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

* chore: audit new detect-secrets baseline entries as false positives

The baseline regeneration reset is_secret to null for entries whose
line numbers shifted. Mark all 17 unaudited entries as is_secret=false
(test data, example configs, fake credentials) to pass the
--fail-on-unaudited pre-commit check.

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>

---------

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>
Signed-off-by: Jonathan Springer <jps@s390x.com>
Co-authored-by: Jonathan Springer <jps@s390x.com>

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* feat(discovery): add automatic tool discovery with hot/cold classification (#3839)

Implement automatic tool discovery for upstream MCP servers via
usage-aware adaptive polling. The gateway can now continuously
synchronise tool lists from registered servers without manual
intervention.

Server classification (hot/cold):
- Classify servers based on MCP session pool usage patterns
- Hot servers (top 20% by recent usage): polled at 1x base interval
- Cold servers (remaining 80%): polled at 3x base interval
- Classification is deterministic: sorted by recency, active sessions,
  use count, and URL for tie-breaking
- Leader election via Redis with TTL renewal for multi-worker
  coordination
- Falls back to local-only operation without Redis

Integration with GatewayService:
- Health checks respect hot/cold classification intervals
- Auto-refresh of tools/resources/prompts respects classification
- Fail-open on classification errors (poll anyway)
- Poll timestamps tracked via Redis with TTL expiry
- Uses base gateway URL (pre-auth) for classification lookups to
  avoid leaking query-param auth secrets to Redis

Configuration:
- AUTO_REFRESH_SERVERS=true enables automatic tool sync (default: false)
- GATEWAY_AUTO_REFRESH_INTERVAL=300 sets base polling interval
- HOT_COLD_CLASSIFICATION_ENABLED=false (opt-in, requires Redis)

Includes comprehensive tests with 100% coverage on the new
ServerClassificationService and integration tests for the
GatewayService hot/cold polling paths.

Closes #3734

Signed-off-by: Lang-Akshay <akshay.shinde26@ibm.com>
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* refactor(plugins): replace in-tree rate_limiter with cpex-rate-limiter package

Remove the in-tree rate_limiter plugin and replace it with the
cpex-rate-limiter PyPI package, a compiled Rust extension providing
the same RateLimiterPlugin class with additional algorithms
(sliding-window, token-bucket) alongside the original fixed-window.

- Add cpex-rate-limiter>=0.0.2 as a [plugins] optional dependency
- Update Containerfile.lite to install the plugins extra
- Remove plugins/rate_limiter/ source directory
- Remove unit and integration tests that imported plugin internals
- Update all config files to use cpex_rate_limiter.RateLimiterPlugin
- Disable RateLimiterPlugin in test fixture config (package not
  available in unit test environment)
- Update documentation to reflect the external package

Signed-off-by: Jonathan Springer <jps@s390x.com>

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* refactor(plugins): update build, CI, and docs for PyPI plugin migration

Remove all plugins_rust/ build infrastructure and update references
across Containerfiles, Makefile, CI workflows, pre-commit configs,
CODEOWNERS, and documentation to reflect that plugins are now
distributed as PyPI packages (cpex-*) via the [plugins] optional extra.

- Remove Rust plugin builder stages from all Containerfiles
- Remove ~100 lines of rust-* plugin Makefile targets (keep mcp-runtime)
- Add --extra plugins to CI pytest workflow
- Add [plugins] extra to install-dev Makefile target
- Update tool_service.py import to use cpex_retry_with_backoff
- Update plugin kind paths in 7 doc files to cpex_pii_filter.*
- Clean up pre-commit, CODEOWNERS, MANIFEST.in, whitesource, .gitignore

Signed-off-by: Jonathan Springer <jps@s390x.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>

* fix(plugins): address PR review findings on PyPI plugin migration

Round 1 (blockers + high):
- Restore exclude-newer = "10 days" in pyproject.toml; replace stale
  langchain/requests pins with cpex-* per-package overrides anchored
  to 2026-04-09 so the plugins resolve newer than the global window
- Guard cpex_retry_with_backoff import in tool_service.py with
  try/except ImportError; falls back to (None, True) for the Python
  pipeline when the optional [plugins] extra is not installed
- Delete orphaned .github/workflows/rust-plugins.yml and the
  associated test cases in tests/unit/test_rust_plugins_workflow.py;
  drop the workflow card from docs/docs/architecture/explorer.html
- Delete orphaned docs/docs/using/plugins/rust-plugins.md and remove
  it from docs/docs/using/plugins/.pages mkdocs nav
- Harden docker-entrypoint.sh install_plugin_requirements:
  canonicalize /app and the resolved requirements path with
  readlink -f and require the path to live under /app/, log
  non-comment lines from the requirements file before pip runs,
  and skip cleanly on validation failure
- Delete PLUGIN-MIGRATION-PLAN.md (one-time planning doc)
- Add COPY plugins/requirements.txt to Containerfile.scratch (the
  layered Containerfile.lite already had it; the broad COPY . in
  Containerfile already includes it)

Round 2 (medium + low):
- Bump cpex-* version pin floors in pyproject.toml [plugins] to
  match resolved versions in uv.lock (cpex-rate-limiter>=0.0.3,
  cpex-encoded-exfil-detection>=0.2.0, cpex-pii-filter>=0.2.0,
  cpex-url-reputation>=0.1.1)
- Add Prerequisites section to tests/performance/PLUGIN_PROFILING.md
  documenting the [plugins] extra requirement
- Add Status: Partially superseded note to ADR-041 explaining that
  plugins_rust/ was removed when in-tree Rust plugins migrated to
  PyPI packages
- Document upgrade semantics in plugins/requirements.txt header
  (pip without --upgrade skips already-satisfied constraints)
- Add importlib.util.find_spec() precheck to
  tests/performance/test_plugins_performance.py main(); the script
  now skips cleanly with an actionable message if any of the five
  cpex packages referenced by the perf config are missing
- Rename tests/unit/test_rust_plugins_workflow.py to
  test_go_toolchain_pinning.py to match its remaining contents
  (Go workflow pin and Makefile toolchain assertion)

Follow-ups tracked in #4116 and
IBM/cpex-plugins#21 for the longer-term tool_service.py refactor
that will eliminate the cross-package import entirely.

Signed-off-by: Jonathan Springer <jps@s390x.com>

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* revert: restore tests changes from PR #3965

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* fix(ci): align plugin tests with PyPI migration

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: remove legacy plugin test skip infrastructure

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: align packaged plugin tests with rust shims

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: cover retry policy import path in tool service

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* fix: harden cpex plugin migration paths

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: cover retry policy parser branches

Signed-off-by: lucarlig <luca.carlig@ibm.com>

* test: cover plugin requirements entrypoint path

Signed-off-by: lucarlig <luca.carlig@ibm.com>

---------

Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: Jonathan Springer <jps@s390x.com>
Co-authored-by: Pratik Gandhi <gandhipratik203@gmail.com>
Co-authored-by: Lang-Akshay <akshay.shinde26@ibm.com>
Co-authored-by: lucarlig <luca.carlig@ibm.com>

Author: 4 people

.claude/skills/pr-risk-scoring/score_prs.py

@@ -11,6 +11,7 @@
     approvals_csv  Optional CSV with columns: pr_number,approvals
 """
 
+# Standard
 import csv
 import json
 import os
@@ -167,7 +168,7 @@ def compute_security_score(files, labels):
     return min(score, 5)
 
 
-PROD_PREFIXES = ("mcpgateway/", "plugins/", "plugins_rust/", "a2a-agents/", "mcp-servers/", "tools_rust/")
+PROD_PREFIXES = ("mcpgateway/", "plugins/", "a2a-agents/", "mcp-servers/", "tools_rust/")
 
 
 def compute_test_score(files):

.github/CODEOWNERS

@@ -6,7 +6,6 @@
 /mcpgateway/plugins @araujof @terylt @jonpspri
 
 # Rust projects
-/plugins_rust/ @lucarlig @dima-zakharov
 /tools_rust/ @lucarlig @dima-zakharov
 /mcp-servers/rust/ @lucarlig @dima-zakharov
 

.github/workflows/pytest.yml

@@ -83,31 +83,17 @@ jobs:
           python-version: ${{ matrix.python }}
 
       # -----------------------------------------------------------
-      # 2.5  Setup Rust toolchain for Rust plugins
+      # Note: Rust plugin builds removed - all plugins now distributed as PyPI packages
+      # Rust MCP runtime (tools_rust/mcp_runtime) not needed for main pytest suite
+      # (e2e_rust tests are excluded and run in separate workflow)
       # -----------------------------------------------------------
-      - name: 🦀  Install Rust stable
-        run: rustup default stable
-
-      - name: 📦  Cache Cargo dependencies
-        uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
-        with:
-          path: |
-            ~/.cargo/registry
-            ~/.cargo/git
-            plugins_rust/*/target
-          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
-          restore-keys: |
-            ${{ runner.os }}-cargo-
-
-      - name: 🔨  Build Rust plugins (clean, install, verify stubs)
-        run: make rust-clean-stubs && make rust-install && make rust-verify-stubs
 
       # -----------------------------------------------------------
       # 3️⃣  Run the tests with coverage (fail under 95 %total  coverage)
       # -----------------------------------------------------------
       - name: 🧪  Run pytest
         run: |
-          uv run pytest -n auto \
+          uv run --extra plugins pytest -n auto \
             --durations=5 \
             --ignore=tests/fuzz \
             --ignore=tests/e2e/test_entra_id_integration.py \
@@ -117,8 +103,6 @@ jobs:
             --cov-report=term \
             --cov-branch \
             --cov-fail-under=95
-        env:
-          REQUIRE_RUST: "1"
 
       # -----------------------------------------------------------
       # 3.5  Diff-cover: enforce 93% coverage on changed lines (PRs only)

.github/workflows/rust-plugins.yml

@@ -262,11 +262,6 @@ uv.lock
 .uv-cache/
 .uv-tmp/
 
-# ========================================
-# Rust (plugins_rust)
-# ========================================
-plugins_rust/target/
-plugins_rust/*/target/
 *.rs.bk
 
 # ========================================

.gitignore

@@ -36,7 +36,7 @@ repos:
       - id: detect-private-key
         name: 🔐 Detect Private Key
         description: Detects the presence of private keys.
-        exclude: (mcpgateway/utils/generate_keys|tests/unit/mcpgateway/utils/test_generate_keys|tests/unit/mcpgateway/plugins/framework/external/mcp/test_tls_utils)\.py|plugins_rust/secrets_detection/examples/heavy_workload\.rs|plugins_rust/secrets_detection/src/scanner\.rs|plugins_rust/secrets_detection/src/patterns\.rs
+        exclude: (mcpgateway/utils/generate_keys|tests/unit/mcpgateway/utils/test_generate_keys|tests/unit/mcpgateway/plugins/framework/external/mcp/test_tls_utils)\.py
         types: [text]
 
   # -----------------------------------------------------------------------------
@@ -248,7 +248,6 @@ repos:
         name: ✅ Check Shebang Scripts Are Executable
         description: Ensures that (non-binary) files with a shebang are executable.
         types: [text]
-        exclude: plugins_rust/secrets_detection/compare_performance\.py
         stages: [pre-commit, pre-push, manual]
 
       - id: forbid-new-submodules