5142 UI rewrite create a new v1toolsgenerate schemas from openapi endpoint

[ja8zyjits]

✨ Feature / Enhancement PR

🔗 Epic / Issue

Link to the epic or parent issue:
Closes #5142

---

🚀 Summary (1-2 sentences)

What does this PR add or change?

Adds new versioned REST API endpoint POST /v1/tools/generate-schemas-from-openapi that generates MCP tool input/output schemas from OpenAPI specifications. Mirrors admin endpoint functionality without CSRF protection, enabling API consumers and integrations to programmatically generate schemas.

---

🧪 Checks

• make lint passes
• make test passes
• CHANGELOG updated (if user-facing)

---

📓 Notes (optional)

Design sketch, screenshots, or extra context.

Implementation Details

New Files:

• mcpgateway/routers/openapi_schema_router.py (125 lines) - Router implementation
• tests/unit/mcpgateway/routers/test_openapi_schema_router.py (17 tests) - Comprehensive test coverage

Modified Files:

• mcpgateway/main.py - Router registration
• tests/utils/rbac_mocks.py - Fixed mock decorators to accept allow_admin_bypass parameter
• mcpgateway/routers/oauth_router.py - Added missing docstring for popup parameter in initiate_oauth_flow() (one-line documentation improvement, low risk)

Key Features

• RBAC: Requires tools.create permission, no admin bypass
• No CSRF: Versioned API endpoint suitable for programmatic access
• Service Delegation: Reuses existing fetch_and_extract_schemas() service
• Error Handling: Identical to admin endpoint (400, 404, 502, 500 responses)
• Security: URL validation via SecurityValidator

Request/Response Format

Request:

{
  "url": "https://api.example.com/v1/calculate",
  "request_type": "POST",
  "openapi_url": ""
}

Response (Success):

{
  "message": "Schemas generated successfully from OpenAPI spec",
  "success": true,
  "input_schema": { "type": "object", "properties": {...} },
  "output_schema": { "type": "object", "properties": {...} },
  "spec_url": "https://api.example.com/openapi.json"
}

Test Coverage

All 17 tests passed covering (16 unit tests + 1 TestClient integration test):

• Happy path scenarios (all fields, minimal fields, auto-discovery)
• Input validation (invalid JSON, missing fields, type errors)
• Service error mapping (ValueError, KeyError, HTTPStatusError, HTTPError, Exception)
• RBAC enforcement
• Default value handling
• URL parsing

Architecture

flowchart TD
    A[API Client] -->|POST /v1/tools/generate-schemas-from-openapi| B(OpenAPI Schema Router)
    B -->|RBAC Check| C{tools.create permission?}
    C -->|No| D[403 Forbidden]
    C -->|Yes| E[Validate Request]
    E -->|Invalid| F[400 Bad Request]
    E -->|Valid| G[SecurityValidator.validate_url]
    G -->|Invalid| F
    G -->|Valid| H[fetch_and_extract_schemas]
    H -->|ValueError| F
    H -->|KeyError| I[404 Not Found]
    H -->|HTTPError| J[502 Bad Gateway]
    H -->|Exception| K[500 Internal Error]
    H -->|Success| L[200 OK with schemas]

Loading

Comparison: Admin vs Versioned API Endpoint

Aspect	Admin Endpoint	Versioned API Endpoint
Path	/admin/tools/generate-schemas-from-openapi	/v1/tools/generate-schemas-from-openapi
Router	admin_router	openapi_schema_router
CSRF	Required	None
RBAC	tools.create, no admin bypass	Same
Service	fetch_and_extract_schemas()	Same
Error Handling	Comprehensive	Identical
Use Case	Admin UI	API consumers, integrations

Error Response Differences

Validation Errors: The REST API endpoint returns 422 Unprocessable Entity (FastAPI/Pydantic standard) for invalid input, while the admin endpoint returns 400 Bad Request (HTMX form compatibility). This is intentional — the REST API follows HTTP semantics where 422 indicates the request was well-formed but semantically invalid.

Example:

• REST API: POST /v1/tools/generate-schemas-from-openapi with missing url → 422
• Admin UI: POST /admin/tools/generate-schemas-from-openapi with missing url → 400

[marekdano]

@ja8zyjits, thanks for this PR!!!

List of my findings:

I cannot see the option for Request body in docs#/Resources/create_resource_resources_post that we can update

[marekdano]

Eager import from mcpgateway.admin defeats lazy loading

openapi_schema_router.py line 26 imports a private function from the admin module:

from mcpgateway.admin import _read_request_json

main.py comments explicitly state admin is a "large module (~19k lines, ~120ms cold)" that is lazy-imported to avoid startup cost. This top-level import loads the full admin module on every startup, regardless of MCPGATEWAY_ADMIN_API_ENABLED. _read_request_json is also a private symbol (_ prefix) — importing private internals across module boundaries is fragile.

Suggestion: Move _read_request_json (or an equivalent body reader) to mcpgateway/utils/request_helpers.py and import from there, or inline the logic directly in the router.

[marekdano]

Unrelated removal of set_logging_service in main.py

The diff removes both the import and the call:

-    from mcpgateway.admin import admin_router, enforce_admin_csrf, set_logging_service, validate_section_permissions
+    from mcpgateway.admin import admin_router, enforce_admin_csrf, validate_section_permissions

-    set_logging_service(logging_service)

set_logging_service still exists in admin.py (line 710). Dropping the call means the admin module's logging service is never initialized — a silent regression in admin logging. This change is unrelated to the issue and appears accidental; it should be reverted or explained.

[marekdano]

Please remove

[marekdano]

Please remove

[marekdano]

Path naming is inconsistent with the admin endpoint

Endpoint	Path
Admin	/admin/tools/generate-schemas-from-openapi (plural)
New (this PR)	/v1/tools/generate-schema-from-openapi (singular)

The issue title also uses plural (generate-schemas-from-openapi). Worth aligning before the endpoint is published.

[marekdano]

Should use a Pydantic request model for a versioned API endpoint

Using _read_request_json + manual field validation is the pattern for the admin's HTML/HTMX forms. For a versioned REST API endpoint, a Pydantic model is the right convention — it auto-generates correct OpenAPI docs, provides cleaner validation error messages, and matches every other router in mcpgateway/routers/:

class GenerateSchemaRequest(BaseModel):
    url: str
    request_type: str = "GET"
    openapi_url: str = ""

@router.post("/generate-schema-from-openapi")
async def generate_schema_from_openapi(body: GenerateSchemaRequest, ...):

[marekdano]

RBAC test coverage is hollow

patch_rbac_decorators() is called at import time, so @require_permission is bypassed for every test. There is no test that verifies the endpoint returns 403 when the caller lacks tools.create. Add at least one test using a real TestClient with a mocked permission check that returns False, to confirm the decorator is wired correctly.

[marekdano]

The PR looks good!

LGTM 🚀

[msureshkumar88]

Thanks for adding the versioned API path for schema generation — reusing fetch_and_extract_schemas() instead of re-implementing the OpenAPI parsing is the right call, and the SSRF protection in the underlying service (fetch_openapi_spec) correctly covers openapi_url for both the admin and new v1 paths.

A few things worth resolving before merge:

Blocking

1. Error-response shape diverges from the admin endpoint and from the PR's own "Identical" claim. Admin (admin.py:12059-12091) manually validates the raw JSON body and returns a custom 400 ({"message": ..., "success": false}) for missing/invalid fields. The new router uses Pydantic (Body(...) + GenerateSchemaRequest), so the same malformed-input cases now return FastAPI's default 422 instead. Could you either align the response shape with the admin endpoint, or update the PR description/table to call out this intentional difference?
2. No test exercises the route through the actual ASGI app. All 16 tests call router_mod.generate_schemas_from_openapi(...) directly rather than going through TestClient. That means the route registration (app.include_router(openapi_schema_router) in main.py), the real Pydantic validation path (422 responses), and the explicit "no CSRF" design goal have zero coverage. Worth adding at least one TestClient test that POSTs to /v1/tools/generate-schemas-from-openapi to lock in routing + the no-CSRF behavior so a future global CSRF change doesn't silently break this.
3. mcpgateway/routers/oauth_router.py docstring change looks unrelated to this PR's scope (OpenAPI schema generation). Worth splitting into a separate commit/PR, or dropping if it was picked up accidentally during a rebase.

Suggestions

4. Significant duplication between openapi_schema_router.py:103-184 and admin.py:12036-12151 (same validate → fetch → error-mapping ladder). Extracting the shared logic into one helper (e.g. in openapi_service.py) would prevent the kind of drift seen in finding #1 from recurring.
5. Worth a security sign-off on scope expansion: this moves an SSRF-protected outbound-fetch capability from an admin-UI/CSRF-gated surface to any bearer-token holder with tools.create. Not a code defect, but the blast radius for who can trigger outbound requests from the gateway just widened — would be good to confirm which roles carry tools.create in the default role set as part of this review.
6. No caching on the OpenAPI spec fetch — every call re-fetches and re-parses the full spec. Pre-existing behavior, but more exposed now with a broader caller base; a short-TTL cache keyed on spec_url could reduce load.
7. Endpoint has no dedicated rate limit, and it triggers a real outbound HTTP call per request to a caller-supplied target — could be used as a low-cost amplifier for outbound requests.

Minor

8. PR description's endpoint path (/v1/tools/generate-schema-from-openapi, singular "schema") doesn't match the actual implemented/tested path (/v1/tools/generate-schemas-from-openapi, plural). Worth fixing the description so it's not confusing for future readers.
9. Test count in the PR description says 17; the diff has 16 tests.
10. No docs update for the new public /v1/... route, and the CHANGELOG checkbox is unchecked — flagging since this is user-facing.

Not a breaking change — purely additive, and the rbac_mocks.py signature changes add an optional kwarg with a default so existing callers are unaffected. Issue reference (Closes #5142) is present and correct.

[msureshkumar88]

Good progress addressing the earlier feedback — the Pydantic model, plural path, TestClient route-registration test, and CHANGELOG are all in good shape now.

One open thread still needs attention before merge:

RBAC 403 enforcement — direct call vs. TestClient

@marekdano's thread on line 23 of test_openapi_schema_router.py is still unresolved. The existing test_generate_schemas_403_when_permission_denied calls router_mod.generate_schemas_from_openapi(body, _user=...) directly and asserts pytest.raises(HTTPException). That verifies the permission-check logic in isolation, but the reviewer specifically asked for a TestClient test — i.e., a request that travels through ASGI routing — to confirm the @require_permission decorator is wired correctly on the registered route, not just on the bare function.

The test_endpoint_via_testclient_validates_route_registration test (added in the latest commit) uses dependency_overrides to bypass auth entirely, so it can't catch a misconfigured decorator either.

A couple of approaches worth considering:

1. Extend the existing TestClient test: after the happy-path assertions, add a sub-case that doesn't override get_current_user_with_permissions and sends a request with no credentials — you'd get either 401 or a CSRF error (pre-existing global behavior), which at least proves the route isn't accidentally left open.

2. Convert test_generate_schemas_403_when_permission_denied to use TestClient: the reload pattern already in that test (restore_rbac_decorators → importlib.reload(router_mod)) gives you a fresh router with the real @require_permission. Mounting router_mod.router on a throwaway FastAPI() and overriding get_current_user_with_permissions to return an unprivileged user dict, while keeping PermissionService monkeypatched to DenyAll, would drive the 403 through ASGI and close the thread cleanly.

Option 2 is the closer match to what the reviewer asked for. Happy to provide a code sketch if that's helpful.

Everything else looks good — no other blocking concerns from my side.

[msureshkumar88]

E2E Verification Results

Ran an automated E2E script against a live gateway instance to verify the new endpoint works end-to-end through real ASGI routing with real authentication and RBAC.

Environment / Config

Setting	Value
Gateway	uvicorn mcpgateway.main:app --host 127.0.0.1 --port 9444
Database	SQLite (fresh bootstrap via python -m mcpgateway.bootstrap_db)
Auth	AUTH_REQUIRED=true, JWT HS256
Plugins	PLUGINS_ENABLED=false
Target API	Minimal FastAPI app on :9123 exposing POST /calculate (two float inputs → float result) with auto-generated /openapi.json

Setup Commands

# Bootstrap fresh DB
python -m mcpgateway.bootstrap_db

# Start gateway
uvicorn mcpgateway.main:app --host 127.0.0.1 --port 9444 &

# Start target OpenAPI server
uvicorn target_app:app --host 127.0.0.1 --port 9123 &

# Generate tokens
ADMIN_TOK=$(python -m mcpgateway.utils.create_jwt_token \
  --username admin@example.com --exp 60 --secret "$JWT_SECRET_KEY" --admin)

VIEWER_TOK=$(python -m mcpgateway.utils.create_jwt_token \
  --username viewer@example.com --exp 60 --secret "$JWT_SECRET_KEY")

Viewer user inserted into DB with viewer role (tools.read only, no tools.create).

Test Commands & Results

# 1. Happy path — auto-discovery, admin token
curl -s -X POST http://127.0.0.1:9444/v1/tools/generate-schemas-from-openapi \
  -H "Authorization: Bearer $ADMIN_TOK" \
  -H "Content-Type: application/json" \
  -d '{"url":"http://127.0.0.1:9123/calculate","request_type":"POST","openapi_url":""}'
# → 200 {"success":true,"input_schema":{...},"output_schema":{...},"spec_url":"http://127.0.0.1:9123/openapi.json"}

# 2. Explicit openapi_url
-d '{"url":"http://127.0.0.1:9123/calculate","request_type":"POST","openapi_url":"http://127.0.0.1:9123/openapi.json"}'
# → 200 ✅

# 3. RBAC — viewer role (tools.read only, no tools.create)
curl -s -X POST http://127.0.0.1:9444/v1/tools/generate-schemas-from-openapi \
  -H "Authorization: Bearer $VIEWER_TOK" \
  -d '{"url":"http://127.0.0.1:9123/calculate","request_type":"POST"}'
# → 403 {"detail":"Access denied"}  ✅

# 4. Admin still gets through (has tools.create via platform_admin role)
# → 200 {"success":true,...}  ✅  (allow_admin_bypass=False doesn't block users who have the perm explicitly)

# 5. No CSRF needed — Bearer-only request succeeds
# (same request as #1, no X-CSRF-Token header)
# → 200 ✅  Core PR claim confirmed

# Admin endpoint same conditions:
curl -s -X POST http://127.0.0.1:9444/admin/tools/generate-schemas-from-openapi \
  -H "Authorization: Bearer $ADMIN_TOK" -d '{...}'
# → 303 (redirected to login — requires session cookie + CSRF)
# New v1 endpoint and admin endpoint confirmed to be distinct auth surfaces.

# 6. Unknown user JWT (not in DB)
# → 401 {"detail":"Invalid authentication credentials"}  ✅

# 7. Missing url field
-d '{"request_type":"POST"}'
# → 422 (Pydantic validation)  ✅

# 8. Unreachable target host
-d '{"url":"http://127.0.0.1:9999/nope","request_type":"GET"}'
# → 502 {"message":"Failed to fetch OpenAPI spec from the provided URL","success":false}  ✅

# 9. Malicious URL scheme
-d '{"url":"javascript:alert(1)","request_type":"GET"}'
# → 400 {"message":"Tool URL must start with one of: http://, https://, ws://, wss://","success":false}  ✅

# 10. Path not present in spec
-d '{"url":"http://127.0.0.1:9123/does-not-exist","request_type":"GET"}'
# → 404 {"message":"\"Path '/does-not-exist' not found in OpenAPI spec\"","success":false}  ✅

Summary

Test	Expected	Result
Happy path (auto-discovery)	200 + schemas	✅
Explicit openapi_url	200 + schemas	✅
Viewer role (no tools.create)	403	✅
Admin role (has tools.create)	200	✅
No CSRF token with Bearer	200 (CSRF not required)	✅
Admin endpoint same conditions	303 (CSRF/session gated)	✅
Unknown user JWT	401	✅
Missing url field	422	✅
Unreachable host	502	✅
Malicious URL scheme	400	✅
Path absent from spec	404	✅

All 11 probes passed. RBAC enforcement verified live through ASGI with real DB users.

Note: Unauthenticated POST (no Bearer token at all) returns 403 CSRF_TOKEN_INVALID rather than 401 — the global CSRF middleware short-circuits before the route's auth dependency runs for requests without a Bearer token. This is pre-existing gateway-wide behavior, not introduced by this PR.

[msureshkumar88]

All blocking items resolved:

• test_generate_schemas_403_when_permission_denied now drives through TestClient/ASGI — mounts the reloaded router with a real @require_permission, overrides get_current_user_with_permissions to return an unprivileged user, and asserts 403. Exactly what was asked for.
• set_logging_service removal from main.py was reverted — no regression in admin logging.
• Plural path, Pydantic model, CHANGELOG, route-registration TestClient test, and eager-import fix were all addressed in the prior round.

LGTM 🚀

[marekdano]

Thanks for working through all the feedback — the Pydantic model, plural path, eager-import fix, set_logging_service revert, and the TestClient-based RBAC test all look good.

LGTM 🚀

[marekdano]

LGTM 🚀