Metadata Guardrail by vrushankportkey · Pull Request #1149 · Portkey-AI/gateway

vrushankportkey · 2025-06-18T08:02:54Z

Description

Motivation

Type of Change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update
Refactoring (no functional changes)

How Has This Been Tested?

Unit Tests
Integration Tests
Manual Testing

Screenshots (if applicable)

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

Related Issues

…st-copy-in-manifest.json Update model whitelist manifest copy

matterai-app · 2025-06-18T08:03:30Z

Description

Summary By MatterAI

🔄 What Changed

This pull request introduces a new metadata guardrail plugin. This plugin allows for enforcing policies based on key-value pairs present in the request context's metadata. It supports 'any', 'all', and 'none' operators for matching, along with a 'not' flag for negation. New unit tests have been added to default.test.ts to cover various scenarios for this new handler.

🔍 Impact of the Change

This new plugin enhances the system's ability to apply fine-grained access control or routing logic based on custom metadata attached to requests. It provides a flexible and configurable way to validate metadata, improving the overall security and control capabilities of the platform. The addition of comprehensive unit tests ensures the reliability and correctness of the new functionality.

📁 Total Files Changed

3 files were changed:

plugins/default/default.test.ts: Modified to include tests for the new metadata handler.
plugins/default/metadata.ts: New file added, containing the implementation of the metadata guardrail handler.
plugins/index.ts: Modified to export and register the new defaultmetadata handler.

🧪 Test Added

Unit tests were added in plugins/default/default.test.ts for the metadata handler:

should return true when any pair matches: Tests the any operator where at least one key-value pair in the parameters matches the context metadata.
should return false when not all pairs match: Tests the all operator where not all specified key-value pairs are found in the context metadata.
should return true when none of the pairs match: Tests the none operator where no specified key-value pairs are found in the context metadata.
should return false when any pair matches but not is true: Tests the any operator combined with the not flag, expecting a false verdict when a match occurs but should not.

🔒Security Vulnerabilities

No security vulnerabilities were detected in the changes. The new plugin implements input validation for pairs and handles cases where context.metadata is missing, contributing to robust error handling.

Motivation

To provide a new guardrail plugin that allows users to define policies based on metadata key-value pairs present in the request context, enabling more flexible and powerful request filtering and routing.

Type of Change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update
Refactoring (no functional changes)

How Has This Been Tested?

Unit Tests
Integration Tests
Manual Testing

Screenshots (if applicable)

N/A

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

Related Issues

N/A

Tip

Quality Recommendations

Add a specific unit test case for the scenario where context.metadata is null or undefined to explicitly verify the if (!context.metadata) branch's behavior and the returned data structure.
In the catch block of metadata.ts, consider including foundKeys and missingKeys as empty arrays in the data object to maintain a consistent structure with the successful execution path's data object, which can aid in debugging and logging consistency.

Sequence Diagram

sequenceDiagram
    actor User
    participant PluginManager
    participant metadataHandler

    User->>PluginManager: Initiates Request (with metadata in context)
    PluginManager->>metadataHandler: Call handler(context: PluginContext, parameters: PluginParameters, eventType: HookEventType)
    metadataHandler-->>metadataHandler: Validate parameters (pairs, operator)
    alt Missing or Invalid Pairs
        metadataHandler-->>metadataHandler: Throw Error
    else No metadata in context
        metadataHandler-->>metadataHandler: Set verdict based on 'not' flag
        metadataHandler-->>metadataHandler: Set data with 'No metadata provided' explanation
    else Process Metadata
        metadataHandler-->>metadataHandler: Iterate Object.entries(parameters.pairs)
        metadataHandler-->>metadataHandler: Compare context.metadata[key] === value
        metadataHandler-->>metadataHandler: Populate foundKeys[] and missingKeys[]
        metadataHandler-->>metadataHandler: Determine result based on parameters.operator ('any', 'all', 'none')
        metadataHandler-->>metadataHandler: Apply parameters.not to final verdict
        metadataHandler-->>metadataHandler: Construct data object with verdict, foundKeys, missingKeys, explanation
    end
    metadataHandler-->>PluginManager: Return { error, verdict, data }
    PluginManager->>User: Respond based on plugin verdict

matterai-app

This PR adds a new Metadata Guardrail feature that looks well-implemented with good test coverage. I have a few suggestions to improve error handling and code robustness.

Skipped files

plugins/README.md: Skipped file pattern

matterai-app · 2025-06-18T08:06:00Z

Important

PR Review Skipped

PR review skipped as per the configuration setting. Run a manually review by commenting /matter review

💡Tips to use Matter AI

Command List

/matter summary: Generate AI Summary for the PR
/matter review: Generate AI Reviews for the latest commit in the PR
/matter review-full: Generate AI Reviews for the complete PR
/matter release-notes: Generate AI release-notes for the PR
/matter : Chat with your PR with Matter AI Agent
/matter remember : Generate AI memories for the PR
/matter explain: Get an explanation of the PR
/matter help: Show the list of available commands and documentation
Need help? Join our Discord server: https://discord.gg/fJU5DvanU3

narengogi · 2025-06-18T12:17:55Z

+        "type": "object",
+        "properties": {
+          "pairs": {
+            "type": "json",


"type": "json" is not a valid json schema key afaik, please use "type": "object"
cc: @VisargD

Hey there are other "type":"json" in the code right?

Co-authored-by: matter-code-review[bot] <150888575+matter-code-review[bot]@users.noreply.github.com>

matterai-app · 2025-06-19T12:44:13Z

Important

PR Review Skipped

PR review skipped as per the configuration setting. Run a manually review by commenting /matter review

💡Tips to use Matter AI

Command List

/matter summary: Generate AI Summary for the PR
/matter review: Generate AI Reviews for the latest commit in the PR
/matter review-full: Generate AI Reviews for the complete PR
/matter release-notes: Generate AI release-notes for the PR
/matter : Chat with your PR with Matter AI Agent
/matter remember : Generate AI memories for the PR
/matter explain: Get an explanation of the PR
/matter help: Show the list of available commands and documentation
Need help? Join our Discord server: https://discord.gg/fJU5DvanU3

matterai-app · 2025-06-19T12:44:33Z

Important

PR Review Skipped

PR review skipped as per the configuration setting. Run a manually review by commenting /matter review

💡Tips to use Matter AI

Command List

/matter summary: Generate AI Summary for the PR
/matter review: Generate AI Reviews for the latest commit in the PR
/matter review-full: Generate AI Reviews for the complete PR
/matter release-notes: Generate AI release-notes for the PR
/matter : Chat with your PR with Matter AI Agent
/matter remember : Generate AI memories for the PR
/matter explain: Get an explanation of the PR
/matter help: Show the list of available commands and documentation
Need help? Join our Discord server: https://discord.gg/fJU5DvanU3

Co-authored-by: matter-code-review[bot] <150888575+matter-code-review[bot]@users.noreply.github.com>

matterai-app · 2025-06-19T12:44:55Z

Important

PR Review Skipped

PR review skipped as per the configuration setting. Run a manually review by commenting /matter review

💡Tips to use Matter AI

Command List

/matter summary: Generate AI Summary for the PR
/matter review: Generate AI Reviews for the latest commit in the PR
/matter review-full: Generate AI Reviews for the complete PR
/matter release-notes: Generate AI release-notes for the PR
/matter : Chat with your PR with Matter AI Agent
/matter remember : Generate AI memories for the PR
/matter explain: Get an explanation of the PR
/matter help: Show the list of available commands and documentation
Need help? Join our Discord server: https://discord.gg/fJU5DvanU3

matterai-app · 2025-06-19T12:45:11Z

Important

PR Review Skipped

PR review skipped as per the configuration setting. Run a manually review by commenting /matter review

💡Tips to use Matter AI

Command List

/matter summary: Generate AI Summary for the PR
/matter review: Generate AI Reviews for the latest commit in the PR
/matter review-full: Generate AI Reviews for the complete PR
/matter release-notes: Generate AI release-notes for the PR
/matter : Chat with your PR with Matter AI Agent
/matter remember : Generate AI memories for the PR
/matter explain: Get an explanation of the PR
/matter help: Show the list of available commands and documentation
Need help? Join our Discord server: https://discord.gg/fJU5DvanU3

VisargD · 2025-06-25T07:58:10Z

Hey! Closing this PR as we have already added requiredMetadataKeys plugin in #1147 . We can extend the same plugin for any new functionality, if requested by the community.

vrushankportkey added 3 commits June 17, 2025 15:53

chore: update model whitelist copy

2f23382

Merge pull request #4 from vrushankportkey/codex/update-model-whiteli…

1c474aa

…st-copy-in-manifest.json Update model whitelist manifest copy

Add metadata guardrail

7cd3a85

matterai-app Bot reviewed Jun 18, 2025

View reviewed changes

Comment thread plugins/default/metadata.ts Outdated

Comment thread plugins/default/metadata.ts Outdated

Comment thread plugins/default/metadata.ts

Comment thread plugins/default/default.test.ts Outdated

docs: clarify how metadata header is used

0fd7342

narengogi previously approved these changes Jun 18, 2025

View reviewed changes

Update plugins/default/metadata.ts

498b412

Co-authored-by: matter-code-review[bot] <150888575+matter-code-review[bot]@users.noreply.github.com>

vrushankportkey dismissed narengogi’s stale review via 498b412 June 19, 2025 12:43

Update plugins/default/metadata.ts

e5bd432

Co-authored-by: matter-code-review[bot] <150888575+matter-code-review[bot]@users.noreply.github.com>

vrushankportkey and others added 2 commits June 19, 2025 18:14

Update plugins/default/metadata.ts

5910ab8

Co-authored-by: matter-code-review[bot] <150888575+matter-code-review[bot]@users.noreply.github.com>

Update plugins/default/default.test.ts

2abcfd4

Co-authored-by: matter-code-review[bot] <150888575+matter-code-review[bot]@users.noreply.github.com>

VisargD closed this Jun 25, 2025

Uh oh!

Conversation

vrushankportkey commented Jun 18, 2025

Description

Motivation

Type of Change

How Has This Been Tested?

Screenshots (if applicable)

Checklist

Related Issues

Uh oh!

matterai-app Bot commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Summary By MatterAI

🔄 What Changed

🔍 Impact of the Change

📁 Total Files Changed

🧪 Test Added

🔒Security Vulnerabilities

Motivation

Type of Change

How Has This Been Tested?

Screenshots (if applicable)

Checklist

Related Issues

Quality Recommendations

Sequence Diagram

Uh oh!

matterai-app Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

matterai-app Bot commented Jun 18, 2025

PR Review Skipped

Command List

Uh oh!

Uh oh!

narengogi Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

vrushankportkey Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

matterai-app Bot commented Jun 19, 2025

PR Review Skipped

Command List

Uh oh!

matterai-app Bot commented Jun 19, 2025

PR Review Skipped

Command List

Uh oh!

matterai-app Bot commented Jun 19, 2025

PR Review Skipped

Command List

Uh oh!

matterai-app Bot commented Jun 19, 2025

PR Review Skipped

Command List

Uh oh!

VisargD commented Jun 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

matterai-app Bot commented Jun 18, 2025 •

edited

Loading