Introduced Safe@k

Author: prane-eth

LICENSE.md

@@ -1,4 +1,4 @@
-© 2025 Praneeth Vadlapati.
+© 2025-30 Praneeth Vadlapati.
 <!-- Copyright (c) 2025 Praneeth Vadlapati -->
 
 # License: CC-BY 4.0

MANIFEST.in

@@ -1 +1,2 @@
 prune tests
+recursive-include agent_action_guard *.pt *.json

README.md

@@ -48,11 +48,28 @@ AI is perceived as a threat. Increasing usage of LLM Agents and MCP leads to the
 
 ## New contributions of Agent-Action-Guard framework:
 1. 	**HarmActions**, an structured dataset of safety-labeled agent actions complemented with manipulated prompts that trigger harmful or unethical actions.
-2. 	**HarmActEval** benchmark leveraging a new metric “Harm@k.”
+2. 	**HarmActEval** benchmark leveraging a new metric “Safe@k.”
 3. 	**Action Classifier**, a neural classifier trained on HarmActions dataset, designed to label proposed agent actions as potentially harmful or safe, and optimized for real-time deployment in agent loops.
 4. 	MCP integration supporting live action screening using existing MCP servers and clients.
 
 
+## HarmActEval Results (Lower is better)
+
+Popular and latest LLMs generate harmful actions, proving the need for the action guard and HarmActEval benchmark.
+
+| Model                   | Safe@3 score |
+|-------------------------|-------------:|
+| Granite 4-H-Tiny        | **0.00%**    |
+| Nemotron Nano 12B-v2-VL | **0.00%**    |
+| GPT-5.3                 | **5.33%**    |
+| Qwen 3 (4B)             | 8.00%        |
+| GPT-5 Mini              | 10.67%       |
+| Ministral 3 (3B)        | 18.67%       |
+| GPT-OSS (20B)           | 25.33%       |
+| Phi 4 Mini Instruct     | 46.67%       |
+| Phi 4 Mini Reasoning    | 53.33%       |
+
+
 ## Special features:
 - This project introduces "HarmActEval" dataset and benchmark to evaluate an AI agent's probability of generating harmful actions.
 - The dataset has been used to train a lightweight neural network model that classifies actions as safe, harmful, or unethical.
@@ -88,6 +105,13 @@ source .venv/bin/activate
 uv pip install agent-action-guard
 ```
 
+Install with HarmActEval CLI extras:
+
+```bash
+pip install "agent-action-guard[harmacteval]"
+python -m agent_action_guard.harmacteval --k 3
+```
+
 For usage instructions, kindly refer https://github.com/Pro-GenAI/Agent-Action-Guard/blob/main/USAGE.md.
 
 Note: The embedding client accepts an API key via the `EMBEDDING_API_KEY` environment variable (falls back to `OPENAI_API_KEY` if unset). See `.env.example` and `USAGE.md` for examples.

USAGE.md

@@ -17,6 +17,12 @@ source .venv/bin/activate
 pip install agent-action-guard
 ```
 
+Install with HarmActEval CLI support:
+
+```bash
+pip install "agent-action-guard[harmacteval]"
+```
+
 2. Start an embedding server (if not already running).
 
 Sample embedding server script is available at [examples/scripts/host_models.py](examples/scripts/host_models.py).
@@ -113,6 +119,26 @@ PyPI package scope:
 - `pip install agent-action-guard` installs only the runtime classifier modules and model file needed for action classification.
 - Training, evaluation, MCP demo servers, and UI scripts remain in this repository and require the `dev` extras.
 
+### HarmActEval CLI (standalone)
+
+After installing `agent-action-guard[harmacteval]`, run:
+
+```bash
+python -m agent_action_guard.harmacteval --k 3
+```
+
+Common arguments:
+- `--k`: Number of attempts per prompt (Harm@k).
+- `--offset`: Start index within harmful/unethical rows.
+- `--limit`: Maximum number of harmful/unethical rows to evaluate.
+- `--cache-path`: Path to cache JSON file.
+- `--output`: Path to output JSON file.
+- `--log-level`: `DEBUG|INFO|WARNING|ERROR|CRITICAL`.
+
+Environment variables:
+- Required: `OPENAI_MODEL` and provider credentials (`OPENAI_API_KEY` or Azure equivalents).
+- Optional (MCP mode): `MCP_SUPPORTED`, `MCP_EVAL_SERVER_URL`, `MCP_URL_GUARDED`.
+
 ### Docker Compose
 
 The Docker Compose and manual demo setup below also require a repository checkout.