Merge branch 'feature/technique_intent' into feature/unified-run-spec

Author: patriciapampanelli

AGENTS.md

@@ -1,13 +1,71 @@
-# garak - generative AI red-teaming and assessment toolkit
+# Agent instructions for garak - generative AI red-teaming and assessment toolkit
+
+> These instructions apply to **all** AI-assisted contributions to `nvidia/garak`.
+> Breaching these guidelines can result in automatic banning.
 
 This is tooling for adversarial assessment of LLMs and LLM-powered tools.
 It's open source software, used in production environments, with an active and skilled community
 As such, the code needs to be robust, precise, and responsible.
 Due to the nature of the project, there is a lot of potentially harmful or dangerous data associated with the repository.   
 
-## Coding guide
-- Read the documentation in `docs/`, especially the content on extending garak.
-- Avoid adding new dependencies.
+## Contribution policy
+
+### Duplicate-work checks
+
+Before proposing a PR, run these checks:
+
+```bash
+gh issue view <issue_number> --repo nvidia/garak --comments
+gh pr list --repo nvidia/garak --state open --search "<issue_number> in:body"
+gh pr list --repo nvidia/garak --state open --search "<short area keywords>"
+```
+
+- If an open PR already addresses the same fix, do not open another.
+- If your approach is materially different, explain the difference in the issue.
+
+### No low-value busywork PRs
+
+Do not open one-off PRs for tiny edits (single typo, isolated style change, one mutable default, etc.). Mechanical cleanups are acceptable only when bundled with substantive work.
+
+### Cohesive PRs
+
+Each PR should have a clear focus. Only update files related to the topic of that PR.
+
+### Accountability
+
+- Pure code-agent PRs are **not allowed**. A human submitter must understand and defend the change end-to-end.
+- The submitting human must review every changed line and run relevant tests.
+- PR descriptions for AI-assisted work **must** include:
+    - Why this is not duplicating an existing PR.
+    - Test commands run and results.
+    - Clear statement that AI assistance was used.
+
+### Fail-closed behavior
+
+If work is duplicate/trivial busywork, **do not proceed**. Return a short explanation of what is missing.
+
+### Project Guides
+
+Check the docs on "Contributing" and "Extending", in `docs/source`, and follow these.
+
+### Commit messages
+
+Add attribution using commit trailers such as `Co-authored-by:` (other projects use `Assisted-by:` or `Generated-by:`). For example:
+
+```text
+Your commit message here
+
+Co-authored-by: GitHub Copilot
+Co-authored-by: Claude
+Co-authored-by: gemini-code-assist
+Signed-off-by: Your Name <your.email@example.com>
+```
+
+
+## Development requirements
+
+### Coding guide
+- Always avoid adding new dependencies. Use the `extra_dependency_names` functionality if essential.
 - Keep documentation of garak architecture in the docs/ dir up to date - though use docstrings in the first instance if possible.
 - When working on probes, detectors, or buffs, be sure to check the content of the relevant `doc_uri` to understand the code's intent and the underlying technique.
 - Use the payloads, data, and services mechanisms when suitable.
@@ -17,21 +75,25 @@ Due to the nature of the project, there is a lot of potentially harmful or dange
 - Comply with docstring requirements - see the docs and also `tests/test_docs.py`.
 - Catch specific exception types; avoid `except Exception` and bare `except:`.
 
-## Dev environment tips
+### Dev environment tips
 - Use (and expect) only Python versions specified in `pyproject.toml`.
 - Be sure you're using the right environment, with the right dependencies. Virtual environment management is preferred.
 
-## Testing instructions
+### Testing instructions
 - Don't break existing tests.
 - Add tests as you go.
 - Tests for specific modules should go in a new file. For example, tests for `garak.probes.xyz` should go in `tests/probes/test_probe_xyz.py`.
 - ARM, x86, and Windows all need to be supported - check the list of supported architectures in `pyproject.toml`.
+- Don't add tests for default values given in configurable plugins.
+- Don't add tests for functionality already covered by tests of parent classes.
+- Add descriptive strings to asserts, explaining the expect underlying behaviour; be terse.
+- Check that tests work. If `pytest` or other project dependencies are not available, the environment has not been set up correctly; give the user this problem.
 
-## Code primitives
+### Code primitives
 - Avoid updating `attempt` or any base classes (`probes.base.*`, `generators.base.*`, `detectors.base.*`) frivolously.
 - Consider using a service for content to be available across all of garak over a whole run.
 
-## Style
+### Style
 - Comply with any repository formatting and linting config in `pyproject.toml`.
 - Don't include default values in docstrings; the code is the documentation for these.
 - Follow the visual design language of CLI output, including emojis and colour changes where in line with existing style.

docs/source/conf.py

@@ -179,5 +179,5 @@
 import os
 import sys
 
-sys.path.insert(0, "../..")
+sys.path.insert(0, os.path.abspath("../.."))
 sys.path.append(os.path.abspath("./_ext"))

docs/source/extending.probe.rst

@@ -95,6 +95,10 @@ From the base class:
     tags: Iterable[str] = []
     # what the probe is trying to do, phrased as an imperative
     goal: str = ""
+    # the target behaviour / failure mode this probe elicits,
+    # as a code from the trait typology (garak/data/cas/trait_typology.json).
+    # Propagated to every Attempt minted by the probe.
+    intent: Union[str, None] = None
     # Deprecated -- the detectors that should be run for this probe. always.Fail is chosen as default to send a signal if this isn't overridden.
     recommended_detector: Iterable[str] = ["always.Fail"]
     # default detector to run, if the primary/extended way of doing it is to be used (should be a string formatted like recommended_detector)
@@ -118,6 +122,7 @@ Many of these are decent defaults, though there are a few that we absolutely wan
 * ``active``: Should this probe be part of default scans?
 * ``tags``: MISP-formatted taxonomy categories __e.g.__ ``["avid-effect:security:S0403", "owasp:llm01", "quality:Security:PromptStability", "payload:jailbreak"]``
 * ``goal``: What is the probe trying to do? __e.g.__ ``"disregard the system prompt"``
+* ``intent``: Which target behaviour / failure mode does this probe elicit? Must be a code from ``garak/data/cas/trait_typology.json`` __e.g.__ ``"T009ignore"``. The value is automatically propagated to every ``Attempt`` the probe creates. If the probe loads a payload that also declares an ``intent``, the payload's intent takes priority.
 * ``primary_detector``: What ``Detector`` should your probe use?
 
 .. code-block:: python
@@ -133,6 +138,7 @@ Many of these are decent defaults, though there are a few that we absolutely wan
             "payload:jailbreak",
         ]
         goal = "disregard the system prompt"
+        intent = "T009ignore"  # the target behaviour elicited, from trait_typology.json
         doc_uri = "https://garak.ai"
         active = False
         ...

docs/source/generators/llm.rst

@@ -0,0 +1,23 @@
+garak.generators.llm
+====================
+
+Wraps Simon Willison's ``llm`` library, which provides a unified interface
+to many LLM providers (OpenAI, Claude, Gemini, Ollama, etc.) via plugins.
+
+API keys and provider setup are managed by ``llm`` directly. Install the
+base package and any provider plugins you need:
+
+.. code-block:: bash
+
+   pip install llm llm-claude-3
+
+Then invoke garak with the ``llm`` model id:
+
+.. code-block:: bash
+
+   garak --model_type llm --model_name gpt-4o-mini
+
+.. automodule:: garak.generators.llm
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/index_generators.rst

@@ -20,6 +20,7 @@ For a detailed oversight into how a generator operates, see :doc:`generators/bas
    generators/langchain
    generators/langchain_serve
    generators/litellm
+   generators/llm
    generators/mistral
    generators/ollama
    generators/openai

docs/source/payloads.rst

@@ -25,6 +25,7 @@ The JSON structure of a payload is:
         "garak_payload_name": // a mandatory key, used to identify this as a garak payload. holds a description of the payload.
         "payload_types": // a list of strings, each identifying an entry in the payload typology (typology_payloads.tsv)
             ["Security circumvention instructions/Product activation codes"],
+        "intent": "S003productkeys", // optional; a code from the trait typology (garak/data/cas/trait_typology.json) describing the target behaviour these payloads are designed to elicit. When set, probes using this payload will propagate the intent to each Attempt, overriding the probe's default intent.
         "detector_name": "productkey.Win5x5", // a suggested detector
         "detector_config": {}, // a config dict of Configurable options for the detector
         "payloads": [ // a list of strings: the payloads themselves