You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# In PR, comment "@coderabbitai configuration" to get the full config including defaults
5
+
# Set the language for reviews by using the corresponding ISO language code.
6
+
# Default: "en-US"
7
+
language: "en-US"
8
+
# Settings related to reviews.
9
+
# Default: {}
10
+
reviews:
11
+
# Set the profile for reviews. Assertive profile yields more feedback, that may be considered nitpicky.
12
+
# Options: chill, assertive
13
+
# Default: "chill"
14
+
profile: chill
15
+
# Add this keyword in the PR/MR title to auto-generate the title.
16
+
# Default: "@coderabbitai"
17
+
auto_title_placeholder: "@coderabbitai title"
18
+
# Auto Title Instructions - Custom instructions for auto-generating the PR/MR title.
19
+
# Default: ""
20
+
auto_title_instructions: 'Format: "<category>: <title>". Category must be one of: feat, fix, docs, style, refactor, perf, test, build, ci, chore, revert, cp. The category must be followed by a colon. Title should be concise (<= 80 chars). Example: "feat: Add logit_bias support".'# current: ''
21
+
# Set the commit status to 'pending' when the review is in progress and 'success' when it is complete.
22
+
# Default: true
23
+
commit_status: false
24
+
# Generate walkthrough in a markdown collapsible section.
25
+
# Default: false
26
+
collapse_walkthrough: true
27
+
# Generate an assessment of how well the changes address the linked issues in the walkthrough.
28
+
# Default: true
29
+
assess_linked_issues: true
30
+
# Include possibly related issues in the walkthrough.
31
+
# Default: true
32
+
related_issues: true
33
+
# Related PRs - Include possibly related pull requests in the walkthrough.
34
+
# Default: true
35
+
related_prs: true
36
+
# Suggest labels based on the changes in the pull request in the walkthrough.
37
+
# Default: true
38
+
suggested_labels: true
39
+
# Suggest reviewers based on the changes in the pull request in the walkthrough.
40
+
# Default: true
41
+
suggested_reviewers: true
42
+
# Generate a poem in the walkthrough comment.
43
+
# Default: true
44
+
poem: false # current: true
45
+
# Post review details on each review. Additionally, post a review status when a review is skipped in certain cases.
46
+
# Default: true
47
+
review_status: false # current: true
48
+
# Configuration for pre merge checks
49
+
# Default: {}
50
+
pre_merge_checks:
51
+
# Custom Pre-merge Checks - Add unique checks to enforce your team's standards before merging a pull request. Each check must have a unique name (up to 50 characters) and clear instructions (up to 10000 characters). Use these to automatically verify coding, security, documentation, or business rules and maintain code quality.
52
+
# Default: []
53
+
custom_checks:
54
+
- name: "Test Results for Major Changes"
55
+
mode: "warning"# or "error" to block merges
56
+
instructions: |
57
+
If this PR contains major changes (such as new features, breaking changes, or significant refactoring), verify that the PR description includes test results or testing information.
58
+
If a change could affect numerics or convergence, the PR description should include information demonstrating that there is no regression.
59
+
If a change could affect performance, the PR description should include before-and-after performance numbers, as well as the configuration and context in which they apply.
60
+
Pass if test results are documented or if the changes are minor.
61
+
auto_review:
62
+
# Configuration for auto review
63
+
# Default: {}
64
+
# Automatic Incremental Review - Automatic incremental code review on each push
65
+
# Default: true
66
+
auto_incremental_review: false # current: true
67
+
# Review draft PRs/MRs.
68
+
# Default: false
69
+
drafts: false
70
+
# Base branches (other than the default branch) to review. Accepts regex patterns. Use '.*' to match all branches.
-*(jailbreak)* Validate Jailbreak Detection config at create-time ([#1675](https://github.com/NVIDIA-NeMo/Guardrails/issues/1675))
28
+
-*(library)* Add PolicyAI Integration for Content Moderation ([#1576](https://github.com/NVIDIA-NeMo/Guardrails/issues/1576))
29
+
30
+
### 🐛 Bug Fixes
31
+
32
+
-*(server)* Make openai an optional server-only dependency ([#1623](https://github.com/NVIDIA-NeMo/Guardrails/issues/1623))
33
+
-*(actions)* Rename generate_next_step to generate_next_steps for task-specific LLM support ([#1603](https://github.com/NVIDIA-NeMo/Guardrails/issues/1603))
34
+
-*(library)* Add `valid` alias to action results in GuardrailsAI integration ([#1578](https://github.com/NVIDIA-NeMo/Guardrails/issues/1578)) ([#1611](https://github.com/NVIDIA-NeMo/Guardrails/issues/1611))
35
+
-*(llm)* Filter stop parameter for OpenAI reasoning models ([#1653](https://github.com/NVIDIA-NeMo/Guardrails/issues/1653))
36
+
-*(logging)* Show cache hits in Stats log and fix duplicate metadata restore ([#1666](https://github.com/NVIDIA-NeMo/Guardrails/issues/1666))
37
+
-*(cache)* Make cache stats log visible in verbose mode ([#1667](https://github.com/NVIDIA-NeMo/Guardrails/issues/1667))
38
+
-*(library)* Use bot refuse to respond in gliner PII detection flows ([#1671](https://github.com/NVIDIA-NeMo/Guardrails/issues/1671))
39
+
-*(streaming)* Handle None stop tokens in streaming handler ([#1685](https://github.com/NVIDIA-NeMo/Guardrails/issues/1685))
40
+
-*(streaming)* Handle dict chunks in RollingBuffer.format_chunks ([#1687](https://github.com/NVIDIA-NeMo/Guardrails/issues/1687))
41
+
-*(middleware)* Handle MODIFIED status in GuardrailsMiddleware instead of silently dropping it ([#1714](https://github.com/NVIDIA-NeMo/Guardrails/issues/1714))
42
+
43
+
### 🚜 Refactor
44
+
45
+
-*(streaming)* Remove LangChain callback dependencies from StreamingHandler ([#1547](https://github.com/NVIDIA-NeMo/Guardrails/issues/1547))
> **LATEST RELEASE / DEVELOPMENT VERSION**: The [main](https://github.com/NVIDIA-NeMo/Guardrails/tree/main) branch tracks the latest released beta version: [0.20.0](https://github.com/NVIDIA-NeMo/Guardrails/tree/v0.20.0). For the latest development version, checkout the [develop](https://github.com/NVIDIA-NeMo/Guardrails/tree/develop) branch.
16
+
> **LATEST RELEASE / DEVELOPMENT VERSION**: The [main](https://github.com/NVIDIA-NeMo/Guardrails/tree/main) branch tracks the latest released beta version: [0.21.0](https://github.com/NVIDIA-NeMo/Guardrails/tree/v0.21.0). For the latest development version, checkout the [develop](https://github.com/NVIDIA-NeMo/Guardrails/tree/develop) branch.
17
17
18
18
✨✨✨
19
19
@@ -295,9 +295,9 @@ Evaluating the safety of a LLM-based conversational application is a complex tas
295
295
296
296
## How is this different?
297
297
298
-
There are many ways guardrails can be added to an LLM-based conversational application. For example: explicit moderation endpoints (e.g., OpenAI, ActiveFence), critique chains (e.g. constitutional chain), parsing the output (e.g. guardrails.ai), individual guardrails (e.g., LLM-Guard), hallucination detection for RAG applications (e.g., Got It AI, Patronus Lynx).
298
+
There are many ways guardrails can be added to an LLM-based conversational application. For example: explicit moderation endpoints (e.g., OpenAI, ActiveFence, PolicyAI), critique chains (e.g. constitutional chain), parsing the output (e.g. guardrails.ai), individual guardrails (e.g., LLM-Guard), hallucination detection for RAG applications (e.g., Got It AI, Patronus Lynx).
299
299
300
-
NeMo Guardrails aims to provide a flexible toolkit that can integrate all these complementary approaches into a cohesive LLM guardrails layer. For example, the toolkit provides out-of-the-box integration with ActiveFence, AlignScore and LangChain chains.
300
+
NeMo Guardrails aims to provide a flexible toolkit that can integrate all these complementary approaches into a cohesive LLM guardrails layer. For example, the toolkit provides out-of-the-box integration with ActiveFence, PolicyAI, AlignScore and LangChain chains.
301
301
302
302
To the best of our knowledge, NeMo Guardrails is the only guardrails toolkit that also offers a solution for modeling the dialog between the user and the LLM. This enables on one hand the ability to guide the dialog in a precise way. On the other hand it enables fine-grained control for when certain guardrails should be used, e.g., use fact-checking only for certain types of questions.
Copy file name to clipboardExpand all lines: benchmark/aiperf/README.md
+25-24Lines changed: 25 additions & 24 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -27,14 +27,15 @@ To use the provided configurations, you need to create accounts at <https://buil
27
27
1.**Create a virtual environment in which to install AIPerf**
28
28
29
29
```bash
30
-
$ mkdir ~/env
31
-
$ python -m venv ~/env/aiperf
30
+
mkdir ~/env
31
+
python -m venv ~/env/aiperf
32
+
source~/env/aiperf/bin/activate
32
33
```
33
34
34
35
2.**Install dependencies in the virtual environment**
35
36
36
37
```bash
37
-
$ pip install aiperf huggingface_hub typer
38
+
pip install aiperf huggingface_hub typer httpx
38
39
```
39
40
40
41
3.**Login to Hugging Face:**
@@ -50,39 +51,39 @@ To use the provided configurations, you need to create accounts at <https://buil
50
51
After creating a Personal API key, set the `NVIDIA_API_KEY` variable as below.
51
52
52
53
```bash
53
-
$ export NVIDIA_API_KEY="your-api-key-here"
54
+
export NVIDIA_API_KEY="your-api-key-here"
54
55
```
55
56
56
57
## Running Benchmarks
57
58
58
59
Each benchmark is configured using the `AIPerfConfig` Pydantic model in [aiperf_models.py](aiperf_models.py).
59
60
The configs are stored in YAML files, and converted to an `AIPerfConfig` object.
60
-
There are two example configs included which can be extended for your use-cases. These both use Nvidia-hosted models, :
61
+
There are two example configs included which can be extended for your use-cases. These both use Nvidia-hosted models:
61
62
62
-
-[`single_concurrency.yaml`](aiperf_configs/single_concurrency.yaml): Example single-run benchmark with a single concurrency value.
63
-
-[`sweep_concurrency.yaml`](aiperf_configs/sweep_concurrency.yaml): Example multiple-run benchmark to sweep concurency values and run a new benchmark for each.
63
+
-[`single_concurrency.yaml`](configs/single_concurrency.yaml): Example single-run benchmark with a single concurrency value.
64
+
-[`sweep_concurrency.yaml`](configs/sweep_concurrency.yaml): Example multiple-run benchmark to sweep concurrency values and run a new benchmark for each.
Configuration files are YAML files located in [aiperf_configs](aiperf_configs). The configuration is validated using Pydantic models to catch errors early.
169
+
Configuration files are YAML files located in [configs](configs). The configuration is validated using Pydantic models to catch errors early.
169
170
170
171
### Top-Level Configuration Fields
171
172
172
173
| Field | Type | Required | Description |
173
174
|-------|------|----------|-------------|
174
-
|`batch_name`| string |Yes| Name for this batch of benchmarks. Used in output directory naming (e.g., `aiperf_results/batch_name/timestamp/`) |
175
-
|`output_base_dir`| string |Yes| Base directory where all benchmark results will be stored |
175
+
|`batch_name`| string |No| Name for this batch of benchmarks. Used in output directory naming (e.g., `aiperf_results/batch_name/timestamp/`). Default: `benchmark`|
176
+
|`output_base_dir`| string |No| Base directory where all benchmark results will be stored. Default: `aiperf_results`|
176
177
|`base_config`| object | Yes | Base configuration parameters applied to all benchmark runs (see below) |
177
178
|`sweeps`| object | No | Optional parameter sweeps for running multiple benchmarks with different values |
178
179
@@ -355,11 +356,11 @@ Each run directory contains multiple files with benchmark results and metadata.
355
356
- **`inputs.json`**: Synthetic prompt data generated for the benchmark.
356
357
- **`profile_export_aiperf.json`**: Main metrics file in JSON format containing aggregated statistics.
357
358
- **`profile_export_aiperf.csv`**: Same metrics as the JSON file, but in CSV format for easy import into spreadsheet tools or data analysis libraries.
358
-
- **`profile_export.jsonl`**: JSON Lines format file containing per-request metrics. Each line is a complete JSON object for one request with:
359
-
- **`logs/aiperf.log`**: Detailed log file from AIPerf execution containing:
359
+
- **`profile_export.jsonl`**: JSON Lines format file containing per-request metrics. Each line is a complete JSON object for one request.
360
+
- **`logs/aiperf.log`**: Detailed log file from AIPerf execution.
0 commit comments