Skip to content

Commit 8670c91

Browse files
authored
docs: guardrail config's streamProcessingMode (#564)
1 parent a740efa commit 8670c91

2 files changed

Lines changed: 6 additions & 0 deletions

File tree

content/docs/configuration/librechat_yaml/object_structure/aws_bedrock.mdx

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ endpoints:
1919
guardrailIdentifier: "your-guardrail-id"
2020
guardrailVersion: "1"
2121
trace: "enabled"
22+
streamProcessingMode: "sync"
2223
```
2324
2425
> **Note:** AWS Bedrock endpoint supports all [Shared Endpoint Settings](/docs/configuration/librechat_yaml/object_structure/shared_endpoint_settings), including `streamRate`, `titleModel`, `titleMethod`, `titlePrompt`, `titlePromptTemplate`, and `titleEndpoint`. The settings shown below are specific to Bedrock or have Bedrock-specific defaults.
@@ -148,6 +149,7 @@ endpoints:
148149
['guardrailIdentifier', 'String', 'The unique identifier of the guardrail to apply.', 'Required when using guardrails.'],
149150
['guardrailVersion', 'String', 'The version of the guardrail to use.', 'Required when using guardrails.'],
150151
['trace', 'String', 'Controls guardrail trace output for debugging. Options: "enabled", "enabled_full", or "disabled".', 'Optional. Default: "disabled"'],
152+
['streamProcessingMode', 'String', 'Controls guardrail stream processing mode. Options: "sync" or "async".', 'Optional. Default: "sync"'],
151153
]}
152154
/>
153155

@@ -159,12 +161,14 @@ endpoints:
159161
guardrailIdentifier: "abc123xyz"
160162
guardrailVersion: "1"
161163
trace: "enabled"
164+
streamProcessingMode: "sync"
162165
```
163166

164167
**Notes:**
165168
- Guardrails help ensure responsible AI usage by filtering harmful content, PII, and other sensitive information
166169
- The `guardrailIdentifier` can be found in the AWS Bedrock console under Guardrails
167170
- Set `trace` to `"enabled"` or `"enabled_full"` during development to see which guardrail policies are triggered
171+
- Set `streamProcessingMode` to `"async"` to stream responses faster (at the cost of guardrail possibly allowing inappropriate content through until its scan completes)
168172
- For production, set `trace` to `"disabled"` to reduce response payload size
169173

170174
## Notes

content/docs/configuration/pre_configured_ai/bedrock.mdx

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,7 @@ endpoints:
6464
guardrailIdentifier: "abc123xyz"
6565
guardrailVersion: "1"
6666
trace: "enabled"
67+
streamProcessingMode: "sync"
6768
```
6869
6970
- `streamRate`: (Optional) Set the rate of processing each new token in milliseconds.
@@ -81,6 +82,7 @@ endpoints:
8182
- `guardrailIdentifier`: The guardrail ID or ARN from your AWS Bedrock Console.
8283
- `guardrailVersion`: The guardrail version number (e.g., `"1"`) or `"DRAFT"`.
8384
- `trace`: (Optional) Enable trace logging: `"enabled"`, `"disabled"`, or `"enabled_full"`.
85+
- `streamProcessingMode`: (Optional) Set stream processing mode: `"sync"` or `"async"` (defaults to `"sync"`).
8486
- See [AWS Bedrock Guardrails documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-how.html) for creating and managing guardrails.
8587

8688
## Inference Profiles

0 commit comments

Comments
 (0)