Skip to content

Commit 0e924ae

Browse files
rafainnch1kim0n1
andauthored
FEAT: Threshold and ratio configuration and testing file for optimal threshold and ratio configuration (#21)
* improved systems * FIX: Refactor negative_to_positive_ratio parameter to be optional in SentinelLocalIndex. FEAT: created a testing tool for best threshold and ratio analysis * DOCS: Add section on testing optimal thresholds and data ratios in README * FEAT: Added a review mode for detailed information regarding each case with optional flags. DOCS: Updated relevent documentation with these fixes * FEAT: Enhance threshold testing with performance metrics and results-only mode * FEAT: Set default negative_to_positive_ratio to 5.0 and add error handling and fallback for ratio adjustments * FIX: Update user profile creation to exclude sexual content examples Feat: Adjusts message metrics dynamically * FIX: Comment out sexual content examples in user profile creation to prevent inclusion * FEAT: Added caching and caching tests to the main model load, reduced load time of model exponentially after the first caching. TESTS: Updated embedding tests to include caching and its management functions FEAT: Updated the example script for testing purposes to include caching mechanics * DOCS: Fixed formatting for better PEP8 format * STYLE: Apply PEP 8 formatting and update examples - Apply PEP 8 formatting to Example_Threshold_Script.py - Update embeddings.safetensors - Update sentinel_against_hate.ipynb - Fixed line length violations (max 79 characters) - Corrected indentation and spacing - Enhanced readability while maintaining functionality * FEAT: Add caching option to SentinelLocalIndex for improved load performance * REFACTOR: Refactor user profile creation to use external test data for speech examples * TEST: Update mean_of_positives tests to return 0.0 for negative and empty score arrays, fixes edge case, NaN returns. * CHORE: Remove redundant Cache_Model variable in `calculate_rare_class_affinity`, update example file to use path/to/index rather than local path * TEST: Add edge case tests for aggregation functions and contrastive_components in score_formulae and SentinelLocalIndex * Chore: removed redundant imports * chore: Changed caching to be false by default, to preserve old functionality, removed redundant exports, added no-cache flag to the testing script * All requested changes made * fix capitalisation issue * The should fix thee build issues for CI tests as pandas ^1.0.0 didn't have support for PEP 517 builds hence swapped to ^2.0.0 which is compatable - may require further testing however didn't impact functionality of code --------- Co-authored-by: Vladislav Kondratyev <chikimoni61@gmail.com>
1 parent e80915d commit 0e924ae

14 files changed

Lines changed: 1783 additions & 76 deletions

README.md

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,22 @@ Roblox Sentinel, part of the Roblox Safety Toolkit, is a Python library designed
1111

1212
By prioritizing recall over precision, Sentinel serves as a high-recall candidate generator for more thorough investigation. This approach is particularly effective for applications where rare patterns are critical to identify. Rather than treating each message in isolation, Sentinel analyzes patterns across messages to identify concerning behavior.
1313

14+
## What’s New: Aggregation options and Explainability
15+
16+
Sentinel now includes multiple aggregation strategies and built‑in explainability to help you tune for your use case and understand why a score was assigned.
17+
18+
- Aggregators (in `sentinel.score_formulae`):
19+
- `skewness(scores, min_size_of_scores=10)`: default, pattern‑oriented and robust to message count
20+
- `top_k_mean(scores, k=3)`: focuses on the strongest signals
21+
- `percentile_score(scores, q=90.0)`: robust to outliers via a percentile over positives
22+
- `softmax_weighted_mean(scores, temperature=1.0)`: smoothly emphasizes higher scores
23+
- `max_score(scores)`: simplest, picks the highest positive score
24+
25+
- Explainability (in results):
26+
- Each call to `calculate_rare_class_affinity` returns a `RareClassAffinityResult` with:
27+
- `aggregation_name`, `aggregation_stats`: which aggregator was used and key params
28+
- `explanations`: per‑text details including top‑K positive/negative similarities, contrastive components, and neighbor snippets (when available)
29+
1430
## Terminology
1531

1632
In Sentinel's codebase:
@@ -65,6 +81,16 @@ print(f"Overall rare class affinity score: {overall_score:.4f}")
6581
for message, score in result.observation_scores.items():
6682
risk_level = "High" if score > 0.5 else "Medium" if score > 0.1 else "Low"
6783
print(f"'{message}' - Score: {score:.4f} - Risk: {risk_level}")
84+
85+
# Inspect explainability
86+
print("Aggregator:", result.aggregation_name)
87+
print("Aggregation stats:", result.aggregation_stats)
88+
for message, ex in result.explanations.items():
89+
print("--", message)
90+
print(" topk_positive:", ex["topk_positive"]) # scaled similarities
91+
print(" topk_negative:", ex["topk_negative"]) # scaled similarities
92+
print(" contrastive:", ex["contrastive"]) # positive_term, negative_term, log_ratio_unclipped
93+
print(" neighbors (sample):", ex["neighbors"][:2] if ex["neighbors"] else None)
6894
```
6995

7096
## Creating a New Index
@@ -109,6 +135,37 @@ saved_config = index.save(
109135
aws_access_key_id="YOUR_ACCESS_KEY_ID", # Optional if using environment credentials
110136
aws_secret_access_key="YOUR_SECRET_ACCESS_KEY" # Optional if using environment credentials
111137
)
138+
139+
## Testing for optimal Thresholds and data ratio's
140+
141+
Usage of the 'examples\Example_Threshold_Script.py' script will allow for quick threshold checks for a variety of ratios, by default these are 10:1, 5:1 and 1:1 ratios. This has predefined example chat logs, and should, show optimal settings for the dataset being used based on an average score and average detection count.
142+
You will be able to get detailed information output in the ratios with the -r or --review flags.
143+
144+
## Choosing an aggregation strategy
145+
146+
Different deployments optimize for different trade‑offs. You can swap in any aggregator using the `aggregation_function` argument:
147+
148+
```python
149+
from sentinel.score_formulae import top_k_mean, percentile_score, softmax_weighted_mean, max_score
150+
151+
texts = ["msg a", "msg b", "msg c"]
152+
153+
# Focus on the strongest few signals
154+
res1 = index.calculate_rare_class_affinity(texts, aggregation_function=lambda arr: top_k_mean(arr, k=3))
155+
156+
# Robust to outliers
157+
res2 = index.calculate_rare_class_affinity(texts, aggregation_function=lambda arr: percentile_score(arr, q=90))
158+
159+
# Smoothly emphasize higher scores
160+
res3 = index.calculate_rare_class_affinity(texts, aggregation_function=lambda arr: softmax_weighted_mean(arr, temperature=0.5))
161+
162+
# Simplest, picks the maximum
163+
res4 = index.calculate_rare_class_affinity(texts, aggregation_function=max_score)
164+
```
165+
166+
Notes:
167+
- All aggregators operate over per‑observation scores where non‑confident observations are already clipped to 0.
168+
- The default `skewness` remains a good choice when user activity volume varies widely.
112169
```
113170

114171
## How It Works

0 commit comments

Comments
 (0)