Skip to content

Add configurable severity thresholds and custom rule profiles #18

@rudra496

Description

@rudra496

Summary

Allow users to create custom rule profiles with configurable severity thresholds.

What to implement

1. Rule Profiles

# .aitrust.yaml
profiles:
  strict:
    min_score: 90
    fail_on: [critical, high, medium]
  standard:
    min_score: 70
    fail_on: [critical, high]
  relaxed:
    min_score: 50
    fail_on: [critical]

2. CLI Usage

aitrust validate src/ --profile strict
aitrust validate src/ --profile standard   # default

3. Per-Category Thresholds

thresholds:
  security:
    fail_below: 80    # Security must score 80+
  hallucination:
    fail_below: 70
  logic:
    fail_below: 60
  best_practices:
    fail_below: 50

4. Custom Severity Override

overrides:
  # Promote bare-except from medium to high in this project
  bare_except: high
  # Demote mutable-default from medium to low
  mutable_default: low

Acceptance Criteria

  • Built-in profiles: strict, standard, relaxed
  • Custom profiles via config file
  • Per-category thresholds work
  • Severity overrides work
  • Tests

Difficulty

Beginner — configuration handling, the validation engine already supports scoring.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions