Phase 6: Guardrails & Validation - Enforce Quality and Safety Constraints

[BPMSoftwareSolutions]

Phase 6: Guardrails & Validation

Overview

Implement comprehensive guardrails and validation mechanisms to ensure resume quality, prevent hallucinations, enforce ATS compatibility, and maintain data integrity throughout the tailoring pipeline.

Objectives

• Implement truth filter guardrail
• Create hallucination detection
• Build JSON schema validation with repair
• Enforce length and ATS formatting constraints
• Demonstrate robust error handling and recovery

Deliverables

1. Truth Filter Guardrail

• Create truth guardrail: src/guardrails/truth_guardrail.py
  • Every bullet must map to a retrieved snippet or numeric evidence
  • Verify each bullet against retrieved snippets
  • Flag unverifiable claims
  • Reject resume if TruthScore < 95%
  • Suggest corrections based on retrieved snippets

2. Hallucination Detection

• Create hallucination detector: src/guardrails/hallucination_detector.py
  • Detect claims that don't match user's actual experience
  • Check for:
    • Skills not in master resume
    • Experience not in experience corpus
    • Metrics that seem inflated
    • Technologies not mentioned in retrieved snippets
  • Flag suspicious claims
  • Suggest alternatives from retrieved snippets

3. JSON Schema Validation with Repair

• Create schema validator: src/guardrails/schema_validator.py
  • Validate resume JSON against strict schema
  • On validation failure:
    • Identify missing/invalid fields
    • Generate repair prompt
    • Re-prompt LLM to fix issues
    • Validate repaired JSON
    • Repeat up to 3 times
  • Fail closed: reject if repair fails

4. Length & ATS Enforcement

• Create ATS enforcer: src/guardrails/ats_enforcer.py
  • Enforce max tokens per section:
    • Summary: 500 chars max
    • Each bullet: 150 chars max
    • Total resume: 5000 chars max
  • Enforce standard section order: summary → skills → experience
  • Enforce keyword coverage: > 90% of JD keywords present
  • Remove special characters that break ATS parsing
  • Enforce consistent formatting (no mixed case, no unicode)

5. Integrated Guardrails Pipeline

• Create guardrails orchestrator: src/guardrails/guardrails_pipeline.py
  • Orchestrate all guardrails in sequence:
    1. JSON schema validation (fail closed)
    2. Truth filter (reject if TruthScore < 95%)
    3. Hallucination detection (flag and suggest corrections)
    4. ATS enforcement (enforce constraints)
  • Track guardrail violations
  • Generate detailed guardrails report

6. Error Handling & Recovery

• Create error handler: src/guardrails/error_handler.py
  • Graceful error handling for all guardrail failures
  • Automatic recovery strategies:
    • Schema validation failure → repair loop
    • Truth filter failure → suggest corrections
    • Hallucination detection → flag and suggest alternatives
    • ATS enforcement failure → auto-correct formatting
  • Fallback to baseline resume if all recovery attempts fail
  • Log all errors and recovery attempts

7. Demonstration & Testing

• Create demo script: scripts/demo_guardrails.py
  • Generate sample tailored resume (with intentional violations)
  • Run through guardrails pipeline
  • Show violations detected
  • Show corrections applied
  • Display before/after comparison
  • Show guardrails report
• Create unit tests: tests/test_guardrails.py
  • Test truth filter accuracy
  • Test hallucination detection
  • Test schema validation and repair
  • Test ATS enforcement
  • Test error handling and recovery
  • Test edge cases (empty fields, invalid JSON, etc.)

Success Criteria

• Truth filter correctly identifies unverifiable claims (> 95% accuracy)
• Hallucination detector catches suspicious claims (> 90% accuracy)
• Schema validation catches all invalid JSON
• Repair loop fixes invalid JSON in < 3 attempts (> 95% success rate)
• ATS enforcement correctly formats resumes
• Guardrails pipeline rejects < 5% of valid resumes
• Error handling recovers from failures > 95% of the time
• Demo shows clear guardrail enforcement
• All unit tests pass
• No hallucinations in final resumes (TruthScore > 95%)
• All resumes pass ATS validation

Files to Create/Modify

New Files:

• src/guardrails/__init__.py
• src/guardrails/truth_guardrail.py
• src/guardrails/hallucination_detector.py
• src/guardrails/schema_validator.py
• src/guardrails/ats_enforcer.py
• src/guardrails/guardrails_pipeline.py
• src/guardrails/error_handler.py
• scripts/demo_guardrails.py
• tests/test_guardrails.py

Modified Files:

• src/tailor.py - Integrate guardrails pipeline
• README.md - Document guardrails

Related Issue

• Parent: Implement LLM Training Strategy: RAG + LoRA Fine-Tuning for Agentic Resume Tailoring #45 - Implement LLM Training Strategy: RAG + LoRA Fine-Tuning
• Depends on: Phase 1: RAG Integration - Index and Retrieve Experience Snippets #46 (Phase 1), Phase 2: LoRA Fine-Tuning - Train on Curated Resume Tailoring Pairs #47 (Phase 2), Phase 3: Critique/Verifier Agent - Truth Filters and Self-Critique Loop #48 (Phase 3), Phase 4: DPO Preference Optimization - Rank and Learn from Resume Variants #49 (Phase 4), Phase 5: Metrics & Monitoring - Track Resume Quality Indicators #50 (Phase 5)

Acceptance Criteria

1. ✅ Truth filter validates bullets against retrieved snippets
2. ✅ Hallucination detector catches suspicious claims
3. ✅ Schema validation catches and repairs invalid JSON
4. ✅ ATS enforcement correctly formats resumes
5. ✅ Error handling recovers from failures > 95% of the time
6. ✅ Guardrails pipeline rejects < 5% of valid resumes
7. ✅ Demo shows clear guardrail enforcement
8. ✅ All tests pass with > 90% accuracy
9. ✅ No hallucinations in final resumes (TruthScore > 95%)
10. ✅ All resumes pass ATS validation
11. ✅ Documentation updated with guardrails examples