Skip to content

Latest commit

 

History

History
488 lines (341 loc) · 11.3 KB

File metadata and controls

488 lines (341 loc) · 11.3 KB

Thread-Sentry

High-Performance Deadlock and Race Condition Detection Engine
Production-ready concurrent safety monitoring with < 5% overhead

License Rust Performance

中文文档 | Usage Guide | Performance Report


🎯 Why Thread-Sentry?

The Problem

Writing concurrent programs in Rust/C++/Go is challenging. The two most dreaded issues:

  • Deadlocks: Program freezes after running for days
  • Data Races: Data corruption without any visible symptoms

Current Solutions

ThreadSanitizer (TSan):

  • ✅ Detects issues
  • 10-50x performance overhead
  • 5-10x memory overhead
  • Cannot deploy to production

The Gap: No tool exists for real-time production monitoring of concurrent safety issues.

Thread-Sentry's Solution

< 5% performance overhead - Production-ready
Real-time detection - Immediate alerts
Precise localization - File, line, thread
Zero-intrusion - Just replace Mutex type
Comprehensive - Both deadlock and race detection


🚀 Quick Start

Installation

[dependencies]
thread-sentry = "0.1"

Three Ways to Detect Races

1️⃣ Guard Auto-Detection (Simplest)

use thread_sentry::{Mutex, init, report_issues};

fn main() {
    init();
    
    let data = Arc::new(Mutex::new(0u64));
    
    // Just replace std::sync::Mutex → thread_sentry::Mutex
    thread::spawn(|| {
        let mut guard = data.lock();
        *guard = 100;  // Auto-detect + Auto-print
    });
    
    report_issues();
}

2️⃣ SentryField Tracking (Recommended)

use thread_sentry::{Mutex, SentryField, init};

struct SharedData {
    counter: SentryField<u64>,  // Auto-track field access
}

fn main() {
    init();
    
    let data = Arc::new(Mutex::new(SharedData::new()));
    
    thread::spawn(|| {
        let mut guard = data.lock();
        guard.counter.set(100);  // Auto-detect + Auto-print
    });
    
    report_issues();
}

3️⃣ Manual Registration (unsafe code)

use thread_sentry::{RaceDetector, AccessType};

unsafe {
    *raw_ptr = value;
    
    // Manual registration for unsafe code
    RaceDetector::record_access_manual(
        addr, thread_id, AccessType::Write, lock_id, size
    );
}

See USAGE_GUIDE.md for detailed examples.


📊 Performance

Comparison with Alternatives

Tool Performance Overhead Memory Overhead Production Ready
TSan 500-5000% 5-10x ❌ No
Helgrind 2000-3000% 3-5x ❌ No
Thread-Sentry < 5% < 2x Yes

Benchmarks

Metric std::sync parking_lot Thread-Sentry
Latency 50ns 38ns (-24%) 48ns (-4%)
Throughput 20M ops/sec 26M ops/sec (+30%) 21M ops/sec (+5%)

Key Finding: Thread-Sentry has only 4% overhead vs std::sync::Mutex.

See Performance Report for detailed benchmarks.


✨ Features

1. Deadlock Detection

Automatically detects circular lock dependencies:

let lock1 = Mutex::new(0);
let lock2 = Mutex::new(0);

// Thread 1: lock1 -> lock2
// Thread 2: lock2 -> lock1
// Thread-Sentry detects the cycle and reports immediately!

Output:

╔══════════════════════════════════════════════════════════╗
║ ⚠️  DEADLOCK DETECTED                                    ║
╚══════════════════════════════════════════════════════════╝

Cycle Length: 2 locks
Lock Chain:
  [1] Lock #1 held by Thread 1
    Backtrace:
      1. main.rs:15 - transfer_money()
      
  [2] Lock #2 held by Thread 2
    Backtrace:
      1. main.rs:22 - process_transaction()

2. Race Condition Detection

Detects unsynchronized concurrent memory access:

// Thread 1: write without lock
x = 100;  // Write, no lock

// Thread 2: read without lock
read x;   // Read, no lock

// Thread-Sentry detects the race condition!

Output:

╔══════════════════════════════════════════════════════════╗
║ ⚡ RACE CONDITION DETECTED                               ║
╚══════════════════════════════════════════════════════════╝

Memory Address: 0x7f8a3c001000

Access 1: Write at bank.rs:30 (Thread 1, no lock)
Access 2: Read at bank.rs:45 (Thread 2, no lock)

🏗️ Architecture

Thread-Sentry is built on three layers:

┌─────────────────────────────────────────┐
│        Application Layer                 │
│  SentinelMutex / SentinelRwLock         │  ← Replace standard locks
├─────────────────────────────────────────┤
│        Monitoring Layer                  │
│  - Lock event tracking                   │  ← Intercept lock/unlock
│  - Thread state management               │
│  - Dependency graph construction         │
├─────────────────────────────────────────┤
│        Detection Layer                   │  ← Real-time analysis
│  - Deadlock detector (cycle detection)   │
│  - Race detector (access tracking)       │
└─────────────────────────────────────────┘

See Architecture Documentation for details.


📖 Documentation


🔧 Advanced Usage

Selective Monitoring

// Development: Enable full monitoring
#[cfg(debug_assertions)]
use thread_sentry::Mutex;

// Production: Selective monitoring
#[cfg(not(debug_assertions))]
{
    thread_sentry::init();
    // Use Thread-Sentry for critical locks only
    // Use parking_lot for high-frequency paths
}

Custom Configuration

// Initialize with custom settings
thread_sentry::init();

// Use different lock types
let critical_lock = Mutex::new(data);  // Full monitoring
let fast_lock = RwLock::new(cache);    // Read-write monitoring

🧪 Testing

Run Tests

cargo test

Run Examples

# Demo
cargo run --example demo

# Benchmark
cargo run --example benchmark

# Real-world scenario
cargo run --example real_world

Performance Testing

# See docs/PERFORMANCE_TESTING_GUIDE.md for details
scripts\run_benchmarks.bat

🤝 Comparison with parking_lot

parking_lot deadlock_detection

Advantages:

  • ✅ < 1% overhead (very low)
  • ✅ Basic deadlock detection

Limitations:

  • ❌ No race condition detection
  • ❌ 10-second polling delay
  • ❌ Limited information (only lock IDs)
  • ❌ Not suitable for CI/CD

Thread-Sentry

Advantages:

  • ✅ Real-time detection (0 delay)
  • ✅ Race condition detection
  • ✅ Precise localization (file, line, thread)
  • ✅ Suitable for development and CI/CD
  • ✅ Production-ready (< 5% overhead)

Trade-off:

  • Higher overhead (6% vs < 1%)

Best Practice

Use both:

  • Development: Thread-Sentry (comprehensive diagnosis)
  • Production: parking_lot (long-term monitoring)
  • Critical paths: Thread-Sentry (safety priority)
  • High-frequency: parking_lot (performance priority)

🎯 Use Cases

Ideal Scenarios

High-concurrency services

  • Web servers, API endpoints
  • Database connection pools
  • Message queue systems

Long-running applications

  • Background services
  • Scheduled tasks
  • Event-driven systems

Critical business logic

  • Financial transactions
  • Inventory management
  • Order processing

Development and debugging

  • Real-time feedback
  • CI/CD integration
  • Early problem detection

Not Suitable For

  • Single-threaded applications
  • Extremely performance-sensitive scenarios (< 5% overhead unacceptable)
  • Already using other detection tools

📦 Examples

Banking System

use thread_sentry::Mutex;
use std::sync::Arc;

struct BankAccount {
    id: u64,
    balance: f64,
}

fn transfer(from: Arc<Mutex<BankAccount>>, to: Arc<Mutex<BankAccount>>, amount: f64) {
    let from_account = from.lock();
    let to_account = to.lock();  // Thread-Sentry checks for deadlock
    
    from_account.balance -= amount;
    to_account.balance += amount;
}

Producer-Consumer

use thread_sentry::Mutex;

let buffer = Arc::new(Mutex::new(Vec::new()));

// Producer
thread::spawn(|| {
    let mut buf = buffer.lock();
    buf.push(item);
});

// Consumer
thread::spawn(|| {
    let mut buf = buffer.lock();
    if let Some(item) = buf.pop() {
        process(item);
    }
});

🔍 Technical Highlights

Low Overhead Design

  • DashMap: Lock-free concurrent hash map
  • SmallVec: Stack-allocated small arrays
  • Incremental detection: Only check new edges
  • Lazy backtrace: Collect only when issues found

Detection Algorithms

Deadlock:

  • Build dependency graph
  • DFS cycle detection
  • Real-time graph updates

Race Condition:

  • Track memory access history
  • Happens-before analysis
  • Conflict detection

📈 Roadmap

Short-term

  • Async lock support (tokio::sync::Mutex)
  • Statistical sampling (< 1% overhead)
  • Visualization tools (dependency graph SVG)
  • Structured logging output

Long-term

  • Distributed deadlock detection (cross-service)
  • ML-based problem prediction
  • GPU-accelerated graph analysis
  • IDE plugin integration

🤝 Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.

Development Setup

git clone https://github.com/yourusername/thread-sentry.git
cd thread-sentry
cargo build
cargo test

📄 License

MIT License


🙏 Acknowledgments

Built with amazing Rust community libraries:

  • parking_lot: High-performance lock implementation
  • dashmap: Concurrent hash map
  • crossbeam: Concurrent primitives
  • backtrace: Stack trace collection

📞 Support


Thread-Sentry: Making concurrent programming safer, one lock at a time. 🛡️