Skip to content

failure-tracking.md

Latest commit

 

History

History
411 lines (301 loc) · 9.36 KB

failure-tracking.md

File metadata and controls

411 lines (301 loc) · 9.36 KB
Raw

Failure Tracking

FailureTracker aggregates recurring fingerprints in process memory.

Basic Usage

FailureTracker tracker = new FailureTracker();

tracker.capture(firstFailure);
tracker.capture(firstFailure);
tracker.capture(secondFailure);

System.out.println(tracker.report());

Example:

2 unique failure signatures

BUGDNA-001
Count: 2

BUGDNA-002
Count: 1

capture(Throwable) generates and returns the fingerprint. Existing fingerprints can be recorded with capture(Fingerprint).

PII-Safe Fingerprinting

BugDNA normalizes common high-cardinality values before they are used as fingerprint evidence. Numeric tokens become {NUMBER} and email addresses become {EMAIL}:

Account 123456 -> Account {NUMBER}
Account 654321 -> Account {NUMBER}
john@email.com -> {EMAIL}
alice@email.com -> {EMAIL}

This keeps fingerprints stable when failure messages contain account IDs, order IDs, user IDs, or email addresses.

JUnit-Friendly Assertions

Use BugDnaAssertions.assertThat(Fingerprint) in automated tests to verify failure classification without introducing another assertion dependency:

import static io.github.bugdna.BugDnaAssertions.assertThat;

assertThat(fingerprint)
        .hasCategory(FailureCategory.DATABASE)
        .hasRootCause(SQLTimeoutException.class);

The fluent methods throw AssertionError, so they work with JUnit and other JVM test runners.

Failure Dependency Graph

Use BugDna.dependencyGraph(Throwable) to fingerprint each exception in a causal chain and render how failures depend on one another:

FailureDependencyGraph graph = BugDna.dependencyGraph(failure);
System.out.println(graph.report());
BUGDNA-001
 └─ BUGDNA-014
      └─ BUGDNA-022

The graph follows Throwable.getCause() from the outer failure to the deepest cause. This is useful in Spring Batch and Kafka flows where a batch step, consumer handler, and downstream client failure can each have distinct fingerprints but still belong to one incident chain.

Top Failure Report

System.out.println(tracker.topFailureReport());
Top 10 Failure Signatures
BUGDNA-001
Count: 1003
BUGDNA-002
Count: 512
BUGDNA-003
Count: 201

Use a custom limit:

tracker.topFailureReport(5);
List<FailureAggregate> top = tracker.topFailures(5);

Consume structured aggregates when text output is not appropriate:

for (FailureAggregate failure : tracker.topFailures(5)) {
    dashboard.record(
            failure.getId(),
            failure.getOccurrences(),
            failure.getFingerprint().getCategory()
    );
}

Results are ordered by occurrence count descending, then fingerprint ID for deterministic ties.

Root-Cause Clustering

Different fingerprints can belong to one operational family:

tracker.capture(connectionRefused);
tracker.capture(socketTimeout);
tracker.capture(poolExhausted);

System.out.println(tracker.familyReport());
Root Cause Families

Family: DATABASE_CONNECTIVITY
Occurrences: 3
Unique Failures: 3
BUGDNA-001 (1)
BUGDNA-014 (1)
BUGDNA-027 (1)

Consume structured family aggregates:

for (FailureFamilyAggregate family : tracker.families()) {
    family.getFamily();
    family.getOccurrences();
    family.getUniqueFailures();
    family.getFailures();
}

Use topFamilies(int), topFamilyReport(), or topFamilyReport(int) for bounded views. Family clustering is operational metadata; individual fingerprints remain the stable identity for exact failure signatures.

Failure Timeline

Every tracker capture records a timestamp. Normal captures use the current instant; explicit timestamps are available for imported logs or deterministic processing:

tracker.capture(fingerprint, Instant.parse("2026-06-13T09:01:00Z"));
tracker.capture(fingerprint, Instant.parse("2026-06-13T09:02:00Z"));

System.out.println(tracker.timelineReport(ZoneId.of("Asia/Kolkata")));
14:31 BUGDNA-001
14:32 BUGDNA-001

timeline() returns immutable FailureOccurrence values in chronological order. Each occurrence exposes getOccurredAt(), getFingerprint(), and getId().

Timeline retention is bounded independently from lifetime aggregate counts:

FailureTracker tracker = new FailureTracker(25_000);

The default limit is 10,000 events. When full, the oldest retained event is removed. getTotalOccurrences() still reports all captures since construction or clear().

Burst Detection

Detect fingerprints whose retained timeline reaches a minimum rate:

List<FailureBurst> bursts = tracker.bursts(300);
System.out.println(tracker.burstReport(300, ZoneId.of("UTC")));
BUGDNA-001 burst detected

First Seen: 09:01
Peak Rate: 312/min
Duration: 22 min

Peak rate is the highest number of occurrences in one UTC minute. By default, a gap longer than one minute starts a new burst, preventing unrelated incidents from being reported as one long duration. Use bursts(long, Duration) to choose another idle boundary.

Each FailureBurst exposes:

  • getId() and getFingerprint()
  • getFirstSeen() and getLastSeen()
  • getPeakRatePerMinute()
  • getDuration()
  • getOccurrences()

Fingerprint Drift Detection

Deployment comparisons detect when a recurring fingerprint ID changes signature shape after a release:

DeploymentComparison comparison = RegressionDetector.compare(
        previousSnapshot,
        currentSnapshot
);

for (FingerprintDrift drift : comparison.getFingerprintDrifts()) {
    System.out.println(drift.report());
}

Example:

BUGDNA-001
Signature Drift: 73%
Possible code path change detected

Drift is based on the origin class, origin method, and normalized call path. A high score means the same fingerprint ID is recurring through a meaningfully different shape, which often points to a release-time code path change.

Available Counts

tracker.getTotalOccurrences();
tracker.getUniqueFailures();
tracker.getUniqueFamilies();
tracker.failures();
tracker.families();

For 500 captured exceptions grouped into three IDs:

getTotalOccurrences() -> 500
getUniqueFailures()   -> 3
failures().size()     -> 3

failures() returns an immutable snapshot of all aggregates.

Concurrency

The implementation uses ConcurrentHashMap and concurrent counters. Multiple application threads can call capture(...) safely.

Snapshots are point-in-time views. Captures may continue while a report is being built, so a report is operationally consistent but is not a global transaction.

Lifecycle

The tracker:

  • Stores no data outside the process
  • Loses counts when the process restarts
  • Grows with the number of unique fingerprint IDs
  • Retains only the configured number of recent timeline events
  • Resets when clear() is called

Choose persistent monitoring or storage when counts must survive restarts.

Spring Boot

The starter registers the same core FailureTracker as a bean:

@Component
class BatchFailureReport {

    private final FailureTracker tracker;

    BatchFailureReport(FailureTracker tracker) {
        this.tracker = tracker;
    }

    void print() {
        System.out.println(tracker.topFailureReport());
    }
}

Automatic MVC captures and BugDnaSpringService.fingerprint(...) update the shared tracker.

Skip Reason Analysis

SkipReasonAnalyzer identifies the failure signature responsible for the most skipped items:

SkipReasonAnalyzer analyzer = new SkipReasonAnalyzer();

for (Throwable skippedFailure : skippedFailures) {
    analyzer.record(skippedFailure);
}

System.out.println(analyzer.report());
Most Common Failure

BUGDNA-001

Count:
421

Use it from a Spring Batch SkipPolicy without adding a Spring Batch dependency to BugDNA:

class AnalyzingSkipPolicy implements SkipPolicy {
    private final SkipReasonAnalyzer analyzer;

    AnalyzingSkipPolicy(SkipReasonAnalyzer analyzer) {
        this.analyzer = analyzer;
    }

    @Override
    public boolean shouldSkip(Throwable failure, long skipCount) {
        boolean skipped = isSkippable(failure, skipCount);
        if (skipped) {
            analyzer.record(failure);
        }
        return skipped;
    }
}

getMostCommonFailure() returns the structured FailureAggregate. report() returns None with count 0 before any skips are recorded.

Consumer Failure Tracking

ConsumerFailureTracker captures topic, partition, offset, and fingerprint without depending on a specific messaging client:

ConsumerFailureTracker tracker = new ConsumerFailureTracker();

try {
    process(record);
} catch (RuntimeException failure) {
    tracker.capture(
            record.topic(),
            record.partition(),
            record.offset(),
            failure
    );
}

System.out.println(tracker.report());
BUGDNA-021

Topic:
payment-events

Occurrences:
203

Failures are grouped by topic and fingerprint. This keeps the same fingerprint on payment-events and refund-events as separate aggregates. Each ConsumerFailureAggregate exposes the latest captured partition and offset:

ConsumerFailureAggregate failure = tracker.failures().get(0);

failure.getTopic();
failure.getPartition();
failure.getOffset();
failure.getFingerprint();
failure.getOccurrences();

Use the overload accepting Fingerprint when the failure was fingerprinted earlier:

tracker.capture("payment-events", 2, 9812L, fingerprint);