Skip to content

perf/resilience: reduce OTP polling latency and add runtime resilience gates#493

Merged
DennisDyallo merged 28 commits into
yubikit-composite-device-newfrom
yubikit-performance
Jul 1, 2026
Merged

perf/resilience: reduce OTP polling latency and add runtime resilience gates#493
DennisDyallo merged 28 commits into
yubikit-composite-device-newfrom
yubikit-performance

Conversation

@DennisDyallo

@DennisDyallo DennisDyallo commented Jun 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

  • Add a BenchmarkDotNet performance harness for discovery and management paths over SmartCard, FIDO HID, and OTP HID.
  • Fix OTP HID write-ready polling by checking readiness immediately and polling only while the device is busy.
  • Harden SmartCard listener native-failure handling with deterministic no-hardware fault-injection tests.
  • Fix SmartCard listener context disposal on restart/stop so fake native handles return to baseline.
  • Add Category=RuntimeResilience gates and a fast local/CI runner: dotnet toolchain.cs -- resilience --fast.
  • Add phase learning documents so AI agents and humans can recover context, understand proof quality, and continue the workflow without re-discovering prior decisions.

Why This Expanded

The original benchmark work found and fixed a large OTP latency defect. Follow-up profiling showed no other obvious 10x foreground performance target, but it exposed a more important adjacent class: runtime resilience and flakiness risks around native listener loops, stale handles, retry storms, and lifecycle cleanup.

This PR now turns that discovery into a repeatable agent/human workflow: find a real or seeded bug class, add a deterministic no-hardware invariant, red-green prove the detector, review proof quality, document learnings, commit the phase, and only then move on.

Benchmark Findings

OTP HID after the polling fix:

  • CreateManagementSessionOverOtpHid: ~1.039 s to ~28.0 ms, about 37x faster.
  • GetDeviceInfoOverOtpHid: ~2.075 s to ~55.9 ms, about 37x faster.
  • ThreadPool completed work items dropped to 0 for these OTP paths.

Other profiled paths after cleanup:

  • Discovery: cached ~356 ns, forced rescan ~59.6 ms, cold ~96.0 ms.
  • FIDO HID: connect ~51 ms, management session ~70 ms, get info ~86 ms.
  • SmartCard after killing scdaemon: connect ~6.1 ms, select management ~7.2 ms, get info ~21.1 ms.

Conclusion: BenchmarkDotNet remains the discovery tool. Confirmed hotspots get converted into cheap deterministic gates rather than requiring BDN in normal CI.

Runtime Resilience Gates

Added a fast no-hardware runtime resilience suite covering:

  • SmartCard native failure retry storms from persistent SCardGetStatusChange / SCardListReaders failures.
  • SmartCard context re-establishment success and failure paths.
  • SmartCard context leak on Start(); Stop(); Start(); Stop(); Dispose();.
  • OTP HID sleep-first ready-to-write polling regression.
  • Static scanner fixtures for risky listener/native-loop shapes:
    • ignored native result inside loop,
    • native failure continue without backoff/exit/handler,
    • catch/retry loop without backoff,
    • sleep-before-ready-to-write polling.

The suite is selected with Category=RuntimeResilience and runs through:

dotnet toolchain.cs -- resilience --fast

AI/Human Collaboration Workflow

This SDK is expected to be developed heavily by AI agents and humans together, so the work optimizes for shared executable context:

  • One canonical fast command for agents and humans: dotnet toolchain.cs -- resilience --fast.
  • No-hardware default gates, so agents can validate without a YubiKey, PC/SC state, touch prompts, or local hardware coordination.
  • Red-green proof for detectors, so future agents know tests are meaningful rather than merely passing.
  • Phase learning docs under docs/learnings/runtime-resilience/ explaining what was found, how it was found, what worked, what failed, what review changed, and what remains risky.
  • A phase closeout gate in docs/plans/runtime-resilience-harness.md: implement, verify, review when relevant, write learnings, commit, then continue.
  • Explicit deferrals for diagnostics project and audit skill creation, so agents do not add shelfware before there is a real orchestration need.

CI Integration

The main workflow now includes a dedicated resilience step after unit tests and before packing:

- name: Run runtime resilience checks
  run: dotnet toolchain.cs -- resilience --fast

This keeps resilience failures visible as their own CI signal instead of hiding them inside generic unit-test output.

Red-Green Proof

The fast runner was deliberately tested against seeded regressions:

  • Temporarily reintroducing OTP sleep-first write polling made dotnet toolchain.cs -- resilience --fast fail with both static scanner and timing-guard failures.
  • Temporarily reintroducing the SmartCard context leak made the runner fail with the context-release invariant.
  • Restoring the fixes made the runner pass again.

Phase Decisions

  • Phase 1: SmartCard no-hardware native failure gate added.
  • Phase 2: OTP polling regression guard red-green verified.
  • Phase 3: Static scanner prototype added with positive and safe-negative fixtures.
  • Phase 4: SmartCard context leak found and fixed.
  • Phase 5: Minimal fast runner added.
  • Phase 6: Dedicated diagnostics project deferred until live diagnostics or cross-module orchestration justify it.
  • Phase 7: Reusable audit skill deferred until there is more than one command/mode to orchestrate.

Verification

Local verification before updating this PR:

  • dotnet toolchain.cs -- resilience --fast
    • 13 runtime-resilience tests passed.
    • Latest local run: ~8.0s.
  • dotnet toolchain.cs -- test --project Core --filter "Category!=RequiresUserPresence"
    • 523 total, 521 succeeded, 2 expected hardware-related skips, 0 failed.
  • dotnet format --verify-no-changes
    • No formatting errors.
    • Existing Tests.TestProject IL2026 / IL3050 warnings remain unrelated.

Earlier verification also included:

  • dotnet toolchain.cs -- build --project Core
  • dotnet toolchain.cs -- pack --package-version 2.0.0-preview.perf.1
  • BenchmarkDotNet before/after runs for OTP HID management paths.

Notes

  • .gitignore and docs/plans/composite-device/rust-csharp-composite-comparison.md remain unrelated local worktree changes and are not part of this PR.
  • Live-hardware diagnostics are intentionally not part of the default path. Future live diagnostics must be read-only by default and explicitly opt in to mutating scenarios.

@DennisDyallo DennisDyallo changed the title perf(otp): reduce HID polling latency perf/resilience: reduce OTP polling latency and add runtime resilience gates Jun 16, 2026
DennisDyallo and others added 11 commits June 16, 2026 21:08
@DennisDyallo
docs: fold runtime resilience learnings forward
58c4ee9
@DennisDyallo
docs(core): require resilience runner for loop changes
2a4c67b
@DennisDyallo
feat(core): add Windows HID transport
b933d2a
@DennisDyallo
fix(core): document Windows HID access failures
db41b6a
@DennisDyallo
fix(fido2): align CTAP status values with CTAP 2.2
32158c9
@DennisDyallo
fix(core): use full PIV application identifier
85d6f0b
@DennisDyallo
docs(fido2): clarify CTAP status coverage
23bdbfb
@DennisDyallo
chore: code style
d265372
@DennisDyallo @claude
fix(core): audit and fix Windows HID tests
de5f0ed 
- Fix HidEnumerationTests: correct RequiresHardware trait, replace
  broken DeviceId prefix filter with SupportsConnection, strengthen
  trivially-true assertions, show Windows advisory only when 0 devices
- Fix HidDeviceListenerDisposalTests: narrow catch(Exception) to
  catch(PlatformNotSupportedException), remove duplicate platform
  implementation test, add state assertions to idempotent tests
- Fix HidDeviceListenerIntegrationTests: narrow exception handling,
  fix Create_StatusIsStarted (never called Start), start listener
  before testing for spurious events and disposal, remove duplicate
  Dispose_MultipleTimes test
- Add HidInterfaceClassifierTests: 9 unit tests covering Classify,
  IsSupported, and GetReportType for FIDO, OTP, and Unknown types

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DennisDyallo
refactor: code style and warnings
51ac5b2
@DennisDyallo
Delete src/Core/tests/Yubico.YubiKit.Core.UnitTests/Sessions/Applicat…
630c697 
…ionIdsTests.cs
Base automatically changed from codex-core-layout-refactor to yubikit-composite-device-new June 25, 2026 08:53
DennisDyallo and others added 3 commits June 26, 2026 14:30
@DennisDyallo
Merge pull request #501 from Yubico/yubikit-piv-9-byte-aid
e8141ce 
feat: Use full PIV application identifier
@github-actions @DennisDyallo
refactor(fido2): remove PuvathRequired backwards compat alias
a0b78a6 
Co-authored-by: Dennis Dyallo <DennisDyallo@users.noreply.github.com>
@DennisDyallo
Merge pull request #505 from Yubico/yubikit-ctap-status-22-alignment
f9f649e 
feat: Align CTAP status values with CTAP 2.2
@DennisDyallo DennisDyallo requested a review from Copilot June 29, 2026 23:35
@github-actions @DennisDyallo
fix(core): use ArgumentException for HID report buffer length validation
fd7d983 
SetFeatureReport and SetOutputReport were throwing InvalidOperationException
for caller-supplied buffers of the wrong length, which misrepresents the
failure as object state rather than invalid input. Switch to ArgumentException
and include expected vs actual lengths for actionable diagnostics.

Co-authored-by: Dennis Dyallo <DennisDyallo@users.noreply.github.com>
Copilot AI reviewed Jun 29, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a “runtime resilience” workflow for the SDK (primarily Core) by fixing a major OTP HID latency regression, hardening SmartCard listener retry/lifecycle behavior, and adding fast no-hardware gates (plus supporting benchmarks/docs) to prevent these bug classes from returning.

Changes:

  • Add a new dotnet toolchain.cs -- resilience --fast target that runs Category=RuntimeResilience gates in Core and wire it into CI.
  • Fix OTP HID write-ready polling to check readiness immediately and only poll while busy; add a no-hardware regression guard.
  • Add SmartCard listener resilience seams/tests (fault injection, backoff boundedness, context lifecycle invariants) plus a lightweight static scanner gate.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
toolchain.cs Adds resilience target and --fast flag wiring for fast runtime-resilience gates.
src/WebAuthn/src/Extensions/PreviewSign/PreviewSignErrors.cs Updates CTAP error mapping for PuatRequired and adds PinTokenExpired mapping.
src/WebAuthn/src/Client/WebAuthnClient.cs Updates CTAP→WebAuthn error mapping to include PuatRequired/PinTokenExpired.
src/Fido2/tests/Yubico.YubiKit.Fido2.UnitTests/CtapExceptionTests.cs Adds CTAP 2.2 status value/message assertions.
src/Fido2/tests/Yubico.YubiKit.Fido2.IntegrationTests/FidoMakeCredentialTests.cs Updates PIN/UV-required integration test naming and expected status handling.
src/Fido2/src/Ctap/CtapStatus.cs Renames/adds CTAP status codes (PuatRequired, PinTokenExpired) and updates spec references.
src/Fido2/src/Ctap/CtapException.cs Updates exception message mapping for new/renamed CTAP statuses.
src/Core/tests/Yubico.YubiKit.Core.UnitTests/Transports/SmartCard/DesktopSmartCardDeviceListenerSCardErrorTests.cs New no-hardware SmartCard fault-injection tests for backoff/recovery and context disposal invariants.
src/Core/tests/Yubico.YubiKit.Core.UnitTests/RuntimeResilience/RuntimeResilienceStaticScanTests.cs Adds static “risky loop shape” scanner tests plus current-source smoke scan.
src/Core/tests/Yubico.YubiKit.Core.UnitTests/Protocols/Otp/Hid/OtpHidProtocolTests.cs Adds runtime-resilience timing guard for sleep-first OTP regression.
src/Core/src/Transports/SmartCard/ISCardApi.cs Introduces injectable PC/SC seam (ISCardApi + NativeSCardApi) for deterministic tests.
src/Core/src/Transports/SmartCard/DesktopSmartCardDeviceListener.cs Refactors listener to use seam, adds failure handling/backoff/recovery, and improves lifecycle cleanup.
src/Core/src/Sessions/ApplicationIds.cs Updates PIV AID constant to full form.
src/Core/src/Protocols/Otp/Hid/OtpHidProtocol.cs Fixes OTP HID ready-to-write polling (immediate check + busy-only polling).
src/Core/CLAUDE.md Documents listener/native retry-loop resilience expectations and recommended gate command.
docs/plans/runtime-resilience-harness.md Adds a phased plan for resilience gates/diagnostics workflow.
docs/learnings/runtime-resilience/README.md Adds index for runtime-resilience phase learnings.
docs/learnings/runtime-resilience/phase-01-smartcard-listener.md Phase 1 closeout: SmartCard listener recovery learnings.
docs/learnings/runtime-resilience/phase-02-otp-polling.md Phase 2 closeout: OTP polling regression learnings.
docs/learnings/runtime-resilience/phase-03-static-scanner.md Phase 3 closeout: static scanner learnings.
docs/learnings/runtime-resilience/phase-04-smartcard-context-leak.md Phase 4 closeout: SmartCard context leak invariant learnings.
docs/learnings/runtime-resilience/phase-05-minimal-fast-runner.md Phase 5 closeout: minimal runner learnings.
docs/learnings/runtime-resilience/phase-06-diagnostics-project-deferred.md Phase 6 closeout: diagnostics project deferral rationale.
docs/learnings/runtime-resilience/phase-07-audit-skill-deferred.md Phase 7 closeout: audit skill deferral rationale.
Directory.Packages.props Adds central package version for BenchmarkDotNet.
benchmarks/Yubico.YubiKit.PerformanceBenchmarks/Yubico.YubiKit.PerformanceBenchmarks.csproj New BenchmarkDotNet harness project definition.
benchmarks/Yubico.YubiKit.PerformanceBenchmarks/Program.cs Implements discovery/management benchmarks over SmartCard/FIDO/OTP.
AGENTS.md Adds guidance to run resilience gates when touching Core runtime loops.
.github/workflows/build.yml Adds CI step for dotnet toolchain.cs -- resilience --fast.

Comment thread src/Core/src/Transports/SmartCard/DesktopSmartCardDeviceListener.cs
Comment thread benchmarks/Yubico.YubiKit.PerformanceBenchmarks/Yubico.YubiKit.PerformanceBenchmarks.csproj
@DennisDyallo DennisDyallo merged commit 41c554e into yubikit-composite-device-new Jul 1, 2026
3 checks passed
@DennisDyallo DennisDyallo deleted the yubikit-performance branch July 1, 2026 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

Assignees

No one assigned

Labels

v2

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@DennisDyallo