Attributable failures#9888
Conversation
|
Important Review skippedAuto reviews are limited to specific labels. 🏷️ Labels to auto review (1)
Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
f713f3c to
6640a01
Compare
|
Bumped lightning-onion and rebased on |
6640a01 to
1dd8765
Compare
|
The linter failure is only about the |
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces support for attributable failures, which is a significant and complex change to the error handling and encryption logic. The core refactoring to separate shared secret extraction from error encrypter creation is well-executed and improves the design.
I've found a few issues related to the implementation:
- An incorrect unit calculation for HTLC hold times, which deviates from the BOLT specification.
- A logic error in handling failures for introduction nodes in a blinded path, where the wrong message type is created and attribution data is discarded.
- A minor opportunity for code simplification in the new logging logic.
Overall, the changes are well-structured, but the identified issues, particularly the correctness ones, should be addressed. The PR description is clear about the work-in-progress nature, and this review should help in finalizing the implementation.
1dd8765 to
1fc33bd
Compare
|
Rebased |
1fc33bd to
a5b9c2a
Compare
|
Rebased |
a5b9c2a to
832693a
Compare
|
Added an itest to verify reported hold times from a multi hop error |
832693a to
4ff0dd6
Compare
4ff0dd6 to
51e069f
Compare
🔴 PR Severity: CRITICAL
🔴 Critical (16 files)
🟠 High (3 files)
🟡 Medium (2 files)
AnalysisThis PR introduces attributable payment failures - a mechanism for identifying which hop along a payment route caused a failure. The changes span multiple critical subsystems simultaneously:
Severity bump triggers (all three fired; CRITICAL is already the maximum):
Changes to the HTLC error encryption path and wire protocol are security-sensitive - incorrect attribution data could leak routing information or cause hard-to-diagnose payment failures. To override, add a |
be35c47 to
517ee50
Compare
|
Rebased on |
Co-authored-by: Joost Jager <joost.jager@gmail.com>
Bump lightning-onion dependency to include attributable error support.
Preparation for the instantiation of an attributable error encrypter. This gets rid of the sphinx encrypter instantiation in OnionProcessor. This would otherwise be problematic when an attributable encrypter would need to be created there without having access to the error structure. Co-authored-by: Joost Jager <joost.jager@gmail.com>
We add some simple helpers for the purpose of parsing the attribution data from and to the wire format.
When propagating a failure backwards along a route we want to persist the received ExtraData of the downstream link so that any handling can occur on our node before returning the updated ExtraData to our upstream link.
We now call into the new encryption/decryption methods and provide the received attribution data as an argument. The result is then also returned to our upstream peer.
Add test coverage for the new attributable failures feature: - lnwire/attr_data_test.go: round-trip, TLV type, empty/nil cases - hop/error_encryptor_test.go: encode/decode, backward compat, hold time, reextract, introduction/relaying encrypter round-trips - htlcswitch/attributable_failure_test.go: end-to-end encrypt/decrypt with attribution data, backward compat without attr data, hold times
517ee50 to
9030e8b
Compare
Add a hold_times field to the Failure proto message so that callers of SendPaymentV2 can see per-hop hold times reported via attributable errors. The hold times are persisted in the KV store (backward compatible via trailing optional read) and populated in both the database and streaming RPC paths. Changes: - lnrpc/lightning.proto: add repeated uint32 hold_times = 10 - payments/db: add HoldTimes to HTLCFailInfo + KV serialization - routing: copy HoldTimes from ForwardingError to HTLCFailInfo - lnrpc/routerrpc: populate HoldTimes in both marshallHtlcFailure and marshallError
Add an integration test that sets up Alice -> Bob -> Carol, triggers a payment failure at Carol (unknown payment hash), and verifies that Alice receives hold times in the RPC failure response. The test checks: - hold_times has one entry per route hop (1:1 correspondence) - failure source index matches Carol (index 2) - each hold time is within a reasonable bound
9030e8b to
437390c
Compare
|
@GeorgeTsagk, remember to re-request review from reviewers when ready |
|
/gateway review |
|
✅ Review posted: #9888 (review) 11 finding(s); 11 inline, 0 in body. 🔁 Need a re-review after pushing changes? Reply with |
There was a problem hiding this comment.
This PR implements BOLT #1044 attributable failures: it splits error-encrypter extraction into a shared-secret generator plus an encrypter constructor, threads attribution data and per-hop hold times through the encrypt/decrypt/forward paths, and persists hold times in both the kv and SQL payment stores. The refactor is coherent and the signature changes are propagated consistently, with new unit and integration tests. The design choice of non-strict attribution decryption during the upgrade grace period is sound and well-documented.
The most serious issue is the hold-time clock handling: CreatedAt is not guarded against a zero/invalid value, so HTLCs persisted by a pre-upgrade node (and any future-dated clock) produce a wildly wrong hold time that gets MAC'd into the attribution error and reported to the sender. Several error paths in the switch and mailbox also mishandle failures — one forwards an all-zero undecryptable failure, another silently abandons a fail-back and can strand an HTLC. Finally, the go.mod replace pointing at a personal lightning-onion fork must be resolved before merge.
The remaining findings concern breaking-API hygiene, mislabeled unit comments in crypto code, unbounded remote-reported values surfaced over RPC, and test-coverage gaps on new branches.
Findings: 🔴 0 Blocker · 🟠 5 Major · 🟡 6 Minor · 🔵 0 Nit
| go 1.25.5 | ||
|
|
||
| // Temporary replace until dependent PR is merged in lightning-onion. | ||
| replace github.com/lightningnetwork/lightning-onion => github.com/GeorgeTsagk/lightning-onion v0.0.0-20260311122656-2ec96bf9d0e9 |
There was a problem hiding this comment.
🟠 Major F1: go.mod pins lightning-onion to a personal fork
The added replace github.com/lightningnetwork/lightning-onion => github.com/GeorgeTsagk/lightning-onion v0.0.0-... is self-described as temporary until the dependent lightning-onion PR merges, and it is the cause of the failing lint. This is a merge blocker: the dependency must be repointed to an upstream lightning-onion tag before this can land. Flagging so it is not merged in this state.
| } | ||
|
|
||
| // getHoldTime returns the hold time in decaseconds since the first | ||
| // instantiation of this sphinx error encrypter. |
There was a problem hiding this comment.
🟠 Major F2: Zero/invalid CreatedAt yields garbage hold time
getHoldTime computes uint32(time.Since(s.CreatedAt).Milliseconds() / 100) with no guard on CreatedAt. For a forwarded HTLC persisted by a pre-#9888 node, restoration goes Decode → Reextract: Decode's legacy branch returns early on len(typeMap) == 0, leaving CreatedAt at the zero time.Time (asserted by TestSphinxErrorEncrypterDecodeBackwardCompat), and Reextract only calls initialize(sharedSecret) — it never sets CreatedAt. time.Since(time.Time{}) overflows and the uint32 cast produces a nonsensical multi-decade hold time that is reported to the sender. The same cast wraps negatively if CreatedAt is ever in the future (clock skew, or a corrupt creationTime restored via time.Unix(0, ...) with no validation). Guard getHoldTime to return 0 when CreatedAt.IsZero() or the duration is negative, and default CreatedAt to now in Reextract/the legacy Decode branch.
| // encrypted failure reason. | ||
| func (f *interceptedForward) Fail(reason []byte) error { | ||
| obfuscatedReason := f.packet.obfuscator.IntermediateEncrypt(reason) | ||
| obfuscatedReason, _, err := f.packet.obfuscator.IntermediateEncrypt( |
There was a problem hiding this comment.
🟠 Major F3: Intercepted Fail path drops attribution data
interceptedForward.Fail does obfuscatedReason, _, err := f.packet.obfuscator.IntermediateEncrypt(reason, nil) and resolves with &lnwire.UpdateFailHTLC{Reason: obfuscatedReason}, discarding the returned attrData and never setting ExtraData. Every other forward-failure site (mailbox.FailAdd, switch.handlePacketFail default case, FailWithCode) converts the returned attrData via lnwire.AttrDataToExtraData and populates ExtraData. As written, HTLCs failed through the interceptor lose attribution, silently breaking the feature for that path. Capture the attrData and set ExtraData on the UpdateFailHTLC.
|
|
||
| extraData, err := lnwire.AttrDataToExtraData(attrData) | ||
| if err != nil { | ||
| log.Errorf("Failed to convert attr data: %v", err) |
There was a problem hiding this comment.
🟠 Major F4: FailAdd abandons fail-back on attr-data conversion error, stranding the HTLC
In FailAdd, after EncryptFirstHop succeeds, extraData, err := lnwire.AttrDataToExtraData(attrData) is followed by if err != nil { log.Errorf(...); return }. The bare return aborts before the UpdateFailHTLC is ever queued, so the HTLC is neither settled nor failed — it stays live until timeout and forces an on-chain close, locking funds and a commitment slot. An incoming HTLC that cannot be forwarded is the attacker-influenced input to this path. The conversion error should still drive a fail-back (e.g. fail without attribution data) rather than silently returning.
| // TODO(roasbeef): don't need to pass actually? | ||
| failure := &lnwire.FailPermanentChannelFailure{} | ||
| htlc.Reason, err = circuit.ErrorEncrypter.EncryptFirstHop( | ||
| reason, attrData, err := circuit.ErrorEncrypter.EncryptFirstHop( |
There was a problem hiding this comment.
🟠 Major F5: handlePacketFail proceeds with nil reason after EncryptFirstHop error
In the FailPermanentChannelFailure branch, reason, attrData, err := circuit.ErrorEncrypter.EncryptFirstHop(failure) only logs on error (if err != nil { log.Error(err) }) and falls through, assigning htlc.Reason = reason (zero-value OpaqueReason) and htlc.ExtraData from empty attrData. Both sibling branches (UpdateFailMalformedHTLC, the default forwarded case) return on encrypt error. The result is an all-zero, undecryptable UpdateFailHTLC sent upstream for a real HTLC, masking the true failure and breaking attribution. Add return err after the log to match the other branches.
|
|
||
| return encrypter | ||
| } | ||
|
|
There was a problem hiding this comment.
🟡 Minor F7: Hold-time unit comments mislabel the encoded unit
The getHoldTime doc comment says "decaseconds" (10s) and the AttrErrorStruct comment says the payload holds "a 32-bit hold time in milliseconds," but the implementation encodes 100ms units (Milliseconds() / 100, i.e. deciseconds), which is what every other comment (failure.go, lightning.proto, payments/db/payment.go) and the ht*100 render agree on. Fix these two comments to say "100ms units" to match the implementation. Minor cleanup in the same constructor: time.Now().Truncate(time.Nanosecond) is a no-op since nanosecond is already time.Time's base resolution.
| // constructor. | ||
| func NewSphinxErrorEncrypter(ephemeralKey *btcec.PublicKey, | ||
| sharedSecret sphinx.Hash256) *SphinxErrorEncrypter { | ||
|
|
There was a problem hiding this comment.
🟡 Minor F8: Constructor and interface semantics changed under the same names
NewSphinxErrorEncrypter changed meaning, not just signature: it was a no-arg blank constructor and is now an initialized constructor taking (ephemeralKey, sharedSecret), with the blank behavior moved to NewSphinxErrorEncrypterUninitialized (same for the Introduction/Relaying variants). A mechanically-remapped call site that needed the old deferred-Reextract semantics but got the initialized constructor (or vice versa) is a silent regression the type checker cannot catch — enumerate the former call sites and confirm each was mapped to the constructor matching its prior initialization state. The ErrorEncrypter interface also changed shape (the Encrypt* methods now return []byte, ErrorEncrypterExtracter → SharedSecretGenerator); this is an intentional breaking change that warrants a migration note in the release notes for out-of-tree implementers.
| } | ||
|
|
||
| return WriteElements(w, byte(f.Reason), f.FailureSourceIndex) | ||
| err := WriteElements(w, byte(f.Reason), f.FailureSourceIndex) |
There was a problem hiding this comment.
🟡 Minor F9: Two independent hold-time serialization formats on recently-churned DB code
Hold times are encoded twice by hand — WriteElements (uint16 count + uint32 each) in serializeHTLCFailInfo, and manual binary.BigEndian in sql_converters.encodeHoldTimes — feeding the same HTLCFailInfo.HoldTimes. Both land on DB code that was itself just modified (kv path PR #10153 Aug 2025; SQL hold_times column PR #10604 Oct 2025). If only one backend is exercised in tests, a format divergence ships undetected. Add a cross-backend round-trip assertion, or factor the two onto a shared encoder so they cannot drift.
| import "github.com/lightningnetwork/lnd/tlv" | ||
|
|
||
| // AttrDataTlvType is the TlvType that hosts the attribution data in the | ||
| // update_fail_htlc wire message. |
There was a problem hiding this comment.
🟡 Minor F10: Exported mutable AttrDataTlvType and nil/empty normalization
var AttrDataTlvType tlv.TlvType1 is an exported, mutable package var that determines the on-wire attribution TLV type — a protocol constant that any importer can silently reassign, breaking interop. Prefer a typed constant exposed via a getter. Also note AttrDataToExtraData(nil) and ([]byte{}) both round-trip to an empty record and ExtraDataToAttrData always returns nil for the empty case, so "no attribution present" and "present but empty" are indistinguishable; document this normalization (or return an fn.Option) so callers don't branch on == nil vs len == 0 incorrectly.
| DecodeHopIterators: p.cfg.SphinxPayment.DecodeHopIterators, | ||
| ExtractErrorEncrypter: p.cfg.SphinxPayment.ExtractErrorEncrypter, | ||
| Peer: p, | ||
| DecodeHopIterators: p.cfg.SphinxPayment.DecodeHopIterators, |
There was a problem hiding this comment.
🟡 Minor F11: New/relocated branches lack test coverage
Two behavior-carrying changes need coverage. First, the blinded-path role selection (introduction vs relaying vs plain), which was only recently introduced in iterator.ExtractErrorEncrypter (PR #8485, 2024), is relocated into the CreateErrorEncrypter closure here as switch { case isIntroduction: ...; case hasBlindingPoint: ...; default: ... }; a swapped case reverts blinded-error behavior with no compile error. Add a test that exercises the introduction case with a blinding point also present (order-sensitivity). Second, IntermediateEncrypt's new errors.Is(err, sphinx.ErrInvalidAttrStructure) recovery branch (regenerating fresh attribution) has no visible test — add one asserting the reset behavior.
|
🤖 gateway audit metadata for this PR — auto-generated, please don't edit. |
Description
Adds support for Attributable failures (feature 36/37). This changes the error encryption and the data encoding for the
update_fail_htlcmessage. We now include the attribution structure in the extra data of the peer message, which helps with assigning blame across the route and also extracting the reported HTLC hold times for each node across the route.For now we default to using non-strict attribution decryption, which means that we're basically preserving the old behavior in order to give time to nodes to upgrade. Otherwise this could lead to premature blame of our direct peers, possibly blacklisting all of our channels early in the payment lifecycle.
This is based on the lightning-onion PR which is also updated to reflect the latest state of attributable failures Bolt PR.
Testing
Current unit/integration tests seem to be passing, which validates that we are preserving the old behavior.
Todo:
Even though the todo list above is important to consider this PR ready to merge, this PR at current state is still good for a first round of review.