[pull] main from firecrawl:main#321
Merged
Merged
Conversation
* refactor(api): confirm Exchange billing from the billing worker The billing confirmation moves from an awaited call in the scrape worker to the billing pipeline, so confirmed means the debit actually committed: - billScrapeJob attaches the access event id to the bill_team job and no longer awaits anything Exchange-related - the billing worker threads it into the batch operation with the billing job id as the reference - the batch processor confirms each covered access once the group's debit commits, and voids it when the commit fails or refunds - outcome reporting rides the billing pipeline's retries and never throws, so it cannot affect billing itself The scrape worker's failure-path void can now race the async confirm; the Exchange resolves this authoritatively (void -> confirmed is a legal repair, confirmed -> void stays rejected), so a paid access can never end up voided. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(exchange): never void on ambiguous commits, retry outcome reports - Failed or thrown billing commits no longer void the Exchange access event: supaBillTeam maps transport errors to success: false, and the debit may have committed before the error, so voiding could erase a real debit. The event stays pending for reconciliation. - Confirmation delivery moved outside the batch lock so an Exchange outage can never hold the 30s lease past expiry and overlap another batch run. - reportExchangeBilling now retries transient failures (3 attempts, backoff), treats 4xx as definitive, and returns success. - Billing-queue enqueue failure now voids the access event (the debit can never commit; a lost-ack enqueue is repaired by void -> confirmed). - Index-worker guards against an empty-string access event ID. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(exchange): jittered backoff, bounded confirm concurrency, retryable 429 - Retry backoff gains full jitter (capped at 15s) so a batch of reports failing together does not retry in synchronized bursts against a degraded Exchange. - Batch confirmations run at most 16 at a time instead of one unbounded Promise.all over the whole batch. - 429 is treated as transient (honoring Retry-After when sane) instead of a definitive rejection, so rate-limited confirmations reconcile once the limit expires. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(exchange): worker-level confirm semaphore, HTTP-date Retry-After, true full jitter - The 16-slot confirmation concurrency budget is now shared across the worker via a small semaphore, so overlapping post-lock deliveries cannot stack beyond the bound. - Retry-After parsing accepts the HTTP-date form in addition to delta-seconds. - The retry delay is now genuine full jitter (0..backoff) with Retry-After as the lower bound, matching the comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(exchange): pull-based confirm worker pool Confirmation delivery now uses a small worker pool that pulls operations one at a time, so a slow Exchange queues at most 16 waiters per batch on the shared budget instead of one closure per operation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )