Skip to content

fix: recover after flush handler exceptions#200

Merged
ioannisj merged 4 commits into
mainfrom
fix/async-batch-handler-flush-recovery
May 11, 2026
Merged

fix: recover after flush handler exceptions#200
ioannisj merged 4 commits into
mainfrom
fix/async-batch-handler-flush-recovery

Conversation

@ioannisj

@ioannisj ioannisj commented May 7, 2026

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

Fixes #199.

AsyncBatchHandler.HandleFlushSignal previously wrapped the entire flush loop in a single try/catch. If a batch send threw after HTTP retries were exhausted, the exception was logged but the background flush processor exited permanently. Future timer/FlushAt signals would release the semaphore, but nothing consumed those signals, so events could
silently accumulate until a restart.

This change:

  • keeps HandleFlushSignal running after per-flush exceptions by catching/logging inside the loop
  • only treats OperationCanceledException as loop termination when the handler's own cancellation token is canceled, so HTTP timeouts or other cancellation-like send failures do not permanently stop the loop
  • replaces the _flushing interlocked gate with a SemaphoreSlim flush lock
  • preserves background flush coalescing when another flush is already running
  • makes explicit FlushAsync() wait for an in-progress flush instead of returning early without doing work

💚 How did you test it?

  • Added regression coverage:
    • BackgroundFlushContinuesAfterException
    • FlushAsyncWaitsForInProgressFlush
  • Verified the new regression test fails on origin/main before the fix.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran pnpm changeset to generate a changeset file

@ioannisj
ioannisj requested review from a team and haacked as code owners May 7, 2026 08:29
@github-actions

github-actions Bot commented May 7, 2026

Copy link
Copy Markdown
Contributor

posthog-dotnet Compliance Report

Date: 2026-05-11 16:32:52 UTC
Duration: 522ms

⚠️ Some Tests Failed

2/16 tests passed, 14 failed


Feature_Flags Tests

⚠️ 2/16 tests passed, 14 failed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id 41ms
Request Payload.Flags Request Uses V2 Query Param 21ms
Request Payload.Flags Request Hits Flags Path Not Decide 5ms
Request Payload.Flags Request Omits Authorization Header 5ms
Request Payload.Token In Flags Body Matches Init 5ms
Request Payload.Groups Round Trip 4ms
Request Payload.Groups Default To Empty Object 5ms
Request Payload.Person Properties Distinct Id Auto Populated When Caller Omits It 5ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False 4ms
Request Payload.Disable Geoip Omitted Defaults To False 5ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key 5ms
Request Lifecycle.No Flags Request On Init Alone 3ms
Request Lifecycle.No Flags Request On Normal Capture 177ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests 6ms
Request Lifecycle.Mock Response Value Is Returned To Caller 5ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event 4ms

Failures

request_payload.request_with_person_properties_device_id

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag'

request_payload.flags_request_uses_v2_query_param

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag'

request_payload.flags_request_hits_flags_path_not_decide

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag'

request_payload.flags_request_omits_authorization_header

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag'

request_payload.token_in_flags_body_matches_init

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag'

request_payload.groups_round_trip

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag'

request_payload.groups_default_to_empty_object

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag'

request_payload.person_properties_distinct_id_auto_populated_when_caller_omits_it

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag'

request_payload.disable_geoip_false_propagates_as_geoip_disable_false

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag'

request_payload.disable_geoip_omitted_defaults_to_false

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag'

request_payload.flag_keys_to_evaluate_contains_only_requested_key

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag'

request_lifecycle.two_flag_calls_produce_two_remote_requests

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag'

request_lifecycle.mock_response_value_is_returned_to_caller

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag'

side_effect_events.get_feature_flag_captures_feature_flag_called_event

404, message='Not Found', url='http://sdk-adapter:8080/get_feature_flag'

@greptile-apps

greptile-apps Bot commented May 7, 2026

Copy link
Copy Markdown
Contributor

Comments Outside Diff (1)

  1. src/PostHog/Library/AsyncBatchHandler.cs, line 231-240 (link)

    P2 The return; at the end of DrainBatchesAsync is superfluous — it follows the while loop and precedes only the local SendBatch helper, so it does nothing. Removing it satisfies simplicity rule 4 ("Has no superfluous parts").

    Prompt To Fix With AI
    This is a comment left during a code review.
    Path: src/PostHog/Library/AsyncBatchHandler.cs
    Line: 231-240
    
    Comment:
    The `return;` at the end of `DrainBatchesAsync` is superfluous — it follows the `while` loop and precedes only the local `SendBatch` helper, so it does nothing. Removing it satisfies simplicity rule 4 ("Has no superfluous parts").
    
    
    
    How can I resolve this? If you propose a fix, please make it concise.
Prompt To Fix All With AI
Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
src/PostHog/Library/AsyncBatchHandler.cs:212-216
Using the synchronous `_flushLock.Wait(0)` inside an `async` method is inconsistent with the project's async patterns and can be surprising to readers — though it never actually blocks since the timeout is 0. Prefer `await _flushLock.WaitAsync(0)` to stay consistent with `FlushBatchesAsync` (which uses `WaitAsync`) and to avoid inadvertently capturing a thread-pool thread in environments where the synchronous SemaphoreSlim path has side effects.

```suggestion
        // Background flushes coalesce when another flush is already in progress.
        if (!await _flushLock.WaitAsync(0))
        {
            return;
        }
```

### Issue 2 of 2
src/PostHog/Library/AsyncBatchHandler.cs:231-240
The `return;` at the end of `DrainBatchesAsync` is superfluous — it follows the `while` loop and precedes only the local `SendBatch` helper, so it does nothing. Removing it satisfies simplicity rule 4 ("Has no superfluous parts").

```suggestion
        while (_channel.Reader.TryReadBatch(_options.Value.MaxBatchSize, out var batch))
        {
            var tasks = batch.Select(item => item.ResolveItem(batchContext));
            var resolved = await Task.WhenAll(tasks);
            await SendBatch(resolved);
        }

        async Task SendBatch(IReadOnlyCollection<TItem> batch)
```

Reviews (1): Last reviewed commit: "chore: add changeset" | Re-trigger Greptile

Comment thread src/PostHog/Library/AsyncBatchHandler.cs
@ioannisj
ioannisj enabled auto-merge (squash) May 11, 2026 16:32
@ioannisj
ioannisj merged commit 16583c8 into main May 11, 2026
14 checks passed
@ioannisj
ioannisj deleted the fix/async-batch-handler-flush-recovery branch May 11, 2026 16:34
@greptile-apps

greptile-apps Bot commented May 11, 2026

Copy link
Copy Markdown
Contributor

Reviews (2): Last reviewed commit: "Merge branch 'main' into fix/async-batch..." | Re-trigger Greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AsyncBatchHandler.HandleFlushSignal exits permanently on HTTP exception, silently stopping all background flushes

2 participants