chore: only err RMQ retries when over 80% mark

[mnafees]

Description

We want to get rid of noisy errors. This PR adjusts how we error out RMQ errors.

Type of change

• Chore (changes which are not directly related to any business logic)

What's Changed

• Adds conditional to only ERR when > 80% of max death count

Checklist

Changes have been:

• Tested (unit, integration, or manually with steps specified)
• Linted and formatted

---

🤖 AI Disclosure

• I acknowledge that an LLM was used in the creation of this Pull Request, in accordance with Hatchet's AI_POLICY.md.

• Details: [e.g. generating tests, writing docs]

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hatchet-docs	Ready	Preview, Comment	Jun 24, 2026 10:07am

[Copilot]

Pull request overview

This PR reduces noisy RabbitMQ retry logging by downgrading most retry logs from Error to Warn, only escalating to Error when the retry/death count is near the configured maximum.

Changes:

• Introduces a logger that defaults to Warn() and switches to Error() once the message retry count crosses an 80% threshold of maxDeathCount.
• Keeps the existing “only log after 5 retries” gate, but changes the severity level when logging does occur.

[github-actions]

Benchmark results

goos: linux
goarch: amd64
pkg: github.com/hatchet-dev/hatchet/internal/msgqueue/rabbitmq
cpu: AMD Ryzen 9 7950X3D 16-Core Processor          
                              │ /tmp/old.txt │            /tmp/new.txt            │
                              │    sec/op    │   sec/op     vs base               │
CompressPayloads_1x10KiB-8       74.69µ ± 7%   72.84µ ± 4%        ~ (p=0.180 n=6)
CompressPayloads_10x10KiB-8      847.3µ ± 2%   873.2µ ± 2%   +3.06% (p=0.004 n=6)
CompressPayloads_10x100KiB-8     10.25m ± 4%   10.11m ± 3%        ~ (p=0.394 n=6)
CompressPayloads_Concurrent-8    54.68µ ± 2%   66.01µ ± 8%  +20.73% (p=0.002 n=6)
geomean                          433.9µ        453.9µ        +4.61%

                              │ /tmp/old.txt │            /tmp/new.txt            │
                              │     B/op     │     B/op      vs base              │
CompressPayloads_1x10KiB-8      10.92Ki ± 1%   10.92Ki ± 0%       ~ (p=0.446 n=6)
CompressPayloads_10x10KiB-8     108.5Ki ± 1%   108.6Ki ± 1%       ~ (p=0.818 n=6)
CompressPayloads_10x100KiB-8    2.920Mi ± 0%   2.920Mi ± 0%       ~ (p=0.485 n=6)
CompressPayloads_Concurrent-8   54.17Ki ± 0%   54.36Ki ± 0%  +0.35% (p=0.015 n=6)
geomean                         117.7Ki        117.8Ki       +0.09%

                              │ /tmp/old.txt │            /tmp/new.txt            │
                              │  allocs/op   │ allocs/op   vs base                │
CompressPayloads_1x10KiB-8        5.000 ± 0%   5.000 ± 0%       ~ (p=1.000 n=6) ¹
CompressPayloads_10x10KiB-8       32.00 ± 0%   32.00 ± 0%       ~ (p=1.000 n=6) ¹
CompressPayloads_10x100KiB-8      63.00 ± 2%   63.00 ± 0%       ~ (p=1.000 n=6)
CompressPayloads_Concurrent-8     17.00 ± 0%   17.00 ± 0%       ~ (p=1.000 n=6) ¹
geomean                           20.35        20.35       +0.00%
¹ all samples are equal

_{Compared against main (3e34b71)}

[mrkaye97]

I'm not too sure we want to change this default?