chore(source-hubspot): Upgrade SDM base image to pre-release CDK with heartbeat diagnostics and deadlock fix#75589
Conversation
…heartbeat diagnostics Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
|
/publish-connectors-prerelease
|
|
|
Pinning pre-release |
…K with thread dumps and queue stats Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
|
/publish-connectors-prerelease
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| - api.hubapi.com | ||
| connectorBuildOptions: | ||
| baseImage: docker.io/airbyte/source-declarative-manifest:7.14.0@sha256:5b95dd9e71defeece4a1efb27688b15e482f18c82c745498611846466b72403e | ||
| baseImage: docker.io/airbyte/source-declarative-manifest:7.13.0.post13.dev23806774478@sha256:3454e270979f4e239c538cc9200cc872d54815f60752d997d419ed51434ff7bf |
There was a problem hiding this comment.
🔴 GA connector base image downgraded from stable release to pre-release dev build
The base image is changed from a stable release (source-declarative-manifest:7.14.0) to a pre-release development build (source-declarative-manifest:7.13.0.post13.dev23806774478). This is both a version downgrade (7.14.0 → 7.13.0-based) and a switch to an unstable .dev build for a connector marked as releaseStage: generally_available (airbyte-integrations/connectors/source-hubspot/metadata.yaml:43). If merged, this GA connector — used by production users — would be built on an unreleased, potentially unstable base image. Development builds may contain incomplete features, known bugs, or experimental changes not intended for production use.
Was this helpful? React with 👍 or 👎 to provide feedback.
Updates to source-declarative-manifest:7.13.0.post14.dev23855054325 which includes the deadlock prevention fix in ConcurrentMessageRepository. This version prevents the main thread from blocking on queue.put() when the queue is full, which was confirmed as the root cause of sync stalls in job 77457394. Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
devin-ai-integration
Bot
changed the title
|
/publish-connectors-prerelease
|
2 participants
Caution
Testing only — do not merge to master. This PR pins source-hubspot to a pre-release SDM image for diagnostic testing and deadlock fix validation.
What
Updates the source-hubspot connector's base SDM image to a pre-release version of
source-declarative-manifestbuilt from airbytehq/airbyte-python-cdk#953. That CDK PR adds:queue.put()when the shared queue is full, which was confirmed as the root cause of sync stalls (job 77457394).Heartbeat output (normal):
Heartbeat output (stall detected — after 90s of frozen message count):
The queue stats instantly reveal the deadlock type:
queue_size=0+msgs frozen→ workers stuck on HTTP/IOqueue_size=10000+print_blocked=YES→ classic pipe deadlock (stdout backpressure)queue_size=10000+print_blocked=NO→ main thread not consuming (the deadlock this CDK version fixes)How
Single-line change in
metadata.yaml: theconnectorBuildOptions.baseImageis updated from the stable7.14.0SDM tag to the pre-release7.13.0.post14.dev23855054325tag (with its corresponding sha256 digest). No other files are modified.Updates since last revision
Updated the SDM base image from
7.13.0.post13.dev23806774478to7.13.0.post14.dev23855054325. The new CDK pre-release adds:ConcurrentMessageRepository— the main thread (sole queue consumer) now uses non-blockingput(block=False)when emitting messages to the shared queue. If the queue is full, messages are buffered in adequeand drained viaconsume_queue(). Worker threads are unchanged (blockingput()for backpressure). This prevents the confirmed deadlock where three code paths in_handle_itemcaused the main thread to block on its own queue:PartitionCompleteSentinel→_on_stream_is_done→ensure_at_least_one_state_emitted→queue.put(state)PartitionGenerationCompletedSentinel→_on_stream_is_done→ same pathPartition→on_partition→emit_message(slice_log)→queue.put(log)Previous revisions added queue stats, automatic thread dumps, and heartbeat diagnostics (see CDK PR for full history).
Review guide
airbyte-integrations/connectors/source-hubspot/metadata.yaml— only thebaseImageline changed.7.13.0.post14.dev23855054325) and digest (sha256:c5a8b477...) look correct.dockerImageTag, registry overrides, or other fields.7.14.0to a7.13.0-based pre-release — intentional for testing.User Impact
STDOUT_HEARTBEATlines to stderr every 30s with queue statsqueue.put()path that caused sync stalls in job 77457394Can this PR be safely reverted and rolled back?
Link to Devin session: https://app.devin.ai/sessions/ad184113df474f0ba37ede09cdac7eaf