feat(source-pinterest): Update CDK to v6 by tolik0 · Pull Request #65960 · airbytehq/airbyte

Anatolii Yatsuk (tolik0) · 2025-09-05T16:55:56Z

What

Update CDK to v6.
Migration to v7 is currently blocked, as version 7 expects all streams — including those in Python implementations — to align with the concurrent CDK.

{"type":"LOG","log":{"level":"FATAL","message":"'NoneType' object has no attribute 'ensure_at_least_one_state_emitted'\nTraceback (most recent call last):\n  File \"/home/anatolii/airbytehq/airbyte/airbyte-integrations/connectors/source-pinterest/main.py\", line 9, in <module>\n    run()\n  File \"/home/anatolii/airbytehq/airbyte/airbyte-integrations/connectors/source-pinterest/source_pinterest/run.py\", line 54, in run\n    launch(source, _args)\n  File \"/home/anatolii/.cache/pypoetry/virtualenvs/source-pinterest-_YSa_W83-py3.11/lib/python3.11/site-packages/airbyte_cdk/entrypoint.py\", line 377, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/home/anatolii/.cache/pypoetry/virtualenvs/source-pinterest-_YSa_W83-py3.11/lib/python3.11/site-packages/airbyte_cdk/entrypoint.py\", line 207, in run\n    yield from map(\n  File \"/home/anatolii/.cache/pypoetry/virtualenvs/source-pinterest-_YSa_W83-py3.11/lib/python3.11/site-packages/airbyte_cdk/entrypoint.py\", line 280, in read\n    for message in self.source.read(self.logger, config, catalog, state):\n  File \"/home/anatolii/.cache/pypoetry/virtualenvs/source-pinterest-_YSa_W83-py3.11/lib/python3.11/site-packages/airbyte_cdk/sources/declarative/concurrent_declarative_source.py\", line 379, in read\n    yield from self._concurrent_source.read(selected_concurrent_streams)\n  File \"/home/anatolii/.cache/pypoetry/virtualenvs/source-pinterest-_YSa_W83-py3.11/lib/python3.11/site-packages/airbyte_cdk/sources/concurrent_source/concurrent_source.py\", line 126, in read\n    yield from self._consume_from_queue(\n  File \"/home/anatolii/.cache/pypoetry/virtualenvs/source-pinterest-_YSa_W83-py3.11/lib/python3.11/site-packages/airbyte_cdk/sources/concurrent_source/concurrent_source.py\", line 147, in _consume_from_queue\n    yield from self._handle_item(\n  File \"/home/anatolii/.cache/pypoetry/virtualenvs/source-pinterest-_YSa_W83-py3.11/lib/python3.11/site-packages/airbyte_cdk/sources/concurrent_source/concurrent_source.py\", line 167, in _handle_item\n    yield from concurrent_stream_processor.on_partition_generation_completed(queue_item)\n  File \"/home/anatolii/.cache/pypoetry/virtualenvs/source-pinterest-_YSa_W83-py3.11/lib/python3.11/site-packages/airbyte_cdk/sources/concurrent_source/concurrent_read_processor.py\", line 86, in on_partition_generation_completed\n    yield from self._on_stream_is_done(stream_name)\n  File \"/home/anatolii/.cache/pypoetry/virtualenvs/source-pinterest-_YSa_W83-py3.11/lib/python3.11/site-packages/airbyte_cdk/sources/concurrent_source/concurrent_read_processor.py\", line 239, in _on_stream_is_done\n    stream.cursor.ensure_at_least_one_state_emitted()\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAttributeError: 'NoneType' object has no attribute 'ensure_at_least_one_state_emitted'"}}

Resolves: https://github.com/airbytehq/airbyte-internal-issues/issues/14217

How

Review guide

User Impact

Can this PR be safely reverted and rolled back?

YES 💚
NO ❌

github-actions · 2025-09-05T16:56:21Z

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Helpful Resources

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

/format-fix - Fixes most formatting issues.
/bump-version - Bumps connector versions.
- You can specify a custom changelog by passing changelog. Example: /bump-version changelog="My cool update"
- Leaving the changelog arg blank will auto-populate the changelog from the PR title.
/run-cat-tests - Runs legacy CAT tests (Connector Acceptance Tests)
/build-connector-images - Builds and publishes a pre-release docker image for the modified connector(s).
/poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
/poe source example lock - Alias for /poe connector source-example lock.
/poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
/poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.

📝 Edit this welcome message.

Anatolii Yatsuk (tolik0) · 2025-09-05T16:56:39Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (7d8b8c9)

github-actions · 2025-09-05T16:58:25Z

`source-pinterest` Connector Test Results

83 tests 78 ✅ 55s ⏱️
2 suites 5 💤
2 files 0 ❌

Results for commit 412b505.

♻️ This comment has been updated with latest results.

Anatolii Yatsuk (tolik0) · 2025-09-05T16:58:56Z

/bump-version

Bump Version job started... Check job output.

✅ Changes applied successfully. (d174c5c)

github-actions · 2025-09-05T17:08:46Z

Deploy preview for airbyte-docs ready!

✅ Preview
https://airbyte-docs-5o12vdfyt-airbyte-growth.vercel.app

Built with commit 412b505.
This pull request is being automatically deployed with vercel-action

Anatolii Yatsuk (tolik0) · 2025-09-08T11:40:54Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (7ddb72b)

Anatolii Yatsuk (tolik0) · 2025-09-08T14:22:41Z

/poe source pinterest use-cdk-latest

Running poe source pinterest use-cdk-latest...

Link to job logs.

🤖 Auto-commit successful: f08bbaa

🟦 Poe command source pinterest use-cdk-latest completed successfully.

…est`

Anatolii Yatsuk (tolik0) · 2025-09-08T14:57:49Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (ad32260)

Anatolii Yatsuk (tolik0) · 2025-09-08T19:17:43Z

/poe source pinterest use-cdk-latest

Running poe source pinterest use-cdk-latest...

Link to job logs.

🤖 Auto-commit successful: 00fba70

🟦 Poe command source pinterest use-cdk-latest completed successfully.

…est`

Brian Lai (brianjlai)

added some comments around the shift back to v6 CDK. feel free to ping me if you want to discuss more about the changes. but yeah as max mentioned, v7 would have some shenanigans. but once its fully ready to migrate to manifest-only most of the pain points should no longer be relevant

Brian Lai (brianjlai) · 2025-09-08T18:15:42Z

 python = "^3.10,<3.12"
 pendulum = "==2.1.2"
-airbyte-cdk = "^4"
+airbyte-cdk = "^7.0.1"


this would work, but we've usually just used ^7. Did you have to do the full version for some reason to get the latest CDK?

It is just how the /poe source pinterest use-cdk-latest updated it, because I had a failed CI check with ^7

Brian Lai (brianjlai) · 2025-09-08T18:17:01Z

        title: "Account ID"
        description: "The Pinterest account ID you want to fetch data for. This ID must be provided to filter the data for a specific account."
        examples: ["1234567890"]
+      num_workers:


as per the conversation during retro about the confusion between Airbyte workers and connector workers, lets change this to num_threads and in the title and description let's just say "Number of concurrent threadsandThe number of parallel threads to use for the sync.`

Brian Lai (brianjlai) · 2025-09-08T18:17:49Z

+concurrency_level:
+  type: ConcurrencyLevel
+  default_concurrency: "{{ config['num_workers'] or 2 }}"
+  max_concurrency: 20


can you add a comment about where the boundaries for concurrency level were determined

Brian Lai (brianjlai) · 2025-09-08T19:35:52Z


        return declarative_streams + report_streams

-    def _create_ad_accounts_stream(self, config):


unfortunately, I suspect that because we're still going to be using v6 of the CDK here, we might still run into this same issue since RFR on python streams is still auto-enabled.

If we were going to v7, then you are right that RFR w/ synthetic cursors is not used anymore and we could share the streams. It may be safer to leave this and once we go to v7 + manifest-only, this whole section will get deleted anyway

I encountered the same issue with version 6 — AdAccounts was created as a DefaultStream, but we’re iterating through it using stream_slices. I tested it and confirmed that we’re still receiving all AdAccounts records when running a sync for multiple report streams.
Would it be safer to explicitly set is_resumable = False?

I tested in the Cloud and got the same number of records for the old and new versions:
https://cloud.airbyte.com/workspaces/9ef0ca9a-d759-484c-8499-93565374eaea/connections/dbcd0609-3073-4bd6-b313-eaea6089c698/timeline
https://cloud.airbyte.com/workspaces/9ef0ca9a-d759-484c-8499-93565374eaea/connections/246f95fc-fe40-43f5-bd70-43eca230b1c5/timeline

ah yeah you're right we have part of the changes to return back DefaultStream still in v6, so that is still a real scenario especially for a simple stream like ad accounts. its fine how you have it.

Would it be safer to explicitly set is_resumable = False?

Yeah I think it would be. So let's disable it. Either way once we go to v7 that setting won't matter any more (once on v7 we'll resume incremental syncs running as full refresh, but full refresh streams will only checkpoint once at the end). This should hopefully just be a short term behavior anyway

Its possible the reason records matched was because we would only see this appear if we have more than one page of ad accounts, but I imagine most customers don't have many pages of ad accounts.

Brian Lai (brianjlai) · 2025-09-08T20:19:42Z

+  dockerImageTag: 2.1.9-rc.1
  dockerRepository: airbyte/source-pinterest
  connectorBuildOptions:
    baseImage: docker.io/airbyte/python-connector-base:4.0.0@sha256:d9894b6895923b379f3006fa251147806919c62b7d9021b5cd125bb67d7bbe22


small nit, but I think we should also be bumping the base image to 4.0.2 since we had some recent updates to the image to get some additional security fixes: https://hub.docker.com/layers/airbyte/python-connector-base/4.0.2/images/sha256-f24aa7ddd043ec3c21851d814e95736deb3e1e657b74def0f1dec14e4d51504e

#62909

Brian Lai (brianjlai) · 2025-09-08T20:39:02Z

                yield record


+class AdAccounts(PinterestStream):


pending the above comment about not being able to reuse the same AdAccounts stream, would it make sense to continue to use the low-code version of the stream instead?

i assume we needed this change since in v7 we would get back a DefaultStream which might incompatible with the python streams, but now back on v6 it should be a DeclarativeStream.

Anatolii Yatsuk (tolik0) · 2025-09-09T16:45:58Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (0a4ed49)

Brian Lai (brianjlai)

lgtm!

Brian Lai (brianjlai) · 2025-09-10T17:20:19Z


        return declarative_streams + report_streams

-    def _create_ad_accounts_stream(self, config):


ah yeah you're right we have part of the changes to return back DefaultStream still in v6, so that is still a real scenario especially for a simple stream like ad accounts. its fine how you have it.

Would it be safer to explicitly set is_resumable = False?

Yeah I think it would be. So let's disable it. Either way once we go to v7 that setting won't matter any more (once on v7 we'll resume incremental syncs running as full refresh, but full refresh streams will only checkpoint once at the end). This should hopefully just be a short term behavior anyway

Its possible the reason records matched was because we would only see this appear if we have more than one page of ad accounts, but I imagine most customers don't have many pages of ad accounts.

Brian Lai (brianjlai) · 2025-09-10T17:50:51Z

 description = "Source implementation for Pinterest."
 authors = [ "Airbyte <contact@airbyte.io>",]
-license = "MIT"
+license = "ELv2"


good catch!

Update CDK to v6

98b05dc

Anatolii Yatsuk (tolik0) marked this pull request as ready for review September 5, 2025 16:56

Anatolii Yatsuk (tolik0) self-assigned this Sep 5, 2025

chore: auto-fix lint and format issues

7d8b8c9

Octavia Squidington III (octavia-squidington-iii) and others added 2 commits September 5, 2025 17:02

chore: bump-version

d174c5c

Add progressive rollout

c71cf15

Update license

ac7ce72

Octavia Squidington III (octavia-squidington-iii) added connectors/source/pinterest labels Sep 5, 2025

Fix unit tests

33f122f

Octavia Squidington III (octavia-squidington-iii) and others added 2 commits September 8, 2025 11:42

chore: auto-fix lint and format issues

7ddb72b

Update CDK version

d515514

Anatolii Yatsuk (tolik0) changed the title ~~feat(source-pinterest): Update CDK to v6~~ feat(source-pinterest): Update CDK to v7 Sep 8, 2025

octavia-bot Bot and others added 2 commits September 8, 2025 14:24

Auto-committed changes from Poe command `source pinterest use-cdk-lat…

f08bbaa

…est`

Fix unit tests

b6ed42f

chore: auto-fix lint and format issues

ad32260

Anatolii Yatsuk (tolik0) requested a review from a team September 8, 2025 15:39

Rollback changes for CDK v7

bcdc4c0

octavia-bot Bot and others added 3 commits September 8, 2025 19:19

Auto-committed changes from Poe command `source pinterest use-cdk-lat…

00fba70

…est`

Fix incremental unit tests

5056c41

Pin CDK to v6

2fd6d92

Brian Lai (brianjlai) reviewed Sep 8, 2025

View reviewed changes

Anatolii Yatsuk (tolik0) added 2 commits September 9, 2025 19:39

Fix comment from review

a2d8e88

Update authenticator to avoid getting new access token for every stream

22a27ec

Octavia Squidington III (octavia-squidington-iii) and others added 3 commits September 9, 2025 16:47

chore: auto-fix lint and format issues

0a4ed49

Fix base image

1fba931

Merge branch 'master' into tolik0/source-pinterest/update-cdk-to-v6

84d6b20

Anatolii Yatsuk (tolik0) requested a review from Brian Lai (brianjlai) September 10, 2025 12:06

Merge branch 'master' into tolik0/source-pinterest/update-cdk-to-v6

412b505

Brian Lai (brianjlai) approved these changes Sep 10, 2025

View reviewed changes

Anatolii Yatsuk (tolik0) merged commit aadfd9a into master Sep 10, 2025
29 of 30 checks passed

Anatolii Yatsuk (tolik0) deleted the tolik0/source-pinterest/update-cdk-to-v6 branch September 10, 2025 20:32

Maxime Carbonneau-Leclerc (maxi297) changed the title ~~feat(source-pinterest): Update CDK to v7~~ feat(source-pinterest): Update CDK to v6 Sep 11, 2025


		return declarative_streams + report_streams

		def _create_ad_accounts_stream(self, config):

Conversation

Anatolii Yatsuk (tolik0) commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Review guide

User Impact

Can this PR be safely reverted and rolled back?

Uh oh!

github-actions Bot commented Sep 5, 2025

👋 Greetings, Airbyte Team Member!

Helpful Resources

PR Slash Commands

Uh oh!

Anatolii Yatsuk (tolik0) commented Sep 5, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

source-pinterest Connector Test Results

Uh oh!

Anatolii Yatsuk (tolik0) commented Sep 5, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Anatolii Yatsuk (tolik0) commented Sep 8, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Anatolii Yatsuk (tolik0) commented Sep 8, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Anatolii Yatsuk (tolik0) commented Sep 8, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Anatolii Yatsuk (tolik0) commented Sep 8, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Brian Lai (brianjlai) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Anatolii Yatsuk (tolik0) commented Sep 9, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Brian Lai (brianjlai) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

Anatolii Yatsuk (tolik0) commented Sep 5, 2025 •

edited

Loading

Anatolii Yatsuk (tolik0) commented Sep 5, 2025 •

edited by github-actions Bot

Loading

github-actions Bot commented Sep 5, 2025 •

edited

Loading

`source-pinterest` Connector Test Results

Anatolii Yatsuk (tolik0) commented Sep 5, 2025 •

edited by github-actions Bot

Loading

github-actions Bot commented Sep 5, 2025 •

edited

Loading

Anatolii Yatsuk (tolik0) commented Sep 8, 2025 •

edited by github-actions Bot

Loading

Anatolii Yatsuk (tolik0) commented Sep 8, 2025 •

edited by github-actions Bot

Loading

Anatolii Yatsuk (tolik0) commented Sep 8, 2025 •

edited by github-actions Bot

Loading

Anatolii Yatsuk (tolik0) commented Sep 8, 2025 •

edited by github-actions Bot

Loading

Anatolii Yatsuk (tolik0) commented Sep 9, 2025 •

edited by github-actions Bot

Loading