feat(source-zendesk-chat): migrate to incremental cursors#54151
feat(source-zendesk-chat): migrate to incremental cursors#54151Artem Inzhyyants (artem1205) merged 20 commits into
Conversation
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Octavia Squidington III (octavia-squidington-iii)
added
area/connectors
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
This reverts commit 7b9f91d.
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
…ndesk-chat-update-cursor # Conflicts: # airbyte-integrations/connectors/source-zendesk-chat/poetry.lock
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Octavia Squidington III (octavia-squidington-iii)
added
the
area/documentation
| type: CursorPagination | ||
| page_size: 1000 | ||
| cursor_value: '{{ response.get("end_time", {}) }}' | ||
| stop_condition: "{{ last_page_size < 1_000 }}" |
There was a problem hiding this comment.
maybe: last_page_size < page_size?
There was a problem hiding this comment.
yeah, but we do not pass page_size to the jinja interpolation, that's why i had to pass it as int 1_000
Artem Inzhyyants (artem1205)
requested review from
Baz (bazarnov) and
Maxime Carbonneau-Leclerc (maxi297)
Maxime Carbonneau-Leclerc (maxi297)
left a comment
Maxime Carbonneau-Leclerc (maxi297)
left a comment
There was a problem hiding this comment.
Have we run regression test on this one? Can I check the results?
|
|
||
| concurrency_level: | ||
| type: ConcurrencyLevel | ||
| default_concurrency: 10 |
There was a problem hiding this comment.
The limits from Zendesk Chat seem somewhat low i.e. 200 requests per minute for the team plan. I fear we might hit this limit with 10 concurrent threads. Should we set API budget at least?
There was a problem hiding this comment.
👍 added api budget, 200/60 = 3.3 -> we can set default concurrency to 4 later if 10 is too high.
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
after checking the results I realised that we need to implement epoch_microseconds for dateparser as well, because it is used by stateconverte, PR: airbytehq/airbyte-python-cdk#388 |
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
|
Cool, ping me again once the change is in this PR and regression test results are ready ❤️ |
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
Artem Inzhyyants (artem1205)
requested a review
from Maxime Carbonneau-Leclerc (maxi297)
|
regression tests failed expectedly records differ on Item root['properties']['last_login'] only. records differ on Item root['properties']['last_login'] only. |
Maxime Carbonneau-Leclerc (maxi297)
left a comment
Maxime Carbonneau-Leclerc (maxi297)
left a comment
There was a problem hiding this comment.
Pre-emptively approving assuming the full refresh regression tests passes
Artem Inzhyyants (artem1205)
deleted the
artem1205/source-zendesk-chat-update-cursor
branch
4 participants
What
Resolve https://github.com/airbytehq/airbyte-internal-issues/issues/11530
How
Review guide
airbyte-integrations/connectors/source-zendesk-chat/source_zendesk_chat/manifest.yamlUser Impact
Can this PR be safely reverted and rolled back?