Skip to content

Commit 5472f5c

Browse files
authored
Merge branch 'main' into devin/1762664873-improve-parameters-schema-description
2 parents fd9aba8 + daf7d48 commit 5472f5c

38 files changed

Lines changed: 1168 additions & 1342 deletions

.github/pr-welcome-community.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,5 +27,6 @@ As needed or by request, Airbyte Maintainers can execute the following slash com
2727
- `/autofix` - Fixes most formatting and linting issues
2828
- `/poetry-lock` - Updates poetry.lock file
2929
- `/test` - Runs connector tests with the updated CDK
30+
- `/prerelease` - Triggers a prerelease publish with default arguments
3031

3132
If you have any questions, feel free to ask in the PR comments or join our [Slack community](https://airbytehq.slack.com/).

.github/pr-welcome-internal.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ Airbyte Maintainers can execute the following slash commands on your PR:
2626
- `/autofix` - Fixes most formatting and linting issues
2727
- `/poetry-lock` - Updates poetry.lock file
2828
- `/test` - Runs connector tests with the updated CDK
29+
- `/prerelease` - Triggers a prerelease publish with default arguments
2930
- `/poe build` - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
3031
- `/poe <command>` - Runs any poe command in the CDK environment
3132

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
name: On-Demand Prerelease
2+
3+
# Minimal permissions for security (addresses GitHub Advanced Security feedback)
4+
permissions:
5+
contents: read
6+
pull-requests: write
7+
issues: write
8+
9+
on:
10+
workflow_dispatch:
11+
inputs:
12+
pr:
13+
description: "PR Number"
14+
type: string
15+
required: false
16+
comment-id:
17+
description: "Comment ID (Optional)"
18+
type: string
19+
required: false
20+
21+
jobs:
22+
prerelease-on-demand:
23+
name: Trigger Prerelease Publish
24+
runs-on: ubuntu-24.04
25+
steps:
26+
- name: Authenticate as GitHub App
27+
uses: actions/create-github-app-token@v2
28+
id: get-app-token
29+
with:
30+
owner: "airbytehq"
31+
repositories: "airbyte-python-cdk"
32+
app-id: ${{ secrets.OCTAVIA_BOT_APP_ID }}
33+
private-key: ${{ secrets.OCTAVIA_BOT_PRIVATE_KEY }}
34+
35+
- name: Create URL to the run output
36+
id: vars
37+
run: echo "run-url=https://github.com/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" >> $GITHUB_OUTPUT
38+
39+
- name: Check that PR number is provided
40+
if: github.event.inputs.pr == ''
41+
run: |
42+
echo "Error: /prerelease command must be invoked on a pull request, not an issue."
43+
exit 1
44+
45+
- name: Get PR info
46+
id: pr-info
47+
run: |
48+
PR_JSON=$(gh api repos/${{ github.repository }}/pulls/${{ github.event.inputs.pr }})
49+
HEAD_REF=$(echo "$PR_JSON" | jq -r .head.ref)
50+
HEAD_REPO=$(echo "$PR_JSON" | jq -r .head.repo.full_name)
51+
echo "head-ref=${HEAD_REF}" >> $GITHUB_OUTPUT
52+
echo "head-repo=${HEAD_REPO}" >> $GITHUB_OUTPUT
53+
echo "PR branch: ${HEAD_REF} from ${HEAD_REPO}"
54+
env:
55+
GH_TOKEN: ${{ steps.get-app-token.outputs.token }}
56+
57+
- name: Check that PR is from this repository (not a fork)
58+
if: steps.pr-info.outputs.head-repo != github.repository
59+
run: |
60+
echo "Error: /prerelease only works for branches in this repository, not forks."
61+
echo "PR is from: ${{ steps.pr-info.outputs.head-repo }}"
62+
echo "Expected: ${{ github.repository }}"
63+
exit 1
64+
65+
- name: Append comment with job run link
66+
if: github.event.inputs.comment-id
67+
id: first-comment-action
68+
uses: peter-evans/create-or-update-comment@v4
69+
with:
70+
comment-id: ${{ github.event.inputs.comment-id }}
71+
issue-number: ${{ github.event.inputs.pr }}
72+
body: |
73+
> **Prerelease Job Info**
74+
>
75+
> This job triggers the publish workflow with default arguments to create a prerelease.
76+
>
77+
> Prerelease job started... [Check job output.][1]
78+
79+
[1]: ${{ steps.vars.outputs.run-url }}
80+
81+
- name: Trigger publish workflow
82+
id: trigger-publish
83+
uses: the-actions-org/workflow-dispatch@v4
84+
with:
85+
workflow: publish.yml
86+
token: ${{ steps.get-app-token.outputs.token }}
87+
ref: ${{ steps.pr-info.outputs.head-ref }}
88+
wait-for-completion: true
89+
display-workflow-run-url: false
90+
inputs: >-
91+
{
92+
"publish_to_pypi": "true",
93+
"publish_to_dockerhub": "true",
94+
"publish_manifest_server": "false",
95+
"update_connector_builder": "false"
96+
}
97+
98+
- name: Append success comment
99+
if: github.event.inputs.comment-id
100+
uses: peter-evans/create-or-update-comment@v4
101+
with:
102+
comment-id: ${{ steps.first-comment-action.outputs.comment-id }}
103+
reactions: hooray
104+
body: |
105+
> ✅ Prerelease workflow triggered successfully.
106+
>
107+
> View the publish workflow run: ${{ steps.trigger-publish.outputs.workflow-url }}
108+
109+
- name: Append failure comment
110+
if: failure() && github.event.inputs.comment-id
111+
uses: peter-evans/create-or-update-comment@v4
112+
with:
113+
comment-id: ${{ steps.first-comment-action.outputs.comment-id }}
114+
reactions: confused
115+
body: |
116+
> ❌ Failed to trigger prerelease workflow.

.github/workflows/publish.yml

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,10 @@ on:
2020
Note that this workflow is intended for prereleases. For public-facing stable releases,
2121
please use the GitHub Releases workflow instead:
2222
https://github.com/airbytehq/airbyte-python-cdk/blob/main/docs/RELEASES.md.
23-
For prereleases, please leave the version blank to use the detected version. Alternatively,
24-
you can override the dynamic versioning for special use cases.
23+
For prereleases, you can use the /prerelease slash command in a PR comment to trigger
24+
this workflow with default arguments. Alternatively, you can manually trigger this workflow
25+
and leave the version blank to use the detected version, or override the dynamic versioning
26+
for special use cases.
2527
required: false
2628
publish_to_pypi:
2729
description: "Publish to PyPI. If true, the workflow will publish to PyPI."

.github/workflows/slash_command_dispatch.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ jobs:
3535
test
3636
poetry-lock
3737
poe
38+
prerelease
3839
3940
# Notes regarding static-args:
4041
# - Slash commands can be invoked from both issues and comments.
@@ -63,6 +64,7 @@ jobs:
6364
- \`/autofix\` - Corrects any linting or formatting issues
6465
- \`/test\` - Runs the test suite
6566
- \`/poetry-lock\` - Re-locks dependencies and updates the poetry.lock file
67+
- \`/prerelease\` - Triggers a prerelease publish with default arguments
6668
- \`/help\` - Shows this help message"
6769
6870
if [[ "${{ github.event.comment.body }}" == "/help" ]]; then

airbyte_cdk/legacy/sources/declarative/declarative_stream.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55
from dataclasses import InitVar, dataclass, field
66
from typing import Any, Iterable, List, Mapping, MutableMapping, Optional, Union
77

8+
from typing_extensions import deprecated
9+
810
from airbyte_cdk.legacy.sources.declarative.incremental import (
911
GlobalSubstreamCursor,
1012
PerPartitionCursor,
@@ -13,7 +15,6 @@
1315
from airbyte_cdk.models import SyncMode
1416
from airbyte_cdk.sources.declarative.interpolation import InterpolatedString
1517
from airbyte_cdk.sources.declarative.migrations.state_migration import StateMigration
16-
from airbyte_cdk.sources.declarative.retrievers import SimpleRetriever
1718
from airbyte_cdk.sources.declarative.retrievers.async_retriever import AsyncRetriever
1819
from airbyte_cdk.sources.declarative.retrievers.retriever import Retriever
1920
from airbyte_cdk.sources.declarative.schema import DefaultSchemaLoader
@@ -28,6 +29,7 @@
2829
from airbyte_cdk.sources.types import Config, StreamSlice
2930

3031

32+
@deprecated("DeclarativeStream has been deprecated in favor of the concurrent DefaultStream")
3133
@dataclass
3234
class DeclarativeStream(Stream):
3335
"""
@@ -198,8 +200,6 @@ def state_checkpoint_interval(self) -> Optional[int]:
198200
return None
199201

200202
def get_cursor(self) -> Optional[Cursor]:
201-
if self.retriever and isinstance(self.retriever, SimpleRetriever):
202-
return self.retriever.cursor
203203
return None
204204

205205
def _get_checkpoint_reader(

airbyte_cdk/sources/declarative/auth/oauth.py

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
import logging
66
from dataclasses import InitVar, dataclass, field
77
from datetime import datetime, timedelta
8-
from typing import Any, List, Mapping, Optional, Union
8+
from typing import Any, List, Mapping, Optional, Tuple, Union
99

1010
from airbyte_cdk.sources.declarative.auth.declarative_authenticator import DeclarativeAuthenticator
1111
from airbyte_cdk.sources.declarative.interpolation.interpolated_boolean import InterpolatedBoolean
@@ -46,6 +46,9 @@ class DeclarativeOauth2Authenticator(AbstractOauth2Authenticator, DeclarativeAut
4646
refresh_request_headers (Optional[Mapping[str, Any]]): The request headers to send in the refresh request
4747
grant_type: The grant_type to request for access_token. If set to refresh_token, the refresh_token parameter has to be provided
4848
message_repository (MessageRepository): the message repository used to emit logs on HTTP requests
49+
refresh_token_error_status_codes (Tuple[int, ...]): Status codes to identify refresh token errors in response
50+
refresh_token_error_key (str): Key to identify refresh token error in response
51+
refresh_token_error_values (Tuple[str, ...]): List of values to check for exception during token refresh process
4952
"""
5053

5154
config: Mapping[str, Any]
@@ -72,9 +75,16 @@ class DeclarativeOauth2Authenticator(AbstractOauth2Authenticator, DeclarativeAut
7275
message_repository: MessageRepository = NoopMessageRepository()
7376
profile_assertion: Optional[DeclarativeAuthenticator] = None
7477
use_profile_assertion: Optional[Union[InterpolatedBoolean, str, bool]] = False
78+
refresh_token_error_status_codes: Tuple[int, ...] = ()
79+
refresh_token_error_key: str = ""
80+
refresh_token_error_values: Tuple[str, ...] = ()
7581

7682
def __post_init__(self, parameters: Mapping[str, Any]) -> None:
77-
super().__init__()
83+
super().__init__(
84+
refresh_token_error_status_codes=self.refresh_token_error_status_codes,
85+
refresh_token_error_key=self.refresh_token_error_key,
86+
refresh_token_error_values=self.refresh_token_error_values,
87+
)
7888
if self.token_refresh_endpoint is not None:
7989
self._token_refresh_endpoint: Optional[InterpolatedString] = InterpolatedString.create(
8090
self.token_refresh_endpoint, parameters=parameters

airbyte_cdk/sources/declarative/declarative_component_schema.yaml

Lines changed: 33 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -807,6 +807,10 @@ definitions:
807807
examples:
808808
- "created_at"
809809
- "{{ config['record_cursor'] }}"
810+
allow_catalog_defined_cursor_field:
811+
title: Allow Catalog Defined Cursor Field
812+
description: Whether the cursor allows users to override the default cursor_field when configuring their connection. The user defined cursor field will be specified from within the configured catalog.
813+
type: boolean
810814
start_value:
811815
title: Start Value
812816
description: The value that determines the earliest record that should be synced.
@@ -868,6 +872,10 @@ definitions:
868872
examples:
869873
- "created_at"
870874
- "{{ config['record_cursor'] }}"
875+
allow_catalog_defined_cursor_field:
876+
title: Allow Catalog Defined Cursor Field
877+
description: Whether the cursor allows users to override the default cursor_field when configuring their connection. The user defined cursor field will be specified from within the configured catalog.
878+
type: boolean
871879
cursor_datetime_formats:
872880
title: Cursor Datetime Formats
873881
type: array
@@ -1378,6 +1386,28 @@ definitions:
13781386
type: string
13791387
examples:
13801388
- "%Y-%m-%d %H:%M:%S.%f+00:00"
1389+
refresh_token_error_status_codes:
1390+
title: Refresh Token Error Status Codes
1391+
description: Status Codes to Identify refresh token error in response (Refresh Token Error Key and Refresh Token Error Values should be also specified). Responses with one of the error status code and containing an error value will be flagged as a config error
1392+
type: array
1393+
items:
1394+
type: integer
1395+
examples:
1396+
- [400, 500]
1397+
refresh_token_error_key:
1398+
title: Refresh Token Error Key
1399+
description: Key to Identify refresh token error in response (Refresh Token Error Status Codes and Refresh Token Error Values should be also specified).
1400+
type: string
1401+
examples:
1402+
- "error"
1403+
refresh_token_error_values:
1404+
title: Refresh Token Error Values
1405+
description: 'List of values to check for exception during token refresh process. Used to check if the error found in the response matches the key from the Refresh Token Error Key field (e.g. response={"error": "invalid_grant"}). Only responses with one of the error status code and containing an error value will be flagged as a config error'
1406+
type: array
1407+
items:
1408+
type: string
1409+
examples:
1410+
- ["invalid_grant", "invalid_permissions"]
13811411
refresh_token_updater:
13821412
title: Refresh Token Updater
13831413
description: When the refresh token updater is defined, new refresh tokens, access tokens and the access token expiry date are written back from the authentication response to the config object. This is important if the refresh token can only used once.
@@ -1419,7 +1449,7 @@ definitions:
14191449
examples:
14201450
- ["credentials", "token_expiry_date"]
14211451
refresh_token_error_status_codes:
1422-
title: Refresh Token Error Status Codes
1452+
title: (Deprecated - Use the same field on the OAuthAuthenticator level) Refresh Token Error Status Codes
14231453
description: Status Codes to Identify refresh token error in response (Refresh Token Error Key and Refresh Token Error Values should be also specified). Responses with one of the error status code and containing an error value will be flagged as a config error
14241454
type: array
14251455
items:
@@ -1428,14 +1458,14 @@ definitions:
14281458
examples:
14291459
- [400, 500]
14301460
refresh_token_error_key:
1431-
title: Refresh Token Error Key
1461+
title: (Deprecated - Use the same field on the OAuthAuthenticator level) Refresh Token Error Key
14321462
description: Key to Identify refresh token error in response (Refresh Token Error Status Codes and Refresh Token Error Values should be also specified).
14331463
type: string
14341464
default: ""
14351465
examples:
14361466
- "error"
14371467
refresh_token_error_values:
1438-
title: Refresh Token Error Values
1468+
title: (Deprecated - Use the same field on the OAuthAuthenticator level) Refresh Token Error Values
14391469
description: 'List of values to check for exception during token refresh process. Used to check if the error found in the response matches the key from the Refresh Token Error Key field (e.g. response={"error": "invalid_grant"}). Only responses with one of the error status code and containing an error value will be flagged as a config error'
14401470
type: array
14411471
items:

0 commit comments

Comments
 (0)