feat: add send_refresh_request_as_query_params toggle to OAuth authenticators#1041
Conversation
Add support for sending OAuth refresh token request parameters as URL query string parameters instead of in the POST body. This is required by OAuth providers like Gong that document their refresh endpoint with grant_type and refresh_token on the URL query string. Changes: - Add get_refresh_request_params() to AbstractOauth2Authenticator - Add build_refresh_request_query_params() that returns params or None - Pass params= to requests.request() in _make_handled_request() - Exclude query param keys from request body to prevent duplication - Add refresh_request_params to Oauth2Authenticator and SingleUseRefreshTokenOauth2Authenticator - Add refresh_request_query_params to declarative schema, Pydantic model, DeclarativeOauth2Authenticator, and model_to_component_factory - Add unit tests covering query params behavior for both native and declarative authenticators Co-Authored-By: bot_apk <apk@cognition.ai>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This CDK VersionYou can test this version of the CDK using the following: # Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1780092927-oauth-refresh-request-query-params#egg=airbyte-python-cdk[dev]' --help
# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1780092927-oauth-refresh-request-query-paramsPR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
|
PyTest Results (Fast)4 100 tests +6 4 088 ✅ +5 7m 16s ⏱️ - 1m 10s Results for commit 947e145. ± Comparison against base commit a7b0929. This pull request skips 1 test.♻️ This comment has been updated with latest results. |
There was a problem hiding this comment.
Pull request overview
Adds support for sending OAuth refresh-token parameters on the URL query string (in addition to the form body), required by providers like Gong whose refresh endpoint rejects body-only parameters. The change is additive — when the new field is unset, behavior is unchanged.
Changes:
- Adds
get_refresh_request_params()/build_refresh_request_query_params()onAbstractOauth2Authenticator, wiresparams=into the refresh HTTP call, and removes any duplicated keys from the body. - Threads a new
refresh_request_paramskwarg throughOauth2AuthenticatorandSingleUseRefreshTokenOauth2Authenticator, and a newrefresh_request_query_paramsfield through the declarative authenticator, schema, generated model, and factory. - Adds unit tests for both stacks and updates
mock_requesthelpers to accept the newparams=kwarg.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| airbyte_cdk/sources/streams/http/requests_native_auth/abstract_oauth.py | Adds query-param plumbing in refresh request build/send path and dedupes body keys. |
| airbyte_cdk/sources/streams/http/requests_native_auth/oauth.py | Adds refresh_request_params kwarg + getter on Oauth2Authenticator; forwarded by SingleUseRefreshTokenOauth2Authenticator. |
| airbyte_cdk/sources/declarative/auth/oauth.py | Adds refresh_request_query_params dataclass field interpolated via InterpolatedMapping. |
| airbyte_cdk/sources/declarative/declarative_component_schema.yaml | Exposes new optional refresh_request_query_params field on OAuthAuthenticator. |
| airbyte_cdk/sources/declarative/models/declarative_component_schema.py | Regenerated Pydantic model with the new field. |
| airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py | Wires new field into both declarative OAuth authenticator constructions. |
| unit_tests/sources/streams/http/requests_native_auth/test_requests_native_auth.py | Tests query-param emission, default-None behavior, and body de-duplication; updates mock_request signature. |
| unit_tests/sources/declarative/auth/test_oauth.py | Equivalent declarative tests and mock_request signature update. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Previously `build_refresh_request_query_params()` returned the static mapping provided at construction time. For `SingleUseRefreshTokenOauth2Authenticator` this meant the rotated refresh_token was persisted to config and emitted as a control message, but the next refresh request still carried the original (now-invalidated) token in the URL query string, breaking long-running syncs. `build_refresh_request_body()` already avoids this trap by reading from the dynamic getters on every call. Apply the same pattern here for the refresh token field specifically — other custom params (e.g. user-overridden `grant_type`) are still passed through unchanged. Regression test exercises two back-to-back refreshes against `SingleUseRefreshTokenOauth2Authenticator` and asserts the URL on the second request carries the rotated refresh token. Co-Authored-By: bot_apk <apk@cognition.ai>
|
One naming nit worth addressing before merge: the declarative manifest field is Since it's a new kwarg with a |
…y_params Per review feedback, rename the Python-level kwarg on the OAuth authenticators to match the manifest field name. The kwarg was added in this PR with a None default, so this is a non-breaking rename: - Oauth2Authenticator and SingleUseRefreshTokenOauth2Authenticator constructor kwarg refresh_request_params -> refresh_request_query_params - get_refresh_request_params() -> get_refresh_request_query_params() - Instance attribute _refresh_request_params -> _refresh_request_query_params - Factory and tests updated to match Co-Authored-By: bot_apk <apk@cognition.ai>
|
sophiecuiy — addressed the naming nit in bc78486. Renamed throughout: kwarg Non-breaking: the kwarg was added in this PR with a |
…h_request_as_query_params boolean Per maintainer feedback (#1041), collapse the user API to a single boolean toggle: - DROP `refresh_request_query_params: Mapping[str, Any]` (kwarg + declarative field). The Mapping forced users to manually re-author standard OAuth refresh args with interpolation strings and made wire shape data-driven instead of policy-driven. - ADD `send_refresh_request_as_query_params: bool = False`. When `True`, the standard refresh args (`grant_type`, `refresh_token`, client credentials unless an `Authorization` header is set, scopes, plus any `refresh_request_body` extras) are emitted on the URL query string and the request body is empty. When `False` (default), behavior is unchanged. Implementation: shared `_build_standard_refresh_args()` helper used by both `build_refresh_request_body()` and `build_refresh_request_query_params()`, gated by `should_send_refresh_request_as_query_params()`. Rotation persistence (the bug the previous design exposed) is now automatic because the body builder already pulls from live getters (`get_refresh_token()`, `get_grant_type()`, etc.) — no separate fix needed. Co-Authored-By: bot_apk <apk@cognition.ai>
Replaces the explicit `refresh_request_query_params` mapping with a single boolean toggle that opts the OAuth refresh into the URL-query-string shape required by Gong's documented refresh endpoint. Behavior on the wire is unchanged. Tracks the upstream CDK API redesign in airbytehq/airbyte-python-cdk#1041. Co-Authored-By: bot_apk <apk@cognition.ai>
Co-Authored-By: bot_apk <apk@cognition.ai>
refresh_request_query_params to OAuth authenticatorssend_refresh_request_as_query_params toggle to OAuth authenticators
|
sophiecuiy — redesigned the API per your suggestion. Single boolean toggle, no Mapping. Pushed in 53cbd43 + 63b1650. API change # before
refresh_request_query_params:
grant_type: refresh_token
refresh_token: "{{ config['credentials']['refresh_token'] }}"
# after
send_refresh_request_as_query_params: trueWhen Implementation: shared helper, gated by the boolean def _build_standard_refresh_args(self) -> MutableMapping[str, Any]:
args = {self.get_grant_type_name(): self.get_grant_type()}
if not credentials_in_header(self):
args[self.get_client_id_name()] = self.get_client_id()
args[self.get_client_secret_name()] = self.get_client_secret()
args[self.get_refresh_token_name()] = self.get_refresh_token()
if self.get_scopes(): args["scopes"] = self.get_scopes()
for k, v in (self.get_refresh_request_body() or {}).items():
if k not in args: args[k] = v
return args
def build_refresh_request_body(self) -> Mapping[str, Any]:
return {} if self.should_send_refresh_request_as_query_params() \
else self._build_standard_refresh_args()
def build_refresh_request_query_params(self) -> Mapping[str, Any] | None:
return self._build_standard_refresh_args() \
if self.should_send_refresh_request_as_query_params() else NoneBecause Live-tenant verification (production Gong, this PR's branch + the matching source-gong manifest in airbytehq/airbyte#78521)
PR title and description updated to reflect the new design. |
Reverts an incidental variable rename (payload -> args) so the diff against main stays minimal in the body-building logic. The helper itself is named _build_standard_refresh_args, but the local variable that historically held the body dict stays named payload, matching the convention in the rest of this file. Co-Authored-By: bot_apk <apk@cognition.ai>
|
sophiecuiy — fair nit on the 75 OAuth unit tests still pass; ruff format / lint / mypy clean. |
CDK PR airbytehq/airbyte-python-cdk#1041 has been merged and released as v7.22.0, which publishes airbyte/source-declarative-manifest:7.22.0 (linux/amd64+linux/arm64 manifest list digest sha256:02df50154030837bc6de45d9d5362d7f038a507040fb0049d57a9a64a6ac4906). Reverts the temporary prerelease pin (7.21.0.post6.dev27226613720@sha256:525e53...) used during pre-merge verification and points at the stable released SDM that contains the `send_refresh_request_as_query_params` field this manifest depends on. Co-Authored-By: bot_apk <apk@cognition.ai>
Summary
Adds an opt-in boolean —
send_refresh_request_as_query_params— that routes the standard OAuth refresh args (grant_type,refresh_token, client credentials when not in anAuthorizationheader, scopes, plus anyrefresh_request_bodyextras) onto the URL query string of the refresh request instead of the form-encoded POST body.This unblocks providers like Gong that document their refresh endpoint as
POSTwith refresh args on the URL query string and an empty body. Today the CDK always sends those args in the body, which Gong's refresh endpoint rejects.Resolves airbytehq/airbyte-internal-issues#16467.
What the toggle does
send_refresh_request_as_query_paramsfalse(default)trueWhen
true,client_id/client_secretare still omitted from the query string if a refreshAuthorizationheader is set, mirroring the pre-existing body-shaping logic.Implementation
A shared helper
_build_standard_refresh_args()(inAbstractOauth2Authenticator) is the single source of truth for "what goes into a refresh request" — it's used by bothbuild_refresh_request_body()andbuild_refresh_request_query_params(), gated by a new abstract gettershould_send_refresh_request_as_query_params() -> bool(defaultFalse):Because the helper pulls live values from
get_refresh_token()etc., rotation persistence is automatic — rotated refresh tokens are picked up on every refresh, no separate fix needed (the live test below exercises this end-to-end against Gong).Files touched
airbyte_cdk/sources/streams/http/requests_native_auth/abstract_oauth.py— new_build_standard_refresh_args()helper, new abstract gettershould_send_refresh_request_as_query_params(), body / query-param builders rewritten in terms of the helper.airbyte_cdk/sources/streams/http/requests_native_auth/oauth.py—Oauth2AuthenticatorandSingleUseRefreshTokenOauth2Authenticatoracceptsend_refresh_request_as_query_params: bool = Falseand override the getter.airbyte_cdk/sources/declarative/auth/oauth.py—DeclarativeOauth2Authenticatorexposessend_refresh_request_as_query_params: bool = False(noInterpolatedMappingneeded) and threads it into the concrete authenticator.airbyte_cdk/sources/declarative/declarative_component_schema.yaml+ regenerated Pydantic model — adds the boolean field onOAuthAuthenticator.airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py— passes the boolean through for both declarative authenticator code paths.params=anddata=kwargs on the underlyingrequests.request(...)call (not just the helper return values), including a rotation regression test onSingleUseRefreshTokenOauth2Authenticator.Live-tenant verification (production Gong, this PR's CDK + the matching source-gong manifest in airbytehq/airbyte#78521)
checksFIkaCoMTE/4(bootstrap)5m9iFtGDqndSCONNECTION_STATUS: SUCCEEDEDcheck5m9iFtGDqndS(rotated)f0rMIv0V0eRJCONNECTION_STATUS: SUCCEEDEDdiscoverf0rMIv0V0eRJCATALOGwith 6 streamsread users(backdated expiry)f0rMIv0V0eRJ→ forced refreshaHePkxlMUgq1check#2 is the smoking gun for rotation persistence: Gong invalidates the previous refresh token on rotation, so without the live-getter behavior described above it would have 401'd.Backward compatibility
False, so all existing authenticators retain their previous behavior (params in body, no query string on the refresh URL).refresh_request_bodyfield is unchanged and still flows into the body when the toggle isFalse. When the toggle isTrueit flows through_build_standard_refresh_args()into the query string.Link to Devin session: https://app.devin.ai/sessions/f35f567405264ba4b512e57abf9be8cd