Skip to content

refactor: connectors can launch themselves; deprecate: AirbyteEntrypoint and launch(), add SerDe methods on critical protocol classes#562

Draft
Aaron ("AJ") Steers (aaronsteers) wants to merge 15 commits intomainfrom
aj/refactor/connectors-can-launch-themselvs
Draft

refactor: connectors can launch themselves; deprecate: AirbyteEntrypoint and launch(), add SerDe methods on critical protocol classes#562
Aaron ("AJ") Steers (aaronsteers) wants to merge 15 commits intomainfrom
aj/refactor/connectors-can-launch-themselvs

Conversation

@aaronsteers
Copy link
Copy Markdown
Member

@aaronsteers Aaron ("AJ") Steers (aaronsteers) commented May 21, 2025

Request: Early Feedback.

Goal here is to detangle the tightly-coupled interdependencies:

  • AirbyteEntrypoint class
  • launch() method
  • Connector and Source classes themselves.

Enumerated changes:

  • parse_cli_args() made a dedicated helper function. Moved to its own utils module, ensuring no circular dependencies between the airbyte_entrypoint module and the connector class modules.
  • All Connector classes inherit class method called launch_with_cli_args()
  • The launch() function is deprecated (but not removed).
    • Internally, launch() it is now implemented via source.launch_with_cli_args() Calling it will emit a deprecation warning.
  • Goal is to fully deprecate and remove AirbyteEntrypoint (although probably not in this iteration).

Context

I finally accepted taking on this work, as it is needed to unblock:

The problem is: the AirbyteEntrypoint currently gatekeeps access to the connector class itself. And AirbyteEntrypoint doesn't have flexibility into calling results into memory vs pushing results to STDOUT.

Related: the airbyte_cdk.test.entrypoint_output.EntrypointOutput in some ways tries to mitigate this, but it has to in some cases fully re-invent the AirbyteEntrypoint implementation.

And since neither of these gives any flexibility in modes of execution or control of RAM overflow, this PR consolidates the execution to have fewer parts, with the (hopefully) the right interface layers to accomplish execution with minimal wheel-reinventing for callers.

Comment thread airbyte_cdk/__init__.py Outdated
@aaronsteers Aaron ("AJ") Steers (aaronsteers) changed the title refactor: connectors can launch themselves; deprecate: AirbyteEntrypoint and launch() refactor: connectors can launch themselves; deprecate: AirbyteEntrypoint and launch(), add SerDe methods on critical protocol classes May 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant