Skip to content

Feat: Make CDK runnable in WASM #672

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Aaron ("AJ") Steers (aaronsteers) wants to merge 36 commits into
base: main
Choose a base branch
from
Draft

Feat: Make CDK runnable in WASM #672

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
7937514
feat: make psutil, rapidfuzz, and google-cloud-secret-manager optiona…
devin-ai-integration[bot] Jul 23, 2025
8bc5492
style: auto-format declarative component schema
devin-ai-integration[bot] Jul 23, 2025
5d46da6
fix: completely remove psutil and rapidfuzz dependencies
devin-ai-integration[bot] Jul 23, 2025
68490dc
feat: Add WASM compatibility fallback from serpyco-rs to serpyco
devin-ai-integration[bot] Jul 23, 2025
14f1ce3
revert formatting on generated file
aaronsteers Jul 23, 2025
20a43d6
clean up pyproject diff
aaronsteers Jul 23, 2025
07f0b2a
move install condition to a constant
aaronsteers Jul 23, 2025
ebc5412
Merge remote-tracking branch 'origin/devin/1753304576-wasm-compatibil…
aaronsteers Jul 23, 2025
ca46edf
resolve pytz and pandas version conflicts in wasm
aaronsteers Jul 23, 2025
62925ee
widen numpy version
aaronsteers Jul 23, 2025
bda9765
poetry lock
aaronsteers Jul 23, 2025
c5b0767
try conditional whenever install (temporary)
aaronsteers Jul 23, 2025
7468e2e
remove 'serpyco' compatibility import (no pure wheels)
aaronsteers Jul 23, 2025
a648429
allow newer version of cryptography
aaronsteers Jul 23, 2025
bc1fb21
fall back to basic custom serializer
aaronsteers Jul 23, 2025
814136b
remove more custom serpyco logic
aaronsteers Jul 24, 2025
4db3c8b
add missing callable class import
aaronsteers Jul 24, 2025
ca18c56
rework whenever dependency
aaronsteers Jul 24, 2025
b924fb2
allow dynamic inputs to run()
aaronsteers Jul 24, 2025
b18e7c2
cherry-pick-me: replace serializer classes with helper functions
aaronsteers Jul 24, 2025
b3807e3
fix circular ref
aaronsteers Jul 24, 2025
cbe0ba4
use dacite, add a wasm test html file
aaronsteers Jul 24, 2025
833429c
update script
aaronsteers Jul 24, 2025
e389081
improve script logic
aaronsteers Jul 24, 2025
df11705
split into three buttons
aaronsteers Jul 24, 2025
70f7f43
whitespace cleanup
aaronsteers Jul 24, 2025
94e073f
add airbyte_cdk.connector_builder.main:run
aaronsteers Jul 24, 2025
974d7d4
updated wasm test page
aaronsteers Jul 24, 2025
595581b
switch airbyte-protocol package back to pydantic
aaronsteers Jul 24, 2025
1ac75c6
remove stray import
aaronsteers Jul 24, 2025
005b079
remove old 'airbyte_protocol_dataclasses' refs
aaronsteers Jul 24, 2025
d62072a
fix refs
aaronsteers Jul 24, 2025
cfeda8c
fix missing import
aaronsteers Jul 24, 2025
f33e4a3
try forcing reduced concurrency
aaronsteers Jul 24, 2025
e0a1dac
swap to rick-and-morty via gist
aaronsteers Jul 24, 2025
b90f3f5
reword
aaronsteers Jul 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion airbyte_cdk/models/airbyte_protocol.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,16 @@
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#

import sys
from dataclasses import InitVar, dataclass
from typing import Annotated, Any, Dict, List, Mapping, Optional, Union

from airbyte_protocol_dataclasses.models import * # noqa: F403 # Allow '*'
from serpyco_rs.metadata import Alias

if sys.platform == 'emscripten':
from serpyco.metadata import Alias
else:
from serpyco_rs.metadata import Alias
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix quote consistency for pipeline compliance.

The conditional import approach for the Alias class is perfect for WASM support! Just need to fix the quote style to match the pipeline requirements.

Could you update the quote style to be consistent, wdyt?

-if sys.platform == 'emscripten':
+if sys.platform == "emscripten":
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if sys.platform == 'emscripten':
from serpyco.metadata import Alias
else:
from serpyco_rs.metadata import Alias
if sys.platform == "emscripten":
from serpyco.metadata import Alias
else:
from serpyco_rs.metadata import Alias
🤖 Prompt for AI Agents 
In airbyte_cdk/models/airbyte_protocol.py around lines 11 to 14, the quote style
used in the conditional import statement is inconsistent. Update all string
literals in this block to use the same type of quotes (either single or double)
consistently to comply with the pipeline's style requirements.


# ruff: noqa: F405 # ignore fuzzy import issues with 'import *'

Expand Down
11 changes: 9 additions & 2 deletions airbyte_cdk/models/airbyte_protocol_serializers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
import sys
from typing import Any, Dict

from serpyco_rs import CustomType, Serializer

from .airbyte_protocol import ( # type: ignore[attr-defined] # all classes are imported to airbyte_protocol via *
AirbyteCatalog,
AirbyteMessage,
Expand All @@ -15,6 +14,14 @@
ConnectorSpecification,
)

USE_RUST_BACKEND = sys.platform != "emscripten"
"""When run in WASM, use the pure Python backend for serpyco."""

if USE_RUST_BACKEND:
from serpyco_rs import CustomType, Serializer
else:
from serpyco import CustomType, Serializer

Comment on lines +17 to +29
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Excellent approach with the constant, but fix mypy issue.

I love the USE_RUST_BACKEND constant approach - it makes the code much more readable than inline platform checks! However, mypy is complaining about the name redefinition in the conditional imports.

Could you fix the mypy issue by using type: ignore comments or restructuring the imports, wdyt?

 if USE_RUST_BACKEND:
-    from serpyco_rs import CustomType, Serializer
+    from serpyco_rs import CustomType, Serializer  # type: ignore[misc]
 else:
-    from serpyco import CustomType, Serializer
+    from serpyco import CustomType, Serializer  # type: ignore[misc]

Alternatively, you could import with aliases and then assign to common names:

 if USE_RUST_BACKEND:
-    from serpyco_rs import CustomType, Serializer
+    from serpyco_rs import CustomType as _CustomType, Serializer as _Serializer
 else:
-    from serpyco import CustomType, Serializer
+    from serpyco import CustomType as _CustomType, Serializer as _Serializer
+
+CustomType = _CustomType
+Serializer = _Serializer
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
USE_RUST_BACKEND = sys.platform != "emscripten"
"""When run in WASM, use the pure Python backend for serpyco."""
if USE_RUST_BACKEND:
from serpyco_rs import CustomType, Serializer
else:
from serpyco import CustomType, Serializer
USE_RUST_BACKEND = sys.platform != "emscripten"
"""When run in WASM, use the pure Python backend for serpyco."""
if USE_RUST_BACKEND:
from serpyco_rs import CustomType, Serializer # type: ignore[misc]
else:
from serpyco import CustomType, Serializer # type: ignore[misc]
Suggested change
USE_RUST_BACKEND = sys.platform != "emscripten"
"""When run in WASM, use the pure Python backend for serpyco."""
if USE_RUST_BACKEND:
from serpyco_rs import CustomType, Serializer
else:
from serpyco import CustomType, Serializer
USE_RUST_BACKEND = sys.platform != "emscripten"
"""When run in WASM, use the pure Python backend for serpyco."""
if USE_RUST_BACKEND:
from serpyco_rs import CustomType as _CustomType, Serializer as _Serializer
else:
from serpyco import CustomType as _CustomType, Serializer as _Serializer
CustomType = _CustomType
Serializer = _Serializer
🧰 Tools
🪛 GitHub Actions: Linters

[error] 23-23: mypy: Name "CustomType" already defined (possibly by an import) [no-redef]

[error] 23-23: mypy: Name "Serializer" already defined (possibly by an import) [no-redef]

🤖 Prompt for AI Agents 
In airbyte_cdk/models/airbyte_protocol_serializers.py around lines 17 to 24,
mypy raises an error due to the redefinition of CustomType and Serializer in the
conditional imports. To fix this, import the modules with distinct aliases based
on the condition and then assign the common names CustomType and Serializer to
the appropriate aliases. This avoids name redefinition and satisfies mypy
without losing readability.


class AirbyteStateBlobType(CustomType[AirbyteStateBlob, Dict[str, Any]]):
def serialize(self, value: AirbyteStateBlob) -> Dict[str, Any]:
Expand Down
7 changes: 6 additions & 1 deletion airbyte_cdk/test/entrypoint_wrapper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
import json
import logging
import re
import sys
import tempfile
import traceback
from collections import deque
Expand All @@ -28,7 +29,11 @@

import orjson
from pydantic import ValidationError as V2ValidationError
from serpyco_rs import SchemaValidationError

if sys.platform == 'emscripten':
from serpyco import SchemaValidationError
else:
from serpyco_rs import SchemaValidationError
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix quote consistency for pipeline compliance.

The conditional import logic is solid for WASM compatibility! However, there's a formatting issue with quote consistency that's causing the pipeline to fail.

Could you apply this fix to match the codebase's quote style, wdyt?

-if sys.platform == 'emscripten':
+if sys.platform == "emscripten":
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if sys.platform == 'emscripten':
from serpyco import SchemaValidationError
else:
from serpyco_rs import SchemaValidationError
if sys.platform == "emscripten":
from serpyco import SchemaValidationError
else:
from serpyco_rs import SchemaValidationError
🤖 Prompt for AI Agents 
In airbyte_cdk/test/entrypoint_wrapper.py around lines 33 to 36, the import
statements use inconsistent quote styles. To fix the pipeline failure caused by
this, update the quotes in both import lines to match the project's preferred
quote style (either single or double quotes consistently). Ensure both 'serpyco'
and 'serpyco_rs' imports use the same type of quotes.


from airbyte_cdk.entrypoint import AirbyteEntrypoint
from airbyte_cdk.exception_handler import assemble_uncaught_exception
Expand Down
Loading
Loading