Add type-aware custom object serialization

[andystaples]

Summary

Reworks user-payload JSON serialization to be secure and type-aware while
keeping the on-the-wire format stable. Custom objects (dataclasses and
from_json()-capable types) now round-trip everywhere in the programming model,
and deserialization is driven by caller-supplied or annotation-discovered types
rather than by the payload itself.

Motivation

The previous codec (shared.to_json / from_json) had real problems:

• Nested dataclasses didn't round-trip — only the top-level object got the
  AUTO_SERIALIZED marker, so nested dataclasses/namedtuples decoded
  inconsistently, and everything came back as a SimpleNamespace rather than the
  original type.
• Payload-driven type selection is unsafe — extending the marker approach to
  reconstruct arbitrary classes named in the payload is classic insecure
  deserialization (OWASP A08). The secure model is type-directed decoding,
  where the destination type comes from the caller/annotations, never the wire.

This also aligns the SDK with the direction of azure-functions-durable and the
.NET DataConverter (plain JSON + caller-supplied target type).

What changed

Type-directed deserialization

• call_activity, call_sub_orchestrator, and call_entity gain an optional
  return_type; wait_for_external_event gains data_type. When provided, the
  result/event is coerced to that type (dataclasses incl. nested / Optional /
  list fields, and from_json()-capable types). These also refine the static
  type of the returned task via @overload
  (e.g. call_activity(..., return_type=Foo) -> CompletableTask[Foo]).
• New typed accessors on client.OrchestrationState: get_input(),
  get_output(), get_custom_status() — each takes an optional expected_type
  and is overloaded to return T | None. Raw serialized_* fields are retained.
• Entity get_state(intended_type=...) now routes through the shared codec
  (dataclass + from_json support).

Annotation-based discovery (new durabletask/internal/type_discovery.py)

• Inbound payloads (orchestrator / activity / entity-operation inputs) and
  call_activity results are reconstructed from the function's type annotations
  when no explicit type is supplied. Discovery is best-effort and
  conservative: builtins and unannotated/unknown types pass through unchanged,
  and a payload that fails to coerce falls back to the raw value.

Codec / behavior

• to_json now emits plain JSON (no internal type marker). Objects exposing
  to_json() are serializable.
• Serialization failures raise a TypeError that chains the original cause and
  names the offending type.
• Back-compat: payloads produced by older SDK versions (carrying the legacy
  AUTO_SERIALIZED marker) still deserialize — into a SimpleNamespace when no
  type is supplied, or stripped and coerced when an expected_type is given — so
  in-flight orchestrations replay cleanly across the upgrade.

Behavior change to note

Decoding without a type now yields a plain dict/list (previously a
SimpleNamespace for marked payloads). Callers that want the original type
should pass return_type / data_type / expected_type, or rely on annotation
discovery. Documented under ## Unreleased in CHANGELOG.md.

Tests & validation

• New unit suites: test_serialization.py, test_type_discovery.py,
  test_orchestration_state.py, plus additions to the entity/orchestration
  executor tests (codec round-trips, nested dataclasses, expected_type
  precedence, legacy-marker back-compat, error chaining, annotation discovery,
  static-type inference).
• Full local gate: strict pyright (0 errors), flake8 (source + tests),
  pytest (non-e2e), and pymarkdown all green. durabletask-azuremanaged
  unit tests pass (it reuses the core client/worker).

Note

Opening as a draft for review. Static-type inference is refined on the task
object; the result = yield ctx.call_activity(...) pattern still yields Any
due to Python generator send-type limitations (would require an await-style API).

[berndverst]

Expanding on the DataConverter-style abstraction idea from the review — and what it would look like with pydantic specifically.

The gap today

Serialization is currently hardcoded to json.dumps / json.loads with per-type, duck-typed to_json() / from_json() hooks. That works, but it bakes the policy into the SDK and gives users no single place to control encoding: custom datetime/Decimal formats, enum-by-name, set/bytes handling, or whole model frameworks (pydantic, attrs, msgspec) all have to bend to fit the hook convention. .NET solves this with a pluggable DataConverter that both the worker and client consume, defaulting to JsonDataConverter (System.Text.Json, IncludeFields = true). The destination type is always supplied by the caller (Deserialize(data, targetType) / Deserialize<T>(data)) — exactly the secure, type-directed model this PR is moving toward.

A Python-equivalent shape

# durabletask/serialization.py (new public API)
from abc import ABC, abstractmethod
from typing import Any


class DataConverter(ABC):
    @abstractmethod
    def serialize(self, value: Any) -> str | None:
        ...

    @abstractmethod
    def deserialize(self, data: str | None, target_type: type | None = None) -> Any:
        ...

The default preserves today's behavior, so it's a non-breaking refactor:

class JsonDataConverter(DataConverter):
    def serialize(self, value):
        return None if value is None else shared.to_json(value)

    def deserialize(self, data, target_type=None):
        return None if data is None else shared.from_json(data, target_type)

Then wire it through the worker/client once (a constructor arg defaulting to JsonDataConverter()), and route every existing shared.to_json(...) / shared.from_json(..., expected_type) call site through self._converter. All the type-directed plumbing this PR already adds — return_type=, data_type=, and annotation discovery — keeps working unchanged; it just supplies target_type to the converter, which decides how the bytes become that type.

Pydantic example

A team using pydantic models could drop in a converter that gives full validation on the way in and pydantic's JSON encoding on the way out, while still falling back to stdlib JSON for plain dicts/dataclasses:

import pydantic

from durabletask.internal import shared
from durabletask.serialization import DataConverter


class PydanticDataConverter(DataConverter):
    """Round-trips pydantic models; falls back to stdlib JSON for everything else."""

    def serialize(self, value):
        if value is None:
            return None
        if isinstance(value, pydantic.BaseModel):
            return value.model_dump_json()            # pydantic v2
        return shared.to_json(value)                  # dataclasses, dicts, ...

    def deserialize(self, data, target_type=None):
        if data is None:
            return None
        if (
            isinstance(target_type, type)
            and issubclass(target_type, pydantic.BaseModel)
        ):
            return target_type.model_validate_json(data)   # pydantic v2, validates
        return shared.from_json(data, target_type)

Register it once on each end:

converter = PydanticDataConverter()
worker = TaskHubGrpcWorker(data_converter=converter)
client = TaskHubGrpcClient(data_converter=converter)

…and pydantic models round-trip everywhere through the same type-directed flow already in this PR:

class Order(pydantic.BaseModel):
    customer: str
    total: float


class Receipt(pydantic.BaseModel):
    confirmation_id: str


def orchestrator(ctx, order: Order):                       # discovery -> Order (validated)
    receipt = yield ctx.call_activity(charge, input=order, return_type=Receipt)
    return receipt.confirmation_id

Here charge's result is validated against Receipt (ValidationError on bad data, instead of a silent partial dataclass), and order is validated when the orchestrator/activity receives it — using pydantic's own coercion rules rather than the SDK's coerce_to_type.

Why this is worth considering over the duck-typed hooks

• One documented extension point instead of a method-name convention. As noted in the line comments, to_json() / from_json() are ambiguous: the encode hook must return a structure (not a JSON string, despite the name), and the decode hook receives a parsed dict (not a JSON string, which is what many libraries' from_json expects). A DataConverter makes the contract explicit and avoids the collision.
• No SDK dependency on pydantic/attrs/msgspec — users opt in by supplying a converter.
• Validation and custom encoding become first-class (dates, decimals, enums, unions) rather than something each type has to re-implement in a hook.
• Polyglot parity with the .NET mental model, which helps teams running both stacks.

This is complementary to the PR, not a blocker: the return_type / discovery / coerce_to_type work here is precisely what a converter would consume. It could land as a fast-follow that swaps the hardcoded shared.to_json / shared.from_json calls for a converter indirection, with JsonDataConverter as the default so nothing changes for existing users.

Note

One open question for the design: whether the converter is global (one per worker/client) like .NET, or can also be overridden per call (e.g. call_activity(..., data_converter=...)). .NET is global; per-call would add flexibility but more surface area.

[berndverst]

One follow-up on the breaking-change cleanup: examples/in_memory_backend_example was missed and is broken by this PR (the human_interaction.py fix is the model to follow here).

That example predates this PR and is unchanged by it, but it still relies on the old "top-level object → SimpleNamespace" behavior that this PR removes. Its orchestrators/activities take unannotated params and use top-level attribute access:

def process_order(ctx, order):          # no annotation
    yield ctx.call_activity(validate_order, input=order)
    total = yield ctx.call_activity(calculate_total, input=order.items)   # <-- here
    ...

def validate_order(ctx, order):         # no annotation
    if not order.items:                 # <-- here
        ...

After this PR, an unannotated payload decodes to a plain dict, so order.items resolves to the built-in dict.items method rather than the list — and for item in order.items raises a confusing TypeError: 'builtin_function_or_method' object is not iterable instead of a clean AttributeError. Reproduced with this PR's own json_codec:

>>> wire = json_codec.to_json(Order("Alice", [OrderItem("Widget", 2, 10.0)]))
>>> got = json_codec.from_json(wire)        # no target type -> dict
>>> type(got).__name__
'dict'
>>> got.items
<built-in method items of dict object at 0x...>
>>> for item in got.items: ...
TypeError: 'builtin_function_or_method' object is not iterable

A few things worth noting:

• The module + dataclass docstrings still describe the old model ("top-level objects become SimpleNamespace instances while nested objects become plain dicts"), so they're now actively misleading.
• The example's test/test_workflows.py drives the full worker → in-memory backend → worker path and would fail — but testpaths = ["tests"] means example tests don't run in CI, so this break is silent until someone runs the example.

The fix needs to be coherent rather than one line, because the example currently mixes attribute access at the top level with dict access on nested items (item["quantity"]):

• Annotate the params so they reconstruct as dataclasses: def process_order(ctx, order: Order), def validate_order(ctx, order: Order), def calculate_total(ctx, items: list[OrderItem]).
• Flip the nested access from item["quantity"] / item["unit_price"] / item["name"] to attribute access (item.quantity, …) now that nested items reconstruct as OrderItem.
• Update the module/dataclass docstrings (and the README if it repeats the claim) to describe the new type-directed model.

Not a blocker, but since this PR is explicitly cleaning up the breaking-change surface and already fixed human_interaction.py, this example deserves the same treatment so the shipped sample doesn't break on upgrade.

[Copilot]

Pull request overview

This PR modernizes Durable Task Python’s payload handling by introducing secure, type-directed JSON deserialization (caller/annotation supplied types, not payload-driven), while keeping the wire format as plain JSON and maintaining back-compat for legacy marker payloads.

Changes:

• Added a pluggable DataConverter abstraction (default JsonDataConverter) and routed all payload boundaries through it (worker + client + entities).
• Implemented type-directed reconstruction via explicit return_type / data_type parameters and best-effort annotation-based type discovery for inbound payloads.
• Replaced legacy marker-based JSON codec with a plain-JSON codec that still recognizes legacy payloads for replay compatibility, plus expanded tests/docs/examples/changelog.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
durabletask/internal/json_codec.py	New plain-JSON codec with type-directed coercion and legacy-marker back-compat.
durabletask/serialization.py	Introduces DataConverter + default JsonDataConverter used across SDK boundaries.
durabletask/internal/type_discovery.py	Adds conservative type-hint discovery for orchestrator/activity/entity inputs and activity outputs.
durabletask/internal/shared.py	Keeps older imports working via re-exports to the new JSON codec.
durabletask/worker.py	Routes orchestration/activity/entity execution through DataConverter; adds typed task completion + event payload typing.
durabletask/task.py	Updates public abstract context APIs with overloads and typed return_type / data_type parameters.
durabletask/client.py	Threads DataConverter into clients and adds typed OrchestrationState accessors.
durabletask/internal/client_helpers.py	Uses the converter for request payload serialization (start/terminate/event/entity signal).
durabletask/internal/entity_state_shim.py	Routes entity runtime state coercion through the converter while preserving strictness expectations.
durabletask/entities/entity_context.py	Uses converter for entity-side scheduling/signaling payloads.
durabletask/entities/entity_metadata.py	Adds get_typed_state() for typed state deserialization via the converter.
durabletask/extensions/history_export/client.py	Switches history-export entity state parsing to typed state access.
tests/durabletask/extensions/history_export/test_entity.py	Aligns tests with typed state access for history-export entities.
tests/durabletask/test_serialization.py	New unit tests for plain-JSON codec, hooks, coercion, and legacy-marker behavior.
tests/durabletask/test_type_discovery.py	New tests for annotation discovery + inbound coercion paths.
tests/durabletask/test_orchestration_state.py	New tests for typed OrchestrationState accessors.
tests/durabletask/test_orchestration_executor.py	Adds executor coverage for return_type and annotation-discovered activity results.
tests/durabletask/test_entity_executor.py	Adds coverage for StateShim coercion behavior via the converter.
tests/durabletask/test_data_converter.py	New tests for the DataConverter abstraction and default implementation behavior.
examples/in_memory_backend_example/src/workflows.py	Updates example to rely on annotated dataclass reconstruction instead of dict access.
examples/human_interaction.py	Updates example to use data_type for external-event payload reconstruction.
docs/supported-patterns.md	Documents using data_type / return_type and annotations to reconstruct original types.
CHANGELOG.md	Documents new converter + typed APIs and the behavioral shift toward plain dict/list when no type is supplied.