Skip to content

bug: structured output via act() silently returns JSON string, not Pydantic instance #1273

@planetf1

Description

@planetf1

Context: Writing a first real Mellea application — a script that classifies GitHub activity items using Granite 4.1 3B via Ollama, with structured Pydantic output. The docs pointed at format= as the right approach. The type system gave no way to express the correct type, so I suppressed its complaint with cast(ActivityRelevance, result.value). Pyright was happy.

The script ran. It produced output. Every item was classified as "medium" with reason "classification unavailable" — the fallback. The bug was only discovered during code review, not at runtime, not with any test. The app had been silently using the fallback path on every single call.

Root cause: when format= is passed to session.act(), result.value is a JSON string, not a parsed Pydantic instance. The correct unwrapping is MyModel.model_validate_json(str(result)), which is documented in the instruct(format=) how-to but not on the act() path.

Reproduction:

from pydantic import BaseModel
from typing import cast
from mellea import start_session
from mellea.stdlib.components import Instruction

class Result(BaseModel):
    label: str

with start_session("ollama") as m:
    result = m.act(Instruction("Say yes or no"), format=Result)
    classification = cast(Result, result.value)  # Pyright accepts this
    print(classification.label)                  # AttributeError: 'str' object has no attribute 'label'

The AttributeError is typically swallowed by a broad except Exception handler, so the app falls back silently on every call rather than raising.

Why not @generative? @generative requires a stable function signature; it's harder to apply to dynamically-built prompts in a loop, which is the natural pattern for batch classification scripts.

Proposed fix (compatibility-safe):

Add .parsed as a new property on ComputedModelOutputThunk that runs model_validate_json when a format= was set. .value keeps its current string contract — existing code is untouched.

result = m.act(Instruction("Say yes or no"), format=Result)
r = result.parsed   # -> Result instance, statically typed, no cast needed
# result.value still returns the JSON string for existing callers

Implementation note: the backend currently stores _format in generate_log.extra as a logging artefact, not on the thunk itself. For .parsed to work, _format would need to be stored on the ComputedModelOutputThunk during setup so the property can call self._format.model_validate_json(self.value).

Short-term (hours): add an explicit docstring note to act() and ModelOutputThunk.value stating that .value is always a string when format= is set, and pointing at model_validate_json. This eliminates the silent-failure risk immediately without any code change.

Related: companion type-overload issue (to be filed) shares the root cause — fixing both together removes cast entirely and turns this into a compile-time error.

Metadata

Metadata

Assignees

Labels

area/stdlibCore abstractions: Context, MOT, SamplingStrategy, formatters, serializationbugSomething isn't workingneeds-designProblem is clear; implementation approach needs design discussion before work starts

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions