Skip to content

bug: Document.from_dict() fails to deserialize flattened documents when metadata contains a "meta" key #12211

Description

@aryanp160

🐛 Describe the bug

Document.from_dict() cannot correctly deserialize certain dictionaries produced by Document.to_dict() using the default flatten=True behavior.

When a document's metadata contains a key literally named "meta" alongside additional metadata keys, the serialized representation is valid, but deserialization raises a ValueError instead of reconstructing the original Document.

This breaks the expected round-trip invariant:

Document.from_dict(document.to_dict()) == document

❌ Error message

ValueError: You can pass either the 'meta' parameter or flattened metadata keys as keyword arguments, but currently you're passing both. Pass either the 'meta' parameter or flattened metadata keys.

✅ Expected behavior

Any document serialized using Document.to_dict() should successfully deserialize with Document.from_dict(), regardless of the names of metadata keys.

The following should always succeed:

restored = Document.from_dict(document.to_dict())

assert restored == document

🔁 Steps to reproduce

from haystack import Document

doc = Document(
    content="Hello",
    meta={
        "meta": {"nested": "value"},
        "author": "Alice",
    },
)

serialized = doc.to_dict()  # flatten=True (default)

Document.from_dict(serialized)

Current result

ValueError: You can pass either the 'meta' parameter or flattened metadata keys...

Expected result

restored = Document.from_dict(serialized)

assert restored == doc
assert restored.meta == {
    "meta": {"nested": "value"},
    "author": "Alice",
}

🔍 Root cause

For flattened documents, the serialized output becomes:

{
    "id": "...",
    "content": "Hello",
    "meta": {"nested": "value"},
    "author": "Alice",
}

Document.from_dict() interprets the top-level "meta" entry as the root metadata container instead of a flattened metadata key.

When the remaining flattened metadata ("author") is processed, the implementation believes both flattened and non-flattened metadata were supplied simultaneously and raises a ValueError.


📝 Related context

PR #11952 fixed the round-trip case where the metadata key "meta" contains a non-mapping value.

This issue concerns the remaining edge case where the "meta" metadata key itself contains a mapping and additional flattened metadata keys are present.


📋 Environment

Component Value
OS Windows 11
Haystack Version main
Commit 1bc569f66ef2495a6ce676dcb2037d3c189769be

☑️ Checklist

  • Reproduced on the latest main branch
  • Reviewed existing issues and related PRs
  • Checked the FAQ

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions