Error with Union of nested dataclasses: Does not validate against any of the Union subtypes ... Not a valid subclass of

[ovidiu-eremia]

🐛 Bug report

I am trying to move from simple-parsing to jsonargparse and I have this error for a config with dataclasses that have Union of other nested dataclasses as a type:

Does not validate against any of the Union subtypes 
Not a valid subclass of

To reproduce

2. Manually constructing a parser

import sys
from dataclasses import dataclass, field
from datetime import UTC, date, datetime
from enum import StrEnum, auto

from jsonargparse import ArgumentParser
from jsonargparse.typing import register_type


def date_serializer(value: date) -> str:
    return value.isoformat()


def date_deserializer(value: str) -> date:
    return datetime.strptime(value, "%Y-%m-%d").astimezone(UTC).date()


register_type(date, date_serializer, date_deserializer)


class Environment(StrEnum):
    dev = auto()
    tst = auto()
    val = auto()
    prd = ""
    dc_prd = "prd"


class Layer(StrEnum):
    landing = auto()
    bronze = auto()
    silver = auto()
    gold = auto()
    any = ""


class FileFormat(StrEnum):
    parquet = auto()
    delta = auto()
    csv = auto()
    json = auto()


@dataclass
class TablePath:
    catalog: str
    schema: str
    table: str

@dataclass
class File:
    name: str
    format: FileFormat
@dataclass
class FilePath:
    location: str
    folder: str
    file: File


@dataclass
class TableArgs:
    path: TablePath | FilePath
    env: Environment
    layer: Layer = Layer.any
    columns: list[str] = field(default_factory=list)
    pk_cols: list[str] = field(default_factory=list)


@dataclass
class ConfigArgs:
    main_catalog: str
    dates_dimension_start_date: date = field(
        default_factory=lambda: datetime(2010, 1, 1, tzinfo=UTC).date(),
    )


@dataclass
class MainArgs:
    config: ConfigArgs
    source: TableArgs
    target: TableArgs


def parse_args(cli_args: list[str]) -> MainArgs:
    parser = ArgumentParser()
    parser.add_class_arguments(
        MainArgs,
        as_group=False,
        as_positional=False,
    )
    print(cli_args)
    result = parser.parse_args(cli_args)
    print(parser.dump(result))

    ns = parser.instantiate_classes(result)
    return MainArgs(**ns.as_dict())


class TestJsonargparse:

    def test_table_source(self) -> None:
        """Parse table source (format not set)."""
        cli_args = [
            "--config.main_catalog",
            "spark_catalog",
            "--config.dates_dimension_start_date",
            "2010-01-01",
            "--source.path.catalog",
            "source_cat",
            "--source.path.schema",
            "source_sch",
            "--source.path.table",
            "source_tbl",
            "--source.env",
            "dev",
            "--target.path.catalog",
            "target_cat",
            "--target.path.schema",
            "target_sch",
            "--target.path.table",
            "target_tbl",
            "--target.env",
            "dev",
            "--target.columns",
            "['col1', 'col2']",
        ]

        result = parse_args(cli_args)

        assert result.source.path.catalog == "source_cat"
        assert result.source.path.schema == "source_sch"
        assert result.target.columns == ["col1", "col2"]

    def test_file_source(self) -> None:
        """Parse file source (format is set)."""
        cli_args = [
            "--config.main_catalog",
            "spark_catalog",
            "--source.path.location",
            "/data/file.csv",
            "--source.path.folder",
            "/data/file.csv",
            "--source.path.file.name",
            "file.csv",
            "--source.path.file.format",
            "csv",
            "--source.env",
            "dev",
            "--source.layer",
            "any",
            "--target.path.catalog",
            "target_cat",
            "--target.path.schema",
            "target_sch",
            "--target.path.table",
            "target_tbl",
            "--target.env",
            "dev",
        ]

        result = parse_args(cli_args)

        assert result.source.path.location == "/data/file.csv"
        assert result.source.path.folder == "/data/file.csv"
        assert result.source.layer == Layer.any


if __name__ == "__main__":
    parse_args(sys.argv)

Running help with FilePath gives me this:

python test_json_parse.py --source.path.help FilePath 
['test_json_parse.py', '--source.path.help', 'FilePath']
usage: test_json_parse.py --source.path.location LOCATION --source.path.folder FOLDER --source.path.file.name NAME --source.path.file.format {parquet,delta,csv,json}

Help for --source.path.help=__main__.FilePath

FilePath(location: str, folder: str, file: __main__.File):
  --source.path.location LOCATION
                        (required, type: str)
  --source.path.folder FOLDER
                        (required, type: str)

File(name: str, format: __main__.FileFormat):
  --source.path.file.name NAME
                        (required, type: str)
  --source.path.file.format {parquet,delta,csv,json}
                        (required, type: FileFormat)

Expected behavior

The first test that uses TablePath passes. But the second test that uses FilePath fails with this error:

ex         = TypeError('Parser key "source.path":\n  Does not validate against any of the Union subtypes\n  Subtypes: [<class \'components.qm.task_args.test_json_parse.TablePath\'>, <class \'components.qm.task_args.test_json_parse.FilePath\'>]\n  Errors:\n    - Not a valid subclass of TablePath\n      Subclass types expect one of:\n      - a class path (str)\n      - a dict with class_path entry\n      - a dict without class_path but with init_args entry (class path given previously)\n      - a dict with parameters accepted by the base class (implicit class_path)\n    - Not a valid subclass of FilePath\n      Subclass types expect one of:\n      - a class path (str)\n      - a dict with class_path entry\n      - a dict without class_path but with init_args entry (class path given previously)\n      - a dict with parameters accepted by the base class (implicit class_path)\n  Given value type: <class \'jsonargparse._namespace.Namespace\'>\n  Given value: Namespace(location=\'/data/file.csv\', folder=\'/data/file.csv\', file=Namespace(name=None, format=None), init_args=Namespace(location=\'/data/file.csv\', folder=\'/data/file.csv\', file=Namespace(name=\'file.csv\', format=<FileFormat.csv: \'csv\'>)), class_path=\'components.qm.task_args.test_json_parse.FilePath\')')
help_action = _HelpAction(option_strings=['-h', '--help'], dest='help', nargs=0, const=None, default='==SUPPRESS==', type=None, choices=None, required=False, help='show this help message and exit', metavar=None)
message    = ('Parser key "source.path":\n'
 '  Does not validate against any of the Union subtypes\n'
 "  Subtypes: [<class 'components.qm.task_args.test_json_parse.TablePath'>, "
 "<class 'components.qm.task_args.test_json_parse.FilePath'>]\n"
 '  Errors:\n'
 '    - Not a valid subclass of TablePath\n'
 '      Subclass types expect one of:\n'
 '      - a class path (str)\n'
 '      - a dict with class_path entry\n'
 '      - a dict without class_path but with init_args entry (class path given '
 'previously)\n'
 '      - a dict with parameters accepted by the base class (implicit '
 'class_path)\n'
 '    - Not a valid subclass of FilePath\n'
 '      Subclass types expect one of:\n'
 '      - a class path (str)\n'
 '      - a dict with class_path entry\n'
 '      - a dict without class_path but with init_args entry (class path given '
 'previously)\n'
 '      - a dict with parameters accepted by the base class (implicit '
 'class_path)\n'
 "  Given value type: <class 'jsonargparse._namespace.Namespace'>\n"
 "  Given value: Namespace(location='/data/file.csv', folder='/data/file.csv', "
 'file=Namespace(name=None, format=None), '
 "init_args=Namespace(location='/data/file.csv', folder='/data/file.csv', "
 "file=Namespace(name='file.csv', format=<FileFormat.csv: 'csv'>)), "
 "class_path='components.qm.task_args.test_json_parse.FilePath')")
parser     = ArgumentParser(prog='run_pytest_script.py', usage=None, description=None, formatter_class=<class 'jsonargparse._formatters.DefaultHelpFormatter'>, conflict_handler='error', add_help=True)
prog       = 'run_pytest_script.py'
self       = ArgumentParser(prog='run_pytest_script.py', usage=None, description=None, formatter_class=<class 'jsonargparse._formatters.DefaultHelpFormatter'>, conflict_handler='error', add_help=True)

../.venv/lib/python3.12/site-packages/jsonargparse/_core.py:1069: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = ArgumentParser(prog='run_pytest_script.py', usage=None, description=None, formatter_class=<class 'jsonargparse._formatters.DefaultHelpFormatter'>, conflict_handler='error', add_help=True)
status = 2, message = None

    def exit(self, status=0, message=None):
        if message:
            self._print_message(message, _sys.stderr)
>       _sys.exit(status)
E       SystemExit: 2

message    = None
self       = ArgumentParser(prog='run_pytest_script.py', usage=None, description=None, formatter_class=<class 'jsonargparse._formatters.DefaultHelpFormatter'>, conflict_handler='error', add_help=True)
status     = 2

../../../.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/argparse.py:2637: SystemExit
----------------------------- Captured stdout call -----------------------------
['--config.main_catalog', 'spark_catalog', '--source.path.location', '/data/file.csv', '--source.path.folder', '/data/file.csv', '--source.path.file.name', 'file.csv', '--source.path.file.format', 'csv', '--source.env', 'dev', '--source.layer', 'any', '--target.path.catalog', 'target_cat', '--target.path.schema', 'target_sch', '--target.path.table', 'target_tbl', '--target.env', 'dev']
----------------------------- Captured stderr call -----------------------------
usage: run_pytest_script.py [--config CONFIG]
                            --config.main_catalog MAIN_CATALOG
                            [--config.dates_dimension_start_date DATES_DIMENSION_START_DATE]
                            [--source CONFIG] --source.path PATH
                            --source.env {dev,tst,val,prd,dc_prd}
                            [--source.layer {landing,bronze,silver,gold,any}]
                            [--source.columns [ITEM,...]]
                            [--source.pk_cols [ITEM,...]] [--target CONFIG]
                            --target.path PATH
                            --target.env {dev,tst,val,prd,dc_prd}
                            [--target.layer {landing,bronze,silver,gold,any}]
                            [--target.columns [ITEM,...]]
                            [--target.pk_cols [ITEM,...]]
tip: For details of accepted options run: run_pytest_script.py --help
error: Parser key "source.path":
  Does not validate against any of the Union subtypes
  Subtypes: [<class 'components.qm.task_args.test_json_parse.TablePath'>, <class 'components.qm.task_args.test_json_parse.FilePath'>]
  Errors:
    - Not a valid subclass of TablePath
      Subclass types expect one of:
      - a class path (str)
      - a dict with class_path entry
      - a dict without class_path but with init_args entry (class path given previously)
      - a dict with parameters accepted by the base class (implicit class_path)
    - Not a valid subclass of FilePath
      Subclass types expect one of:
      - a class path (str)
      - a dict with class_path entry
      - a dict without class_path but with init_args entry (class path given previously)
      - a dict with parameters accepted by the base class (implicit class_path)
  Given value type: <class 'jsonargparse._namespace.Namespace'>
  Given value: Namespace(location='/data/file.csv', folder='/data/file.csv', file=Namespace(name=None, format=None), init_args=Namespace(location='/data/file.csv', folder='/data/file.csv', file=Namespace(name='file.csv', format=<FileFormat.csv: 'csv'>)), class_path='components.qm.task_args.test_json_parse.FilePath')

Environment

• jsonargparse version: 4.45.0
• Python version: 3.12
• How jsonargparse was installed: through uv
• OS: Ubuntu 24.04

[mauvilsa]

Thank you for reporting! I will take a look.

[mauvilsa]

This should have been fixed with #839.

You could also have a look at #829. Unions of dataclasses are tricky. Another option is to configure the dataclasses part of the union so that they support subclasses. This feature hasn't been released yet, but once it does, for this case it would be:

set_parsing_settings(subclasses_enabled=[TablePath, FilePath])