Hive: DECIMAL(p,s) nested in ARRAY<STRUCT<...>> / MAP<...> fails column ingestion — table lands with zero columns while the workflow reports 100% success

[FreakingMind]

Connector

Hive

Feature area

Metadata ingestion

Describe the bug

A Hive column whose type nests a DECIMAL(p,s) inside a complex type — ARRAY<STRUCT<... DECIMAL(p,s) ...>> or MAP<..., DECIMAL(p,s)> — makes get_columns() raise:

ValueError: invalid literal for int() with base 10: '16,4'

The exception is caught in sql_column_handler.py:297 and only logged as a WARNING (the traceback is visible only at loggerLevel: DEBUG). As a result:

• the ingestion workflow finishes with Workflow Success %: 100.0, Errors: 0, Warnings: 0,
• but the table is created in OpenMetadata with an empty column list.

The failure is therefore silent: the pipeline reports success, and nothing in its status indicates that all column metadata for that table was dropped. We only noticed because a table showed up in the UI with no columns.

A top-level STRUCT<... DECIMAL(p,s) ...> works fine — this is the case that was fixed in #13429 by adding an attype.startswith("struct") branch to hive/utils.py. That fix, however, only covers struct: when the same DECIMAL sits inside an ARRAY or a MAP, the outer type is no longer struct and the same ValueError comes back. So this looks like an incomplete fix of #13429 rather than a new defect.

Where it happens — ingestion/src/metadata/ingestion/source/database/hive/utils.py, get_columns():

col_raw_type = col_type
attype = re.sub(r"\(.*\)", "", col_type)          # greedy: eats everything between the FIRST "(" and the LAST ")"
col_type = re.search(r"^\w+", col_type).group(0)  # outer type name -> "array"
...
charlen = re.search(r"\(([\d,]+)\)", col_raw_type.lower())   # matches the NESTED decimal's params -> "16,4"
if charlen:
    charlen = charlen.group(1)
    if attype == "decimal":
        prec, scale = charlen.split(",")
        args = (int(prec), int(scale))
    elif attype.startswith("struct"):             # only STRUCT is special-cased (added in #13429)
        args = []
    else:
        args = (int(charlen),)                    # int("16,4") -> ValueError
    coltype = coltype(*args)

Two things combine:

1. charlen is searched over the whole raw type string, so for a complex type it picks up the parameters of a nested type (decimal(16,4) → "16,4").
2. Whether the type is complex is decided from attype, which the greedy re.sub has already mangled: for array<struct<fee_a:decimal(16,4),...>> it becomes "array<struct<fee_a:decimal,...", so attype.startswith("struct") is False and execution falls through to int(charlen).

To Reproduce

1. Create the tables below in Hive (the last three are controls that ingest fine):

CREATE DATABASE IF NOT EXISTS test_nested_types;

-- FAILS: DECIMAL(p,s) nested inside ARRAY<STRUCT<...>>
CREATE EXTERNAL TABLE test_nested_types.array_struct_decimal(
    id STRING,
    items ARRAY<STRUCT<fee_a: DECIMAL(16,4), fee_b: DECIMAL(16,4), amount: BIGINT, item_name: STRING>>
)
PARTITIONED BY (dt STRING)
STORED AS PARQUET;

-- FAILS: DECIMAL(p,s) nested inside MAP<...>
CREATE EXTERNAL TABLE test_nested_types.map_decimal(
    id STRING,
    fees MAP<STRING, DECIMAL(10,2)>
) STORED AS PARQUET;

-- OK (control): top-level STRUCT with a DECIMAL (the #13429 case)
CREATE EXTERNAL TABLE test_nested_types.struct_decimal(
    id STRING,
    fees STRUCT<fee_a: DECIMAL(16,4), amount: BIGINT>
) STORED AS PARQUET;

-- OK (control): complex type without a nested DECIMAL
CREATE EXTERNAL TABLE test_nested_types.array_struct_bigint(
    id STRING,
    items ARRAY<STRUCT<amount: BIGINT, item_name: STRING>>
) STORED AS PARQUET;

-- OK (control): top-level DECIMAL
CREATE EXTERNAL TABLE test_nested_types.plain_decimal(
    id STRING,
    fee DECIMAL(16,4),
    name VARCHAR(255)
) STORED AS PARQUET;

The type string the metastore returns for the failing column is:

array<struct<fee_a:decimal(16,4),fee_b:decimal(16,4),amount:bigint,item_name:string>>

2. Run a DatabaseMetadata ingestion against that schema (config below).
3. Observe that the workflow reports 100% success, and then check the ingested columns:

curl -s -H "Authorization: Bearer $TOKEN" \
  "$OM/api/v1/tables/name/hive_test.default.test_nested_types.array_struct_decimal?fields=columns" \
  | jq '.columns | length'   # -> 0

Result per table:

Table	Column type	Columns ingested
array_struct_decimal	array<struct<...decimal(16,4)...>>	0 ❌
map_decimal	map<string,decimal(10,2)>	0 ❌
struct_decimal	struct<...decimal(16,4)...>	2 ✅
array_struct_bigint	array<struct<...bigint...>>	2 ✅
plain_decimal	decimal(16,4)	3 ✅

Expected behavior

array_struct_decimal and map_decimal should ingest with their full column list, with the nested STRUCT fields as children — the same way struct_decimal and array_struct_bigint already do.

Separately: even if column reflection does fail, the pipeline should not report Errors: 0 / Success 100% while silently dropping every column of a table. A table ingested with zero columns is indistinguishable from a healthy run in the pipeline status.

Connection / ingestion config

source:
  type: hive
  serviceName: hive_test
  serviceConnection:
    config:
      type: Hive
      username: <redacted>
      hostPort: <redacted>:10000
      auth: NONE
  sourceConfig:
    config:
      type: DatabaseMetadata
      includeTables: true
      schemaFilterPattern:
        includes:
          - test_nested_types
sink:
  type: metadata-rest
  config: {}
workflowConfig:
  loggerLevel: DEBUG
  openMetadataServerConfig:
    hostPort: <redacted>/api
    authProvider: openmetadata
    securityConfig:
      jwtToken: <redacted>

Logs

[2026-07-14 21:13:13] DEBUG    {metadata.Ingestion:sql_column_handler:296} - Traceback (most recent call last):
  File ".../site-packages/metadata/ingestion/source/database/sql_column_handler.py", line 292, in get_columns_and_constraints
    columns = self._get_columns_internal(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/metadata/ingestion/source/database/sql_column_handler.py", line 238, in _get_columns_internal
    return inspector.get_columns(
           ^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/sqlalchemy/engine/reflection.py", line 497, in get_columns
    col_defs = self.dialect.get_columns(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/metadata/ingestion/source/database/hive/utils.py", line 73, in get_columns
    args = (int(charlen),)
            ^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: '16,4'

[2026-07-14 21:13:13] WARNING  {metadata.Ingestion:sql_column_handler:297} - Unexpected exception getting columns for table [array_struct_decimal] (schema: 'test_nested_types', db: 'default'): invalid literal for int() with base 10: '16,4'

# ... and the very same run finishes with:
Processed records: 10
Updated records: 0
Warnings: 0
Errors: 0
Success %: 100.0
Workflow Success %: 100.0

The identical trace is raised for map_decimal with '10,2'.

OS

macOS 14.4 (Docker); also seen on Linux

Python version

3.12

OpenMetadata version

1.12.5

OpenMetadata Ingestion package version

openmetadata-ingestion==1.12.12

Additional context

• Reproduced against Hive 4.0.1 (HiveServer2 + standalone Metastore on PostgreSQL), connector talking to HiveServer2 on port 10000.
• The code in question is unchanged in main and in the 1.13 branch, so this should reproduce there as well.
• Related: Hive ingestion fails on complex struct types #13429 fixed exactly this crash for a top-level STRUCT. The array / map cases were not covered, and they still hit the same line.
• Minor, possibly worth a separate issue: ColumnTypeParser._parse_primitive_datatype_string() does not set precision / scale for nested decimals — it puts the precision into dataLength instead (dataLength=16, precision=None, scale=None for a decimal(16,4) struct field). Top-level decimal columns are handled correctly by check_col_precision().

I have a working fix and a self-contained Docker repro; happy to open a PR if that helps.

Pre-submission checklist

• I searched for duplicate issues.
• I removed credentials, hostnames, emails, and other sensitive data from logs and config.