Commit 1ae9c3d
[SPARK-57458][SQL] Support nanosecond-precision timestamp types in the XML datasource
### What changes were proposed in this pull request?
This PR adds support for nanosecond-precision timestamp types (`TimestampLTZNanosType` and `TimestampNTZNanosType`) in the XML datasource, covering:
- Writing: `StaxXmlGenerator` handles the two nano timestamp types by calling the nanosecond-aware formatter methods (formatNanos, formatWithoutTimeZoneNanos).
- Reading: `StaxXmlParser` routes nano timestamp types to the corresponding `parseNanos` / `parseWithoutTimeZoneNanos` formatter methods in the schema-directed path, and delegates to castTo in the type-coercion path.
- Schema inference: `XmlInferSchema` now infers `TimestampNTZNanosType(9)` when a field value carries sub-microsecond fractional seconds (>6 digits) and `TIMESTAMP_NANOS_TYPES_ENABLED` is on.
The compatibleType widening function is extended to merge two nano timestamp types (taking the higher precision), downgrade `TimestampNTZNanosType` + `TimestampNTZType` to `TimestampNTZType`, and fall back to `TimestampType` for any other nano/non-nano combination. The `StructType` and `ArrayType` recursive cases are moved into the pre-TypeCoercion block so that nested fields also benefit from the nano-widening logic.
### Why are the changes needed?
`xml` rejected nanos timestamp types in its datasource capability checks and lacked the conversions to round-trip them, so these columns could not be written or read through `xml`.
### Does this PR introduce _any_ user-facing change?
Yes. With `spark.sql.timestampNanosTypes.enabled` set to `true`:
- XML files can now be written and read back with `TimestampLTZNanosType` and `TimestampNTZNanosType` columns without error.
- Schema inference promotes a timestamp field to `TimestampNTZNanosType(9)` when its string value contains more than 6 fractional-second digits .
### How was this patch tested?
`XmlSuite`: Added two new inference tests — one verifying that a 9-digit NTZ timestamp string is inferred as `TimestampNTZNanosType(9)`, and one verifying that a mix of micro-precision and nano-precision NTZ rows in the same file degrades to `TimestampNTZType`
Updated `FileBasedDataSourceSuite` and `XmlFunctionsSuite`
### Was this patch authored or co-authored using generative AI tooling?
Yes, Generated-by: Claude (Sonnet 4.6)
Closes #56854 from vinodkc/SPARK-57458.
Authored-by: Vinod KC <vinod.kc.in@gmail.com>
Signed-off-by: Max Gekk <max.gekk@gmail.com>
(cherry picked from commit fe6bc95)
Signed-off-by: Max Gekk <max.gekk@gmail.com>1 parent 4a70d92 commit 1ae9c3d
7 files changed
Lines changed: 245 additions & 39 deletions
File tree
- sql
- catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml
- core/src
- main/scala/org/apache/spark/sql/execution/datasources/xml
- test/scala/org/apache/spark/sql
- execution/datasources/xml
Lines changed: 5 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
35 | | - | |
| 35 | + | |
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
| |||
200 | 200 | | |
201 | 201 | | |
202 | 202 | | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
203 | 207 | | |
204 | 208 | | |
205 | 209 | | |
| |||
Lines changed: 5 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
601 | 601 | | |
602 | 602 | | |
603 | 603 | | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
604 | 607 | | |
605 | 608 | | |
606 | 609 | | |
| |||
652 | 655 | | |
653 | 656 | | |
654 | 657 | | |
| 658 | + | |
| 659 | + | |
655 | 660 | | |
656 | 661 | | |
657 | 662 | | |
| |||
Lines changed: 47 additions & 20 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
542 | 542 | | |
543 | 543 | | |
544 | 544 | | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
545 | 554 | | |
546 | 555 | | |
547 | 556 | | |
| |||
641 | 650 | | |
642 | 651 | | |
643 | 652 | | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
644 | 690 | | |
645 | | - | |
| 691 | + | |
646 | 692 | | |
647 | 693 | | |
648 | 694 | | |
| |||
661 | 707 | | |
662 | 708 | | |
663 | 709 | | |
664 | | - | |
665 | | - | |
666 | | - | |
667 | | - | |
668 | | - | |
669 | | - | |
670 | | - | |
671 | | - | |
672 | | - | |
673 | | - | |
674 | | - | |
675 | | - | |
676 | | - | |
677 | | - | |
678 | | - | |
679 | | - | |
680 | | - | |
681 | | - | |
682 | | - | |
683 | 710 | | |
684 | 711 | | |
685 | 712 | | |
| |||
Lines changed: 0 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
149 | 149 | | |
150 | 150 | | |
151 | 151 | | |
152 | | - | |
153 | | - | |
154 | | - | |
155 | 152 | | |
156 | 153 | | |
157 | 154 | | |
| |||
Lines changed: 84 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1341 | 1341 | | |
1342 | 1342 | | |
1343 | 1343 | | |
1344 | | - | |
| 1344 | + | |
1345 | 1345 | | |
1346 | 1346 | | |
1347 | 1347 | | |
| |||
1635 | 1635 | | |
1636 | 1636 | | |
1637 | 1637 | | |
| 1638 | + | |
| 1639 | + | |
| 1640 | + | |
| 1641 | + | |
| 1642 | + | |
| 1643 | + | |
| 1644 | + | |
| 1645 | + | |
| 1646 | + | |
| 1647 | + | |
| 1648 | + | |
| 1649 | + | |
| 1650 | + | |
| 1651 | + | |
| 1652 | + | |
| 1653 | + | |
| 1654 | + | |
| 1655 | + | |
| 1656 | + | |
| 1657 | + | |
| 1658 | + | |
| 1659 | + | |
| 1660 | + | |
| 1661 | + | |
| 1662 | + | |
| 1663 | + | |
| 1664 | + | |
| 1665 | + | |
| 1666 | + | |
| 1667 | + | |
| 1668 | + | |
| 1669 | + | |
| 1670 | + | |
| 1671 | + | |
| 1672 | + | |
| 1673 | + | |
| 1674 | + | |
| 1675 | + | |
| 1676 | + | |
| 1677 | + | |
| 1678 | + | |
| 1679 | + | |
| 1680 | + | |
| 1681 | + | |
| 1682 | + | |
| 1683 | + | |
| 1684 | + | |
| 1685 | + | |
| 1686 | + | |
| 1687 | + | |
| 1688 | + | |
| 1689 | + | |
| 1690 | + | |
| 1691 | + | |
| 1692 | + | |
| 1693 | + | |
| 1694 | + | |
| 1695 | + | |
| 1696 | + | |
| 1697 | + | |
| 1698 | + | |
| 1699 | + | |
| 1700 | + | |
| 1701 | + | |
| 1702 | + | |
| 1703 | + | |
| 1704 | + | |
| 1705 | + | |
| 1706 | + | |
| 1707 | + | |
| 1708 | + | |
| 1709 | + | |
| 1710 | + | |
| 1711 | + | |
| 1712 | + | |
| 1713 | + | |
| 1714 | + | |
| 1715 | + | |
| 1716 | + | |
| 1717 | + | |
| 1718 | + | |
| 1719 | + | |
| 1720 | + | |
1638 | 1721 | | |
1639 | 1722 | | |
1640 | 1723 | | |
| |||
Lines changed: 36 additions & 14 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| 27 | + | |
26 | 28 | | |
27 | 29 | | |
28 | 30 | | |
| |||
45 | 47 | | |
46 | 48 | | |
47 | 49 | | |
| 50 | + | |
48 | 51 | | |
49 | | - | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
50 | 55 | | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
51 | 60 | | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
57 | 68 | | |
58 | 69 | | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
67 | 89 | | |
68 | 90 | | |
69 | 91 | | |
| |||
Lines changed: 68 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3694 | 3694 | | |
3695 | 3695 | | |
3696 | 3696 | | |
| 3697 | + | |
| 3698 | + | |
| 3699 | + | |
| 3700 | + | |
| 3701 | + | |
| 3702 | + | |
| 3703 | + | |
| 3704 | + | |
| 3705 | + | |
| 3706 | + | |
| 3707 | + | |
| 3708 | + | |
| 3709 | + | |
| 3710 | + | |
| 3711 | + | |
| 3712 | + | |
| 3713 | + | |
| 3714 | + | |
| 3715 | + | |
| 3716 | + | |
| 3717 | + | |
| 3718 | + | |
| 3719 | + | |
| 3720 | + | |
| 3721 | + | |
| 3722 | + | |
| 3723 | + | |
| 3724 | + | |
| 3725 | + | |
| 3726 | + | |
| 3727 | + | |
| 3728 | + | |
| 3729 | + | |
| 3730 | + | |
| 3731 | + | |
| 3732 | + | |
| 3733 | + | |
| 3734 | + | |
| 3735 | + | |
| 3736 | + | |
| 3737 | + | |
| 3738 | + | |
| 3739 | + | |
| 3740 | + | |
| 3741 | + | |
| 3742 | + | |
| 3743 | + | |
| 3744 | + | |
| 3745 | + | |
| 3746 | + | |
| 3747 | + | |
| 3748 | + | |
| 3749 | + | |
| 3750 | + | |
| 3751 | + | |
| 3752 | + | |
| 3753 | + | |
| 3754 | + | |
| 3755 | + | |
| 3756 | + | |
| 3757 | + | |
| 3758 | + | |
| 3759 | + | |
| 3760 | + | |
| 3761 | + | |
| 3762 | + | |
| 3763 | + | |
| 3764 | + | |
3697 | 3765 | | |
3698 | 3766 | | |
3699 | 3767 | | |
| |||
0 commit comments