Surface date-only and time-only fields as SQL DATE / TIME on the analytics-engine route

[ahkcs]

Description

OpenSearch stores every date field as an epoch-millisecond timestamp regardless of its mapping format. On the analytics-engine route, OpenSearchSchemaBuilder.mapFieldType typed every date/date_nanos column as TIMESTAMP and discarded the format, so a date-only field (e.g. basic_date, yyyy-MM-dd) surfaced as 1984-04-12 00:00:00 and a time-only field (e.g. hour, HH:mm:ss) as 1970-01-01 09:00:00, instead of the SQL DATE (1984-04-12) / TIME (09:00:00) the v2 / Calcite path produces.

This closes both gaps (DATE + TIME).

Approach

• Add DateType and TimeType Calcite UDTs, each backed by SqlTypeName.TIMESTAMP and carrying the same precision a plain timestamp does — mirroring the existing IpType / BinaryType UDTs. Because the backing type and precision are identical to a plain timestamp, storage, Substrait serialization, and the DataFusion scan are unchanged: the column is still Timestamp(MILLISECOND) on disk and binds against the parquet column exactly as before. Only the planner-side logical type changes, so the SQL plugin can render it as DATE / TIME.
• Add DateFormatClassifier, which narrows a date/date_nanos field to DATE or TIME by its format (named formats, custom patterns like yyyy-MM-dd / HH:mm:ss, and ||-combined alternatives). It mirrors OpenSearchDateType.getExprTypeFromFormatString in the SQL plugin.
• OpenSearchSchemaBuilder.buildLeafType becomes format-aware and returns DateType / TimeType accordingly; datetime/epoch formats keep TIMESTAMP.

Companion SQL-plugin change (consumes the UDTs for the response schema label and renders the value as a LocalDate / LocalTime) is in opensearch-project/sql and depends on this change being published, exactly as the IpType/BinaryType consumers do.

Failure this fixes (cluster log, before)

A naive UDT without an explicit precision hit:

This feature is not implemented: Unsupported Substrait precision -1, for PrecisionTimestamp

The fix carries the plain-timestamp precision so the type serializes to an identical Substrait PrecisionTimestamp.

Testing

• New unit tests: DateFormatClassifierTests (named date/time/datetime formats, custom patterns, 'T'-literal stripping, combined alternatives, null/blank) — all pass.
• Verified end-to-end against a local 9-plugin analytics cluster with the companion SQL change, -Dtests.analytics.parquet_indices=true:

SQL-plugin IT (analytics route)	before	after
CalcitePPLAggregationIT.testCountByDateTypeSpanForDifferentFormats	❌	✅
CalcitePPLAggregationIT.testCountByDateTypeSpanWithDifferentUnits	❌	✅
CalcitePPLAggregationIT.testCountByTimeTypeSpanForDifferentFormats	❌	✅
CalcitePPLAggregationIT.testCountByTimeTypeSpanWithDifferentUnits	❌	✅
CalcitePPLAggregationIT.testCountByNullableTimeSpan	❌	✅
CalcitePPLAggregationIT.testCountBySpanForCustomFormats	❌	✅
CalciteAnalyticsDatetimeWireFormatIT (date + time cols)	❌	✅

No regressions: the change only alters columns whose mapping format is date-only or time-only; datetime/epoch columns are untouched. Other pre-existing analytics-route failures (percentile-approx, count-distinct-approx, nested fields, date_add) are date/time-independent.

• ./gradlew :sandbox:libs:analytics-api:{spotlessJavaCheck,forbiddenApisMain,licenseHeaders} pass.

Check List

• New functionality includes testing.
• Commits are signed per the DCO using --signoff.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 2a88264)

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

📝 TODO sections sandbox/libs/analytics-api/src/main/java/org/opensearch/analytics/schema/DateFormatClassifier.java [32]: extract shared classifier so analytics and v2 paths cannot drift

🔀 No multiple PR themes

⚡ Recommended focus areas for review Escaped Quote Handling The stripLiterals method removes single-quoted sections with replaceAll("'[^']*'", ""), but does not handle escaped quotes ('') within literals. In java.time patterns, two consecutive single quotes represent a literal single quote. For example, the pattern yyyy-MM-dd''HH:mm:ss (intending a literal quote between date and time) would be incorrectly stripped to yyyy-MM-ddHH:mm:ss, causing it to be classified as TIMESTAMP when it might be intended as date-only or time-only depending on the actual pattern structure. This could lead to incorrect type classification for patterns containing escaped quotes. private static String stripLiterals(String pattern) { return pattern.replaceAll("'[^']*'", ""); } Null Check After Use At line 340, buildLeafType(fieldType, format, typeFactory) is called and assigned to columnType. The null check for columnType occurs at line 341, but this variable was already declared earlier in the method (line 339 shows the assignment). If buildLeafType returns null for an unsupported type, the subsequent null check correctly handles it. However, the code structure suggests columnType might have been used before this point in the original code. Verify that no code path between the variable's first declaration and this assignment attempts to use columnType before it's reassigned, as that would cause a NullPointerException. RelDataType columnType = buildLeafType(fieldType, format, typeFactory); if (columnType == null) { // Unsupported (geo_point/shape/completion/…) or unknown plugin type. Drop the

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to 2a88264

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Handle escaped quotes in patterns The regex '[^']*' doesn't handle escaped single quotes ('') within literals correctly. In Java date patterns, two consecutive single quotes represent a literal single quote. This could cause incorrect classification when patterns contain escaped quotes. sandbox/libs/analytics-api/src/main/java/org/opensearch/analytics/schema/DateFormatClassifier.java [154-156] private static String stripLiterals(String pattern) { - return pattern.replaceAll("'[^']*'", ""); + return pattern.replaceAll("''|'[^']*'", ""); } Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies that the regex doesn't handle escaped single quotes ('') in Java date patterns. This is a valid edge case that could lead to incorrect classification when patterns contain escaped quotes, though it's a relatively uncommon scenario.	Medium
General	Clarify timestamp fallthrough logic When DateFormatClassifier.classify(format) returns SqlTypeName.TIMESTAMP, the code falls through to the generic mapFieldType logic. However, this doesn't explicitly handle the date_nanos type, which may require special precision handling compared to regular date fields. sandbox/libs/analytics-api/src/main/java/org/opensearch/analytics/schema/OpenSearchSchemaBuilder.java [266-275] if ("date".equals(opensearchType) || "date_nanos".equals(opensearchType)) { SqlTypeName logical = DateFormatClassifier.classify(format); - if (logical == SqlTypeName.DATE || logical == SqlTypeName.TIME) { - int precision = typeFactory.createSqlType(SqlTypeName.TIMESTAMP).getPrecision(); - return logical == SqlTypeName.DATE ? new DateType(true, precision) : new TimeType(true, precision); + int precision = typeFactory.createSqlType(SqlTypeName.TIMESTAMP).getPrecision(); + if (logical == SqlTypeName.DATE) { + return new DateType(true, precision); + } else if (logical == SqlTypeName.TIME) { + return new TimeType(true, precision); } + // Fall through to mapFieldType for TIMESTAMP classification } Suggestion importance[1-10]: 4 __ Why: The suggestion restructures the code for clarity but doesn't change the functional behavior. The precision variable is moved outside the conditional, which is a minor improvement in code organization. However, the claim about date_nanos requiring special precision handling is not substantiated, as both date and date_nanos fall through to mapFieldType for TIMESTAMP classification in the current implementation.	Low

---

Previous suggestions

Suggestions up to commit 5a51c07

Category	Suggestion	Impact
General	Handle escaped quotes in patterns The regex '[^']*' doesn't handle escaped single quotes ('') within literals correctly. In Java date patterns, two consecutive single quotes represent a literal single quote. This could cause incorrect classification when patterns contain escaped quotes. sandbox/libs/analytics-api/src/main/java/org/opensearch/analytics/schema/DateFormatClassifier.java [154-156] private static String stripLiterals(String pattern) { - return pattern.replaceAll("'[^']*'", ""); + return pattern.replaceAll("''|'[^']*'", ""); } Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies that the regex doesn't handle escaped single quotes ('') in Java date patterns. This is a valid edge case that could lead to incorrect classification when patterns contain escaped quotes, though it's a relatively uncommon scenario.	Medium
	Early exit when result determined The loop continues processing after allDate and allTime both become false, which means the result is already determined to be TIMESTAMP. Add an early exit when both flags are false to avoid unnecessary iterations and classifications. sandbox/libs/analytics-api/src/main/java/org/opensearch/analytics/schema/DateFormatClassifier.java [107-116] for (String piece : format.split(FORMAT_DELIMITER)) { String trimmed = piece.trim(); if (trimmed.isEmpty()) { continue; } sawComponent = true; Component c = classifyPiece(trimmed); allDate &= (c == Component.DATE); allTime &= (c == Component.TIME); + if (!allDate && !allTime) { + break; + } } Suggestion importance[1-10]: 5 __ Why: The suggestion is correct that adding an early exit when both allDate and allTime are false would avoid unnecessary iterations. This is a minor performance optimization that could be beneficial when processing formats with many alternatives, though the impact is likely small in typical use cases.	Low

Suggestions up to commit 50722ca

Category	Suggestion	Impact
Possible issue	Validate format parameter type safety The format parameter could be a non-String type from the mapping (e.g., a Map or List for complex formats). Passing it directly to DateFormatClassifier.classify() which expects a String could cause a ClassCastException. Verify the type before classification. sandbox/libs/analytics-api/src/main/java/org/opensearch/analytics/schema/OpenSearchSchemaBuilder.java [265-271] if (("date".equals(opensearchType) || "date_nanos".equals(opensearchType)) - && DateFormatClassifier.classify(format) == SqlTypeName.DATE) { + && format instanceof String + && DateFormatClassifier.classify((String) format) == SqlTypeName.DATE) { int precision = typeFactory.createSqlType(SqlTypeName.TIMESTAMP).getPrecision(); return new DateType(true, precision); } Suggestion importance[1-10]: 8 __ Why: The suggestion identifies a potential ClassCastException risk. At line 314, format is extracted as (String) fieldProps.get("format") without type checking, and if it's not a String, passing it to DateFormatClassifier.classify() could fail. Adding an instanceof check prevents runtime errors.	Medium
	Handle escaped quotes in patterns The regex '[^']*' doesn't handle escaped single quotes ('') within literals correctly. In java.time patterns, two consecutive single quotes represent a literal single quote. This could cause incorrect classification when patterns contain escaped quotes. sandbox/libs/analytics-api/src/main/java/org/opensearch/analytics/schema/DateFormatClassifier.java [154-156] private static String stripLiterals(String pattern) { - return pattern.replaceAll("'[^']*'", ""); + return pattern.replaceAll("''|'[^']*'", ""); } Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies that the regex doesn't handle escaped single quotes ('') in java.time patterns. This is a valid edge case that could lead to incorrect classification when patterns contain escaped quotes, though it's a relatively uncommon scenario.	Medium

[github-actions]

❌ Gradle check result for 50722ca: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

Persistent review updated to latest commit 5a51c07

[github-actions]

❌ Gradle check result for 5a51c07: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

Persistent review updated to latest commit 2a88264

[github-actions]

✅ Gradle check result for 2a88264: SUCCESS

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.36%. Comparing base (ce2e409) to head (2a88264).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #22062      +/-   ##
============================================
- Coverage     73.44%   73.36%   -0.08%     
+ Complexity    75617    75545      -72     
============================================
  Files          6038     6038              
  Lines        342958   342958              
  Branches      49340    49340              
============================================
- Hits         251880   251623     -257     
- Misses        70997    71283     +286     
+ Partials      20081    20052      -29     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ahkcs]

Closing as superseded.

The date-only / time-only DATE/TIME surfacing this PR adds has already landed in main via #22045 ("[analytics-engine] PPL date/time/timestamp fixes", merged 2026-06-09, 47a44bf). There the UDTs were renamed DateType → DateOnlyType and TimeType → TimeOnlyType, with DateFormatClassifier and the format-aware OpenSearchSchemaBuilder.buildLeafType(type, format, factory) in place. Rebasing this branch would only re-add duplicate DateType / TimeType classes alongside the merged *OnlyType ones (which is what the last Merge branch 'main' here turned red on).

Verified against current main on the analytics-engine (parquet-primary) route: CalcitePPLAggregationIT and CalcitePPLAggregationPaginatingIT both pass all 15 date/time-span tests (90/100 overall; the 10 remaining failures are pre-existing and unrelated — approx-percentile, approx-count-distinct, and nested-field aggregations).

SQL-plugin consumer side is covered by the merged opensearch-project/sql#5521.