[2.43] fix: Fixes in analytics SQL engine for ClickHouse compatibility [DHIS2-21417]

* fix: Fixes in analytics SQL engine for ClickHouse compatibility [DHIS2-21417] (#23810)

* fix: [DHIS2-21418] lowercase period column names in event SQL emission

PeriodTypeEnum.getName() returns display-cased names (e.g. "Monthly"); the analytics tables are created with lowercase period columns. Postgres folds unquoted identifiers, Doris is case-insensitive, ClickHouse is not.

Lowercase with Locale.ROOT at every emission point so the reference matches the table column on every engine.

* fix: [DHIS2-21419] lowercase program UID in analytics_event/_enrollment table names

DHIS2 UIDs aremixed-case but the analytics tables themselves
are created with lowercased names (analytics_event_iphinat79uw).
Multiple emission sites concatenated "analytics_event_" + program.getUid() without
lowercasing. Invisible on Postgres (case-folded) and Doris
(lower_case_table_names=1), broken on ClickHouse (UNKNOWN_TABLE).

Apply .toLowerCase() at every emission site.

* fix: [DHIS2-21421] emit JOIN form for date-period-structure lookup on ClickHouse

DateFieldPeriodBucketColumnResolver already supports two emission shapes for the `analytics_rs_dateperiodstructure` lookup, switched on
sqlBuilder.useJoinForDatePeriodStructureLookup(): a correlated scalar subquery (Postgres) or a LEFT JOIN (Doris). ClickHouse's analyzer rejects the correlated form ("Resolve identifier 'eb.lastupdated' from parent scope only supported for constants and CTE"), so override the flag to true.

The JOIN form works on every engine, so this is a strict improvement with no regression risk on Postgres/Doris.

* fix: [DHIS2-21422] identity-alias prefixed enrollment column in aggregate CTEs

ClickHouse's analyzer keeps the original table prefix attached to a
projected column when no explicit output alias is given. So a CTE projecting ax.enrollment produces a column the analyzer still binds to ax - the outer query's eb.enrollment from the enrollment_aggr_base CTE alias is then
unresolvable.
Postgres and Doris implicitly drop the prefix and re-scope under the CTE alias.

Add explicit identity aliases at two emission sites in
JdbcEnrollmentAnalyticsManager: the enrollment_aggr_base CTE projection and
the inline derived-table evf event-date subquery. The alias re-binds the
column under the CTE / derived-table alias on ClickHouse and is a no-op for Postgres/Doris.

* fix: [DHIS2-21423] NULL-safe date cast for analytics joins

ClickHouse's toDate(NULL) and CAST(NULL AS Date) raise
CANNOT_INSERT_NULL_IN_ORDINARY_COLUMN, and toDateOrNull only accepts String
input. Aggregate queries that join to analytics_rs_dateperiodstructure on a
nullable data-element column (e.g. cast(ax."<deUid>" as date)) crash.

Introduce AnalyticsSqlBuilder.castAsDate(String). Default emits ANSI `cast(... as date)` - correct for Postgres and Doris. ClickHouse overrides to
toDateOrNull(toString(...)), which is both NULL-safe and accepts every input type the cast might receive (Date / DateTime / DateTime64 / String / Nullable variants).

JdbcEventAnalyticsManager's hasTimeField LEFT JOIN now routes through sqlBuilder.castAsDate. Output for Postgres/Doris is byte-identical.

* fix: [DHIS2-21428] align dimension columns at SELECT for ClickHouse

Two fixes for the same family of case-sensitivity issue:

1) AggregatedRowBuilder.addDimensionData lowercases the dimension name with Locale.ROOT before the result-set lookup. Period-column SQL emission was
   already lowercased; the row builder still asked the result set for
   "Quarterly" instead of "quarterly", which Postgres/MySQL JDBC normalised
   silently but ClickHouse JDBC rejected with InvalidResultSetAccessException.

2) Add explicit "as <col>" alias to dimension projections in the SELECT context so the result-set column has a canonical name regardless of how the engine reports the underlying expression. ClickHouse JDBC retains  the table prefix on unaliased prefixed columns (ax.uidlevel2) in result-set metadata; Postgres/MySQL strip it.

* fix: [DHIS2-21430] generate legend-set columns on ClickHouse via inline CASE

JdbcEventAnalyticsTableManager.getColumnFromDataElementWithLegendSet
early-returned an empty list on engines without correlated-subquery support
(ClickHouse). The companion <deUid>_<lsUid> column was never created in the
ClickHouse analytics tables, but the query path still emitted SQL referring
to it — every aggregate query grouped or filtered by legend bucket failed
with UNKNOWN_IDENTIFIER.

Replace the early-return with a per-row CASE expression built from the static legend ranges. Same final value populated into the column, no correlated subquery, ClickHouse-friendly. Postgres and Doris are unaffected:
supportsCorrelatedSubquery() returns true on both, so the existing correlated-subquery emission path runs unchanged.

* fix: [DHIS2-21431] populate OU-name support column on ClickHouse via JOIN

Populate the <deUid>_name support column on ClickHouse via a LEFT JOIN to organisationunit instead of a correlated
  subquery (which ClickHouse rejects). The column was previously not created at all, causing UNKNOWN_IDENTIFIER errors in
  aggregate queries that group by an org-unit-typed data element.

* Fix formatting

* Fix failing unit test

* More Unit Test fixes (lower case program)

* refactor: [DHIS2-21420] centralise analytics_event/_enrollment table names

Introduce AnalyticsTableNames in dhis-support-sql with eventTable(Program)
and enrollmentTable(Program) helpers. Replace every inline
"analytics_event_" + program.getUid().toLowerCase() and equivalent
"analytics_enrollment_" + ... concatenation with a call through the helper.

* fix: [DHIS2-21422] identity-alias prefixed enrollment column in PI value CTE

Same ClickHouse-analyzer issue as the prior fix in JdbcEnrollmentAnalyticsManager:
a CTE projecting `subax.enrollment` without an explicit output alias keeps the
column bound to `subax` in ClickHouse's scope, so the outer query's
`<alias>.enrollment` reference cannot be resolved. Postgres and Doris re-scope
the column under the CTE alias implicitly.

Add an explicit identity alias on the projection in
DefaultProgramIndicatorSubqueryBuilder so the column is re-bound under the CTE
alias on ClickHouse. No-op for Postgres/Doris.

* remove doc for tracking issues

* fix: [DHIS2-21441] fix rejected correlated scalar subqueries in Events query

* fix: [DHIS2-21486] fix event PI queries by replacing correlated subqueries with event-keyed CTE joins

* fix: [DHIS2-21487] Fix enrollment stage display-name projections

* fix: [DHIS2-21488] fix relationship count CTE SQL for enrollment PIs

* fix: [DHIS2-21493] Fix ClickHouse dateDiff literal casting

* fix: [DHIS2-21495] Fix option-set stage CTE value projection

* Use correlated subqueries when engine supports it

* fix: scope ClickHouse event PI CTEs to candidate events

Add an internal event_pi_candidates CTE for EVENT analytics queries when
the database does not support correlated subqueries and EVENT
program-indicator CTEs are generated.

The candidate CTE mirrors the outer event query scope, including base
filters such as period, org unit, stage, and non-PI query item filters.
EVENT PI CTEs now read from this candidate source instead of scanning
the full analytics event table. PI filters stay outside the candidate
CTE to avoid circular SQL dependencies.

Postgres and other correlated-subquery paths remain unchanged.
Enrollment PI CTEs keep their existing source behavior.

* fix: keep event query enrollment PIs on CTE path

Narrow the Event analytics correlated-subquery fallback so it only
applies to EVENT-type program indicators. ENROLLMENT-type program
indicators used from the Event endpoint still need the CTE path because
it expands generated placeholders such as `FUNC_CTE_VAR`,
`__PSDE_CTE_PLACEHOLDER__`, and `__D2FUNC__`.

* refactor: [DHIS2-21488] replace relationship-count regex with d2:relationshipCount placeholder pipeline

* fix: use ClickHouse age() for year/month/week date differences

ClickHouse dateDiff() counts boundary crossings, so program indicators
using d2:yearsBetween, d2:monthsBetween and d2:weeksBetween returned
values that differed from PostgreSQL, which counts completed units via
age(). For example dateDiff('year', '2023-12-31', '2024-01-01') is 1
where PostgreSQL reports 0 completed years.

Switch the years, months and weeks cases of
ClickHouseSqlBuilder.dateDifference to age() (available since ClickHouse
23.1), keeping the same operand order. Days and minutes stay on
dateDiff() because PostgreSQL computes them with date subtraction and
elapsed time, which already match.

* fix: normalize empty text to NULL in ClickHouse enrollment aggregate CTEs

Enrollment aggregate queries with program-stage data element dimensions
  returned more groups on ClickHouse than on Postgres. ClickHouse
analytics tables store empty strings for absent text values where
Postgres stores NULL, so GROUP BY split empty and NULL into separate
buckets.
Add AnalyticsSqlBuilder.nullIfEmpty, a no-op by default and overridden
in `ClickHouseAnalyticsSqlBuilder` to wrap the column in `nullif(column,
'')`.

* fix: make ClickHouse safeConcat null-safe for analytics display names

* fix: align ClickHouse event query columns and join semantics with Postgres

* fix: treat ClickHouse empty-string coordinates as null in coordinatesOnly filter

* fix: fold ClickHouse empty-string text dimensions to NULL in event aggregate

* fix: alias wrapped ClickHouse aggregate text columns to keep result-set label

* fix: join ClickHouse relationship-count CTE by alias, not registry key

Author: luciano-fiandesio

dhis-2/dhis-api/src/main/java/org/hisp/dhis/program/ProgramIndicatorService.java

@@ -213,6 +213,28 @@ String getAnalyticsSql(
       Date endDate,
       String tableAlias);
 
+  /**
+   * Same as {@link #getAnalyticsSql(String, DataType, ProgramIndicator, Date, Date, String)} but
+   * leaves the {@code d2:relationshipCount} placeholder unexpanded so callers can apply a
+   * dialect-specific expansion (e.g. a {@code LEFT JOIN} on engines that do not support correlated
+   * subqueries).
+   *
+   * @param expression the expression.
+   * @param dataType the data type to return.
+   * @param programIndicator the program indicator to evaluate.
+   * @param startDate the start date.
+   * @param endDate the end date.
+   * @param tableAlias use this table alias for expression returning a inner query
+   * @return the SQL string with {@code __D2RELCNT__(uid='...')__} placeholders intact.
+   */
+  String getAnalyticsSqlDeferRelationshipCount(
+      String expression,
+      DataType dataType,
+      ProgramIndicator programIndicator,
+      Date startDate,
+      Date endDate,
+      String tableAlias);
+
   /**
    * Gets the analytics SQL clause of an expression. The SQL does not substitute null values.
    *

dhis-2/dhis-services/dhis-service-analytics/src/main/java/org/hisp/dhis/analytics/OrgUnitField.java

@@ -230,7 +230,10 @@ private String ouQuote(String tableAlias, String col, boolean noColumnAlias) {
           + ((noColumnAlias) ? "" : " as " + col);
     }
 
-    return sqlBuilder.quote(tableAlias, col);
+    // Always add the column alias when one is requested. Without it, JDBC drivers that retain the
+    // table prefix in result-set metadata (e.g. ClickHouse) report the column as
+    // "<tableAlias>.<col>", and downstream lookups by bare column name fail.
+    return sqlBuilder.quote(tableAlias, col) + ((noColumnAlias) ? "" : " as " + col);
   }
 
   @Override

dhis-2/dhis-services/dhis-service-analytics/src/main/java/org/hisp/dhis/analytics/common/CteContext.java

@@ -52,11 +52,15 @@
  */
 @Slf4j
 public class CteContext {
+  public static final String EVENT_PROGRAM_INDICATOR_CANDIDATES = "event_pi_candidates";
+
   private final Map<String, CteDefinition> cteDefinitions = new LinkedHashMap<>();
 
   /** The type of analytics query being executed. This can be either EVENT or ENROLLMENT. */
   @Getter private final EndpointItem endpointItem;
 
+  @Getter private String eventProgramIndicatorSourceTable;
+
   public CteDefinition getDefinitionByItemUid(String itemUid) {
     return cteDefinitions.get(itemUid);
   }
@@ -100,6 +104,28 @@ public void addCte(
       int offset,
       boolean isRowContext,
       boolean hasFilter) {
+    addCte(programStage, item, cteDefinition, offset, isRowContext, hasFilter, false);
+  }
+
+  /**
+   * Adds a CTE definition to the context.
+   *
+   * @param programStage The program stage
+   * @param item The query item
+   * @param cteDefinition The CTE definition (the SQL query)
+   * @param offset The calculated offset
+   * @param isRowContext Whether the CTE is a row context
+   * @param hasFilter Whether the query item has a filter requiring non-null CTE values
+   * @param hasValueName Whether the CTE exposes the display-name column as value_name
+   */
+  public void addCte(
+      ProgramStage programStage,
+      QueryItem item,
+      String cteDefinition,
+      int offset,
+      boolean isRowContext,
+      boolean hasFilter,
+      boolean hasValueName) {
     String key = computeKey(item);
     if (cteDefinitions.containsKey(key)) {
       cteDefinitions.get(key).getOffsets().add(offset);
@@ -111,7 +137,8 @@ public void addCte(
               cteDefinition,
               offset,
               isRowContext,
-              hasFilter);
+              hasFilter,
+              hasValueName);
       cteDefinitions.put(key, cteDef);
     }
   }
@@ -154,16 +181,21 @@ public void addExistsCte(ProgramStage programStage, QueryItem item, String cteDe
    * @param cteDefinition The CTE definition (the SQL query)
    * @param functionRequiresCoalesce Whether the function requires to be "wrapped" in coalesce to
    *     avoid null values (e.g. avg, sum)
+   * @param joinColumn The column exposed by the CTE for joining it back to the outer query
    */
   public void addProgramIndicatorCte(
-      ProgramIndicator programIndicator, String cteDefinition, boolean functionRequiresCoalesce) {
+      ProgramIndicator programIndicator,
+      String cteDefinition,
+      boolean functionRequiresCoalesce,
+      String joinColumn) {
     cteDefinitions.put(
         programIndicator.getUid(),
         CteDefinition.forProgramIndicator(
             programIndicator.getUid(),
             programIndicator.getAnalyticsType(),
             cteDefinition,
-            functionRequiresCoalesce));
+            functionRequiresCoalesce,
+            joinColumn));
   }
 
   /**
@@ -258,12 +290,43 @@ public void addD2FunctionCte(String key, CteDefinition cteDefinition) {
     }
   }
 
+  /**
+   * Registers a {@link CteDefinition.CteType#D2_RELATIONSHIP_COUNT} CTE. No-op if a CTE with the
+   * same key was already registered, so multiple PIs (or multiple references inside one PI) for the
+   * same relationship type share a single CTE.
+   */
+  public void addRelationshipCountCte(String key, CteDefinition cteDefinition) {
+    if (cteDefinition != null
+        && key != null
+        && cteDefinition.getCteType() == CteDefinition.CteType.D2_RELATIONSHIP_COUNT) {
+      cteDefinitions.putIfAbsent(key, cteDefinition);
+    } else {
+      log.warn("Attempted to add invalid relationship count CTE definition for key: {}", key);
+    }
+  }
+
   public void addShadowCte(String tableName, String sql, CteDefinition.CteType cteType) {
     // Use a simple CteDefinition for shadow CTEs
     CteDefinition shadowCte = CteDefinition.forShadowTable(tableName, sql, cteType);
     cteDefinitions.put(tableName, shadowCte);
   }
 
+  public void useEventProgramIndicatorCandidateSource() {
+    eventProgramIndicatorSourceTable = EVENT_PROGRAM_INDICATOR_CANDIDATES;
+  }
+
+  public void addEventProgramIndicatorCandidatesCte(String sql) {
+    addShadowCte(
+        EVENT_PROGRAM_INDICATOR_CANDIDATES,
+        sql,
+        CteDefinition.CteType.EVENT_PROGRAM_INDICATOR_CANDIDATES);
+  }
+
+  public boolean hasEventProgramIndicatorCtes() {
+    return cteDefinitions.values().stream()
+        .anyMatch(def -> def.getCteType() == CteDefinition.CteType.PROGRAM_INDICATOR_EVENT);
+  }
+
   public CteDefinition getBaseAggregatedCte() {
     return cteDefinitions.get(ENROLLMENT_AGGR_BASE);
   }

dhis-2/dhis-services/dhis-service-analytics/src/main/java/org/hisp/dhis/analytics/common/CteDefinition.java

@@ -68,8 +68,12 @@ public enum CteType {
     PROGRAM_STAGE_DATE_ELEMENT(10),
     /** CTE replacing a d2:function(...) subquery (like d2:countIfValue). */
     D2_FUNCTION(10),
+    /** Per-trackedEntity relationship count CTE backing {@code d2:relationshipCount(...)}. */
+    D2_RELATIONSHIP_COUNT(10),
     /** CTE for checking existence (rowContext=true). */
     EXISTS(10),
+    /** Candidate event rows used as input for EVENT program indicator CTEs. */
+    EVENT_PROGRAM_INDICATOR_CANDIDATES(4),
     /** Special CTE for pre-aggregated enrollments. */
     TOP_ENROLLMENTS(1),
     SHADOW_ENROLLMENT_TABLE(2),
@@ -110,6 +114,9 @@ public enum CteType {
   /** Whether the query item has a filter that requires non-null values from this CTE */
   private final boolean hasFilter;
 
+  /** Whether the CTE exposes a display-name column as value_name. */
+  private final boolean hasValueName;
+
   private CteDefinition(
       String itemId,
       String programStageUid,
@@ -123,7 +130,8 @@ private CteDefinition(
       String joinColumn,
       Integer targetRank,
       CteType cteType,
-      boolean hasFilter) {
+      boolean hasFilter,
+      boolean hasValueName) {
     this.itemId = itemId;
     this.programStageUid = programStageUid;
     this.programIndicatorUid = programIndicatorUid;
@@ -137,6 +145,7 @@ private CteDefinition(
     this.targetRank = targetRank;
     this.cteType = cteType;
     this.hasFilter = hasFilter;
+    this.hasValueName = hasValueName;
   }
 
   /** Creates a CTE definition for program stage data elements. */
@@ -155,7 +164,8 @@ public CteDefinition(
         null, // joinColumn
         null, // targetRank
         CteType.PROGRAM_STAGE, // Set type
-        false); // hasFilter
+        false, // hasFilter
+        false); // hasValueName
     this.offsets.add(offset);
   }
 
@@ -177,6 +187,18 @@ public CteDefinition(
       int offset,
       boolean isRowContext,
       boolean hasFilter) {
+    this(programStageUid, queryItemId, cteDefinition, offset, isRowContext, hasFilter, false);
+  }
+
+  // Constructor for standard Program Stage CTEs with rowContext, hasFilter, and value_name
+  public CteDefinition(
+      String programStageUid,
+      String queryItemId,
+      String cteDefinition,
+      int offset,
+      boolean isRowContext,
+      boolean hasFilter,
+      boolean hasValueName) {
     this(
         queryItemId,
         programStageUid,
@@ -190,7 +212,8 @@ public CteDefinition(
         null, // joinColumn
         null, // targetRank
         CteType.PROGRAM_STAGE, // Set type
-        hasFilter); // Pass hasFilter
+        hasFilter, // Pass hasFilter
+        hasValueName); // Pass hasValueName
     this.offsets.add(offset);
   }
 
@@ -208,7 +231,8 @@ public CteDefinition(String cteDefinition, String aggregateWhereClause) {
         null, // joinColumn
         null, // targetRank
         CteType.BASE_AGGREGATION, // Set type
-        false); // hasFilter
+        false, // hasFilter
+        false); // hasValueName
   }
 
   /** Creates a CTE definition for program indicators. */
@@ -217,6 +241,17 @@ public static CteDefinition forProgramIndicator(
       AnalyticsType programIndicatorType,
       String cteDefinition,
       boolean requiresCoalesce) {
+    return forProgramIndicator(
+        programIndicatorUid, programIndicatorType, cteDefinition, requiresCoalesce, null);
+  }
+
+  /** Creates a CTE definition for program indicators. */
+  public static CteDefinition forProgramIndicator(
+      String programIndicatorUid,
+      AnalyticsType programIndicatorType,
+      String cteDefinition,
+      boolean requiresCoalesce,
+      String joinColumn) {
     // Calls private constructor, passing CteType.PROGRAM_INDICATOR
     return new CteDefinition(
         programIndicatorUid, // itemId
@@ -228,12 +263,13 @@ public static CteDefinition forProgramIndicator(
         false, // isExists
         requiresCoalesce, // requiresCoalesce
         null, // aggregateWhereClause
-        null, // joinColumn
+        joinColumn, // joinColumn
         null, // targetRank
         programIndicatorType == AnalyticsType.EVENT
             ? PROGRAM_INDICATOR_EVENT
             : PROGRAM_INDICATOR_ENROLLMENT, // Set type
-        false); // hasFilter
+        false, // hasFilter
+        false); // hasValueName
   }
 
   /** Creates a CTE definition for filter CTEs (replacing filter subqueries). */
@@ -253,7 +289,8 @@ public static CteDefinition forFilter(
         null, // joinColumn
         null, // targetRank
         CteType.FILTER, // Set type
-        false); // hasFilter
+        false, // hasFilter
+        false); // hasValueName
   }
 
   /**
@@ -280,7 +317,8 @@ public static CteDefinition forVariable(String key, String cteDefinitionSql, Str
         joinColumn, // Pass joinColumn
         null, // targetRank
         CteType.VARIABLE, // Set type
-        false); // hasFilter
+        false, // hasFilter
+        false); // hasValueName
   }
 
   /**
@@ -302,7 +340,8 @@ public static CteDefinition forProgramStageDataElement(
         joinColumn, // Pass joinColumn
         targetRank, // Pass targetRank
         CteType.PROGRAM_STAGE_DATE_ELEMENT, // Set type
-        false); // hasFilter
+        false, // hasFilter
+        false); // hasValueName
   }
 
   /**
@@ -328,7 +367,34 @@ public static CteDefinition forD2Function(
         joinColumn,
         null, // targetRank
         CteType.D2_FUNCTION,
-        false); // hasFilter
+        false, // hasFilter
+        false); // hasValueName
+  }
+
+  /**
+   * Creates a CTE definition for a d2:relationshipCount aggregation, joined per trackedEntity.
+   *
+   * @param key A unique key identifying this relationship-count CTE instance.
+   * @param cteDefinitionSql The SQL body for the CTE (selects {@code trackedentityid} and {@code
+   *     value}).
+   * @return The created CteDefinition.
+   */
+  public static CteDefinition forRelationshipCount(String key, String cteDefinitionSql) {
+    return new CteDefinition(
+        key,
+        null,
+        null,
+        cteDefinitionSql,
+        generateRandomAlias(),
+        false,
+        false,
+        false,
+        null,
+        "trackedentity",
+        null,
+        CteType.D2_RELATIONSHIP_COUNT,
+        false,
+        false);
   }
 
   public static CteDefinition forShadowTable(String tableName, String sql, CteType cteType) {
@@ -343,7 +409,8 @@ public static CteDefinition forShadowTable(String tableName, String sql, CteType
         null, // aggregateWhereClause
         null, // joinColumn
         null, // targetRank
-        cteType, false); // hasFilter
+        cteType, false, // hasFilter
+        false); // hasValueName
   }
 
   public CteDefinition setExists(boolean exists) {
@@ -374,6 +441,10 @@ public boolean isProgramStage() {
     return this.cteType == CteType.PROGRAM_STAGE && !isExists;
   }
 
+  public boolean hasValueName() {
+    return hasValueName;
+  }
+
   private static String generateRandomAlias() {
     // Use the same alias generation as before
     return new RandomStringGenerator.Builder().withinRange('a', 'z').get().generate(5);