Lateral pg19 (#13)

* ext: enable LATERAL — drop gate, plumb NL nest-params, block correlated commute

PG-style LATERAL in the FROM clause now goes through ORCA instead of falling
back to the standard planner.

Three changes are needed together to make this work:

1. gpopt/translate/CTranslatorQueryToDXL.cpp
   Remove the blanket "LATERAL unsupported" raise. Outer references from a
   LATERAL RTE's inner Query already resolve via the parent translator's
   CMappingVarColId (initialised in the constructor) — no separate plumbing
   needed for the Var-to-ColRef mapping.

2. libgpopt/src/translate/CTranslatorExprToDXL.cpp (PdxlnNLJoin)
   Generalise the existing DPE-PartitionSelector branch that turns a NL into
   an IndexNLJ when the inner side has outer refs to the outer child's
   output. Any inner subtree whose outer references intersect the outer
   child's output now goes the same route: outer_refs get registered in
   m_phmcrdxlnIndexLookup so the inner scalar translator emits a resolved
   Ident, and the NL emits PARAM_EXEC nest params (under
   EopttraceIndexedNLJOuterRefAsParams, which pg_orca enables by default).
   Also drops the GPOS_ASSERT_IMP that forbade outer refs in non-index NL
   inner children — that condition is now legal.

3. libgpopt/src/xforms/CXformInnerJoinCommutativity.{cpp,h}
   Tighten the xform's promise: refuse to commute an InnerJoin when one
   child's outer references reach into the other child's output columns. A
   plain CLogicalInnerJoin is symmetric in ORCA's algebra, so without this
   guard the join enumerator happily produced the swapped orientation for a
   LATERAL-shaped join. Putting the correlated side on the NL outer is
   unexecutable: the executor opens outer first and the outer-ref columns
   are still unbound.

After this commit the six LATERAL shapes (uncorrelated, equi-correlated,
constant projection, TVF outer-ref arg, LIMIT, LEFT JOIN LATERAL) all
optimise to a correct ORCA plan, though equi-correlated cases still emit
NL+Materialize rather than HashJoin — that decorrelation comes in the
follow-up commit.

* ext: lower equi-correlated LATERAL to Apply for HashJoin decorrelation

LATERAL with a top-level CLogicalSelect on the right side (the typical
"FROM x, LATERAL (SELECT ... WHERE inner = x.col)" pattern) was being
turned into a plain CLogicalInnerJoin, which forced the optimizer onto
the NL+Materialize path. PG's own planner pulls these LATERALs up into
a HashJoin; ORCA can do the same once the algebra is right.

libgpopt/src/translate/CTranslatorDXLToExpr.cpp (PexprLogicalJoin)
  Heuristic gate: if the right child's top operator is a CLogicalSelect
  whose outer references intersect the left child's output, rebuild the
  join as CLogicalInnerApply / CLogicalLeftOuterApply instead of a plain
  Join. CXformInnerApply2InnerJoin / CXformLeftOuterApply2LeftOuterJoin
  then pull the correlated predicate out of the Select and lower the
  result to a Join, which the cost model picks as HashJoin.

  Restricting to the "top is CLogicalSelect" shape keeps the other
  LATERAL forms (TVF outer-ref args, LIMIT/Sort, constant projection)
  on the plain-Join path, where the previous commit's commutativity
  guard handles them via NL + nest params.

libgpopt/include/gpopt/xforms/CXformApply2Join.h (CreateCorrelatedApply)
  Guard against TApply::PdrgPcrInner() == nullptr. Apply objects built
  from LATERAL have no inner scalar colref (LATERAL returns a relation,
  not a scalar), so the scalar-subquery-shaped correlated-apply form
  here doesn't apply — no-op instead of dereferencing the null pointer.

Effect on the six LATERAL cases: equi-correlated inner-LATERAL drops
from NL+Materialize to a clean HashJoin matching PG; LEFT LATERAL drops
to HashRightJoin (still carries a triplicated Hash Cond pending the
LOJ inferred-pred dedup in the next commit). Other shapes unchanged.

* orca: dedup inferred predicates on LeftOuterJoin

PexprInferPredicates extends a join's predicate with extras derived from
constraint propagation (e.g. the commuted form of an equality, transitive
closures, etc). After predicate push-through, MakeJoinWithoutInferredPreds
strips the redundant ones back out via PexprRemoveImpliedConjuncts, keyed
on equivalence classes.

CanRemoveInferredPredicates was hard-coded to InnerJoin only — left over
from when LOJ semantics were considered too tricky to dedupe. The original
note "currently, only inner join is included, but we can add more later"
acknowledged the limitation. LeftOuterJoin's null-preserving side cares
about which qualifying tuples pair up, not about how many copies of an
equivalent equality predicate the matcher evaluates, so the dedup is
semantically safe.

Symptom: LEFT JOIN LATERAL (... WHERE inner = outer.col) produced a
HashRightJoin whose Hash Cond was a 3-way AND of equivalent equalities:

  Hash Cond: ((lt2.a = lt1.x) AND (lt2.a = lt1.x) AND (lt2.a = lt1.x))

Three forms entered the predicate during preprocessing — original a=x,
commuted x=a from constraint inference, and a re-pushed copy — and the
LOJ branch of MakeJoinWithoutInferredPreds was a no-op. After this
commit the predicate is the single (lt2.a = lt1.x).

* orca: preserve sibling-correlated outer refs during computed-col pruning

PexprPruneUnusedComputedColsRecursive walks the expression top-down with
a required-columns set, dropping CScalarProjectElements that nothing
upstream consumes. The required set was built from each operator's own
PcrsLocalUsed and its scalar children's used columns (via
CExpressionHandle::PcrsUsedColumns) — neither of which captures outer
references that one relational child holds against another sibling.

For a LATERAL whose inner references a computed column from a derived
table on the outer side, the chain is:

  LogicalApply / LogicalJoin
    ├── LogicalProject(dv = val * 2)
    │    └── LogicalGet(na)
    └── LogicalSelect(filter: nb.id = dv)
         └── LogicalGet(nb)

The inner Select's DeriveOuterReferences() = {dv}, but the Apply's
PcrsUsedColumns() returns only the columns from the scalar predicate
(true) and PcrsLocalUsed (empty). The pruner descends into the outer
Project with `dv` absent from pcrsReqd → defined - required = {dv} →
the Project gets stripped. The dangling CScalarIdent "dv" then crashes
DXL→PlStmt translation with "Attribute number N not found".

Fix: before recursing into children, fold each relational child's
DeriveOuterReferences() into pcrsReqd. Those refs are columns the child
needs from its siblings, so siblings' producers must be preserved.

Includes outer refs that escape this operator entirely (genuine refs to
the grandparent) — those just stay in pcrsReqd as we descend; they have
no producer at this level and pruning logic only acts on Project /
GbAgg defined columns, so the extra entries are harmless.

Symptom: `SELECT count(*) FROM (SELECT id, val*2 AS dv FROM t) a,
LATERAL (SELECT * FROM s WHERE s.id = a.dv) x` fell back to PG with
"DXL-to-PlStmt Translation: Attribute number 8 not found in project list".
After this commit the query lowers to a clean HashJoin under ORCA.

* test: lock in LATERAL coverage with 18 cases

Adds a dedicated lateral.sql / lateral.out regression test under
test/schedule covering the six base LATERAL shapes (uncorrelated,
equi-correlated, scalar projection, TVF outer-ref arg, LIMIT, LEFT
JOIN LATERAL) plus twelve nested / composite variants:

- 2-level and 3-level chains (each LATERAL references its immediate
  outer; or one LATERAL references both outer and middle)
- LATERAL nested inside another LATERAL
- LATERAL containing an inner JOIN (decorrelates to HashJoin chain)
- LEFT JOIN LATERAL with a nested LATERAL + LIMIT
- LATERAL containing an aggregate
- LATERAL with non-equi range correlation
- 3-level chain ending in a TVF
- LATERAL inside EXISTS
- LEFT LATERAL with an inner filter that excludes everything
- LATERAL referencing a derived-table computed column under an
  aggregate (regression for the sibling-correlated outer-ref pruning
  bug fixed in PexprPruneUnusedComputedColsRecursive)

The expected file pins the actual plan shape (Hash Join / Hash Right
Join / Function Scan / NL with nest params, etc.), so any future
change that regresses the commutativity guard, the selective Apply
conversion, the LOJ inferred-pred dedup, or the preprocessor outer-
ref preservation will diff visibly. Three back-to-back fresh-instance
runs show the plans are deterministic at ~150 ms.

* orca: PopCopyWithRemappedColumns nullptr guard for LATERAL Apply

CLogicalInnerApply / CLogicalLeftOuterApply built from LATERAL have no
inner scalar colref (LATERAL returns a relation, not a scalar), so
m_pdrgpcrInner is nullptr. PopCopyWithRemappedColumns blindly called
CUtils::PdrgpcrRemap on it, which dereferences the null array in
Release builds and SIGSEGVs (Debug builds catch it at the assert).

The crash is reached whenever ORCA needs to deep-copy a LATERAL-derived
Apply with column remapping, e.g. when a CLogicalCTEConsumer inlines a
producer whose body contains the Apply:

  WITH t AS (SELECT * FROM a, LATERAL (SELECT * FROM b WHERE ...) s)
  SELECT count(*) FROM t;

Guard against nullptr m_pdrgpcrInner and rebuild the copy with the
1-arg ctor (the 2-arg form asserts pdrgpcrInner is non-null+non-empty).

Found by walking the LATERAL edge-case matrix; lateral.sql now covers
this shape under E2.

* test: extend lateral.sql with 12 edge cases

Adds an "Edge cases" section to lateral.sql covering shapes that came
out of an LATERAL edge-case sweep:

  E1   varlevelsup=2: LATERAL nested in a correlated scalar subquery
  E2   LATERAL inside a CTE body (locks in the InnerApply nullptr-
       PdrgPcrInner copy-with-remap crash that this commit pairs with)
  E3   VALUES + LATERAL
  E4   LATERAL + GROUP BY at outer
  E5   LATERAL + GROUPING SETS
  E6   LATERAL + window function
  E7   UNION ALL inside the LATERAL body
  E8   LATERAL with DISTINCT outside
  E9   LATERAL top-N per outer (ORDER BY ... LIMIT 1)
  E10  INSERT...SELECT with LATERAL
  E11  3-level LATERAL where the grandchild references the outermost
  E12  LATERAL unnest(array)

All twelve go through ORCA without fallback. Three back-to-back
fresh-instance runs settle at ~211 ms; the plan shapes are stable.

Edge cases that intentionally do NOT land here:
  - CTE inside a LATERAL body -> pre-existing ORCA limitation
    "Operator CTE with outer references not supported"
  - PREPARE/EXECUTE with $params -> pre-existing limitation
    (requires optimizer_enable_query_parameter, not wired in pg_orca)
  - FOR UPDATE on an aggregate query -> SQL-level rejection

* test: import PG join.sql LATERAL section as pg_lateral regression

Adds test/sql/pg_lateral.sql — a verbatim port of the "Test LATERAL"
block from PostgreSQL upstream's src/test/regress/sql/join.sql (the
chunk introduced by the "-- Test LATERAL" header), plus inline setup
for int2_tbl / int4_tbl / int8_tbl / tenk1 / onerow mirroring upstream's
test_setup.sql and the top of join.sql. tenk1's 10000-row payload is
loaded via COPY from $PG_REGRESS_SQL/data/tenk.data so the test stays
in sync with PG when that file changes.

Locks in roughly 95 queries covering:
  - basic equi-correlated LATERAL with tenk1 / int4_tbl / int8_tbl
  - lateral-versus-parent scope resolution (the int8_tbl q1/q2 case)
  - LATERAL with TVF args, UNION ALL, VALUES, GROUPING SETS-adjacent
    aggregates, JOIN inside LATERAL
  - lateral references requiring pullup at outer-join boundaries
  - PlaceHolderVar nesting and the bug #9041 postponed-quals case
  - dummy/empty inner rels (bug #15694)
  - LATERAL with VALUES tuple containing outer refs to both sides of
    an enclosing LEFT JOIN
  - intentional SQL-level rejections (missing LATERAL keyword,
    RIGHT/FULL JOIN with LATERAL, ambiguous column refs, UPDATE/DELETE
    LATERAL restrictions)

Current ORCA coverage on this set: 15 queries optimised by ORCA (visible
Optimizer: pg_orca marker), 22 queries fall back to PG, all correctness
preserved.  Fallback breakdown:
  - 14 "DXL-to-PlStmt Translation: Attribute number N not found in
    project list" — same class as the sibling-correlated outer-ref bug
    fixed in CExpressionPreprocessor, but in more complex shapes
    involving LeftOuter + LATERAL VALUES referencing both sides of
    the outer join; ORCA's enumerator places the LATERAL on a side that
    can't see one of its referenced columns at execution time
  - 2 "Whole-row variable" (pre-existing ORCA limitation, e.g.
    coalesce(i) on a record type)
  - 2 "no plan has been computed for required properties" (enum gap)
  - 4 intentional PG-side SQL rejections

Three back-to-back fresh-instance runs land at ~370 ms with stable
plans. Future fixes to the still-falling-back patterns will show up as
fewer fallback INFO lines and more Optimizer: pg_orca markers in the
expected output diff.

Statement timeout of 20s guards against ORCA picking a pathologically
bad plan that would otherwise hang the whole regression suite.

* orca: enforce LATERAL sibling visibility in DPv2 join enumeration

CJoinOrderDPv2 enumerates join orders by combining subsets of the
NAryJoin's atoms via dynamic programming. It tracked LOJ right-child
dependencies (an LOJ's right side must be paired with its left), but
did not track the more general LATERAL-style dependency where one
atom holds outer references to another sibling atom's output columns.

Without this check the enumerator happily formed subsets like
{x, lateral_ref_to_y} from a query

  SELECT * FROM
    int8_tbl x LEFT JOIN (SELECT q1, coalesce(q2,0) q2 FROM int8_tbl) y
                       ON x.q2 = y.q1,
    LATERAL (VALUES (x.q1, y.q1, y.q2)) v(xq1, yq1, yq2);

The chosen physical plan placed the LATERAL VALUES inside a Nested
Loop whose outer was just x — but the VALUES references colids
produced by y, which is on a different side of the enclosing
LeftOuter. At execution time those refs are unbound; DXL→PlStmt
catches it as "Attribute number N not found in project list" and
falls back to the PG planner.

Precompute per-atom sibling requirements in the DPv2 constructor:

  outer_refs_i  = atom_i.DeriveOuterReferences()
  sibling_refs  = outer_refs_i − m_outer_refs     // refs to NAryJoin
                                                  // siblings, not refs
                                                  // escaping the join
  sibling_required[i] = { j | sibling_refs ∩ atom_j.DeriveOutputColumns() }

In GetJoinExpr, reject any candidate join whose combined atom set is
missing a required sibling of one of its members. The DP table then
never enumerates the unexecutable subset, and the LATERAL atom can
only enter the join once all its required siblings are already in.

Effect on test/sql/pg_lateral.sql (the PostgreSQL upstream LATERAL
section ported over): fallbacks drop from 22 to 12, ORCA-handled
EXPLAINs go from 15 to 17, and the "DXL-to-PlStmt Translation:
Attribute number N not found" pattern from this query shape
disappears.  Other ORCA tests, the PG --pg-tests suite, and
cost_align.sh are unchanged.

Three back-to-back fresh-instance pg_lateral runs land at ~358 ms
with deterministic plans.

* test: relock pg_lateral expected — 10 fewer fallbacks after DPv2 fix

Re-captures test/expected/pg_lateral.out after the DPv2 sibling-
visibility enforcement. Bottom-line change:

  fallback INFO lines: 22 -> 12   (-10)
  "Optimizer: pg_orca" markers: 15 -> 17   (+2)

The "DXL-to-PlStmt Translation: Attribute number N not found in
project list" class of failure that came from the LeftOuter +
LATERAL VALUES atom-subset bug is now gone. Remaining 12 fallbacks
are unrelated patterns (Whole-row variable, "no plan computed for
required properties", and the four intentional PG-side SQL errors
from upstream's join.sql).

Author: yjhjstz

gpopt/translate/CTranslatorQueryToDXL.cpp

@@ -3269,11 +3269,12 @@ CTranslatorQueryToDXL::TranslateFromClauseToDXL(Node *node)
 
 			/* forceDistRandom is GPDB-only */
 
-		if (rte->lateral)
-		{
-			GPOS_RAISE(gpdxl::ExmaDXL, gpdxl::ExmiQuery2DXLUnsupportedFeature,
-					   GPOS_WSZ_LIT("LATERAL"));
-		}
+		// LATERAL is supported. Outer references from a LATERAL RTE's inner
+		// query resolve against the parent's CMappingVarColId (set up by the
+		// translator ctor) via varlevelsup; the optimizer side requires the
+		// commutativity guard in CXformInnerJoinCommutativity and the
+		// outer-refs-as-PARAM_EXEC plumbing in PdxlnNLJoin / PdxlnHashJoin to
+		// execute correlated joins correctly.
 
 		if (rte->funcordinality)
 		{

libgpopt/include/gpopt/xforms/CJoinOrderDPv2.h

@@ -441,6 +441,14 @@ class CJoinOrderDPv2 : public CJoinOrder,
 	// outer references, if any
 	CColRefSet *m_outer_refs;
 
+	// LATERAL-style sibling requirements: m_atom_sibling_required[i] is the
+	// bitset of atom indices whose output columns atom i references. An atom
+	// can only appear in a join subset that also contains all of its
+	// requirements; otherwise the subset is unexecutable (the atom's outer
+	// refs would be unbound). Populated in the constructor and consulted in
+	// GetJoinExpr.
+	CBitSetArray *m_atom_sibling_required;
+
 	CMemoryPool *m_mp;
 
 	SLevelInfo *

libgpopt/include/gpopt/xforms/CXformApply2Join.h

@@ -68,6 +68,17 @@ class CXformApply2Join : public CXformExploration
 			return;
 		}
 
+		TApply *popApply = TApply::PopConvert(pexprApply->Pop());
+		CColRefArray *colref_array = popApply->PdrgPcrInner();
+		if (nullptr == colref_array)
+		{
+			// Apply created from LATERAL (or any non-scalar-subquery source)
+			// has no inner scalar colref; the correlated-apply form built
+			// here is scalar-subquery-shaped and does not apply. Skip.
+			return;
+		}
+		GPOS_ASSERT(1 == colref_array->Size());
+
 		CExpression *pexprInner = (*pexprApply)[1];
 		CExpression *pexprOuter = (*pexprApply)[0];
 		CExpression *pexprScalar = (*pexprApply)[2];
@@ -77,11 +88,6 @@ class CXformApply2Join : public CXformExploration
 		pexprScalar->AddRef();
 		CExpression *pexprResult = nullptr;
 
-		TApply *popApply = TApply::PopConvert(pexprApply->Pop());
-		CColRefArray *colref_array = popApply->PdrgPcrInner();
-		GPOS_ASSERT(nullptr != colref_array);
-		GPOS_ASSERT(1 == colref_array->Size());
-
 		colref_array->AddRef();
 		COperator::EOperatorId eopidSubq = popApply->EopidOriginSubq();
 		COperator::EOperatorId op_id = pexprApply->Pop()->Eopid();

libgpopt/include/gpopt/xforms/CXformInnerJoinCommutativity.h

@@ -57,12 +57,7 @@ class CXformInnerJoinCommutativity : public CXformExploration
 	BOOL FCompatible(CXform::EXformId exfid) override;
 
 	// compute xform promise for a given expression handle
-	EXformPromise
-	Exfp(CExpressionHandle &  // exprhdl
-	) const override
-	{
-		return CXform::ExfpHigh;
-	}
+	EXformPromise Exfp(CExpressionHandle &exprhdl) const override;
 
 	// actual transform
 	void Transform(CXformContext *pxfctxt, CXformResult *pxfres,

libgpopt/src/base/CUtils.cpp

@@ -4928,7 +4928,8 @@ CUtils::PexprMatchEqualityOrINDF(
 CExpression *
 CUtils::MakeJoinWithoutInferredPreds(CMemoryPool *mp, CExpression *join_expr)
 {
-	GPOS_ASSERT(COperator::EopLogicalInnerJoin == join_expr->Pop()->Eopid());
+	GPOS_ASSERT(COperator::EopLogicalInnerJoin == join_expr->Pop()->Eopid() ||
+				COperator::EopLogicalLeftOuterJoin == join_expr->Pop()->Eopid());
 
 	CExpressionHandle expression_handle(mp);
 	expression_handle.Attach(join_expr);
@@ -5047,11 +5048,11 @@ CUtils::Equals(const IMDId *mdid, const IMDId *other_mdid)
 }
 
 // operators from which the inferred predicates can be removed
-// NB: currently, only inner join is included, but we can add more later.
 BOOL
 CUtils::CanRemoveInferredPredicates(COperator::EOperatorId op_id)
 {
-	return op_id == COperator::EopLogicalInnerJoin;
+	return op_id == COperator::EopLogicalInnerJoin ||
+		   op_id == COperator::EopLogicalLeftOuterJoin;
 }
 
 CExpressionArrays *

libgpopt/src/operators/CExpressionPreprocessor.cpp

@@ -2486,6 +2486,23 @@ CExpressionPreprocessor::PexprPruneUnusedComputedColsRecursive(
 	CExpressionArray *pdrgpexpr = GPOS_NEW(mp) CExpressionArray(mp);
 	const ULONG ulChildren = pexpr->Arity();
 
+	// Sibling-correlated references: a relational child (e.g. the inner side
+	// of an Apply or a LATERAL-style Join) may reference columns produced by
+	// another relational child. Those references look like outer references
+	// to the inner child but are satisfied by the sibling, so they must be
+	// kept in pcrsReqd to prevent pruning the producing sibling's project
+	// list. Without this, derived-table computed columns (LATERAL inner ref
+	// to `a.val*2 AS dv`) get dropped from the outer side and leave dangling
+	// CScalarIdent in the inner predicate.
+	for (ULONG ul = 0; ul < ulChildren; ul++)
+	{
+		CExpression *pexprChild = (*pexpr)[ul];
+		if (pexprChild->Pop()->FLogical())
+		{
+			pcrsReqd->Include(pexprChild->DeriveOuterReferences());
+		}
+	}
+
 	for (ULONG ul = 0; ul < ulChildren; ul++)
 	{
 		CExpression *pexprChild =

libgpopt/src/operators/CLogicalInnerApply.cpp

@@ -111,6 +111,14 @@ CLogicalInnerApply::PopCopyWithRemappedColumns(CMemoryPool *mp,
 											   UlongToColRefMap *colref_mapping,
 											   BOOL must_exist)
 {
+	if (nullptr == m_pdrgpcrInner)
+	{
+		// LATERAL-derived Apply has no inner scalar colref. The 2-arg ctor
+		// asserts pdrgpcrInner is non-null+non-empty, so use the 1-arg
+		// form which preserves the nullptr.
+		return GPOS_NEW(mp) CLogicalInnerApply(mp);
+	}
+
 	CColRefArray *pdrgpcrInner =
 		CUtils::PdrgpcrRemap(mp, m_pdrgpcrInner, colref_mapping, must_exist);
 

libgpopt/src/operators/CLogicalLeftOuterApply.cpp

@@ -109,6 +109,13 @@ COperator *
 CLogicalLeftOuterApply::PopCopyWithRemappedColumns(
 	CMemoryPool *mp, UlongToColRefMap *colref_mapping, BOOL must_exist)
 {
+	if (nullptr == m_pdrgpcrInner)
+	{
+		// LATERAL-derived Apply: no inner scalar colref. Use the 1-arg ctor
+		// (the 2-arg form asserts pdrgpcrInner is non-null+non-empty).
+		return GPOS_NEW(mp) CLogicalLeftOuterApply(mp);
+	}
+
 	CColRefArray *pdrgpcrInner =
 		CUtils::PdrgpcrRemap(mp, m_pdrgpcrInner, colref_mapping, must_exist);
 

libgpopt/src/translate/CTranslatorDXLToExpr.cpp

@@ -37,7 +37,9 @@
 #include "gpopt/operators/CLogicalForeignGet.h"
 #include "gpopt/operators/CLogicalGbAgg.h"
 #include "gpopt/operators/CLogicalGet.h"
+#include "gpopt/operators/CLogicalInnerApply.h"
 #include "gpopt/operators/CLogicalInsert.h"
+#include "gpopt/operators/CLogicalLeftOuterApply.h"
 #include "gpopt/operators/CLogicalIntersect.h"
 #include "gpopt/operators/CLogicalIntersectAll.h"
 #include "gpopt/operators/CLogicalLimit.h"
@@ -2094,6 +2096,42 @@ CTranslatorDXLToExpr::PexprLogicalJoin(const CDXLNode *dxlnode)
 	// get the scalar condition and then translate it
 	CDXLNode *pdxlnCond = (*dxlnode)[ulChildCount - 1];
 	CExpression *pexprCond = PexprScalar(pdxlnCond);
+
+	// LATERAL fast-path: when the right child's top operator is a
+	// CLogicalSelect whose predicate references the left child's output
+	// columns, rebuild the join as CLogicalApply. The standard
+	// CXformInnerApply2InnerJoin / CXformLeftOuterApply2LeftOuterJoin path
+	// then pulls the correlated equi predicate out of the Select and lowers
+	// the result to a plain CLogicalInnerJoin / CLogicalLeftOuterJoin —
+	// which downstream cost-picks HashJoin (matching PG's lateral pullup).
+	// Other LATERAL shapes (TVF args, ConstTableGet, Limit, Sort) stay on
+	// the plain-Join path and rely on the commutativity guard.
+	if ((EdxljtInner == join_type || EdxljtLeft == join_type) &&
+		2 == pdrgpexprChildren->Size())
+	{
+		CExpression *pexprLeft = (*pdrgpexprChildren)[0];
+		CExpression *pexprRight = (*pdrgpexprChildren)[1];
+		if (COperator::EopLogicalSelect == pexprRight->Pop()->Eopid())
+		{
+			CColRefSet *pcrsRightOuterRefs =
+				pexprRight->DeriveOuterReferences();
+			CColRefSet *pcrsLeftOutput = pexprLeft->DeriveOutputColumns();
+			if (!pcrsRightOuterRefs->IsDisjoint(pcrsLeftOutput))
+			{
+				pexprLeft->AddRef();
+				pexprRight->AddRef();
+				pdrgpexprChildren->Release();
+				if (EdxljtInner == join_type)
+				{
+					return CUtils::PexprLogicalApply<CLogicalInnerApply>(
+						m_mp, pexprLeft, pexprRight, pexprCond);
+				}
+				return CUtils::PexprLogicalApply<CLogicalLeftOuterApply>(
+					m_mp, pexprLeft, pexprRight, pexprCond);
+			}
+		}
+	}
+
 	pdrgpexprChildren->Append(pexprCond);
 
 	return CUtils::PexprLogicalJoin(m_mp, join_type, pdrgpexprChildren);

libgpopt/src/translate/CTranslatorExprToDXL.cpp

@@ -4513,22 +4513,11 @@ CTranslatorExprToDXL::PdxlnNLJoin(CExpression *pexprInnerNLJ,
 	CExpression *pexprScalar = (*pexprInnerNLJ)[2];
 
 
-#ifdef GPOS_DEBUG
-	// Allow outer refs in inner child when:
-	//  (a) it's an index NLJ (outer refs are explicit and expected), or
-	//  (b) inner child is PartitionSelector for DPE NLJ (outer refs are
-	//      partition-key predicates referencing the probe child).
-	GPOS_ASSERT_IMP(
-		COperator::EopPhysicalInnerIndexNLJoin != pop->Eopid() &&
-			COperator::EopPhysicalLeftOuterIndexNLJoin != pop->Eopid() &&
-			COperator::EopPhysicalLeftSemiIndexNLJoin != pop->Eopid() &&
-			COperator::EopPhysicalLeftAntiSemiIndexNLJoin != pop->Eopid() &&
-			COperator::EopPhysicalPartitionSelector !=
-				pexprInnerChild->Pop()->Eopid(),
-		pexprInnerChild->DeriveOuterReferences()->IsDisjoint(
-			pexprOuterChild->DeriveOutputColumns()) &&
-			"detected outer references in NL inner child");
-#endif	// GPOS_DEBUG
+	// Outer refs in NL inner child are legal when handled below: (a) explicit
+	// index NLJ flavors, (b) DPE PartitionSelector, or (c) general LATERAL /
+	// correlated NL where the inner side references outer's output columns —
+	// in all three cases the refs are bound to PARAM_EXEC slots via the
+	// nest-params machinery in PdxlnNLJoin.
 
 	EdxlJoinType join_type = EdxljtSentinel;
 	BOOL is_index_nlj = false;
@@ -4591,6 +4580,42 @@ CTranslatorExprToDXL::PdxlnNLJoin(CExpression *pexprInnerNLJ,
 			GPOS_ASSERT(!"Invalid join type");
 	}
 
+	// General correlated NLJ (LATERAL, decorrelated subquery, or any case where
+	// ORCA picked plain CPhysicalInnerNLJoin / LeftOuterNLJoin with outer
+	// references in the inner subtree pointing back at the outer child's
+	// output columns). Treat as an index NLJ so the refs are bound via
+	// PARAM_EXEC nest params, matching the IndexNLJ code path below.
+	if (!is_index_nlj &&
+		COperator::EopPhysicalPartitionSelector !=
+			pexprInnerChild->Pop()->Eopid())
+	{
+		CColRefSet *pcrsInnerOuterRefs =
+			pexprInnerChild->DeriveOuterReferences();
+		CColRefSet *pcrsOuterOutput = pexprOuterChild->DeriveOutputColumns();
+		if (!pcrsInnerOuterRefs->IsDisjoint(pcrsOuterOutput))
+		{
+			CColRefSet *pcrsIntersect =
+				GPOS_NEW(m_mp) CColRefSet(m_mp, *pcrsInnerOuterRefs);
+			pcrsIntersect->Intersection(pcrsOuterOutput);
+			outer_refs = pcrsIntersect->Pdrgpcr(m_mp);
+			pcrsIntersect->Release();
+			is_index_nlj = true;
+			// Pre-populate the ident map so PdxlnScalar resolves these outer
+			// refs while emitting scalar DXL inside the inner subtree.
+			for (ULONG ul = 0; ul < outer_refs->Size(); ul++)
+			{
+				CColRef *pcr = (*outer_refs)[ul];
+				if (nullptr == m_phmcrdxlnIndexLookup->Find(pcr))
+				{
+					CDXLNode *dxlnode = CTranslatorExprToDXLUtils::PdxlnIdent(
+						m_mp, m_phmcrdxln, m_phmcrdxlnIndexLookup,
+						m_phmcrulPartColId, pcr);
+					m_phmcrdxlnIndexLookup->Insert(pcr, dxlnode);
+				}
+			}
+		}
+	}
+
 	// DPE NLJ: inner child is PartitionSelector wrapping AppendTableScan.
 	// Treat this like an index NLJ so outer (probe) column refs in the
 	// PartitionSelector filter are passed as PARAM_EXEC to the inner side,