ShaderCompiler: fix 3-argument texture() rejected by glslang (#1755)

## Summary

The sampler vertical-flip applied on the **D3D / Metal / Vulkan**
backends was injected as a **2-argument** function-like macro:

```glsl
#define texture(x,y) texture(x, flip(y))
#define textureLod(x,y,z) textureLod(x, flip(y), z)
```

This cannot match the **3-argument** bias form `texture(sampler, uv,
bias)` emitted by some Babylon.js shaders (e.g. GreasedLine:
`texture(grl_colors, uv, 0.)`), and glslang's preprocessor has **no
variadic-macro support** (`Too many args in macro`), so those shaders
failed to compile — surfacing as a `BJS - Error compiling effect` retry
loop that hangs the test.

## Fix

Replace the `texture()`/`textureLod()` macros with a
`FlipSamplerCoordinates` **AST traverser** that rewrites a 2-component
sample coordinate to `vec2(u, 1.0 - v)` for any call arity and sampler
type. It runs only on the backends that already apply
`ProcessSamplerFlip` (D3D, Metal, Vulkan); the OpenGL backend shares
bgfx's V-orientation and is unaffected. `texelFetch` keeps its macro
(integer texel coordinates), and 3D/cube/array (vec3+) coordinates are
left untouched, matching the previous `flip(vec2)`/`flip(vec3)`
behavior.

Fixes
[ThePirateCove#1656](BabylonJS/ThePirateCove#1656).

## Verification (D3D11 Debug)

- GreasedLine idx 149 / 150 now **compile and render** — no hang, no
`Error compiling effect`.
- **No regression** across 20 diverse texture / postprocess / shadow /
refraction / glow / cubemap tests (idx 5, 8, 25, 35, 37, 155, 373, 396,
410, 487, 4, 23, 32, 36, 49, 131, 135, 254, 343, 394).
- OpenGL backend build-checked (`GRAPHICS_API=OpenGLWindowsDevOnly`);
the unchanged sampler path compiles clean.

## Note

This PR originally also bumped the bgfx transient vertex-buffer limits
for
[ThePirateCove#1570](BabylonJS/ThePirateCove#1570).
On investigation that turned out to be a misdiagnosis (the overflow is a
nanovg Canvas burst, not Gaussian Splatting / vertex pulling), so that
change was dropped — details posted on #1570.

Co-authored-by: Branimir Karadzic <branimirkaradzic@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: 3 people

Plugins/ShaderCompiler/Source/ShaderCompilerCommon.cpp

@@ -35,18 +35,14 @@ namespace Babylon::ShaderCompilerCommon
             throw std::runtime_error{"ProcessSamplerFlip: Could not find shader name define."};
         }
 
+        // The vertical (V) flip applied to texture()/textureLod() sample coordinates is performed by
+        // the FlipSamplerCoordinates AST traverser, not by a preprocessor macro. A 2-argument
+        // function-like macro (`#define texture(x,y) texture(x, flip(y))`) cannot match the
+        // 3-argument bias form `texture(sampler, uv, bias)` emitted by some Babylon.js shaders (e.g.
+        // GreasedLine), and glslang's preprocessor has no variadic-macro support, so those shaders
+        // failed to compile. texelFetch keeps its macro because it takes integer texel coordinates,
+        // which the float-coordinate AST flip does not handle.
         static const auto textureSamplerFunctions = R"(
-            highp vec2 flip(highp vec2 uv)
-            {
-                return vec2(uv.x, 1. - uv.y);
-            }
-            highp vec3 flip(highp vec3 uv)
-            {
-                return uv;
-            }
-
-            #define texture(x,y) texture(x, flip(y))
-            #define textureLod(x,y,z) textureLod(x, flip(y), z)
             #define texelFetch(tex, uv, lod) texelFetch((tex), ivec2((uv).x, textureSize((tex), (lod)).y - 1 - (uv).y), (lod))
             #define SHADER_NAME)";
 

Plugins/ShaderCompiler/Source/ShaderCompilerDXBC.cpp

@@ -102,6 +102,8 @@ namespace Babylon::Plugins
         }
 
         ShaderCompilerTraversers::IdGenerator ids{};
+        // Flip 2D texture sample coordinates (replaces the former ProcessSamplerFlip texture() macro).
+        ShaderCompilerTraversers::FlipSamplerCoordinates(program);
         auto cutScope = ShaderCompilerTraversers::ChangeUniformTypes(program, ids);
         auto utstScope = ShaderCompilerTraversers::MoveNonSamplerUniformsIntoStruct(program, ids);
         std::map<std::string, std::string> vertexAttributeRenaming = {};

Plugins/ShaderCompiler/Source/ShaderCompilerDXIL.cpp

@@ -178,6 +178,8 @@ namespace Babylon::Plugins
         }
 
         ShaderCompilerTraversers::IdGenerator ids{};
+        // Flip 2D texture sample coordinates (replaces the former ProcessSamplerFlip texture() macro).
+        ShaderCompilerTraversers::FlipSamplerCoordinates(program);
         auto cutScope = ShaderCompilerTraversers::ChangeUniformTypes(program, ids);
         auto utstScope = ShaderCompilerTraversers::MoveNonSamplerUniformsIntoStruct(program, ids);
         std::map<std::string, std::string> vertexAttributeRenaming = {};

Plugins/ShaderCompiler/Source/ShaderCompilerMetal.cpp

@@ -103,6 +103,8 @@ namespace Babylon::Plugins
         }
 
         ShaderCompilerTraversers::IdGenerator ids{};
+        // Flip 2D texture sample coordinates (replaces the former ProcessSamplerFlip texture() macro).
+        ShaderCompilerTraversers::FlipSamplerCoordinates(program);
         auto cutScope = ShaderCompilerTraversers::ChangeUniformTypes(program, ids);
         auto utstScope = ShaderCompilerTraversers::MoveNonSamplerUniformsIntoStruct(program, ids);
         std::map<std::string, std::string> vertexAttributeRenaming = {};

Plugins/ShaderCompiler/Source/ShaderCompilerTraversers.cpp

@@ -1673,6 +1673,76 @@ namespace Babylon::ShaderCompilerTraversers
 
             TIntermediate* m_intermediate{};
         };
+
+        class FlipSamplerCoordinatesTraverser : public TIntermTraverser
+        {
+        public:
+            static void Traverse(TProgram& program)
+            {
+                for (auto stage : {EShLangVertex, EShLangFragment})
+                {
+                    auto* intermediate{program.getIntermediate(stage)};
+                    if (intermediate != nullptr)
+                    {
+                        FlipSamplerCoordinatesTraverser traverser{intermediate};
+                        intermediate->getTreeRoot()->traverse(&traverser);
+                    }
+                }
+            }
+
+        protected:
+            virtual bool visitAggregate(TVisit visit, TIntermAggregate* node) override
+            {
+                if (visit == EvPreVisit && (node->getOp() == EOpTexture || node->getOp() == EOpTextureLod))
+                {
+                    auto& sequence = node->getSequence();
+                    if (sequence.size() >= 2)
+                    {
+                        // The coordinate is the operand right after the sampler. Only 2-component
+                        // float coordinates (sampler2D-style) are flipped; cube/array/3D coordinates
+                        // (vec3+) are left untouched, matching the original flip(vec2)/flip(vec3).
+                        auto* coordinate = sequence[1]->getAsTyped();
+                        if (coordinate != nullptr &&
+                            coordinate->getType().getBasicType() == EbtFloat &&
+                            !coordinate->getType().isArray() &&
+                            coordinate->getType().getVectorSize() == 2)
+                        {
+                            sequence[1] = FlipVerticalCoordinate(coordinate);
+                        }
+                    }
+                }
+
+                return true;
+            }
+
+        private:
+            FlipSamplerCoordinatesTraverser(TIntermediate* intermediate)
+                : m_intermediate{intermediate}
+            {
+            }
+
+            // Builds `coordinate * vec2(1.0, -1.0) + vec2(0.0, 1.0)`, i.e. (u, 1.0 - v).
+            TIntermTyped* FlipVerticalCoordinate(TIntermTyped* coordinate)
+            {
+                const TSourceLoc& loc{coordinate->getLoc()};
+                TType vec2Type{EbtFloat, EvqConst, 2};
+
+                TConstUnionArray scaleValues{2};
+                scaleValues[0].setDConst(1.0);
+                scaleValues[1].setDConst(-1.0);
+                TIntermTyped* scale{m_intermediate->addConstantUnion(scaleValues, vec2Type, loc)};
+
+                TConstUnionArray offsetValues{2};
+                offsetValues[0].setDConst(0.0);
+                offsetValues[1].setDConst(1.0);
+                TIntermTyped* offset{m_intermediate->addConstantUnion(offsetValues, vec2Type, loc)};
+
+                TIntermTyped* scaled{m_intermediate->addBinaryMath(EOpMul, coordinate, scale, loc)};
+                return m_intermediate->addBinaryMath(EOpAdd, scaled, offset, loc);
+            }
+
+            TIntermediate* m_intermediate{};
+        };
     }
 
     ScopeT MoveNonSamplerUniformsIntoStruct(TProgram& program, IdGenerator& ids)
@@ -1719,4 +1789,9 @@ namespace Babylon::ShaderCompilerTraversers
     {
         InvertYDerivativeOperandsTraverser::Traverse(program);
     }
+
+    void FlipSamplerCoordinates(TProgram& program)
+    {
+        FlipSamplerCoordinatesTraverser::Traverse(program);
+    }
 }

Plugins/ShaderCompiler/Source/ShaderCompilerTraversers.h

@@ -134,4 +134,19 @@ namespace Babylon::ShaderCompilerTraversers
     /// https://github.com/bkaradzic/bgfx/blob/7be225bf490bb1cd231cfb4abf7e617bf35b59cb/src/bgfx_shader.sh#L44-L45
     /// https://github.com/bkaradzic/bgfx/blob/7be225bf490bb1cd231cfb4abf7e617bf35b59cb/src/bgfx_shader.sh#L62-L65
     void InvertYDerivativeOperands(glslang::TProgram& program);
+
+    /// Flip the vertical (V) component of 2D texture sample coordinates, i.e. rewrite the
+    /// coordinate `uv` of every `texture()`/`textureLod()` call to `vec2(uv.x, 1.0 - uv.y)`.
+    ///
+    /// bgfx's D3D/Metal/Vulkan backends sample textures with the opposite V-orientation from
+    /// WebGL/OpenGL. This was historically corrected by injecting a `#define texture(x,y)
+    /// texture(x, flip(y))` preprocessor macro into the shader source, but a 2-argument
+    /// function-like macro cannot match the 3-argument bias form `texture(sampler, uv, bias)`
+    /// that some Babylon.js shaders emit (e.g. GreasedLine), and glslang's preprocessor has no
+    /// variadic-macro support. Performing the flip on the AST handles every texture()/textureLod()
+    /// arity and sampler type. Only 2-component (sampler2D-style) coordinates are flipped, matching
+    /// the previous `flip(vec2)`/`flip(vec3)` overloads where vec3+ coordinates were left untouched.
+    /// Must only be used on the backends that apply ProcessSamplerFlip (D3D, Metal, Vulkan); the
+    /// OpenGL backend shares bgfx's V-orientation and must not flip.
+    void FlipSamplerCoordinates(glslang::TProgram& program);
 }

Plugins/ShaderCompiler/Source/ShaderCompilerVulkan.cpp

@@ -78,6 +78,8 @@ namespace Babylon::Plugins
         }
 
         ShaderCompilerTraversers::IdGenerator ids{};
+        // Flip 2D texture sample coordinates (replaces the former ProcessSamplerFlip texture() macro).
+        ShaderCompilerTraversers::FlipSamplerCoordinates(program);
         auto cutScope = ShaderCompilerTraversers::ChangeUniformTypes(program, ids);
         auto utstScope = ShaderCompilerTraversers::MoveNonSamplerUniformsIntoStruct(program, ids);
         std::map<std::string, std::string> vertexAttributeRenaming = {};