Fix GL pixel shader debug failure with gl_FrontFacing

[fengT-T]

Description

Pixel shader debug fails on any fragment shader that uses gl_FrontFacing in a
condition, like:

vec3 N = gl_FrontFacing ? normalize(v_view_normal) : -normalize(v_view_normal);                                                  

Turns out CreateInputFetcher() injects a macro:

  #define gl_FrontFacing (gl_FrontFacing ? 1u : 0u)                                                                                

which was meant to convert the built-in bool to uint for the SSBO output.
But it's global, so it also changes every use of gl_FrontFacing in the user's
own shader code. The ternary above becomes:

  (gl_FrontFacing ? 1u : 0u) ? normalize(...) : -normalize(...)                                                                    

GLSL needs a bool condition there, not uint, so the replacement shader fails
to compile.

Fix

Two lines:

1. Drop the macro. gl_FrontFacing stays as the native bool built-in.
2. Convert explicitly at the only spot that actually needs uint:
   isFrontFace = gl_FrontFacing ? 1u : 0u;

The inputs.gl_FrontFacing assignment is generated from GL reflection data
which already gets the type right — no conversion needed there.

Tested on

AMD RX 9070, RadeonSI driver, Linux, OpenGL 3.3 core.

[baldurk]

This doesn't make sense to me, can you please explain in more detail exactly what's going wrong? you briefly say "But it's global" about the macro but that's not how macros work, and the input fetcher shader is entirely generated and does not include user shader code.

How is that macro affecting any other shaders?

[fengT-T]

I’m sorry about the confusion earlier — I didn’t explain things clearly. I’m still pretty new to the RenderDoc codebase, so my understanding of the terminology and how everything flows is still rough. I just found that this change fixed my issue.

Here’s what caused it:

I have a fragment shader that uses gl_FrontFacing in a ternary:

vec3 N = gl_FrontFacing ? normalize(v_view_normal) : -normalize(v_view_normal);

Standard stuff, renders fine. But when I click the "Debug" button for pixel shader debugging, it always fails with "no write to the pixel at the current event." If I don’t use gl_FrontFacing at all, everything works normally.

Searching around for gl_FrontFacing, I found this macro:

#define gl_FrontFacing (gl_FrontFacing ? 1u : 0u)

After a quick look at how RenderDoc works, I guessed that this macro is causing a type mismatch that breaks debugging. So I tried changing it, and after the modification, I was able to debug shaders that use gl_FrontFacing without any problem.

[baldurk]

If you are making the change semi-blind and you don't understand how the code works I think it's best that you file an issue with a bug report, and include a full and reproducible test case. Then I can investigate and ensure the correct fix is made.