Unexpected Y-axis translation when using `ortho()` and `resetMatrix()`

[SableRaf]

Note

This issue is a repost of processing/processing#6175 by @usuallyannoyed, updated and edited for tone and clarity.

Description

The ortho() function does not behave as expected.

Expected Behavior

ortho(0, width, 0, height) with a cleared modelview matrix (resetMatrix) should allow users to draw within the specified bounds (e.g., 0 to width on the x-axis and 0 to height on the y-axis).

Current Behavior

Instead, the drawing area behaves incorrectly, requiring adjustments (e.g., translating -height on the y-axis) to achieve the expected results. This behavior makes it unintuitive for users who expect the specified bounds to align with the visible viewport.

Steps to Reproduce

void setup() {
  size(500, 500, P3D);
}

void draw() {
  ortho(0, width, 0, height);
  resetMatrix(); // clear modelview matrix
  fill(255);
  
  // Uncomment the line below to 'fix' the behavior
  // translate(0, -height); 
  
  beginShape(TRIANGLES);
  vertex(640, 640);
  vertex(0, 640);
  vertex(0, 0);
  endShape();
}

Observed Result

The triangle only becomes visible when compensating for the unexpected offset by translating the y-axis.

Expected Result

The triangle should appear within the specified ortho bounds without additional transformations.

Environment

• Operating System: macOS 13.4
• Processing Version: 4.3.1

Possible Causes / Solutions

The issue appears to stem from the following block of code in the Processing source:

processing4/core/src/processing/opengl/PGraphicsOpenGL.java

Lines 4482 to 4486 in 937f528

	// The minus sign is needed to invert the Y axis.
	projection.set(x, 0, 0, tx,
	0, -y, 0, ty,
	0, 0, z, tz,
	0, 0, 0, 1);

Despite the comment saying that The minus sign is needed to invert the Y axis., this implementation does not fully invert the y-axis as intended. Instead, it flips the y-axis around zero, resulting in all the y-coordinates being offset into negative space.

Suggested Fix

To invert the y-axis properly in normalized device coordinates (NDC), pre-multiply the projection matrix with a -1 scale for the y-axis. This can be achieved by modifying the projection matrix calculation as follows:

// Add a pre-multiplied -1 y scale to flip the y axis in NDC.
    projection.set(x,  0, 0, tx,
                   0, -y, 0, -ty,
                   0,  0, z, tz,
                   0,  0, 0,  1);

This approach ensures the y-axis behaves as expected without requiring additional transformations.

Additional comments

This behavior stems from Processing's choice to have the Y-axis increase downward, consistent with traditional computer graphics. While reasonable, the implementation relies on ad-hoc adjustments throughout the code. Centralizing it at a lower level (NDC or viewport) and exposing it as an optional flag would provide more clarity and flexibility.

[SableRaf]

Reopening this as the original fix was reverted in c83f44c

The suggested fix (implemented in #954) fixed this particular issue but caused unexpected behavior in almost every shader example.

[IIITM-Jay]

Hi @SableRaf , while giving some thoughts to this issue, I have few ideas and approaches to share and discuss here first.

1. Using Shader Uniform

To ensure that shaders handle the Y-axis orientation correctly, we could declare a uniform boolean and then applying in vertex shader, followed by setting from Processing, something like-

uniform bool uFlipY;


if (uFlipY) {
    pos.y = -pos.y;
}
gl_Position = pos;

// In `PShader`
shader.set("uFlipY", true);

2. Using Internal Flag for Y-Axis Inversion

We could declare a boolean something like - boolean flipYAxis = false; // Default: no flip and then using in ortho() to behave accordingly:

float yScale = flipYAxis ? -y : y;
float yTranslate = flipYAxis ? -ty : ty;

projection.set(x,  0, 0, tx,
               0, yScale, 0, yTranslate,
               0, 0, z, tz,
               0, 0, 0, 1);

and then a user can simply use like-

g.flipYAxis = true;
ortho(0, width, 0, height);
resetMatrix();

3. Adding a whole new method to handle this

We could add a new method next to existing ortho(...) method -

public void orthoTopDown(float left, float right, float bottom, float top) {
  float x = 2.0f / (right - left);
  float y = -2.0f / (top - bottom);     // Flipping y here (as compared to existing method)
  float z = -2.0f / (far - near);
  float tx, ty, tz = (all as-is with no change)
 

  projection.set(x, 0, 0, tx,
                 0, y, 0, ty,
                 0, 0, z, tz,
                 0, 0, 0, 1);

  updateProjmodelview();
}

Let me know your views on above.

[IIITM-Jay]

Hi @SableRaf and @Stefterv , after giving some more time and observations to this, I think we should use both the Approach 2 which is of using internal flag and Approach 1 using Shader Uniform. Because, ortho() only affects the projection matrix — not the programmable shader pipeline. When we modify the Y-axis via the projection matrix (Approach 2), we're flipping coordinates at the CPU/OpenGL matrix level. But if we use shaders (which bypass fixed-function pipeline behaviors), then gl_Position is computed in the shader, not by ortho(). As for custom vertex shader, we're explicitly telling the GPU how to calculate the final screen-space position (gl_Position) of each vertex. So ortho() flips won't affect what the vertex shader does unless we replicate the logic.

In my opinion, for 2D/3D sketches with default P3D renderer, Only Approach 2 (flipYAxis in ortho()) will suffice, but for Custom shaders with PShader, we must also use Approach 1 (pass uFlipY to shader) for full support.

What you folks think of 🤔 ...

[IIITM-Jay]

Hi @SableRaf and @Stefterv, If you don't mind, may I request for suggestions and guidance on above to proceed ahead.

Thanks once again!

[IIITM-Jay]

Hi @SableRaf let me know among the methods that I have discussed above, which one would be the best suitable approach to fix this or some other concerns I should be taking care of and keep in mind to check upon before working on this. I would love to discuss feedback as well as other solutions that you think of if any.

[IIITM-Jay]

Hi @SableRaf and @Stefterv am I thinking in right direction as discussed in above comments:

I think we should use both the Approach 2 which is of using internal flag and Approach 1 using Shader Uniform. Because, ortho() only affects the projection matrix — not the programmable shader pipeline. When we modify the Y-axis via the projection matrix (Approach 2), we're flipping coordinates at the CPU/OpenGL matrix level.

I would be more than happy to raise a PR if you folks think that I am approaching in a correct way. Will revise the PR as per the feedback and changes as required.
I can understand the life of maintainers and the busy schedule of them and how much requests to handle ... just like the one here by me! 😅 Pardon me for this.

Thanks!

[Stefterv]

Hi @IIITM-Jay, thanks so much for your thorough work and patience!

After merging the earlier PR, we realized that—even though the current behavior is technically incorrect—many things have already been built on top of it. We’d like to fix it properly in the future, but don’t currently have the bandwidth to approach this with the care it needs.

Really appreciate your effort here, and we’ll definitely revisit your suggestions when we’re ready to address it fully.