[fix] PSS auto salt-length verify failed due leading zeros

RSABlindedEngine.processBlock() converts the raw modular-exponentiation
result back through BigInteger, which strips leading zero bytes.

When the encoded message (EM) happened to start with 0x00 — roughly a
1-in-256 chance per signature — the returned array was shorter than the
expected emLen = ceil((modBits-1)/8).  The hand-rolled PSS parser in
pssAutoSaltLength() used the array length as emLen, which shifted every
subsequent offset (maskedDB, H, the 0x01 separator), causing the
salt-length recovery to fail and verify_pss to return false.

Add a deterministic regression test with a hardcoded signature whose EM
is known to have a leading 0x00 byte for the rsa2048 test fixture key.

Author: kares

src/main/java/org/jruby/ext/openssl/PKeyRSA.java

@@ -1095,10 +1095,22 @@ private static int pssAutoSaltLength(RSAPublicKey pubKey, byte[] sigBytes, Strin
         RSAKeyParameters bcPubKey = new RSAKeyParameters(false, pubKey.getModulus(), pubKey.getPublicExponent());
         RSABlindedEngine rsa = new RSABlindedEngine();
         rsa.init(false, bcPubKey);
-        byte[] em = rsa.processBlock(sigBytes, 0, sigBytes.length);
+        byte[] raw = rsa.processBlock(sigBytes, 0, sigBytes.length);
+
+        // emLen = ceil((modBits - 1) / 8) per RFC 8017 §9.1.1
+        int emLen = (pubKey.getModulus().bitLength() - 1 + 7) / 8;
+
+        // RSA raw output may be shorter than emLen if leading bytes are zero;
+        // left-pad with zeros to the expected length.
+        byte[] em;
+        if (raw.length < emLen) {
+            em = new byte[emLen];
+            System.arraycopy(raw, 0, em, emLen - raw.length, raw.length);
+        } else {
+            em = raw;
+        }
 
         int hLen  = getDigestLength(digestAlg);
-        int emLen = em.length;
         if (emLen < hLen + 2 || em[emLen - 1] != (byte) 0xBC) return -1;
 
         int dbLen = emLen - hLen - 1;

src/test/ruby/rsa/test_rsa.rb

@@ -288,6 +288,29 @@ def test_sign_verify_pss
     }
   end
 
+  # Regression test: verify_pss with salt_length: :auto must handle RSA raw
+  # output shorter than emLen (leading zero bytes stripped by BigInteger).
+  # The hardcoded signature below was produced by sign_pss with the rsa2048
+  # fixture key and salt_length: :digest; its RSA public-key recovery yields
+  # an encoded message (EM) with a leading 0x00 byte (255 bytes instead of
+  # the expected emLen=256), which triggers the leading-zero edge case.
+  def test_sign_verify_pss_auto_salt_leading_zero
+    key = Fixtures.pkey("rsa2048")
+    data = "test-data-6"
+    signature = Base64.decode64(
+      "X0U4VKxjVFN6sXKJaZdpGKOwSPo9L1VNbKyTqJ5P7nNDOz8flimVFJwe6H969zYM" \
+      "xdbP2hLVLHtgzLeQd6N85DiYLDqObUIAhm94kMhGNh2bc3mutYTU96huKbL8YfhG/" \
+      "+1SxAJUaMBQqRQ7UIaPJHuShD5rQ9SHRRASNbWpzRGtMzRgMorykF2+nAnMlbQxxNW" \
+      "s7Ia1rsR9OGUS/Q9rQqUuB5i9IR513Xf/O39AxgiaoEehdYAdmbh1W/hQqGGnb8Xt" \
+      "fTdQlsbiyuMvTnbNqIlwVNh4FU2AZr9u4DHd3zQt6csYI2UPcE00vv3e0RWY4C4EA" \
+      "TGVk92gZ59VEVeJEQ=="
+    )
+    assert_equal true,
+                 key.verify_pss("SHA256", signature, data, salt_length: 32, mgf1_hash: "SHA256")
+    assert_equal true,
+                 key.verify_pss("SHA256", signature, data, salt_length: :auto, mgf1_hash: "SHA256")
+  end
+
   def test_rsa_param_accessors
     key_file = File.join(File.dirname(__FILE__), 'private_key.pem')
     key = OpenSSL::PKey::RSA.new(File.read(key_file))