[Security] Default maxFrameSize=Integer.MAX_VALUE -> declared-length pre-allocation -> OOM whole-server crash (Draft_6455.java:557)

[afldl]

Description

Summary

Java-WebSocket's default server (Draft_6455) can be crashed process-wide by
any unauthenticated peer with a single 10-14 byte frame: the decoder
allocates up to ~2 GiB from the declared frame length
(ByteBuffer.allocate(checkAlloc(payloadlength)), Draft_6455.java:557)
because the default maxFrameSize is Integer.MAX_VALUE (:210) and
checkAlloc (Draft.java:322) only rejects negative counts. OutOfMemoryError
is an Error (not caught by the exception handlers) whole-server crash.
Pre-auth.

Root cause

Draft_6455.java:

public Draft_6455() { this(inputExtensions, inputProtocols, Integer.MAX_VALUE); }  // :210  default no cap
...
ByteBuffer payload = ByteBuffer.allocate(checkAlloc(payloadlength));               // :557

Draft.java:322:

public int checkAlloc(int bytecount) {
    if (bytecount < 0) throw new InvalidDataException(...);
    return bytecount;       // only rejects negative; 1GB passes
}

PoC (end-to-end)

Attachment: gist https://gist.github.com/afldl/bed08861681eb87b4254c08bc7c5b0e7

File	Purpose
Harness.java	Reproducer: declared payload 1023MB through checkAlloc
output.txt	Real run output: OOM Error

Reproduction:

javac Harness.java && java Harness

Expected output (real):

attacker declares payload length = 1073741823 (1023 MB)
maxFrameSize check: 1073741823 <= 2147483647   PASSES
OutOfMemoryError: allocation exhausted heap   whole-server crash (Error not caught)

Suggested fix

Default maxFrameSize to a sane value (e.g. 65536, like other libraries) and
have checkAlloc enforce it (or make the allocation lazy/streamed).

Credit

Reported by afldl, 2026-07.