JS binary serializer mishandles non-ASCII metadata fields

[lukaszsamson]

Environment

• Elixir version (elixir -v): all
• Phoenix version (mix deps): 1.8.5
• Operating system: all

Actual behavior

Repro:

1. Clone https://github.com/lukaszsamson/phoenix-serializer-bug-repro
2. Install deps
3. Start server and load the main route
4. Run window.runRoundtrip() in browser console

Result:

Browser:

Understand this error
frame:1829 📡 [info] CONNECTED TO CtfWeb.UserSocket in 14µs  Transport: :websocket  Serializer: Phoenix.Socket.V2.JSONSerializer  Parameters: %{"vsn" => "2.0.0"}
frame:1829 📡 [info] JOINED echo:room:café in 13µs  Parameters: %{}
app.js:63 Uncaught (in promise) Error: push timeout
    at Object.callback (app.js:63:40)
    at push.js:76:23
    at Array.forEach (<anonymous>)
    at Push.matchReceive (push.js:76:8)
    at Object.callback (push.js:107:12)
    at Channel.trigger (channel.js:278:12)
    at Push.trigger (push.js:126:18)
    at push.js:111:12

Server:

[info] CONNECTED TO CtfWeb.UserSocket in 19µs
  Transport: :websocket
  Serializer: Phoenix.Socket.V2.JSONSerializer
  Parameters: %{"vsn" => "2.0.0"}
[info] JOINED echo:room:café in 35µs
  Parameters: %{}
[error] GenServer #PID<0.2030.0> terminating
** (Jason.EncodeError) invalid byte 0xE9 in <<101, 99, 104, 111, 58, 114, 111, 111, 109, 58, 99, 97, 102, 233>>
    (jason 1.4.4) lib/jason.ex:213: Jason.encode_to_iodata!/2
    (phoenix 1.8.5) lib/phoenix/socket/serializers/v2_json_serializer.ex:72: Phoenix.Socket.V2.JSONSerializer.encode!/1
    (phoenix 1.8.5) lib/phoenix/socket.ex:839: Phoenix.Socket.encode_reply/2
    (phoenix 1.8.5) lib/phoenix/socket.ex:803: Phoenix.Socket.handle_in/4
    (bandit 1.10.4) lib/bandit/websocket/connection.ex:79: Bandit.WebSocket.Connection.handle_frame/3
    (bandit 1.10.4) lib/bandit/websocket/handler.ex:50: Bandit.WebSocket.Handler.pop_frame/3
    (bandit 1.10.4) lib/bandit/delegating_handler.ex:18: Bandit.DelegatingHandler.handle_data/3
    (bandit 1.10.4) lib/bandit/delegating_handler.ex:8: Bandit.DelegatingHandler.handle_info/2
    (stdlib 7.2) gen_server.erl:2434: :gen_server.try_handle_info/3
    (stdlib 7.2) gen_server.erl:2420: :gen_server.handle_msg/3
    (stdlib 7.2) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Process Label: {Phoenix.Socket, CtfWeb.UserSocket, nil}
Last message: {:tcp, #Port<0.53>, <<130, 169, 213, 124, 173, 153, 213, 125, 172, 151, 221, 79, 153, 252, 182, 20, 194, 163, 167, 19, 194, 244, 239, 31, 204, 255, 60, 25, 206, 241, 186, 35, 207, 240, 187, 179, 45, 180, 177, 29, 217, 248, 248, 140, 50, 3, 85>>}
State: {%ThousandIsland.Socket{socket: #Port<0.53>, transport_module: ThousandIsland.Transports.TCP, read_timeout: 60000, silent_terminate_on_error: false, span: %ThousandIsland.Telemetry{span_name: :connection, telemetry_span_context: #Reference<0.3970928401.616824834.101520>, start_time: -576460735585235292, start_metadata: %{handler: Bandit.DelegatingHandler, telemetry_span_context: #Reference<0.3970928401.616824834.101520>, remote_port: 49506, remote_address: {127, 0, 0, 1}, parent_telemetry_span_context: #Reference<0.3970928401.616824837.104216>}, handler: Bandit.DelegatingHandler, span_metadata: %{handler: Bandit.DelegatingHandler, telemetry_span_context: #Reference<0.3970928401.616824834.101520>}}}, %{connection: %Bandit.WebSocket.Connection{websock: CtfWeb.UserSocket, websock_state: {%{channels: %{"echo:room:café" => {#PID<0.2038.0>, #Reference<0.3970928401.616824834.101600>, :joined}}, channels_inverse: %{#PID<0.2038.0> => {"echo:room:café", "3"}}}, %Phoenix.Socket{assigns: %{}, channel: nil, channel_pid: nil, endpoint: CtfWeb.Endpoint, handler: CtfWeb.UserSocket, id: nil, joined: false, join_ref: nil, private: %{}, pubsub_server: Ctf.PubSub, ref: nil, serializer: Phoenix.Socket.V2.JSONSerializer, topic: nil, transport: :websocket, transport_pid: #PID<0.2030.0>}}, state: :open, compress: nil, opts: [compress: nil, connect_info: [], path: "/websocket", serializer: [{Phoenix.Socket.V1.JSONSerializer, "~> 1.0.0"}, {Phoenix.Socket.V2.JSONSerializer, "~> 2.0.0"}], error_handler: {Phoenix.Transports.WebSocket, :handle_error, []}, timeout: 60000, transport_log: false, auth_token: nil], fragment_frame: nil, span: %Bandit.Telemetry{span_name: :websocket, telemetry_span_context: #Reference<0.3970928401.616824838.102521>, start_time: -576460735521315875, start_metadata: %{websock: CtfWeb.UserSocket, telemetry_span_context: #Reference<0.3970928401.616824838.102521>, connection_telemetry_span_context: #Reference<0.3970928401.616824834.101520>}}, metrics: %{send_text_frame_bytes: 69, send_text_frame_count: 1, recv_text_frame_bytes: 41, recv_text_frame_count: 1}}, handler_module: Bandit.WebSocket.Handler, extractor: %Bandit.Extractor{header: "", payload: [], payload_length: 0, required_length: 0, mode: :header_parsing, max_frame_size: 0, frame_parser: Bandit.WebSocket.Frame, primitive_ops_module: Bandit.PrimitiveOps.WebSocket}}}
[error] GenServer #PID<0.2038.0> terminating
** (Jason.EncodeError) invalid byte 0xE9 in <<101, 99, 104, 111, 58, 114, 111, 111, 109, 58, 99, 97, 102, 233>>
    (jason 1.4.4) lib/jason.ex:213: Jason.encode_to_iodata!/2
    (phoenix 1.8.5) lib/phoenix/socket/serializers/v2_json_serializer.ex:72: Phoenix.Socket.V2.JSONSerializer.encode!/1
    (phoenix 1.8.5) lib/phoenix/socket.ex:839: Phoenix.Socket.encode_reply/2
    (phoenix 1.8.5) lib/phoenix/socket.ex:803: Phoenix.Socket.handle_in/4
    (bandit 1.10.4) lib/bandit/websocket/connection.ex:79: Bandit.WebSocket.Connection.handle_frame/3
    (bandit 1.10.4) lib/bandit/websocket/handler.ex:50: Bandit.WebSocket.Handler.pop_frame/3
    (bandit 1.10.4) lib/bandit/delegating_handler.ex:18: Bandit.DelegatingHandler.handle_data/3
    (bandit 1.10.4) lib/bandit/delegating_handler.ex:8: Bandit.DelegatingHandler.handle_info/2
    (stdlib 7.2) gen_server.erl:2434: :gen_server.try_handle_info/3
    (stdlib 7.2) gen_server.erl:2420: :gen_server.handle_msg/3
    (stdlib 7.2) proc_lib.erl:333: :proc_lib.init_p_do_apply/3
Process Label: {Phoenix.Channel, CtfWeb.EchoChannel, "echo:room:café"}
Last message: {:DOWN, #Reference<0.3970928401.616824838.102530>, :process, #PID<0.2030.0>, {%Jason.EncodeError{message: "invalid byte 0xE9 in <<101, 99, 104, 111, 58, 114, 111, 111, 109, 58, 99, 97, 102, 233>>"}, [{Jason, :encode_to_iodata!, 2, [file: ~c"lib/jason.ex", line: 213, error_info: %{module: Exception}]}, {Phoenix.Socket.V2.JSONSerializer, :encode!, 1, [file: ~c"lib/phoenix/socket/serializers/v2_json_serializer.ex", line: 72]}, {Phoenix.Socket, :encode_reply, 2, [file: ~c"lib/phoenix/socket.ex", line: 839]}, {Phoenix.Socket, :handle_in, 4, [file: ~c"lib/phoenix/socket.ex", line: 803]}, {Bandit.WebSocket.Connection, :handle_frame, 3, [file: ~c"lib/bandit/websocket/connection.ex", line: 79]}, {Bandit.WebSocket.Handler, :pop_frame, 3, [file: ~c"lib/bandit/websocket/handler.ex", line: 50]}, {Bandit.DelegatingHandler, :handle_data, 3, [file: ~c"lib/bandit/delegating_handler.ex", line: 18]}, {Bandit.DelegatingHandler, :handle_info, 2, [file: ~c"lib/bandit/delegating_handler.ex", line: 8]}, {:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 2434]}, {:gen_server, :handle_msg, 3, [file: ~c"gen_server.erl", line: 2420]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 333]}]}}
State: %Phoenix.Socket{assigns: %{}, channel: CtfWeb.EchoChannel, channel_pid: #PID<0.2038.0>, endpoint: CtfWeb.Endpoint, handler: CtfWeb.UserSocket, id: nil, joined: true, join_ref: "3", private: %{log_join: :info, log_handle_in: :debug}, pubsub_server: Ctf.PubSub, ref: nil, serializer: Phoenix.Socket.V2.JSONSerializer, topic: "echo:room:café", transport: :websocket, transport_pid: #PID<0.2030.0>}

The repro constructs message with non ASCII characters in metadata. The serializer fails to encode it correctly and the server errors on invalid message so the client never receives response.

Reason:

phoenix/assets/js/phoenix/serializer.js

Lines 31 to 53 in 6ceade4

	binaryEncode(message){
	let {join_ref, ref, event, topic, payload} = message
	let metaLength = this.META_LENGTH + join_ref.length + ref.length + topic.length + event.length
	let header = new ArrayBuffer(this.HEADER_LENGTH + metaLength)
	let view = new DataView(header)
	let offset = 0
	view.setUint8(offset++, this.KINDS.push) // kind
	view.setUint8(offset++, join_ref.length)
	view.setUint8(offset++, ref.length)
	view.setUint8(offset++, topic.length)
	view.setUint8(offset++, event.length)
	Array.from(join_ref, char => view.setUint8(offset++, char.charCodeAt(0)))
	Array.from(ref, char => view.setUint8(offset++, char.charCodeAt(0)))
	Array.from(topic, char => view.setUint8(offset++, char.charCodeAt(0)))
	Array.from(event, char => view.setUint8(offset++, char.charCodeAt(0)))
	var combined = new Uint8Array(header.byteLength + payload.byteLength)
	combined.set(new Uint8Array(header), 0)
	combined.set(new Uint8Array(payload), header.byteLength)
	return combined.buffer
	},

phoenix/lib/phoenix/socket/serializers/v2_json_serializer.ex

Lines 47 to 58 in 6ceade4

	bin = <<
	@reply::size(8),
	join_ref_size::size(8),
	ref_size::size(8),
	topic_size::size(8),
	status_size::size(8),
	join_ref::binary-size(join_ref_size),
	ref::binary-size(ref_size),
	reply.topic::binary-size(topic_size),
	status::binary-size(status_size),
	data::binary
	>>

join_ref.length and others are JS string (UTF-16 code unit) lengths, not UTF-8 byte lengths. Then on the server side byte_size of the decoded binary is incorrect. char.charCodeAt(0) returns the UTF-16 code unit of the first character, not the UTF-8 bytes. This means any non ASCII char gets mangled. Server decoder reads join_ref::binary-size(join_ref_size) using the first-byte length field, then interprets it as UTF-8 text.

Expected behavior

No timeout, correct serialization, roundtrip working