Problem

The Pro Node Renderer currently communicates over HTTP/2, but parts of the request protocol still use encodings that may add avoidable escaping/parsing overhead.

The main hypothesis is not that binary formats magically shrink all payloads. The useful hypothesis is narrower:

Non-streaming render requests currently send the renderingRequest JavaScript source string through application/x-www-form-urlencoded, which percent-encodes and decodes the whole source string. A raw-body request, JSON envelope, or MessagePack envelope may reduce escaping/unescaping overhead while preserving the current HTTP/2 transport and renderer protocol.

This issue should evaluate that hypothesis before considering deeper protocol changes.

Relationship to sibling investigations

This is one of three issues that micro-optimize the same render-request hot path and are gated on the same question — is anything other than React SSR a material cost?: #3582 (replace Fastify), #3583 (the HTTP/2-over-Unix-socket transport issue), and this one (wire encoding). Rather than each independently building a profiler and likely reaching the same "SSR dominates" conclusion, a single shared Phase 0 profiling spike should produce the go/no-go for all three. See "Benchmark Plan → Phase 0". If that spike shows transport + encoding combined are below the materiality threshold, all three can close together.

Affected package areas:

• Ruby request construction: react_on_rails_pro/lib/react_on_rails_pro/request.rb
• Ruby HTTP body construction: react_on_rails_pro/lib/react_on_rails_pro/renderer_http_client.rb
• Node renderer body parsing: packages/react-on-rails-pro-node-renderer/src/worker.ts
• Incremental stream parser: packages/react-on-rails-pro-node-renderer/src/worker/handleIncrementalRenderStream.ts

Current Protocol Shape

Unary render request

The common hot path is:

1. Rails builds a renderingRequest string. This is executable JavaScript source.
2. Rails sends renderingRequest plus protocol/auth fields in a form body.
3. If no bundle file is included, the body is application/x-www-form-urlencoded.
4. On the Node side, Fastify parses the form body and the renderer executes renderingRequest in the VM.

Relevant code:

• Request#form_with_code adds renderingRequest and only attaches bundle/assets when send_bundle is true.
• RendererHttpClient.build_form_body uses urlencoded form encoding unless any field is a file.
• Bundle upload is rare: normal render requests do not include the bundle; cache misses trigger a 410 retry with bundle upload, and PreSeedRendererCache can avoid that cold path at deploy time.

Incremental render request

Incremental render uses bidirectional HTTP/2 streaming:

1. Rails opens a writable HTTP/2 request body with Content-Type: application/x-ndjson.
2. Rails writes the initial render request as one JSON line.
3. Rails writes async prop updates as additional JSON lines.
4. Node parses each line with JSON.parse.
5. Node streams HTML chunks back on the HTTP response.

Relevant code:

• Request#render_code_with_incremental_updates writes initial_data.to_json.
• AsyncPropsEmitter writes update chunks to the same request stream.
• handleIncrementalRenderStream.ts parses NDJSON lines.

JSON request body already exists for smaller endpoints

/asset-exists already uses an application/json request body, so the Ruby client already has a JSON body path. This is useful for a small compatibility spike, but /asset-exists is not the main render hot path.

Important Distinction: Outer vs Inner Escaping

There are two escaping layers:

1. Outer transport escaping

This is the layer this issue can improve with a relatively small change.

Current unary render requests put the entire renderingRequest source string inside an urlencoded form field. That means Ruby percent-encodes the string and Node/Fastify parses/decodes it before the renderer can use it.

Sending the JS source as a raw body removes this layer entirely (verbatim bytes); a JSON or MessagePack envelope reduces it (JSON still escapes quotes/backslashes/control chars, MessagePack does not). Raw body is therefore the cleanest fix for this layer — see Candidate 1.

2. Inner JavaScript/props escaping

This is not fixed by MessagePack or JSON as an outer envelope.

The renderingRequest itself is still executable JavaScript source. If props/state are serialized and escaped inside that JS string, then a binary outer envelope does not remove that inner serialization/escaping cost.

Removing the inner layer would require a deeper protocol redesign where Rails sends structured render data, such as:

{
  "componentName": "App",
  "props": { "...": "..." },
  "renderOptions": { "...": "..." }
}

and Node builds/invokes the render call instead of receiving arbitrary JS source. That is a larger, more breaking project and should not be part of the first encoding spike.

Candidate Changes

Note on ordering: renderingRequest is one dominant string field; the other fields (auth, protocol version, dependencyBundleTimestamps[]) are tiny. So the most effective outer-encoding change is the one that stops escaping that string at all. Raw body does this with zero new dependencies and is therefore both the performance upper bound and the simplest change — test it first as the discriminator (see Benchmark Plan).

1. Unary render: raw body + metadata in headers (recommended; also the discriminator)

Send the renderingRequest JS source as the raw request body (e.g. Content-Type: application/javascript) and move the small protocol/auth fields to request headers. This includes protocolVersion, gem/package version metadata, authentication fields, and dependency bundle timestamps. The render route already reads bundleTimestamp / renderRequestDigest from the URL path, so reading the remaining small metadata fields from headers is a natural extension.

Possible behavior:

• When no files are attached, send the JS source verbatim as the body; metadata in headers.
• Preserve multipart only for bundle/cache-miss uploads.

Pros:

• Removes all outer escaping of the dominant string — verbatim bytes on the wire (strictly better than JSON, which still escapes quotes/backslashes/control chars in the JS source).
• No new dependency; trivially debuggable (it's just the JS source).
• Equivalent to MessagePack on the dominant string, without a binary dependency — which makes MessagePack largely redundant for this payload.
• Cleanest experimental control: if raw body is not materially faster than urlencoded form, outer encoding genuinely does not matter and the whole investigation can stop with confidence.

Cons:

• Metadata moves to headers (HPACK-compressed under HTTP/2, so small fields are cheap); array fields like dependencyBundleTimestamps[] need a header-friendly representation.
• Node handler must read those fields from headers instead of request.body.
• Needs the same new-vs-old compatibility handling as the JSON option.
• Likely shares the existing content-length quirk: the buffered-upload path deliberately omits Content-Length because setting it caused Fastify HTTP/2 stream resets in testing (renderer_http_client.rb:249-256). A raw body is also buffered, so the spike should follow the same content-length handling — verify whether the quirk still applies to a raw-body request (and whether it persists if Fastify is later replaced per Investigate replacing Fastify in the Pro Node Renderer #3582) rather than assuming either way.

Header representation options to test for dependencyBundleTimestamps:

• Repeated headers, e.g. x-ror-dependency-bundle-timestamp: <hash> once per dependency. This is HTTP-native for repeated values, but Ruby/Node library behavior around repeated request headers must be verified.
• A single JSON metadata header, e.g. x-ror-renderer-metadata: {"dependencyBundleTimestamps":["..."]}. This is easier to parse consistently and can carry future small metadata, but it reintroduces JSON for metadata. That is probably fine because metadata is tiny and not the dominant escaped string.

2. Unary render: urlencoded form to JSON envelope

A middle option if raw body is impractical for some field-shape reason.

Possible behavior:

• When no files are attached, send unary render requests as application/json.
• Preserve multipart only for bundle/cache-miss uploads.
• Keep the same envelope fields: renderingRequest, auth fields, protocol version, dependency bundle timestamps, etc.
• Node already has Fastify's default JSON parser, so parsing support likely exists without new dependencies.

Pros:

• Removes urlencoded percent-encoding of the renderingRequest string.
• No new binary-format dependency.
• Easy to A/B benchmark against current form encoding.
• Lower implementation complexity than MessagePack.
• Easier debugging than binary payloads.

Cons:

• JSON still escapes quotes, backslashes, and control characters in the JavaScript source string.
• Does not remove inner props/JS escaping inside renderingRequest.
• Requires compatibility handling for older Node renderers if Rails can be upgraded independently.

3. Unary render: MessagePack envelope

Worth testing only if raw body / JSON show outer encoding is material and binary metadata framing is suspected to matter. MessagePack avoids escaping the JS string by length-prefixing raw bytes — which is the same property raw body (candidate 1) provides for free. So for this payload MessagePack's only edge over raw body is compacter encoding of the small metadata, which is unlikely to be worth a Ruby + Node dependency.

Possible behavior:

• Add application/msgpack render request bodies.
• Encode the same envelope currently sent in the form/JSON body.
• Keep multipart for bundle/cache-miss uploads.

Pros:

• Avoids most string escaping at the transport envelope layer.
• Can reduce payload size for object-heavy metadata and update chunks.
• Dynamic object shape fits this protocol better than Protobuf.

Cons:

• Adds Ruby and Node dependencies.
• Binary payloads are harder to inspect/debug.
• Does not remove inner JS/props escaping inside renderingRequest.
• Benefit may be small if VM/React SSR dominates.
• Compatibility/version negotiation is required.

4. Incremental render: NDJSON to MessagePack stream

Possible follow-up after unary render measurements.

Possible behavior:

• Add a new content type, e.g. application/x-msgpack-stream.
• Frame each message with a length prefix followed by MessagePack bytes.
• Preserve HTTP/2 bidirectional streaming and response behavior.

Pros:

• Removes JSON stringify/parse and newline framing for async prop update chunks.
• Avoids malformed-newline edge cases.
• May help if many async prop updates carry large structured data.

Cons:

• Requires a new streaming parser on Node and encoder/framer on Ruby.
• Needs careful handling for partial frames, malformed frames, backpressure, and cancellation.
• Harder to debug than NDJSON.
• Does not help ordinary unary render unless that path also changes.

5. Protobuf/BSON

Not recommended as the first spike.

Protobuf is attractive only if we redesign the renderer protocol into stable typed messages. The current payload has arbitrary props and a large executable JS string, so Protobuf would likely become a typed envelope around untyped string/bytes fields, losing most of its value while adding schema management.

BSON is also not a natural fit unless we specifically need MongoDB-like types. It adds type surface without clearly solving the renderer's main escaping issue better than MessagePack.

Compatibility Strategy

Keep existing encodings as the default until benchmarked and proven.

Suggested rollout path:

1. Add a Node renderer parser path for raw unary render bodies, e.g. Content-Type: application/javascript, where the body is the renderingRequest string and metadata comes from headers.
2. Add a Rails-side opt-in to send unary render requests as raw body when no files are attached.
3. Keep urlencoded/multipart behavior available for compatibility.
4. If raw body is beneficial, consider making raw body the default in a minor or major release depending on renderer compatibility constraints.
5. Evaluate JSON only if raw body has compatibility or field-shape problems.
6. Only add MessagePack after raw-body-vs-form data shows outer escaping/parsing is material and raw body is not sufficient.

Compatibility expectation: users are expected to run matching React on Rails Pro gem and react-on-rails-pro-node-renderer package versions. This change can still be implemented as a backward-compatible protocol extension by keeping old parsers in the new renderer and using protocolVersion / renderer capability checks to avoid sending the new encoding to an older renderer.

Field-shape note: the render handler currently reads request.body populated by @fastify/formbody / @fastify/multipart with attachFieldsToBody: 'keyValues'. Both raw-body (fields in headers) and JSON (fields as a parsed object) change where and how fields arrive — notably array fields like dependencyBundleTimestamps[], which are repeated name[]=... pairs in form encoding but a real array (JSON) or a header-friendly encoding (raw-body). The handler must accept the new shape for every field, not just renderingRequest.

Backward compatibility matrix to verify:

• New Rails client to old Fastify-era Node renderer.
• Old Rails client to new Node renderer.
• New Rails client to new Node renderer.
• protocolVersion is sent and validated correctly in both old and new encodings.
• Matching gem/npm renderer versions use the intended encoding.
• Mismatched versions either fall back safely or fail with the existing clear protocol/version mismatch error.
• All envelope fields (incl. array fields) parse correctly in the new shape.
• Cache-miss bundle upload still uses multipart and still works.
• Incremental render remains NDJSON unless explicitly changed.

Benchmark Plan

Phase 0: measure current cost (shared with #3582 and #3583)

Before adding formats, profile where request time and CPU go:

• Ruby body construction and escaping.
• Node body parsing.
• VM execution.
• React SSR.
• Response streaming.

This is the same profiling step the sibling issues need; run it once and share the result. Stop if outer request encoding is not material — use the same threshold as #3583 (continue only if transport/connect/body I/O, here encoding + parsing combined, accounts for roughly >=5-10% of request wall time or CPU).

Phase 1: raw-body vs form (the discriminator), then JSON

Test raw body first — it removes all outer escaping of the JS string and is therefore the upper bound on achievable gain:

1. Current unary render application/x-www-form-urlencoded.
2. Unary render with the JS source as a raw body, metadata in headers.
3. Unary render application/json envelope.

Decision logic:

• If raw body is not materially better than form → outer encoding does not matter → stop (skip JSON and MessagePack).
• If raw body wins → ship raw body (simplest, no deps). Use JSON only if a field-shape constraint makes raw body impractical.

Use scenarios:

• Small props.
• Large string-heavy props.
• Quote/backslash-heavy props.
• Unicode-heavy props.
• Large but realistic renderingRequest strings.
• Many small render requests.

Measure:

• Ruby encode time.
• Ruby allocated objects/bytes.
• Wire payload size.
• Node parse time.
• Node allocated memory/RSS.
• End-to-end p50/p95/p99 latency.

Phase 2: MessagePack only if justified

Compare:

1. Current urlencoded form.
2. Raw body.
3. JSON envelope.
4. MessagePack envelope.

Use the same unary render scenarios.

Only continue to incremental MessagePack streaming if MessagePack materially beats JSON for the envelope or if incremental NDJSON parse/stringify cost is independently proven significant.

Acceptance Criteria

• Establish whether urlencoded outer escaping/parsing is a meaningful cost in real render requests.
• Determine whether raw-body unary render requests provide a material improvement over urlencoded form.
• Determine whether application/json unary render bodies are useful only as a fallback if raw body is impractical.
• Determine whether MessagePack is worth the additional dependency and debugging cost after raw body is measured.
• Preserve multipart bundle upload behavior.
• Preserve current HTTP/2 transport.
• Preserve existing renderer protocol semantics and error behavior.
• Avoid a structured render-call protocol redesign unless measurements show the inner JS/props serialization layer is the real bottleneck.

Open Questions

• How much time and allocation does URI.encode_www_form add for realistic renderingRequest strings?
• How much time and allocation does Fastify form parsing add compared with Fastify JSON parsing?
• Does JSON escaping of JavaScript source strings still dominate after removing percent-encoding?
• Are quote/backslash-heavy props common enough to matter?
• Can the Rails client safely prefer raw-body render requests while maintaining compatibility with older Node renderer packages?
• Is JSON needed only as a compatibility fallback if raw body has field-shape or parser issues?
• Should content negotiation be explicit through protocolVersion, content type, or a renderer capability check, given that matching gem/npm renderer versions are the expected deployment shape?
• Is incremental NDJSON parse/stringify cost material, or is it lost in async prop computation and React streaming?

Current Recommendation

1. Run the shared Phase 0 profiling spike (with Investigate replacing Fastify in the Pro Node Renderer #3582 and Evaluate HTTP/2-over-Unix-socket for same-host Rails↔Node Renderer #3583). If outer encoding + parsing is not material, stop.
2. If it is material, benchmark raw body (JS source verbatim, metadata in headers) against the current urlencoded form. Raw body is the discriminator: it removes all outer escaping of renderingRequest, so it bounds the achievable gain.
   • If raw body is not materially better than form → stop (do not test JSON or MessagePack — escaping isn't the cost).
   • If raw body wins → ship it (no new dependency, easy to debug). Fall back to a JSON envelope only if a field-shape constraint makes raw body impractical.
3. Do not start with JSON-first-then-stop: JSON still escapes the JS string, so a weak JSON result would wrongly look like "encoding doesn't matter" when raw body would have won.
4. Reach for MessagePack only if raw body is impractical and compact binary metadata is independently shown to matter — unlikely, since msgpack's string handling matches raw body and the metadata is tiny. Do not start with Protobuf/BSON.

Treat a structured render-call protocol as a separate, larger project. It is the path that could remove inner props/JS escaping, but it is much more invasive than changing the outer wire encoding.