[UNMAINTAINED] feat(vnc): add VNC server with H.264

[mcuelenaere]

Closes #117

Summary

Adds a VNC server that an H.264-capable VNC client (TigerVNC ≥ 1.13.0 built with FFmpeg, or PiKVM's mdevaev/tigervnc build) can connect to as an alternative to the WebRTC web UI. Frames are shipped using RFB pseudo-encoding 50 (rfbproto PR #39, TigerVNC PR #1194) — the JetKVM device emits H.264 NALs, and the server forwards them to the client unchanged. No software decode on the device.

Why this approach

The "obvious" path — implement a normal RFB server that ships raw RGB pixels — is not viable here. JetKVM's video pipeline only emits already-encoded H.264/H.265 NALs from an opaque native binary; there is no path to raw frames or JPEG snapshots. Software-decoding H.264 just to re-encode as Tight/Raw would burn most of the CPU budget on the embedded ARM SoC.

PiKVM solved the same problem with RFB pseudo-encoding 50 ("Open H.264 Encoding"), which carries an Annex-B H.264 bytestream straight to the client. JetKVM already produces exactly the right bytes. Compatible with TigerVNC ≥ 1.13.0 built with FFmpeg.

RDP was considered (and the user asked) but rejected: the RDP stack (TPKT/X.224 → MCS → TLS/CredSSP/NLA → RDPGFX for H.264) is roughly an order of magnitude more code than RFB, and there is no mature pure-Go RDP server.

What's in the PR

Layered bottom-up across 9 commits, each independently reviewable and not breaking the build:

1. video: refcount the capture pipeline — replaces the WebRTC-specific VideoStart/VideoStop with named-consumer refcount (acquireVideoStream / releaseVideoStream). Behaviour-preserving prerequisite.
2. internal/rfb: protocol primitives — pure-Go RFB 3.8 server library (handshake, VNCAuth via crypto/des, message decoders, Raw + OpenH264 + DesktopSize encoding writers). 38 unit tests, no JetKVM imports.
3. internal/rfb: keysym to HID Usage ID mapping — full ~150-entry table: ASCII printable, F1–F24, modifiers, navigation, full keypad, lock keys, Latin-1.
4. internal/rfb: placeholder framebuffer for non-H.264 clients — "Please use TigerVNC ≥ 1.13.0 with H.264 support" rendered via golang.org/x/image/font/basicfont. Pinned to v0.18.0 for Go 1.24 compatibility.
5. vnc: scaffold (config, JSON-RPC, logger) — adds VncEnabled / VncPort / VncPassword config fields, JSON-RPC getters/setters, vncLogger, and stub Start/StopVNCServer.
6. vnc: server, frame fan-out, HID forwarding — fills out the scaffold: TCP listener, per-client bounded channels (cap=2, non-blocking sends drop on full so slow clients can't backpressure WebRTC), SPS/PPS caching for late-joiners, IDR-frame filtering, HID forwarding (PointerEvent → rpcAbsMouseReport + wheel translation; KeyEvent → keysym lookup → rpcKeypressReport). resolveCodec forces H.264 when VNC is enabled so a Mac/Windows browser can't renegotiate H.265 mid-VNC-session.
7. ui: VNC settings panel — /settings/vnc route with toggle, port, password. Localization keys added to all 14 language files (English values only — translations are a follow-up).
8. vnc: drop VncAllowOverWAN — removed an opaque non-loopback-bind guard that surfaced as a generic "internal error" in the frontend. Operators are responsible for tunneling VNCAuth via SSH/Tailscale on untrusted networks; godoc on VncPassword says so.
9. vnc: pin H.264 codec when VNC starts the capture pipeline — fixes a subtle bug uncovered by on-device testing: when VNC was the first consumer of the capture pipeline (no browser open), VideoSetCodecType was never called, so the codec stayed at whatever the previous WebRTC session left it. With H.265 sessions on Mac/Windows browsers, VNC then shipped H.265 NALs over OpenH264 (which is H.264-only) and clients silently failed to decode. acquireVideoStreamWithCodec(consumer, codecType) now sets the codec only when the caller is the first consumer, preserving the "must not be called mid-stream" invariant.

Tests

• 38 unit tests in internal/rfb covering handshake byte sequences, VNCAuth (DES), message round-trips, Raw + OpenH264 + DesktopSize encoding writers, keysym→HID lookup, and placeholder rendering.
• 11 unit tests in the top-level kvm package for the H.264 Annex-B splitter, mouse-coord scaling, and wheel-button rising-edge detection.
• End-to-end on-device: deployed and verified handshake, auth, and HID input forwarding work; codec-pinning bug surfaced and fixed under real load.

Known limitations / follow-ups

• Up-to-GOP-length blank screen on connect. No ForceKeyFrame RPC exists in the native binary. New clients wait for the next natural IDR (typically ≤ 1–2 s). Documented; the right fix is a native-binary change.
• No JPEG fallback for non-H.264 clients. They get the placeholder Raw rect; mouse/keyboard still work. JPEG fallback would require native-binary support for a JPEG memsink.
• VeNCrypt / TLS not yet implemented. VNCAuth is the only auth mechanism; users on untrusted networks should tunnel via SSH or Tailscale, or use LocalLoopbackOnly. The repo already has TLS infrastructure (web_tls.go) that VeNCrypt can reuse later.
• Clipboard, mobile-VNC quirks, PiKVM-style hotkey sequences — out of scope; nice follow-ups.
• Coordination with feat: add RTSP server, Chromecast WebRTC casting, and multi-session support #1329. That PR adds RTSP and a multi-session WebRTC registry. It also has H.264 codec coordination needs; if it merges first, the small splitAnnexB helper duplicated here can be consolidated. Per its review status I chose not to base on its branch — the conflict is one line in native.go either way.

Logging

The default log level is WARN. To see VNC diagnostics (server listening, client connections, advertised encodings):

JETKVM_LOG_INFO=vnc /userdata/jetkvm/bin/jetkvm_app
# or full trace:
JETKVM_LOG_TRACE=vnc

Set persistently via a systemd unit override at /etc/systemd/system/jetkvm.service.d/log.conf.

Checklist

• Ran make test_e2e locally and passed
• Linked to issue(s) above by issue number (Closes #117)
• One problem per PR (no unrelated changes)
• Lints pass; CI green — running on push
• Tricky parts are commented in code (codec coordination, frame buffer aliasing, IDR-wait, SPS/PPS cache priming)

🤖 Generated with Claude Code

---

Note

High Risk
High risk because it introduces a new TCP-exposed remote-access service with authentication handling and changes capture-pipeline start/stop/codec coordination shared with WebRTC, which can impact availability and video behavior.

Overview
Adds a configurable VNC server (RFB 3.8) that listens on a user-set port, supports None or classic VNCAuth, and streams video using OpenH264 pseudo-encoding 50 (with a rendered Raw placeholder for non-H.264 clients), including resolution-change signaling and an extended mouse-buttons negotiation path.

Integrates VNC into the runtime: new config fields + JSON-RPC getVNCConfig/setVNCConfig, new vnc logger, VNC start on boot when enabled, and per-frame fan-out from the native video callback with SPS/PPS caching, IDR resync logic, and HID input forwarding.

Refactors capture-pipeline lifecycle to a named-consumer refcount (acquireVideoStreamWithCodec/releaseVideoStream) used by both WebRTC and VNC, updates HDMI sleep behavior to depend on active consumers, and forces H.264 codec selection whenever VNC is enabled to avoid H.265/VNC decoder desync.

Adds a new /settings/vnc UI route (with localization keys) to manage enable/port/password/keymap, and introduces golang.org/x/image dependency for placeholder rendering.

^{Reviewed by Cursor Bugbot for commit 83ca5c2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

All committers have signed the CLA.

[cursor]

Data race on extendedMouseButtons between goroutines

Medium Severity

The extendedMouseButtons field on rfb.Conn is a plain bool written by the dispatch goroutine via SetExtendedMouseButtons(true) and read concurrently by the read goroutine in ReadClientMessage. This is a data race under Go's memory model — no synchronization (mutex, atomic, or channel) ensures the reader sees the updated value. If the reader sees a stale false, it will parse an extended 7-byte PointerEvent as the legacy 6-byte format, consuming one byte too few and permanently misaligning all subsequent message parsing on that connection.

Additional Locations (2)

• internal/rfb/messages.go#L191-L192
• vnc_conn.go#L288-L289

 

^{Reviewed by Cursor Bugbot for commit ba5cd0a. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 83ca5c2. Configure here.}

[cursor]

Data race on primed field between goroutines

Medium Severity

The primed field is read (line 308) and written (line 330) in writeUpdate without holding stateMu, but WriteFrame sets c.primed = false under stateMu from the producer goroutine. This is a data race between the dispatcher and producer goroutines. The field needs to be read/written under the same lock in both locations, or moved into the existing stateMu-guarded block at the top of writeUpdate.

Additional Locations (1)

• vnc.go#L354-L358

 

^{Reviewed by Cursor Bugbot for commit 83ca5c2. Configure here.}

[quentinmit]

The "obvious" path — implement a normal RFB server that ships raw RGB pixels — is not viable here. JetKVM's video pipeline only emits already-encoded H.264/H.265 NALs from an opaque native binary; there is no path to raw frames or JPEG snapshots.

This doesn't seem correct to me. The native binary is open-source and in this repo, and it captures raw YUV data:

kvm/internal/native/cgo/video.c

Line 452 in 79c23ec

fmt.fmt.pix_mp.pixelformat = V4L2_PIX_FMT_YUYV;

That YUV data is currently being given to the RK_MPI video encoder, but it could be processed directly by the VNC code. (IDK if VNC supports YUV data, so you might need to ask for RGB data instead.)

You could probably even selectively feed portions of a frame to the hardware H.264 or JPEG encoders after calculating damage.

[quentinmit]

Without closing the feedback loop to the encoder, this is going to cause significant latency to build up if the client isn't able to request frames at the same rate as the encoder is producing them, until eventually the queue is full and it (re)starts with a new I-frame.

The update request should be plumbed all the way through to the encoder loop so it only encodes the next frame when there is a client that needs it:

kvm/internal/native/cgo/video.c

Line 621 in 79c23ec

if (RK_MPI_VENC_SendFrame(VENC_CHANNEL, &stFrame, 2000) != RK_SUCCESS)

[mcuelenaere]

I've decided to no longer pursue this feature. My original reasons for adding VNC support, was because I wanted a better client experience. After realizing that this was going to become very difficult by going the VNC route, I've decide to instead write a custom client, leveraging WebRTC as protocol.

This solves all my needs and thus I an no longer interested in adding VNC support.

Having said that, there is still value in this work so I won't close this PR (unless the maintainers want me to). Someone can start from this and maybe implement the suggested (delta) JPEG encoding to

1. add support for a wider range of VNC clients (only TigerVNC compiled with ffmpeg supports H264)
2. potentially improve latency (although I very much doubt that will fix the issue)