|
| 1 | +--- |
| 2 | +import BlogLayout from '../../layouts/BlogLayout.astro'; |
| 3 | +
|
| 4 | +const bodyContent = `<p>Building a reliable userspace transport stack from scratch is a formidable challenge, but the Go programming language provides the exact concurrency primitives and standard library depth needed to pull it off. We recently relied entirely on pure Go to build <a href="/">Pilot Protocol</a> (created by Calin Teodor at Vulture Labs), a peer-to-peer overlay network designed to provision autonomous AI agents with persistent, routable 48-bit virtual addresses.</p> |
| 5 | +
|
| 6 | +<p>If you follow the rapidly evolving AI space, you have likely encountered the Model Context Protocol (MCP) or various Agent-to-Agent (A2A) frameworks. While MCP provides an excellent standard for the application layer by defining how AI models manage context and execute tools, it intentionally stops short of solving the underlying networking layer. MCP does not dictate how two autonomous AI agents natively connect when trapped behind disparate corporate firewalls or residential NATs.</p> |
| 7 | +
|
| 8 | +<p>Pilot Protocol fills this exact gap. It leverages UDP hole-punching for efficient NAT traversal, but because agents exchanging structured API data require reliable, stream-oriented connections, raw UDP is not enough. Existing transport protocols like QUIC were too heavy and did not natively support our requirements for custom addressing and bilateral cryptographic trust handshakes.</p> |
| 9 | +
|
| 10 | +<p>Our solution was to build a reliable transport layer entirely in userspace over raw UDP using zero external dependencies. This meant implementing sliding windows, Additive-Increase/Multiplicative-Decrease (AIMD) congestion control, Nagle's algorithm, and Retransmission Timeouts (RTO) from scratch. Here is how we bypassed the kernel and built a full state machine, congestion controller, and crypto-router natively in Go, and how you can apply these lessons to your own high-throughput systems.</p> |
| 11 | +
|
| 12 | +<section> |
| 13 | +<h2>1. Concurrency and Fine-Grained Locking</h2> |
| 14 | +
|
| 15 | +<p>When writing a custom transport protocol, an easy pitfall is locking the entire socket state whenever a packet arrives. At thousands of packets per second, a global lock quickly creates a severe bottleneck.</p> |
| 16 | +
|
| 17 | +<p>Instead, we leveraged Go's concurrency primitives to keep locking highly granular. We utilize a single, lock-free routing goroutine that reads off the <code>net.UDPConn</code> and demultiplexes packets to specific connection handlers via channels:</p> |
| 18 | +
|
| 19 | +<pre><code>func (d *Daemon) routeLoop() { |
| 20 | + for incoming := range d.tunnels.RecvCh() { |
| 21 | + d.handlePacket(incoming.Packet, incoming.From) |
| 22 | + } |
| 23 | +}</code></pre> |
| 24 | +
|
| 25 | +<p>Inside our <code>Connection</code> struct, we partitioned state across multiple specific <code>sync.Mutex</code> instances. Looking through our implementation, you will find <code>conn.Mu</code> for general connection state, <code>conn.RetxMu</code> for the retransmission buffer, <code>conn.NagleMu</code> for write coalescing, <code>conn.AckMu</code> for delayed ACKs, and <code>conn.RecvMu</code> for out-of-order buffers.</p> |
| 26 | +
|
| 27 | +<p>This separation of concerns means that an incoming packet carrying both data and a SACK (Selective Acknowledgement) block can update the receive window and the retransmission queue concurrently without ever stalling the outbound write path.</p> |
| 28 | +</section> |
| 29 | +
|
| 30 | +<section> |
| 31 | +<h2>2. Taming the Garbage Collector with Polled Timers</h2> |
| 32 | +
|
| 33 | +<p>In early iterations of our stack, we noticed a significant source of GC pressure: per-packet timer allocations. Initially, we allocated a new <code>time.AfterFunc</code> for every single packet sent to handle Retransmission Timeouts (RTO). Allocating and tearing down thousands of timer objects per second led to measurable CPU spikes.</p> |
| 34 | +
|
| 35 | +<p>To optimize memory management, we shifted to a polling model. Each connection is assigned exactly one background goroutine running a static <code>time.Ticker</code>:</p> |
| 36 | +
|
| 37 | +<pre><code>func (d *Daemon) retxLoop(conn *Connection) { |
| 38 | + ticker := time.NewTicker(RetxCheckInterval) |
| 39 | + defer ticker.Stop() |
| 40 | +
|
| 41 | + for { |
| 42 | + select { |
| 43 | + case <-conn.RetxStop: |
| 44 | + return |
| 45 | + case <-ticker.C: |
| 46 | + conn.Mu.Lock() |
| 47 | + st := conn.State |
| 48 | + conn.Mu.Unlock() |
| 49 | +
|
| 50 | + if st == StateEstablished || st == StateFinWait { |
| 51 | + d.retransmitUnacked(conn) |
| 52 | + } else if st == StateClosed { |
| 53 | + conn.CloseRecvBuf() |
| 54 | + d.ports.RemoveConnection(conn.ID) |
| 55 | + return |
| 56 | + } else { |
| 57 | + return |
| 58 | + } |
| 59 | + } |
| 60 | + } |
| 61 | +}</code></pre> |
| 62 | +
|
| 63 | +<p>Scanning an array of several hundred inflight packets periodically is highly efficient in Go. In contrast, managing hundreds of individual timer objects places unnecessary strain on the heap. One goroutine, one ticker, one heap allocation for the timer itself — regardless of how many packets are in flight.</p> |
| 64 | +</section> |
| 65 | +
|
| 66 | +<section> |
| 67 | +<h2>3. Modeling Nagle's Algorithm with select</h2> |
| 68 | +
|
| 69 | +<p>Nagle's algorithm coalesces small writes into larger Maximum Segment Size (MSS) packets to reduce network overhead. Implementing this in C often requires complex event loop manipulation. In Go, the native <code>select</code> statement handles this control flow elegantly.</p> |
| 70 | +
|
| 71 | +<p>When our outbound function receives a sub-MSS write, it buffers the data and evaluates when to flush it. The goroutine simply waits for one of three conditions: a 40 ms timeout is reached, the inflight data is acknowledged (meaning the network is clear to send more), or the connection is closed.</p> |
| 72 | +
|
| 73 | +<pre><code>nagleTimer := time.NewTimer(NagleTimeout) |
| 74 | +select { |
| 75 | +case <-conn.NagleCh: |
| 76 | + nagleTimer.Stop() |
| 77 | + // All data ACKed, flush now |
| 78 | +case <-nagleTimer.C: |
| 79 | + // 40ms Timeout reached, flush regardless |
| 80 | +case <-conn.RetxStop: |
| 81 | + nagleTimer.Stop() |
| 82 | + return protocol.ErrConnClosed |
| 83 | +}</code></pre> |
| 84 | +
|
| 85 | +<p>Using standard channels allowed us to implement classic TCP network states directly as control flow, significantly reducing the complexity of the underlying state machine.</p> |
| 86 | +</section> |
| 87 | +
|
| 88 | +<section> |
| 89 | +<h2>4. Native Cryptography: ECDH, AES-GCM, and Ed25519</h2> |
| 90 | +
|
| 91 | +<p>Building a secure network stack usually means wrestling with CGO bindings for OpenSSL or Ring. With Go, we built the entire cryptographic layer natively.</p> |
| 92 | +
|
| 93 | +<p>For the protocol's default tunnel encryption, we use <code>crypto/ecdh</code> to perform an X25519 key exchange, deriving a shared secret (via HKDF-SHA256) that seamlessly feeds into <code>crypto/aes</code> and <code>crypto/cipher</code> for AES-256-GCM authenticated encryption. This means an attacker cannot read or tamper with the payload without detection.</p> |
| 94 | +
|
| 95 | +<p>For node identity and trust handshakes, we rely on <code>crypto/ed25519</code> to issue unique 48-bit virtual addresses. When loading a node's persistent identity from disk upon restart, we use a neat Go trick to verify the keypair's mathematical integrity before trusting it:</p> |
| 96 | +
|
| 97 | +<pre><code>// LoadIdentity reads an identity keypair from a JSON file. |
| 98 | +func LoadIdentity(path string) (*Identity, error) { |
| 99 | + // ... reading file and base64 decoding omitted for brevity ... |
| 100 | +
|
| 101 | + priv, err := DecodePrivateKey(f.PrivateKey) |
| 102 | + pub, err := DecodePublicKey(f.PublicKey) |
| 103 | +
|
| 104 | + // Verify key consistency: the public key stored on disk must match |
| 105 | + // the public key derived from the private key |
| 106 | + derivedPub := priv.Public().(ed25519.PublicKey) |
| 107 | + if !derivedPub.Equal(pub) { |
| 108 | + return nil, fmt.Errorf("identity file corrupted: public key does not match private key") |
| 109 | + } |
| 110 | +
|
| 111 | + return &Identity{PublicKey: pub, PrivateKey: priv}, nil |
| 112 | +}</code></pre> |
| 113 | +
|
| 114 | +<p>This entirely native crypto stack handles thousands of authenticated handshake packets per second without breaking a sweat, all while keeping our binary dependency-free.</p> |
| 115 | +</section> |
| 116 | +
|
| 117 | +<section> |
| 118 | +<h2>5. Integrating with the Standard Library (net.Conn)</h2> |
| 119 | +
|
| 120 | +<p>The ultimate goal of building a custom network stack is making it invisible to the application developer. Because our client <code>Conn</code> implements <code>Read</code>, <code>Write</code>, <code>SetDeadline</code>, and <code>Close</code>, it perfectly satisfies the standard library's <code>net.Conn</code> interface.</p> |
| 121 | +
|
| 122 | +<p>To ensure standard library compatibility (like handling timeouts dynamically when wrapped in a Go <code>http.Server</code>), we utilized a channel broadcasting pattern for <code>SetReadDeadline</code>. This was critical; making Go's HTTP server work correctly over our overlay exposed five distinct bugs in our IPC layer before we got it right. When a new deadline is set, we close a dedicated channel (<code>deadlineCh</code>) to interrupt any blocked reads immediately:</p> |
| 123 | +
|
| 124 | +<pre><code>func (c *Conn) SetReadDeadline(t time.Time) error { |
| 125 | + c.mu.Lock() |
| 126 | + c.readDeadline = t |
| 127 | + // Signal any blocked Read to re-check |
| 128 | + if c.deadlineCh != nil { |
| 129 | + close(c.deadlineCh) |
| 130 | + } |
| 131 | + c.deadlineCh = make(chan struct{}) |
| 132 | + c.mu.Unlock() |
| 133 | + return nil |
| 134 | +}</code></pre> |
| 135 | +
|
| 136 | +<p>The <code>Read</code> method itself then uses a <code>select</code> statement that listens on the network receive channel (<code>c.recvCh</code>), the deadline timer, and the deadline interrupt channel (<code>dch</code>). If the deadline is updated mid-read, the <code>dch</code> case triggers, forcing the function to re-evaluate the clock:</p> |
| 137 | +
|
| 138 | +<pre><code>// Inside Conn.Read... |
| 139 | +select { |
| 140 | +case data, ok := <-c.recvCh: |
| 141 | + if !ok { |
| 142 | + return 0, io.EOF |
| 143 | + } |
| 144 | + n := copy(b, data) |
| 145 | + if n < len(data) { |
| 146 | + c.recvBuf = data[n:] |
| 147 | + } |
| 148 | + return n, nil |
| 149 | +case <-timer: |
| 150 | + return 0, os.ErrDeadlineExceeded |
| 151 | +case <-dch: |
| 152 | + // Deadline was changed, re-check |
| 153 | + return 0, os.ErrDeadlineExceeded |
| 154 | +}</code></pre> |
| 155 | +</section> |
| 156 | +
|
| 157 | +<section> |
| 158 | +<h2>Conclusion</h2> |
| 159 | +
|
| 160 | +<p>Building a reliable transport layer in userspace is a rigorous technical challenge, but Go's standard library provided everything we needed out of the box. By combining the <code>crypto</code> packages for native security, the <code>time</code> and <code>sync</code> packages for efficient memory management, and the deep interface ecosystem of the <code>net</code> package, we built a high-performance overlay network without importing a single third-party dependency.</p> |
| 161 | +
|
| 162 | +<p>If you are interested in networking, custom protocols, or how distributed AI agents will traverse firewalls to communicate in the future, dive into the Go standard library. You will find it is more powerful than you think. Explore the <a href="https://github.com/TeoSlayer/pilotprotocol">Pilot Protocol source on GitHub</a>, or read about <a href="/for/enterprise">running Pilot Protocol as an enterprise private network</a>.</p> |
| 163 | +</section> |
| 164 | +
|
| 165 | +<div class="cta"> |
| 166 | + <h3>Try Pilot Protocol</h3> |
| 167 | + <p>A userspace transport, a 48-bit agent address space, NAT traversal, and end-to-end encryption — all in a single Go binary.</p> |
| 168 | + <a href="/docs/getting-started">Get Started</a> |
| 169 | +</div>`; |
| 170 | +--- |
| 171 | +<BlogLayout |
| 172 | + title="Building a Userspace TCP-over-UDP Stack in Pure Go" |
| 173 | + description="How we built a reliable transport layer in userspace for Pilot Protocol — sliding windows, Nagle, RTO, and AES-GCM — using only Go's standard library." |
| 174 | + date="April 19, 2026" |
| 175 | + tags={["go", "networking", "transport", "systems"]} |
| 176 | + canonicalPath="/blog/userspace-tcp-over-udp-stack-pure-go" |
| 177 | + bannerImage="/blog/banners/why-ai-agents-need-network-stack.webp" |
| 178 | +> |
| 179 | + <Fragment set:html={bodyContent} /> |
| 180 | +</BlogLayout> |
0 commit comments