Message framing

Key takeaways TCP delivers a byte stream, not messages: your three writes can arrive as one read, or five. Framing is the receiver’s rule for finding message boundaries, and there are three families: delimiters (scan for a marker, escape it in data), length prefixes (read N, then N bytes), and fixed-size frames. When framing drifts — one boundary misjudged — everything after parses as garbage, so serious protocols add resynchronization: a way to find the next true boundary. Radio protocols, which lose bits routinely, are the masterclass in framing done defensively.

Here’s the level below everything this module has used so far. HTTP, SSE, WebSockets, gRPC — each one had to answer the same primitive question first: in an endless river of bytes, where does one message end and the next begin? This lesson is that question, and it’s where API engineering meets the problems radio engineers have fought for a century.

The stream has no seams

A TCP connection (TCP & UDP) promises bytes in order — and nothing about grouping. Write {"a":1} then {"b":2}, and the receiver may read {"a":1}{"b": now and 2} later. Any code that assumes “one read = one message” works flawlessly on localhost, then shatters under real network conditions — one of the classic bugs of network programming. The protocol must define boundaries; the bytes won’t.

Framing turns a seamless byte stream back into messages — and one wrong boundary poisons everything downstream until resync.

The three families

Strategy	How it works	Example protocols	Weakness
Delimiter	Scan for a marker byte/sequence	HTTP headers (CRLF + blank line), SSE (blank line), newline-delimited JSON	The marker must never appear in data — so data must be escaped or forbidden
Length prefix	Read a fixed-size length field, then exactly that many bytes	HTTP bodies (`Content-Length`), WebSocket frames, gRPC messages	A corrupted length desynchronizes everything after it
Fixed size	Every frame is exactly N bytes	Audio sample blocks, TDMA radio bursts, cell-based transports	Rigid — every message pays for the largest case

Notice HTTP uses two of them: delimiters for the header section (human-typed text, unbounded) and a length prefix for the body (arbitrary binary data that no delimiter could survive). That hybrid is a sensible general design: delimiters where content is constrained text, lengths where content is arbitrary bytes.

Delimiters carry a subtle tax — escaping. If \n ends a message, a \n inside a message must be encoded (which is partly why newline-delimited JSON works: JSON string escapes already guarantee no raw newlines). Forget the escaping rule on either side and you’ve built a protocol that fails only on the data that happens to contain the delimiter — a bug that hides for months.

When framing drifts

Framing errors are uniquely vicious because they cascade. Misread one length field — a corrupted byte, a bug, a version mismatch — and the receiver starts the “next message” mid-payload. That message parses as garbage or, worse, parses as valid nonsense; its misread length points somewhere else random, and the connection is now permanently desynchronized, producing errors unrelated to the original cause. The symptom appears far from the fault, which is what makes framing bugs so miserable to diagnose.

Protocol designers answer with resynchronization strategies:

Tear down and reconnect. TCP protocols mostly punt: on any framing anomaly, kill the connection; the fresh one starts cleanly framed. Crude, correct, and exactly what gRPC and WebSocket implementations do on a malformed frame.
Sync markers. Put a distinctive pattern at each frame start; on confusion, scan forward for it. This is how radio lives: a P25 or DMR receiver finds a frame sync word — a bit pattern chosen for unmistakability — in a stream that has no connections to reset, and simply hunts for the next sync after any corruption. GopherTrunk’s decoders spend serious engineering on exactly this: on the air, framing drift is not an edge case but the weather.
Self-terminating encodings whose boundaries survive local damage — rarer, costlier, used where neither reset nor markers fit.

Rule of thumb: never build your own framing on TCP if you can borrow one — newline-delimited JSON, WebSocket frames, or gRPC give you battle-tested boundaries. If you must build it, length-prefix it, bound the length (a 2 GB “message” is an attack, not a message), and decide the resync story before the first byte ships.

Why API people should care

Mostly you’ll consume framing others built — but the abstraction leaks upward exactly when things break: a proxy that buffers an SSE stream is a framing middleman misbehaving; a “connection reset” after a malformed chunk is a resync policy firing; a hand-rolled TCP protocol between two of your own services is you, on the hook for everything above. And if you ever look inside the digital radio protocols GopherTrunk decodes, you’ll find this lesson is their whole lower half.

Quick check: a receiver misreads one length prefix in a stream. What's the consequence?

Recap

TCP is a byte stream with no message boundaries; “one write = one read” is a localhost illusion.
Framing comes in three families: delimiters (escape the marker!), length prefixes, and fixed-size frames — HTTP hybridizes the first two.
Framing errors cascade: one wrong boundary poisons everything after it, with symptoms far from the cause.
Serious protocols plan resynchronization — connection reset on TCP, sync words on radio, where drift is routine.
Borrow framing (WebSocket, gRPC, NDJSON) rather than building it; if you build, bound your lengths and design the resync first.

Next up: Schemas & code generation.