From the Issue Tracker, Part 3: Encrypted, Says Who — Four Layers Between a Flag and Its Metadata

By

Part 3 of From the Issue Tracker, postmortems of GopherTrunk bugs that fought back. Part 2 chased a talker alias through cipher land. This one is about a field that was always half right: every encrypted P25 Phase 2 call got flagged encrypted: true, and not one of them ever said *what kind of encrypted. The answer turned out to be four independent defects deep — and the tool that finally ordered them was a log line that fires even when there is nothing to say.*

TL;DR: On a live P25 Phase 2 system, 66 of 66 encrypted calls were flagged encrypted: true while algorithm_id/key_id never populated (#813). The flag and the metadata come from different places, so one can be right while the other is structurally impossible. Underneath sat a four-stage chain: an encryption-sync MAC opcode (0x70) that doesn’t exist on real air, a plausible-but-wrong carrier-recovery diagnosis disproven by a TCXO source failing identically, a 48-bit sync-word constant silently truncated into a 40-bit field, and a missing 2↔3 dibit remap that let superframes lock while every payload decoded to garbage. The technique that cracked the ordering was an unconditional per-call census that logs stage counters even at zero — and the last line of defense is a validity gate that refuses to publish an algorithm ID the standard has never heard of.

Cheat sheet

Fact Detail
Issue #813
Symptom 66/66 encrypted Phase 2 calls flagged encrypted: true; algorithm_id / key_id never populate
Wrong theory Missing carrier recovery — a real defect, genuinely fixed, and not this bug (a TCXO source failed identically)
Real causes Fictional OpEncryptionSync = 0x70; a 48-bit sync constant truncated into a 40-bit field; a missing 2↔3 dibit remap
The diagnostic Unconditional per-call census — stage counters logged even at zero
Field numbers Phase 1: 89% of encrypted calls resolved a valid ALGID; Phase 2: 0.5%, and those were a bit-error smear
Last line of defense p25.AlgorithmKnown(id) validity gate at the composer — absent beats wrong

In this post

  • The symptom as reported — the flag is right; the metadata is structurally impossible.
  • Stage zero: the opcode that never existed — 0x70 was a working model, unit-tested against itself.
  • The census: a log line that fires at zero — the three-way stage disambiguator.
  • The plausible wrong theory: carrier recovery — the right bug class, the wrong bug.
  • Root cause one: a 48-bit constant in a 40-bit field — hunting for a sync word never transmitted.
  • Root cause two: the 2↔3 remap — superframes lock while every payload decodes to garbage.
  • The numbers, and the gate — 89% vs 0.5%, and refusing to publish nonsense.
  • What we keep — the durable rules and their Field Guide entries.

The symptom as reported

The reporter’s setup was deliberately clean: an Airspy on the control channel, a dedicated RTL-SDR voice follower, recordings.skip_encrypted: false so encrypted calls are actually followed, and a generous 5-second metadata window. Thirty minutes of live traffic, then straight to the API rather than the logs:

{"id": 9367, "system": "MMR", "protocol": "p25-phase2", "group_id": 3202,
 "frequency_hz": 468612500, "encrypted": true,
 "end_reason": "encrypted", "talkgroup_alpha": "22-02 WD2"}

66 calls flagged encrypted: true. Zero with an algorithm_id or key_id key at all — omitempty was hiding zero values. The reporter had already ruled out the obvious: not the follow window (1500 → 5000 ms changed nothing), not the recorder short-circuiting (confirmed via log lines that calls were followed), not a logging gap (the API is the stored value).

Why can the flag be right while the metadata never arrives? Because they are unrelated. encrypted: true comes from the grant’s ServiceOptions “protected” bit on the control channel. The algorithm and key IDs must be recovered from the voice channel’s MAC layer. One path worked perfectly; the other, it turned out, had never worked at all.

Stage zero: the opcode that never existed

The extraction code existed and was unit-tested: a MAC PDU handler keyed on OpEncryptionSync = 0x70, wired all the way through to the engine. Every test passed — because every test synthesized a 0x70 PDU. The constant was an explicit working model; the source even carried a note that the relevant spec PDF wasn’t available. On real Phase 2 (TIA-102.BBAC), ALGID/KID/MI don’t ride a standalone opcode: they ride the MAC_PTT message that begins each transmission, identified by slot type, not by MAC opcode. AsEncryptionSync could never match anything on air.

That explained the shape of the symptom — flagged but never populated — and produced fix number one: parse MAC_PTT at the documented offsets. It did nothing, which is how the real diagnostic entered the story.

The census: a log line that fires at zero

The first instrumentation pass added detail to the existing composer: p25p2 mac pdu line. The reporter ran ~19 minutes of live traffic: 202 voice chains started with valid configuration, 33 encrypted calls — and zero of those log lines, for any slot type.

Here is the trap: that line only fires on a successful MAC decode. Its silence is identical whether superframe sync never locked, the ISCH never classified a MAC slot, or the MAC FEC failed every time. The silence of a success-only log line carries no diagnostic information.

The replacement was an unconditional per-call census — one line at the end of every call, even when every counter is zero:

composer: p25p2 call census serial=… system=… superframes=N \
    voice_subframes=N mac_subframes=N mac_pdus=N  slot_Voice4V=… slot_Unknown=…

Read as a three-way stage disambiguator:

Census reading Failing stage
superframes=0 Upstream of MAC entirely — superframe sync never locks
superframes>0, mac_subframes=0 Sync locks but ISCH never yields a MAC slot
mac_subframes>0, mac_pdus=0 MAC FEC chain fails every slot — now byte layouts matter

The next run returned superframes=0 on 67 of 67 calls. Not one superframe ever locked, encrypted or not — on the same dongle that recorded Phase 1 voice cleanly the same night. The MAC_PTT byte-offset question was moot; the failure was upstream of MAC entirely.

The plausible wrong theory: carrier recovery

The Phase 2 H-DQPSK receiver was MatchedFilter → Gardner timing → differential decode — no NCO, no AGC, no Costas loop. A differential decoder cancels a constant carrier phase but not the per-symbol rotation 2π·Δf/baud left by a real tuner’s frequency offset. At 6000 baud, ~750 Hz of offset rotates every symbol a full π/4 into the wrong quadrant. And the hardware split matched perfectly: the Airspy control channel (TCXO, low offset) decoded; the RTL-SDR voice follower (no TCXO) never locked. It was even the same bug class already fixed twice on sibling paths — Phase 1 C4FM got coarse AFC in #275, Phase 1 CQPSK got an NCO seed and Costas loop in #492. Every Phase 2 test had synthesized zero-offset IQ, so CI never noticed.

Carrier recovery was added — coarse seed, NCO, AGC, a rotation-aware Costas loop — and the synthetic stream went from ~72% symbol errors at 1500 Hz offset to 0% across ±5 kHz. A genuine defect, genuinely fixed.

The field result: superframes=0, 56 of 56 calls. Unchanged.

The clincher came later, unprompted: the reporter re-ran with the Airspy R2 — a TCXO source — as the voice follower and got superframes=0 on 20 of 20 unencrypted follows too. A low-offset source failing identically to a high-offset one rules out carrier recovery entirely, and rules out hardware with it. Whatever was broken was hardware-independent.

Root cause one: a 48-bit constant in a 40-bit field

// internal/radio/p25/phase2/sync.go — before
OutboundSyncHex uint64 = 0x575F7DFF77FF  // 48 bits…
SyncDibits             = 20              // …into a 20-dibit (40-bit) field

hexToDibits silently used only the low 40 bits, so the correlator hunted for 0x5F7DFF77FF — neither the standard sync word nor anything that has ever been transmitted. No superframe could lock, on any tuner, ever. And every round-trip test passed, because the test encoder injected sync from the same wrong constant it decoded with. A self-consistent fiction.

The authoritative P25 Phase 2 outbound sync is 0x575D57F7FF (cross-checked against OP25’s frame_sync_magics.h and SDRtrunk, per TIA-102.BBAC). The regression test that pinned it synthesizes the sync independently of the project’s own modulator — the only kind of test that can expose a bad shared constant: zero locks on the old value, locks on the correct one.

Root cause two: the 2↔3 remap

Fixing the sync constant surfaced the fourth layer. The shared DQPSK quadrant slicer assigns the two negative-phase symbols the dibit values 3 and 2 where TIA-102 says 2 and 3. The Phase 1 CQPSK path — verified on real air in #492 — already documents and corrects exactly this transposition:

// internal/radio/p25/phase1/receiver/cqpsk.go
var lsmDibitRemap = [4]uint8{0, 1, 3, 2} // swaps 2↔3 → canonical TIA-102

Phase 2 was missing the equivalent remap. The swap is its own inverse, which made for a satisfying sanity check: applying [0,1,3,2] to the transposed sync 0x565956A6AA yields exactly the authoritative 0x575D57F7FF, and vice versa. The failure signature is distinctive and worth remembering: superframes lock, but every payload decodes to garbage — the regression test reproduces the field symptom precisely, recovering alg=0x75 key=0x555d where 0x84/0x1234 was encoded. The fix canonicalizes the receiver’s dibit output right after the slicer (one point covers sync, ISCH, MAC FEC, and diagnostics together) and restores the standard sync constant, leaving the shared demodulator — used by TETRA and Phase 1 CQPSK — untouched.

The numbers, and the gate

A 7-day quantification from the completed-call webhook made the before/after brutal and the residual honest:

Path Encrypted calls Valid algorithm ID resolved Rate
P25 Phase 1 2,739 2,432 (AES-256) 89%
P25 Phase 2 3,107 15 0.5%

The Phase 1 column is the control group, and it explains why the asymmetry went unnoticed for so long: Phase 1 recovers ALGID and key ID from the LDU2 encryption-sync fields mid-call, a decode path verified on air long before this issue — so anyone spot-checking “does encryption metadata work?” against a Phase 1 call saw it working. The Phase 2 path had never produced a correct value, and even after these fixes, the mid-call KindCallSourceUpdate that backfills a Phase 2 call’s source and encrypted flag carries no algorithm or key — those arrive only through the MAC_PTT chain this post walked.

Worse than “omitted”: the Phase 2 fields were populating with bit-error values — a uniform algorithm-ID smear across 0x00–0xFF, a different key ID every call. A wrong value published confidently is worse than an absent one.

The mitigation is a validity gate, p25.AlgorithmKnown(id), checked against the TIA-102 algorithm registry (0x80, 0x81, 0x83, 0x84, 0x85, 0x86, 0x89, 0x9F, 0xAA) and applied at the composer — the single point that both the recorder (webhooks, call history) and the engine (SSE, TUI) draw from, so every consumer is covered at once. An out-of-set value is provably a mis-decode and is dropped; the fields stay absent rather than lying. One deliberate scope call: the gate does not whitelist the classified Type-1 block (0x00–0x41) — admitting 66 low values would let a large slice of the smear straight through, and a real Type-1 sighting is a one-line registry addition.

One process footnote: the reporter’s raw IQ capture sat as a release asset on a fork, which the project’s offline-replay tooling couldn’t fetch — so every fix in this chain had to be verified against spec-conformant synthetics and code-internal cross-checks against the on-air-verified Phase 1 CQPSK path instead of the reporter’s own air. Host captures somewhere plain; it matters.

What we keep

  • A flag and its metadata can have different truth values. encrypted: true came from the grant; the algorithm ID needed the voice channel’s MAC layer. See encrypted call handling for how the two paths relate.
  • Silence of a success-only log line carries no information. The unconditional per-call census — counters logged even at zero — is the single most transferable technique in this story, and it’s now in the diagnostic playbook.
  • A self-consistent synthetic proves nothing about air. The truncated sync constant survived every round-trip test because encoder and decoder shared the fiction. Regression tests for on-air constants must synthesize independently — the pinned values live in P25 on-air constants.
  • A wrong theory can be a real bug. The carrier-recovery work was correct and necessary — it just wasn’t this bug. The disproof (a TCXO source failing identically) is as valuable as the fix.
  • Never publish a value you can’t validate. The registry gate turns a bit-error smear back into honest absence, at the one choke point every consumer shares.

FAQ

How can encrypted: true be reliable while the algorithm ID never arrives? Because they come from different places. The flag is the grant’s ServiceOptions “protected” bit, decoded on the control channel — a path that worked. The algorithm and key IDs must be recovered from the voice channel’s MAC layer, which had never worked. A flag and its metadata can have entirely different truth values.

The carrier-recovery theory fit perfectly — what disproved it? The hardware split (Airspy TCXO decoded, RTL-SDR didn’t) matched the theory, the fix measurably repaired synthetic offsets, and the field census still read superframes=0. The clincher was the reporter re-running with the Airspy — a low-offset TCXO source — as the voice follower and failing identically. When a low-offset source fails the same way as a high-offset one, carrier recovery is exonerated, and so is the hardware.

Why didn’t unit tests catch the truncated sync constant? Every round-trip test injected sync from the same wrong constant it decoded with, so encoder and decoder agreed perfectly. The regression test that pinned the fix synthesizes the sync word independently of the project’s own modulator — the only kind of test that can expose a bad shared constant.

What does the validity gate actually block? Any algorithm ID outside the TIA-102 registry (0x80–0x86, 0x89, 0x9F, 0xAA). An out-of-set value is provably a mis-decode and is dropped at the composer, the one point both the recorder and the engine draw from. It deliberately does not whitelist the classified Type-1 block (0x00–0x41): admitting 66 low values would let a large slice of the bit-error smear straight through.

Is a wrong published value really worse than an absent one? Yes. An absent field says “unknown”; a smeared field says “algorithm 0x75, key 0x555d” with full confidence to every webhook, API consumer, and UI panel downstream — and no consumer can tell it from a real decode. Honest absence is recoverable; confident garbage propagates.

Series navigation

Part 3 of 22 · ← Part 2: The Talker-Alias Hunt — Three Wrong Transports and an Architectural Gate · Next → Part 4: The Dongle That Heard Nothing — One Line in a Register Table

#from-the-issue-tracker#p25#phase2#encryption#debugging#postmortem