Part 5 of From the Issue Tracker, postmortems of GopherTrunk bugs that fought back. Part 4 ended with a one-line hardware fix. This one ends somewhere stranger: after two real bugs were found and fixed, the remaining symptom turned out not to be in the code at all. It was recorded into the capture files themselves — and proving that required decimating a capture with a resampler that wasn’t ours.
TL;DR: With four P25 control-channel taps on one Airspy R2, 2.5 MS/s could only “see” the tap nearest center — but raising the rate to 10 MS/s made all four taps go dark, including the previously strong one (#764). Two genuine rate-dependent bugs were fixed (per-tap CPU blowup and a hardcoded channelizer bin count), yet the symptom survived into pure offline replay (#771). The reported “AGC stuck 10× above target” was a red herring — the working capture shows the identical value. The decisive experiment: decimate the 10 MS/s capture 4:1 with an independent resampler and feed the proven 2.5 MS/s path — the ~10 dB in-channel deficit survived, so it was baked into the samples. Carrier-clean but modulation-degraded, independent of gain: the signature of sampling-clock phase noise at the Airspy’s native 10 MS/s rate. Not our DSP. Also fixed along the way: this project’s issue-closing discipline.
Cheat sheet
| Fact | Detail |
|---|---|
| Issues | #764 (live daemon), #771 (replay follow-up) |
| Symptom | At 2.5 MS/s only the tap nearest center decoded; at 10 MS/s all four taps went dark — including the previously strong one |
| Real bugs fixed | Per-tap full-rate resampling on one goroutine (USB ring overrun); channelizer bin count hardcoded to 16 |
| Red herring | “AGC stuck 10× above target” — the working capture shows the identical agc_level≈1.47 |
| The decisive experiment | Decimate the 10 MS/s capture 4:1 with an independent Kaiser resampler, feed the proven 2.5 MS/s path — same 9.5 dB SNR |
| Verdict | The ~10 dB in-channel deficit is baked into the samples: sampling-clock phase noise at the R2’s native 10 MS/s rate |
| Process fallout | The issue was closed twice unverified — the project’s issue-closing policy exists because of this thread |
In this post
- The report — the elimination already done: RF present, no clipping, band wider than 2.5 MS/s delivers.
- Two real bugs — the per-tap CPU blowup and the hardcoded channelizer bins.
- Offline replay, and a process failure — the symptom survives with no live component; the double-close.
- The red herring: “AGC stuck 10× high” — check the alarming metric in the passing case.
- The experiment that settled it — an independent resampler as the ultimate control, plus the rule-outs.
- Carrier-clean, modulation-degraded — the phase-noise signature and the hardware verdict.
- What we keep — the durable rules and their Field Guide entries.
The report
The setup: an Airspy R2 in wideband role at 420.9 MHz center, four MMR
control-channel taps spread from −887.5 kHz to +937.5 kHz. At
sample_rate: 2500000, only Mt Anakie (−812.5 kHz, closest to usable passband)
decoded; the other three sat at −82 to −86 dBFS with zero frame-sync hits. The
reporter had already done the elimination themselves: airspy_rx tuned directly
at a failing frequency showed −5.3 dBFS of strong signal (RF is present), clip
ratio was zero (no overload), and a capture + spectrum snapshot at the
daemon’s own settings confirmed only one carrier visible at 2.5 MS/s but 19
carriers at 10 MS/s — including all four taps. The band was simply wider than
2.5 MS/s could deliver.
So: raise the rate. And here the report earns its keep, because the result was the opposite of a partial improvement:
Running the live daemon at
sample_rate: 10000000broke decode for all four sites, including Mt Anakie, which decodes cleanly at 2.5 MS/s. Reverting immediately restored it.
A change that kills the previously strong tap is a different beast from one that fails to help the weak ones. It smells like a systemic, rate-dependent defect — and it was. Two of them.
Two real bugs
Per-tap CPU. Each tap mixed and ran a single-stage rational resampler at the full input rate — a 208:1 decimation per tap at 10 MS/s — all inline on one goroutine. At 10 MS/s × 4 taps the engine could no longer keep real time, the USB ring overran, and dropped samples broke symbol sync on every tap. That is why the strong site died with the weak ones: symbol timing doesn’t care about SNR when the sample stream has holes in it. The fix runs one shared decimation stage down to ~2.5 MS/s before the per-tap mixers, pinning per-tap cost to the proven regime regardless of dongle rate.
Channelizer bins. Separately, the polyphase channelizer’s bin count was hardcoded to 16 — sized for 2.4 MS/s ≈ 150 kHz per bin. At 10 MS/s that became 625 kHz bins, merging adjacent carriers. (A wrinkle worth knowing: these four taps, spaced 25–75 kHz apart, sit below the channelizer’s minimum bin width, so they were routed to the per-tap DDC and never touched the channelizer — the two halves of the issue title were unrelated code paths all along.)
Both fixes merged. The reporter rebuilt, retested — same symptom. And then did the single most useful thing in the whole thread: took the live daemon, the USB bus, and real time out of the loop entirely.
Offline replay, and a process failure
gophertrunk capture at both rates, same antenna, same gain, same center — then
gophertrunk replay against the static files. The 2.5 MS/s capture locked and
decoded the full band plan. The 10 MS/s capture did not lock, at any tap offset.
No USB ring, no CPU pressure, no live component: the fix that targeted the
CPU/USB-overrun mechanism could not have addressed this, because this
reproduces with none of those in play.
Two things went wrong at this point, and both are worth owning in print. First, #764 was closed twice on an unverified fix while the symptom was still live, with close comments that re-stated the original fix instead of engaging the follow-up — which is why #771 exists as an issue at all. That failure directly reshaped the project’s issue-closing policy: no close-as-completed until a failing-first test passes and the symptom is verifiably gone, and follow-ups get addressed on their own terms, never by repeating the original fix description.
Second, a subtler confusion: the merged fix changed the multi-tap wideband
DDCBank (internal/dsp/tuner/ddc.go) — but gophertrunk replay -tune-hz runs
the single-channel ccdecoder.Downconverter (internal/scanner/ccdecoder/ddc.go),
a separate path the fix never touched. The “fix doesn’t work” replay result was
structurally impossible to be about that fix. Two code paths that look like
one is a trap that recurred in later issues — the adjacent-carrier warning of
#815 lives in the
daemon’s decoder and never fires in replay, for the same reason — and it’s now
documented so the next investigation starts by naming which path the repro
actually exercises.
The red herring: “AGC stuck 10× high”
The replay diagnostics showed agc_level≈1.47 against agc_target=0.15708 —
roughly 10× above target, never converging. It looked like a smoking gun for a
gain-normalization bug in the new shared decimation stage, and the follow-up
issue was framed around it.
Then someone checked the passing case. The working 2.5 MS/s capture shows the
identical agc_level≈1.47. That ~9× ratio is simply the symbol-domain AGC’s
normal operating gain — the matched filter has gain proportional to
samples-per-symbol, and the AGC compensates. The metric was converging fine in
both captures.
A metric that looks alarming in the failing case must be checked in the passing case before it’s allowed to be evidence.
That one sentence would have saved days. It’s now a standing rule.
The experiment that settled it
If the replay path is rate-invariant (and a synthetic harness said it was — same ±24 kHz passband, same 48 kHz internal rate, unity DC gain at both input rates), the only remaining variable is the captured data itself. The decisive control: take the 10 MS/s capture, decimate it 4:1 with an independent Kaiser polyphase resampler — not GopherTrunk’s DSP at all — and feed the result through the proven 2.5 MS/s path, the exact path that locks the native 2.5 MS/s capture.
| Capture | Demod EVM | Demod SNR | Result |
|---|---|---|---|
| 2.5 MS/s native | 7.4% | 19.7 dB | Locks — full band-plan decode |
| 10 MS/s native | 22.5% | 9.5 dB | No lock |
| 10 MS/s → independent 4:1 decimate → 2.5 MS/s path | 22.5% | 9.5 dB | No lock |
The ~10 dB in-channel deficit survived a clean decimation through a completely
different DSP chain. It was baked into the 10 MS/s samples before any of our code
touched them. That exonerates the DDC, the AGC, and the decode path in one move —
and a regression test (TestDownconverterSNRInvariantAcrossRate) now pins the
invariant so a rate-dependent gain-staging bug can’t sneak back in silently.
Alongside it, the systematic rule-outs — each a specific mechanism with a specific test, so the exoneration is checkable rather than asserted:
| Hypothesis | Test | Result |
|---|---|---|
| Aliasing of strong neighbors beyond ±1.25 MHz (present at 10 MS/s, absent at 2.5) | Band-limit the 10 MS/s input before decimation, then decode | No change — not aliasing |
| float32 precision loss in the recursive NCO or the ~2,500-tap polyphase filter | Recompute both in float64 and compare | Bit-identical output — not precision |
| Channel-filter width admitting extra noise at the higher rate | Sweep the filter from 24 kHz down to 6.25 kHz | <0.5 dB change in post-discriminator SNR — not filtering |
Three plausible rate-dependent mechanisms, three clean negatives — which is what gave the independent-resampler result its weight: by the time it ran, it was the last variable standing.
Carrier-clean, modulation-degraded
So what is wrong with the 10 MS/s samples? Not clipping — both captures peak near −48 dBFS, nowhere close to full scale, so overload and intermod are out. And here is the genuinely counterintuitive part: by wideband FFT, the carrier looks cleaner at 10 MS/s (SNR 36.2 dB) than at 2.5 MS/s (33.1 dB). The energy is all there. Only the in-channel modulation quality is ~10 dB worse — and that deficit sits co-band with the signal, inside ±2.4 kHz of the discriminator output, where no filter can remove it.
Carrier-clean but modulation-degraded is the classic signature of reciprocal mixing — phase noise on the sampling clock, smearing the modulation without touching the carrier’s apparent strength. The confirmation came from a gain-independence test the reporter ran on request: gain 600 gave EVM 22.5% / SNR 9.5 dB; gain 300 gave 22.7% / 9.4 dB, despite ~6 dB less capture power. A deficit that doesn’t track gain isn’t compression or IMD. It’s fixed to the clock path.
The hardware picture closes the loop: the Airspy R2 exposes exactly two native
rates, 10 and 2.5 MS/s — and 2.5 is the FPGA’s decimate-by-4 of the same 10 MS/s
ADC, which averages down the clock phase noise. The deficit appeared at the R2’s
top native rate. That generalizes into a purchasing caution: the Airspy Mini
shares the R820T2 tuner but samples on a different clock, and its 6 MS/s ceiling
is the same untested top-rate regime — measure before you rely on it. The
qualification recipe (capture the same channel at 2.4 MS/s and the rate under
test, replay both with -diag, compare the EVM/SNR line) is now in
Airspy rate selection.
What shipped, besides the two real fixes: a no-lock reason string in replay
(“EVM 22.5% / demod SNR 9.5 dB is below the lock threshold — SNR-limited capture,
not a tuning or AGC issue”), so the next person sees the cause instead of a
misleading AGC number, and an opt-in -soft-sync mode that extends reach into
marginal captures without being able to manufacture a false lock. The reporter’s
verdict: staying on 2.5 MS/s for this site — which the analysis confirms is the
right call on this hardware.
What we keep
- Check the alarming metric in the passing case. The “AGC stuck 10× high” reading was identical in the capture that locked. This rule alone reorders most investigations, and it opens the diagnostic playbook.
- An independent implementation is the ultimate control. Decimating with a resampler that isn’t yours separates “our DSP degrades the data” from “the data arrived degraded” in one experiment.
- Carrier-clean + modulation-degraded = clock phase noise. The wideband FFT can look better while the channel is unusable; the deficit lives co-band and ignores gain. See signal signatures.
- Know which code path your repro exercises. The daemon’s wideband
DDCBankand replay’s single-channelDownconverterare separate; a fix to one proves nothing about the other. Name the path in the log line. - A dongle’s top native rate is its least-tested regime. Qualify it empirically per Airspy rate selection before building a site plan on it.
- Don’t close what you haven’t verified. This issue was closed twice while the symptom was live. The policy that came out of it — failing-first test, reporter confirmation, address the latest follow-up — is why later postmortems in this series are shorter.
FAQ
If the data was bad all along, did the two code fixes matter? Yes. The per-tap CPU blowup and the hardcoded channelizer bin count were real, rate-dependent defects that would have broken any wideband rate raise, clean clock or not — the shared decimation stage is what makes higher capture rates viable at all. They just weren’t the whole story, which is exactly why closing on them without verifying the symptom was the process failure.
How can the carrier look cleaner at 10 MS/s while decode is worse? The wideband FFT measures carrier power against the broadband floor; decode depends on in-channel modulation quality. Reciprocal mixing — clock phase noise convolved onto the signal — smears the modulation while leaving apparent carrier strength intact, so FFT SNR rose (33.1 → 36.2 dB) while demod SNR fell ~10 dB. The two metrics answer different questions.
Why is 2.5 MS/s cleaner when it comes from the same ADC? The R2’s 2.5 MS/s output is the FPGA’s decimate-by-4 of the same 10 MS/s ADC path — and that averaging suppresses the clock phase noise that dominates at the native rate. The top native rate is the one regime with no decimation gain to hide behind.
Could more (or less) gain have rescued the 10 MS/s capture? No — that was the point of the gain-independence test. Gain 600 gave EVM 22.5% / SNR 9.5 dB; gain 300 gave 22.7% / 9.4 dB despite ~6 dB less capture power. A deficit that doesn’t track gain isn’t compression or intermod; it’s fixed to the clock path, and no gain setting touches it.
Should Airspy owners avoid the high rates entirely?
Not categorically — qualify empirically. Capture the same channel at 2.4 MS/s
and the rate under test, replay both with -diag, and compare the EVM/SNR line;
the recipe is in
Airspy rate selection.
The Airspy Mini’s 6 MS/s ceiling is the same untested top-rate regime on a
different clock — measure before building a site plan on it.
Series navigation
Part 5 of 22 · ← Part 4: The Dongle That Heard Nothing — One Line in a Register Table · Next → Part 6: CQPSK in Four Acts — Fixing the Linear Path One Layer at a Time