Field Guide · concept

Also known as: round-trip trap, self-consistent synthetic trap

The self-consistent test trap is the failure mode of round-trip testing: encode something, decode it back, assert you got the original — and never notice that the encoder and decoder share the wrong constant, the wrong table, or the wrong convention, so the mistake cancels itself out.1 The test is real, the assertions are strict, the suite is green, and the code is wrong in a way this test can never see, because a round trip is blind to any error it makes twice. It is a test-oracle problem in disguise: the decoder is being graded by an oracle that inherited its own misconceptions.

encoder decoder PASS ✓ shared wrong constant
The assertion checks that decode inverts encode — which stays true no matter what the shared constant says. The bug lives below the round trip, where no assertion looks.

How it bites

The trap recurs wherever a codebase implements both directions of a transform, and GopherTrunk’s issue tracker documents it in several independent dresses:

  • A skipped step, skipped twice. The TETRA direct-mode voice path skipped descrambling at colour code 0 — wrong, since colour-0 scrambling is non-identity — but the synthetic tests scrambled and descrambled consistently at colour 0, so they passed either way and real on-air traffic decoded at the chance floor, misread as encryption (#1003).
  • A wrong constant, mirrored in the fake. A SoapyRemote RPC call used opcode 600 where the protocol says 501 — and the fake test server switched on the same constant, so both sides moved together and every unit test passed while real hardware silently ignored the call.
  • A wrong table, validated against itself. Placeholder TETRA sync-training constants correlated perfectly against signals synthesized from those constants, and never once against real air (#553).
  • An estimator grading its own homework. A blind channel estimate contaminated by interference “verified” an interference-rejection combiner on synthetic scenes built from the same estimate’s assumptions.

The common thread: the test’s ground truth was manufactured by the code under test, so the test measured self-consistency — a property even wrong code has.

Escaping it

The antidote is always the same in structure — inject an independent source of truth — and comes in escalating strengths:

  1. Independent references. Decode fixtures produced by a different implementation (a reference codec, another project’s decoder, upstream protocol literals), or check constants against the spec’s numbers rather than your own header. This is golden test vectors with the emphasis on whose gold.
  2. Asymmetric fixtures. Make the encode side unconditional where reality is unconditional — GopherTrunk’s regression now scrambles on encode always, so a decoder that skips descrambling fails the test the way it fails the air.
  3. Consumption asserts. A fake server that requires every request byte to be consumed catches argument-shape drift a switch-on-shared-constants fake cannot.
  4. Reality gates. For signal-processing code, no synthetic pass closes the loop: the change is not verified until it works against a capture of the real phenomenon. A green round trip is a necessary condition, never a sufficient one.

In the bigger picture

Round-trip tests remain excellent — fast, deterministic, great at catching asymmetric regressions — and none of this argues for deleting them. The discipline is knowing what they cannot certify: any property both directions share. Unit tests isolate code from its collaborators; this trap is what happens when a test fails to isolate code from its assumptions. Budget at least one test per transform whose expected values were produced by something that is not your code.

Sources

  1. Test oracle — Wikipedia, on the problem of obtaining correct expected outputs independently of the system under test. ↩

See also

From the blog