Field Guide · algorithm

Also known as: MLSE, maximum-likelihood sequence estimation, Viterbi equalizer

Maximum-likelihood sequence estimation (MLSE) is the optimal way to combat intersymbol interference: rather than filtering each symbol independently, it searches over all possible transmitted sequences and picks the one most likely to have produced the received waveform, given a known channel.1 Because the ISI channel has finite memory, that search is carried out efficiently by the Viterbi algorithm over a trellis whose states are the recent symbol history — which is why MLSE is often called a Viterbi equalizer.

statestime →
MLSE builds a trellis whose states encode recent symbol history; the Viterbi algorithm keeps the best path into each state and the heavy surviving path is the most-likely transmitted sequence.

How it works

The channel with memory L (its impulse response spans L+1 symbols) acts like a convolutional encoder over the transmitted symbols, so the received signal can be modelled by a finite-state trellis:

  • Each state is the last L symbols the channel “remembers.” For an alphabet of M symbols there are M^L states.
  • Each branch between states corresponds to one candidate new symbol and carries a branch metric — the squared distance between the received sample and the noiseless sample that symbol-plus-history would produce through the known channel.
  • The Viterbi algorithm accumulates these metrics, keeping only the single best (smallest-distance) surviving path into each state at each step. The path with the lowest total metric at the end is the maximum-likelihood sequence.

Minimising total squared distance to the received signal is exactly maximising the likelihood under additive Gaussian noise — hence “maximum-likelihood sequence estimation.” A matched filter (or whitened matched filter) front end typically precedes the trellis search, and the channel estimate that defines the branch metrics is measured from a known training/sync sequence.

Cost and trade-offs

MLSE is optimal — no equalizer achieves a lower sequence error probability — but its M^L state count grows exponentially with the channel memory and the modulation order. For a binary signal over a 4-tap channel that is only 8 states, but for higher-order constellations or long delay spreads it explodes, which is why practical systems bound L (and often shorten the channel with a prefilter) or fall back on reduced-state variants (DDFSE, RSSE) that prune the trellis. Where the channel is too long or unknown, the nonlinear DFE or a linear MMSE equalizer trades some performance for far lower cost.

Relevance to SDR

The classic deployment is GSM, whose receivers use a Viterbi equalizer (MLSE) with the midamble training sequence to defeat the multipath ISI of the mobile channel — the same Viterbi engine GSM already uses to decode its convolutional FEC. MLSE also appears in magnetic-recording read channels (PRML) and other severe-ISI links. It sharpens detection well beyond what a linear equalizer manages when ISI is heavy. GopherTrunk’s target systems (P25, DMR, NXDN) use narrowband, RRC-shaped modulations decoded via matched filtering and synchronisation rather than an MLSE equalizer — though the Viterbi algorithm itself is central to their error-correction decoding — so MLSE is presented here as the optimal equalization benchmark from the wider RF world.

Sources

  1. Maximum likelihood sequence estimation — Wikipedia, on trellis-based ML detection over ISI channels and the Viterbi equalizer. ↩

See also