Lesson 18 of 30 advanced 5 min read

Concurrency patterns

Key takeaways Three shapes cover most concurrent Go. A pipeline chains stages, each a goroutine reading one channel and writing the next. A worker pool runs a fixed number of goroutines off one job channel to cap concurrency. Fan-out/fan-in parallelizes a slow stage across many goroutines, then merges the results. All three are just goroutines and channels arranged deliberately.

Once you have goroutines, channels, and context, real programs assemble them into a few recurring structures. This lesson shows the shapes GopherTrunk’s sample pipeline is built from.

The pipeline

A pipeline is a chain of stages connected by channels. Each stage receives on its input, does one job, and sends on its output — then closes it:

func demod(in <-chan complex64) <-chan byte {
    out := make(chan byte)
    go func() {
        defer close(out)
        for s := range in {
            out <- symbol(s)
        }
    }()
    return out
}

symbols := demod(samples)   // stages compose: decode(demod(samples))

Each stage runs concurrently, so raw samples, demodulated symbols, and decoded frames all flow at once — exactly how an SDR engine keeps up with a live radio.

A pipeline: each stage is a goroutine that reads one channel and writes the next; all stages run at once.

The worker pool

When one stage is the bottleneck, run several copies of it — but a fixed number, so concurrency stays bounded:

jobs := make(chan Frame)
results := make(chan Decoded)

for i := 0; i < runtime.NumCPU(); i++ {   // N workers
    go func() {
        for f := range jobs {
            results <- decode(f)          // whichever worker is free takes the job
        }
    }()
}

The pool self-balances: fast jobs free a worker to grab the next one immediately. Cap the count at something sensible (often runtime.NumCPU()) rather than spawning one goroutine per job.

Fan-out and fan-in

Fan-out is starting several workers on the same input channel; fan-in merges their outputs back into one, typically with a sync.WaitGroup:

func fanIn(cs ...<-chan Decoded) <-chan Decoded {
    out := make(chan Decoded)
    var wg sync.WaitGroup
    for _, c := range cs {
        wg.Add(1)
        go func(c <-chan Decoded) {
            defer wg.Done()
            for v := range c {
                out <- v
            }
        }(c)
    }
    go func() { wg.Wait(); close(out) }()   // close once all inputs drain
    return out
}

Pattern	Shape	Use it when
Pipeline	stage → channel → stage	Work is a series of transforms
Worker pool	N goroutines, one job channel	You must cap concurrency
Fan-out/fan-in	split to many, merge to one	One stage is the bottleneck

Thread a context (previous lesson) through every stage so a single cancel tears the whole structure down cleanly.

Quick check: what makes a worker pool different from spawning a goroutine per job?

Recap

A pipeline chains goroutine stages with channels; each closes its output when done.
A worker pool runs a fixed number of goroutines off one job channel to cap concurrency.
Fan-out/fan-in parallelizes a slow stage, then merges with a WaitGroup.
Thread a context through every stage for clean shutdown.

Next up: how Go organizes code into packages and modules.

Frequently asked questions

What is a worker pool?

A fixed number of goroutines all reading jobs from one channel and writing results to another. It caps concurrency at a chosen size so you don’t spawn an unbounded number of goroutines, and it spreads work evenly because whichever worker is free grabs the next job.

What is fan-out/fan-in?

Fan-out starts several goroutines reading from the same input channel to parallelize a slow stage. Fan-in merges their several output channels back into one. Together they let one busy stage of a pipeline run on many cores while the rest stays a single stream.