Recording, Composition & Streaming, Part 11: Retention & Housekeeping

By

Part 11 of Recording, Composition & Streaming, following the 3 p.m. dispatch on talkgroup 101 from PCM to a Broadcastify upload. By now that call is two artifacts: a WAV the recorder wrote and a call_log row the persistence layer wrote. Multiply that by every call, every day, and a scanner left running fills its disk. This post is about the janitor — the one background job whose entire purpose is to make sure that yesterday’s dispatch doesn’t outlive its welcome, without ever touching a file it wasn’t told to.

TL;DR: storage.Retention is a single idempotent sweeper that runs at startup and then every Interval. It has three independent lanes — old call_log rows, old rows in an allow-listed set of decoder-log tables, and old .wav / .raw files under a FilesRoot — and each lane runs only if its own max-age is set. File deletion is doubly opt-in: it needs both a FilesRoot and a non-zero FilesMaxAge, and it deletes only recording artifacts by extension, never a config or CSV parked nearby. The daemon builds the sweeper at all only when at least one retention window is configured.

Key takeaways

  • Three lanes, three gates. Call rows, decoder-log tables, and files are each swept only when their respective max-age is greater than zero. A zero age is a disabled lane, not “delete everything.”
  • Opt-in by config. No retention: window in the YAML → the daemon never constructs a Retention at all. File deletion additionally requires an explicit FilesRoot.
  • Deletes only recording artifacts. The filesystem walk removes a file only if its extension is .wav or .raw and its mtime is past the cutoff — configs and talkgroup CSVs in the same tree are left alone.
  • Idempotent and crash-tolerant. Every sweep recomputes cutoffs from time.Now(); errors are logged and swallowed so a transient FS or DB hiccup never kills the loop.

Cheat sheet

Thing What it does Where
Retention The sweeper: three age-gated deletion lanes internal/storage/retention.go
RetentionOptions Per-lane max-ages, FilesRoot, Interval internal/storage/retention.go
SweepOnce Runs the configured deletions once internal/storage/retention.go
deleteOldRows DELETE FROM call_log WHERE started_at < cutoff internal/storage/retention.go
deleteOldLogRows / deleteOldByColumn Age-out one decoder-log table by its timestamp column internal/storage/retention.go
deleteOldFiles / isRecordingArtifact filepath.Walk + extension + mtime check internal/storage/retention.go
decoderLogTables The allow-list of sweepable log tables internal/storage/retention.go
daemon wiring Builds Retention only if a window is set cmd/gophertrunk/daemon.go

In this post

  • The shape of a sweeper — Run, SweepOnce, and the three gated lanes.
  • Aging out rows — the call-log lane and the decoder-log allow-list.
  • Aging out files — the walk, the extension guard, and the mtime cutoff.
  • Opt-in by config — how the daemon decides whether a sweeper exists at all.
  • Idempotence — why re-running the sweep is always safe.

A janitor, not a pipeline

Retention is the one subsystem in this series that isn’t a bus subscriber. It reacts to a timer, not to call events — nothing about a call’s life triggers a sweep. Run does one sweep immediately at startup (so a daemon that was down over a retention boundary catches up on boot) and then ticks on Interval:

// internal/storage/retention.go (shape)
func (r *Retention) Run(ctx context.Context) error {
    r.SweepOnce(ctx)
    tick := time.NewTicker(r.interval)
    defer tick.Stop()
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-tick.C:
            r.SweepOnce(ctx)
        }
    }
}

SweepOnce is the whole policy in one function. It has three lanes, and each is guarded by a two-part check: the relevant backend must exist (r.db != nil, or a non-empty r.files) and that lane’s max-age must be positive. A zero age means the lane is off:

// internal/storage/retention.go (shape)
func (r *Retention) SweepOnce(ctx context.Context) {
    if r.db != nil && r.dbAge > 0 {
        r.deleteOldRows(ctx) // call_log rows
    }
    if r.db != nil && r.logAge > 0 {
        for _, table := range decoderLogTables {
            r.deleteOldLogRows(ctx, table) // pager_log, aprs_log, …
        }
    }
    if r.files != "" && r.filesAge > 0 {
        r.deleteOldFiles() // .wav / .raw under FilesRoot
    }
}

That two-part gate is the design’s spine: presence of a backend is not consent to delete. A daemon with a database attached but no CallRowMaxAge sweeps zero rows. Errors inside each lane are logged and swallowed — a locked file or a transient DB error must not tear down the ticker — so the sweeper self-heals on the next tick.

SweepOnce timer-driven, not events call_log rows gate: dbAge > 0 decoder-log tables gate: logAge > 0 · allow-list .wav / .raw files gate: FilesRoot ≠ "" & filesAge > 0 started_at < cutoff received_at < cutoff mtime < cutoff Each lane is independent — a zero max-age disables just that lane.
Three sweep lanes, three independent gates. A missing max-age silences one lane without affecting the others; nothing here is all-or-nothing.

Aging out rows

The database lanes are almost too simple to describe, which is the point. The call-log lane computes a cutoff from now on every sweep and deletes anything older:

// internal/storage/retention.go (shape)
func (r *Retention) deleteOldRows(ctx context.Context) (int64, error) {
    cutoff := time.Now().Add(-r.dbAge).UnixNano()
    res, err := r.db.sql.ExecContext(ctx,
        `DELETE FROM call_log WHERE started_at < ?`, cutoff)
    // …return RowsAffected…
}

TestRetentionDeletesOldRows pins the boundary: with a 24-hour age, a 25-hour-old row is deleted while 10-hour and 1-hour rows survive.

The decoder-log lane is where the allow-list matters. GopherTrunk decodes far more than trunked voice — POCSAG pagers, APRS, AIS vessels, DSC, ADS-B aircraft, MDC1200, M17 — and each writes an append-only log table keyed by a received_at nanosecond column. The sweeper knows exactly which tables it may touch:

// internal/storage/retention.go (shape)
var decoderLogTables = []string{
    "pager_log", "aprs_log", "vessel_log", "dsc_log",
    "aircraft_log", "mdc1200_log", "m17_log",
}

deleteOldLogRows is a thin wrapper over deleteOldByColumn(ctx, table, "received_at"), which interpolates the table and column names directly into the DELETE. That string-building is safe precisely because the inputs are in-code constants — the allow-list slice or a literal — never user input. The comment in the source says so explicitly, and it’s the reason location_log (which keys on reported_at, not received_at) is swept by a separate call rather than being bolted into the loop. TestRetentionLogSweepEveryTable runs the delete against every allow-listed table plus the location_log special case, so a typo or a table missing its timestamp column would fail the build’s tests.

Note the two log lanes share one logAge: every decoder-log table ages out on the same window. TestRetentionLogSweepDisabledByDefault confirms the gate — with CallRowMaxAge set but LogRowMaxAge zero, a 100-day-old m17_log row survives untouched.

Aging out files

The filesystem lane is the one that needs the most care, because a wrong deletion here is unrecoverable and the recordings directory is a place operators park other things. deleteOldFiles walks FilesRoot and applies two filters to every file it finds — an extension check and an mtime check — deleting only when both pass:

// internal/storage/retention.go (shape)
func (r *Retention) deleteOldFiles() (int, error) {
    cutoff := time.Now().Add(-r.filesAge)
    return filepath.Walk(r.files, func(path string, info os.FileInfo, err error) error {
        if err != nil { return nil }        // tolerate transient FS errors
        if info.IsDir() { return nil }
        if !isRecordingArtifact(path) { return nil } // only .wav / .raw
        if info.ModTime().After(cutoff) { return nil } // still fresh
        os.Remove(path)
        return nil
    })
}

func isRecordingArtifact(path string) bool {
    switch strings.ToLower(filepath.Ext(path)) {
    case ".wav", ".raw":
        return true
    }
    return false
}

The extension guard is deliberate defensive coding. As the source comment notes, it’s there so the sweep doesn’t clobber configs or talkgroup CSVs an operator parked near the recordings. TestRetentionDeletesOldFiles makes the point concretely: in a tree containing an old .wav, an old .raw, a fresh .wav, and an old config.yaml, the sweep removes the two old artifacts, keeps the fresh one, and leaves the config.yaml alone even though it’s well past the cutoff — the extension check saves it.

Two more details worth noting. The walk swallows per-entry errors by returning nil from the walk function, so one unreadable directory doesn’t abort the whole sweep. And the mtime comparison uses the file’s own modification time, not any database record — the file lane needs no DB at all, which is why NewRetention accepts a FilesRoot-only configuration with no DB.

walk entry IsDir? yes skip no ext .wav / .raw? no preserve yes mtime > cutoff? yes (fresh) skip no (old) os.Remove
The per-file decision tree. A file is deleted only after passing both the extension guard and the mtime cutoff; every other path preserves it.

Opt-in by config

None of this exists unless the operator asks for it. In cmd/gophertrunk/daemon.go, the sweeper is constructed only when at least one retention window is set:

// cmd/gophertrunk/daemon.go (shape)
if cfg.Retention.CallLogDays > 0 ||
    cfg.Retention.LogDays > 0 ||
    cfg.Retention.FilesDays > 0 {
    interval, _ := retentionInterval(cfg.Retention.Interval)
    r, _ := storage.NewRetention(storage.RetentionOptions{
        DB:            db,
        FilesRoot:     cfg.Recordings.Dir,
        CallRowMaxAge: time.Duration(cfg.Retention.CallLogDays) * 24 * time.Hour,
        FilesMaxAge:   time.Duration(cfg.Retention.FilesDays) * 24 * time.Hour,
        Interval:      interval,
    })
    d.retention = r
}

This is the same config-driven lazy init rule that runs through the whole output half (see Part 1): with no retention days configured, d.retention stays nil and the daemon never spawns the sweep goroutine. The FilesRoot is wired from the recordings directory, so file deletion piggybacks on wherever the recorder is already writing — but only takes effect when FilesDays (hence FilesMaxAge) is non-zero, keeping file deletion doubly opt-in. retentionInterval parses the optional interval string, defaulting to one hour when it’s empty, mirroring NewRetention’s own default.

NewRetention itself refuses a pointless configuration: TestRetentionRequiresAtLeastOne confirms it errors when neither a DB nor a FilesRoot is supplied — there’d be nothing to sweep.

Idempotence

The reason this sweeper can run at startup, on every tick, and even concurrently with the writers is that it holds no state between runs. Every lane recomputes its cutoff from time.Now() at the moment it executes; a DELETE ... WHERE ts < cutoff that finds nothing to delete is a no-op, and re-deleting an already-deleted row or file simply affects zero rows or is skipped by the walk. There is no “mark-then-sweep” bookkeeping to get out of sync.

Concurrency with the call log writer is safe because SQLite serialises writers — the store caps the pool to a single connection, so the sweeper’s DELETE and the call log’s UPDATE take turns rather than racing. TestRetentionRunStopsOnContextCancel confirms the loop exits cleanly on context cancellation, returning the cancellation error like every other long-lived subsystem in the daemon.

Where this goes next

That’s the persistence and housekeeping pair complete — the call is now recorded, indexed, and destined to be cleaned up on schedule. The next posts turn to the live side of the output half. Part 12 covers live listening: how the same decoded PCM that fills the WAV is fanned out to a browser or gRPC listener in real time, with no file and no database involved.

FAQ

Does GopherTrunk delete my recordings by default? No. File deletion is doubly opt-in: it needs both a recordings directory (FilesRoot) and a non-zero retention.files_days. With no retention: window configured at all, the daemon never even constructs the sweeper, so nothing is ever deleted.

Will the sweeper remove non-recording files in the recordings folder? No. isRecordingArtifact only matches .wav and .raw extensions. A config.yaml, a talkgroup CSV, or any other file in the same tree is skipped regardless of age — the extension guard exists specifically to protect files operators park nearby.

How do the three retention lanes relate? They’re independent. Call-row age (call_log_days), decoder-log age (log_days), and file age (files_days) each gate their own lane; setting one does nothing to the others, and a zero value disables just that lane rather than meaning “delete everything.”

Is it safe to run retention while calls are being logged? Yes. The sweep is idempotent — it recomputes cutoffs from the current time each run and re-running deletes nothing already gone — and SQLite serialises writers against a single connection, so the sweeper’s deletes and the call log’s writes take turns instead of racing.

Series navigation

Part 11 of 14 · ← Part 10: The Call Log — SQLite Persistence & Never-Downgrade Updates · Next → Part 12: Live Listening

#storage#retention#housekeeping#sqlite#go#filesystem