Pipes & redirection

Key takeaways Every command has three streams: stdin (input), stdout (normal output), and stderr (errors). Redirection wires those streams to files — > overwrites, >> appends, and < feeds a file in as input. The pipe | wires one command’s output straight into the next command’s input. This is where the shell stops being a place to type single commands and becomes a place to build things: chain small tools together and send their output to and from files.

Up to now each command has stood alone: you type it, it prints to the screen, done. The real power of the shell is connecting commands to each other and to files. Once this clicks, a single line can answer questions that would take a whole program otherwise.

Three streams

Every command you run has three standard streams attached to it:

  • standard input (stdin) — where the command reads from.
  • standard output (stdout) — where its normal results go.
  • standard error (stderr) — a separate channel for error and warning messages.

By default, input comes from your keyboard and both output streams go to your screen. That is why a command just prints its results and you just type at it — the streams are pointed at you. Redirection and pipes are nothing more than pointing those streams somewhere else.

Errors travelling on their own stream is the clever part: it means you can capture a command’s results in a file while its error messages still appear on screen, or send each to a different place.

Redirecting output

The > operator sends stdout to a file instead of the screen. It overwrites the file — whatever was there is gone:

ls > files.txt        # save the listing to files.txt (replaces it)

Use >> to append instead, adding to the end and keeping the existing contents:

echo "run started" >> log.txt   # add a line without wiping the file

Errors ride the separate stderr stream, so > does not capture them. Redirect those with 2> (the 2 is stderr’s stream number):

find / -name '*.cfile' 2> errors.log   # results to screen, errors to a file

To capture normal output and errors together in one file, use &>:

make test &> build.log          # everything, stdout and stderr, into build.log

Redirecting input

The mirror image of > is <, which feeds a file in as a command’s stdin — as if you had typed the file’s contents at the keyboard:

sort < names.txt      # sort reads its input from the file

You will reach for < less often than the others, because most tools also accept a filename directly, but it is worth recognising when you see it.

Pipes

The pipe | is the star. It connects one command’s stdout directly to the next command’s stdin — no file in between. The output of the left-hand command becomes the input of the right-hand command:

ls | less             # page through a long listing instead of it scrolling past
cat log.txt | grep ERROR   # keep only the lines containing ERROR

You can chain as many stages as you like, each one transforming what the last handed it. That is how you compose small tools into something bigger without writing a single line of code.

The Unix philosophy

There is a design idea behind all of this: write small programs that each do one thing well, then combine them with pipes. Rather than one giant program with a setting for everything, Unix gives you a toolbox of sharp little tools — one lists files, one filters lines, one counts, one sorts — and the pipe lets you snap them together for the job in front of you.

This is the shell’s real superpower, and it is why the command line stays useful for tasks nobody anticipated. The next lesson on text-processing tools is where those sharp little tools really pay off.

Worked examples

A pipeline is best understood by reading it left to right, one stage at a time. Count how many lines in a log mention an error:

cat log.txt | grep ERROR | wc -l

Read it as a sentence: cat pours the file into the pipe, grep ERROR keeps only the matching lines, and wc -l counts what survives. The screen shows a single number — the answer to “how many errors?”.

Here is another, finding which processes are using the most of something without scrolling by hand:

ps aux | grep gophertrunk

ps aux lists every running process; grep gophertrunk throws away every line that does not mention it. You asked a real question and answered it in one line — and each piece was a tool you already knew.

Quick check: what does ls | grep .txt do?

Recap

  • Every command has three streams: stdin, stdout, and stderr.
  • > writes stdout to a file (overwrites); >> appends to it.
  • 2> redirects errors; &> captures both output and errors together.
  • < feeds a file in as a command’s input.
  • The pipe | connects one command’s output straight into the next’s input.
  • Small tools chained with pipes — the Unix philosophy — is the shell’s real power.

Next up: text-processing tools