Pipeline and Redirection in Linux


In the Linux operating system, Pipeline and Redirection are mostly useful and powerful techniques that helps in sending the output of one program to a file or as an input of another program. These two techniques are highly recommended to be learned as a beginner because they are needed to write scripts and other stuff in day-to-day life with Linux when you are in immediate stage. With the previous articles you had already worked with “|” which is pipeline even without being aware of it.

First we need to learn the three data streams in Linux operating system.  Simply they are as below.

  • STDIN (0) – Standard input represents the way data is input to the the program
  • STDOUT (1) – Standard output represents the output of the program ( what screen displays )
  • STDERR (2) – Standard error represents the errors of the output
Above three data streams in a rough image is as below.
Data streams on Linux OS. STDIN, STDOUT, STDERR
STDOUT – In all of the Linux operating systems, the output of a command ( program ) is going to standard output stream which is the screen or the display. This can be changed with the help of pipeline and redirection.
STDIN – Normally standard input is the stream of data inputting to the program or the console. Mostly that is your keyboard input. BTW this also can be changed with pipeline and redirection technique.
STDERR – Standard error stream is the data line which delivers errors of your program. Normally if you got errors in your program output, console displays the errors in your display. But with redirection we can pass the errors to another file without displaying on the screen.

Pipeline ( Pipes )

It’s symbol is “|” . We can use pipeline to send the output of one program to another program as an input. Simply, Pipes allows to send one commands output as an input of another command. The second one should be a command, it can’t be a file. 
eg_1 :
Here we have created a directory named “pipeline” and inside of it there are a lots of text files and png files have created.

And if we need to list all those files with command “ll“, it will display only the last few files in the display. But we need to list them from the top.

We can use pipeline to display all the files from the top. The command is “ll | less” or “ll | more”. What does this command do is ? The output of the ll command is sent to the command less as an input with the help of pipeline. So less command takes that input and displays the output from the top of the results.

* I will discuss another Linux command that will be helpful for you and also which can integrate with pipeline.
How to count the number of lines, word and ext…
We can we use the command “wc” for counting purposes in Linux.
syntax : wc {options} filenames
  1. wc -l: Prints the number of lines in a file
  2. wc -w: Prints the number of words in a file
  3. wc -c: Prints the count of  bytes of a file
  4. wc -m: Prints the count of characters of a file
eg_2 :
If we need to find how many files in a directory, we can use “ll | wc -l“.
wc command in Linux
Above is another way of using pipeline with the commands. ll command lists the files and it’s output will be sent to wc -l command as an input. So number of lines will be displayed.
Summary of Pipeline :
|  sends the output of one program to as an input of another program. 


Redirection is very similar to Pipe technique. But except sending the output to a program, Redirection sends the output to a file. Normally as we discussed in the top of this article standard output of a command ( program ) is the display. But with redirection we can send the output to another file using “>” and “>>“.
eg_3 : 
In Linux operating system, /etc/passwd file is containing all the details about it’s users. We will discuss about those files in next articles.
Guess we need to copy the content of /etc/passwd file to a new file as my_password_file.
First we will check our current /etc/passwd file with command cat /etc/passwd
/etc/passwd file in linux

In order to get the content of  /etc/passwd to a new file named “my_password_file”, you can use the command as below.

Redirection in Linux
 So, Redirection helps to send the output of a command to be stored in another file as above. 
* If we can use redirection with a single “>”, why do we need “>>” ? It is for appending. Which means, When we use a single > , it tells the command to send the output to the mentioned file. But >> will tell the command to sent the content without deleting the content of the new file if it was not blank. >> will copy the content to the new file without replacing it when > will do the copy while replacing the new file.
We will discuss it with an example.
eg_4 : 
Firstly, we will create a backup file of our already created file “my_password_file”.
Before continuing the above example, let me teach you another important command in Linux.
echo : This command prints on the display whatever you writes .
syntax : echo “Your text”
echo command in Linux
Back to our example ex_4,
We will send a simple message to “my_password_file”.
* As above, with a single “>”, echo will send some text to my_password_file and single > has replaced the old content with new content.
Here, we will use  “>>” for redirection.

And the output will be as below. >> will be used for appending. The old content will not be deleted or replaced with the new content. “Hello World” has appended with my old content.

Summary of Redirection :
> will send the output to  a new file
>> will append the output to a new file
On the top of this article, we talked about three data streams. There we noted STDERR ( standard error ).
ex_5 : 
We will try to find some log files under root directory. The find commands will be discussed in future articles.
find / -iname *.log will find all the files ending with .log extention under / directory.
find command in Linux

The output will be as below. You can see there are errors on the display. This is the standard error.

With redirection we can send the errors to another file or to /dev/null ( /dev/null is a garbage location in linux. Whatever file passed to /dev/null will be destroyed )
stderr data stream in Linux

On the above is the way of calling for standard errors to be sent to another file. There 2> errorfile, 2 is used because standard errors are identified with number 2 in data streams of Linux OS. So, The above command will ask to find all log files under / and send all the errors to errorfile.

The errorfile has stored the standard error output of the above find command.

* You could use the below example as well  for sending errors to /dev/null
find / -iname *.log 2> /dev/null
Now you have a clear idea about Pipeline and Redirection. Additionally this article discussed about wc and echo command. Please refer more and more on these topic.
Now i guess you should have gained some knowledge about pipeline and redirection in linux. Read more and keep growing.
Other resources:

Leave a Reply