Introduction
The uniq command is a powerful tool in Linux that can be used to remove duplicate lines from a file or stream. It is a very useful command for cleaning up data and making sure that only unique lines are present in a file. In this article, we will look at 8 examples of how to use the uniq command to remove duplicate lines in Linux. We will cover how to use the uniq command to remove duplicate lines from a file, how to ignore certain characters when comparing lines, and how to count the number of unique lines in a file. We will also look at how to use the uniq command to compare two files and how to use it to sort lines in a file. By the end of this article, you should have a good understanding of how to use the uniq command to remove duplicate lines in Linux.
8 Uniq Command Examples [Remove Duplicate Lines in Linux]
1. To remove duplicate lines from a file:
$ uniq file.txt
2. To remove duplicate lines and display only the unique lines:
$ uniq -u file.txt
3. To display only the duplicate lines:
$ uniq -d file.txt
4. To ignore case while comparing lines:
$ uniq -i file.txt
5. To ignore the first N characters while comparing lines:
$ uniq -f N file.txt
6. To ignore the last N characters while comparing lines:
$ uniq -s N file.txt
7. To display the number of times each line is repeated:
$ uniq -c file.txt
8. To display the lines that are not repeated:
$ uniq -u file.txt
[ad_1]
As Linux users, we interact with various types of files on a regular basis. One of the most common file types on any computer system is a plain text file. Oftentimes, it is a very common requirement to find the required text in these files.
However, this simple task quickly becomes annoying if the file contains duplicate entries. In such cases, we can use the uniq command to filter duplicate text efficiently.
In Linux, we can use the uniq command that comes in handy when we want to list or remove duplicate lines that present adjacently.
Apart from this, we can also use the uniq command to count duplicate entries. It is important to note that, the uniq command works only when duplicate entries are adjacent.
In this simple guide, we will discuss the uniq command in-depth with practical examples in Linux.
uniq Command Syntax
The syntax of the uniq command is very easy to understand and is similar to other Linux commands:
$ uniq [OPTIONS] [INPUT] [OUTPUT]
It is important to note that, all the options and parameters of the uniq command are optional.
Creating Sample Text File
To begin, first, let’s create a simple text file with a vi editor and add the following duplicate contents located in the adjacent lines.
$ vi linux-distributions.txt $ cat linux-distributions.txt
Now, let’s use this file to understand the usage of the uniq command.
1. Remove Duplicate Lines from a Text File
One of the common uses of the uniq command is to remove the adjacent duplicate lines from the text file as shown.
$ uniq linux-distributions.txt
In the above output, we can see that the uniq command has successfully eliminated the duplicated lines.
2. Count Duplicated Lines in a Text File
In the previous example, we saw how to remove duplicate lines. However, sometimes we also want to know how many times the duplicate line appears.
We can achieve this using the -c option as shown in the below example:
$ uniq -c linux-distributions.txt
In the above output, the first column represents the number of times the line is repeated.
3. Remove Duplicates with Case Insensitive
By default, the uniq command works in a case-sensitive way. However, we can disable this default behavior by using the -i option as shown.
$ uniq -i linux-distributions.txt
In this example, we can observe that now, the string Ubuntu and UBUNTU is treated as same. Along with this, the same happens with the string CentOS and CENTOS.
4. Print Only Duplicate Lines from a File
Sometimes, we want just want to print the duplicate lines from a text file, in that case, you can use the -d option as shown.
$ uniq -d linux-distributions.txt
In the above output, we can see that the uniq command shows the duplicate entry from each group.
5. Print All Duplicate Lines from a File
In the previous example, we saw how to display a duplicate line from each group. In a similar way, we can also show all the duplicates lines using the -D option:
$ uniq -D linux-distributions.txt
The above output doesn’t show the text UBUNTU, CENTOS, and Alpine as those are uniq lines.
6. Show Duplicate Lines By Groups in a New Line
In the previous example, we printed all duplicate lines. However, we can make the same output more readable by separating each group by a new line.
Let’s use the --all-repeated=separate option to achieve the same:
$ uniq --all-repeated=separate linux-distributions.txt
In the above output, we can see that each repeated group is separated by a new line delimiter.
7. Print Only Unique Lines from a File
In previous examples, we saw how to print duplicate lines. Similarly, we can also instruct the uniq command to print non-duplicate lines only.
Now, let’s use the -u option to print unique lines only:
$ uniq -u linux-distributions.txt
Here, we can see that the uniq command displays the lines that aren’t duplicated.
8. Remove Non-adjacent Duplicate Lines in File
One of the trivial limitations of the uniq command is that it only removes adjacent duplicate entries. However, sometimes we want to remove the duplicate entries regardless of their order in the given file.
In such cases, first, we can sort the file contents and then pipe that output to the uniq command as shown.
$ sort linux-distributions.txt | uniq
In this example, we used the sort and uniq commands without any options. However, we can also combine other supported options with these commands.
Conclusion
In this guide, we learned the uniq command using practical examples. Do you know of any other best example of the uniq command in Linux? Let us know your views in the comments below.
[ad_2]
8 Uniq Command Examples [Remove Duplicate Lines in Linux]
The uniq command is a useful tool for removing duplicate lines from a file or stream in Linux. It can be used to quickly identify and remove duplicate lines from a file, or to compare two files and identify any lines that are unique to each file. In this article, we will discuss 8 examples of the uniq command.
1. Remove Duplicate Lines from a File
The most basic use of the uniq command is to remove duplicate lines from a file. To do this, simply pipe the contents of the file into the uniq command:
cat file.txt | uniq > new_file.txtThis will create a new file, new_file.txt, which contains only the unique lines from the original file.
2. Count the Number of Duplicate Lines
The uniq command can also be used to count the number of duplicate lines in a file. To do this, use the -c option:
cat file.txt | uniq -cThis will output the number of duplicate lines in the file, along with the line itself.
3. Ignore Case When Comparing Lines
By default, the uniq command is case-sensitive when comparing lines. To ignore case when comparing lines, use the -i option:
cat file.txt | uniq -i4. Ignore Certain Fields When Comparing Lines
The uniq command can also be used to ignore certain fields when comparing lines. To do this, use the -f option, followed by the number of fields to ignore:
cat file.txt | uniq -f 2This will ignore the first two fields when comparing lines.
5. Output Unique Lines Only
By default, the uniq command will output both unique and duplicate lines. To output only the unique lines, use the -u option:
cat file.txt | uniq -u6. Output Duplicate Lines Only
To output only the duplicate lines, use the -d option:
cat file.txt | uniq -d7. Compare Two Files
The uniq command can also be used to compare two files and identify any lines that are unique to each file. To do this, use the -s option:
uniq -s file1.txt file2.txtThis will output any lines that are unique to each file.
8. Output the Number of Unique Lines
Finally, the uniq command can be used to output the number of unique lines in a file. To do this, use the -z option:
cat file.txt | uniq -zThis will output the number of unique lines in the file.
In this article, we discussed 8 examples of the uniq command. We discussed how to remove duplicate lines from a file, count the number of duplicate lines, ignore case when comparing lines, ignore certain fields when comparing lines, output unique lines only, output duplicate lines only, compare two files, and output the number of unique lines.
