uniq, often used with
sort, allows a user to report, or filter, repeated lines that are adjacent in a file.
$ sort in.txt | uniq line1 line2 line3
Useful Options / Examples
Some common and useful flags include
-u for listing lines that are duplicated or that are unique, respectfully. As well as
-c for counting the number of times a line was duplicated.
$ sort in.txt | uniq -c 2 line1 3 line2 1 line3
-c flag counts the number of times an adjacent line was duplicated. It displays the count followed by the line.
$ sort in.txt | uniq -d line1 line2
-d flag lists only the lines in the file that had a duplicate line adjacent to it. Considering the counts from the above
-c example, we can see that only lines with counts greater than 1 were listed.
$ sort in.txt | uniq -u line3
-u flag lists only lines in the file that did not have a duplicate line adjacent to it. It is the inverse of the
$ sort in.txt | uniq -i line1 line2 line3
-i flag tells uniq to be case insensitive when comparing adjacent lines. With the
-i flag given, ‘line1’ and ‘LINE1’ will be conisered duplictes if adjacent in the file and which ever string appeared first in the file is the one that will be displayed.
$ sort in.txt | uniq -c -f 1 6 line1
-f num flag tells
uniq to ignore the first
num ‘fields’ in each line. A ‘field’ is a string in the line seperated by whitespace. In this example, since each line had only one field and we gave
num a value of 1, the command would compare only the empty space in each line, which of course would be duplicates.