UniQ

Uniq is a filter utility found on Unix and Unix-like systems. It takes input text and outputs only unique lines by removing adjacent duplicates. Because it only detects duplicates that appear consecutively, uniq is most effective when the input is sorted or otherwise arranged so that identical lines are next to each other.

Usage generally follows the form: uniq [OPTION]... [INPUT [OUTPUT]]. If INPUT is omitted, uniq reads from standard input. If OUTPUT is omitted, the results are written to standard output. A common pattern is to use uniq in a pipeline with sort to collapse all duplicates in a file.

Common options include:

- -c, which prefixes each output line with the number of occurrences.

- -d, which prints only lines that have duplicates.

- -u, which prints only lines that are unique (no duplicates).

- -i, which performs case-insensitive comparisons.

- -f N, which ignores the first N fields when determining duplicates.

- -s N, which ignores the first N characters when determining duplicates.

Examples:

- Remove duplicates from a sorted file: sort data.txt | uniq > unique.txt

- Count occurrences of each line: sort data.txt | uniq -c

- Show only lines that appear more than once: sort data.txt | uniq -d

Notes:

Uniq does not remove non-adjacent duplicates unless the input is sorted or otherwise arranged. It is included in POSIX and is implemented in various forms across GNU coreutils, BSD, and other Unix-like systems. For files with non-adjacent duplicates, a combination with sort (and sometimes additional processing) is typically used.