You can combine the sort and uniq command line programs to remove duplicate lines from a text file. In the example below, it will produce a new file without the dups while preserving the original.
sort original.txt | uniq > dups_removed.txt
You can combine the sort and uniq command line programs to remove duplicate lines from a text file. In the example below, it will produce a new file without the dups while preserving the original.
sort original.txt | uniq > dups_removed.txt