Path: utzoo!utgpu!news-server.csri.toronto.edu!clyde.concordia.ca!uunet!tut.cis.ohio-state.edu!ucsd!network.ucsd.edu!weber!pbiron
From: pbiron@weber.ucsd.edu (Paul Biron)
Newsgroups: comp.sources.wanted
Subject: Re: removing duplicate lines from a text file???
Keywords: duplicate lines, sort(1)
Message-ID: <2309@network.ucsd.edu>
Date: 25 Mar 90 23:44:15 GMT
References: <1990Mar25.182039.25565@jarvis.csri.toronto.edu>
Sender: nobody@network.ucsd.edu
Reply-To: pbiron@weber.ucsd.edu (Paul Biron)
Organization: Division of Social Sciences, UCSD
Lines: 22

In article <1990Mar25.182039.25565@jarvis.csri.toronto.edu> stevep@dgp.toronto.edu (Steve Portigal) writes:
>
>    Is there any simple way to remove duplicate lines from a text file?  If
>anyone could suggest even what command to use, I would appreciate it.
>  thanks very much, Steve Portigal
>-- 
>******************************************************************
>I can't be held repsonsible for anything...I am brain dead from all
>this terminal time.
>******************************************************************
>"Black and Huge" -- GWAR

The easiest way is
    sort -u orig_file > new_file

The -u flag to sort(1) says "keep only the Unigue lines".


Paul Biron     (pbiron@ucsd.edu)       (619) 534-5758
Social Sciences DataBase Project, Central University Library
University of California, San Diego
La Jolla, Ca. 92093