Path: utzoo!utgpu!news-server.csri.toronto.edu!clyde.concordia.ca!uunet!cs.utexas.edu!wuarchive!rex!uflorida!haven!aplcen!wb3ffv!fallst!tkevans
From: tkevans@fallst.UUCP (Tim Evans)
Newsgroups: comp.sources.wanted
Subject: Re: removing duplicate lines from a text file???
Summary: Use 'uniq'
Keywords: duplicate lines, sort(1)
Message-ID: <1608@fallst.UUCP>
Date: 29 Mar 90 01:46:21 GMT
References: <1990Mar25.182039.25565@jarvis.csri.toronto.edu> <90Mar26.232441est.2199@smoke.cs.toronto.edu>
Organization: Fallston, MD
Lines: 16

In article <90Mar26.232441est.2199@smoke.cs.toronto.edu>, moraes@cs.toronto.edu (Mark Moraes) writes:
> >>>    Is there any simple way to remove duplicate lines from a text file?
> >>    sort -u orig_file > new_file
> 
From TFM:

	uniq reads the input file comparing adjacent lines.  In the
	normal case, the second and succeeding copies of repeated
	lines are removed; the remainder is written on the output file.

Note that this says _adjacent_ lines.

-- 
UUCP:		{rutgers|ames|uunet}!mimsy!woodb!fallst!tkevans
INTERNET:	tkevans%fallst@wb3ffv.ampr.org
Tim Evans	2201 Brookhaven Ct, Fallston, MD 21047  (301) 965-3286