Path: utzoo!utgpu!news-server.csri.toronto.edu!smoke.cs.toronto.edu!neat.cs.toronto.edu!moraes
Newsgroups: comp.sources.wanted
From: moraes@cs.toronto.edu (Mark Moraes)
Subject: Re: removing duplicate lines from a text file???
Message-ID: <90Mar26.232441est.2199@smoke.cs.toronto.edu>
Keywords: duplicate lines, sort(1)
Organization: Department of Computer Science, University of Toronto
References: <1990Mar25.182039.25565@jarvis.csri.toronto.edu> <2309@network.ucsd.edu> <3081@auspex.auspex.com>
Date: 27 Mar 90 04:25:45 GMT
Lines: 14

>>>    Is there any simple way to remove duplicate lines from a text file?
>>    sort -u orig_file > new_file
>Assuming, of course, that the order of the lines in the file isn't
>important.

In that case, perhaps something like

awk '{printf "%8d %s\n", NR, $0}' | 
	sort -u +1 | 
	sort -n | 
	sed 's/^.........//'

assuming, of course, that none of the lines are longer than the
maximum lengths your awk/sed can handle.