Path: utzoo!utgpu!jarvis.csri.toronto.edu!rutgers!usc!elroy.jpl.nasa.gov!jato!herron.uucp!jbrown
From: jbrown@herron.uucp (Jordan Brown)
Newsgroups: comp.databases
Subject: Re: dBASE IV again - a reply to a response (and vice versa :-))
Message-ID: <102@herron.uucp>
Date: 25 Jul 89 07:21:38 GMT
References: <Jul.22.21.05.45.1989.3462@yes.rutgers.edu> <9467@xanth.cs.odu.edu> <3520004@hpindda.HP.COM> <Jul.18.20.09.45.1989.2100@yes.rutgers.edu> <349@rls.UUCP>
Reply-To: jbrown@jato.jpl.nasa.gov
Lines: 72

emuleomo@yes.rutgers.edu (Emuleomo) writes:
> Although I don't know why you are experiencing that problem,
> I DO KNOW that Filters are a VERY BAD IDEA.  They are performance HOGS.
> If you remove the filter statement and simulate it, you will be SURPRISED 
> how much FASTER your application will run!!!

People keep saying this, and it just isn't true.  Filters are stupid, yes,
but they aren't any slower than doing the SAME STUPID THING that they do.
If you truly simulate a filter, you'll get the same speed (only slower).

Sure, if you use indexes you'll win big.  If you "know" that after
a certain point you can stop examining records you'll win big.
But for a general-purpose function with no knowlege of the behavior of
the query, filters don't do anything particularily horrible.  With some
clever query optimization they could be a lot better.  They'd probably
be buggy and require more memory too.

SET FILTER TO <cond>
LIST

should be *exactly* as fast as

LIST FOR <cond>

Similarly, 

SET FILTER TO <cond>
GO TOP
DO WHILE .NOT.EOF()
   ... process ...
   SKIP
ENDDO

should be *exactly* as fast as

LOCATE FOR <cond>
DO WHILE .NOT.EOF()
   ... process ...
   CONTINUE
ENDDO

and faster than "simulating" it by

GO TOP
DO WHILE .NOT.EOF()
   IF <cond>
      ... process ...
   ENDIF
   SKIP
ENDDO

(Which, by the way, is nothing like a full simulation as it doesn't
handle backing up, seeking, etc.)

A filter is an absolutely abysmal way to pick out a particular invoice from
a line items table.  It's not a bad way, however, to pick out all the
people in California in name order using an existing "last+first" index.

Filters are no stupider or slower than FOR clauses or LOCATE commands.
I guess people just think that because they look like they happen at
a lower level they should be magical.

(Disclaimer, sort of... I don't have anything to do with AT any more,
other than having some friends there.  I'm even a competitor in some
senses.  However, I had a fair amount to do with the initial
implementation of filters in dBASE III, and I'm tired of people griping
that filters are slow when they are certainly NO WORSE than any of the
other equivalent things in the language.  They aren't the magical
solution to all your searching needs.  They are the WRONG WAY to
do many things.  If you want to argue that filters and FOR should
do query optimization, fine, but recognize that query optimization
is still an unsolved problem and is not trivial to do.)