Path: utzoo!attcan!utgpu!jarvis.csri.toronto.edu!mailrus!tut.cis.ohio-state.edu!pt.cs.cmu.edu!a.gp.cs.cmu.edu!koopman
From: koopman@a.gp.cs.cmu.edu (Philip Koopman)
Newsgroups: comp.lang.forth
Subject: Re: Benchmarks wanted
Summary: lies, damn lies, and benchmarks
Keywords: Performance measuring, profiling, debugging, playing around.
Message-ID: <7306@pt.cs.cmu.edu>
Date: 12 Dec 89 11:57:59 GMT
References: <1475@massormetrix.ida.liu.se>
Organization: Carnegie-Mellon University, CS/RI
Lines: 34

In article <1475@massormetrix.ida.liu.se>, mip@massormetrix.ida.liu.se (Mikael Patel) writes:
> I'am looking for some appropriate benchmarks for a C-based Forth I'am
> working on. I have run the sieve benchmark (both versions) and the
> fibonacci function and now looking around for some more.
> 
> Do you have any others? Please post them! 
> 
> Also how should we deal with comparison of threading methods, processor, 
> cache, clock rate, bit-width, etc? Some suggestions? For instance, a 
> poor threading method (as using C as the implementation language) can 
> be "compensated" for by using a faster processor e.g. workstation. 
> 
> Last, how should we construct Forth benchmarks? What properties do we
> want to measure? And how can we measure them accuratly?

Benchmarks are really tough in any language on any processor.
Strictly speaking, sieve and fib are more toys than benchmarks.
They fit completely into cache on almost any machine having
cache, and have a very limited mix of instructions.
Synthetic benchmarks such as Dhrystone aren't very useful,
because they are tuned to conventional languages such as C
and don't make sense as Forth code.

I don't have an answer for you, but you should know that
you are embarking on a long and arduous journey if you
are looking for real, meaningful benchmarks.  This is a problem
that even the conventional computing community has been struggling
with for a long time.

  Phil Koopman                koopman@greyhound.ece.cmu.edu   Arpanet
  2525A Wexford Run Rd.
  Wexford, PA  15090
Senior Scientist at Harris Semiconductor.
I don't speak for them, and they don't speak for me.