Xref: utzoo comp.ai:5032 comp.ai.neural-nets:1087
Path: utzoo!attcan!utgpu!jarvis.csri.toronto.edu!rutgers!tut.cis.ohio-state.edu!purdue!haven!mimsy!brillig.umd.edu!lynne
From: lynne@brillig.umd.edu (Lynne D'Autrechy)
Newsgroups: comp.ai,comp.ai.neural-nets
Subject: NETtalk results
Summary: two different criterion used to judge performance
Keywords: Neural Networks, NETtalk
Message-ID: <20676@mimsy.umd.edu>
Date: 10 Nov 89 13:06:58 GMT
References: <1690@cod.NOSC.MIL> <77404@linus.UUCP> <13659@orstcs.CS.ORST.EDU> <1989Nov9.160406.14658@Neon.Stanford.EDU>
Sender: news@mimsy.umd.edu
Reply-To: lynne@brillig.umd.edu.UUCP (Lynne D'Autrechy)
Distribution: usa
Organization: U of Maryland, Dept. of Computer Science,
Lines: 23

In a JHU TR (EECS-86/01) published before the Sejnowski and Rosenberg Complex 
Systems paper, they talk about two criteria that were used to judge the 
performance of the network.  The first criterion is a "perfect match" criterion
and the second criterion is a "best guess" criterion.  Quoting from the TR,

	The output was considered a "perfect match" if the value of
	each articulatory feature was within a margin of 0.1 of its
	correct value.  This was a much stricter criterion than the 
	"best guess", which was the phoneme making the smallest angle
	with the output vector.


As reported in the TR, two types of input were used -- continuous informal
speech and words taken from the dictionary.  For the first type of input,
informal speech, the percentage of correct best guesses after learning was
95% while the percentage of perfect matches was 55%.  For the second type
of input, with a network of 120 hidden units, the best guess performance
was 98% while the perfect match performance was about 52%.

Only the "best guess" statistics were reported in the Complex Systems
article.  In summary, the impressiveness of the results achieved by
NETtalk depends on which criterion you use to judge the performance
of the network.