Xref: utzoo comp.ai.neural-nets:945 alt.cyb-sys:25
Path: utzoo!utgpu!jarvis.csri.toronto.edu!rutgers!sunybcs!bingvaxu!cjoslyn
From: cjoslyn@bingvaxu.cc.binghamton.edu (Cliff Joslyn)
Newsgroups: comp.ai.neural-nets,alt.cyb-sys
Subject: Re: Generalization Criteria
Message-ID: <2448@bingvaxu.cc.binghamton.edu>
Date: 20 Sep 89 00:10:44 GMT
References: <506@uvaee.ee.virginia.EDU>
Reply-To: cjoslyn@bingvaxu.cc.binghamton.edu.cc.binghamton.edu (Cliff Joslyn)
Organization: SUNY Binghamton, NY
Lines: 71

In article <506@uvaee.ee.virginia.EDU> aam9n@uvaee.ee.virginia.EDU (Ali Minai) writes:
>Given a *finite* set of input-output values, an estimator/approximator
>is used to induce a "reasonable" model for the system that generated
>the data. While there are many measures of "goodness of fit" with
>respect to the given data set (e.g. mean squared error), there seems
>to be no universally accepted measures for the "generalization"
achieved.

I'm not familiar with this specific application, but it seems likely to
me that it might be appropriate to use Maximum Entropy (MaxEnt) Theory. 
On this view, given that the input-output sets form a distribution, the
model is constructed so as to maximize the entropy relative to those
distributions, which act as constraints.  

The philosophy of this is that this is the *only* model which is
completely parsimonious relative to that data, that is uses exactly all
the data provided, making no further assumptions or ignoring any data. 
This view is strongly related to traditional information theory and
Bayesian statistics.  Most applications are in data analysis, especially
image reconstruction.

Christensen, Ronald: (1985) "Entropy Minimax Multivariate Statistical 
	Modeling", /Int. J. Gen. Sys./, v. 11

Cox, RT: "Probability, Frequency, and Reasonable Expectation",
     /American J. of Physics/, v. 14, pp. 1-13

Erickson, Gary J: , ed.(1988) /Maximum-Entropy and Bayesian Methods in
     Sci. + Eng./, v. 1,2, Kluwer

Jaynes, ET: (1957) "Information Theory and Statistical Mechanics",
     /Physical Review/, v. 106,108, pp. 620-630

     (1978) "Where Do We Stand on Maximum Entropy?", in: /Maximum Entropy
     Formalism/, ed. RD Levine, pp. 211-314, Cambridge, Cambridge,Mass

     (1990) /Probability Theory as Logic/, NOTE: To appear

Jumarie, Guy: (1987) "New Concepts in Information Theory", /Physica
     Scripta/, v. 35:2, pp. 220-224

     (1988) /Relative Information: Theories and Applications/,
     Springer-Verlag, Berlin

Kapur, JN: (1989) /Maximum Entropy Models in Science and Engineering/,
     John Wiley, New York

Polya, George: (1954) /Patterns of Plausible Inference/, Princeton U.
     Press, Princeton

Skilling, John: (1989) "Classic Maximum Entropy", in: /Maximum Entropy
     + Bayesian Methods/, ed. J. Skilling, pp. 45-52, Kluwer, New York

     , ed.(1989) /Maximum-Entropy and Bayesian Methods/, Kluwer

     (1990) "Quantified Maximum Entropy", in: /Proc. 9th MaxEnt Workshop/

Smith, C Ray: (1990) "From Rationality + Consistency to Bayesian
     Probability", in: /Proc. 9th MaxEnt Workshop/

Tribus, Myron: (1969) /Rational Descriptions, Decisions, and Designs/, 
	Pergamon, New York

Tribus, Myron, and Levine, RD: , eds.(1979) /Maximum Entropy
     Formalism/, MIT Press, Cambridge

-- 
O---------------------------------------------------------------------->
| Cliff Joslyn, Cybernetician at Large
| Systems Science, SUNY Binghamton, cjoslyn@bingvaxu.cc.binghamton.edu
V All the world is biscuit shaped. . .