Path: utzoo!utgpu!news-server.csri.toronto.edu!rutgers!uwm.edu!ux1.cso.uiuc.edu!ux1.cso.uiuc.edu!m.cs.uiuc.edu!forbus
From: forbus@m.cs.uiuc.edu
Newsgroups: comp.ai
Subject: Re: AAAI Reviews
Message-ID: <3200025@m.cs.uiuc.edu>
Date: 17 Jul 90 03:22:00 GMT
References: <13@<Jul>
Lines: 73
Nf-ID: #R:<Jul:13:m.cs.uiuc.edu:3200025:000:3920
Nf-From: m.cs.uiuc.edu!forbus    Jul 16 22:22:00 1990


Speaking as a former program chair for AAAI, let me stick in the
following $0.02.  [These opinions are, of course, my own, and should
not be construed as an official statement of the AAAI.]

0. If you think you have received an irresponsible review, PLEASE get
in touch with the program chair(s).  Such feedback is viewed as
extremely important.  While it may or may not affect the outcome in
your particular case, there is deep concern about quality control in
the reviewing process.  For example, reviewers who screw up often end
up getting dropped from future program committees.

1. When I was program chair, my very own paper was rejected.  If it is
a "closed shop" or an "old boy's network", it certainly didn't work
very well for me!  (And, yes, I thought it was a great paper.
Furthermore, it had been flushed the year before and I carefully
re-wrote it along the lines the referees had suggested.  Still lost.
See point 4 below.)

2. Writing conference papers is a special skill, akin to writing
haiku.  A good conference paper, due to space limitations, is
typically about one idea and some of its ramifications.  System
descriptions rarely work. (The paper mentioned above finally saw the
light of day as a journal article.  There was just no way it could fit
in the constraints of a conference paper.)  Empirical papers where the
experiments don't all lead to some coherent point often don't work.

And space limitations aren't the only problem.  The time constraints
means that, unlike a journal, if it isn't pretty much all there the
first time, there won't be time to fix it.  Now, add to these
constraints the fact that people tend to finish their papers at the
last minute (50% of them arrive the day of the deadline, on the
average), and you see why writing for this forum is tough.

Empirically, many AAAI (and IJCAI) submissions are poorly written.
One may say this is a "subjective matter", but that's dodging the
issue.  In point of fact, conference papers really need to be
understandable by the reviewers, who after all are part of the
community one hopes to address.

3. In my experience, the reviewing provided by AAAI usually results in
more accurate appraisals than the more distributed IJCAI method.  The
face-to-face approach really helps reviewers converge more accurately.
Many conflicts get resolved by finding another expert in the area and
having them read the paper, and either providing extra advice or
writing an additional review.  (The most controversial case garnered
five reviews.  I don't remember if the paper got in or not.)

The methodological bias issue, to me, is a red herring.  I'm sure
there are biases.  I'm also sure that everyone who reviews works very
hard to make sure that their biases are kept under control.
Interestingly, I've seen many examples of people from radically
different "camps" in a subarea of AI agree, right down the line, on
accept/reject decisions on papers in their area.

So, personally, I'm much less likely to believe a paper was rejected
for dogmatic reasons, and more inclined to believe that the referees
just honestly thought it wasn't over threshold for that conference.  I
often get papers rejected.  It doesn't feel good.  Sometimes in
retrospect I see what the reviewers meant.  Sometimes I don't.  While
I have seen cases of irresponsible reviews, in my experience they are
extremely rare.

4. In any filtering system some noise is inevitable.  Ultimately, you
have to realize that there are simply honest differences of opinion.
People differ on when a point is well-enough established, on what
constitutes sufficient rigor, on what kinds of discussion of potential
impact and literature survey suffices, etc.  Usually the really great
papers all get accepted, the really bad papers all get rejected, and
the stuff in between gets sorted out more or less correctly.  That's
really the best one can hope for, I think.

	Ken Forbus