Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!sdd.hp.com!ucsd!ucbvax!ulysses!ether From: ether@tzero.usa (David Etherington) Newsgroups: comp.ai Subject: Re: AAAI Reviews Message-ID: Date: 18 Jul 90 20:32:54 GMT References: <82157@tut.cis.ohio-state.edu> Sender: netnews@ulysses.att.com Organization: /usr/ether/.organization Lines: 35 In-reply-to: byland@iris.cis.ohio-state.edu's message of 16 Jul 90 15:15:23 GMT There are lots of problems with the AAAI reviewing process, but perhaps things aren't as bad as they may seem annectdotally. First of all, all the annectdotes you hear are bad--nobody tells you how good the reviews they got were! Secondly, each reviewer meets with the other reviewer(s) face to face and discusses the paper (in some cases, assuming that they didn't already agree with a cut and dried accept or reject) to hash out a decision. Thus, given paper where one reviewer who thinks there is no content and the other thinks it is acceptable, some negotiation goes on. Thus, you have to have 2 irresponsible reviewers to get an irresponsible JUDGEMENT. Unfortunately, the process doesn't then FORCE the reviewers to go back and write a good review. So you end up with reviews that don't reflect the final judgement adequately. The area chairs are supposed to try to check the reviews to filter out the inflammatory ("Why do you keep sending in this kind of crap?" ones, but it doesn't always work (there were about 350 reviews done in 2 days in the KR area!). However, there were at least a couple of irresponsible reviewers (IMHO) this year, and they may have shared a couple of papers. There should be a mechanism to review reviewers (publicly? :-) so that the flakes get weeded out. Don't know how you could do it in such a way that you'd be free of lawsuits, though. All in all, the AAAI review process seems much better than the IJCAI process, taken in the large, since there is confrontation on the reviews. I rarely see what I'd consider a well-written paper with good content rejected. (If anything, I see papers accepted that need lots of work.) Unfortunately, less-well-written but significant papers sometimes suffer. Maybe that should encourage us to write better, rather than trying to get the paper together in time to make the Monday FedEx deadline!