Path: utzoo!utgpu!news-server.csri.toronto.edu!mailrus!tut.cis.ohio-state.edu!zaphod.mps.ohio-state.edu!usc!rutgers!aramis.rutgers.edu!nash.rutgers.edu!mccarty From: mccarty@nash.rutgers.edu (Thorne McCarty) Newsgroups: comp.ai Subject: AAAI Reviews Message-ID: Date: 13 Jul 90 15:50:25 GMT Organization: Rutgers Univ., New Brunswick, N.J. Lines: 246 Was your paper rejected by AAAI this year? Were the reviews, in your opinion, incompetent and irresponsible? I was so outraged by one review of our paper that I sent a long message of protest to Tom Dietterich, the Program Co-Chair. Dietterich apologized for the poor quality review, but he insisted that this was an aberration. Most reviews are of high quality, he claims. I do not think this review was an aberration. From the anecdotal evidence, there are many AAAI reviews each year that are thoroughly unprofessional. Worse, according to the anecdotal evidence, the selection process itself is biased. It is difficult to prove these statements, of course, since most rejected authors simply grumble to their friends and then drop the matter. I would suggest the following: If you have a serious complaint about the quality of a AAAI review, send a critique to the Program Committee. (This year's Co-Chairs are Tom Dietterich: tgd@turing.cs.orst.edu, and Bill Swartout: swartout@vaxa.isi.edu.) Better yet, publicize your discontent. If enough people protest, perhaps AAAI will improve its act. For the record, here is my response to the reviews. To set the context, I have included an abstract of our paper. I would be happy to send a copy of the full paper to anyone who is interested in reading it. L. Thorne McCarty Computer Science Department Rutgers University _____________________________________________________________________________ \documentstyle[12pt]{article} \setlength{\parskip}{\medskipamount} \newcommand{\Px}{\mbox{$P({\bf x})$}} \newcommand{\Gix}{\mbox{$G_{i}({\bf x})$}} \newcommand{\Djx}{\mbox{$D_{j}({\bf x})$}} \let\If=\Leftarrow \let\And=\wedge \let\Or=\vee \let\Not=\neg \mathchardef\Fail="0218 \begin{document} \begin{center} \Large\bf The Case for Explicit Exceptions \end{center} \begin{center} \large L. Thorne McCarty\\ William W. Cohen \end{center} \begin{quotation} {\bf Abstract:} Most of the work on inheritance hierarchies in recent years has had as its goal the design of general purpose algorithms that depend only on the topology of the inheritance network. This research has produced some important observations about the various strategies used in human common sense reasoning, but it has also produced a proliferation of incompatible systems. In this paper, we resurrect the alternative technique, originally proposed by Etherington and Reiter, of explicitly encoding exceptions to default rules. The main technical innovation is the use of a different logical framework: a logic programming language based on {\em intuitionistic logic.} Using a combination of full intuitionistic negation plus negation-as-failure to encode default rules, we obtain analogues of the normal, seminormal and nonnormal defaults of Reiter's default logic. The advantage of our approach is that, whereas there is no adequate proof theory in classical logic for seminormal defaults, the analogous queries to an intuitionistic default rulebase can be answered by a simple top-down goal-directed interpreter. The claim that a default rulebase with explicit exceptions is easy to write and debug has been substantiated by encoding more than 40 standard examples from the literature. \end{quotation} \subsection*{First Review (Handwritten):} This is not a serious review, but a diatribe against logic programming. The reviewer simply refuses to evaluate the paper because the work was done in PROLOG! Some sample comments: \begin{quotation} This paper belongs in a conference on logic programming, rather than AI. \end{quotation} \noindent The paper certainly belongs in a conference on AI. It addresses problems that have been dominant in AI for many years. If the logic programming community can offer solutions to these problems, then the (American!) AI community ought to pay attention. This sort of insularity has seriously weakened the field in the past. \begin{quotation} The author labors under the misconception that PROLOG is a good language for communicating to other humans. \end{quotation} \noindent One of the main points of the paper is that our proof procedure for default rules can be encoded in a very simple PROLOG interpreter. To make this point, naturally, we describe the interpreter. It is very simple to understand. At Rutgers, we teach our first-year graduate students both LISP and PROLOG, and they should all be able to follow this discussion. If the reviewer is unable to do so, then he (or she) is not qualified to review the paper. Another claim of the paper is that it is very simple to encode the standard benchmark problems in the literature in our system, and run them through our interpreter. The reviewer addresses this claim as follows: \begin{quotation} The paper does substantiate his [sic] claims with benchmark examples, but unfortunately, most of these are contained in an appendix that exceeded the legal page limit. \end{quotation} \noindent This statement is false. Our paper conforms exactly to the page limits listed in the Call for Papers for {\it AAAI-90\/}: eleven pages of text including appendices, but not including the bibliography. This last comment is a transparent device to avoid addressing the substantive claims of the paper. When the reviewer gets the facts wrong, however, he (or she) reveals the full extent of his (or her) bias. It is obvious that the reviewer has decided to reject the paper from the start, without ever reading it carefully, and is simply grasping at straws to justify this decision. This person should never be allowed to review a {\it AAAI\/} paper again. \subsection*{Second Review (Typed):} On the surface, this is a responsible review. The reviewer appreciates the main claims of the paper: \begin{quotation} The most appealing part of this work is the exceedingly simple proof procedure which implements the intuitionistic theorem proving. The authors are justifiably proud of this aspect of their approach! \ldots The proof procedure is especially elegant for interacting defaults. \end{quotation} \noindent From these comments alone, I would have anticipated an acceptance rather than a rejection. The only complaint about the paper is the alleged ``obscurity of its writing style.'' It does appear that the reviewer had some difficulty following portions of the paper. But let us see if we can identify some of the reasons for this difficulty. Here is one specific point that the reviewer claims is unclear: \begin{quotation} Also, (4) and (10) can't possibly be equivalent under the classical (= Boolean) interpretation of $\If$ and $\Fail$, although perhaps under classical (= negation-as-failure) of $\Fail$. This needs to be clarified. \end{quotation} \noindent But is it true that (4) and (10) cannot possibly be equivalent under the classical interpretation of $\If$ and $\Fail$? Note the following: \begin{displaymath} \Px \;\If\; \bigwedge_{i}\Gix \;\And\; \bigwedge_{j}\Fail\Not\Djx \end{displaymath} \begin{displaymath} \left[\Px \;\If\; \bigwedge_{i}\Gix\right] \;\If\; \bigwedge_{j}\Fail\Not\Djx \end{displaymath} \begin{displaymath} \left[\Px \;\If\; \bigwedge_{i}\Gix\right] \;\Or\; \Fail\bigwedge_{j}\Fail\Not\Djx \end{displaymath} \begin{displaymath} \left[\Px \;\If\; \bigwedge_{i}\Gix\right] \;\Or\; \bigvee_{j}\Fail\Fail\Not\Djx \end{displaymath} \begin{displaymath} \left[\Px \;\If\; \bigwedge_{i}\Gix\right] \;\Or\; \bigvee_{j}\Not\Djx \end{displaymath} Each of these steps is an equivalence in classical logic! So what needs to be clarified? We could have included this derivation in our paper, of course, but we assumed that this equivalence could be verified by any first-year graduate student in AI! The reviewer also complains about the citation of various results in intuitionistic logic, as needed in the paper. There is nothing particularly obscure about intuitionistic logic. It has been thoroughly studied since the 1930's, and we have included references to some of the main sources [12, 21]. We have also included references to the major papers on intuitionistic logic programming, by myself and others [2, 14, 15, 23, 24, 26]. All the stated results can be found in these sources. There is a comment by the reviewer that the main facts about intuitionistic logic should be collected in a single place, ``up front.'' I think this a reasonable editorial suggestion, although I think an appendix would be a more appropriate location for such material. However, given the problem with (4) and (10) above, should we also include an appendix collecting the main facts about {\it classical\/} logic? The other specific examples of ``obscure writing'' are difficult to track down. I have searched for ``inconsistent uses of the symbol script R,'' and I cannot find them. The following statement is itself very obscure: \begin{quotation} \ldots it is not at all clear without repeated scrutiny that the authors are primarily {\it reinterpreting\/} the workings of the proof procedure. At first it really seems like they're really recoding the default rules. \end{quotation} \noindent In a sense, the point of giving a model-theoretic semantics is precisely to ``reinterpret the workings of the proof procedure,'' so it is not clear why the reviewer has a problem here. And what does he (or she) mean by the phrase ``recoding the default rules''? This needs to be clarified! In general, it is very difficult to evaluate a claim that a paper is obscurely written. What is obscure to one reader may be a model of clarity to another. I have shown this paper to several people, and they have all found it comprehensible. I sent the paper to one person (a specialist in natural language processing) prior to receiving the {\it AAAI\/} reviews, and she volunteered the following comment: ``One thing that I would like to add, is that I think you write very clearly, so that even novices in nonmon like me have a chance.'' \subsection*{Overall Assessment:} When the Program Committee Chair (or the Topic/Subtopic Chair) received these reviews, what should he (or she) have done? The first review should have been discarded. It should have been clear from a superficial examination of the paper that these were irresponsible comments. That would have left only one review, which is positive on the substance of the paper and negative on the writing style. At this point, it should have been mandatory that the Committee Chair solicit an additional review. Moreover, given the very subjective nature of an opinion about writing style, two additional reviews should have been required in this case. The Committee did not take these steps. I stand by my previous statement that the quality of reviewing at {\it AAAI\/} is very poor. \end{document}