Path: utzoo!utgpu!jarvis.csri.toronto.edu!rutgers!sun-barr!cs.utexas.edu!uunet!mcvax!ukc!etive!aipna!edai!cam From: cam@edai.ed.ac.uk (Chris Malcolm cam@uk.ac.ed.edai 031 667 1011 x2550) Newsgroups: comp.ai Subject: Re: Turing Test and Subject Bias Message-ID: <399@edai.ed.ac.uk> Date: 31 May 89 14:40:25 GMT References: <3018@crete.cs.glasgow.ac.uk> Reply-To: cam@edai (Chris Malcolm) Organization: University of Edinburgh, Edinburgh Lines: 51 In article <3018@crete.cs.glasgow.ac.uk> gilbert@cs.glasgow.ac.uk (Gilbert Cockton) writes: >Just what sort of Science did young Mr. Turing have in mind when he >decided that subjective opinion could ever be a measure of system >performance? Young Mr Turing suggested in his 1950 paper "Computing Machinery and Intelligence" that it would be 50 years before what we now call "the Turing Test" and he called "the imitation game" could be applied. I see no reason to revise that estimate downwards. He also made it clear that he considered that everyday notions of "machinery" and "thinking" then carried too much excess semantic baggage for reasonable discussion of the question "can machines think" to be profitable, and consequently suggested the "imitation game" as a Gedanken experiment with which to clear the philosophical air. While there have been a few occasions in AI research when people were asked to play a Turing-like game to assess a program whose specific purpose was to articulate a model of a certain kind of of human behaviour, such as Colby's "PARRY", or the ranking of MYCIN's diagnostic responses compared to a panel of experts, it remains true that the primary purpose of the Turing Test in AI is as a gedanken experiment. As Turing pointed out in the paper in question: "The popular view that scientists proceed inexorably from well-established fact to well-established fact, never being influenced by any improved conjecture, is quite mistaken. Provided it is made clear which are proved facts and which are conjectures, no harm can result. Conjectures are of great importance since they suggest useful lines of research." The Turing Test was suggested by its originator as a source of useful conjecture. It still serves that purpose, and that tradition in AI is continued by Searle's Chinese Room argument, and Harnad's Total Turing Test, to name two examples recently ventilated on comp.ai. >How do AI types *REALLY* test their systems? This question seems to suggest that AI types _pretend_ to be using something like the Turing Test, but actually in the privacy of their labs are up to something quite different. I thought, Gilbert, that you had once in your career suffered some education in AI? The answer to how we AI types test our systems is as various as the kinds of system we build, and the reasons we build them. In many cases, as I am sure you know, what is interesting about our systems is why they don't work :-) That is not the kind of thing which is established by testing a system. -- Chris Malcolm cam@uk.ac.ed.edai 031 667 1011 x2550 Department of Artificial Intelligence, Edinburgh University 5 Forrest Hill, Edinburgh, EH1 2QL, UK