Path: utzoo!utgpu!news-server.csri.toronto.edu!rutgers!usc!snorkelwacker!bloom-beacon!eru!hagbard!sunic!news.funet.fi!funic!santra!cs.hut.fi!juh From: juh@cs.hut.fi (Juha Hyv|nen) Newsgroups: comp.edu Subject: Re: Automatic checking the students' answers Message-ID: Date: 19 Oct 90 16:09:24 GMT References: Sender: news@santra.uucp (Cnews - USENET news system) Distribution: comp Organization: Helsinki University of Technology, Finland Lines: 202 In-Reply-To: eibo@rzsun3.informatik.uni-hamburg.de's message of 15 Oct 90 14:19:36 GMT From: eibo@rzsun3.informatik.uni-hamburg.de (Eibo Thieme) Subject: Re: Automatic checking the students' answers Date: 15 Oct 90 14:19:36 GMT +------------------------ ! juh@cs.hut.fi (Juha Hyv|nen) writes: ! ! > Student Jack sends a request by e-mail to a mail server asking ! > for question # 1. The server sends him back the following. ! ! > "Show the heap after each insertion of the keys ! > A N E X A M P L E into an initially empty heap." ! [...stuff deleted by juh...] ! ! > When Jack has solved the problem, he sends back the answer. ! > Now, the "answer analyzer" checks his answer and gives points ! > to him. ! ! As I read this article I felt quite uncomfortable, as I couldn't ! appreciate this vision as much as Juha Hyv|nen appears to do. Being ! a student of informatics in Germany I am far too close to the intended ! group to stay calm on such a topic. ! ! Automated testing is IMHO one of the most inappropriate methods for ! evaluating peoples knowledge. Some points to mention are: !........................ I realize I did not mention anything about the intended use of the proposed system. We are not trying to evaluate the students' knowledge. We are trying to make sure they do their homework (and learn while doing it). In order to pass the course, you have to do (quite a lot of) homework. That consists of questions like question #1 above. In addition to that, you have to pass a final exam. To make sure everyone is doing their homework, it has to be checked. Homework is devided in five parts, each of which about one major section of the book (R. Sedgewick, Algorithms). The student must get enough points from every part. And those who do their homework very well get a bonus that rises their final exam grade, e.g., from 2 to 3 (but not 0 -failed- to 1; we use the scale of 0...5). Presently, checking the homework is done by a person. Last spring, over 500 students took part in the course. That meant that a total of 500 students x 5 parts x 4 questions = 10,000 answers had to be checked (and the results registered). At the rate of checking one answer a minute it would take over 160 hours to check all the answers. That is one working month! I think that the time should be spent teaching than doing a routine job that could be done by a computer. +------------------------ ! 1. The set of questions askable is confined to the area ! of reproducing memorized facts and application of ! memorized rules. (Anybody out there in AI-land daring ! to oppose ? :-) !........................ We are not asking the students to reproduce memorized facts. But students must know these facts (and rules) in order to be able to answer (simple) questions like "Which of the following arrays can represent a heap? 1) a b f h k o p 2) c a g u y z 3) a h b k m r t " To answer that, you have to know AND understand the "heap condition" (and the rule for representing a heap as an array). Of course intelligent questions like "Why" or "Explain" or "Suggest" cannot be automatically checked. But currently, we are not asking that kind of questions. Could you give some examples of questions we should ask? +------------------------ ! 2. Knowledge is *always* embedded, being part of a person ! acting in the world. To pay attention to this *central* ! aspect of knowledge it requires direct interaction, ! which in this context means aural examinations. !........................ We do not have the resources for direct interaction other than the classes (lectures). (Most students skip the classes -- attending is voluntary.) Here are some other observations: The students could ask questions. They do not. The students are asked questions. They do not want to answer. If the students are made to answer, even more of them tend to skip the class. (We could start a new thread on discussing why. Here is one answer: they are afraid of being wrong and letting everybody to know it. "It is better the keep quiet and let everybody THINK you do not know the answer than to open your mouth and make sure they KNOW that." I believe this shyness (?) among us Finns has been discussed in some newsgroups -- maybe in soc.culture.nordic.) Conclusion: direct interaction (with too limited resources) does not work (with 500 Finns). What is "aural examination"? Do you mean "oral"? +------------------------ ! 3. People tend to forget the nature of learning, believing ! it being nothing more than is required. The net effect ! is a very poor standard of education. !........................ To teach e.g. the quicksort we do not have a homework question: "Learn the quicksort." We have a question: "Solve this problem using the quicksort algorithm." What seems to be required: "Solve this problem". What actually is required: "Learn quicksort". So, the student learns an algorithm by doing in a small scale the same that a program is supposed to do in a large scale (learning-by-doing). +------------------------ ! 4. Much energy will be spent to conform system requirements, !........................ There are (at least two) different kinds of requirements. The mail server must be able to - determine who sent the message - extract the answers from the message These are what we call "external format" of the answers. Then there are the requirements the analyzer program sets for the answers. Of course, answers must be given in a simple format (that the analyzer can understand). The graphical format used in the text book cannot be used. Still, the problems should be solved using that format. The conversion from the graphical into the required format is (or should be) very simple. The format of each answer is called "internal format". The internal format reflects the program's implementation of the data structure. The result is that students learn one implementation of an abstract data structure. A disadvantage is that they may think that is THE implementation. +------------------------ ! ingenious ideas will be considered false. !........................ I can remember only one "ingenious idea" in all my checking of the final exams & homework. It was solving a problem using a method not discribed in the book. By the way, several students came up with it. Therefore, I do not think that ingenious ideas are a problem. Besides, when the student complains, the answer is checked by a person. +------------------------ ! Again, people will see the method, not the contents. !........................ Take the quicksort example. What is the content of a sorting problem? The result (sorted array)? But everyone knows the result! You cannot mean that. I think you mean that we should let the student choose the tool (i.e, the sorting method) best suited for the problem? Actually, we would like to. But we also want to make sure s/he knows all tools and how they work. And that is the main purpose of this particular course. +------------------------ ! Please note, that this is only about education, which is a very reduced ! view onto learning, i.e. attaining wisdom. And mark, that my arguments ! are applicable to any automated testing including multiple-choice-tests, ! not only computer aided testing. I am willing to explicate myself further ! if there is any interest. !........................ I certainly am interested. / (.___o .. /_/ ___/ Juha Hyvonen ! / ! !/ ) ! juh@hutcs.hut.fi ------ PS. I got a warning about some bugs in the algorithms in the book (thanks Pat, those were new ones to me). Does anyone have a list of other known bugs? If not, does anyone happen to know the e-mail address of Robert Sedgewick (the author of the book)? PS2. The course is valued (by the university) as being a "three-week" course, i.e., the amount of work needed to learn and understand all issues is about 3 working weeks (120 hours) -- including classes, homework, and additional personal effort. The course deals with sections 1...23 and 29...32 (Fundamentals, Sorting, Searching, String Processing, and Graph Algorithms) of the book R. Sedgewick, Algorithms, 2nd ed. 650 p. Addison-Wesley, 1988. ISBN 0-201-06673-4 Do you (the net people) think that 3 weeks is reasonable? All the algorithms are supposed to be totally new to the students. Every student here seem to think that it is a joke.