Path: utzoo!attcan!uunet!wuarchive!zaphod.mps.ohio-state.edu!rpi!uupsi!sunic!news.funet.fi!funic!santra!cs.hut.fi!juh From: juh@cs.hut.fi (Juha Hyv|nen) Newsgroups: comp.edu Subject: Re: Automatic checking the students' answers Message-ID: Date: 22 Oct 90 15:18:34 GMT References: <1990Oct19.103605.5035@maths.nott.ac.uk> Sender: news@santra.uucp (Cnews - USENET news system) Organization: Helsinki University of Technology, Finland Lines: 105 In-Reply-To: anw@maths.nott.ac.uk's message of 19 Oct 90 10:36:05 GMT From: anw@maths.nott.ac.uk (Dr A. N. Walker) Subject: Re: Automatic checking the students' answers Date: 19 Oct 90 10:36:05 GMT +------------------------ ! In article eibo@rzsun3.informatik.uni-hamburg.de ! (Eibo Thieme) writes: ! ! > 1. The set of questions askable is confined to the area ! > of reproducing memorized facts and application of ! > memorized rules. ! ! ... but this is not true. The difficulty ! lies in making the answers easily parsable. I have been using such ! systems since 1970 to set and mark problems in Numerical Analysis. ! The questions were askable because they conformed to a simple template ! in which certain elements could be randomised. What facts the students ! had memorised, or what rules they used was in no way a factor in ! the system, which worked because the computer was at least able to ! assess the answers. !........................ In numerical analysis, the (unambiguous) answer to a problem cannot be known if you do not know any method to solve it. In our case, the "answers" are trivial (e.g., a sorted array). We are not interested in the answer. We are interested in the method (e.g. quicksort). That makes it more difficult because the representation of the *method* must be parsable (and as natural for the student as possible). +------------------------ ! [Initially, the computer was used primarily to generate a ! "random" question sheet (for the student) and the corresponding correct ! answers (for me); once the computer power became available, the ! whole system became interactive, and the student responses were ! assessed by the computer. As the responses were always numbers ! or (occasionally) a menu selection, the computer either understood ! the response or could reject it as "ungrammatical". All interactions ! were logged, and students could include comments/queries/complaints ! in the log for me to deal with.] !........................ Your system seems to be similar to those discribed in papers that I have found. They all seem to be dealing with cases where generating tests is easy and checking only the answer is sufficient (mainly in mathematics). I have found no references to automated interactive or *non-interactive* testing that could deal with the method used in getting the (already known) answer. I believe that setting up a non-interactive testing system in numerical analysis would be rather simple because there is (almost) no syntax within the answers. I have found references to papers (and read some of them) about automated test generation automated program marking interactive tests (with automated marking) interactive programming tutors (self-paced learning) (I am looking for more...) By the way, I do not see much difference in interactive testing and interactive tutoring in general. Interactive testing tools can be used in (simple) tutoring if they provide instant feedback. And with little effort, it should be possible to include automatic marking into interactive tutoring tools. +------------------------ ! I wanted to find out whether the students could *do* NA. ! The course *also* (naturally) included theoretical knowledge, ! assessed in the traditional ways, but *practical* knowledge can ! often, with imagination, be assessed mechanically. !........................ And we want to know whether the students can solve a problem using a particular *method*. That is also practical knowledge. And you hit the point with imagination. That was one reason for me to post my question [about automatic marking], to (use your imagination to help me) find good and suitable questions for automatic marking and their answer formats. Nobody has suggested a single one yet :-( +------------------------ ! In a practical test, the method and the ingenuity ! are irrelevant. The computer sees the results. If I ask you to ! solve an equation, or to perform a quadrature, and you get the ! [unambiguous!] right answer in a reasonable time, fine, no matter ! how you do it. If the answer is wrong, it's no mitigation that ! you used a very ingenious method. !........................ In our case (of practical test), the results are irrelevant. We want the computer to see the method. The syntax of representing the method is the key issue. And the answer is not necessarily unambiguous: one (trivial) mistake does not mean that it is completely wrong. Intelligently dealing with partly correct answers is perhaps the hardest requirement to meet. The answer should therefore consist of distinct steps. If the result of one intermediate step is wrong, it should be taken as a basis of the next step and not the correct result (of the step). (Could your equations be solved in a single step?) / (.___o .. /_/ ___/ Juha Hyvonen ! / ! !/ ) ! juh@hutcs.hut.fi ------