Xref: utzoo sci.bio:1470 sci.misc:2338
Path: utzoo!attcan!uunet!seismo!sundc!pitstop!sun!decwrl!labrea!rutgers!bellcore!faline!thumper!ulysses!andante!alice!dmr
From: dmr@alice.UUCP
Newsgroups: sci.bio,sci.misc
Subject: Re: Strange results in Nature article
Message-ID: <8092@alice.UUCP>
Date: 7 Aug 88 07:28:24 GMT
Organization: AT&T Bell Laboratories, Murray Hill NJ
Lines: 101

This affair is unfortunate.  Although I thought well of Nature to
publish the paper, and considered the cautionary comments and
editorial printed with it entirely appropriate, their investigation
and the report they printed of it (Nature 334, 28 July) were indeed
badly handled.

The report (by Maddox, Randi, and Stewart) was much too hastily
prepared and is nearly incomprehensible because of editorial errors in
its preparation.  It is not the sort of thing that in the ordinary
course of events would be accepted by Nature.  Much of it is narrative
in flavor:  "Benveniste ...  offered to predict where the peaks and
troughs would fall in the data....  But his predictions proved to be
entirely wrong." Later, "Opening sealed envelopes is Randi's
expertise."

At the same time, it attempts to convey actual results through graphs,
but they appear to have botched the presentation; if someone can
explain to me what Figs 3, 4, 5 actually represent, I would be
grateful.

To give examples of the problems:  Fig 3 has two graphs of
"degranulation" vs.  "dilution".  The figure's caption begins "Records
for the first two blind experiments (5-7 inclusive)...." What does
this mean?  The top graph is labelled "Blind expt X" and has two data
sets on it, with a legend saying that the x marks are "EX" and black
circles "EY".  The bottom graph has no such legends, and its two data
sets are marked with black circles and open squares.  Although the
dispersion of the top and bottom graphs are similar, the ordinate
scales differ, so that one notices that the bottom graph looks
bouncier than the top one, though in fact they are about the same.
The text does not explain what the Expt X, Y, EX, EY mean, nor the
"two experiments" vs "5-7 inclusive."

In spite of the labels, the abscissas must really be log dilution, not
just dilution.  The ordinate values in Fig.  3 have had something
subtracted from them (their values are sometimes negative) and are
clearly not "degranulation;" the first reference to Fig 3 in the text
talks about subtracting the mean.  I guess that Fig 3 probably
represents normalized counts (not degranulation), but assignment of the
labels and referring the graph points to actual experiments is pure
guesswork.

The second reference to Fig. 3 in the text appears to be talking about
a combination of Figs. 4 and 5, graphs showing the same kind of data
as each other (normalized deviations of basophil counts from their
mean) but which are drawn in somewhat different styles, obstructing
comparison.

Fig 6 probably has a wrongly labelled ordinate (fractional
degranulation rather than percent as the label says), and its caption
says "Two duplicate Italian runs showing high degranulation, but
discordantly" while the text says "Figure 6 is typical of the data
from Milan.  While there are no duplicate data measurements, and
therefore no direct evidence of sampling error, there is also some
evidence of degranulation at high dilution." What are the two meanings
of "duplicate" here?  And there is no evidence for degranulation at
high dilution, because there is no way to know what the dilution
is--the abcissa in this graph is merely "tube number."

Aside from procedural matters recounted anecdotally, the two killer
criticisms are 1) Benveniste et al. got smaller sampling errors from
counting the basophils than could be expected, suggesting the
possibility of something fishy in their counting; 2) The whole effect
disappeared when blind counting (counter didn't know what was in the
tube) was used.  But the report so screws up the presentation that you
must trust the authors of the report that these observations are true;
their graphs are meaningless.

The report ends with 5 specific criticisms, of which the first, "The
care with which the experiments have been carried out does not match
the extraordinary character of the claims made in their
interpretation," summarizes the rest, which go into a bit more detail.
Unfortunately, the care with which the investigation and the report
were prepared do not match the expectations engendered by the
extraordinary publicity the affair has received.  The criticisms that
the report makes may well be trenchant, but its presentation is so
bady flawed that an unbiased reader would find it hard to take
seriously except in faith that Maddox, Randi, and Stewart can be
trusted.  My guess (as a biased reader) is that a proper presentation
of their findings would support their conclusions.

Immediately following the report is a reply by Benveniste.  Just as
unfortunately, he raves when he should be measured and modest.  Much of
what he says sounds like Usenet, in fact:  "Salem witchhunts or
McCarthy-like prosecutions will kill science.  Science flourishes only
in freedom.  We must not let, at any price, fear, blackmail, anonymous
accusation, libel and deceit nest in our labs." He also recounts
activities that were, at least, inappropriate-- "Stewart imposed a
deadly silence in the counting room, yet loud laughter was heard where
he was filling chambers.  There, during this critical process, was
Randi playing tricks, distracting the technician in charge of its
supervision!"

It seems to me that Nature misread the issue and overreacted.  The
observations that the original paper makes are so far from what can be
explained, and should be so easily checkable, that it should have been
sufficient to let others try to reproduce the results.  At any rate, if
an investigatory team was to be sent, it should have been done with
less haste and greater care.

        Dennis Ritchie