Path: utzoo!mnetor!uunet!lll-winken!lll-lcc!ames!ll-xn!mit-eddie!husc6!bbn!uwmcsd1!ig!agate!ucbvax!ACC.ARPA!art
From: art@ACC.ARPA
Newsgroups: comp.protocols.tcp-ip
Subject: protocol verification
Message-ID: <8802090008.AA08913@ucbvax.Berkeley.EDU>
Date: 8 Feb 88 21:59:00 GMT
Sender: daemon@ucbvax.BERKELEY.EDU
Reply-To: <art@acc.arpa>
Organization: The Internet
Lines: 22



With the recent discussion about TCP testing and verification, I thought
that I would pass on an observation.  I don't know if this reflects on
just the specific instances or protocol testing in general.

I was involved with GM in testing of MAP protocols for Autofact '84 and '85.
Both times, all of the vendors eventually were able to successfully pass
the conformance test suite.  When interoperability testing began, MOST of
the vendors found problems affecting interoperability.  On other occasions,
I have seen systems working on real networks fail conformance tests and
systems which passed the conformance tests fail on the real network.

I feel that this probably stems from a tendency by the test writer(s)
to concentrate more on the protocol details (such as which diagnostic
error code should be returned), than on the basic functionality of the
protocol.  The protocol writer may then end up spending effort just trying
to make the test suite happy at the expense of a robust design.

						Art Berggreen
						art@acc.arpa

------