Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!swrinde!zaphod.mps.ohio-state.edu!magnus.ircc.ohio-state.edu!csn!boulder!daemon
From: Daniel.Karrenberg@cwi.nl (Daniel Karrenberg)
Newsgroups: comp.dcom.sys.cisco
Subject: Appliques can fail .....
Message-ID: <32714@boulder.Colorado.EDU>
Date: 27 Feb 91 11:51:30 GMT
Sender: news@boulder.Colorado.EDU
Lines: 98

Appliques can fail .....
	... and it can be hard to detect.


This is long but I wish I had read something like it before I had to
deal with this one!

I have come across a very hard to detect serial line problem recently.
Our setup is as follows (place names included to add local flavour):


     KTH Stockholm Sweden                     CWI Amsterdam Netherlands

   +--------+    +--------+                   +--------+    +--------+
   |        |V.35|        |                   |        |V.35|        |
   | CISCO  +====+  MUX   +-- Various Telcos -+ MODEM  +====+ CISCO  |
   |        |    |        |                   |        |    |        |
   +--------+    +--------+                   +--------+    +--------+

      (1)           (2)                          (3)            (4)


We were having some strange problems with that link. The keepalives were
getting thru OK and small IP packets were doing reasonably well (1-2% loss).
Large IP packets (1000B) weren't getting thru at all.

So we went to tackle the problem:

	Step 1:   Make local loop at (3).
		  Ping yourself from 4: OK.
		  Conclusion: Cisco (4) plus cable to (3) and digital
		  interface of (3) are OK.

	Step 2:   Make local loop at (2).
		  Ping yourself from 1: OK.
		  Conclusion: Cisco (1) plus cable to (2) and digital
		  interface of (2) are OK.

At this point both Sweden and Holland conclude it is the Telcos (we call
them PTTs) again and call in the problem. Note: PTT demarc is the
digital interface of the PTT owned Modem/Mux (call it CSU/DSU if you like).
The PTTs make various loops, say they found something and declare the
line OK. Testing revealed that the problem persisted albeit a little 
less severe, only 95% of 1000 byte packets between (1) and (4) were dropped 
:-) :-(.

We repeat steps 1 and 2 above with the same results. We start suspecting
a clocking problem. So Sweden gets a BERT tester (actually a Vitalink)
and we go to 

	Step 3: Connect BERT tester to (2), remote loop at (3).
		Run a few minutes of BERT pattern: Works fine.
		Conclusion: The PTTs are not really to blame for
		this one.

	Step 4: Reset remote loop at (3), shut down interface
		at (4). 
		Run a few minutes of BERT pattern: Works fine.
		Conclusion: Cable and applique in (4) OK.

	Step 5: Reconnect (1), interface at (4) remains shut down.
		Have (1) ping itself: Works fine.
		Conclusion: Nothing wrong in Sweden, really.

	Step 6: Shut down interface in (1), enable interface in (4).
		Have (4) ping iself: All large packets get dropped.
		Conclusion: Head scratching. Must be in (4) or
		some weird clocking problem between (3) and (4).

	Step 7: Swap MCI card in (4).
		Have (4) ping iself: All large packets get dropped.
		Conclusion: More head scratching. Should it be the
		applique or the flat cable in the cisco ?????!?!?!?!?

	Step 8: Swap back MCI card and use different applique in (4).
		Have (4) ping iself: Works!
		Conclusion: Happiness and disbelief.

We subsequently swapped a few apliques with the conclusion that
old ones (bar code serial <10000) consistently don't work on this line.
New ones do work although the link is not 100% stable yet but this might
be due to other problems.

I am still not 100% sure what makes the old qppliques not work since 
i haven't had the time to put a scope on the line. Any ideas?

Lessons learned:

	1) In some (rare) circumstances local loopbacks
	   do not detect local problems.

	2) A spare parts pool needs to include appliques.

	3) Problems like this can only be found by both
	   ends testing synchroneously with an open telephone
	   connection to discuss things as they go.

Daniel