Path: utzoo!attcan!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!sdd.hp.com!caen!elm.engin.umich.edu!zarnuk From: zarnuk@caen.engin.umich.edu (Paul Steven Mccarthy) Newsgroups: comp.software-eng Subject: Effect of execution-speed on reliability/testing Summary: Extremely slow programs are difficult to test thoroughly Keywords: speed testing Message-ID: <1990Dec19.102005.11830@engin.umich.edu> Date: 19 Dec 90 10:20:05 GMT Sender: news@engin.umich.edu (CAEN Netnews) Organization: University of Michigan Engineering, Ann Arbor Lines: 121 I know this may be an unpopular opinion, but I have personal experiences to back up the position: R | /---------------\ e | / \ l | / \ i | / \ a | / \ b | / \ i | / \ l | / \ i | / \ t | / \ y |/ \ +------------------------------------------> slow Execution Speed faster I don't know what the fine detail of this graph should look like, but personal experience has shown the general trends to be true. Namely, both extremenly fast and extremely slow programs suffer from reliability problems. Every textbook that I have read, and every software engineering professor that I have spoken with has emphasized the right-hand side of this graph, but they have all neglected the left-hand side. The impression that I have gotten from most professionals in the software trade is something like: "Speed isn't important. The hardware will speed up and take care of our inefficiencies." I beg to differ. Speed has a direct impact on reliability. Systems that are too slow to test thoroughly, just don't get tested. Case in point: Last August I rewrote a significant portion of one of our systems in C, (After hours, on my own time, of course.) Here are some rough statistics comparing the two implementations: C Database Package *1* Lines of Code: 5k 5k *2* Reusability: 20% 2% Development time: 4wks ? *3* Testing time: 3days 2wks Number of Files: 7 60 *4* Service Calls: 1 28 Speed N=10k: 20min. 90min. Speed N=50k: 45min. 360min. (6 hrs!) *5* Speed N=250k: 120min. 1440min. (24 hrs!) *1* These estimates include comments and blank lines, which are heavy in the C version, and non-existent in the database version. The original database version, which is used for these comparisons, was very poorly written. A cleaned-up implementation, written with the same database package had only about 3k lines. *2* About 1k of the C version is a btree-module which has been LINKed into about 20 programs by myself and others since this system was released, including ?-many programs in 3 separate production systems, and numerous "one-shots". Certain sections of the database version are duplicated in other parts of the system. I don't know which came first. They are all cut-and-paste copies due to the inter- pretative nature of the database package, and every single copy has to be manually updated, re-"compiled", and retested whenever a bug is detected, or dependent factors change. I don't rate this kind of reuse very highly. (I tried to promote a macro- preprocessor when I came on board here, but management is too conservative. They don't want to *change* anything. Apparently, employing 5 full-time programmers to continually patch their baling-wire and bubble-gum 70k lines of source without adding any functionality is cost-effective.) *3* I don't think the database version was ever thoroughly tested, but it took two weeks to use it to generate the output for comparison with the C version. *4* The C version was installed at the end of August. One bug has been detected so far (fence-post) in three and a half months of daily use at four sites. The database version was generating at least two service calls a week due to: its dependencies on numerous files; its slow speed (forcing users to abort it while running to handle priority items, leaving it exposed to power-loss problems for extended periods of time, etc., etc.); and the numerous latent bugs that it contained which were never uncovered because it was too slow to test thoroughly. *5* Times taken from production runs. Average production run: N=100k. I put a lot of extra effort into the C implementation to match the output from the database system byte-for-byte, duplicating bugs and all, (to automate testing with file-comparisons,) so the systems are functionally identical. Before I undertook the task of rewriting this application in C, I reimplemented it with the database package first. I gained roughly a factor of two in performance by cleaning up the code, but it was still so incredibly slow that it would have taken at least four weeks to run all the necessary tests. What's more, I tried three or four different approaches, using both the highest- and the lowest- level features of the package to try to squeeze more performance out of it. -- It just wasn't there. I was able to run 5-8 tests of the C version in the amount of time that it took to run 1 test of the database version. I was able to test the C version much more thoroughly than the database version had ever been tested. Because the C version ran so much faster than the database version, it was far less vulnerable to user-aborts, power-losses, etc.. Many people in the software field promote an attitude that performance is not important, or that increasing performance degrades reliability. These statements are misleading. Highly optimized code which sacrifices legibility for performance can seriously degrade reliability, but extremely slow code is just as bad -- or worse. Regards, --- Paul...