Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!asuvax!ncar!midway!clout!chinet!les From: les@chinet.chi.il.us (Leslie Mikesell) Newsgroups: comp.mail.misc Subject: Re: BITFTP grief! (UUNET email-ftp?) Message-ID: <1991May24.170708.3850@chinet.chi.il.us> Date: 24 May 91 17:07:08 GMT References: <1991May22.081251.1026@uunet.uu.net> <1991May22.174325.6230@chinet.chi.il.us> Organization: Chinet - Chicago Public Access UNIX Lines: 36 In article emv@msen.com (Ed Vielmetti) writes: >If you can specify exactly where the files you are interested in are, >it is straightforward to maintain a reasonably up to date shadow >directory of the remote FTP site and make that directory available for >UUCP. Updates can be handled in a number of ways [...] >Naturally you'd like to keep whatever mechanisms there are for making >sure that the file you fetch is the same as the latest version on the >server somewhat hidden from the general "rftp" user, so that your >caching will have a reasonable chance to work. No sense in going off >to bother Columbia if you have a copy handy that's the same as what >they have. If the file changes frequently and is not requested often, caching would just be a waste. Besides, bitftp at Princeton was able to scarf a 500K file from Columbia within a second of asking, according to the FTP log they send back - is it really worth storing a copy to save that second? On the other hand, it is pretty inconvenient for me to determine when that file has been changed. What you really need is a new transfer protocol that tells you up front the date and length of a file and a CRC value for the contents. If you have a matching copy, then you don't need to do the transfer at all. If your copy is shorter, you would send a CRC and the length of the piece you have and the other end would check it against the corresponding length of its copy. If the CRC matched, you could just pick up the new portion of the file, otherwise the old copy would be completely overwritten. Ideally, this protocol would work through one or more "relay" sites that would be allowed to cache locally, passing on only the initial check to the remote end point. Les Mikesell les@chinet.chi.il.us