Xref: utzoo comp.protocols.nfs:2037 comp.arch:21532 Path: utzoo!utgpu!news-server.csri.toronto.edu!rutgers!ucsd!swrinde!mips!mash From: mash@mips.com (John Mashey) Newsgroups: comp.protocols.nfs,comp.arch Subject: Re: Offloading I/O [was:Incremental sync()s ...] Message-ID: <1088@spim.mips.COM> Date: 18 Mar 91 05:13:53 GMT References: <3254@crdos1.crd.ge.COM> <19838@cbmvax.commodore.com> Sender: news@mips.COM Followup-To: comp.protocols.nfs Organization: MIPS Computer Systems, Inc. Lines: 159 Nntp-Posting-Host: winchester.mips.com In article peter@ficc.ferranti.com (Peter da Silva) writes: >Here we're talking about putting file systems in smart processors. How about >putting other stuff there? > > Erase and kill processing. (some PC smart cards do this, > as did the old Berkeley Bussiplexer) > Window management. (all the way from NeWS servers > with Postscript in the terminal, > down through X terminals and Blits, > to the 82786 graphics chip) > Network processing. (Intel, at least, is big on doing > lots of this in cards, to the point > where the small memory on the cards > becomes a problem... they do tend > to handle high network loads nicely) These have been done, in various ways, and have at least fairly often been good ideas, although some of the requirements keep changing on you. In general, these OFTEN have the characteristics that make it reasonable to distribute the processing, although people argue about the window management case. The relevant characteristics are: 1: The ratio of # of device interactions to bandwidth or processing is high, i.e., there are many interactions, but most of them neither move much data, nor require much processing. 2: There is relatively little global data needed to process the device interactions. 3: There is a reasonable protocol, such that the work can be split between I/O processor and main CPU(s), such that the main CPU(s) normally interact with the I/O processor much less frequently than the IOP interacts with the devices. I.e., if the IOP does NOT lessen the frequency of interaction, it may well get in the road more than not. 4: The devices demand lower response latency than can cost-effectively be done by the main CPU. [Note that my argument against doing this for disk file systems was that disk I/ O was the opposite of 1 (much data, few requests), and disobeyed 2 (much global data).] Consider some of the paths through which asynch serial I/O has gone through (for concreteness, on UNIX): 1. Simple UARTS for serial ports, with no special queueing. CPU polls for input or gets 1 interrupt per char CPU emits output on scheduled basis, or 1 interrupt per char Cheap, but needs main CP Uthat can stand high interrupt rates if supporting many and/or fast lines. CPU does echoing EX: DEC DL11 (yes, that goes a way back) (in this case just 1 line) 2. #1 plus input silo EX: DEC DJ11: 64-entry silo, handling 16 lines. CPU can either: poll the device regularly to see if there is any input available, gather 1-64 entries and parcel them out to the processes they belong to. OR ask for an interrupt whenbver a character appears, thus possibly trading away overhead ot minimize latency. CPU does echoing 3. #2 plus more control EX: DEC DH11: like DJ11, but for example, you could ask for an interrupt only if there were N characters in the silo. Also, it could do auto-echo 4. #3 plus DMA output EX: (I think): DEC DZ11 + DQS11 com processor (My memory gets vague on this one. The DQS was a com processor with good speed, albeit "interesting" to code for.) 5. Input silos with settable parameters, output silos; maybe DMA There have been lots of these: no special processing, but: a) On input, poll, or ask for an interrupt if the silo has N characters, or possibly, ask for interrupt if even 1 character is there, and M milliseconds have elapsed. if something is set up as a comm processor geared for non-terminal use, it may well do DMA, to support fast lines. b) On output, either have deep silos that you can stuff many characters into in bunches, or else do DMA out. 6. Move echo, erase, and line kill into the IOP. This has been done in various commercial UNIXes (maybe people can post examples). For line-oriented input, this is not a bad idea, and especially as cheap micros became available, became much more reasonable. (Note that in the DQS/DZ era, anything like a micro was NOT cheap.) The CPU: needs a protocol to hand to the IOP definitions of erase & kill characters (if the user is allowed to change them), and in fact, any other parameters of relevance The IOP: needs more local storage (to hold parameters) needs even more local storage, because the natural unit of buffering is now a complete input line Must deal with all of the UNIX escape sequences and conversions, as well as interrupts. (For example, if an interrupt comes, it may want to terminate output in progress to that line.) BENEFITS: CPU gets one interrupt for an entire line of input. CPU need not do echoing. DRAWBACKS: Often, for cost reasons, the IOP could just barely do what was needed, and then along would come additional requirements. For example, suppose you had a definite idea of "Interrupt" and that was wired into the IOP, and then UNIX starts letting people change that, etc. In general, to make this work, the IOP must almost always be able to deal with every input character, without bothering the main CPU. Of course, historically, about the same time such things became widespreadly practical, they became almost useless in some environments, because people were using CRTs, not with line mode, but with visually oriented terminals. At that point, everybody started running the terminals in "raw" mode, thus bypassing all of the above processing! So, the next step: 7. Make the IOP smart enough to distinguish between "ordinary" characters which should be echoed, and stuck in an input queue, and other characters and/or sequiences, which demanded response from the CPU. The theory here, is that, for example, you sit typing text into vi or emacs fairly efficiently, and still get the various control sequences to do something, without making the IOP huge. Of course, since every program might well use different special sequences, this may require more interaction with the main CPU. I think this one has been done. 8. Put termcap in the IOP. Implications of this left as an exercise :-) One clear lesson: When doing an IOP, be careful that the interface requirements don't change on you, especially if they often force you into a transparent pass-thru mode where the IOP really is doing nothing. Another one: watch out if you trying to make the same IOP handle both terminal I/O and fast serial lines to connect systems I believe that most systems tend to use devices in groups 1-5 these days, because the processing requirements have gotten complex enough that you might as well just have the main CPU do it, irritating though it may be. Anyway, those are a few samples. Maybe people would post examples of these various approaches, or others, with comments on: a) How well they worked b) How long they lasted c) What the weirder bugs and problems were -- -john mashey DISCLAIMER: UUCP: mash@mips.com OR {ames,decwrl,prls,pyramid}!mips!mash DDD: 408-524-7015, 524-8253 or (main number) 408-720-1700 USPS: MIPS Computer Systems MS 1/05, 930 E. Arques, Sunnyvale, CA 94086