Path: utzoo!utgpu!news-server.csri.toronto.edu!bonnie.concordia.ca!nstn.ns.ca!news.cs.indiana.edu!att!linac!pacific.mps.ohio-state.edu!zaphod.mps.ohio-state.edu!think.com!spool.mu.edu!munnari.oz.au!manuel!cmf851
From: cmf851@anu.oz.au (Albert Langer)
Newsgroups: comp.periphs.scsi
Subject: Re: Random Access DAT
Message-ID: <1991Jun17.171231.6601@newshost.anu.edu.au>
Date: 17 Jun 91 17:12:31 GMT
References: <1991Jun14.211246.4340@newshost.anu.edu.au> <9850028@hpcpbla.HP.COM>
Sender: news@newshost.anu.edu.au
Organization: Computer Services Centre, Australian National University, Canberra, Australia.
Lines: 191

Thanks for the detailed and helpful answers to my questions Kevin.
Now here's some follow ups:

In article <9850028@hpcpbla.HP.COM> kev@hpcpbla.HP.COM (Kevin Jones) writes:

>> 1. Now that DAT is available with fairly high speed random access, is
>> there any software available to actually USE this? [...]
>>
>> 2. Do the drivers normally provided on Unix systems (e.g. ISC 386/ix 2.2)
>> allow for random seeks or would new drivers as well as application
>> software be needed? (If so, are any such drivers available?).

> [detailed explanation of DDS and SCSI operation omitted]

>A drive will automatically use "fast search" mode when told to
>skip blocks/filemarks/setmarks (SCSI "SPACE" command), or to
>position to an absolute location on tape (SCSI "LOCATE" command).
>Most drivers out there should be able to handle block/mark skipping.

I take it that, and the detail I have omitted about how it works, means
the answer to my Q2 is "Yes" i.e. a Unix device driver normally used
just for backup streaming can be expected to properly implement
commands to seek to any given location. One would not expect such a driver
to be implemented like a tty or lineprinter that doesn't know what "seek"
means (even though that would be sufficient for backup streaming). Please
confirm.

>The problem with fast searching is knowing where to go, thus either
>you have a smart backup application which keeps track of the order in
>which it puts files onto tape, or you have a dumb backup application
>and keep track of the order of files yourself. You could then use
>something like the 'mt' command to position the tape in front of
>a file you want to read.

I take it, from the lack of any other answer to my Q1, that you are
unaware of any software that uses the DDS facilities available,
although you are familiar with the technical details of what can
be done. This seems strange. What's the point of providing this stuff
if nobody uses it? This also relates to Q7 - you seem to know your
stuff and work for HP on DDS DAT - why don't they produce some
sample random access software instead of just selling it as a 
backup streamer?

>> 3. Main application I have in mind is for offline archiving of large
>> volumes of files to be available for mail requests. [...]
>>
>> 4. Is it practical to do this with filemarks instead of fully
>> random seeking? (i.e. THOUSANDS of filemarks?) Seems undesirable
>> to me as there would be a stop-start after each logical file was
>
>With DDS, a filemark uses 4 bytes of space, and a drive can "write"
>one in approx 5 milli-seconds. (this assumes you write the filemark
>in "immediate report" mode to prevent a time/space wasting flush of
>the drive's data buffe to tape).
>Thus there is not neccesarily a stop-start after a filemark is
>written.

I take it from this, and the reference to mt in your answer to my Q1
and Q2, that your answer to my Q4 is "Yes". i.e. that one could
put literally thousands of files separated by filemarks on tape
in the normal way using successive outputs to a non-rewind device 
driver and just use "mt" for skipping to the appropriate file for
retrieval and still get the full benefits of fast tape movement
without having to use any "seek" commands at block level. This
implies that the application described in Q3 may as well use
that simple filemark method (Q4) and there is no point bothering
about seeks to specific block numbers for it. Please confirm.

>> 5. A nice way to do it might be something that makes a directory
>> showing the byte offset to each file as files are being streamed
>> out through cpio or tar. Then to read you just sort the requests.
>> Normal recovery through cpio or tar would be available,
>> and faster direct access as well.
>
>You mean: have a number of cpio/tar archives on tape interspersed with
>filemarks, and skip to the archive you're interested in ???
>I guess this might work, provided cpio/tar doesn't cause a rewind
>when it "opens" the device for read/write. (I don't know much about
>this sort of stuff, so I skip to the next question ...........)

The way I understand it, cpio and tar just write to stdout. When the 
cpio or tar process finishes and closes stdout it is not cpio or tar but 
the device driver that causes a rewind unless it is a non-rewind device driver
- usually both are available (using a 1 bit difference in minor
device numbers). Thus storing a series of cpio or tar files separated
by filemarks should be absolutely no different in principle from just
storing any other files (e.g. using cat or cp).

What I meant was using a SINGLE tape file produced by cpio or tar to
hold thousands of individual files and EITHER recover individual files
or sets of them using cpio or tar (which implies slowly reading through
the whole tape file), OR seek directly to the tape blocks that hold
the individual file required. The latter alternative would require
building some sort of index to the cpio or tar file as it is being
produced. Seems to me that since you say there is no performance
penalty for using filemarks I may as well just use the "mt" approach.

[Q6. Thanks for finally convincing me there is no way around the
limitation of two append only partitions by providing full details.]

>> [Q9 and Q10 re juke boxes]

>I havn't seen any coming from Japan. There are some "stacker" devices
>for DAT on the market. Typical capacity approx 12 cassettes.

I take it the distinction between "stacker" and "juke box" is that
with a "stacker" you cannot randomly select the next cassette but
only get them one after the other in the order they were stacked
(and you would have to reload manually after getting to number 12). 
Please confirm.

If so, that would still be very useful for some computer purposes,
though less useful than a fully random access juke box. It could
even be used for the file archive application responding to mail
requests that I mentioned in Q3 (as well as for ordinary very
large backups etc.) There would just be a batch cycle proceeding
through all 12 tapes to fill any mail requests (though having
to restack after each batch is unfortunate, it could be done
only once or twice per day). Can you please provide any details 
you have available on who is marketing "stackers" and
at what price? Also, are they audio or computer devices and if audio,
would there be any difficulty using the stacker mechanism with a 
computer DAT drive? 

>> 11. The Wangtek DAT I am using has a JVC drive and an entirely
>> separate Wangtek SCSI controller which is merely screwed together
>> to make it look like a single full height drive. Would the JVC
>> drive be an ordinary audio DAT drive and would there be some
>> simple way one could feed output from an ADC to it or input to
>> a DAC from it (at least standalone audio DACs are readily
>> available for use with high end CD players)?
>
>A lot of DAT drives use mechanisms developed for audio. The similarity
>ends at this point. Most of the electronics (doens to the read/write
>amplifiers in some cases) will have undergone radical change to
>support computer data. Bolting on ADC's/DAC's becomes a non-trivial
>operation.

I don't understand this answer. My understanding was that the audio DAT
format is fully digital and the DDS format is just a particular form of
it that defines block numbers etc. If so, why should only the mechanism
be the same and not also the digital electronics (including read/write
amplifiers)? If there is a digital signal being recorded and played 
back, why should accessing that for audio be any more difficult than
recording an audio CD (digitally) on a DAT? I can understand there
might be extra stuff for fast searches etc but surely more than the
mechanism is the same.

>> 12. Likewise, could one just add a SCSI controller board to
>> a consumer audio DAT drive to get a computer DAT (as I
>> believe Wangtek may have done)? [...]

>In other words, "designing and adding a SCSI controller" may be
>all that WangDAT or any other DAT vendor has done, but that's where
>the difficulty is, and where a lot of the development and product
>costs are incurred. That's what you're paying any perceived "extra"
>for.

I just saw a note about SCSI DATs available for USD $1500, c.f.
$2500 when the one I'm using was bought. I guess the current
prices are closer to reflecting the design costs you describe.

>Price "erosion" (I think that might be a marketing term) will occur.
>I wouldn't expect DAT to become as cheap/cheaper than CD rom.
>There's an awfull lot more mechanism and electronics in a DAT
>drive than in a CD ROM.

Hmm, well CD is becoming VERY cheap and I can see how the mechanism
differences alone will always keep DAT more expensive. But electronics
is becoming so cheap that I think one can hope for SCSI controllers
for audio DAT drives that don't add much to the cost (with the
code and integration development costs amortized over many thousands
of users, what's 0.5 MB of RAM and a few VLSI chips these days :-)

If audio DAT takes off as a mass market competitor to ordinary audio
cassettes (and CDs) then prices would have to end up comparable to
CDs even though a bit greater (and less than VCRs which have similar
mechanisms and are now getting to be fairly cheap).

It will be VERY interesting when a typical western style home with
hi fi equipment has 1.2 Gb or 2.4 Gb of tape storage sitting there
waiting to be used, along with the 550 MB CD audio/CD ROM and the
workstation quality HDTV (and 2 x 64 kbs circuit switched + 1 x 16 kbs 
packet switched ISDN line) - adding a keyboard and CPU box to get
a pretty powerful computing environment won't be such a big deal :-)

Anyway, thanks again for the helpful answers.

--
Opinions disclaimed (Authoritative answer from opinion server)
Header reply address wrong. Use cmf851@csc2.anu.edu.au