Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!swrinde!mips!apple!amdahl!kim From: kim@uts.amdahl.com (Kim DeVaughn) Newsgroups: comp.sys.amiga.tech Subject: Re: when is a block not a block? Summary: a block by any other name ... Message-ID: <648H02l0b8KM01@amdahl.uts.amdahl.com> Date: 15 Jul 90 23:01:35 GMT References: <6498.269a4527@vax1.tcd.ie> Organization: Amdahl Corporation, Sunnyvale, CA 94086 Lines: 260 In article <6498.269a4527@vax1.tcd.ie>, cpmurphy@vax1.tcd.ie writes: > >So I had a good look at the new 'ls' program that was >in comp.sources.amiga recently. There I was alarmed to see that there is a bug >in 1.3 to do with file blocks. Apparently using Examine() or ExNext() with a >FileInfoBlock structure will not always give the correct number of blocks taken >up by a file. Well ... the real "bug" is that the only place it's "officially documented" (that I could find) is in an old AmigaMail from CBM (the Nov/Dec 1988 issue, in an article by Steve Beats on the differences between the FFS and the OFS). To quote from the "struct FileHeaderBlock" listed on page 5 of that issue: ULONG HighSeq; /* total blocks used in file (not updated by FFS, only the old filing system) */ ULONG DataSize; /* number of data blocks used for this file */ (sic) One other mention of this is made by Betty Clay in her fine article on the FFS in an issue of the now (sadly) defunct TrasnAmi journal, from about the same time. In it she mentiond that a USENET posting describes this, but I was unable to locate any such from my "archives" of "probably useful information". As may be, once you figure out that what we're really talking about is a part of what is known as the "struct FileInfoBlock" as defined in the dos.h/dos.i include files, and you equate "HighSeq" with "fib_NumBlocks", and you decide to ignore the comment associated with "DataSize" above as a typo, things start to make sense ... :-) Oh yes, one other thing, the comment on "HighSeq" (fib_NumBlocks) is wrong. The field *is* updated by the FFS ... it's just not accurate (usually off by 2 or 3 blocks, but that depends on the file's size). For example, a file I have is 135046 bytes long. On an FFS device, the fib_NumBlocks entry says 267 data blocks, whereas the file actually has only 264 data blocks. And no, fib_NumBlocks isn't adding in the file's header and extension blocks for you. The file actually occupies 268 blocks total (264 data, 1 header, 3 extensions). BTW, on an OFS device (floppys, for example), this file will have 277 data blocks, and occupy 281 blocks total. >The way to get around it, so it says, is to use Info(). You use the call to Info() to find out the number of data-bytes per data-block (id_BytesPerBlock in the InfoData struct). You use this to "correct" the value found in fib_NumBlocks (see below). > I'm a little bit confused by all of this. Could someone please enlighten me. I can't imagine why ... :-) Actually, this whole area is one of the most poorly "documented" areas in the entire AmigaOS (along with the console/CLI interface). Due, I'm sure, to the braindead TriPOS/BCPL crap that had to be grafted on top of an otherwise pretty nice OS at the last moment, many years ago. CBM has been working hard to exorcize this stuff, and I really hope to see the interfaces and documentation improve with 2.0. From the few "bits'n'pieces" that have been mentioned here at times, I don't think I'll be disappointed. One other problem is that none of this information is in the RKM's. They just point you to the Bantam AmigaDOS manual, which tells you very little, and what it does say is quite out of date. >How do I get the "real" number of blocks taken up by a file so as to avoid this >bug? Say I have a file on a HD and I want to copy it to a floppy. How do I >ensure that there is enough space on the floppy for this file? Is it to convert >it's size in bytes to "floppy blocks", then compare it to the Info() from the >floppy? Well, assuming your HD partition is an FFS one, you could just say: "ls -lB488 filename" using ls v4.0k, take a look at the number of blocks reported, and see if your floppy has that many blocks left on it (with the AmigaOS command "Info"). See the ls docs for further examples. Or you could use Mike Meyer's "willfit" pgm. I think he's calculating things correctly, as we hashed this over in email awhile ago. Don't use the recently posted "du" (v1.2), as it does not always give the correct number of blocks on large files. I've reported this to the author. If it's for your own code, you can use/modify the routines that handle this in ls ... fixNumBlocks() and blkalloc(). You first call fixNumBlocks(), then blkalloc() for each file of interest (though if the file of interest is on an OFS device, it isn't strictly necessary to do the fixNumBlocks() call). I've appended these two little routines below (edited somewhat to remove some "extraneous" comments :-) ). In subsequent postings, Mike Meyer said: > >There isn't a bug in 1.3 about the number of blocks a file uses. >There's just a difference of opinion between OFS and FFS about how >what that phrase means. One of them counts the number of data blocks, >the other counts all the blocks used by the file, both data and >overhead. (Question to CATS: Is this considered a bug? And is it fixed >in 2.0?) Not quite, Mike (see the example above). Basically, you cannot depend on the value in fib_NumBlocks to tell you anything meaningful if the device is an FFS one. And no, it's not a "bug". Since it's "documented", that makes it a "feature", right? :-) What *is* a bug, is that the official 1.3 includes don't tell you that fib_NumBlocks is meaningless on FFS (nor do the AutoDocs, RKM's, etc). >I haven't checked 2.0 yet, but on 1.3 there's no way (that I >know of) to determine how many data blocks/header block. On the other >hand, everything uses 72. There are a couple of "interesting" items that could be of further use in Steve Beat's AmigaMail article: "Please note that these structures assume a fixed, 512 byte disk block. This is a valid assumption for filing systems up to V1.3. However this will NOT be the case under V1.4 Kickstart which will support full variable sized disk blocks. " and: "Hash table size will always be set to 72 for instance, this can be used as one of the consistency checks before proceeding to directly read from or write to the disk. " and (in BOLD face): "Never use this knowledge in an application program. It is guaranteed to break under V1.4 Kickstart. " and (of possible real use): "struct RootBlock { [...] ULONG HTSize; /* Size of the hash table in longwords, must be set to 72 */ " Nevermind that I have yet to find a RootBlock (etc.) struct in any of the includes or headers. At least there is a place for the information to reside. Whether or not it will actually be used is another story ... though it would seem to be a shame to keep the 72 blocks/extent limitation if the block sizes on some future (hypothetical) filesystem were to be of 4K size, or even 1K for that matter. Until this stuff gets *documented*, going with "72" is about the best you can do, I guess. Perhaps the "Keeper of AmigaDOS" could comment (Hi, Randell) ...? /kim The routines fixNumBlocks() and blkalloc() from ls v4.0k: #define MAX_BLKS_PER_EXTENT 72 struct InfoData *CurID = 0; /* Global InfoData for Info() */ /* * fixNumBlocks() - A hack to fix the fib_NumBlocks field so it is correct. * * It was busted with the introduction of the FFS in AmigaDOS 1.3. Don't * ask where this is documented ... as far as I know, it isn't; not in the * 1.3 includes, autodocs, readme's, user's manual, RKM's, or DevCon notes. * It is mentioned (sort of) in an AmigaMail, and I'm told it was mentioned * once in a message on USENET. * * As may be ... since fib_NumBlocks was in use thruout this code before * this was discovered, it was easiest to just fixup the fib_NumBlocks when * the file/dir gets initially Examine/ExNext'd, with a call to Info() for * *all* files (which means alot of unnecessary calls get made [for files * all in the same dir, etc]). The performance penalty ends up being about * 1 sec, for a tree with 537 entries (which seems acceptable to me). * * Helluva way to run a ship ... * * /kim /\;;/\ * */ /* CurID is allocated in _main() during initialization, etc. */ VOID fixNumBlocks(lock, fib) struct FileLock *lock; FIB *fib; { LONG *nb; LONG bsize; static int errflag = 0; if ((LSFlagsX & NOFIXNUMBLOCKS) != 0) return; if (blksize == 0) { /* Try to fill InfoData, bomb if not readable for some reason (like */ if (Info((BPTR)lock, CurID) == 0) /* ls'ing a "pathass'd" assign) */ { /* Print error msg only once, and only if we'll be printing block counts */ if ((errflag == 0) && ((LSFlags & (LONGLIST | TOTALIZE)) != 0)) { errflag++; asprintf(workstr, "\nls: warning (%ld): block count(s) may be inaccurate - see ls.doc\n\n", IoErr()); WSTR(workstr); } return; } else bsize = CurID->id_BytesPerBlock; } else bsize = blksize; /* non-0 blksize used to force the size of your */ /* choice ... it is set from the cmd args parse */ /* routine for the -B option */ nb = (LONG *)&(fib->fib_NumBlocks); *nb = (fib->fib_Size / bsize) + ((fib->fib_Size % bsize) ? 1 : 0); } /* * blkalloc() - Returns the actual number of blocks allocated by a file/dir * (or just the data blocks, if DATABLKSONLY is set). * * Assumes 1.3 original or fast filesystem (and a fixed up fib_NumBlocks). * */ LONG blkalloc(fib) register FIB *fib; { if ((LSFlagsX & DATABLKSONLY) == 0) { return(fib->fib_NumBlocks + (fib->fib_NumBlocks / MAX_BLKS_PER_EXTENT) + ((fib->fib_NumBlocks % MAX_BLKS_PER_EXTENT) ? 1 : 0) + ((fib->fib_Size == 0) ? 1 : 0) /* kludge for 0-len files (and dirs) */ ); } else { return(fib->fib_NumBlocks); } } -- UUCP: kim@uts.amdahl.com -OR- ked01@juts.ccc.amdahl.com or: {sun,decwrl,hplabs,pyramid,uunet,oliveb,ames}!amdahl!kim DDD: 408-746-8462 USPS: Amdahl Corp. M/S 249, 1250 E. Arques Av, Sunnyvale, CA 94086 BIX: kdevaughn GEnie: K.DEVAUGHN CIS: 76535,25