Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!swrinde!elroy.jpl.nasa.gov!decwrl!pa.dec.com!shodha.enet.dec.com!alan From: alan@shodha.enet.dec.com ( Alan's Home for Wayward Notes File.) Newsgroups: comp.sys.dec Subject: Re: rz57 and block size Summary: I think I said that... Message-ID: <3433@shodha.enet.dec.com> Date: 20 Jun 91 23:58:50 GMT References: Distribution: comp Organization: Digital Equipment Corp. - Colorado Springs, CO. Lines: 108 But I'm not sure how it got here. In article , dascher@stanley.cis.brown.edu (David Ascher) writes: > Hi all. > > I'm about to start a very large newsfeed and use a good chunk of an > rz57 as storage medium... As usual, the problem that comes up is > blocksize, since small blocksizes allow for an efficient storage of > thousands of very small articles. Problem is, it isn't "legal" to > format an rz57 with a blocksize of less than 4k... Which is a real > problem. When the question was asked originally (internally) I wondered at the specific problem. The first, and one I think the one you percieve is that it would nice to have block size of 2 KB. Sorry. 4 KB and 8 KB are the only ones supported. The other problem, the answer to which you quote, is that on some disks many small files means you'll run out of inodes. There is a newfs (mkfs) parameter that lets you control the number of inodes created. The default value is 2048 bytes of disk/inode. I've had exceptional success making it larger and getting fewer inodes, but not as much getting more. That was a few versions ago and it may well have been broken. The work-around to this problem is: > > Some people at DEC have told me that the only way around this is, and > I quote: > > "The solution to this problem is to lie about the geometry of the disk > and tell it that there are more, but smaller, cylinders". More cylinders mean more groups of inodes and therefore more inodes. If you have a habit of running out of inodes it's a reasonable solution. The real solution is the invention of the "Fat" file system. > > Have any of you ever done this? Any experts out there who can either > tell me if my fears are unfounded, or where problems may crop up? Interesting question. Are you're fears indeed unfounded? I looked my news disk (currently an RA82) and discovered 161,581 files. The average filesize is 2,978 bytes. The distribution goes something like: Number <= KB 31 0 (.04%) 38613 1 (24%) 74794 2 (46%) 25050 3 (16%) 9537 4 (6%) 4766 5 (and so on...) 2551 6 1210 7 798 8 2095 16 499 24 305 32 221 40 309 48 304 56 426 64 71 > 64 KB (largest 1,204,224) So, a lot of the files will be less than 2 KB. But a lot will be larger. Now consider the fragment allocation scheme. In one of the ULTRIX V4 releases we added (or at least documented) a flag to mkfs(8), to set the file system optimization flag. The choices are to optimize for space or time. In the case of space, the file system prefers to allocate a file fragment with existing fragments, rather than allocate a new fragment. So, two 3 KB files will only use one 8 block on disk with 2 KB leftover. In the time case it allocates a new fragment. This way if the file expands, it doesn't have to be copied to a new fragment with enough space. For news, I think the choice would be to optimize for space, which is also the default. So, if you could choose a file system block size of 2 KB you constrain the file system to doing 2 KB I/O for any file, big or small. Hundreds of small files will still packed as tightly together as possible. If you use a larger block size, then reading back the larger files get the benefit of the larger block size. I/O for the small files should only a block size that is "enough". I think your fears are unfounded. If you can get enough inodes. > > Thanks for any info. > > -david ascher > Brown University Computing and Information Services > David_Ascher@Brown.EDU -- Alan Rollow alan@nabeth.cxn.dec.com