Path: utzoo!censor!geac!torsqnt!news-server.csri.toronto.edu!cs.utexas.edu!wuarchive!zaphod.mps.ohio-state.edu!maverick.ksu.ksu.edu!ux1.cso.uiuc.edu!news
From: olson@sax.cs.uiuc.edu (Robert Olson)
Newsgroups: comp.lang.perl
Subject: Large, sparse dbm databases
Message-ID: <1991Feb7.044015.10585@ux1.cso.uiuc.edu>
Date: 7 Feb 91 04:40:15 GMT
Sender: news@ux1.cso.uiuc.edu (News)
Organization: University of Illinois at Urbana
Lines: 68

I am using a dbm database (via dbmopen) to create a rather large,
sparse database. My data fits nicely in the form...
	$db{key1}
	$db{key1, key2}
	$db{key1, key2, key3}
	etc
where there are relatively few values of each of the first couple
keys. 

(Perhaps an example is in order. The data is information about classes
gleaned from the g++ compiler.

	$db{"class"} = "X,Y,Z"	
	$db{"class", "X"} = "method,field"
	$db{"class", "X", "field" } = "field1,field2,field3"
	$db{"class", "X", "field", "field1"} = "type,offset"
	$db{"class", "X", "field", "field1", "type" } = "int"

As you can guess, there are a large number of different keys in the
second and fourth positions and relatively few in the first and third.
)

Some of the values in the db are list which I just represent as comma
separated lists of strings. It is easiest given the data I create the
database from to store the initial value of the list in the database,
and append succeeding values to the list as I read them from the
input.

Problem: I get errors such as

  dbm store returned -1, errno 28, key "class^\InfrastructureManager^\method"
	at db.pl line 42, <> line 810.

I get errno 28 to be No space left on device. This shouldn't have been
the case; there were >50M free on the disk. BUT, I guess this is
telling:

% ls -sl class*
  28 -rw-rw-r--  1 olson       61440 Feb  6 22:25 classinfo.dir
4840 -rw-rw-r--  1 olson    491334656 Feb  6 22:25 classinfo.pag

Not much real storage, but lotsa holes.... (Note: this is for the
preallocated version (see below). The unpreallocated versions showed
similar behavior but with the apparant size only in the 100M range).

The first thing I tried was to make each key a constant length, so
that instead of $db{"class", "X"} I had $db{"class", "X", '', '', ''}.

This didn't seem to change much. Suspecting that appending to database
elements (eg $db{"class"} = $db{"class"} . ',' . $elt) was hurting me,
I tried preallocating the value via
	$db{"class"} = ' ' x 100; $db{"class"} = $firstValue
but that didn't seem to help either.

From what I did it may be obvious that I do not understand the
workings of dbm files...

Any suggestions?

--bob

PS. Apologies if this gets posted twice; the first one didn't show up on the
nntp server...
--
Bob Olson			University of Illinois at Urbana/Champaign
Internet: rolson@uiuc.edu	UUCP:  {uunet|convex|pur-ee}!uiucdcs!olson
UIUC NeXT Campus Consultant	NeXT mail: olson@fazer.champaign.il.us
"You can't win a game of chess with an action figure!" AMA #522687 DoD #28