Xref: utzoo comp.unix.wizards:11511 comp.unix.questions:9559
Path: utzoo!utgpu!water!watmath!clyde!att!osu-cis!tut.cis.ohio-state.edu!bloom-beacon!think!ames!amdahl!pyramid!prls!mips!dce
From: dce@mips.COM (David Elliott)
Newsgroups: comp.unix.wizards,comp.unix.questions
Subject: Re: Redirecting output in AWK
Keywords: 10 file limit ?, close
Message-ID: <4449@dunkshot.mips.COM>
Date: 4 Oct 88 14:28:33 GMT
References: <816@mergvax>
Reply-To: dce@dunkshot.UUCP (David Elliott)
Distribution: na
Organization: MIPS Computer Systems, Sunnyvale, CA
Lines: 75

In article <816@mergvax> rkxyv@mergvax (Rob Kedoin) writes:
>
>I am trying to use AWK to split one file into many formatted, smaller files.
>
>The problem I am having is that I cannot output to more than 10 files.  I am
>doing my output in a manner similar to:
>	MY_FILE="foo";
>	printf("abc\n") >> MY_FILE;
>	close(MY_FILE);
>Even though I am executing the close, I am still running out of files.  Does
>anyone know of a fix to this ?

I tested awk (SVR3.1) using '>' on our system and it works just fine.
Are you sure that '>>' is affected by close?

Anyway, in case you can't get a fix in time, here are some tricks you
can use to get the job done:

1. Use awk to generate a set of lines of the form:

	filename text

   and pipe it to a shell "while read" loop.  The code would look
   like:

	awk '{ ... }' | sed 's/\\/\\\\/g' | while read file line
	do
		echo "$line" >> "$file"
	done

   (The sed script is required if your 'read' processes backslashes; mine
   does.)

2. Make multiple passes on the input, processing only 9 files per pass,
   putting error messages in the 10th.  The code would look like:

	INPUT=/usr/tmp/script.input.$$
	TEMPF=/usr/tmp/script.tempf.$$
	ERRORS=/usr/tmp/script.errs.$$
	cp input $INPUT
	while [ -s $INPUT ]  # while input is not empty
	do
		awk 'BEGIN {
			nfiles = 0;
			errfile = "'$ERRORS'";
		}
		(nfiles > 9) {
			print;
			next;
		}
		{
			process input, putting errors into errfile and
			incrementing nfiles when you open a new file
		}' $INPUT > $TEMPF
		if [ -s $ERRORS ]
		then
			cat $ERRORS 1>&2
			break
		fi
		cp $TEMPF $INPUT
	done
	rm -f $INPUT $TEMPF $ERRORS

3. Combine these two, doing 9 files of output in the awk, and printing
   the "filename text" lines to stdout for the while loop to process.
   If you have fewer than 10 files, you get all the work done in awk,
   and if you have more, you still come out OK, but slower.

   This should be easy to do if you define a function to do your
   outputting, letting it keep track of the number of output files
   and deciding whether to put the output into a file or to format
   it to stdout.

-- 
David Elliott		dce@mips.com  or  {ames,prls,pyramid,decwrl}!mips!dce