Xref: utzoo comp.sources.d:3992 comp.os.vms:17112
Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!ncar!noao!arizona!naucse!jdc
From: jdc@naucse.UUCP (John Campbell)
Newsgroups: comp.sources.d,comp.os.vms
Subject: Re: Flex (latest 2.1 beta) on VMS
Message-ID: <1664@naucse.UUCP>
Date: 25 Aug 89 17:26:57 GMT
References: <13603@bloom-beacon.MIT.EDU>
Organization: Northern Arizona University, Flagstaff, AZ
Lines: 48

From article <13603@bloom-beacon.MIT.EDU>, by scs@adam.pika.mit.edu (Steve Summit):
: In article <1629@naucse.UUCP> jdc@naucse.UUCP (John Campbell) writes:
:>The new flex reads a large chunck at a
:>time.  With VMS STREAM-LF files this works just fine--but with "normal"
:>VFC editor text files (darn these RMS things) the VMS 'C' rtl will only
:>return at most 1 record full of characters for any large read() byte
:>request.
:>During processing on a flex input file, flex complains of a "NULL in
:>input."  This seems to be because yyunput() wants to "shift things up
:>to make room" and assumes that the end of the valid buffer is around
:>YY_BUF_SIZE deep.
: 
: This sounds like a bug in flex.  If I understand the complaint
: correctly, the code gets confused when the buffer is not (?)
: substantially full.  (This sounds odd; code usually fails when
: buffers fill up, not when they stay relatively empty.)

I'm the original poster of the first article.  To date I have not proven
that there is a bug in flex.  I still believe this is true, but I haven't
had the time to make the obvious tests in other environments.  I still
believe that yyunput() is in error in some way, but I would like to have
some follow up information before I bug Vern Paxson with my worries.

I can tell you that using VMS fread() instead of read() for initscan.c
seems to work just fine on all format of input files:  

#define YY_INPUT(buf,result,max_size) \
        if ( (result = fread(buf, 1, max_size, yyin)) == 0 ) \
            if (ferror(yyin))\
               YY_FATAL_ERROR( "fread() in flex scanner failed" );


I'm a little confused because, with no change to the yyunput() routine, I 
can run one of my old programs which has to replace YY_INPUT() with my
own read routine to toss out '\0's.  I find that this routine, which does
not try to fill up to max_size works for the regular expressions that
I analyze.  At this point I assume there is something more complicated
going on in the lexer for flex itself than in the small ditty I wrote.

Anyway, if anyone would make some YY_INPUT() substitutions (like use
getc()) and see what happens when they recompile and test flex itself
we'd all be very happy.  I work for a university that is about to start
classes so I'm overwhelmed right now.

-- 
	John Campbell               ...!arizona!naucse!jdc
                                    CAMPBELL@NAUVAX.bitnet
	unix?  Sure send me a dozen, all different colors.