Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!swrinde!mips!news.cs.indiana.edu!rutgers!psuvax1!news
From: flee@cs.psu.edu (Felix Lee)
Newsgroups: comp.lang.perl
Subject: Re: ptags - perl tags in perl
Message-ID: <lt5Gizfq1@cs.psu.edu>
Date: 16 Apr 91 01:24:49 GMT
References: <128366@uunet.UU.NET> <1991Apr11.212214.27164@convex.com>
	<oebG+9jo1@cs.psu.edu> <129114@uunet.UU.NET>
Sender: news@cs.psu.edu (Usenet)
Organization: ~/News/org
Lines: 79
Nntp-Posting-Host: guardian.cs.psu.edu

>Hmmmm, what's a token?

A token is a lexical unit.  What is a lexical unit?  Something
convenient for the parser.

I lied.  It's not just a stand-alone scanner.  There's also a
stand-alone parser.  One of my subgoals in building the thing was to
make the pieces easily decomposable.  So the stand-alone scanner is
just a small wrapper around gettoken().

>How do you parse: $foo =~ s/$bar/tr;a-z;A-Z;/e
	dvar "foo"
	"=~"
	substitute-begin "/"
	dvar "bar"
	substitute-to
	translate-begin ";"
	string "a-z"
	translate-to
	string "A-Z"
	translate-end
	substitute-end

This needs a little explanation.  A double-quoted string like
	"$a is $b"
becomes the token sequence
	dquote-begin
	dvar "a"
	string " is "
	dvar "b"
	dquote-end
and things inside s//, m//, and tr// are treated similarly.

For s///, the scanner always returns an expression for the second
clause.  So if you don't have a trailing "e", the scanner adds a
dquote-begin/dquote-end pair.
	s/foo/bar/
becomes
	substitute-begin "/"
	string "foo"
	substitute-to
	dquote-begin
	string "bar"
	dquote-end
	substitute-end

>How do you deal with: &qaz(<<foo,<<bar)
	call
	string "qaz"
	"("
	dquote-begin "foo"
	string "..."
	dquote-end
	","
	dquote-end "bar"
	string "..."
	dquote-end
	")"

<< quoting and formats require multiply buffered input.  Input is
buffered in a "line stream".  The line stream is buffered in a
"character stream".  The scanner normally reads from the character
stream, but << quoting and formats will read from the line stream.

>What about q (or $) followed by newline or tab?

Left as an exercise to the reader.

Warning.  I haven't actually implemented all of the above, so some of
what I just described may be unwieldy in practice.

The last thing I did, about a month ago, was add full expression
parsing to the yacc grammar.  The grammar has no s/r or r/r conflicts
and uses no precedence rules, but I'm not certain it actually parses
the same language that "perl" does.  I was in middle of rewriting the
scanner to handle the new parser requirements when the project dropped
by the wayside.
--
Felix Lee	flee@cs.psu.edu