Newsgroups: comp.std.c
Path: utzoo!henry
From: henry@zoo.toronto.edu (Henry Spencer)
Subject: Re: The Preprocessor and tokens
Message-ID: <1991Mar25.190133.16977@zoo.toronto.edu>
Date: Mon, 25 Mar 1991 19:01:33 GMT
References: <3137@wyse.wyse.com>
Organization: U of Toronto Zoology

In article <3137@wyse.wyse.com> cmills@wyse.wyse.com () writes:
>...I was under the impression
>that the preprocessor and the compiler agreed on what constituted a
>token... 

Unfortunately, no.  There is a separate notion of a "preprocessing token",
which is like a real token in a lot of ways but has a much more generous
syntax for numbers.  This is what you're running afoul of.

The problem is that there are demented people who want to use vile and
disgusting perversions like token concatenation to actually put numbers
together out of pieces at preprocessing time.  The lexical syntax of C
numbers is really ugly, and trying to produce a lexical definition of a
valid *piece* of a number is agonizing.  X3J11 therefore came up with
the notion of a "preprocessing number" which covers anything vaguely
number-like, and stipulated full lexical validation of numbers only
after preprocessing.
-- 
"[Some people] positively *wish* to     | Henry Spencer @ U of Toronto Zoology
believe ill of the modern world."-R.Peto|  henry@zoo.toronto.edu  utzoo!henry