Xref: utzoo comp.std.c:874 comp.lang.c:16785
Path: utzoo!utgpu!jarvis.csri.toronto.edu!mailrus!tut.cis.ohio-state.edu!ucbvax!decwrl!sun!imagen!atari!portal!cup.portal.com!Kevin_P_McCarty
From: Kevin_P_McCarty@cup.portal.com
Newsgroups: comp.std.c,comp.lang.c
Subject: detecting invalid pointers
Message-ID: <15495@cup.portal.com>
Date: 7 Mar 89 09:35:15 GMT
Distribution: usa
Organization: The Portal System (TM)
Lines: 52

Is there any guaranteed way to detect an out of range pointer,
i.e., one which is supposed to point into an array but might not?

For example, a friend had something like

     int  x[TABLESIZE];
     int  *p;
     ...
     checkrange(p, __FILE__, __LINE__);

where

     checkrange(int *p, char *fname, int lineno)
     {
          if ((p < x) || (p >= x+TABLESIZE)) {
               /* error message: out of range pointer */
          }
     }

My first reaction was that one should not rely on plausible behavior
of an out of range pointer in a comparison, since that is undefined.
For example, the null pointer need not pass either comparison.  That's
easy to remedy; append `|| (p == NULL)' to the condition.

It is conceivable, and I can't find anything that would rule it out,
that a non-null pointer p which did not point into x might fail
(p < &x[0]) and/or fail (p >= &x[TABLESIZE]).  Trichotomy can fail
because pointers need not have a global linear order.

My initial response was to recommend what I thought was a stronger,
more reliable test, namely

     if ((p != NULL) && (p >= x) && (p < x+TABLESIZE)
          /* p is in-range */
     else
          /* p is invalid */

But this is little better.  While it is conceivable that if p and q
are non-null and incomparable (don't point into the same array),
none of (p < q), (p == q), (p > q), (p < q+n) holds,  it is harder to
conceive the possibility that (p > q) *and* (p < q+n) could hold, but
I can't find anything to rule that out either.  While perhaps
almost all implementations would behave reasonably here, what
guarantees that one of these comparisons must fail?

A test like
     if ((p != NULL) && (p-x >= 0) && (p-x < TABLESIZE)
is subject to the same doubts.

What's the best way to test this?

Kevin McCarty