Path: utzoo!attcan!uunet!cs.utexas.edu!asuvax!stjhmc!p6.f18.n114.z1.fidonet.org!will.summers
From: will.summers@p6.f18.n114.z1.fidonet.org (will summers)
Newsgroups: comp.lang.c
Subject: Turbo C large character array
Message-ID: <24241.26BED998@stjhmc.fidonet.org>
Date: 7 Aug 90 15:00:01 GMT
Sender: ufgate@stjhmc.fidonet.org (newsout1.26)
Organization: FidoNet node 1:114/18.6 - Iasd Eng Bbs, Phoenix Az
Lines: 155

In article <1990Jul27.193520.4689@ux1.cso.uiuc.edu> gordon@osiris.cso.uiuc.edu
(John Gordon) writes:

 > I am trying to declare a char array that is rather large, and TCC
 > won't accept it, it says it is too large to fit within
 > available memory.  Bull!  I know for a fact that there is at
 > least 300 or 400K left.  Anyway, here is the declaration:
 >
 >                 char menu[1200][80];

   WARNING: Kludges C programers and compiler writers use
   to cope with the most prevalent hardware archetecture on the
   face of the earth are about to be discussed.  Not a few
   consider it the most twisted archetecture on the face of the
   earth but they probably exagerate a little.  Nevertheless,
   this topic has been known to make grown men cry.  Do not
   read on a full stomach.

The definition of the "huge" memory model varies.  For example, in
Microsoft C, the huge model allows objects over 64K in size.
It's been a long time since I MS C'ed, but if memory serves no
array element could be over 64K (I don't know if this outlaws
a[70000][4], but it does seem to preclude a[4][7000] ) and
large arrays had to have elements whose size was a power of
2 -- ie.  no

    struct x_ { char array[13] } a[10000];     /* WRONG in MSC */

Under Turbo C, the huge model means something quite different.  First,
under Turbo's large model you could have up to 1MB of data, but
only a _total_ of 64K of static data (the rest would have to be
malloc'ed or on the stack).  Under either large or huge models,
you have to be careful dealing with objects over 64K, either by
declaring pointers to such objects as "huge" or by being _very_
careful to ensure that segments do not "wrap" and that
normalization occurs before pointer comparisons.  (yucky in
the extreme).

The change to the huge model relaxes the restriction on a 64K
_total_ of static data, but still restricts the amount of static
data declared in any module (compile file) to 64K.  This
implies that any object of static data must be less than 64K.

Other compilers no doubt have other subtle differences in what is
and what ain't acceptable under their large and huge
models, and how they handle/not handle 'far' and 'huge' pointer
normalization and wrap.  (Does any '86 compiler handle
p +=70000;  in an "unsurprising" way when p is a 'far' pointer?)

There are a couple of ways to get arround this restriction.
The first **which I've never used** is to use farmalloc() and
'huge' pointers.  'huge' pointers are kept "always normalized" and
let you manipulate objects over 64K without having to worry about
"segment wrap".  (At least that is what is advertised:
remember, I've never used the damn things -- the performance
hit is reported to be devastating.  For example ++p generates
a _subroutine_ call!) I don't recommend using them in code you
write, but they may save the day when you are under time pressure to
get code imported from an non-'86 source to run.

A second way is to use farmalloc and far pointers.  Like in the
large model _you_ are responsible for seeing that the pointers
don't "wrap" and that they are normalized before comparisons
are made ( under some compilers, 2 far pointers can point to
the same object but compare _unequal_; under others, a pointer to
a[5] could compare as _less_ _than_ a pointer to a[3]. Neat.).  Not
recommended for any but the most careful, experienced coders, and
then only to hand-optimize code developed and checked out
using 'huge' pointers.  I've never done this either, and hope I'm
never tempted to, because there is a third alternative much cleaner:



I recommend taking a page from the past.  On early
machines there was no multiply instruction.  So multiplication
was done in subroutines.  This was very slow.  So compilers would
pre-calculate the offsets of the rows of an array at compile time
and store the results into an array.  The run-time would access
the array to compute multi-dimensional array offsets without
having to multiply.  These were called "dope vectors", possibly
because they made up for "dope" machines too dumb to be able to
multiply.  I suggest you "roll your own dope vectors" when
dealing with large arrays under the Intel archetecture.

Something along the following lines:

#define NUMLINES 1200
#define LINESIZE  80

  /* declare menu as an array of pointers to char:  */

 char *menu[NUMLINES];    /* You don't need the 'far' keyword if you are
                             compiling in the medium, large or huge models.
                             For portability I suggest leaving it out */

/*  malloc the individual rows of the array   */

   int line;

    for (line=0; line<NUMLINES; line++)  {
        menu[line] = (char *) malloc (LINESIZE * sizeof(char));
        if (menu[line] == (char *) NULL)    {
            punt("Not enough memory for menu\n");
            crapout();
        }
    }

    You can refer to individual bytes as menu[line][column] just as if
    menu were a two-dimensional array, even though menu is a
    single-dimensional array of pointers to 1200 80-character arrays of
    char.

    The performance hit is that you do 2 memory de-references per
    access, one near and one far.  I'd venture to say that on
    some compilers this is _faster_ than "real" two dimensional
    array access in the large model as it avoids a multiply at
    the expense of a near memory access.  (Anyone care
    to post some benchmark results?)

Note: Under this method code is portable to environments
that never heard of memory segmentation without conditional
compilation and without changes to code or header files.  It is
also portable with far less chance of suprises to PC compilers
that have subtle, sometimes quite unexpected, differences in
the way they treat 'far' and 'huge' pointers, "large" arrays, and
large and huge memory models ( Admit it: would you think to
design your code to anticipate a "the size of array elements must
be a power of 2" restriction if no one told you of a compiler
that had such a restriction?)

One difference between this and a "real" two dimensional array is
you can't play pointer punning "games" with the object like
assigning a pointer to char to &menu[0][0] then trying to
increment it through all 96000 elements...  but then you'd not do
anything like that, now, would you?  (If you would, never admit
it!)

A final note: at the expense of portablility, this is a
reasonable way to handle large amounts of data in a small or
compact model program.  You declare menu as an array of _far_
pointers to char.  You use farmalloc (and farfree) instead of
malloc, and you cast the result of malloc to _far_ pointer to
char.  The parts of the program that do not refer to the far data
will run a good deal faster.

                                  \/\/ill

PS.  Ain't the '86 archetecture wunnerful.


 

--  
Uucp: ...{gatech,ames,rutgers}!ncar!asuvax!stjhmc!18.6!will.summers
Internet: will.summers@p6.f18.n114.z1.fidonet.org