Path: utzoo!utgpu!news-server.csri.toronto.edu!rpi!zaphod.mps.ohio-state.edu!mips!daver!leadsv!pyramid!athertn!hemlock!mcgregor
From: mcgregor@hemlock.Atherton.COM (Scott McGregor)
Newsgroups: comp.cog-eng
Subject: Re: Images vs. Text: Syntactic Issues
Message-ID: <34984@athertn.Atherton.COM>
Date: 8 Apr 91 17:34:43 GMT
References: <1991Apr7.184708.22888@colorado.edu> <10292@pitt.UUCP> <1991Apr6.163454.2454@watdragon.waterloo.edu>
Sender: news@athertn.Atherton.COM
Reply-To: mcgregor@hemlock.Atherton.COM (Scott McGregor)
Organization: Atherton Technology -- Sunnyvale, CA
Lines: 180

In article <1991Apr7.184708.22888@colorado.edu>,
ralex@tigger.Colorado.EDU (Repenning Alexander) writes:

> abstract nouns representing concepts: e.g., a loan
>  cannot be mapped to a single "real" object

If an abstract noun has a direct affect on the real world, then it is often
sensible to indicate the abstract noun by an aspect of it which appears in
the real world.  For instance, a loan contract is a physical world thing,
the loan an abstract concept.  Yet it is very common for people to treat,
in WORD, in ACTION, and in PICTURES the physical world representation as
the loan itself (e.g. "we need your signature on the loan" (paper form
is meant but not stated explictly).  This works, because people understand
discourse in terms of context.  A picture of a loan form, or a check, or
of the collatoral might be sufficient in a given context either separately
or together to indicate the abstract loan.  Now the word Loan may well
do this just as well.  One of the nice thing about "graphical" representations
is that they support both text and pictures, and these can be combined
to help convey information more effectively, to more people, than either
alone could.  Of course this is dependent upon the skills of the designer.

> adjectives
>   simple if property is intrinsically visual, e.g., color
>  otherwise, .. hmmm..

Here again, convention and context stand in for grammar, and allow 
adjectives to be represented in other ways.  Standard alternatives
are to use a different visual or temporal attribute.  For instance
a tool for exploring particle physics might represent quarks as circles
and show their "color" by using a corresponding "color" and their "spin"
by a combination of arrows  and/or animation of the spin.  However,
the terms color and spin applied to quarks do not REALLY have any of
these properties in the sense that apples are red (i.e. electro-magnetic
radiation reflection characteristics) or that wheels spin (angular
momentum).  Rather in the case of quarks, they are made up terms for
abstract qualities only having to do with conservation and quantum
numbers, but which have no other macro world meaning.  As you can see
from this example, this use of substitute terms, for things that we
cannot "see" is common not only in graphical renderings, but also in
our verbal language itself. 

In the case of adjectives, color, shape, border characteristics,
physical juxtaposition or arrangement on the screen, connection with
lines or arrows, and temporal juxtaposition including animation are
common techniques for showing adjectival characteristics nonverbally.

>verbs
>  e.g., describe relationships between agent and patient. Without
>  using some sort of animation it is very hard to convey the action
>  associated with a pictorial representation. Agent and/or patient which
>  typically have to be part of the pictorial representation can be
>  missinterpreted as the noun(s) to be represented, e.g., how would you
>  represent the action of sawing? Drawing a saw would most likely imply
>  the noun saw.

Note that your example "saw" is a good case of the ambiguity of text
in isolation of context alone.  Does the word

		"saw"

all by itself mean: 1) the noun for something that cuts wood?  2)
a command for the reader to begin cutting wood? or 3) the past tense
of to see?

Context is critical.  In logic class the statement "Two is blue" is
a category mistake, in poetry class it is a metaphor, and on the
billiards table it is a true statement!  Context is just as critical
for graphic or pictorial images, and the fact that images are ambiguous
in isolation does not make them useless anymore than the fact that
words in isolation are ambigous makes them useless.  The best solution
in both cases is to get them out of isolation and back into a clarifying
context as quickly as possible!

When the need for action is obvious by context, then the noun form of a tool
for doing that action is often very clear to people.  When faced with a
scissors symbol, and a paste pot symbol in a situation where cutting and
pasting are obviously desirable alternatives, people will quite naturally
select the scissors to indicate their desire "to cut" something, or 
the paste pot "to paste" something.  Dragging (a limited form of animation)
can also indicate a verb form. People understand the notion of
pulling a representation of a page to a trash can meaning to delete (throw
away) the underlying object, or pulling it to the printer icon meaning to
print it, or pulling it to the mail box meaning to mail it.  The question
of whether such dragging is superior to doing textual operations is a
complex one that also has to do with competancy with keyboards vs. pointing
devices.  But indicating verbs is quite possible in pictorial representations
when chosen well in a limited context or domain of discourse.

> I'd like to get your opionions. Does it make any sense at all? 

> Maybe part of the problem with picture-oriented representations is
> that, in contrast to English, picture-oriented representations do not
> have any defined syntax. You can of course make up a syntax using
> color, spatial features, explicit relationships (e.g., arcs), etc. 

In fact, though, most textual computer interfaces don't rely on english
syntax, but rather, like pictorial representations, get most of their
meaning from their context, or domain of discourse.  This area, which
is the focus of Wittgenstein's later work, as well as the work of Austin
and Searle, is often called Pragmatics (i.e. Language = Syntactics + Semantics
+ Pragmatics).  Pragmatics is so powerful, that in common working discourse
syntax is often abandoned, but the users fully understand what is meant.
Wittgenstein gives the example of the bricklayer who says only "brick"
to his assistant, but the assistant knows that this is a request that
a brick be retrieved and placed where the mason is working. No syntax
was involved, but no ambiguity was felt by the individuals either.
I think this is one of the reasons that english syntax is not widely used
in computer interfaces; it is just as possible to understand things by a
keyword alone, given the work context.  In fact, when syntax is introduced
into textual computer interfaces, it is likely to be a made up syntax
(e.g. "(verb/command)  -(adjectives/modifier-flags) (nouns/files))
and not traditional english syntax at all!

> Have there been attempts to capture a "natural" syntax for images used by 
> people.

Yes, it is formally taught in graphic design classes and informally learned
by most people. See below for an example.

> What does it mean on an abstract level when somebody draws a
> bunch of circles, boxes and lines? 

There are also many graphical representation "conventions" that play the
part of a pre-known syntax for their interpreters. Many of us may have
gone through school without learning these conventions formally, the way
we may have learned our written language grammar.  But we know them just
the same, just as illiterate speakers of the language often speak it
quite well with no syntax errors despite their lack of formal eduation.
For just one example, consider the principles of composition, as applied
to advertising. Consider 3 soaps. The creators of "Brand I" want it to
have cerebral/traditional appeal.  "Brand C" is designed to give the
feeling of a sensual experience.  The object for "Brand Z" is to come across
and dramatic/modern/dynamic.

If you ask a hundred people to design packaging for these soaps based on
these descriptions (and I have done this!) you will find that almost 
everyone will come up with designs that follow these rules, even if they have
had no art training:

Brand I will have vertical and/or horizontal lines on it and all text
on the package will be written horizonally.  Text fonts will be roman,
not italic (i.e. vertical ascenders and descenders, not slanted).
The package will probably be a rectangular prism.

Brand C will feature curving lines, swirls and/or circles.  Type font
selection is very likely to be a script or cursive style (i.e, curved
ascenders and descenders).  The package is likely to be an oval or
circle.

Brand Z will have lots of diagonal (nonhorizontal, nonvertical) lines 
and shapes (e.g. starburst polygons) Text is written in italics 
(slanting ascenders and descenders) and text is often written on an
angle with respect to natural base of the packaging.  Angular packaging,
such as triangles, trapezoids, or rhomboids are often selected.

Why do people come up with these designs over and over again?  The
principles of composition in graphic art dictate exactly these choices.
Without going into whether graphic principles are any more innate than
textual or verbal syntax, let us observe that in both cases people are
free to break the rules, but almost always choose not to. They may 
learn these rules formally in a classroom or may pick them up informally
through observation.  But they do learn them, internalize them, and use
them, and by doing so increase their ability to communicate in the
chosen medium.

> What is the minimal set of graphical primitives?

It is hard to say what this means.  Minimal to what context?  What
can be said is that the graphical conventions understood by indivduals
are very rich indeed, perhaps as rich as syntactic conventions used
in language. There aren't just a few things to learn if you want
to learn them formally, just as there is no short catalog of things
to learn to formally understand grammar.  We take years to learn these
things informally (especially as children), and it takes a long time
to learn them formally too.  But if you want to learn more about these,
I suggest a course of study in of "speech act" analysis and pragmatics,
plus a course of study in principles of graphic design. 

Scott McGregor
Atherton Technology