| |
Introduction
3D computer graphics routines can be characterised on a spectrum from physics
to faking it. The more physics involved the more realistic, though slow,
while the more faking the faster. We also see that the more faking required,
the more the sensibilities of the artist are needed. Real-time virtual reality,
with its high demands on processing power, involves more faking it than
in other 3D computer graphics applications, but the fact is that the artist's
eye is needed across the spectrum, and this is only for appearances:
the artist's wider cultural antennae are also vital for content.
This paper examines the range of the artistic skills necessary in VR, and
at the same time looks at some of the philosophical implications of the
technology.
What is VR?
There are different ways of defining virtual reality. For the purposes of
this paper I will consider it simply as an interactive technology that recreates
some aspect of human experience. Not all would agree that interactivity
is essential to virtual reality systems, but without it we leave out a prime
element in human experience: free will. Free will, from a philosophical
standpoint, is a difficult thing to pin down, but a virtual reality system
that leaves out any possibility for the user to make an input turns it into
a version of cinema. However, as this paper focuses on the visual clues
that make us accept a computer-generated version of reality as worth engaging
with, the interactivity question is not so central. How systems can model
aspects of human experience and what 'real' reality is becomes relevant
however.
Reality and the Role of the Brain
One of the reasons why the subject of virtual reality is so interesting
is that it can challenge our thinking about ordinary reality. In fact, the
question of what 'reality' is and to what extent the mind and brain construct
it for us is one that has engaged thinkers all through history, and recent
research has made the answers no easier to come by. Some groups, often with
a religious or mystical background, suggest that the world we encounter
through our sense perceptions is less real than their essences (Plato's
'forms' [1]), while others propose a spirit world which is more
'real' to them than the material (Rudolf Steiner for example [2]). The Scottish philosopher David Hume was much taken
up with the problem of how we seem to construct an orderly world from disparate
and jumbled sense impressions; he found that the more one concentrated on
them the more the world seemed to fall apart in fact. In the end he despaired
of a solution and suggested (perhaps ironically) that 'the only remedy was
inattention'. The work of contemporary philosopher Mary Warnock has explored
this theme further, with reference also to Kant [3].
Most empiricists however are quite certain that there is a real object world
'out there'. Empiricists, unlike Plato and his followers, believe that knowledge
of the world can only come from careful observation of it as presented through
the five senses, and through instruments, such as the telescope or microscope,
which extend them. Interestingly not all scientists hold this view, Erwin
Schroedinger for example saying that 'the world is given to me only once,
not one existing and one perceived' [4].
To firmly hold either one or the other view seems unreasonable to me however;
it is more interesting to suspend final judgement for the moment and look
at what open-minded research is telling us.
One interesting area of research that poses problems for the question of
a 'reality-out-there' is dream studies. Brain researchers have found that
the chemical and electrical activities in the human brain are different
and distinct in the three main states of consciousness: waking, REM sleep,
and dreamless sleep. (REM sleep is named after the Rapid Eye Movements which
takes place in dreaming). Research has shown that the visual cortex of the
brain (the part involved in vision) is stimulated in dreaming as if impulses
were coming from the eyes: the brain is 'constructing' a visual reality
along with the tactile and emotional drama of the dream. Even more surprising
is the fact that the dreamer responds to the dream environment by generating
motor signals that would normally cause walking, running, fighting and so
on. A chemical that floods the brain during sleeping normally inhibits the
transmission of these motor impulses to the rest of the body, though with
one exception: the eyes. Technically the body is said to be in a state of
suppressed muscle tone. Sometimes, if the impulses are too strong (as in
a nightmare perhaps) the dreamer may move or speak, or even wrestle with
phantom opponents. Genuine sleepwalking is an extreme case of this. Interestingly,
the same is true of many higher mammals, cats for instance. One can often
observe a cat's tail twitch as it sleeps, and science has shown that it
must be dreaming because all the chemical and electrical states in the cat's
brain correspond closely to those in humans.
A form of dreaming where the participant can control their dream environment
is called lucid dreaming, and has been the subject of recent research. It
has been discovered that most people make good subjects for lucid dreaming
and can be trained to gain degrees of control over their dreams. They have
even been able to receive messages from lucid dreamers via eye movements,
as these do not share the suppressed muscle tone of the rest of the body.
The messages have been used to confirm events in dreams, and these moments
can be correlated with measured brain activity and the dreamer's subsequent
account of events. A group in the US have been working with artists and
lucid dreaming with interesting results on their paintings.[5].
The conclusion of some who are involved in this research is that 'waking
is like dreaming with the imagination suppressed, while dreaming is like
waking with muscle tone suppressed.' The relevance of this to virtual reality
is partly philosophical, and partly practical, as there is no doubt that
the impetus of VR research will be to move inexorably from modern total-immersion
systems involving computer graphics visual systems to what is loosely called
'brain-stem' virtual reality where the user is 'plugged in' directly to
the technology. Such possibilities remain at present in the domain of science
fiction, such as Neuromancer, [6]
or in the wildly speculative physics of Frank Tipler in The Physics of
Immortality. [7]
A Taxonomy of VR
Before contemplating the future possibilities of brain-stem VR, let us draw
up a rough and ready taxonomy of virtual reality systems, so that we can
better see where computer imaging fits in at present.
· text-based, such as Muds and
Moos over the Internet: keyboard and on-screen text - input via keyboard,
feedback via text on screen
· graphics-based, such as games: input via mouse or joystick,
feedback via graphics on screen, sound cues
· professional simulators, such as aircraft, ship, tank and
road vehicle systems: input via mock-driving controls such as steering
wheel, rudder, flight console, feedback via graphics on wide screen, with
the addition of sound and possibly kinetics such as cockpit judder, lean
etc.
· low-level immersion, such as homebrew VR kits, arcade VR
games consoles: input via a range of devices including the dataglove, bodysuit,
spatial positioning sensors, feedback from head-up display, audio cues.
· high-level or 'total' immersion: same as the previous
category but using state of the art devices and supercomputer processing
power.
· brain-stem VR: a direct connection between a computer and
our brain via synaptic connections (an idea popularised by William Gibson
in Neuromancer).
· up-loading the mind: leaving the body behind altogether
and digitally encoding (uploading) our personalities into the computer
(a highly speculative idea proposed by the physicist Frank Tipler).
Constructing a World
Whatever level of virtual reality we deal with the creators of the system
need to construct an artificial or imaginary world. With the exception of
text-based systems this world is presented via computer-generated imagery
of varying sophistication. At its simplest a world is 'modelled', that is
a database of elements and their relationships and behaviours are constructed,
and then 'rendered', that is made visible through the use of computer imaging.
As our normal world is three-dimensional most VR worlds are modelled in
3D. In text-based systems the model is described textually, and the 'rendering'
of it takes place in the user's imagination, ranging from a typical dungeon
in a castle to the hot-tub in LambdaMoo.
Total immersion: Brains in Vats?
Before looking more closely at computer graphics issues in VR, I want to
make another philosophical detour. Daniel Denett tells us in his Consciousness
Explained that philosophers in the last century were interested in the
'brain in a vat' problem [8]. The idea here
is to explore the philosophical implications of a scenario where a mad (or
evil) scientist has scooped out our brain, placed it in a vat, and connected
us up to electrical apparatus which receives and returns synaptic impulses
so as to fool us that we were living an ordinary life. Denett briefly mentions
VR in this context, but we are in a position here to pursue this a little
further. The main technical objection to the scenario is in what is known
as 'combinatorial explosion'. Because the brain's owner may make over a
period of time so many unpredictable decisions (because it must still be
allowed to exercise 'free will' for the illusion to work), it would become
impossible to calculate all the possible activities of the person, and so
be able to provide electrical impulses that covered all the eventualities.
For example, one might walk the same route to work every day, which means
that the mad scientist only needs to construct part of a city (in computer
terms: model and render). But what if our person in the vat decides, on
impulse, to turn down a side-street one day? Or, far worse, suddenly decides
to go on a holiday to a foreign city and, again on impulse, turn into an
obscure alleyway? Our scientist would need to reconstruct the entire planet
and all its inhabitants in order to guarantee that the illusion never broke
down.
The scenario was dropped by philosophers because of this basic objection
to it, but with VR we have to consider it again. Even the most basic of
computer games programmers is in the mad scientist's position: how much
of a 'world' is one going to construct for the interactive participant?
We find even at this crude level of VR an answer to combinatorial explosion:
procedural modelling. This is a technique whereby, for example, cities
can be constructed using a rule-based system: by abstracting out the main
principles whereby cities grow and their elements are constructed and appear
to us, we can generate cities (or whatever) 'on the fly'. In addition we
need (in visual terms) to be able to render any view of these constructed
environments on the fly, but this is a separate problem requiring only that
there is adequate processing power. (For the brain in the vat an inadequate
external processing system might result in 'picture loss' if turning one's
virtual head rapidly.) Recent advances in computer graphics modelling and
rendering techniques mean that the philosophers will need to revisit the
brain in the vat problem.
In The Physics of Immortality the physicist Frank Tipler makes the
assumption that computers in the far future will be powerful enough to serve
a vast number of 'brains in vats', in fact every one who ever lived (or
might have lived). Tipler does not in fact suggest preserving peoples' brains:
his theory proposes that our personalities (or souls or whatever) are uploaded
into the computer itself, and that the machine will be powerful enough to
simulate life for everyone who every lived a digital immortality. This is
the most extreme form of VR yet proposed, and, although Tipler argues for
it in great detail (with considerable support from equations and physics)
most scientists are highly sceptical of his ideas. Probably the biggest
stumbling block is the question of the computability of mind, that is whether
the mind/brain functioning can be simulated, even in principle, by a computer
(or technically a Turing machine). This is a very interesting debate which
I discuss further in Artificial Consciousness Artificial Art [9]. The next section will throw up further doubts about
Tipler's proposal.
Physics vs. Faking it
Apart from text-based and 2D VR systems we are mainly dealing with 3D environments,
all of whose features can be expressed, understood, and coded via the laws
of physics. In practice however this is rarely done because of the amount
of computing power required, and the whole evolution of computer graphics
has in fact been a history of 'faking it.'
Faking starts at the modelling stage, where an internal representation or
3D database is built of the artificial world. Various techniques exist,
but the commonest method is to represent objects using polygons. A great
deal of skill is involved in modelling objects to an appropriate level of
accuracy using the minimum number of polygons: in the games industry for
example typical 3D scenes may have a maximum polygon count in the low hundreds
to a few thousands. An artists' job working on such a game may involve the
modelling of a realistic looking car with just 30 polygons. This is a skill
that artists develop from their traditional training in the arts, and involves
an intuitive understanding of how visual clues work, and how to abstract
the key visual characteristics from complex natural phenomena. What if we
were to turn to physics to help us in the modelling stage? If we really
wanted to build a virtual reality that imitated beyond any doubt the real
world then we probably would have to use physics right down to the molecular
level. The appearance and behaviour of objects depends on this: the exact
distribution of mass in a car for example determines the way it corners;
the exact distribution of pigments and carriers in the car's paintwork determines
its finish (and whether the car looks new and expensive or old and cheap).
This, for me, is another of the objections I have to Tipler's virtual universe:
a convincing reality requires modelling at the molecular (or even atomic)
level, and you would need a processor for every molecule or atom. 'Molecular
computing' as it is called does look in fact like a possibility, though
a remote one, but even if we could build an information processor at the
molecular size, we would land up needing one per molecule in our model:
in other words you would need a whole universe to model a universe!
A more realistic use of physics in modelling has been mentioned earlier:
procedural modelling. This requires again that we abstract out some high-level
behaviour from a natural phenomenon, for instance rules about the way plants
grow, clouds form, and so on. One common technique in this area is called
particle systems, where simple physical laws are used to control large numbers
of particles (represented usually only by a location and a small amount
of other data such as mass, direction, speed, and colour). This is a solution
somewhere between (in terms of the processing power needed) molecular representation
and polygon modelling.
However, modelling plays only a small part of the overall process of producing
'cheap' but convincing VR. Turning now to the rendering stage of computer
visualisation, we find that even more faking it goes on, though the physics
in fact is well-understood. There are two main areas for faking: firstly
in a range of texturing techniques that are used to make up deficiencies
in the modelling, and secondly in the rendering itself. The earliest
faking of a 'realistic' scene derived from the polygon representation of
objects: simply shade each polygon according to how much light falls on
it. This gives a crude faceted realism to a scene that a wire-frame rendering
can never have. Progress in rendering involved better faking: if shading
levels are averaged across the polygons (Gouraud shading) there is a better
illusion of smoothness, if polygon normals are interpolated (Phong shading)
specular highlights can be introduced to make things look shiny. These are
called shading models, and they work in conjunction with lighting
models. The simplest lighting model is a classical bit of faking-it given
the grand title of Lambert's lighting law. What was needed was a simple
mathematical relationship that took the angle between the light direction
falling at a point on the surface of an object and the normal to the plane
and calculated a light intensity. When this angle is zero the intensity
should be a maximum, and when the angle is ninety degrees it should be zero;
what, asked Mr Lambert, is a simple mathematical function that does this?
Answer: the cosine. The result works, that is it gives reasonably
realistic 3D shaded objects, but the maths used bears no relation to the
physics of light involved.
A more sophisticated strategy called ray-tracing allows more of the laws
of physics to be used, specifically optics. This is a shading technique
that can be used with different lighting models, as with the previous shading
techniques, but much more realistic results can be obtained. Ray-tracing
follows imaginary rays of light through a scene, but in reverse, starting
with the eye-point, moving through a pixel of the final image into the 3D
space where it may encounter objects, bounce off them or pass through them
in differing proportions, and finally encounter a light source. A development
of this called radiosity allows for subtle lighting effects to be computed
based on the equilibrium of light energy in confined spaces. Ray-tracing
takes much more processing power than the rendering techniques described
above, and radiosity calculations that much more again. The kind of imagery
obtainable is described as photo-realistic, and, as this name suggests,
the artistic sensibilities required to work successfully with it are closer
to that of the photographer than painter.
Because ray-tracing and radiosity techniques are so processor-intensive
they tend to be used for still imaging and animations with high budgets.
VR, with its requirements for real-time rendering (absolute minimum 6 frames
a second, but preferably 25 up to 60) rules out both of these techniques
for the time being. In fact, even more conventional rendering techniques
are often too slow, which is where texturing comes in. In any VR system
there will be a polygon 'ceiling', that is a maximum number of polygons
that the system can render at acceptable frame rates. To improve on the
crudity that this would otherwise impose on the look of the virtual world,
digital images known as maps are added to the polygons. The most important
are texture maps (images such as wood or brick), transparency maps (providing
holes in objects) and reflection maps (making objects look shiny). In each
case the application of the map reduces the polygon count, often quite dramatically.
In a typical top-end flight simulator one might need trees at the end of
a runway for realism, but modelling a realistic-looking tree with thousands
of polygons is out of the question. What is the minimum number of polygons
that we can use to simulate a tree and get away with it? The surprising
answer (in the specific context of the flight simulator) is one! A single
polygon with the appropriate texture map (showing trunk, branches and leaves)
and an appropriate transparency map (a silhouette of the tree) makes for
an acceptable tree. This works because the angle of approach to the tree
when landing will have only minute variations otherwise it would soon be
apparent that it was a cardboard cut-out. The artistic skills here, as in
a range of VR applications with limited view-points, is more that of the
stage-set designer.
If we can use ray-tracing and radiosity we can obtain very good shiny-looking
objects because of the accurate reflections in them. Interestingly, even
here, a degree of faking is often needed because of the cost of modelling
enough world to make realistic reflections. A spoon, held up to the face
in a busy street, reflects large chunks of city especially if moved a modelling
task that can be avoided if the reflections don't have to be accurate. The
fact is, and artists know this, that the reflection in the spoon is read
by the eye/brain merely as a visual clue: the eye is not focused even on
the reflected image but uses it in determining that it is a shiny metal
(and whether or not it is clean and suitable for eating with, for example).
The geniuses of faking-it quickly spotted that almost any reflected image
would do, and many systems merely provide a typical indoor reflected image
(usually containing a window) and a typical outdoor reflected image (usually
with a sky and ground). You've guessed it: another map is used, the reflection
map. (For a good discussion of these techniques and how 'chrome' has come
to symbolise modern computer graphics, and how the artist's sensibilities
are involved in this see 'The Chrome Age' by Sonya Shannon. [10])
If we consider now what this spectrum of physics-to-faking-it implies we
realise that it is all about trade-offs. Faking it gives us speed; physics
needs a lot of processing power. Hence the demands of VR for real-time graphics
means, for the foreseeable future that more faking than physics is needed.
Does our spectrum imply as processing power increases (it will need to by
orders of magnitude) that the artistic input will be less? I don't believe
so, but the type of artistic skill will change: real-time ray-tracing
and radiosity will require the eye more of the photographer than the painter.
Artists' Visual Clues
We have shown that both the modelling and the
rendering stage of VR graphics require the artists' sensibilities, given
that until Tipler's world becomes true we cannot rely on physics alone to
build our virtual reality systems. Let us explore how pre-digital art has
exploited a similar strategy of visual clues, albeit rendered in paint or
other artistic media. Artists have always abstracted key visual features
of the natural and man-made objects that make up their compositions, though
it is only in this century that the degree of abstraction has made a quantum
leap beyond its previous parameters. If we stick to representational art
however, it is clear that the artist somehow 'models' the objects to be
represented in their minds, and then uses visual clues drawn from a stock
of such clues that comprise a language of the visual.
When an artist wishes to represent a scene, whether drawn from real life
or imaginary, they have to consider how it is lit: for the picture to be
convincing there will generally have to be a global light direction
which determines which side of objects will be brighter than others, and
how shadows fall. Many decisions regarding light have to be taken (perhaps
unconsciously), but they are all readily translatable into 3D computer graphics.
While a painter has a greater freedom and flexibility to use light in a
personal way than with a 3D computer system, he or she still has to be consistent
and follow certain rules otherwise the marks made on the canvas will not
be readable. Visual clues can cancel each other out (though of course illusionists
such as Escher can make use of this.)
Once global decisions are made about lighting, from which many of the necessary
visual clues and their execution follow in paint, or other media, objects
within the scene have to be differentiated using different surface qualities.
Thus apples are shiny (a dab of white is required for the highlight), a
carpet is dull but textured, a person's face rich in subtle colouring, wet
things reflect other objects.
Reflection in water and other reflective surfaces has long been a stock
trick for artists; it is instructive to look at master cartoonists like
Giles or Steve Bell to see how, with a few simple vertical strokes, they
transform a dry city pavement into a wet, sodden, slippery surface. Refraction,
the bending of light through transparent objects, is another easy trick:
just shift around the background a bit, or, if it is a severe case, like
a bottle or glass sphere, just use background elements in a murky swirl.
(Incidentally the long-running Smirnov vodka ads use no refraction for the
scene seen through the bottle probably for clarity , and it does not detract
from the message.)
Artists also use what is known as depth cueing to give the impression of
space: objects in the distance are reduced in clarity because of atmospheric
effects such as haze or fog. The simplest visual clue here is a desaturation
of colour (see Shannon's article again for a discussion of this). One
of the difficult problems in painting is to give the elements of the composition
sufficient separation, because, lacking stereo vision and parallax, the
viewers' normal strategies for disentangling objects are restricted (this
comes back to Hume's perceptive comments about how on earth we extract discrete
objects from an environment served up to us as a 'porridge' of visual sense
impressions). Some painters use thick black lines around objects, while
others use a variety of depth clues, including of course the natural one
of scale.
Finally, objects that move rapidly in a scene can be given the illusion
of motion by blurring them. All these techniques, including this motion
blurring, are translatable into software techniques for VR, and all of them
require the artists' eye to work successfully.
3D Studio
Given that 3D graphics systems whether for VR or anything else, have to
provide the user with a balance on the physics-to-faking-it spectrum combined
with a reasonable estimate of the artistic skill the user will have, how
do well-known systems compare? How do their designers pitch the functionality
of their systems? They also need to take into account how much physics a
user knows.
Materials: just provide a name, e.g. marble, brushed aluminium, and leave
it to the system? Or provide the tools to create these? The exact balance
is difficult, and is matched with the difficulty throughout the modelling
and rendering options that can be provided. At London Guildhall University
we use a 3D modelling and animation package called 3D Studio [11], which shows an interesting range of choices in the
modelling and rendering functions offered. The software is designed to run
on a fast PC, such as a Pentium, but the performance of the average machine
is way below that provided by workstations such as the Silicon Graphics
machines. Hence ray-tracing is only provided for accurate shadows, and the
rest is rendered by a scan-line algorithm.
Virtual Worlds and the Artist
So far we have discussed mainly the artisanship of the artist in
constructing acceptable virtual worlds, acceptable from a visual standpoint
that is. The real challenge of VR is to harness the available technology
at any stage in its development to provide an experience that really engages.
It is obvious from people like Tipler that there is a desire to play God,
to 'improve' on real reality. This impulse may be the same as the Utopian
one that appears through history from Plato to Aldous Huxley, but it may
well be that the more utopian the virtual world the more boring it will
be. In other words the ground-rules for life in the real world seem to provide
for a pretty engaging drama, and it is more likely that good VR scenarios
will not abandon the age-old bugbears of the utopian dreamers, such as evil,
profit and loss, and death. The artists' role in VR is much greater than
the merely visual, encompassing all the creative arts, many of which are
akin to those found in theatre or film production. The visual artist has,
however, a key role in the look and feel it of it.
Conclusions
We have seen that virtual reality systems comprise a spectrum of applications,
most of which rely heavily on the visual. Because of the processing power
necessary to provide sophisticated computer-generated imagery, much of it
is achieved through cheaper forms that can be called faking. Computer imaging
techniques are themselves drawn from a spectrum that can be described as
'from physics to faking it', and the right choice is critical for a workable
VR scenario. The skills of the visual artist are essential in working across
this spectrum; at one end the visual clues used by painters and illustrators
have to be understood, while at the other end the sensibilities of the photographer
are essential.
The philosophical implications of VR are another challenging aspect to the
artist, in that the designer of a virtual world has to have a profound grasp
of the drama of the real world. I leave you with a philosophical thought:
would you really find a virtual world designed by utopian thinkers like
Plato or Tipler more interesting than one designed by a dramatist like Shakespeare
or Hitchcock?
References
[1] See
for example Plato, The Last Days of Socrates, Trans.: Hugh Tredennick,
Harmondsworth: Penguin, 1969, around p. 110 or Plato, The Republic,
Trans.: Desmond Lee, London: Penguin Books, 1987, around p. 252
[2] See
for example Steiner, Rudolf. The Evolution of Consciousness, Rudolf
Steiner Press, 1966
[3] Warnock,
Mary, Imagination, London: Faber, 1980 (This deals extensively with
Hume as well)
[4] See
Wilber, Ken, Quantum Questions - Mystical Writings of the World's Great
Physicists, Boston and London: Shambhala, 1985, p. 79, or Schroedinger's
own writings.
[5] Bogzaran,
F. 'Images of the Lucid Mind' in Consciousness Research Abstracts,
proceedings of the "Tucson II" conference (Journal of Consciousness
Studies) University of Arizona, 1996
[6] Gibson,
William, Neuromancer, London: Harper Collins, 1993
[7] Tipler,
Frank J. The Physics of Immortality - Modern Cosmology, God and the Resurrection
of the Dead, London: Macmillan, 1994
[8] Dennet,
Daniel C., Consciousness Explained, Allen Lane, The Penguin Press,
1991, p. 3-4
[9] King,
Mike, 'Artificial Consciousness - Artificial Art', chapter for a book Art
and Computers, (Ed. Stuart Mealing), to be published by Intellect. See
also a shorter version in Proceedings, International Symposium on Electronic
Art 1995, Montreal, ISEA'95 Montreal, 1995, p. 137 - 140
[10] Shannon,
Sonya, 'The Chrome Age: Dawn of Virtual Reality' in Leonardo (3rd
Annual New York Digital Salon), Vol 28, N. 5, pp. 369 - 380
[11] Produced
by Autodesk. The most recent version, 3DS Max, runs under Windows NT and
is a competitor to high-end systems like Softimage.
|
