Li S.Z., Jain A.K. (eds.) Encyclopedia of Biometrics

Подождите немного. Документ загружается.

of incidence . Generally, in Computer Graphics, the re-

ﬂectance used is a combination of diffuse and specular

reﬂectance. The most well-known model is the Phong

model:

phong

i;model

¼ðc h



;



liþK h



v; ri



Þl  S



where c is the albedo at point x,



r is the reﬂection

direction (depending on the normal and the lighting

direction, as shown on the sketch), K is the fraction

of energy specularly reﬂected, and n is an index that

controls the ‘‘tightness’’ of the specular highlight (note

that there is one such equation for each color channel).

In this equation, the ﬁrst summand inside the brackets

is the diffuse (i.e. Lambertian) part and the second

Face Sample Synthesis. Figure 1 Renderings using different illumination models. The 3D model includes 24,367 vertices

and 48,660 triangles.

380

Face Sample Synthesis

one is the specular part. Modeling the specular high-

light is important for human skin, as it often has a

thin layer of oil or sweat above the pigmented cells.

Figure 1 a–d show examples of face rendering w ith a

Phong model.

The Pong model assumes smooth surfaces, but, in

reality, surfaces are imperfect and exhibit microgeome-

try. There exist more complex BRDF models that rep-

resent the surface as composed of micro-facets that can

shadow and mask each other. Another effect is the

off-specular highlight: When the angle of incidence is

grazing (near 90

∘

), some materials (such as human

skin) reﬂect much more light than is absorbed, causing

the color of the point to approach that of the light. This

is accounted for by the Fresnel term that is the ratio

between reﬂected and absorbed light. Some of the more

complex BRDF model that accounts for these two

effects are Blinn [10], Cook-Torrance [11], Torrance-

Sparrow [12], and more recently Lafortune [13].

Note that this is not the end of the story, yet. The

BRDF assumes that the outgoing light at one point

results only from the incoming light at the same point.

This is in fact an approximation as it neglects the

scattering of light within the material. This phenome-

non is modeled by the bidirectional surface scattering

reﬂectance distribution function (BSSRDF) [14]of

which the BRDF is a special case. Human skin does

show some important subsurface scattering effects and

to reach photo-realism these effects should not be

neglected. This is for instance apparent when the ear

is illuminated from the back. It then looks translucent

which results from the subsurface scattering.

Normals

H uman skin is not a smooth surface. P or es and

wrinkles induce very small scale variations of the surface.

Representing these variations with 3D vertices would

require a very ﬁne sampling of the head resulting in an

overly large number of vertices making the rotation or

visibility test computationally inefﬁcient. The concept of

‘‘normal mapping’’ was developed precisely for this rea-

son. Instead of computing a normal from the shape (by

interpolating the normals of the triangles corners in

which a pixel is projected from), the normals are com-

puted from a dense normal map (for which interpolation

is carried similarly to the texture map). A normal map is

generally acquired by photometric stereo [15]: Several

photographs of the face of a subject are taken with

different light directions. The subject and the camera

must be perfectly still during this acquisition process (a

pixel must be the projection of exactly the same point

on the subject face for all photographs). Each photo-

graph yields one measurement of the BRDF of a sur-

face point. Using several measurements a BRDF model

can be ﬁtted, thereby recovering the normal of the

point. Often, for its simplicity, a Lambertian mode l is

used, in which case, the operator must choose the light

direction such as to minimize specular reﬂections.

Figure 1 shows different t ypes of rendering from

the most simple (left) to more complex and realistic

(right).

Identity Modeling

So far, an overview of the face image synthesis process

from a 3D model of the face of an individual is pre-

sented. This 3D model can be acquired by a 3D scanner

or can be manually crafted with a modeling software.

These processes can be tedious and expensive. There-

fore, it is desirable to be able to generate the 3D shape

and texture (i.e., albedo) from any individual. This can

be done by deﬁning a vector space of shapes and of

textures and probability distributions in these spaces.

This is accomplished by learning typical face variations

from an example set of 3D faces. The vector space is

deﬁned by densely registering the examples with a

reference face, thereby deﬁn ing a label for each vertex.

Once the vector spaces are deﬁned, linear combination

of the example shapes and textures are made to gene-

rate the shape and texture of new (i.e., out of the

example set) individuals. The coefﬁcients of these lin-

ear combinations are the parameters of the identity

model. One individual is coded by a speciﬁc value for

each parameter. However, some variations are more

ty pical than others and probability distributions in

the vector space must be used in order to ensure the

plausibility of a novel individual. If the probability

distributions of the human faces in the vector spaces

are assumed Gaussian, then the most efﬁcient coding is

yielded by a Principal Component Analysis [16] of the

examples. These principles are used by the 3D Morph-

able Model [3], the state of the art identity generic

human face model.

Denoting by X the 3  N matrix with the 3D

position of N vertices (hence the position of a vertex

Face Sample Synthesis

381

x in the ‘‘Finite Projective Camera Model’’ section is

a column of the matrix X) and by C the 3  N matrix

with the RGB albedo of N vertices, a novel 3D face is

yielded by the following equations:

X ¼

; C ¼

; ð8Þ

where X

and C

are the M shape and texture principal

components and a

and b

the shape and texture

parameters.

Conclusion

In conclusion, the set of parameters y of an analysis by

synthesis method (1) is composed of nin e parameters

for the projection. For illumination, using a Phong

reﬂectance model with one light source and with nor-

mals computed from the shape, seven parameters must

be estimated: three parameters for the intensity of the

colored light, two parameters for its direction along

with the specular coefﬁcient K and the Phong expo-

nent n. Additionally, 2M parameters must be recovered

for the 3D shape and the texture.

Using a normal map model (generic for all indivi-

duals) and an environment map for analysis by syn-

thesis has never been attempted. Indeed, it would

result in a tremendously complicated problem for the

following reasons: It is unclear how to model normal

maps that would generalize for any individual. A simple

linear combination as is used for the shape and texture

cannot be used for normals because a normal vector is a

unit length vector and the sum or the mean of two unit

length vectors does not result in a unit length vector.

Moreover, deﬁning correspondences for pores and wrin-

kles (which would be required to make a vector space

and avoid blur results) is for the moment unsolved. As

for the light sources, estimating the direction and inten-

sity of a single light source from a single facial image is

already an ill-posed problem (there is not enough infor-

mation in one image to completely constraint the solu-

tion), let alone with a large number of light sources as is

the case when using an environment map.

Summary

The motivation of face sample synthesis is not only

to generate face images from a small number of

parameters but also to analyze them using an analysis

by synthesis approach. In this article, the two main

sources of face image variations (pose and illumina-

tion) are accounted for by a ﬁnite projective camera

model. Illumination modeling is more complicated

and requires the operator to choose the type of illumi-

nation sources, the type of reﬂectance function, and

the manner to generate normals (either from the shape

or acquired by a photometric stereo method). Finally,

identity variations can be obtained by a linear combi-

nation of examples.

Related Entries

▶ Deformable Models

▶ Face Pose Analysis

References

1. Kanade, T.: Computer Recognition of Human Faces. Birkha

user

Verlag, Stuttgart, Germany (1973)

2. Cootes, T., Edwards, G., Taylor, C.: Active appearance model.

Fifth European Conference on Computer Vision. Freiburg,

Germany (1998)

3. Blanz, V., Vetter, T.: A morphable model for the synthesis of

3D-faces. SIGGRAPH 99. Los Angeles, California, USA (1999)

4. Romdhani, S., Vetter, T.: Estimating 3D shape and texture using

pixel intensity, edges, specular highlights, texture constraints

and a prior. CVPR. San Diego, CA, USA (2005)

5. Parke, F.I., Waters, K.: Computer Facial Animation. AKPeters

Wellesley, Massachusetts, USA (1996)

6. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer

Vision. Cambridge University Press (2000)

7. Woo, A., Poulin, P., Fournier, A.: A survey of shadow algorithms.

IEEE Comput. Graph. Appl. 10(6), 13–32 (1990)

8. Greene, N.: Environment mapping and other applications of

world projections. IEEE Comput. Graph. Appl. 6(11) (1986)

9. Debevec, P., Malik, J.: Recovering high dynamic range radiance

maps from photographs. Siggraph. Los Angeles, California, USA

(1997)

10. Blinn, J.F.: Models of light reﬂection for computer synthesized

pictures. SIGGRAPH ’77: Proceedings of the Fourth Annual

Conference on Computer Graphics and Interactive Techniques,

pp. 192–198. New York, NY, USA, ACM (1977)

11. Cook, R.L., Torrance, K.E.: A reﬂectance model for computer

graphics. ACM Trans. Graph. 1(1), 7–24 (1982)

12. Torrance, K.E., Sparrow, E.M.: Theory for off-specular reﬂection

from roughened surfaces, pp. 32–41. Radiometry (1992)

13. Lafortune, E.P.F., Foo, S.C., Torrance, K.E., Greenberg, D.P.:

Non-linear approximation of reﬂectance functions. SIGGRAPH

’97: Proceedings of the 24th Annual Conference on Computer

382

Face Sample Synthesis

Graphics and Interactive Techniques, pp. 117–126. New York,

NY, USA, ACM /Addison-Wesley (1997)

14. Jensen, H.W., Marschner, S.R., Levoy, M., Hanrahan, P.: A prac-

tical model for subsurface light transport. SIGGRAPH, ACM/

Addison-Wesley (2001)

15. Barsky, S., Petrou, M.: Colour photometric stereo: simultaneous

reconstruction of local gradient and colour of rough textured

surfaces. ICCV p. 600 (2001)

16. Jolliffe, I.T.: Principal Component Analysis. Springer, Berlin

(2002)

Face Sample Utility

▶ Face Sample Quality

Face Sketching

A face sketching is a parsimonious yet expressive rep-

resentation of face. It depicts concise sketches of face

that captures the most essential perceptual informa-

tion with a number of strokes.

▶ And-Or Graph Model for Faces

Face Tracking

AMIT K. ROY-CHOWDHURY,YILEI XU

Department of Electrical Engine ering, University of

California, Riverside, CA, USA

Synonym

Facial motion estimation

Definition

In many face r ecognition systems, the input is a video

sequence consisting of one or more faces. It is necessary

to track each face over this video sequence so as to

extract the information that will be processed by the

recognition system. Tracking is also necessary for 3D

model-based rec ognition systems, where the 3D model

is estimated from the input video. Face tracking can be

divided along different lines depending upon the meth-

od used, e.g., head tracking, feature tracking, image-

based tracking, model-based tracking. The output of

the face tracker can be the 2D position of the face in

each image of the video (2D tracking), the 3D pose of the

face (3D tracking), or the location of features on the face.

Some trackers are also able to output other parameters

related to lighting or expression. The major challenges

encountered by face tracking systems ar e robustness to

pose changes, lighting variations, and facial deforma-

tions due to changes of expression, occlusions of the

face to be tracked and clutter in the scene that makes

it difﬁcult to distinguish the face from the other objects.

Introduction

Tracking, which is essentially ▶ motion estimation,is

an integral part of most face processing systems. If the

input to a face recognition system is a video sequence, as

obtained from a surveillance camera, tracking is needed

to obtain correspondence between the observed faces

in the different frames and to align the faces. It is so

integral to video-based face recognition systems that

some existing methods integrate tracking and recogni-

tion [1]. It is also a necessary step for building 3D face

models. In fact, tracking and 3D modeling are often

treated as two parts of one single problem [2–4].

There are different ways to classify face tracking

algorithms [5]. One such classiﬁcation is based on

whether the entire face is tracked as a single entity

(sometimes referred to as head tracking) or whether

individual facial features are tracked. Sometimes a com-

bination of both is used. Another method of classiﬁca-

tion is based on whether the tracking is in the 2D image

space or in 3D pose space. For the former, the output

(overall head location or facial feature location) is a

region in the 2D image and does not contain informa-

tion about the change in the 3D orientation of the head.

Such methods are usually not very robust to changes

of pose, but are easier to handle computationally. Alter-

natively, 3D tracking methods, which work by ﬁtting a

3D model to each image of the video, can provide

estimates of the 3D pose of the face. However, they are

Face Tracking

383

usually more computationally intensive. Besides, many

advanced face tracking methods are able to handle chal-

lenging situations like facial

▶ deformations, changes of

lighting, and partial occlusions.

A broad overview of the basic mathematical frame-

work of face tracking methods will be given ﬁrst, followed

by a review of the current state-of-the-art and technical

challenges. Next, a few application scenarios will be con-

sidered, like surveillance, face recognition, and face mod-

eling, including discussion of the importance of face

tracking in each of them. Then some examples of face

tracking in challenging situations will be shown, before

conclusion.

Basic Mathematical Framework

An overview of the basic mathematical framework that

explains the process in which most trackers work is

provided here. Let p 2 ℜ

denote a parameter vector,

which is the des ired output of the tracker. It could a 2D

location of the face in the image, the 3D pose of the

face, or a more complex set of quantities that also

include lighting and deformation parameters. Deﬁne

a synthesis function f : ℜ

 ℜ

!ℜ

that can take

an image pixel v 2 ℜ

at time (t1) and transform it

to f (v, p) at time t. For a 2D tracker, this function f

could be a transformation between two images at two

consecutive time instants. For a 3D model-based trac-

ker, this can be considered as a rendering function of

the object at pose p in the camera frame to the pixel

coordinates v in the image plane. Given an input image

I(v), align the synthesized image with it so as to obtain

p ¼ arg min

gfðv; pÞIðvÞðÞ; ð1Þ

where

p denotes the estimated parameter vector for

this input image I(v).

The essence of this approach is the well-known

Lucas–Kanade tracking, an efﬁcient and accurate im-

plementation of which has been proposed using the

inverse compositional approach [6]. Depending on

the choice of v and p, the method is applicable to the

overall face image, a collection of discrete features, or a

3D face model. The

▶ cost function g is often imple -

mented as an L

norm, i.e., the sum of the squares of

the errors over the entire region of interest. However,

other distance metrics may be used. Thus a face tracker

is often implemented as a least-squares

▶ optimization

problem.

Let us consider the problem of estimating the

change, △ p

≜ m

, in the parameter vector between

two consecutive frames, I

(v) and I

t1

(v)as

¼ arg min

f ðv;

t1

þ mÞI

ðvÞðÞ

; ð2Þ

and

t1

: ð3Þ

The optimization of the above equation can be

achieved by assuming a current estimate of m as

known and iteratively solve for increments △ m such

that

f ðv;

t1

þ m þ4mÞI

ðvÞÞ



ð4Þ

is minimized.

Performance Analysis

While the basic idea of the face tracking algorithms is

simple, the challenge comes in being able to perform

the optimization efﬁciently and accurately. The func-

tion, f, will be nonlinear, in general. This is because f

will include camera projection, the 3D pose of the

object, the effect of lighting, the surface reﬂectance,

nonrigid deformations, and other factors. For exam-

ple, in [7], the authors derived a bilinear form for this

function under the assumption of small motion. It

could be signiﬁcantly more complex in general. This

complexity makes it difﬁcult to obtain a global opti-

mum for the optimization function, unless a good

starting point is available. This initialization is often

obtained through a face detection module working on

the ﬁrst frame of the video sequence. For 3D model-

based tracking algo rithms, it also requires registration

of the 3D model to the detected face in the ﬁrst frame.

The need for a good initialization for stable face

tracking is only one of the problems. All trackers suffer

from the problem of drift of the estimates and face

tracking is no exception. Besides, the synthesis func-

tion f may be difﬁcult to deﬁne precisely in many

instances. Examples include partial occlusion of the

face, deformations due to expression changes, and

variations of lighting including cast shadows. Special

care needs to be taken to handle these situations, since

direct optimization of the cost function (2) would give

an incorrect result.

384

Face Tracking

Computational speed is another important issue in

the design of tracking algorithms. Local optimization

methods like gradient descent, Gauss–Newton, and

Levenberg–Marquardt [8] can give a good result if the

starting point is close to the desired solution. However,

the process is often slow because it requires recompu-

tation of derivatives at each iteration. Recently, an

efﬁcient and accurate method of performing the opti-

mization has been proposed by using an inverse

compositional approach, which does not require re-

computation of the gradients at each step [6]. In this

approach, the transformation between two frames is

represented by a

▶ Face Warping function, which is

updated by ﬁrst inverting the incremental warp and

then composing it w ith the current estimate. Our

independent experimental evaluation has shown that

on real-life facial video sequences, the inverse compo-

sitional approach leads to a speed-up by at least one

order of magnitude, and often more, leading to almost

real-time performance in most practical situations.

Challenges in Face Tracking

As mentioned earlier, the main challenges that face

tracking methods have to overcome are (1) variations

of pose and lighting, (2) facial deformations, (3) oc-

clusion and clutter, and (4) facial resolution. These are

the areas where future research in face tracking should

concentrate. Some of the methods proposed to address

these problems will be reviewed brieﬂy below.

1. Robustness to pose and illumination variations. Pose

and

▶ illumination variations often lead to loss of

track. One of the well-known methods for dealing

with illumination variations was presented in [9],

where the authors proposed using a parameterized

function to describe the movement of the image

points, taking into account illumination variation

by modifying the brightness constancy constraint

of optical ﬂow. Illumination invariant 3D tracking

was considered within the active appearance model

(AAM) framework in [10], but the method requires

training images to build the model and the result

depends on the quality and variety of such data. 3D

model based motion estimation algorithms are the

usually robust to pose variations, but often lack

robustness to illumination. In [7], the authors pro-

posed a model-based face tracking method that was

robust to both pose and lighting changes. This was

achieved through an analytically derived mode l for

describing the appearan ce of a face in terms of its

pose, the incident lighting, shape, and surface re-

ﬂectance. Figure 1 shows an example.

2. Tracking through facial deformations. Tracking faces

through changes of expressions, i.e., through facial

deformations, is another challenging problem. An

example of face tracking through changes of ex-

pression and pose is shown in Fig. 2. A survey of

work on facial expression analysis can be found in

[12]. The problem is closely related to modeling of

facial expressions, which has applications beyond

tracking, notably in computer animation. A well-

known work in this area is [13], which has been

used by many researchers for tracking, recognition,

and reconstruction. In contrast to this model-based

approach, the authors in [14] proposed a data-

driven approach for tracking and recognition of

non-rigid facial motion. More recently, the 3D

morphable model [15] has been quite popular in

synthesizing different facial expressions, which

implies that it can also be used for tracking by

posing the problem as estimation of the synthesis

parameters (coefﬁcients of a set of basis functions

representing the morphable model).

3. Occlusion and clutter. As with most tracking pro-

blems, occlusion and clutter affect the performance

Face Tracking. Figure 1 Tracked points on a face through changes of pose and illumination. These points are

projections of a 3D face mesh model.

Face Tracking

385

of most face trackers. One of the robust tracking

approaches in this scenario is the use of particle

ﬁlters [16], which can recover from a loss of track

given a high enough number of particles and obser-

vations. However, in practice, occlusion and clutter

remain serious imp ediments in the design of highly

robust face tr acking systems.

4. Facial resolution. Low resolution will hamper per-

formance of any tracking algorithm, with face

tracking being no exception. In fact, [5] identiﬁed

low resolution to be one of the main impediments

in video-based face recognition. Figure 3 shows an

example of tr acking through scale changes and

illumination. Super-resolution approaches can be

used to overcome these problems to some extent.

However, super-resolution of faces is a challenging

problem by itself because of detailed facial features

that need to be modeled accurately. Recently, [17]

proposed a method for face super-resolution using

AAMs. Super-resolution requires registration of

multiple images, followed by interpolation. Usual-

ly, these two stages are treated separately, i.e., regis-

tration is obtained through a tracking procedure

followed by super-resolution. In a recent paper

Face Tracking. Figure 2 An example of face tracking under changes of pose and expressions. The estimated pose is

shown on the top of the frames. The pose is represented as an unit vector for the rotation axis, and the rotation angle in

degrees, where the reference is taken to be the frontal face.

Face Tracking. Figure 3 Tracked points on a face through changes of scale and illumination.

386

Face Tracking

[18], the authors proposed feeding back the super-

resolved texture in the nth frame for tracking the

(n þ1)th frame. This improves the tracking, which,

in turn, improves the super-resolution output.

This could be an interesting area of future work

taking into consideration issues of stability and

convergence.

Some Applications of Face Tracking

Some applications where face tracking is an important

tool have been highlighted below:

1. Video surveillance. Since faces are often the most

easily recognizable signature of identity and intent

from a distance, vid eo surveillance systems often

focus on the face [5]. This requires tracking the face

over multiple frames.

2. Biometrics. Video-based face recognition systems

require align men t of the faces before they can

be compared. This alignment compensates for

changes of pose. Face tracking, especially 3D

pose estim ation, is therefore an important com-

ponent of such applications. Also, integration of

identity over the entire video sequence requires

tracking the face [1].

3. Face modeling. Reconstruction of the 3D model of a

face from a video sequence using structure from

motion requires tracking. This is because the depth

estimates are related nonlinearly to the 3D motion

of the object. This is a difﬁcult nonlinear estimation

problem and many papers can be found that focus

primarily on this, some examples being [2–4].

4. Video communications and multimedia systems.

Face tracking is also important for applications like

video communications. Motion estimates remove

the interframe redundancy in video compression

schemeslikeMPEGand H.26x.In multimediasystems

like sports videos, face tracking can be used in

conjunction with recognition or reconstruction mod-

ules, orfor focusing on aregion of interestin theimage.

Summary

Face tracking is an important criterion for a number

of applications, like video surveillance, biometrics,

video communications, and so on. A number of meth-

ods have been proposed that work reasonably well under

moderate changes of pose, lighting and scale. The output

of these methods vary from head location in the image

frame to tracked facial features to 3D pose estimation.

The main challenge that future research should address

is robustness to changing environmental conditions,

facial expressions, occlusions, clutter, and resolution.

Related Entries

▶ Face Alignment

▶ Face Recognition

References

1. Zhou, S., Krueger, V., Chellappa, R.: Probabilistic recognition of

human faces from video. Comput. Vision Image Understand. 91,

214–245 (2003)

2. Fua, P.: Regularized bundle-adjustment to model heads from

image sequences without calibration data. Int. J. Comput. Vision

38, 153–171 (2000)

3. Shan, Y., Liu, Z., Zhang, Z.: Model-based bundle adjustment

with application to face modeling. In: Proceedings of IEEE Inter-

national Conference on Computer Vision, pp. 644–651 (2001)

4. Roy-Chowdhury, A., Chellappa, R., Gupta, R.: 3D face modeling

from monocular video sequences. In: Face Processing: Advanced

Modeling and Methods. Academic Press, New York (2005)

5. Zhao, W., Chellappa, R., Phillips, P., Rosenfeld, A.: Face Recog-

nition: A Literature Survey. ACM Transactions (2003)

6. Baker, S., Matthews, I.: Lucas–Kanade 20 years on: A unifying

framework. Int. J. Comput. Vision 56, 221–255 (2004)

7. Xu, Y., Roy-Chowdhury, A.: Integrating motion, illumination

and structure in video sequences, with applications in illumina-

tion-invariant tracking. IEEE Trans. Pattern Anal. Machine

Intell. Vol. 29, 793–806 (2007)

8. Luenburger, D.: Optimization by Vector Space Methods. Wiley,

New York (1969)

9. Hager, G.D., Belhumeur, P.: Efﬁcient region tracking with para-

metric models of geometry and illumination. IEEE Trans. Pat-

tern Anal. Mach. Intell. 20, 1025–1039 (1998)

10. Koterba, S., Baker, S., Matthews, I., Hu, C., Xiao, H., Cohn, J.,

Kanade, T.: Multi-view aam ﬁtting and camera calibration. In:

IEEE Intl. Conf. Comput. Vision (2005)

11. Lepetit, V., Fua, P.: Monocular Model-Based 3D Tracking of

Rigid Objects. Now Publishers Inc. (2005)

12. Fasel, B., Luettin, J.: Automatic facial expression analysis: a

survey. Pattern Recognit. 86, 259–275 (2003)

13. Terzopoulos, D., Waters, K.: Analysis and synthesis of facial

image sequences using physical and anatomical models. IEEE

Trans. Pattern Anal. Mach. Intell. 15, 569–579 (1993)

14. Black, M., Yacoob, Y.: Tracking and recognizing rigid and non-

rigid facial motions using local parametric models of image

motion. In: International Conference on Computer Vision,

pp. 374–381 (1995)

Face Tracking

387

15. Blanz, V., Vetter, T.: Face recognition based on ﬁtting a 3D

morphable model. IEEE Trans. Pattern Anal. Mach. Intell. 25,

1063–1074 (2003)

16. Arulampalam, M., Maskell, A., Gordon, N., Clapp, T.: A tutorial

on particle ﬁlters for online nonlinear/non-Gaussian Bayesian

tracking. IEEE Trans. Signal Process. 50 (2002)

17. Dedeoglu, G., Baker, S., Kanade, T.: Resolution-aware ﬁtting of

active appearance models to low-resolution images. In: European

Conference on Computer Vision (2006)

18. Yu, J., Bhanu, B., Xu, Y., Roy-Chowdhury, A.: Super-resolved

facial texture under changing pose and illumination. In: Inter-

national Conference on Image Processing (2007)

Face Variation

CARLOS D. CASTILLO,DAV I D W. JACO B S

Department of Computer Science, University of

Maryland, College Park, MD, USA

Synonym

Facial changes

Definition

Face variation refers to the way in which the appearance

of the face changes due to changes in viewing conditions

such as illumination or pose, or due to changes in

properties of the face, such as its expression or age.

Introduction

Face recognition is a fundamental problem in bio-

metrics. One of the chief sources of difﬁculty in face

recognition is the large number of variations that can

affect the appearance of faces. These include changes in

lighting, pose, facial expression, makeup, hair, glasses,

facial hair, occlusion by objects that block part of the

face from view, aging, and weight gain or loss. Many

studies suggest that these variations can signiﬁcantly

reduce the performance of recognition algorithms.

Some face recognition systems aimed at cooperative

subjects deal with this problem by attempting to

control these sources of variation. This may be appro-

priate for some applications. In these cases, pose can

be controlled by requiring a subject to look into the

camera, which is kept at a ﬁxed height. Indoors, con-

trolled lighting can be employed. And subjects may be

requested to keep a neutral facial expression, and avoid

variation in occluding objects, such as eye glasses or

scarves. Working under such controlled conditions,

face recognition systems have achieved high levels of

accuracy [1].

However, in many cases large variations in appear-

ance cannot be controlled. One may wish to recognize a

person based on photographs taken some time before,

as when one veriﬁes that a person matches a passport

photograph. In this case, changes in appearance due to

aging, changes in weight, or variations in hair style will

be inevitable. In many applications involving security

or interactions between a computer or robot and a

person, at least a few days may pass between the time

a face is ﬁrst learned and then later recognized. Even

over short time periods there may be variation in a face

due to changes in makeup, or in how recently a subject

has shaved. Finally, in many applications, even lighting,

pose or facial expression cannot be controlled, either

because a subject is uncooperative or because one

wishes to have the ﬂexibility to recognize people as

they move naturally through an environment, changing

their position relative to the camera and lights.

There has been relatively less work on face recogni-

tion in the presence of these variations than for recogni-

tion under controlled conditions. Of these variations,

lighting and pose variation have received the most at-

tention. Many other sources of variation have been the

subject of only a few research efforts. For example, to the

authors’ knowledge, there has been no work explicitly

aimed at accounting for variations due to weight change.

Furthermore, most research has been limited to the case

in which conditions are controlled when subjects are

enrolled into a gallery of known faces, so that, for

example, all gallery images are taken in the same pose

or lighting. Then, research focuses on matching a probe

face viewed under different conditions to the correct

entry in the gallery. Face recognition becomes much

more difﬁcult when the gallery is imaged under heter-

ogenous conditions. For example, there may be a ten-

dency to match a face viewed with side lighting to the

gallery face of a different person viewed with the same

lighting when the gallery face of the correct person is

acquired under very different lighting conditions [2].

However, there are many applications, such as the

organization of personal photos, in which it may not

388

Face Variation

be possible to acquire a gallery of images taken under

controlled conditions.

Illumination

Changes in lig hting can produce signiﬁcant variation

in the appearance of a face. These changes occur due to

an actual variation in lighting, such as the difference

between indoor and outdoor illumination, but they

also occur when a person moves relative to even a

ﬁxed set of lights. Therefore, illumina tion changes

must be accounted for in a wide range of recognition

scenarios. Adini et al. [2 ] has shown that using com-

mon measures of image similarity, there is greater

similarity between two images of different people

taken under the same lighting conditions than between

two images of the same person taken under quite

different lighting conditions. As a consequence, in

spite of a number of research efforts, existing recogni-

tion algorithms show much poorer performance in the

presence of lighting variations than when used with

controlled lighting [1].

A number of approaches have been taken in order

to mitigate the effects of lighting change. Three com-

mon strategies include the following. First, one

can apply image representations that are generically

insensitive to lig hting variation. Second, one can

train a recognition system using sets of imag es that

provide examples of the effects of lighting variation on

images of faces. Third, one can use knowledge of the

three-dimensional shape of faces either to predict the

effect that changes in lighting might have on their

appearance in images, or as a representation that is

unaffected by lighting. These approaches are summar-

ized brieﬂy here; the reader can ﬁnd more details in [ 3,

4, 5], and [6].

Determining the intrinsic properties of a scene

independent of lighting conditions is a classic problem

in computer vision that has been studied for decades.

Multiplicative and additive effects of lighting can be

removed by normalizing the mean and variance of

the image intensities.

▶ Histogram equalization has

been applied to remove lighting effects that produce a

monotonic change in image intensities. Finally, some

representations of images have been shown to be less

sensitive to lig hting variations, including the direction

of image gradients, vectors containing the output

of Gabor ﬁlters [7], or representations that attribute

low frequency components of the image to lighting,

and remove these effects. These and other, related

techniques, have been shown to produce substanti al

improvements in recognition performance compared

to methods that compare raw pixel intensities.

In a second approach, a training set of images is

used to learn the effects of lighting variation on the

appearance of faces. The training set may contain

images of many individuals who are different from

those the system will later try to recognize. These

images show the variation in appearance of each per-

son in the training set as the lighting varies. Methods

such as

▶ Linear Discriminant Analysis may be used to

then ﬁnd representations of faces that best capture the

information that varies between individuals, while dis-

carding information that varies due to light, but not

due to identity [8]. There is also a good deal of evi-

dence that the set of images that a face produces under

a wide range of lighting conditions occupy a low-

dimensional linear subspace in the space of all possible

images. This implies that when the gallery contains

multiple images of each subject, taken under different

lighting conditions, a linear subspace spanned by these

images can be used to represent the subject.

A third set of methods makes use of knowledge

of the 3D structure and surface reﬂectance properties

of faces to predict and compensate for the effects of

lighting [5]. This can involve obtaining a model of

each face to be recognized. Acquisition systems that

can capture the 3D structure of a face, along with the

varying surface properties of eyebrows, lips, and skin

exist. This makes the process of enrollment into the

biometric system more complex, though. An alterna-

tive is to use general knowledge obtained from 3D

scans of a training set of individuals other than the

person to be recognized. In the latter case, a generic

face model may be ﬁt to a gallery image, producing a

model speciﬁc to that person. A model of a person’s

face can be used to solve for the lighting that best

matches that model to the probe image. Recognition

can then be performed by comparing the probe image

to a rendering of the model, produced by computer

graphics. Other approaches may use the model to build

representations of a face’s appearance under diverse

lighting conditions, and compare these to the probe

image. Finally, if one obtains a 3D model as a probe,

this can be directly compared to a 3D model acquired

at enrollment.

Face Variation

389