Li S.Z., Jain A.K. (eds.) Encyclopedia of Biometrics

Подождите немного. Документ загружается.

thermal IR bands. The reﬂected IR band (0.7–2.4 m)

corresponds to reﬂected solar radiation and contains

no information about the thermal properties of

objects. It can be divided into two subbands: the

near-ir (NIR) (0.7–0.9 m) and the short-wave infrared

(SWIR) (0.9–2.4 m) bands.

The thermal IR band (2.4–14.0 m) corresponds to

thermal radiation em itted from objects. Temperature

variations on the surface of an object produce a heat

pattern, called th ermogram, which can be visualized as

a 2D image (i.e., thermal image). The amount of emit-

ted radiation depends both on the temperatu re and the

emissivity of the objects [6]. The thermal IR band can

be divided into two subbands: the mid-wave infrared

(MWIR) w ith range 3.0–5.0 m and long-wave infra-

red (LWIR) with range 8.0–14.0 m. It should be men-

tioned that there are strong atmospheric absorption

bands at 2.4–3.0 m and at 5.0–8.0 m.

Due to the presence of highly distinctive and per-

manent physiological patterns unde r the facial skin

(i.e., vein and tissue structure) [7], thermograms con-

tain important information that can be exploited for face

recognition. The human face emits thermal radiation

both in the MWIR and LWIR bands of the thermal IR

spectrum. However, thermal emissions of the skin are

much higher in the LWIR band than in the MWIR

band. As a result, face images have a much lower

within-class variation in the LWIR spectrum. To ana-

lyze a thermal image,

▶ radiometri c calibration is re-

quired. This is a process which achieves a direct

relation between the value at a pixel of the thermal

image and the absolute amount of thermal emission

from the corresponding physical scene element. The

goal is to standardize thermal IR image s, independent-

ly of environmental condi tions, cameras, and passage

of time [3].

Recognition in the Thermal IR

Face recognition in the visible spectrum exploits the

reﬂectance characteristics of the human face. As a

result, changes in ambient illumination might degrade

recognition performance. Face recognition in the ther-

mal IR spectrum exploits physiological characteristics

of the face by considering the thermal energy emitted

from the face ra ther than the light reﬂected. Therefore,

face recognition in the thermal infrared ( IR) spectrum

is nearly invariant to changes in ambient illumination.

Moreover, it is less sensitive to scattering and absorp-

tion by smoke or dust while the tasks of face detection

and localization can be simpliﬁed considerably due to

the fact that background clutter is typically not visible.

Early overviews of face identiﬁcation in the thermal IR

spectrum can be found in [8–10]. A recent review on

face recognition methods, both in the visible and ther-

mal IR bands, can be found in [11] while a general

review on multispectral face recognition methods, with

emphasis on thermal IR, can be found in [12].

The effectiveness of visible versus IR spectrum

was compared in an early study using several recogni-

tion algorithms in [13]. Using a database of subjects

without eyeglasses, varying facial expression, and

allowing minor lighting changes, it was found that

there are no signiﬁcant performance differences

between visible and IR recognition across all the algo-

rithms tested. In later studies [3, 14, 15], several popu-

lar appearance-based face recognition me thodologies

were tested under vario us lighting conditions and

facial expressions. Results from these studies indicate

superior performance for thermal IR-based recogni-

tion comapred with that for visible-based recognition.

These ﬁndings were conﬁrmed in an operational scenar-

io where images were captured both indoors and out-

doors [16].

The effect of lighting, facial expression, and passage

of time between the gallery and probe images were

examined in [17]. Although IR-based recognition out-

performed visible-based recognition assuming lighting

and facial expression changes, it was found that IR -based

recognition degrades when there is substantial passage of

time between the gallery and probe images. In a related

study [18], however, it was reported that both thermal

IR and visible imagery degrade similarly with time pas-

sage. Improve ments using fusion were reported in

[16, 17]. Recognition using thermal IR was also shown

to be less sensitive to changes in 3D head pose and facial

expression in [4]. In [19], a statistical hypothesis prun-

ing methodology was introduced for face recognition in

thermal IR. First, each thermal IR face image was decom-

posed into spectral features using Gabor ﬁlters. Then,

it was represented by a few parameters by modeling

the marginal density of the Gabor ﬁlter coefﬁcients

using Bessel functions. Recognition was performed in

the space of parameters of the Bessel functions.

Methodologically, the majority of the thermal IR face

recognition methods reported in the earlier sections do

not differ signiﬁcantly from face recognition methods in

360

Face Recognition, Thermal

the visible band (i.e., appearance-based and feature-

based). An exception is the method presented in [7]

which explicitly exploits physiological information

using the bioheat information present in thermal images.

In particular, this method extracts the superﬁcial blood

vessel network which contains contour shapes quite

characteristic of each individual. Matching is based on

the branching points of the skeletonized vascular net-

work. Using physiological features has the potential to

improv e thermal IR-based recognition by making recog-

nition more robust to changes over time.

Limitations of Thermal IR

Despite its advantages, thermal IR has several draw-

backs. First, it is sensitive to temperature changes in

the surrounding environment. Currents of cold or

warm air could inﬂuence the performance of systems

using IR imagery. Second, it is sensitive to variations in

the heat patterns of the face. Factors that could contribute

to these variations include facial expressions (e.g. open

mouth), physical conditions (e.g. lack of sleep, physical

exercise), and psychological conditions (e.g. fear , stress,

excitement). Third, thermal IR is opaque to glass. Glass

blocks a large portion of thermal energy resulting in a loss

of information near the eye region as shown on Fig. 2.

Finally, radi ometric calibration is required every time

the environmental conditions change (e.g., moving the

camera at a different location), a different camera is

used (e.g., even if it is the same model), or data collec-

tions take place at different time intervals.

Fusion of Visible with Thermal IR

Imagery

The beneﬁts of fusing visible with thermal IR imagery

have been documented in a number of studies includ-

ing [3, 17, 20–23 ]. The idea is to combine the strengths

of each spectral band to build more accurate and robust

face recognition systems. For example, increased body

temperature changes the thermal characteristics of

theface,whiletherearenotsigniﬁcantdifferencesin

the visible spectrum. Also, while eyeglasses completely

occlude the eyes in the thermal IR spectrum, the prob-

lem is considerably less severe in the visible spectrum

although visible imagery can suffer from highlights on

the glasses under certain illumination conditions.

A summary of the fusion strategy reported in [21–23]

for improving face recognition performance in the

presence of eyeglasses is as follows.

Objects made of glass act as a temperature screen,

completely hiding the parts located behind them. In

the case of subjects wearing eyeglass es, this poses some

major difﬁculties since the eyes would be occluded

completely due to the fact that eyeglasses block thermal

energy (i.e., see Fig.2). Experimental results, reported

in [21–23], illustrate that face recognition performance

in the thermal IR degrades seriously when eyeglasses

are present in the probe image but not in the gallery

image and vice versa. To address this limitation, fusion

of thermal IR with v isible imagery was employed in

[21–23]. Two different fusion strategies were investi-

gated: pixel-based fusion in the wavelet domain, and

feature-based fusion in the eigenspace domain. In both

cases, fusion was carried out using Genetic Algorithms

(GAs) [24].

The Equinox database [25] was used for experi-

mentation. The database contains frontal faces

under the following scenarios: (1) three different light

directions – frontal and lateral (right and left); (2) three

facial expression – ‘‘frown,’’ ‘‘surprise’’ and ‘‘smile’’; (3)

vocals pronunciation expressions – subjects were asked

to pronounce several vocals from which three represen-

tative frames were chosen; and (4) presence of glasses –

for subjects wearing glasses, all of these scenarios were

repeated with and without glasses. For testing, the data

were divided as follows: EG (expression frames w ith

glasses, all illuminations), EnG (expression frames

without g lasses, all illuminations), EFG (expression

frames with g lasses, frontal illumination), ELG (expres-

sion frames with glasses, lateral illumination), EFnG

(expression frames without glasses, frontal illumina-

tion), ELnG (expression frames without glasses, lateral

illumination). The inclusion relations among these sets

are as follows:

EG ¼ ELG [ EFG;

EnG ¼ ELnG [ EFnG and EG \ EnG ¼;

ð1Þ

Recognition performance was measured by ﬁnd-

ing the percentage of the images in the test set, for

which the top match is an image of the same person

from the gallery. Figures 3 and 4 show the results

obtained. Among the two fusion strategies tested,

fusion in the wavelet domain yielded the best results.

Nevertheless, fusion outperformed each modality

alone in both cases.

Face Recognition, Thermal

361

Milestones

Despite its advantages, face recognition in the thermal

IR spectrum has received relatively little attention

compared to visible spectrum, mostly because of the

following reasons:



Higher cost of thermal sensors



Lower image resolution



Higher image noise



Lack of widely available data sets

Advances in IR imaging technology and the availabil-

ity of publicly available datasets, however, have facili-

tated experimentation with thermal imagery in the

context of face recognition. While the difference in

cost between visible and thermal imaging equipment

is still large, the gap is closing rapidly as new uncooled

microbolometer technologies enter the market. Com-

panies, such as FLIR Systems (http://www.ﬂir.com),

offer a large variety of thermal cameras for different

budgets. The issue of image noise can be addressed

Face Recognition, Thermal. Figure 2 (a, b) Visible images; (c, d) thermal IR images. It should be observed that since

thermal IR is opaque to glass, the presence of eyeglasses blocks the eyes completely.

362

Face Recognition, Thermal

Face Recognition, Thermal. Figure 3 Eyeglasses results in the wavelet domain: (a) same illumination conditions – eyeglasses are not present both in the gallery and

probe sets; (b) eyeglasses are present both in the gallery and probe sets – illumination conditions are different; (c) eyeglasses are not present both in the gallery and probe

sets – illumination conditions are different; (d) similar to (c) except that the gallery and probe sets contain multiple illuminations.

Face Recognition, Thermal

363

using powerful radiometric calibration procedures

while the issue of image resolution can be addressed

using super-resolution techniques or fusing infrared

with visible imagery. For example, Equinox Corporation

(http://www.equinoxsensors.com) has made available

a system for real-time fusion of thermal IR with visible

imagery with image co-registration correction.

In terms of data, things are rather limited com-

pared to the plethora of face databases available in

the visible spectrum [26 , 27]. The most extensive IR

facial database, that is publicly available, is the Equinox

database [25]. This database was created by Equinox

Corporation under DARPA’s HumanID program.

It includes coregister ed visible/LWIR/MWIR/SWIR im-

ages and it is representative of unconstrained frontal

imagery of people’s faces in an indoor environment.

Another, publicly available database, is the Notre

Dame IR face database [28], which includes data cap-

tured at different sessions over time. Obviously, addi-

tional datasets are required to spark more research in

this area, in particular, data depicting scenarios widely

different from the imaging conditions during data

acquisition (e.g., outdoor imagery). Introducing ap-

pearance variability due to various factors (e.g., meta-

bolic activity) would be extremely useful in testing the

robustness of thermal IR for face recognition.

Face Recognition, Thermal. Figure 4 Eyeglasses results in the eigenspace domain: (a) same illumination conditions –

eyeglasses are not present both in the gallery and probe sets; (b) eyeglasses are present both in the gallery and probe sets –

illumination conditions are different; (c) eyeglasses are not present both in the gallery and probe sets – illumination

conditions are different; (d) similar to (c) except that the gallery and probe sets contain multiple illuminations.

364

Face Recognition, Thermal

Summary

While face recognition in the visible band performs

satisfactorily under controlled conditions, thermal IR

face recognition offers more advantages when there is

no control over illumination or for detecting disguised

faces. The passive nature of thermal IR systems lowers

their complexity and improves their reliability. With

dropping prices and technological advances, thermal

IR is becoming more affordable and practical than

before. Although thermal IR has many advantages, it

suffers from several drawbacks including that it is

sensitive to temperature changes and opaque to glass.

A promising approach to deal with these issues is

fusing visible with thermal IR imagery.

Related Entries

▶ Biometrics, Overview

▶ Face Recognition

References

1. Zhao, W., Chellappa, R., Phillips, P., Rosenfeld, A.: Face recog-

nition: A literature survey. ACM Comput. Surveys 35(4),

399–458, (2003)

2. Wolff, L., Socolinsky, D., Eveland, C.: Quantitative measurement

of illumination invariance for face recognition using thermal

infrared imagery. In: IEEE Workshop on Computer Vision Be-

yond the Visible Spectrum. Hawaii (2001)

3. Socolinsky, D., Selinger, A., Neuheisel, J.: Face recognition with

visible and thermal infrared. Comput. Vision Image Understand

91, 72–114 (2003)

4. Friedrich, G., Yeshurun, Y.: Seeing people in the dark: face

recognition in infrared images. In: International Workshop on

Biologically Motivated Computer Vision, pp. 348–359 (2002)

5. Pavlidis, I., Symosek, P.: The imaging issue in an automatic face/

disguise detection system. In: IEEE Workshop on Computer

Vision Beyond the Visible Spectrum, pp. 15–24 (2000)

6. Siegel, R., Howell, J.: Thermal Radiation Heat Transfer, 3rd edn.

Taylor & Francis, London (1992)

7. Buddharaju, P., Pavlidis, I., Tsiamyrtzis, P., Bazakos, M.:

Physiology-based face recognition in the thermal infrared spec-

trum. IEEE Trans. Pattern Analy. Mach. Intell., 29(4), 613–626

(2007)

8. Evans, D.: Infrared facial recognition technology being pushed

toward emerging applications. In: Proceedings of SPIE, vol.

2962, pp. 276–286 (1997)

9. Prokoski, F., Riedel, R.: Infrared identiﬁcations of faces and body

parts. In: Jain, A., Bolle, R., Pankanti, S. (eds.) BIOMETRICS:

Personal Identiﬁcation in Networked Society, pp. 191–212 (1999)

10. Prokoski, F.: History, current status, and future of infrared

identiﬁcation. In: IEEE Workshop on Computer Vision Beyond

the Visible Spectrum. Hilton Head (2000)

11. Kong, S., Heo, J., Abidi, B., Paik, J., Abidi, M.: Recent advances in

visual and infrared face recognition – a review. Comput. Vision

Image Understand. 97, 103–135 (2005)

12. Socolinsky, D.: Multispectral Face Recognition. In: HandBook of

Biometrics, Jain, A., Flynn, P., and Ross, A. (eds.) pp. 293–313

(2008)

13. Wilder, J., Phillips, J., Jiang, C., Wiener, S.: Comparison of visible

and infra-red imagery for face recognition. In: IEEE Internation-

al Conference on Automatic Face and Gesture Recognition, pp.

182–187. Killington (1996)

14. Socolinsky, D., Wolff, L., Neuheisel, J., Eveland, C.: Illumination

invariant face recognition using thermal infrared imagery. In:

Computer Vision and Pattern Recognition Conference. Hawaii

(2001)

15. Socolinsky, D., Selinger, A.: Comparative study of face recogni-

tion performance with visible and thermal infrared imagery. In:

International Conference on Pattern Recognition, pp. 217–222

(2002)

16. Socolinsky, D., Selinger, A.: Thermal face recognition in an

operational scenario. In: IEEE Conference on Computer Vision

and Pattern Recognition (CVPR) (2004)

17. Chen, X., Flynn, P., Bowyer, K.: Ir and visible light face recogni-

tion. Comput. Vision Image Understand. 99, 332–358 (2005)

18. Socolinsky, D., Selinger, A.: Thermal face recognition over time.

In: International Conference on Pattern Recognition (ICPR)

(2004)

19. Srivastana, A., Liu, X.: Statistical hypothesis pruning for recog-

nizing faces from infrared images. Image Vision Comput. 21,

651–661 (2003)

20. Heo, J., Kong, S., Abidi, B., Abidi, M.: Fusion of visual and

thermal signatures with eyeglass removal for robust face recog-

nition. In: Workshop on Object Tracking and Classiﬁcation

Beyond the Visible Spectrum (2004)

21. Gyaourova, A., Bebis, G., Pavlidis, I.: Fusion of infrared and

visible images for face recognition. In: European Computer

Vision Conference (ECCV) (2004)

22. Singh, S., Gyaourova, A., Bebis, G., Pavlidis, I.: Infrared and

visible image fusion for face recognition. In: SPIE Defense and

Security Symposium (Biometric Technology for Human Identi-

ﬁcation) (2004)

23. Bebis, G., Gyaourova, A., Singh, S., Pavlidis, I.: Face recognition

by fusing thermal infrared and visible imagery. Image Vision

Comput. 24(7), 727–742 (2006)

24. Goldberg, D.: Genetic Algorithms in Search, Optimization, and

Machine Learning. Addison Wesley, Reading, MA, USA (1989)

25. Equinox corporation, ir face database. http://www.equinoxsensors.

com/products/HID.html (last visited in June 2008)

26. Gross, R.: Face Databases. In: Li, S.Z., Jain, A.K. (eds) Handbook

of Face Recognition, pp. 301–327 (2005)

27. Face recognition homepage. http://www.face-rec.org/ (last visited

in June, 2008)

28. Computer vision laboratory, university of notre dame, biometrics

database distribution. http://www.nd.edu/cvrl/ (last visited in

June, 2008)

Face Recognition, Thermal

365

Face Recognition, Video-Based

RAMA CHELLAPPA

,GAURAV AGGARWAL

S. K

EVIN ZHOU

University of Maryland, College Park, USA

Siemens Corporate Research, Princeton, NJ, USA

Synonyms

Face recognition from image sequences; Video-

based face recognition

Definition

Video-based face recognition is the technique of estab-

lishing the identity of one or multiple persons present

in a video, based on their facial characteristics. Given

the input face video, a typical video-based face recog-

nition approach combines the temporal characteristics

of facial motion with appearance changes for recogni-

tion. This often involves

▶ temporal characterization

of faces for recognition, building 3D model or a super-

resolution image of the face, or simply learning the

appearance variations from the multiple video frames.

The ability to generalize across po se, illumination,

expression, etc. depends on the choice of combination.

Video-based face recognition is particularly useful in

surveillance scenarios in which it may not be possible

to capture a single good frame as required by most still

image based methods.

Introduction

Face recognition is one of the most successful applica-

tions in the vast amount of research on image analysis

and understanding [1]. The fact that face recognition

can be performed at a distance without subject’s coop-

eration or knowledge makes it particularly attractive as

compared to more reliable biometrics like ﬁngerprints

and iris or retinal scans. Traditionally face recognition

has been limited to still images. Though great leaps

have been made in recognizing faces from still images,

more needs to be done to achieve the goal of recogniz-

ing faces in uncontrolled scenarios. Still image based

approaches often struggle to truly generalize across

variations in pose, expression, illumination, etc., lead-

ing to a not so satisfactory performance on real images.

The advent of inexpensive cameras and increased pro-

cessing power has made it possible to capture and store

videos in real time. Videos have the advantage of

providing more information in the form of multiple

frames making it relatively easier to generalize across

variations that have been difﬁcult with still images.

Moreover, v ideo makes it easier to track (or segment)

faces which can then be fed into a recognition system.

Importantly, psychological evidence indicates that

dynamic information contributes to face recognition

especially under nonoptim al viewing conditions [2].

These reasons form the basis of the recent interest in

using videos for recognizing faces [3–5]. Though video

provides extra information, the video feeds are almost

always uncontrolled making it challenging to track and

hence recognize faces.

Operation of a Video-based Face

Recognition System

A typical Video-based Face Recognition (VFR) system

operates by acquiring video feeds from one or multiple

cameras, tracking and segment ing faces from the

input feed(s), extracting representations to characterize

the identity of the face(s) in the video, and then com-

paring them with the enrolled representations of sub-

jects in the database. This constitutes the test phase of

the system. During the enrollment (or training) phase,

a similar sequence of steps is followed using one or

multiple video feeds per identity and the corresponding

composite representations are stored in the database.

VFR approaches differ in the representation that is

used to characterize the moving faces. An ideal VFR

system performs these operations automatically with-

out any human intervention. Though potentially a

VFR system can operate in either veriﬁcation mode

(one-to-one matching) or identiﬁcation mode (one-

to-many), the real applicati on of such a system lies

in identi fying subjects using surveillance cameras

(say on an airport) without their knowledge. There-

fore, a typical VFR system will often operate in what

is known as watch list mode [6]. The watch

list problem is a generalization of both identiﬁca-

tion and veriﬁcation problems in which the system

only attempts the identiﬁcation of individuals on

the watch list. The performance in this mode is

measured using both identiﬁcation rate and false

alarm rate.

366

Face Recognition, Video-Based

Challenges for Video-based Face

Recognition Systems

Effective utilization/fusion of the information (both

spatial and temporal) present in a v ideo to achieve

better generalization (for each subject) and discrimi-

nability (across different subjects) for improved iden-

tiﬁcation is one of the biggest challenges faced by a

VFR system. The fusion schemes can range from sim-

ple selection of good frames (which are then used for

recognition in a still-image based recognition frame-

work) to estimation of the full 3D structure of a face

which can then be used to generalize across pose,

illumination, etc. The choice may depend primarily

on the operational requirements of the system. For

example, in a surveillance setting, the resolution of

the faces may be too small for reliable shape estima-

tion. The choice also limits the recognition capability

of the system. A simple good frame selection scheme

will not have the capability to generalize appearance

across pose variations and thus requires the test video

to have some pose overlap with the galler y videos.

Effective modeling of subject-speciﬁc facial character-

istics from video data can only be achieved if the

changes in facial appearance during the course of

the video are appropriately attributed to different fac-

tors like pose changes, lighting, expression variations,

etc. Unlike still image based scenarios, these variations

are inherent in a VFR setting and must be accounted

for to reap the beneﬁts of extra information provided

by the video data. In addition, due to the nature of the

input data, VFR is often addressed in conjunction w ith

tracking problem which is a challenging problem by

itself. In fact, more often than not, tracking accuracy

depends on the knowledge of reliable appear ance

model (depends on the identity provided by the recog-

nition module) while recognition result is dependent

on the localization accuracy of the face region in input

video.

Examples of Video-based Face

Recognition Algorithms

Given the potential advantages video provides for the

task of face recognition, relatively little work has been

done to recognize faces in videos. The challenges in

modeling moving faces along with the unavailability of

large standard datasets have hindered the progress of

research on VFR algorithms. Table 1 gives a snapshot

of a few existing VFR algorithms. As clear from the

table, all the approaches have been tested on a very

small sized (often private) datasets. The following dis-

cussion describes them in detail.

1. Simultaneous Tracking and Recognition of Faces:

Traditional tracking-then-recognition approaches

resolve uncertainties in tracking and recognition

sequentially and separately, which often involves dif-

ﬁcult choices (like criteria to select good frames and

Face Recognition, Video-Based. Table 1 A snapshot of a few existing video-based face recognition algorithms

Algorithm Short description Experimental evaluation

Probabilistic recognition of human

faces from video [7]

Simultaneous tracking-and-

recognition using a time series state

space model and sequential

importance sampling

Private: 12 subjects, NIST:

30 subjects, MoBo [8]: 25 subjects

Video-based face recognition using

probabilistic appearance

manifolds [9]

Face modeled using a

low-dimensional appearance

manifold, approximated by

piecewise linear subspaces

Honda-UCSD dataset:

20 subjects (52 videos)

Face verification through tracking

facial features [10]

Tracks facial features defined on a

grid with Gabor attributes using SIS

algorithm

Li dataset: 19 subjects

(2 sequences each)

Video-based face recognition using

adaptive hidden markov models [11]

Statistics of training videos, and their

temporal dynamics learnt by an

HMM

Private: 12 subjects,

Mobo [8]: 25 subjects

A system identification approach for

video-based face recognition [12]

Face modeled as a linear dynamical

system using ARMA model

Honda-UCSD dataset [9]: 30 subjects,

Li dataset [10]: 19 subjects

Face Recognition, Video-Based

367

estimation of registration parameters). Zhou et al.

[7] avoid these issues while resolving uncertainties

in tracking and recognition simultaneously in a

uniﬁed probabilistic framework. The temporal in-

formation present in the video is fused using a

time-series state-space model to characterize the

evolving kinematics and identity. The three basic

components of the model are as follows.



A motion equation governing the kinematic

behavior of the tracking motion vector. In its

most general form, the motion equation can be

written as

¼ g ðy

t1

; u

Þ; t  1; ð1Þ

where u

is the noise that determines the transi-

tion probability p(y

t1

). The function g(.,.)

characterizes the evolving motion. It can either

be a function learned ofﬂine or given a priori.

Choice of y

is application dependent.



An identity equation governing the temporal

evolution of the identity variable.

¼ n

t1

; t  1; ð2Þ



An obser vation equation establishing the link

between the motion vector and the identity

variable.

g¼I

þ n

; t  1; ð3Þ

where n

is the observation noise that deter-

mines the observation likelihood p(z

)

and t

g transforms the observation z

the chosen feature space.

Under the assumption of statistical independence

between all noise variables and prior knowledge of

the distributions p(y

) and p(n

), (1) and (2)

can be combined as follows.

pðx

t1

Þ¼pðn

t1

Þpðy

t1

Þ; ð4Þ

where x

¼ (y

). Given a video sequence, the goal

is to estimate the posterior probability p(n

0:t

The posterior probability is calculated using

Sequential Importance Sampling (SIS) [13]. Using

the SIS technique, the joint probability distribution

of the motion vector and the identity variable is

estimated at each time instant and then propagated

to the next time instant as governed by the motion

and identity equations. The marginal distribution

of the identity variable is estimated to provide the

desired identity result. Fig. 1 shows the performance

of the approach on a NIST dataset consisting of 30

persons gallery.

2. Probabilistic Appearance Manifolds for VFR: Similar

to [7], Lee et al. [9] propose a VFR algorithm that

performs modeling, tracking and recognition in one

integrated framework. This is accomplished using a

probabilistic appearance manifold based representa-

tion that is utilized simultaneously by both tracking

and recognition modules. The recognition module

uses tracker’s output (the location of the face in the

current frame) to update the current internal appear-

ance model that is in turn used by the tracker.

Each face is characterized using a collection of

linear subspaces in the image space which is con-

structed by clustering the exemplars from the input

face videos. Each cluster often contains face images

with similar poses and is represented using a PCA

subspace. The collection of linear subspaces is fur-

ther characterized using a transition matrix that

captures the probabilities of moving from one

pose subspace to another between two consecutive

frames. The transition matrix is used to combine

facial appearance w ith temporal coherency of pose

variations to perform recognition. The approach

has be en tested on 52 video sequences of 20 differ-

ent subjects.

3. 2D Feature Graph based Approach: Li and Chellappa

[10] propose a 2D feature-graph based approach

for VFR in which the intensity model is replaced

by a feature-graph using Gabor transform. The

feature-graph approach is more robust to the

variations in illumination and pose but possibly

requires slightly higher-resolution videos. The track-

ing problem is formulated as a Bayesian inferenc e

problem for which Marko v Chain Monte Carlo

(MCMC) techniques are employed to obtain an em-

pirical solution. A reparameterization is used to facil-

itate empirical estimation and to allo w veriﬁcation to

be addressed simultaneously along with tracking. The

facial features to be tracked are deﬁned on a grid with

Gabor attributes (Fig. 2). The motion of facial fea-

ture points is modeled as a global two-dimensional

afﬁne transformation (to account for head motion)

plus a local deformation to account for the residual

due to expression changes and modeling errors.

The global motion is estimated by impor tance

sampling while the residual motion is handled by

incorporating local deformation into the likelihood

measurement.

368

Face Recognition, Video-Based

The temporal evolution of the jet positions is

modeled as a dynamic system, where tracking is

solved by analyzing this system, which in general,

is non-Gaussian and nonlinear. Note that tracking is

solved by analyzing the dynamic system governing

the evolution of the changes in afﬁne parameters.

This reparameterization originates from a simple

Taylor expansion. However, its novelty comes from

the fact that one can choose different initial states for

different purposes. If the initial state corresponds to

a feature set from the ﬁrst frame of a sequence, then

the reparameterization is suitable for pure tracking.

However, if the initial state represents some template

from a candidate list, then the reparameterization

is naturally good for tracking-for-veriﬁcation. When

a template and the sequence belong to the same

person, tracking results should reﬂect a coherent

motion induced by the same underlying shape.

On the other hand, a more random motion pattern

will often be observed when the template and the

sequence belong to different persons. Thus, with

different templates, such a tracker allows veriﬁcation

to be addressed simultaneously with tracking. The

motion coherence in a shape is evaluated by calcu-

lating the posterior probabilities from the estimated

densities on a region centered on the mean shape.

Fig. 2 shows the tracking and veriﬁcation results

using this approach.

4. Hidden Markov Models for VFR: Li and Chen [11]

propose adaptive Hidden Markov Models (HMM)

to recognize faces in videos. During training, a sepa-

rate HMM is learnt for each subject in the gallery to

characterize appearance statistics and temporal dy-

namics of the facial motion. Recognition is performed

by analyzing the test video by HMMs corresponding

to subjects in the gallery . During the recognition

proce ss, test video sequences are used to update the

gallerymodelsinanunsupervisedfashionbasedon

the recognition result. The approach has been tested

on two datasets with 21 and 24 subjects respec tive ly.

5. 3D Model based Approach: As opposed to most VFR

approaches which model face as a 2D object, the

Face Recognition, Video-Based. Figure 1 Sequential tracking and recognition [7]. Top left: A probe video from the NIST

dataset; Top right: Recognition performance under different models; Bottom: Gallery set.

Face Recognition, Video-Based

369