Li S.Z., Jain A.K. (eds.) Encyclopedia of Biometrics

Подождите немного. Документ загружается.

2. Borda Count. The fused rank scores using Borda

count are computed as follows: m

=(2+3+2)=7,

= (4 + 5 + 3) = 12, m

= (5 + 1 + 4) = 10,

=(1+2+1)=4,m

= (4 + 4 + 5) = 13. Thus ,

is lowest and user D achieves highest combined

ranking (Table 1).

3. Weighted Borda Count. The fused rank scores

=(2 0.5 + 3  0.15 + 2  0.35) = 2.15. Simi-

larly rank fused scores for rest of the users can be

computed and are shown in Table 2.

4. Bayes Fuse. The prior probabilities that each of

the ranks are true (untrue), i.e., belongs to the

genuine (imposter) class, can be obtained from

the training data and are provided in the problem.

The fused rank score for user A can be computed

using (3) as follows: m

= log (0.1/0.9) + log (0.06/

0.94) + log (0.2/0.8) = 6.34. The rest of the fused

rank scores and t he combined rankings are dis-

played in Table 2.

Summary

In the biometrics literatures, one can ﬁnd several

examples [1, 3, 4, 6] of above rank level fusion methods

to consolidate the outputs from different matchers.

Bhatnagar et al. [4] employs a variation of Borda

count method that uses partitioning of templates

to consolidate the combined ranks. Highest rank

method employed by Rautiainen and Seppanen [6], is

referred as lowest rank method since it chooses the

minimum rank from the list of dissimilarity score

instead of conventional maximum rank methods that

employ highest ranks from the list of similarity scores.

Several other variations of Borda count method have

also been developed in the literature [7]; Nenson’s

method that uses successive elimination from Borda

count that are below average Borda count or Quota

Borda method that includes the quota element in

counting ranks. However, they have not yet been inves-

tigated for their utility in the biometrics literature.

A survey of biometrics on various fusion techni-

ques [5] suggests that the rank level fusion method is

less preferred method of fusion while score level fusion

continues to be the most popular method. The rank

level fusion can be more useful in combining decisions

from a large number of biometric matchers and such

large systems has not yet been evaluated in the

biometrics literature.

References

1. Lee, Y., Lee, K., Jee, H., Gil, Y., Choi, W., Ahn, D., Pan, S.:

‘‘Fusion for multimodal biometric identiﬁcation,’’ Proc.

ABVPA 2005, LNCS 3546, 1071–1079 (2005)

Fusion, Rank-Level. Table 1 Example for consolidating ranks using unsupervised rank level fusion methods

Highest rank method Borda count method

User identity

Fused rank score

Combined ranking

Fused rank score

Combined ranking

A m

23m

B m

34m

12 4

C m

11m

10 3

D m

12m

E m

45m

13 5

Fusion, Rank-Level. Table 2 Example for consolidating ranks using supervised rank level fusion methods

Weighted borda count method Bayes fuse method

User identity

Fused rank score

Combined ranking

Fused rank score

Combined ranking

A m

2.15 2 m

6.34 2

B m

3.8 3 m

10.93 4

C m

4.05 4 m

6.48 3

D m

1.15 1 m

1.47 1

E m

4.35 5 m

11.43 5

610

Fusion, Rank-Level

2. Aslam, J.A., Montague, M.: ‘‘Models for metasearch,’’ In: Pro-

ceedings of the 24th ACM SIGIR Conference on Research and

Development in Information Retrieval, Sep. 2001, pp. 379–381

(2001)

3. Ho, T.K., Hull, J.J., Srihari, N.: ‘‘Decision combination in multi-

ple classiﬁer systems.’’ IEEE Trans. Pattern Anal. Mach. Intell. 16,

66–75 (1994)

4. Bhatnagar, J., Kumar, A., Saggar, N.: ‘‘A novel approach to

improve biometric recognition using rank level fusion,’’ Proc.

CVPR 2007, Minneapolis, MN, pp. 1–6, (2007)

5. Ross, A., Nandakumar, K., Jain, A.K.: Handbook of Multibio-

metrics, Springer Verlag (2006)

6. Rautiainen, M. and Seppanen, T.: ‘‘Comparison of visual

features and fusion techniques in automatic detection of

concepts from news video,’’ In: Proceedings of the IEEE Interna-

tional Conference on Multimedia and Expo., ICME 2005,

Amsterdam, pp. 932–935 (2005)

7. Dummett, M.A.E.: Principles of Electoral Reform, Oxford

University, New York (1997)

Fusion, Score-Level

ARU N ROSS

,KARTHIK NANDAKUMAR

West Virginia University, Morgantown, WV, USA

Institute for Infocomm Research A* STAR,

Fusionopolis, Singapore

Synonyms

Fusion at the conﬁdence level; Fusion at the measure-

ment level; Match score fusion

Definition

In score-level fusion the match scores output by mul-

tiple biometric matchers are consolidated in order to

render a decision about the identity of an individual.

Typically, this consolidation procedure results in the

generation of a single scalar score which is subsequently

used by the biometric system. Fusion at this level is the

most commonly discussed approach in the biometric

literature primarily due to the ease of accessing and

processing match scores (compared with the raw bio-

metric data or the feature set extracted from the data).

Fusion methods at this level can be broadly classiﬁed

into three categories: density-based schemes, transfor-

mation-based schemes and classiﬁer-based schemes.

Introduction

A match score is the result of comparing two feature

sets extracted using the same feature extractor. A simi-

larity score denotes how ‘‘similar’’ the two feature sets

are, while a distance score denotes how ‘‘different’’ they

are. Consequently, a high similarity score between a

pair of feature sets indicates a good match whereas

a high distance score indicates a poor match.

In score-level fusion the match scores output by

multiple biometric matchers are combined to generate

a new match score (a scalar) that can be subsequently

used by the veriﬁcation or identiﬁcation modules for

rendering an identit y decision (alternatively, the fusion

process may directly result in a decision). Fusion at this

level is the most commonly discussed approach in the

biometric literature primarily due to the ease of acces-

sing and processing match scores (compared with the

raw biometric data or the feature set extracted from

the data). Fusion methods at this level can be broadly

classiﬁed into three categories [1]: density-based

schemes, transformation-based schemes and classiﬁer-

based schemes.

Density-Based Fusion schemes

Let s ¼½s

; s

; ...; s

 denote the scores emitted by

multiple matchers, w ith s

representing the match

score of the jth matcher, j ¼ 1, ..., R. Further, let

the labels o

and o

denote the genuine and imposto r

classes, respectively. Then, by

▶ Bayes decision theory

[2], the probability of error can be minimized by

adopting the following decision rule. (This is known

as the Bayes decision rule or the minimum-error-rate

classiﬁcation rule under the 0-1 loss function [2]).

Assign s ! o

Pðo

jsÞ > Pðo

jsÞ; i 6¼ j; and i; j ¼ 0; 1: ð1Þ

Here, the a posteriori probability Pðo

jsÞ, i ¼ 0,1,

can be derived from the class-conditional density func-

tion pðsjo

Þ using the Bayes formula, i.e.,

Pðo

jsÞ¼

pðsjo

ÞPðo

pðsÞ

; ð2Þ

where P(o

) is the a priori probability of observing

class o

and pð sÞ denotes the probability of encounter-

ing s. Thus , Eq. (1) can be re-written as

Fusion, Score-Level

611

Assign s ! o

pðsjo

> t; i 6¼ j; and i; j ¼ 0; 1 ð3Þ

where

pðsjo

is known as the likelihood ratio and

t ¼

Pðo

is a predetermined threshold. The density

pðs jo

Þ is typically estimated from a training set of

match score vectors, using parametric or nonparametric

techniques. However, a large number of training sam-

ples are necessary to reliably estimate the joint-density

function pðsjo

Þ especially if the dimensionality of the

feature vector s is large. In the absence of sufﬁcient

number of training samples (which is typically the case

when the multibiometric system is ﬁrst deployed or if its

parameters are subsequently adjusted), it is commonly

assumed that the scalar scores s

, ...s

are generated

by R independent random processes. This assumption

permits the density function to be expressed as

pðsjo

Þ¼

j¼1

pðs

Þ; ð4Þ

where the joint-density function is now replaced by

the product of its marginals. The marginal densities,

p(s

), j ¼ 1, 2, ... R, i ¼ 0, 1, are estimated from a

training set of genuine and impostor scores

corresponding to each of the R biometric matchers.

Equation (4) results in the product rule which com-

bines the scores generated by the R matchers as,

prod

j¼1

pðs

: ð 5Þ

Kittler et al. [3] modify the product rule by further

assuming that the a posteriori probability Pðo

jsÞ of

class o

does not deviate much from its a pr iori proba-

bility P(o

) resulting in the sum rule:

sum

j¼1

pðs

j¼1

pðs

: ð6Þ

Similar expressions can be derived for combining the

match scores using the max, min, and median rules

[1, 3]. All the aforementioned rules implicitly assume

that the match scores are continuous random variables.

Dass et al. [4] relax this assumption and represent

the univariate density functions (i.e., the marginals in

Eq. (4)) as a mixture of discrete as well as continuous

components. The resulting density functions are

referred to as generalized densities. The authors dem-

onstrate that the use of generalized density estimates

(as opposed to continuous density estimates) signi-

ﬁcantly enhances the matching performance of the

fusion algorithm. Furthermore, they use

▶ copula

functions to model the correlation structure between

the match scores s

, s

, ..., s

and, subsequently, deﬁne

a novel fusion rule known as the copula fusion rule.

Transformation-Based Fusion

schemes

Density-based schemes, as stated earlier, require a large

number of training samples (i.e., genuine and impos-

tor match scores) in order to accurately estimate the

density functions. This may not be possible in most

multibiometric systems due to the time, effort, and

cost involved in acquiring labeled multibiometric

data in an operational environment. In such situations,

it may be necessary to directly combine the match

scores generated by multiple matchers using simple

fusion operators (such as the simple sum of scores or

order statistics) without ﬁrst interpreting them in a

probabilistic framework. However, such an approach

is meaningful only when the scores output by the mat-

chers are comparable. To facilitate this, a score normali-

zation process is essential to transform the multiple

match scores into a common domain (it must be

noted, however, that some score normalization schemes

do require a large number of training samples as seen in

the following section). The process of score normaliza-

tion entails changing the location and the scale para-

meters of the underlying match score distributions in

order to ensure compatibility between multiple score

variables. A few of the commonly discussed score nor-

malization methods are described in this article.

The simplest normalization technique is the min–

max normalization. Min–m ax normalization is best

suited for the case where the bounds (maximum and

minimum values) of the scores produced by a matcher

are known. In this case, the minimum and maximum

scores can be easily transformed into 0 and 1, res-

pectively. However, even if the match scores are not

bounded, the minimum and maximum values for the

given set of training match scores can be estimated

prior to applying min–max normalization. Let s

denote the ith match score output by the jth matcher,

612

Fusion, Score-Level

i ¼ 1, 2, ..., N ; j ¼ 1, 2, ..., R (R is the number of

matchers and N is the number of match scores avail-

able in the training set). The min–max normalized

score, ns

, for the test score s

is given by

min

i¼1

max

i¼1

min

i¼1

: ð7Þ

When the minimum and maximum values are esti-

mated from the given set of match scores, this method

is not robust (i.e., the method is sensitive to outliers in

the data used for estimation). Min–max normalization

retains the original distribution of scores except for

a scaling factor and transforms all the scores into a

common range [0, 1]. Distance scores can be trans-

formed into similarity scores by subtracting the nor-

malized score from 1.

Decimal scaling can be applied when the scores

of different matchers are on a logarithmic scale. For

example, if one matcher has scores in the range [0, 10]

and the other has scores in the range [0, 1000], the

following normalization could be applied to trans-

form the scores of both the matchers to the common

[0, 1] range.

; ð8Þ

where n

¼ log

max

i¼1

. In the example with two

matchers where the score ranges are [0, 10] and

[0, 1000], the values of n would be 1 and 3, respectively.

The problems with this approach are the la ck of

robustness and the implicit assumption that the scores

of different matchers vary by a logarithmic factor.

The most commonly used score normali zation

technique is the z-score normalization that uses the

arithmetic mean and standard deviation of the training

data. This scheme can be expected to perform well if

the average and the variance of the score distributions

of the matchers are available. If the values of these two

parameters are not known, then they can be estimated

based on the given training set. The z-score normalized

score is given by

 m

; ð9Þ

where m

is the arithmetic mean and s

is the standard

deviation for the jth matcher. However, both mean

and standard deviation are sensitive to outliers

and hence, this method is not robust. Z-score

normalization does not guarantee a common numer-

ical range for the normalized scores of t he different

matchers. If the distribution of the scores is not

Gaussian, z-score normalization does not preserve

the d istribution of the given set of scores. This is

due to the fa ct that mean an d standard deviation are

the optimal location and scale parameters only for a

Gaussian distribution. While mean and standard de-

viation are reasonable estimates of location and scale,

respectively, they are not optimal for an ar bitrar y

match score distribution.

The median and median absolute deviation (MAD)

statistics are less sensitive to outliers as well as points

in the extreme tails of the distribution. Hence, a nor-

malization scheme using median and MAD would be

relatively robust and is given by

 med

MAD

; ð10Þ

where med

¼ median

i¼1

and MAD

¼ median

i¼1

 med

j. However, the median and the MAD esti-

mators have a low efﬁciency compared to the mean

and the standard deviation estimators, i.e., when the

score distribution is not Gaussian, median and MAD

are poor estimates of the location and scale parameters.

Therefore, this normalization technique does not pre-

serve the input score distribution and does not trans-

form the scores into a common numerical range.

Cappelli et al. [5] use a double sigmoid function for

score normalization in a multibiometric system that

combines different ﬁngerprint matchers. The normal-

ized score is given by

1þexp 2

t



ifs

< t;

1þexp 2

t



otherwise;

ð11Þ

where t is the reference operating point and a

and

denote the left and right edges of the region in

which the function is linear. The double sigmoid

function exhibits linear characteristics in the interval

(ta

, ta

). While the double sigmoid normaliza-

tion scheme transforms the scores into the [0, 1] inter-

val, it requires careful tuning of the parameters t,a

and a

to obtain good efﬁciency. Generally, t is chosen

to be some value falling in the region of overlap be-

tween the genuine and impostor score distribu tions,

and a

are set so that they correspond to the

Fusion, Score-Level

613

extent of overlap between the two distributions toward

the left and right of t, respectively. This normalization

scheme provides a linear transformation of the scores

in the region of overlap, while the scores outside this

region are transformed nonlinearly. The double sig-

moid normalization is very similar to the min–max

normalization followed by the application of a two-

quadrics (QQ) or a logistic (LG) function as suggested

by [6]. When the values of a

and a

are large, the

double sigmoid normalization closely resembles the

QQ-min–max normalization. On the other hand,

the double sigmoid normalization can be made to

approach the LG-min–max normalization by assigning

small values to a

and a

The tanh-estimators introduced by Hampel [7] are

robust and highly efﬁcient. The tanh normalization is

given by

tanh 0:01

 m



þ 1



; ð12Þ

where m

and s

are the mean and standard devia-

tion estimates, respectively, of the genuine score distri-

bution as given by Hampel estimators. Hampel

estimators are based on the following inﬂuence

(c)-function:

c uðÞ¼

u 0 juj < a,

a  signðuÞ a juj < b,

a  signðuÞ

cjuj

cb



b juj < c,

0 jujc,

ð13Þ

where

signfug¼

þ1; if u  0;

1; otherwise:



ð14Þ

The Hampel inﬂuence function reduces the inﬂu-

ence of the scores at the tails of the distribution (iden-

tiﬁed by a, b, and c) during the estimation of the

location and scale parameters. Hence, this method is

not sensitive to outliers. If many of the points that

constitute the tail of the distributions are discarded, the

estimate is robust but not efﬁcient (optimal). On the

other hand, if all the points that constitute the tail of the

distributions are considered, the estimate is not robust

but its efﬁciency increases. Therefore, the parameters a,

b, and c must be carefully chosen depending on the

amount of robustness required which in turn depends

on the amount of noise in the available training data.

Mosteller and Tukey [8] introduce the biweight

location and scale estimators that are robust and efﬁ-

cient. But, the biweight estimators are iterative in nature

(initial estimates of the biweight location and scale para-

meters are chosen, and these estimates are updated based

on the training scores), and are applicable only for Gauss-

ian data. A summary of the characteristics of the different

normalization techniques discussed in this article is

shown in Table 1. The min–max, decimal scaling

and z-score normalization schemes are efﬁcient, but

are not robust to outliers. On the other hand, the

median normalization scheme is robust but inefﬁcient.

Only the double sigmoid and tanh-estimators have

both the desired characteristics, namely, robustness

and efﬁciency.

Once the match scores output by multiple match-

ers are transformed into a common domain they can

be combined using simple fusion operators such as the

sum of scores, product of scores or order statistics (e.g.,

maximum/minimum of scores or median score).

Classifier-Based Fusion schemes

In the veriﬁcation mode of operation, the match scores

generated by the multiple matchers may be input to a

trained pattern classiﬁer, such as a neural network, in

order to determine the class label (genuine or impos-

tor). In this approach, the goal is to directly estimate

the class rather than to compute an intermediate

scalar value. Classiﬁer-based fusion schemes assume

the availability of a large representative number of

genuine and impostor scores during the training

phase of the classiﬁer when its parameters are com-

puted. The component scores do not have to be trans-

formed into a common domain prior to invoking the

classiﬁer.

Fusion, Score-Level. Table 1 Summary of score

normalization techniques.

Normalization technique Robustness Efficiency

Min–max No High

Decimal scaling No High

Z-score No High

Median and MAD Yes Moderate

Double sigmoid Yes High

Tanh-estimators Yes High

614

Fusion, Score-Level

In the biometric literature several classiﬁers have

been used to consolidate the match scores of multiple

matchers. Brunelli and Falavigna [9] use a HyperBF

network to combine matchers based on voice and face

features. Verlinde and Cholet [ 10] compare the relative

performance of three different classiﬁers, namely, the

k-Nearest Neighbor classiﬁer using vector quantiz a-

tion, the decision tree classiﬁer, and a classiﬁer based

on the logistic regression model w hile fusing the match

scores originating from three biometric matchers.

Experiments on the M2VTS database show that the

total error rate (sum of the false accept and false reject

rates) of the multimodal system is an order of magni-

tude less than that of the individual matchers. Chatzis

et al. [11] use classical k-means clustering, fuzzy clus-

tering and median radial basis function (MRBF) algo-

rithms for fusion at the match score level. The

proposed system combines the output of ﬁve different

face and voice matchers. Each matcher provides a

match score and a quality metric indicating the reli-

ability of the match score. These values are concate-

nated to form a ten-dimensional vector that is input to

the classiﬁers. Ben-Yacoub et al. [12] evaluate a num-

ber of classiﬁcation schemes for fusion including sup-

port vector machine (SVM) with polynomial kernels,

SVM with Gaussian kernels, C4.5 decision trees, mul-

tilayer perceptron, Fisher linear discriminant, and

Bayesian classiﬁer. Experimental evaluations on the

XM2VTS database consisting of 295 subjects suggest

the beneﬁt of score level fusion. Bigun et al. [13]

propose a novel algorithm based on the Bayesian clas-

siﬁer that takes into account the estimated accuracy of

the individual classiﬁers (i.e., matchers) during the

fusion process. Sanderson and Paliwal [14] use a sup-

port vector machine (SVM) to combine the scores of

face and speech experts. In order to address noisy

input, they design structurally noise-resistant classi-

ﬁers based on a piece-wise linear classiﬁer and a mod-

iﬁed Bayesian classiﬁer.

Summary

In a multibiometric system, fusion at the score level

offers the best tradeoff between amount of information

that is available and ease of fusion. Hence, score level

fusion is typically adopted by most multibiometric

systems. Although a wide variety of score level fusion

techniques have been proposed in the literature,

these can be grouped into three main categories, viz.,

density-based, transformation-based and classiﬁer-

based schemes. The performance of each scheme

depends on the amount and quality of the available

training data. If a large number of match scores is

available for training the fusion module, then density-

based approaches such as the likelihood ratio test can

be used. Estimating the genuine and impostor distri-

butions may not always be feasible due to the limited

number of training samples that are available. In such

cases, transformation-based schemes are a viable alter-

native. The nonhomogeneity of the match scores pre-

sented by the diffe rent matc hers raises a number of

challenges. Suitable score normali zation schemes are

essential in order to transform these matc h scores into

a comparable domain. The sum of scores fusion meth-

od with simple score normalization (such as min–

max) represents a commonly used transformation-

based scheme. Classiﬁcation-based fusion schemes

consolidate the outputs of different matchers into a

single vector of scores which is input to a trained

classiﬁer. The classiﬁer then determines if this vector

belongs to the ‘‘genuine’’ or ‘‘impostor’’ class.

Related Entries

▶ Fusion, Qualit y-Based

▶ Fusion, User-Speciﬁc

▶ Multibiometrics

References

1. Ross, A., Nandakumar, K., Jain, A.K.: Handbook of Multibio-

metrics. 1st edn. Springer, New York, USA (2006)

2. Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classiﬁcation. Wiley,

New York (2001)

3. Kittler, J., Hatef, M., Duin, R.P., Matas, J.G.: On combining

classiﬁers. IEEE Trans. Pattern Analy. Mach. Intell. 20, 226–239

(1998)

4. Dass, S.C., Nandakumar, K., Jain, A.K.: A principled approach to

score level fusion in multimodal biometric systems. In: Proceed-

ings of Fifth International Conference on Audio- and Video-based

Biometric Person Authentication (AVBPA), pp. 1049–1058. Rye

Brook, USA (2005)

5. Cappelli, R., Maio, D., Maltoni, D.: Combinin g ﬁngerpri nt

classiﬁers. In: Proceedings of First Internat ional Workshop

on Multipl e Classiﬁer Systems, pp. 351–3 61. Cagliari, Italy

(2000)

Fusion, Score-Level

615

6. Snelick, R., Uludag, U., Mink, A., Indovina, M., Jain, A.K.: Large

Scale evaluation of multimodal biometric authentication using

state-of-the-art systems. IEEE Trans. Pattern Analy. Mach. Intell.

27, 450–455 (2005)

7. Hampel, F.R., Rousseeuw, P.J., Ronchetti, E.M., Stahel, W.A.:

Robust Statistics: The Approach Based on Inﬂuence Functions.

Wiley, New York (1986)

8. Mosteller, F., Tukey, J.W.: Data Analysis and Regression: A Sec-

ond Course in Statistics. Addison-Wesley, Reading, MA, USA

(1977)

9. Brunelli, R., Falavigna, D.: Person Identiﬁcation using multiple

cues. IEEE Trans. Pattern Analy. Mach. Intell. 17, 955–966 (1995)

10. Verlinde, P., Cholet, G.: Comparing decision fusion paradigms

using k-NN based classiﬁers, decision trees and logistic regres-

sion in a multi-modal identity veriﬁcation application. In: Pro-

ceedings of Second International Conference on Audio- and

Video-Based Biometric Person Authentication (AVBPA), pp.

188–193. Washington D.C., USA (1999)

11. Chatzis, V., Bors, A.G., Pitas, I.: Multimodal decision-level fu-

sion for person authentication. IEEE Trans. Syst. Man Cybernet.

Part A: Syst. Humans 29, 674–681 (1999)

12. Ben-Yacoub, S., Abdeljaoued, Y., Mayoraz, E.: Fusion of face and

speech data for person identity veriﬁcation. IEEE Trans. Neural

Networks 10, 1065–1075 (1999)

13. Bigun, E.S., Bigun, J., Duc, B., Fischer, S.: Expert Conciliation for

multimodal person authentication systems using bayesian sta-

tistics. In: First International Conference on Audio- and Video-

based Biometric Person Authentication (AVBPA), pp. 291–300.

Crans-Montana, Switzerland (1997)

14. Sanderson, C., Paliwal, K.K.: Information Fusion and Person

Veriﬁcation Using Speech and Face Information. Research

Paper IDIAP-RR 02-33, IDIAP (2002)

Fusion, Sensor-Level

AFZEL NOORE,RICHA SINGH,MAYANK VASTA

West Virginia University, Morgantown, WV, USA

Synonyms

Fusion, data level; Fusion, image level

Definition

Sensor level fusion combines raw biometric informa-

tion that can account for inter-class and intra-class

variability and facilitate decision making based on the

fused raw information. A typical sensor level fusion

algorithm ﬁrst integrates raw biometric data either

obtained from different viewpoints (for example,

mosaicing several ﬁngerprint impressions) or obtained

from different sensors (for example, multimodal bio-

metric images). The integrated data is then processed

and discriminatory biometric features are extracted for

matching. This level of fusion can be operated in both

veriﬁcation and

▶ identiﬁcation modes. Few examples

of sensor level fusion are: ﬁngerprint mosaicing, multi-

spectral face image fusion, and multimodal biometric

image fusion.

Introduction

The concept of biometric information fusion is moti-

vated from classical multi-classiﬁer systems that com-

bine information from different sources and represent

using a single entity. Performance driven systems that

use multiple biometric characteristics are known in

multibiometric system [1]. These systems have several

advantages over unimodal biometric systems such as

tolerance to noise and malfunction, universality, and

improved accuracy. Multibiometric systems are broadly

classiﬁed into ﬁve levels of fusion.

1. Sensor level fusion. Raw data obtained directly from

the sensors are fused without any feature extraction

and represented as a single unit. This level of fusion

is also known as data level fusion or image level

fusion (for image based biometrics).

2. Feature level fusion . Data obtained from different

sensors are ﬁrst subjected to feature extraction

algorithms and the feature sets are combined to

generate a new feature vector which is subsequently

used for recognition.

3. Match score level fusion. Features extracted from

individual biometric modalities are ﬁrst matched

to compute the corresponding match scores. Match

scores obtained from different biometric systems

are then combined to generate a fused match score.

4. Decision level fusion. Decisions of individual

biometric classiﬁers are fused to compute a com-

bined decision. This level of fusion is also known

as abstract level fusion because it is used when there

is access to only decisions from individual classiﬁer’s.

5. Rank level fusion. With identiﬁcation systems, rank

level fusion involves combining identiﬁcation ranks

obtained from multiple unimodal biometrics. The

output of rank level fusion is a consolidated rank

that is used for ﬁnal decision.

616

Fusion, Sensor-Level

This article focuses on sensor level fusion and pro-

vides a comprehensive overview of the methodologies

involved. In this level of fusion, ﬁrst the raw data

obtained from the sensors are combined to generate a

fused data. An application oriented feature extraction

algorithm is then used to compute the features from

the fused data and matching is performed. Figure 1

illustrates the basic concept of sensor level fusion.

Sensor level fusion can be broadly classiﬁed into

three categories: (1) single sensor multi-samples, (2)

multi-sensor, and (3) multimodal. This article is

organized to accentuate various algorithms proposed

in each fusion category.

Sensor Level Fusion: Single Sensor Multi-

Samples

In these systems, multiple samples of a single bio metric

modality are acquired using a single sensor and the

information is combined to account for variations that

can occur in a biometric modality. For example, as

shown in Fig. 2, different proﬁles of a face image can

be combined to obtain a fused representation of face

image that can address the challenges due to pose

variations [2]. In this categ ory of sensor fusion,

image

▶ mosaicing techniques are used for integrati ng

information obtained from several impressions or v iew

points, to augment the biometric content and to en-

hance the veriﬁcation/identiﬁcation performance.

Singh et al. [2] describe the concept of mosaicing in

biometrics as an exercise in information fusion when

multiple images of a subject’s biometric information

are fused into a single entity in the image domain

itself. Therefore, this could be viewed as fusion at the

sensor level.

Mosaicing was ﬁrst introduced in biometrics by

Ratha et al. [3]. A rolled ﬁngerprint image is generated

from several partial ﬁngerprint images using segmen-

tation and blending algorithms assuming that the

partial ﬁngerprints are spatially registered. The per-

formance of this ﬁngerprint mosaicing algorithm is

evaluated using different blending algorithms. The

mosaicing algorithm generates rolled ﬁngerprint image

that is very close to the ground truth and improves

the minutiae count that is useful for recognition. Fur-

ther, Jain and Ross [4] proposed the use of iterative

closest point algorithm to seamlessly register the ridges

of two ﬁngerprint images and to generate a composite

ﬁngerprint image. Recently, Ross et al. [5] employed

thin-plate splines ( TPS) to model the non-linear

deformation in ﬁngerprint imag es and integrate it

Fusion, Sensor-Level. Figure 1 Basic concept of sensor level fusion.

Fusion, Sensor-Level. Figure 2 Combining profile and frontal face images using mosaicing technique. (a) Proﬁle and

frontal face images and (b) Mosaiced face image.

Fusion, Sensor-Level

617

in the mosaicing process. This algorithm ﬁrst aligns

two ﬁngerprint images using coarse alignment (afﬁne

model) followed by TPS based ﬁne alignment. Once

the ﬁngerprint images are registered, a mosaiced ﬁn-

gerprint image is obtained by applying simple pixel

averaging based blending method.

Mosaicing has also been applied to face biometrics.

Yang et al. [ 6] describe an algorithm to create pano-

ramic face mosaics. The acquisition system consists of

ﬁve cameras that simultaneously obtains ﬁve different

views of a subject’s face. Based on the manually marked

control points, the algorithm uses a series of linear

transformations and smoothing operations on com-

ponent images to generate a face mosaic. Unlike ﬁn-

gerprint mosaicing, face mosaicing requires speciﬁc

feature extraction algorithm. Two different schemes

to represent the panoramic image were proposed: one

in the spatial domain and another in the frequency

domain. Experimental evaluation on a database of

12 individuals shows that the face mosaicing algorithm

improves identiﬁcation accuracy in both the spatial

and frequency domains. In [7], Liu and Chen describe

a face mosaicing algorithm in which the human head is

approximated with a 3D ellipsoidal model. The face, at

a certain pose, is v iewed as a 2D projection of this 3D

ellipsoid. All 2D face images of a subject are projected

onto this ellipsoid via geometrical mapping to form a

texture map which is represented by an array of local

patches. Matching is accomplished by adopting a prob-

abilistic model to compute the distance of patches

from an input face image. An identiﬁcation accuracy

of 90% on different databases has been reported.

In [2], Singh et al. proposed a face mosaicing algo-

rithm that can perform mosaicing in v isible spectrum

domain as well as in short wave infrared domain.

The algorithm ﬁrst registers the component face

images using two stage registration algorithm and

then a face mosaic is generated using multi-resolution

splines based blending algorithm. Facial features are

encoded using a generic feedforward hierarchical

model-based feature extraction algorithm that extrac ts

local facial features using the fundamentals of a

biological visual system. Experiments conducted

on three different face databases indicate that the pro-

posed face mosaicing algorithm offers signiﬁcant ben-

eﬁts by accounting for pose variations that are

commonly observed in face images. Moreover, the

mosaicing algorithm requires less time for matching

compared to the score level fusion and also reduces the

memory requirements.

Sensor Level Fusion: Multi-Sensors

In this category of sensor level fusion, multiple samples

of a single biometric modality are obtained using

multiple sensors and the information is combined

such that the fused multi-sensor information improves

the recognition performance. In general, the informa-

tion obtained from multiple sensors are complementary

to each other and can account for the intra-class varia-

bility. For example, as shown in Fig. 3, multi-spectral

face images obtained using visible spectrum and infra-

red sensors can be fused to minimize the intra-class

variations due to illumination and expression.

Multi-spectral face image fusion is the classical

model for this level of fusion. Face recognition algo-

rithms generally use visible spectrum images for rec-

ognition because the reﬂectance property yields a

Fusion, Sensor-Level. Figure 3 Multi-spectral face image fusion. (a) Visible and infrared spectrum images and

(b) Fused image.

618

Fusion, Sensor-Level

clear representation of facial features to differentiate

between two individuals. However, visible spectrum

images also possess several other properties which

affect the performance of recognition algorithms. For

example, changes in lighting affect the representation

of visible spectrum images and can inﬂuence feature

extraction. Other variations in facial appearance such

as hairs, wrinkles, and expression are also evident in

visible spectrum images and these variations increase

the false rejection rate of face recognition algorithms.

To address the challenges posed by visible spectrum

images, researchers have used infrared images for face

recognition [8]. Among all infrared spectrum images,

long wave infrared (LWIR) images possess several

properties that are complementary to visible images.

Visible spectrum captures the electromagnetic energy

in the range 0.4–0.7mm, whereas long wave infrared or

thermal images are captured in the range of 8–12mm.

Thermal images represent the heat pattern of the object

and are invariant to illumination and expression. Face

images captured in long wave infrared spectrum have

less intra-class variation and help to reduce the false

rejection rate of recognition algorithms. These proper-

ties of long wave infrared and visible images can be

combined to improve the performance of face recogni-

tion algorithms.

In literature, researchers have proposed several

multi-spectral face image fusion algorith ms [8]. Bebis

et al. [9] proposed an image fusion algorithm in wave-

let domain using genetic algorithm. In this algorithm,

multi-spectral face images are ﬁrst transformed into

wavelet domain and a multiresolution representation

is obtained. Then, a genetic algorithm is used to select

the most appropriate wavelet coefﬁcients at pixel level.

Finally, inverse

▶ wavelet transform is applied to gen-

erate a fused face image and Eigenface based algorithm

is used fo r feature extraction and matching. The genetic

fusion algorithm suffers from making a good choice of

ﬁtness function. Kong et al. [10] proposed a wavelet

based multi-spectral face image fusion algorithm in

which the visible and infrared spectrum images are

ﬁrst registered using afﬁne transformation. An empiri-

cal weighting scheme is then applied on the registered

multi-spectral face images in wavelet domain to obtain

the composite face image. Although the algorithm

is straightforward, the generic empirical weig hting

scheme is not sufﬁcient to address the inter-class and

intra-class variability in face images. Recently, Singh

et al. [11] proposed a 2n-granular support vector

machine (2n-GSVM ) based multi-spectral face image

fusion algorithm. This algorithm ﬁrst registers multi-

spectral face images using mutual information based

registration algorithm. Then, a 2n-GSVM learning

scheme is invoked in wavelet domain to learn the

properties of the multi-spectral face images at different

resolution and granularity levels, determine optimal

information and combine them to generate a fused

image. Finally, texture features are extracted from the

fused image for recognition. Experimental results show

that 2n-GSVM based fusion algorithm can address the

challenges due to illumination, expression, and occlu-

sion variations. This algorithm provides improved

veriﬁcation accuracy (>94% ) compared to other

image fusion schemes.

Another example of sensor level fusio n with multi-

ple sensors is fusing 2D and 3D facial information.

Lu et al. [12] describe a semi-automatic sensor level

fusion algorithm that integrates range and texture fea-

tures for improved face recognition performance.

Combining 3D shape information with registered 2D

texture information using iterative closest point algo-

rithm improves the face identiﬁcation performance.

The authors report that the algorithm is robust to

arbitrary view, lighting, and facial appearance. How-

ever, the algorithm is computationally expensive and

suffers due to non-rigid variations.

Sensor Level Fusion: Multimodal

In most of the multimodal biometric systems, such as

bimodal system with face and ﬁngerprint, fusion is

performed at match score level or decision level. Very

limited research is undertaken to perform sensor level

fusion in a multimodal system. In this category of

sensor level fusio n, multimodal biometric images are

fused to address issues such as universality, memory

storage, small sample size recognition, and recognition

performance. Further, an efﬁcient sensor level fusion

algorithm has advantages due to the availability of

fused raw information from where the representative

composite biometric features can be extracted and

used for matching. The main challenge lies in develop-

ing fusion algorithm that can account for inter-class

and intra-c lass variability in multimodal biometric

images. An example of multimodal biometric image

fusion is shown in Fig. 4.

Fusion, Sensor-Level

619