Research Project Homepage: Statistical Inference Methods for Understanding Human Identification Systems PIs: Ross Beveridge & Bruce Draper Computer Science, Colorado State University

Project Objective:

The Human ID Program will further develop the science of human identification. Understanding how to use modern statistical methods to answer fundamental questions about human identification algorithms is critical to this new science. We will establish a standard set of statistical methods for performing evaluation within the context of Human ID and illustrate their use in empirical studies that compare top performing face recognition algorithms from the 1996/97 FERET evaluation. A suite of four face recognition algorithms, along with standard image preprocessing tools and performance analysis tools will be developed and distributed in the CSU Face Identification Evaluation System available through the algorithms link on the Evaluation of Face Recognition Algorithms web site.

Project Approach:

Statistical methods applied to understanding human identification algorithms typically answer three kinds of question:

1. Is the difference between two algorithms statistically significant?
2. Is the difference between two algorithms significant relative to variation induced by an internal or external factor?
3. What makes some human subjects, or some images, easier or harder to recognize?

To illustrate the first question, assume algorithms A and B are compared on 1195 people and correctly identify 925 and 886 people respectively, may we conclude this difference is significant? What assumptions are made when concluding the difference is or is not significant? Answering the second question typically demands a more involved experimental protocol than does the first. However, the payoff can be valuable information about the factors contributing to differences in performance,in particular, information that can be used to draw inferences about the relative importance of algorithm design choices as well as the relative importance of external domain factors. The third question gets to the heart of what makes some subjects harder to recognize. For example, using a standard algorithm, are men easier to recognize than women? Our work is developing new protocols for addressing this kind of question.

In Phase I of the Human ID Program we have implemented four human face recognition algorithms along with code to run experiments. All of this code is included in the CSU Face Identification Evaluation System. Over ten papers have been written describing the work and are available throught the papers section the Evaluation of Face Recognition Algorithms web site. Work continues on several fronts. Recent availability of larger face data sets are permitting us to run more comprehensive studies on the standard suite of four algorithms. Considerable attention is also being focused on how to better define and study the factors that make individual subjects harder or easier to recognize.

Recent Accomplishments:

Version 5.0 of the CSU Face Identification Evaluation System was released on May 1, 2003. It includes everything necessary to replicate a significant portion of the FERET 1996/97 evaluation. A researcher who downloads the CSU system and obtains the FERET data from NIST can invoke a single Unix script that will preprocess the FERET face images, run four of the algorithms included in the FERET evaluations, and generate performance summaries called cumulative match curves. This provides researchers with a rich baseline against which to measure improvement.

We have concluded a study of human subject covariates using a standard Principal Components Analysis (PCA) algorithm and 1,072 human subjects from the FERET data. Eleven covariates were considered, including age, gender and race. The paper A Statistical Assessment of Subject Factors in the PCA Recognition of Human Faces summarizes the study and its findings. Of note, subjects wearing glasses, so long as they kept the glasses on, were more easily recognized than subjects without glasses. Also, while other studies have observed algorithms doing better on men than women, our study found no statistically significant gender effect.

A full re-implementation of the USC Elastic Bunch Graph Matching Algorithm was released with version 5.0 of the CSU Face Identification Evaluation System in May 2003. David Bolme's masters thesis includes a detailed description of the algorithm along with a series of experiments illustrating internal design tradeoffs within the algorithm. Two related finding are particularly interesting. First, the facial landmarks are not localized as reliably as we initially expected. Second, it appears recognition performance suffers less from this imprecise localization than we initially expected.

A full re-implementation of the MIT Bayesian Intrapersonal/Extrapersonal Classifier has undergone steady refinements and a new implemenation of the algorithm was included with version 5.0 of the CSU Face Identification Evaluation System in May. One finding of note: a careful analysis of the maximum a-posteriori probability similarity score has revealed that the prior intrapersonal and extrapersonal class proabilities are not needed. The explicit mention of how these prior probabilites were chosen in the original presentation of this algorithm suggests the authors were not aware of this simplification. Another important finding is that using standard FERET training image sets, the complex maximum a-posteriori probability similarity score offers no improvement relative to the much simpler maximum likelihood similarity score. This work is summarized in the paper An Implementation and Study of the Moghaddam and Pentland Intrapersonal/Extrapersonal Image Difference Face Recognition Algorithm.

A thorough comparison of PCA and ICA face recognition algorithms has clarified a critical difference between two common ways of applying independent component analysis to human face recognition. One way yields a relatively high-quality face recognition algorithm and the other does not. Much of the prior literature has been vague relative to the critical distinction. This work is presented in the paper "Recognizing Faces with PCA and ICA" will appear shortly in Vision and Image Understanding.

Current Plan:

Working jointly with NIST, our work on human subject covariates is being expanded to look at much larger data sets. We are also actively studying whether there are image covariates, i.e., simple measurable properties of face images, that enable us to predict when images will be harder or easier to recognize. Finally, basic research into better ways of structuring studies continues with an emphasis on how to move between simpler subject and image specific performance measures to measures defined over large sets of human subjects: for example recognition rate.

Technology Transition:

The Evaluation of Face Recognition Algorithms web site is a common resource for baseline algorithms and statistical evaluation protocols. The CSU Face Identification Evaluation System distributed through our site has received increasing attention, with over 2,500 downloads of the system during the period from November 2002 through July 2003.

Last Revised on Wednesday, August 13, 2003