Paper

David S. Bolme, Michelle Mills Strout, and J. Ross Beveridge. FacePerf: Benchmarks for Face Recognition Algorithms. IEEE International Symposium on Workload Characterization. (Boston, MA, USA). Sept 2007.

Download paper in PDF Format (400KB)

Abstract

In this paper we present a collection of C and C++ biometric performance benchmark algorithms called FacePerf.  The benchmark includes three different face recognition algorithms that are historically important to the face recognition community: Haar-based face detection, Principal Components Analysis, and Elastic Bunch Graph Matching.  The algorithms are fast enough to be useful in realtime systems; however, improving performance would allow the algorithms to process more images or search larger face databases.  Bottlenecks for each phase in the algorithms have been identified. A cosine approximation was able to reduce the execution time of the Elastic Bunch Graph Matching implementation by 32%.

Related Sites

OpenCV

CSU Evaluation of Face Recognition Alogrithms

Software Download

faceperf.1.0 (17MB)

Requirements

This distribution was compiled and tested using the standard GNU build system and the gcc complier.  Most linux distributions should come with all the software need to compile and test the system.  ImageMagick (convert) and bzip2 (bzcat) is required to convert the CMU database and the FERET database to pgm format.  Python is required for the accuracy evaluation of the face detection algorithm but not required to produce benchmark timings.

Please make sure the following command can be run from the command line before compiling the system.

1. •bash
   

2. •make
   

3. •gcc and g++
   

4. •convert (ImageMagick)
   

5. •bzcat
   

Most linux distributions include these utilities.

MacOS 10.4 or later:  Running the benchmark on a Mac requires the Macos X developer tools are installed.  ImageMagick is available through darwinports, fink, or the ImageMagick website.

Quick Start

After downloading, extract the tar file:

> tar xzf  csuFaceIdBenchmark.1.0.0rc1.tgz

> cd csuFaceIdBenchmark

Run make in the top level directory.  This will compile the CSU programs and then recursively call configure and make on OpenCV.  The benchmarks could take 5 to 20 minutes to compile:

> make

If make fails due to OpenCV errors you can try building just the CSU algorithms.

> make stable

The distribution includes a small dataset (csuScrapShots) that was compiled from CSU yearbook dated prior to 1927 who’s copyright had expired.  The benchmarks can be tested on this data set by running the following script:

> ./RuntimeBenchmarksScraps.sh

This script will run each benchmark three times.  The execution times for each run will be recorded in the file: runtime.txt.

Using some larger datasets

The CSU Scrap Shots dataset contains 173 low quality that are not very representative of actual face recognition problems.  We used the FERET and CMU Face Detection datasets in our profiling experiments.  Unfortunately, we cannot redistribute these datasets due to copyright and privacy issues, but they can both be obtained for free. The full benchmark test is performed on 640 images from the FERET dataset and 42 images from the CMU dataset.

FERET grayscale database

The primary dataset used with the CSU face identification algorithms is the FERET grayscale database.  This database can be obtained for free by contacting NIST.  Note: the color database contains the grayscale images as bonus data.

Once you have obtained the data it needs to be unpacked into the csuFaceIdBenchmark/data directory.  The files should end up with paths such as:

csuFaceIdBenchmark/data/FERET/source/pgm/00001fa010_930831.pgm

csuFaceIdBenchmark/data/FERET/source/pgm/00001fb010_930831.pgm

csuFaceIdBenchmark/data/FERET/source/pgm/00002fa001_931230.pgm

csuFaceIdBenchmark/data/FERET/source/pgm/00002fa001d_931230.pgm

csuFaceIdBenchmark/data/FERET/source/pgm/00002fa010_930831.pgm

A script has been included in the distribution that will help to extract the from the CD.  To use this script, insert the FERET DVD 2 and run the following command:

> scripts/Process_FERET_DVD.sh /Volumes/CDROM

“/Volumes/CDROM” should be replaced with the top level directory for the FERET DVD.

CMU Face Detection Database

The CMU face detection database is a standard face detection dataset.  The imagery should also be installed in the csuFaceIdBenchmark/data directory.  The paths should look like this:

csuFaceIdBenchmark/data/images//newtest/addams-family.gif

csuFaceIdBenchmark/data/images//newtest/aeon1a.gif

csuFaceIdBenchmark/data/images//newtest/audrey1.gif

csuFaceIdBenchmark/data/images//newtest/audrey2.gif

csuFaceIdBenchmark/data/images//newtest/audrybt1.gif

To install this data run the following commands.

> mkdir -p data/images

> cd data/images

> tar zxvf images.tgz

Running the larger benchmarks

After installing the large dataset, make sure the system has been compiled by running make:

> make

If the build completes successfully you can now run the bench mark test.

> ./RuntimeBenchmarks.sh

This script will run each executable in the suite 3 times.  The results will be saved to a file called runtime.txt.

Common Problems and Solutions

CSU Only - Some users have encountered trouble compiling or running the OpenCV face detector on some systems.  If this is the case you can compile only the CSU algorithms using: make stable

Cygwin - The CSU algorithms will compile and run under cygwin.  We have encountered dependency issues with OpenCV.  (see also CSU Only above)

Microsoft Compliers - All algorithms have been compiled using Microsoft visual studio.  Porting the benchmarks should not be a big issue.  Microsoft does not properly handle some unix specific headers in the CSU algorithms.  Minor modifications are needed to correct this problem and compile the code.  The OpenCV website contains help for OpenCV related issues.