Skip to main content

Example on how to use the scalable implementation of PLDA and how to reproduce experiments of the article

Project description

This package contains scripts that show how to use the implementation of the scalable formulation of Probabilistic Linear Discriminant Analysis (PLDA), integrated into Bob, as well as how to reproduce experiments of the article mentioned below. It is implemented and maintained via github.

References

If you use this package and/or its results, please cite the following publications:

  1. The original paper with the scalable formulation of PLDA explained in details:

    @article{ElShafey_TPAMI_2013,
      author = {El Shafey, Laurent and McCool, Chris and Wallace, Roy and Marcel, S{\'{e}}bastien},
      title = {A Scalable Formulation of Probabilistic Linear Discriminant Analysis: Applied to Face Recognition},
      year = {2013},
      month = jul,
      journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
      volume = {35},
      number = {7},
      pages = {1788-1794},
      doi = {10.1109/TPAMI.2013.38},
      pdf = {http://publications.idiap.ch/downloads/papers/2013/ElShafey_TPAMI_2013.pdf}
    }
  2. Bob as the core framework used to run the experiments:

    @inproceedings{Anjos_ACMMM_2012,
      author = {A. Anjos and L. El Shafey and R. Wallace and M. G\"unther and C. McCool and S. Marcel},
      title = {Bob: a free signal processing and machine learning toolbox for researchers},
      year = {2012},
      month = oct,
      booktitle = {20th ACM Conference on Multimedia Systems (ACMMM), Nara, Japan},
      publisher = {ACM Press},
      url = {http://publications.idiap.ch/downloads/papers/2012/Anjos_Bob_ACMMM12.pdf},
    }
  3. If you decide to use the Multi-PIE database, you should mention the following paper, where it is introduced:

    @article{Gross_IVC_2010,
      author = {Gross, Ralph and Matthews, Iain and Cohn, Jeffrey and Kanade, Takeo and Baker, Simon},
      title = {Multi-PIE},
      journal = {Image and Vision Computing},
      year = {2010},
      month = may,
      volume = {28},
      number = {5},
      issn = {0262-8856},
      pages = {807--813},
      numpages = {7},
      doi = {10.1016/j.imavis.2009.08.002},
      url = {http://dx.doi.org/10.1016/j.imavis.2009.08.002},
      acmid = {1747071},
    }
  4. If you only use the Multi-PIE annotations, you should cite the following paper since annotations were made for the experiments of this work:

    @article{ElShafey_TPAMI_2013,
      author = {El Shafey, Laurent and McCool, Chris and Wallace, Roy and Marcel, S{\'{e}}bastien},
      title = {A Scalable Formulation of Probabilistic Linear Discriminant Analysis: Applied to Face Recognition},
      year = {2013},
      month = jul,
      journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
      volume = {35},
      number = {7},
      pages = {1788-1794},
      doi = {10.1109/TPAMI.2013.38},
      pdf = {http://publications.idiap.ch/downloads/papers/2013/ElShafey_TPAMI_2013.pdf}
    }
  5. If you decide to use the Labeled Faces in the Wild (LFW) database, you should mention the following paper, where it is introduced:

    @TechReport{LFWTech,
      author = {Gary B. Huang and Manu Ramesh and Tamara Berg and Erik Learned-Miller},
      title = {Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments},
      institution = {University of Massachusetts, Amherst},
      year = {2007},
      number = {07-49},
      month =  oct,
    }

Installation

There are two options you can follow to get this package installed and operational on your computer: you can use automatic installers like pip (or easy_install) or manually download, unpack and use zc.buildout to create a virtual work environment just for this package. In both cases, you must first install Bob (>= 1.2.0), whose installation process is described in the user guide.

Using an automatic installer

Using pip is the easiest (shell commands are marked with a $ signal):

$ pip install xbob.paper.tpami2013

You can also do the same with easy_install:

$ easy_install xbob.paper.tpami2013

This will download and install this package plus any other required dependencies. It will also verify if the version of Bob you have installed is compatible.

This scheme works well with virtual environments by virtualenv or if you have root access to your machine. Otherwise, we recommend you use the next option.

Using zc.buildout

Download the latest version of this package from PyPI and unpack it in your working area:

$ wget http://pypi.python.org/packages/source/x/xbob.paper.tpami2013/xbob.paper.tpami2013-1.0.0.zip
$ unzip xbob.paper.tpami2013-1.0.0.zip
$ cd xbob.paper.tpami2013-1.0.0

The installation of the toolkit itself uses buildout. You don’t need to understand its inner workings to use this package. Here is a recipe to get you started:

$ python bootstrap.py
$ ./bin/buildout

These two commands should download and install all non-installed dependencies and get you a fully operational test and development environment.

PLDA tutorial

The following example consists of a simple script, that makes use of Probabilistic Linear Discriminant Analysis (PLDA) modeling on the Fisher’s iris dataset. It performs the following tasks:

  1. Train a PLDA model using the first two classes of the dataset

  2. Enroll a class-specific PLDA model for the third class of the dataset

  3. Compute (verification) scores for both positive and negative samples

  4. Plot the distribution of the scores and save it into a file

To run this simple example, you just need to execute the following command:

$ ./bin/plda_example_iris.py --output-img plda_example_iris.png

Reproducing experiments

It is currently possible to reproduce all the experiments of the article on both Labeled Faces in the Wild and Multi-PIE using the PLDA algorithm. In particular, the value of the accuracy reported in Table 2, the Figure 2 and the HTER reported on Table 3 can be easily reproduced, by following the steps described below.

Be aware that all the scripts provide several optional arguments that are very useful if you wish to use your own features or your own parameters.

Keep in mind that the results published in the paper were obtained with a pre-release of Bob (older than 1.0.0). You might hence observe slight differences when running the scripts with Bob 1.2.0.

Note for Grid Users

At Idiap, we use the powerful Sun Grid Engine (SGE) to parallelize our job submissions as much as we can. At the Biometrics group, we have developed a little toolbox gridtk that can submit and manage jobs at the Idiap computing grid through SGE.

The following sections will explain how to reproduce the paper results in single (non-gridified) jobs. If you are at Idiap, you could run the following commands on the SGE infrastructure, by applying the ‘–grid’ flag to any command. This may also work on other locations with an SGE infrastructure, but will likely require some configuration changes in the gridtk utility.

Labeled Faces in the Wild dataset

The experiments of this section are performed on the LFW (Labeled Faces in the Wild) protocol. The features are publicly available and will be automatically downloaded from this webpage if you follow the instruction below. They were extracted on the LFW images aligned with the funneling algorithm.

Getting the features and converting them into HDF5 format

The following command will download a tarball with the SIFT features, extract the content of the archive and convert the features into a suitable HDF5 format for Bob:

$ ./bin/lfw_features.py --output-dir /PATH/TO/LFW/DATABASE/

PCA+PLDA toolchain on LFW

Once the features have been extracted, the dimensionality is reduced using Principal Component Analysis (PCA), before applying PLDA modeling. These steps are combined in the following script, that will run the PCA+PLDA toolchain on the specified protocol:

$ ./bin/toolchain_pcaplda.py --features-dir /PATH/TO/LFW/DATABASE/lfw_funneled --protocol view1 --output-dir /PATH/TO/LFW/OUTPUT_DIR/

To report the final performance on LFW, it is required to run 10 experiments on view 2 in a leave-one-out cross validation scheme. We provide the following script for this purpose:

$ ./bin/experiment_pcaplda_lfw.py -features-dir /PATH/TO/LFW/DATABASE/lfw_funneled --output-dir /PATH/TO/LFW/OUTPUT_DIR/

Summarizing the results as in Table 2

Once the previous experiments have successfully completed, you can use the following script to plot Table 2, that will estimate the mean accuracy as well as the standard error of the mean on the 10 experiments of LFW view2:

$ ./bin/plot_table2.py --output-dir /PATH/TO/LFW/OUTPUT_DIR/

Multi-PIE dataset

The experiments of this section are performed on the U protocol of the Multi-PIE dataset. The filelists associated with this protocol can be found on this website.

Getting the data

You first need to buy and download the Multi-PIE database:

http://multipie.org/

and to download the annotations available here:

http://www.idiap.ch/resource/biometric/

Feature extraction

The following command will extract Local Binary Patters (LBP) histograms features. You should set the paths to the data according to your own environment:

$ ./bin/lbph_features.py --image-dir /PATH/TO/MULTIPIE/IMAGES --annotation-dir /PATH/TO/MULTIPIE/ANNOTATIONS --output-dir /PATH/TO/MULTIPIE/OUTPUT_DIR/

Dimensionality reduction

Once the features has been extracted, they are projected into a lower dimensional subspace using Principal Component Analysis (PCA):

$ ./bin/pca_features.py --output-dir /PATH/TO/MULTIPIE/OUTPUT_DIR/

Proposed system: PLDA modeling and scoring

PLDA is then applied on the dimensionality reduced features.

This involves three different steps:
  1. Training

  2. Model enrollment

  3. Scoring

The following command will perform all these steps:

$ ./bin/toolchain_plda.py --output-dir /PATH/TO/MULTIPIE/OUTPUT_DIR/

Then, the HTER on the evaluation set can be obtained using the evaluation script from the bob library as follows:

$ ./bin/bob_compute_perf.py -d /PATH/TO/MULTIPIE/OUTPUT_DIR/U/plda/scores/scores-dev -t /PATH/TO/MULTIPIE/OUTPUT_DIR/U/plda/scores/scores-eval -x

The HTER on the evaluation set, when using the EER on the development set as the criterion for the threshold, corresponds to the PLDA value reported on Table 3 of the article mentioned above.

If you want to reproduce the Figure 2 of the article, you can run the following commands (instead of the previous one):

$ ./bin/experiment_plda_subworld.py --output-dir /PATH/TO/MULTIPIE/OUTPUT_DIR/
$ ./bin/plot_figure2.py --output-dir /PATH/TO/MULTIPIE/OUTPUT_DIR/

The previous commands will run the PLDA toolchain several times for a varying number of training samples. Please note, that this will require a lot of time to complete (a bit less than two days on a recent workstation such as one with an Intel Core i7 CPU).

Then, the value of the HTER on Table 3 of the article (for the PLDA system) corresponds to the one, where the full training set is used, and might similarly be obtained as follows:

$ ./bin/bob_compute_perf.py -d /PATH/TO/MULTIPIE/OUTPUT_DIR/U/plda_subworld_76/scores/scores-dev -t /PATH/TO/MULTIPIE/OUTPUT_DIR/U/plda_subworld_76/scores/scores-eval -x

Baseline 1: PCA on the LBP histograms

The LBP histogram features were used in combination with the PCA classification technique (commonly called Eigenfaces in the face recognition literature).

This involves three different steps:
  1. PCA subspace training

  2. Model enrollment

  3. Scoring (with an Euclidean distance)

The following command will perform all these steps:

$ ./bin/toolchain_pca.py --n-outputs 2048 --output-dir /PATH/TO/MULTIPIE/OUTPUT_DIR/

Then, the HTER on the evaluation set can be obtained using the evaluation script from the bob library as follows:

$ ./bin/bob_compute_perf.py -d /PATH/TO/MULTIPIE/OUTPUT_DIR/U/pca_euclidean/scores/scores-dev -t /PATH/TO/MULTIPIE/OUTPUT_DIR/U/pca_euclidean/scores/scores-eval -x

This value corresponds to the one of the PCA baseline reported on Table 3 of the article (Once more, be aware of differences due to the implementation changes in the feature extraction process and algorithm parameters that have not been kept). These results are obtained for a PCA subspace of rank 2048, which was found to be the optimal PCA subspace size, when we tuned this parameter using the LBPH features.

Baseline 2: LDA on the PCA projected LBP histograms

The PCA projected LBP histogram features considered for the PLDA system were also used in combination with the Fisher’s Linear Discriminant Analysis (LDA) classification technique (commonly called Fisherfaces in the face recognition literature).

This involves three different steps:
  1. LDA subspace training

  2. Model enrollment

  3. Scoring (with an Euclidean distance)

The following command will perform all these steps:

$ ./bin/toolchain_lda.py --output-dir /PATH/TO/MULTIPIE/OUTPUT_DIR/

Then, the HTER on the evaluation set can be obtained using the evaluation script from the bob library as follows:

$ ./bin/bob_compute_perf.py -d /PATH/TO/MULTIPIE/OUTPUT_DIR/U/lda_euclidean/scores/scores-dev -t /PATH/TO/MULTIPIE/OUTPUT_DIR/U/lda_euclidean/scores/scores-eval -x

This value corresponds to the one of the LDA baseline reported on Table 3 of the PLDA article (Once more, be aware of slight differences due to the implementation changes in the feature extraction process). These results are obtained for a LDA subspace of rank 64, which was found to be the optimal LDA subspace size, when we tuned this parameter using the initial LBPH features.

Baseline 3: LBP histogram classification with Chi square scoring

The LBP histogram features might be used in combination with a distance such as the Chi Square distance, to obtain a face recognition system.

This involves two different steps:
  1. Model enrollment

  2. Scoring (with a chi square distance)

The following command will perform all these steps:

$ ./bin/toolchain_lbph.py --output-dir /PATH/TO/MULTIPIE/OUTPUT_DIR/

Then, the HTER on the evaluation set can be obtained using the evaluation script from the bob library as follows:

$ ./bin/bob_compute_perf.py -d /PATH/TO/MULTIPIE/OUTPUT_DIR/U/lbph_chisquare/scores/scores-dev -t /PATH/TO/MULTIPIE/OUTPUT_DIR/U/lbph_chisquare/scores/scores-eval -x

This value corresponds to the one of the LBP histogram (chi square) baseline reported on Table 3 of article (Once more, be aware of slight differences due to the implementation changes on the feature extraction process).

Summarizing the results as in Table 3

If you successfully run all the previous experiments, you could get a summary of the performances, as in Table 3, by running the following command:

$ ./bin/plot_table3.py --output-dir /PATH/TO/MULTIPIE/OUTPUT_DIR/

Reporting bugs

The package is open source and maintained via github.

If you are facing technical issues to be able to run the scripts of this package, please send a message on the Bob’s mailing list.

If you find a problem wrt. this satelitte package, you can file a ticket on the github issue tracker of this satellite package.

If you find a problem wrt. the PLDA implementation, you can file a ticket on Bob’s issue tracker .

Please follow these guidelines when (or even better before) reporting any bug.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

xbob.paper.tpami2013-1.0.0.zip (878.1 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page