This is part of a series of blogs diving into the technical aspects of Veridium’s distributed data model, biometrics, and computer vision research and development by our chief biometric scientist Asem Othman.
Extracting intrinsic information from faces, such as identity, gender, ethnicity, and age, is a task that humans perform routinely and efficiently. Now, machines are beginning to catch on. The availability of powerful, low-cost computing systems has created interest in developing automatic face recognition systems and deploying them in a number of applications, including biometric-based access systems.
A technology once only seen in television dramas – automatic face recognition systems are now deployed and utilized in our daily activities. Commercial applications of automatic face recognition are now abundant, including “tag” suggestions on Facebook, as well as the organization of personal photo collections in Google Photo. Moreover, after Apple’s announcement of Face ID, your face may become the norm for unlocking your phone and for daily payment transactions.
Automatic Face Recognition
Automatic face recognition poses a challenging problem in the field of image analysis and computer vision. Thus, research in face recognition is striving to solve fundamental challenges, such as developing face matching methods that are invariant to age, pose, illumination, and facial expressions. Further, research also seeks to utilize the advances in technologies like digital cameras and mobile devices to perform face recognition in new applications and scenarios. Finally, researchers are looking to fulfill the increased demands on security in numerous practical applications where human identification is needed.
Old School Face Representation
To identify a face in a digital image, the face recognition system should automatically find the face in the image (if there is one), then the recognition occurs by matching the detected face with the face template in a database. Just as in the case of fingerprints, where ridge details were described in a hierarchical order at three different levels, facial features can also be described in a hierarchical order.
Level 1 features are the facial characteristics that can be observed from the general appearance of the face, such as skin color. Level 2 features are the localized characteristics of the face, such as the shape of the face and the relationship among the facial attributes. Finally, level 3 characteristics are the micro features that can be useful for the discrimination of monozygotic (i.e., identical) twins, such as facial marks.
Face matching is the process of measuring the similarity or dissimilarity between two face images based on the extracted features. Level 1 face features are quite analogous to level 1 fingerprint features. Hence, level 1 face features cannot accurately identify an individual over a large population of candidates. Similarly, much like level 2 features of fingerprints, level 2 face features are the most discriminative features and are predominantly used for face recognition approaches. There are two broad categories of approaches to match the detected face images: Appearance-based and feature-based methods.
Appearance-based methods consider the global properties of the face image intensity pattern, such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), and Independent Component Analysis (ICA). Feature-based methods use local features of the face, such as geometric relations between the facial features and local texture features of the face that are invariant to pose and lighting, such as gradient orientations and local binary patterns (LBP).
Meanwhile, level 3 features contain unstructured micro-level features on the face, including scars and other facial marks. These features have been used, along with level 2 features, to identify monozygotic twins.
Recently, deep neural networks have achieved impressive results for many visual recognition tasks, including face recognition. Neural networks are not new, perceptrons were first developed in the 1950s. However, network models with many hidden layers (deep structures) can now be trained due to better regularization strategies and availability of large face databases and processing capabilities. Again, rather than handcrafted features, face representations are learned by deep convolutional neural networks (ConvNets). These are trained to classify identities or verify pairs of face images from large-scale training sets of face images.
The dimensionality of the feature representation is hierarchically reduced due to the structure of convolutional and pooling layers. Both low-level features and global features are learned in a cascaded manner. Usually, the output of the last hidden layer, prior to the classification layer, has been shown to have learned a highly robust face representation for new face images in testing.
However, the success of these deep ConvNets approaches relies on sophisticated learning and large-scale training sets. Therefore, the highly successful face systems that have been developed by Facebook and Google raise the question: did they use my personal images in their training database?
For example, Facebook now has 2 billion monthly users who upload about 350 million photos every day – a “practically infinite” amount of data that Facebook can use to train its facial recognition software, according to a 2014 presentation by an engineer working on DeepFace, Facebook’s in-house facial recognition project. Facebook says publicly it doesn’t have any plans to directly sell its database. However, they used everyone personal images to train their DeepFace.
Despite All This, Face is Not the Answer
Whereas facial features are intrinsic properties of a face, the appearance (the textured look) of a face is subject to several factors, including the facial pose (or camera viewpoint), illumination, facial expression, and occlusions (sunglasses or other coverings). In unconstrained scenarios where face image acquisition is not well controlled, or where subjects may be uncooperative, the factors affecting appearance will confound the performance of face recognition.
Moreover, there may be similarities between the face images of different people, especially if they are genetically related. Such similarities further compound the difficulty of recognizing people based on their faces.
Though face recognition is a challenging task, the advanced deploying of deep learning is helping many companies and university address these challenges.