Face Recognizer using openCV on the iPad

Note:

This project was built using

openCV 2.4.8
xCode 5.0.2
for iOS 7
iPad air

set up instructions here
download the full source here (26.9MB)

I created 2 sections. The recognition screen and the registration screen.

We first start off on the main page.

In the MainMenuViewController, I have 2 variables:

CameraWrapper * cameraWrapper;
DataModel * dataModel;

The cameraWrapper is basically an object that wraps all the functionality of the openCV camera. What it does it it captures an image repeatedly. Each image is a matrix object cv::Mat. It gets passed to you, and it is up to you to process it, see where the detected face is, and do things to it.

The dataModel holds all the face data structures. Namely, we are trying to save all the user faces into this data structure. We also have a user labels data structure that matches the faces. Finally it has a UpdateViewDelegate that calls whatever UIViewController that conforms to this delegate to update its front end whenever something gets updated int our data model.

For the register view controller, we add faces we’ve detected to our carousel. That way whenever a system has finished collecting user’s faces, it can train it, and thus, be able to recognize the user in the future.

In the case of recognition, our UpdateViewDelegate is all about updating user interface items.

@protocol UpdateViewDelegate <NSObject>

@optional

-(void)showMessageBox:(NSString*)title andMessage:(NSString*)msg andCancelTitle:(NSString*)cancelTitle;

// register view controller

-(void)addFacesToCarousel;

-(void)clearFacesInCarousel;

-(void)updateNumOfFacesLabel:(NSString*)strNumber;

// login view controller

-(void)animateTraining:(bool)flag;

-(void)incrementIdentityValue:(float)similarity;

-(void)showSimilarity:(float)similarity;

-(void)setRecognizeButton:(bool)flag;

-(void)setTrainButton:(bool)flag;

-(void)showIdentityInt:(int)identity andName:(NSString*)name;

@end

When you push the registration button you open up a UIViewController to register your face.

#pragma mark ------------------- button responders ----------------------

-(void)loginPage

{

DLOG(@"MainMenuViewController - loginPage");

LoginViewController * loginPage =

//[[LoginViewController alloc] initWithCamera:cameraWrapper andThreshold: UNKNOWN_PERSON_THRESHOLD];

[[LoginViewController alloc] initWithCamera:cameraWrapper andDataModel:dataModel andThreshold:UNKNOWN_PERSON_THRESHOLD];

[self.navigationController pushViewController:loginPage animated:YES];

[loginPage release];

}

-(void)registerPage

{

DLOG(@"MainMenuViewController.m - RegisterPage");

unsigned long totalFaces = [dataModel getTotalFaces];

unsigned long numOfProfiles = [dataModel getNumOfProfiles];

DLOG(@"currently in our data model: there are %lu total faces...and %lu profiles", totalFaces, numOfProfiles);

RegistrationViewController * registerPage =

[[RegistrationViewController alloc] initWithCamera:cameraWrapper andDataModel:dataModel];

[self.navigationController pushViewController:registerPage animated:YES];

[registerPage release];

}

When you push the recognition button you open up a UIViewController to recognize your face.

Both views repeatedly detect your face.

Face Detection and 2 ways of facial lighting

I’ve used 2 main methods of getting different lighting of the face. For example, one way it to have the user move their face around by controlling the position of their nose with a axis and a yellow dot. Have their nose line up with the yellow dot and then you’ll be able to position their face differently. Thus, you’ll get different facial lighting and thus, our training image set can be much more various. This helps in the recognition phase.

A second way is to use accelerometer. Have the user stand in one spot and snap images from 0 – 360 degrees. That way, you will also get various lighting on the user’s facial structures.

Camera leak…fix and usage

In openCV’s module highhui folder, go to cap_ios_abstract_camera.mm file
i.e, /Users/rickytsao/opencv-2.4.8/modules/highgui/src/cap_ios_abstract_camera.mm
and replace the stop method with this:

- (void)stop;

{

NSLog(@"cap_ios_abstract_camera.mm - stop");

running = NO;

// Release any retained subviews of the main view.

// e.g. self.myOutlet = nil;

[self.captureSession stopRunning];

//retain count is 2

[captureSession release];//(++)

//retain count is 1

NSLog(@"(++) 4/24/2014 cap_ios_abstract_camera.mm, -(void)stop, captureSession released because it was retained twice via self. and alloc in createCaptureSession");

self.captureSession = nil;

//retain count is 0, and captureSession is set to nil

self.captureVideoPreviewLayer = nil;

self.videoCaptureConnection = nil;

captureSessionLoaded = NO;

}

Insert the lines where I have (++). This is where we need to release the captureSession one more time. Or else, you will leak memory when you opens and closes the camera one too many times.

Step 1, process the face

Whether we are in registration or recognition, the program is constantly detecting a subject’s face via the method ProcessFace.

Every time camera from the iPad captures an image, it gets passed into our Processface method here. That image is of object Mat.

The method getPreprocessedFace will return the preprocessedFace, which means whether it has detected a valid face or not. Then we see what phase we are in and do recognition, or collect. In our case, since it is registration, we are collecting faces. So whatever face we detect, will be saved into a data structured to be trained later.

-(bool)ProcessFace:(cv::Mat&)image {

// Run the face recognition system on the camera image. It will draw some things onto the given image, so make sure it is not read-only memory!

if(m_mode==MODE_END) {

DLOG(@"ProcessFace: END PROCESSING IMAGES");

return false;

}

cv::Rect faceRect;

//don't modify the original

cv::Mat displayedFrame;

image.copyTo(displayedFrame);

//we work with displayedFrame from now on

cv::Mat preprocessedFace = [self getPreprocessedFace:displayedFrame

withFaceWidth:70

withCascadeClassifier:*faceCascadePtr

withFaceRect:faceRect];

//DRAW THE FACE!

if(faceRect.height > 0){

[self drawRect:faceRect aroundImage:image withColor:cv::Scalar(255, 0, 255)];

}

if (m_mode == MODE_RECOGNITION) {

[self doModeRecognition: preprocessedFace];

}

else if (m_mode == MODE_TRAINING) {

[self doModeTraining];

}

else if(m_mode == MODE_COLLECT_FACES) {

[self doModeCollectFaces:preprocessedFace withFlashRect:faceRect onImage:image];

}

return true;

}

As of now, we are collecting face so we use the MODE_COLLECT_FACES phase. This is where we for each image we snap, we inserted it into a data structure called preprocessedFaces.

Before the insertion, we do 3 things to the image. We shrink it to 70×70. Grayscale it. And equalize it. That way, the size of our image is much smaller and easier to work with. Then we pass it into openCV’s detectMultiScale, and it will return a cv::Rect object, detailing, where the face(s) is on the screen. In our case, we are working only with one face. We detect the largest face on the current image.

faceCascadePtr->detectMultiScale(equalizedImg,

faces,

searchFactor,

minNeighbors,

0|flags,

minFeatureSize);

Then we draw a square around the face.

if(faceRect.height > 0){

[self drawRect:faceRect aroundImage:image withColor:cv::Scalar(255, 0, 255)];

}

Once that’s done, we then see what phase we are in:

MODE_RECOGNITION
MODE_TRAINING
MODE_COLLECT_FACES

Since we are in MODE_COLLECT_FACES, we push our image matrix into a data structure.

preprocessedFaces.push_back(colorMat);

preprocessedFaces.push_back(colorMirroredMat);

faceLabels.push_back(m_userId);

I used iCarousel to add all the images I took and insert them onto an image carousel. That way, we can scroll through and check out all the images we took. These are the images that were added into our preprocessedFaces data structure.

Also, in previous versions, I simply used the yellow circle to match up to the person’s nose. Thus, angling their face JUST a little so that I can collect different angles of their face. In this version, I use the accelerometer, where the user angles their faces horizontally from 0 to 360.

Training

//we pass in grayPreprocessedFaces for paramPreprocessedFaces

-(cv::Ptr<cv::FaceRecognizer>)learnCollectedFacesWithFaces:(cv::vector<cv::Mat>&)paramPreprocessedFaces

withLabels:(cv::vector<int>&)paramFaceLabels

andStrAlgorithm:(NSString*)facerecAlgorithm {

DLOG(@"nLearning the collected faces using the [ %@ ] algorithm ...", facerecAlgorithm);

bool haveContribModule = cv::initModule_contrib();

if(!haveContribModule) {

DLOG(@"nERROR: The 'contrib' module is needed for FaceRecognizer but has not been loaded into OpenCV!");

return nil;

}

model = cv::createEigenFaceRecognizer();

if(model.empty()) {

DLOG(@"nERROR: The FaceRecognizer algorithm [ %@ ] is not available in your version of OpenCV. Please update to OpenCV v2.4.1 or newer.", facerecAlgorithm);

}

NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);

NSString *docs = [paths objectAtIndex:0];

NSString *trainedPath = [docs stringByAppendingPathComponent:@"trained.yml"];

unsigned long numOfFaces = paramPreprocessedFaces.size();

unsigned long numOfLabels = paramFaceLabels.size();

DLOG(@"Training %u faces and %u labels: ", numOfFaces, numOfLabels);

//we train on the gray preprocessed faces

model->train(paramPreprocessedFaces, paramFaceLabels); //original

//MODEL SAVING

model->save([trainedPath UTF8String]);//save the model with current data structure

DLOG(@"nlearnCollectedFacesWithFaces - training complete...thank you for using FaceRecognizer. :)");

return model;

}

After the faces have been collected, we automatically train those images using
method learnCollectedFacesWithFaces:withLabels:andStrAlgorithm: in DataModel.m.

We train it on a cv::Ptr object. Once its trained, we can recognize it by having the phase be changed to recognize in DataModel.m. Once our phase change to recognition, we run the below method, and use

1	identity = model->predict(tanTriggedMat);

to predict which image matrix below to which identity.

//process when we are in recognition mode

-(void)doModeRecognition:(cv::Mat)paramPreprocessedFace {

int identity = -1;

//if ((preprocessedFaces.size() > 0) && (preprocessedFaces.size() == faceLabels.size())) {

if (model && !model.empty()) {

cv::Mat grayedPreprocessedFace = [self grayCVMat:paramPreprocessedFace];

//test it out here

cv::Mat tanTriggedMat = [self tan_triggs_preprocessingWithInputArray:grayedPreprocessedFace andAlpha:0.1 andTau:10.0 andGamma:0.2 andSigmaZero:1 andSigmaOne:2];

tanTriggedMat = [self norm_0_255: tanTriggedMat];

//reconstructFace should take grayscle ONLY: NOTE : FOR EIGENFACES AND FISHERFAC

cv::Mat reconstructedFace = [self reconstructFaceWithCVMat:tanTriggedMat];

double similarity = [self getSimilarityWithMat: tanTriggedMat andMat:reconstructedFace]; //tan triggs

DLOG(@"similarity: %f", similarity);

[self showSimilarityDelegateView:similarity];

if (similarity < self.threshold) //WE HAVE A MATCH OF AN IDENTITY

{

try {

identity = model->predict(tanTriggedMat);

//LET'S PIGEONHOLE OUR IDENTITY. WHEN THE 100% TAKES PLACE, THE HIGHEST IDENTIY IS THE PERSON

NSString * key = [NSString stringWithFormat:@"%u", identity];

NSString * name = [nameKeyTable objectForKey:key];

NSString * count = [recognitionCount objectForKey:name];

int iCount = 0;

if(count == nil)

{

count = @"0";

[recognitionCount setObject:count forKey:name];

iCount = 0;

}

else

{

iCount = (int)[count integerValue];

iCount++;

}

[recognitionCount setObject:[NSString stringWithFormat:@"%i", iCount] forKey:name];

DLOG(@"---------------------------------> int id: %u, name: %@, count: %i",

identity, name, iCount);

}

catch(cv::Exception e) {

e.formatMessage();

}

//tell our UIVIEWCONTROLLER to increment identity approval value (0 to 100) for similarity

[self incrementIdentityValueOnDelegateView:similarity];

//tell our UIVIEWCONTROLLER to show the identity of this person

//[self showIdentity: identity];

}