Face Recognition¶

The Face Recognition batch tool analyzes video thumbnails to automatically detect and tag actors using face recognition. You can access this tool from the start page or by right-clicking a video and selecting Detect Actors... from the context menu.

The tool offers three operation modes:

Learn from existing cast -- Improves recognition accuracy by scanning videos where actors have already been manually tagged. The system detects faces in the video thumbnails and matches them against each actor's existing embeddings. When a strong match is found (well above the normal tagging threshold, so the system never learns from uncertain matches), it saves that face as an additional embedding for the actor. Small, low-quality faces are never learned from. This is useful because faces captured from real video footage (with natural lighting, angles, and resolution) complement the companion photo embeddings and help the system recognize actors more reliably. Prerequisite: Actors must already have at least one embedding from companion images before this mode can work -- actors with no embeddings are skipped.

Detect and add actors to cast -- Scans video thumbnails for faces and automatically adds matching actors to the cast. This is the primary mode for batch face detection.

Remove thumbnails without faces -- Deletes thumbnails where no face is detected in the image.

You can select which videos to process: All videos, the currently selected videos, or the filtered set. You can also choose to skip videos that already have actors in their cast to avoid redundant processing. The frame skip setting controls how many thumbnails to skip between processed frames, where 1 means every frame is processed and higher values are faster.

How It Works¶

The face recognition system uses a 128-dimensional "embedding" as a unique fingerprint for each face. The process works as follows:

Faces are extracted from video thumbnails captured during indexing
A 128-dimensional embedding is generated for each detected face
The faces in the video are grouped into the people in it: faces of the same person, seen in different thumbnails, go together
Each person is compared, by the average of their faces, against each actor's learned looks (see Actors whose appearance changes)
If a person matches an actor above the confidence threshold and is seen in at least the minimum number of faces, the actor is tagged

Matching a person by the average of their faces, rather than each face on its own, means a thumbnail where the face is turned or blurred does not decide anything by itself, and a few poor frames do not keep a clearly recognized actor out. Since version 10.5 this finds more of the actors in a video and adds fewer wrong ones.

Since version 10.5 a person is also compared with a second face model, the one that groups faces into people in the Unnamed people tab, and the two scores are averaged. This needs the second model's description of the faces the actor was learned from; Fast Video Cataloger finds it by itself before it next recognizes faces, from the faces stored in your videos and from the actor's photos. The first time, with many learned actors, that can take a minute. Because the averaged score runs a little lower, a person scored this way needs the confidence threshold minus 0.02 (0.62 at the default 0.64). On a real catalog this found more of the cast actors and added fewer wrong ones; an actor learned only from a face that is no longer in the catalog is matched as before.

Three additional safeguards reduce wrong tags:

Ambiguous matches are skipped. When two actors in your catalog score almost equally well for the same person, the system does not guess -- nobody is tagged. The Face Recognition batch log names the other actor ("too close to ..."), so you can tell this apart from an ordinary miss.
Very small faces are ignored for matching. Faces below roughly 50 pixels are detected, but their embeddings are too unreliable to identify anyone, so they never count towards tagging.
A brief second person needs a clearer match. A person seen in three faces or fewer, never in the same frame as a person already tagged as another actor, is most often a poor view of that same actor: a side view or a blurred frame. Since version 10.5 such a person adds an actor only when they score at least 0.05 above the confidence threshold. A second actor who appears in a frame together with the first, or in more than three faces, needs only the threshold.

Since version 10.5, a face is not described from the thumbnail alone while a video is indexed. The faces are found on the thumbnail, and each one is then described from the same frame at up to 1280 pixels wide. A face only 30 pixels wide in a 640 thumbnail is 60 pixels wide there, which is enough to recognize. Faces too small to describe on the thumbnail are kept when they are at least 30 pixels wide in the larger frame, so a 4K video indexed with small thumbnails loses few faces. A small picture of each face, up to 192 pixels and a few KB, is kept from the larger frame and shown for the face in the Cast window and the Unnamed people tab. Faces indexed before this keep the picture cut from their thumbnail until the video is indexed again. On a video indexed this way, Detect Actors matches the faces as indexing described them, and finds faces on the thumbnail only where indexing found none. For videos indexed before 10.5, Read faces from the video file when it is available (on by default, in the Face Recognition batch tool and the Face Recognition preferences) makes Detect Actors read each frame with a face from the video file and describe the faces from it. When many of a video's thumbnails have faces, their frames are read in one pass through the file, otherwise one at a time. This is slower, and it needs the video file, so offline and encrypted videos use the thumbnails. Indexing a video again does the same, but replaces its thumbnails. Running it again keeps what you decided in the Unnamed people tab: a face's person, Not this person and Not this actor.

Generating Face Embeddings¶

Before the system can recognize an actor, it needs to learn their face by generating face embeddings from companion images.

Learning faces from companion images¶

Open the Edit Actor dialog (double-click an actor)
Add companion images showing the actor's face
Select one or more companion images in the list
Click Learn Face to generate embeddings from the selected images
The embedding count is shown in the Companion Images header, e.g. "(8 face embeddings)"
Images that have been used for learning show a green border

You can also right-click a single companion image and select Learn face from this image.

Tips for best results¶

Select 8 high quality images with clear faces, in different lighting and different angles
Use images with only one person for the first image -- when there are no existing embeddings, the system cannot determine which face belongs to the actor if multiple people are in the image
Use clear, well-lit photos where the face is the main subject
Include a variety of expressions and poses
Avoid sunglasses or heavy makeup that obscures features

The system matches faces in subsequent images against existing embeddings to ensure it's learning the correct person. If multiple faces are detected and no existing embeddings exist, the image is skipped.

Other ways an actor learns a face¶

Naming a person in the Unnamed people tab or the Cast window learns the actor's face from the videos the person is in.
Import > Actor folders learns each imported actor's face from the pictures in the actor's folder, and can name the actor's people in their videos (see Import Actors folders).
Import > Actors and Import > All bring the actors' learned faces from the other catalog (see Import Catalogs).

Actors whose appearance changes over time¶

The learned faces of an actor are automatically grouped into distinct looks. Photos that resemble each other (same age, same hairstyle) are combined into a single averaged reference, while clearly different appearances -- a younger face, a beard, a different hair color -- are kept as separate looks. A face in a video is recognized if it matches any of the actor's looks, so old and new appearances do not compete with each other.

If a person's appearance has changed over the years, learn faces from photos of each period. A couple of photos per look is enough; the system does the grouping for you.

Managing Embeddings¶

To see how many embeddings an actor has, open the Edit Actor dialog -- the count is shown in the Companion Images expander header, e.g. "(8 face embeddings)".

To clear all embeddings for an actor, click Clear All in the Face Recognition section of the Edit Actor dialog. This removes all learned face data for that actor. When you delete an actor, their embeddings are automatically removed.

Unnamed people¶

Face recognition stores every face it finds, also the faces of people who are not actors in your catalog yet. The Unnamed people tab in the Actors window groups those faces into people, so you can name the people who keep turning up in your videos without first finding a photo of each one.

Faces are stored when videos are indexed with face recognition turned on, and when you run Detect Actors. For videos indexed before version 10.5, run Detect Actors over them once to fill the tab.

Faces found while a video is indexed, or while Detect Actors runs, join the person they look most like straight away, or become a new person together with faces of the same person in other videos. The first time you open the tab in a session, it groups all the stored faces again. Click Find people to do that yourself, for example to sort out people that were split or joined wrongly. A person is shown when they appear in at least two videos, or with at least three faces in one video. Each card shows the person's largest face and the number of videos and faces. When the person looks like an actor you already have, that actor's portrait and name are shown under the face; click them to name the person as that actor.

Select a person to see all their faces below the cards. If the person looks like an actor, the actor's portrait is shown beside the faces at the same size, with how alike they are, so you can compare them before you click Same person. If it is someone else, click Not this actor: that actor is not suggested for these faces again, and the next closest actor is shown, if there is one. Double-click a face to select its video in the video list. Right-click a face that belongs to someone else and choose Not this person to take it out. A face taken out is never grouped again. For something that is not a face at all, choose Delete face: it is removed from the catalog and is not found again in that video.

With a person selected:

Show videos (right-click, or double-click the person) shows only the videos the person is in. The person appears as an Unnamed person tile in the Cast section of the Search window and works like one more actor there: a video with any of the people in the list is shown. Select the tile and click Remove Actor to take it out. For a person you have already named, Show videos filters on the actor instead.
New actor... opens the Add Actor dialog with the face as the portrait. When you save the actor, the person is named as that actor. Press Enter in First name or Last name to save.
Add to actor... names the person as an actor you already have. Type the name to find the actor. If nobody by that name is found, or only someone else with the same name, click New actor... in the same dialog: it opens Add Actor with the name you typed and the face as the portrait.
Merge joins the selected people into one, when the same person was found as two or more people, for example once seen from the front and once from the side.
Ignore hides extras, crowds and faces found by mistake. New faces of an ignored person are not shown either. Check Show ignored to see them again, then right-click and choose Stop ignoring.
Delete removes the selected people and all their faces from the catalog, after asking. The faces are not found again in those videos, also when you run Detect Actors or index the videos again. A deleted person can still turn up in videos you add later, as a new person; to keep someone out everywhere, use Ignore instead. A person you have named has to be unnamed before it can be deleted.

Naming a person adds the actor to the cast of every video the person is in. The person's largest face in each video, up to ten videos, also becomes one of the actor's face samples, so face recognition finds the actor in new videos from then on. Faces under about 50 pixels are grouped into people, but are too small to become face samples.

Sort by, at the top of the tab, sets the order of the people: Most videos (the default), Suggestions first, which puts the people who look like one of your actors at the top with the closest match first, so you can go through the suggestions in one pass, or Most faces.

Unnamed people in the Cast window¶

The Cast window lists the unnamed people in the selected videos under the cast, with their largest face in the video and how many faces they have there and how many videos they are in. When a person looks like an actor you already have, the actor's name is shown under the face. Right-click a person to Name as that actor, make them a New actor..., Add to actor..., Ignore or Delete them. Show in Unnamed people, or a double-click, selects the person in the Unnamed people tab, with all their faces. With several videos selected, the people in any of them are listed. People you have named are in the cast itself, and ignored people are not listed.

Taking a naming back¶

Right after you name a person, the status line has an Undo link. To take back a naming later, check Show named: the people you have named are shown with the actor's portrait and name. Right-click one and choose Unname. The actor is taken out of the videos the naming added it to, and the faces it learned from the person are forgotten. Cast entries and face samples that were there before, or that you added yourself, stay. If face recognition has already tagged the actor in other videos with the learned faces, those tags stay; use Filter videos on for the actor to find them. Edit Actor on the same right-click menu opens the actor the person was named as.

In a named person's faces, Not this person also removes the face sample learned from that face, and when it was the person's last face in a video, the actor is taken out of that video.

Detection Methods¶

Two face detection methods are available:

HOG (Histogram of Oriented Gradients)

Faster processing
Works well for clear, front-facing shots
May miss faces at angles or in poor lighting
Good for quick scans of large batches

CNN (Convolutional Neural Network)

Runs in addition to HOG on every frame; the results of both detectors are combined
Finds faces at angles and in difficult lighting that HOG misses
Detection is much slower with CNN enabled
Keep it off in most catalogs. The extra faces it finds are side views and dim or blurry faces, which are hard to tell apart, so in our tests CNN found the right actor no more often than HOG alone and added more wrong actors
Try turning it on only if the actors in your videos seldom face the camera and many are not being found, and check the first results before indexing a large batch

You can select the detection method in the Face Recognition preferences.

Tips for Best Results¶

Quality companion images matter -- Use clear, well-lit photos where the face is prominent
Add multiple embeddings -- Different angles and lighting conditions improve robustness. 8 good embeddings per actor is recommended
Cover every look -- If the actor's appearance has changed over time (age, hairstyle, beard), include photos from each period so every look gets learned
More thumbnails = better detection -- Videos with more thumbnails give more chances to detect faces. Consider increasing thumbnail count for videos with important actors
Use companion images wisely -- The green border shows which images have been used for learning
Review results -- Face recognition isn't perfect. Occasionally review auto-tagged actors for accuracy and remove incorrect tags

See the Face Recognition preferences for confidence thresholds and other settings.

Troubleshooting¶

"No faces detected in image"

The image may be too small -- try a larger, higher resolution image
The face may be partially obscured or at an extreme angle
Try a different photo with a clearer view of the face

"Actors not being detected in videos"

Ensure the actor has at least one embedding (check the Edit Actor dialog)
The video thumbnails may not show the actor's face clearly
Try adding more embeddings from different angles
Verify the actor appears in enough frames (check the minimum frames setting)
If the actor looked different when the video was made, learn faces from photos of that period too

"A face is clearly visible but the actor is not tagged"

If two actors in your catalog look alike, the match may have been skipped as ambiguous -- check the batch tool's log for "ambiguous" lines. Adding more distinctive companion photos for both actors helps the system tell them apart
The face may be too small -- faces below roughly 50 pixels in the frame they were described from are never used for matching. A higher thumbnail resolution helps (see the Video Indexer preferences)

"Too many false positives"

Add more embeddings to help distinguish between similar faces
Remove embeddings generated from low-quality source images
If CNN detection is on, try turning it off: the extra faces it finds are the ones most often matched to the wrong actor

"Detection is very slow"

CNN detection is about five times slower than HOG alone -- turn it off for large batches
Processing time depends on the number of thumbnails per video

Automatic Faceless Thumbnail Removal¶

You can configure the indexing pipeline to automatically remove thumbnails where no face is detected. Enable Remove thumbnails without faces in the Video Indexer preferences tab. This runs after all analysis (face recognition, scene classification, object detection) is complete, so you can also enable But keep if keyworded to preserve faceless thumbnails that have scene or object classifications. This setting does not apply when capturing a single frame.

For best results with face detection, use a thumbnail width of at least 320 pixels (set in the Video Indexer preferences).