Visual Search¶

Find scenes by describing them in words, or by picking a scene and asking for ones that look like it. Nothing has to be tagged first: visual search ranks every indexed thumbnail in the catalog by how well it matches, using the same AI model that image tagging and trained tags work from.

What you need¶

• The Visual search model. Open Manage AI models and download Visual search. It is the text half of the model that image tagging and trained tags share, so if you already have one of those, only the text part is fetched.
• The search index. Since version 10.4 every thumbnail gets a compact description recorded while a video is indexed (Preferences > AI Classification > Visual search index, on by default). Videos indexed before that, or with the option off, are added from AI Scenes with the Visual search index option, which shows how much of the catalog is covered and can be stopped and resumed. The Search window shows the same coverage in the Scene search section, because a search only sees what is indexed.

Describe the scene¶

1. Open the Search window and expand Scene Search. Tick Scene search.
2. Type what you are looking for in Describe the scene, for example a dog running on a beach, close-up of hands typing, drone shot over a city at night.
3. The Scene Search Results window fills with the best matching thumbnails, best first. With results grouped by video, the video holding the best match comes first.

Matches are grouped under the video they came from to start with. Turn Group Scene Search Results off (Preferences > Interface) for one continuous list instead; either way the best match comes first.

The Scene Search Results window has its own zoom slider at the top; Ctrl+mouse wheel over the results changes it too. The size is saved with the layout.

The search starts when you pause typing, or straight away when you press Enter.

Writing a description¶

Plain, concrete descriptions work best: what is in the frame, what it is doing, the setting. A few words are enough. Write them in English.

• Describe what the picture shows. The search compares your words with the image and knows nothing else about the video: not its title, file name, keywords or notes. The name of a person, a client or a place it cannot recognise from the picture does not help; use keywords, actors or face recognition for those, and let them narrow the search as described below.
• Say what is there, not what is missing. A street without cars ranks scenes with a street and cars, because the words that matter are street and cars. Describe what an empty street looks like instead: an empty street at dawn.
• When the results are off, rephrase. Add the detail that sets your scene apart (at night, close-up, from above, in the rain), or take detail away when a long description matches nothing well. A different word for the same thing often ranks differently.

Filters and how many results¶

Every other filter still applies. Narrow the video list by keyword, folder, date or cast and the description ranks only the scenes of those videos; add scene keywords or a note and it ranks only the scenes that pass them.

Two choosers under the description box decide how many results you get:

• Show best caps the list: 200 by default, 50 to 2,000. The results window says when the list was cut ("Showing the 200 best matches of 11,766 scenes"). Tighten the filters, or raise Show best, when what you want is not among them.
• per video caps how many scenes of one video can be among the matches: 1, 2, 3, 5, 10 or All, 3 by default. The thumbnails of one shot look alike and score alike, so without a cap a single long take can fill the whole list and hide every other video.

Both choosers apply to Find similar scenes too, and both are remembered for your next search. They are available once Scene search is ticked.

A saved filter keeps the description, the Similar to scene and both choosers, so saving a visual search brings back the same ranking later.

The tooltip of each result shows its relevance, a number between 0 and 1. The numbers are only comparable within one search: a good match to a description scores around 0.15 to 0.30, while two scenes that look alike score 0.8 and above.

Find similar scenes¶

Right-click a thumbnail, in the video's thumbnail strip or in the Scene Search Results, and choose Find similar scenes. The Search window ticks Scene search, opens Scene Search and shows the chosen scene under Similar to, and the results list the scenes that look most like it across the videos in the filter. The scene you started from is left out of the results. Press Reset next to the picture to stop.

Find similar does not use the model itself — it compares the descriptions already in the index, so it works even before the Visual search model is downloaded. Building the index does need the model.

A description and a Similar to scene are two different rankings, so only one applies at a time: typing a description clears the chosen scene, and Find similar clears the description.

Shared catalogs¶

On a catalog shared through the server, the work is split between the two machines.

• The index is built where the indexing happens. When the server indexes videos it records the descriptions itself, using the model downloaded on the server's own machine. Videos you index from your own computer use yours.
• Describing a scene uses the model on the computer you type on. Fast Video Cataloger turns your words into the search vector before sending the search, so every computer that searches by description needs Visual search downloaded from Manage AI models, whichever catalog it opens. Searching from the web client or through an AI assistant uses the server's copy instead, since there is no program on that end.
• Find similar scenes needs no model on any machine. It compares the descriptions already in the index.

Download the model on the machine that shares the catalog and the server picks it up by itself; there is no need to re-run the Share Video Catalog wizard or restart the service. Before version 10.4 the server only found the model when the wizard happened to run after the download.

Troubleshooting¶

"Visual search model not downloaded" under the description box: download the Visual search model from Manage AI models (the link opens it on the right entry). Until then a description cannot be ranked: if it is your only scene filter the search comes back empty, and with scene keywords or a note set those still answer on their own.

The coverage line says the index covers part of the catalog: the rest was indexed before 10.4 or with the option off. Run AI Scenes with Visual search index on the videos, or on the whole catalog; it can be stopped and resumed. Until then those scenes cannot be found by description.

"This scene is not in the search index yet" when using Find similar: the same cause, for that one thumbnail.

No results for a description that should match: check the video filter first. A description ranks only the videos the rest of the Search window lets through.

One video fills the results: set the per video chooser under the description box to a lower number.

The scene you want is ranked low: see Writing a description.