Stable video indexing in Fast video cataloger 8
We have worked very long to make sure Fast video cataloger indexing is stable and can handle just about all video formats.
Indexing videos are always going to be a pretty time-consuming process since the video file needs to be read and decoded. Just reading through the large files is a time and CPU-consuming process. Having videos on a fast device and having a fast multicore CPU will help but even so, if you have many videos it will take considerable time.
From the start, we made sure you can add a list of files to index so that you can let it work along if you have hundreds or thousands of videos to index. After each video is indexed the current playlist is saved so the program can continue where it left off in case the program crashed, the power went out, Windows decided it was the perfect time to do an update, or the program closed down for any other reason.
We have put in plenty of effort to fix bugs in our software to make this as stable as possible. We also code to handle broken videos, conflicting codecs, and so on. But even so, we have reports of people getting crashes when they index large lists of videos.
So what happens when a video is indexed and what is the problem here?
What happens during video indexing?
To decode a video we first need to load and parse the video file format. A video file, i.e the container, contains video, audio, and perhaps other data. So the first thing we need is to read the data in the container and split it out into its different parts. How the format stores its data is dependent on the video format i.e. mp4, MPEG, Avi, etc. This piece of container parsing code is also sometimes called a splitter as it is splitting up the video file into separate streams of the same data type i.e streams for audio, video, etc. Then we need a video decoder for the video stream. This basically takes the video stream and reads it in compressed format and decompresses it. There are a whole lot of different compressions out there and they are constantly being improved upon. Typical video compression is lossy and uses similar techniques as image compression, like jpeg for each frame in the image and also the fact that not every single pixel changes between frames. Some of the compression algorithms have been implemented in hardware on GPU, so for even speedier decompression the decoder can in some cases use hardware decompression, in that case, you also have a video driver involved and the driver depends on the type of video card you have installed.
After the decompression, we have an uncompressed image of a video frame. This is basically all we need for the indexing process, we compress the image again and it is saved in the video catalog.
What can go wrong?
So to summarize, every time a video is indexed, in addition to Fast video cataloger there is a splitter for the format of the video, a decoder for the compression, and possibly a video driver. How these different components run also depends on parameters given to the container format and compression. So for every video, there is a risk that there is an issue in the splitter, decoder, or driver and the bug might not be an obvious crash. It could be a random memory overwrite that messes it up for the following video or another of the components.
The more videos you index in a row the higher is the risk that the computer encounters something unexpected that goes wrong.
And
There is basically no way we can prevent this from happening.
The Fast video cataloger 8 solution
What we can do is to isolate the whole indexing process from the rest of Fast video cataloger and each indexed video from the next. We basically launch each indexing in a separate process so that we always start from a known state and so that each indexing of a video is separate from the other. As a user you should not see any difference, speed is the same as before and you can follow the progress as the video is indexed.
This has now been implemented in Fast video cataloger 8 that you can download from here: https://videocataloger.com
Settings for indexing videos in fast video cataloger
