How to find all _Bad Frames_ in a long AVI?

[Cela]

How to find all the Bad Frames (akka duplicate/dropped frames) for Fizik's BadFrames plugin in a large avi?

Is there a plug-in or an avs script available to help create "BFidx.txt", a text file with the frame numbers idx of all Bad Frames, i.e. Dropped and Duplicate frames, in an automized manner for the example below?

AviSource("test.avi")
# Assume list of bad frames (one number per line) in file "BFidx.txt"
BadFrames(file="BFidx.txt", blend=true)

The script
# dupfind.avs
#############
AviSource("testfile_mit_Dubs.avi")
##################################
### using mvtools-v2.5.11 ###
# http://forum.doom9.org/showthread.php?p=1391871#post1391871
#############################################
#
colon = ": "
#
### Debug "
WriteFile("dupfind_diff.txt", "current_frame", " colon", "YDifferenceFromPrevious")
#WriteFileIf("dups.txt", "YDifferenceFromPrevious < .001", "current_frame", " colon", "YDifferenceFromPrevious")
#
# Output idx of duplicates into textfile
#
WriteFileIf("dupfind_idx.txt", "YDifferenceFromPrevious < .001", "current_frame")
ConvertToRGB() # to see the avi in VirtualDub

finds many but not all duplicate frames.

[Cela]

No ideas? Too difficult? Too trivial? :confused:

[pbristow]

I'd be interested in such a thing too, and your script there is a good start. In order to refine it, though, you need to see what it is about the currently undetected bad frames in your source that allows *you* to spot them, and then try to adapt your script to spot the same clues. I doubt the process can ever be fully automated, though, since a "bad" frame in one context could equally well be just an "interesting artistic choice" in another.

[Cela]

... I doubt the process can ever be fully automated, though, since a "bad" frame in one context could equally well be just an "interesting artistic choice" in another.If not fully solved, I hope it can be improved.

I found a small sample (1.25MB) [URL="http://www.mediafire.com/?wk06w2c8vkrriod"]
When stepping through this avi from begin to end one can see duplicate frames at idx= 7, 10, 13, 19, 22, 25 (BadFrames_expected).
The duplicate frames idx=7, 10, 19 are found by dupfind.avs and have YDifferenceFromPrevious=0.000000.
The duplicate frames idx= 13, 22, 25 are not found because YDifferenceFromPrevious= 2.143024, 2.080581, 2.152214, respectively, though these frames look identical to their respective preceding frames.

I hope, someone will help to solve the riddle.

[pbristow]

The differences I'm seeing between those frames and their originals are just low-level random noise. If you run a light temporal denoise you'll be able to reduce those differences from 2.1 to something like 1.3 without losing any genuine details - and indeed this is worth doing before any further processing anyway - but you'll also need to raise your detection threshold to a more realistic 1.5 or so.

What's curious is why you have some dups with noise differences, and some without. That means the dups have been created on two different occasions, one before the noise got into the picture and one after.

[Cela]

...If you run a light temporal denoise you'll be able to reduce those differences from 2.1 to something like 1.3 without losing any genuine details - and indeed this is worth doing before any further processing anyway...Thanks! This may be the right way to go. Could you, please, be more specific about a light temporal denoise? ...but you'll also need to raise your detection threshold to a more realistic 1.5 or so. Yes, I also got this idea when I compiled the example. But, since I do not know what is going on under the hood of YDifferenceFromPrevious and since the good detections had a straight zero differences I did not dare to enlarge the threshold. Without denoising it would need to be 2.2+ threshold for this example. Unfortunately, this example was shot under bright light conditions. So, I guess, these duplicates do not suffer low-light noise but may origin from stroboscopic capturing. Other footage is mainly dominated by low-light effects.
...What's curious is why you have some dups with noise differences, and some without. That means the dups have been created on two different occasions, one before the noise got into the picture and one after.I do not know where these noise peaks, restricted to certain frames, could come from. Maybe it is induced by how the screen capture works below the hood.
So, the next step to a pragmatic solution will need a good choice for your first advice, a light temporal denoise. What VirtualDub plugin, with what settings would you suggest?

[pbristow]

"what is going on under the hood of YDifferenceFromPrevious" is as follows:

For each pixel in frame N, subtract it's brightness (the Y value) from the brightness of frame N-1. If the result is negative, switch it to positive. Now average this value over all the pixels in the frame.

YDifferenceFromPrevious = 1.0 means that the average difference in brightness between the pixels in frame N and the corresponding pixels in frame N-1 is 1.0.

If Frame N is noiseless picture, and frame N-1 is the same picture with random noise whose average amplitude is 1.5 "steps", then YDifferenceFromPrevious = 1.5.

If frame N and frame N-1 are the same picture, with e same amount of noise added, but the noise is generated freshly each time, then the YDifferenceFromPrevious will be approximately SQRT( (1.5)^2 + (1.5)^2 ) = about 2.1-ish... which seems to be what's happened here. :) So the amount of noise that's got into your video at some point has an average amplitude of around 1.5 steps.


Here's what I think must have happened:

1. The footage was shot, and for some reason either the shooting or the later playback/transfer created some dropped frames, which showed up as duplicates (alternatively, the frame rate was increased by duplicating some frames).

2. The footage was then either transferred by an analogue method that introduced random noise, or compressed by some means that introduced some near-random noise. The level of noise is low enough to not be detected by eye unless you look really carefully, BUT it shows up when you compare the already duplicated frames, because all the proper picture information is the same, but the added noise is different.

3. Finally, some other process has caused more duplicated frames. However, because these duplications are occuring after the random noise was added, the new duplicates are *identical* to their originals. Both the "proper" picture and the added noise have been duplicated, so there is literally zero difference.

So, its not that there are "noise peaks" as you say, but simply that on the most recently created dups, the noise is "frozen in" and repeated exactly, resulting in exactly zero difference between the two frames; while on the earlier created dups, the noise is new and different on each instance of the frame (just as it is on all the non-duped frames - but they should all have differences well over 2.1 anyway, because of the amount of movement between frames).

So I would suggest that you first establish exactly what the original frame rate of the material was, so that you can work out how best to handle each of the two kinds of dups. For example, if one set of dups was created by bad frames being skipped, then you'll want to re-create those missing frames by motion interpolation. But if the other set of dups were created by a change of frame rate, then the best thing to do is decimate *just those* dups out of the stream and get back to the original frame rate, for the most fluid movement... Otherwise, you're just exchanging one moment of frozen time for two moments of half-speed motion. (Of course, if you're happy with the current frame rate and don't mind the slightly jerky effect of the way it was achieved, then you could just leave that set of dups alone.)

[EDIT:] Oh, nearly forgot! For the temporal denoise, you could start with TemporalSoften(1,3,6,15,2) and see how that works out. It's quick (due to the low radius and the fact that it's not doing complicated motion-estimations like the heavy-duty denoisers), and works fine for low level noise like this. Also you probably don't want to *completely* kill the noise, as a small amount of random noise can help to combat problems like colour-banding being caused by later processing. (See "dither".)

[Cela]

Hi pbristow,
thanks for your very interesting explanation. I can map it to the actual workflow steps:
a) the footage was recorded with an AVCHD camcorder (this footage is perfect, no visible jerks)
b) One of the 0000i.MTS streams (i=0, ..., n) was played with a software player, e.g. mpc, on to the computer monitor
c) the screen play was recorded with a screen recorder to yield an Xvid avi.
I guess, the problem comes from screen recording, which may have an input and an output side and the monitor setup involved?
Monitor setup: 1920x1200 pixel, True Color 32-Bit, 60 Hertz
Screen recorder setup: 30 fps, Xvid, 10000kbps, I-frame only, input RGB, output AVI, pixel aspect ratio 1:1
The 2 problem steps may be hidden:
1. All footage is compressed by the Xvid compression algorithm.
2. The 30 fps snapshots may produce moving pictures with what I call a stroboscopic effect, known from old films where the wheels of the cars seem to turn backwards while the car drives forward. So it can happen that there could be two snapshots from the same input picture - just exchanging one moment of frozen time for two moments of half-speed motion.
In fact, the camcorder freezes time with 50i, 50fps, interleaved, TFF.
The graphics card plays with 60 fps. The screen recorder samples to progressive footage with 30fps. Thus, I conclude it is a matter of digitizing a smooth movement first by 50 fps interleaved and then reproducing by 30 fps progressive. I could also reproduce by 25 fps progressive, but this leads to an even more jerky result. So the 30 fps are practically already a small step into the desired direction to regain, as much as possible, the smoothness of the originally smooth movements.
Shouldn't that be possible by some sort of interpolation or blending?
If so, the crucial first step remains to find all jerky parts, i.e. all parts with duplicate looking frames irrespective of almost invisible noise.

To make it not too simple, each camcorder recording appears to get its own noise level, mainly depending on lighting condition when recording the shot. This means, for each footage the adequate threshold has to be establishes before doing anything else. And, how to distinguish between 'is it just invisible noise' or 'is it no noise but some tiny change somewhere in the proper picture'?

To regain the smoothness of the originally smooth movements turns out to be not as easy as it should.

Thanks for your help!