How to find and interpolate duplicate frames

[Cela]

Edit:
Thanks to helpful expert Gavino there developed an excellent solution with two avs procedures:
#find_duplicates_(by_Gavino):
DirectshowSource("raw.avi",30,true,false,true)
ConvertToYV12() # YV12 needed for MVTools and for CCE Basic
#colon = ": "
#WriteFile("dups.txt", "current_frame", "colon", "YDifferenceFromPrevious") # for testing
WriteFileIf("dups.txt", "YDifferenceFromPrevious < .001", "current_frame", " colon", "YDifferenceFromPrevious")#Interpolation_for_second_frame_of_duplicate_(by_Gavino).avs
DirectshowSource("raw.avi",30,true,false,true)
ConvertToYV12()
super = MSuper()
backward_vectors = MAnalyse(super, isb = true, delta=2)
forward_vectors = MAnalyse(super, isb = false, delta=2)
inter = MFlowInter(super, backward_vectors, forward_vectors, time=50, ml=70)
ConditionalFilter(inter.Loop(2,0,0), last, "YDifferenceFromPrevious", "<", ".001")
ConvertToRGB24() # for easy display in VirtualDub

If you are interested in how it developed you are welcome to go on reading. ;)

Capturing video (moving pictures/progressive frames) with respect to its audio timeline sometimes produces randomly distributed duplicate frames which are not in the original video stream.
These false duplicates are not so easy to locate in the captured video stream and degrade the replay quality of the captured steam (like dropped frames).

Example:
.............abc...........opq......xyz......... displayed moving pictures (good frames)
.............aac...........ooq......xxz......... captured moving pictures (missing good frames b,p,y in this example)
.............a-c...........o-q......x-z......... so the duplicate frames are effectively false duplicate frames (here symbolized by '-').


Are there any options to improve the captured video stream?

1. Help quickly finding where these false_duplicates are:
1.1 Add a 'Next/Prev duplicate frames' feature in the 'Go' menu of VirtualDub.
1.2 Add sort of a DGIndex feature 'false_duplicates_at.txt' list with the frame numbers of the duplicates.

Feature 1.2 and 1.1 would support quick access the false duplicates spots in the video by VirtualDub's Go feature.

2. Sort of 'multidecimate' the duplicates
.............aac...........ooq......xxz.........
.............ac...........oq......xz.........<<< missing video frames!

Thus, 'multidecimate' leads to a speed up the video by dropping the false duplicate frames, making the video stream shorter and thereby losing audio sync if audio is not shortened accordingly.
Does 'multidecimate' cope with random patterns?

3. Interpolation of missed moving pictures (is it possible? how?)
3.1 Interpolation
.............abc...........opq......xyz.........
.............a-c...........o-q......x-z.........
.............aBc...........oPq......xYz.........

Would be cool if it was feasable.
This remedy could also be applied for dropped frames?

3.2 'Smart interpolation' would mean that only the ~0.31% missing frames get re-calculated and the ~99.69% good frames are not re-calculated but simply copied (values provided from a 45 minutes video sample with 250 false duplicates).

Example data: "false_duplicates_example_data.rar"
http://rapidshare.com/files/376117955/false_duplicates_example_data.rar
VirtualDub cut of a 98 frame section with 10 false duplicates,
taken from 30fps camcorder AVCHD recording,
replayed with a software player with 60 Hz monitor refresh rate
and captured to 30 fps I-frame-only progressive Xvid.
The capture program did not report any dropped frames.
VirtualDub does not find any 'D'ropped frames either.
But there are 10 false duplicate frames in this sample.
It seems similar to representing a smooth curve by a
'staircase'-polygon.

Please forgive the long post reflecting the point of view of a simple user of VirtualDub (and sometimes Avisynth).

Ideas for quickly finding where these duplicates are would already be a great help for me!

Thanks in advance!

[Ghitulescu]

I prefer to keep the audio synch to video and not vice versa. Never had a dropped or duplicated frame. Audio corrections, should they be necessary, are much easier to be done, especially for home/camcorder movies.

[Cela]

@Ghitulescu
Thank you, Ghitulescu, for your information. That is a good start!
I also know (manual) workarounds and means to avoid the issue.

@ the doom9 experts:
In the course of time, as a user who does a lot manually by using fine programs like VirtualDub, I developed two sorts of interests in these issues:
a) a practical interest, how to save boring manual labour, and
b) sort of an academic interest, how it could be done better.
Please consider it as a feature request.

I suppose there are many bright guys browsing through this forum, who develop x264 encoders and other highly sophisticated solutions, analyze frame contents, etc. I am sure these experts, if they came across my two simple tasks, would make their life easier by setting up automatized procedures for these tasks:

Task 1:
Find the next spot of duplicates (two or more pictures practically alike) in a sequence of pictures.
The sequence of pictures is supposed to represent a smooth movement.
Write the start and end frame number of that duplicates spot into the next line of a text file, (separated by a tab character, which would make import into two Excel columns easier).

Almost all duplicate spots consist of only two pictures which are alike (or practically alike).
Every spot is ended by the next picture being significantly different than the previous.
Thus the pattern is
...aac... (aa are alike, c is significantly different) for most of the spots or
...aaaaaaaaaac... (with more consecutive alike pictures 'a' ended by a significantly different picture 'c')

In most recordings these spots are randomly distributed.
These spots can be easily located by manually stepping through the avi, frame by frame, with VirtualDub. If you consider that a 45 minutes 30 fps movie sums up to 81000 frames, this easy work tends to get boring.

Task 2:
Interpolate for the missing 'b' for the 99.9% case with pattern
...aac... where the second 'a' actually should be a 'b' to build up the supposed smooth movement
...abc... I guess there will be procedures available to .

To summarize it:
I am sure doom9 experts will have automated procedures (e.g. VirtualDub or Avisynth plugins) ready for use for these two tasks
1. find next duplicate
2. make an educated guess (interpolate) for the missing intermediate frame 'b'.

Please share youre expertise!

[Gavino]

You could use Avisynth to write out a text file with the frame numbers of (potential) duplicate frames, with something like:
WriteFileIf (http://avisynth.org/mediawiki/WriteFileIf)("C:\dups.txt", "YDifferenceFromPrevious (http://avisynth.org/mediawiki/ScriptClip#Runtime_Functions) < 3", "current_frame")
You might have to tune the luma difference threshold (here 3) to get accurate detection of duplicates.

Avisynth could also help with interpolation - look into MVFlowFPS (also ConditionalFilter for selectively replacing frames according to some criterion).

[Cela]

@Gavino
This is phantastic! It works great! Thank you so much!!!
It is as simple as that!!!
You really have and shared the "know-how"! :)

Now I have a "find_duplicates.avs"
DirectshowSource("Test.avi",30,true,false,true)
ConvertToYV12()
WriteFileIf("dups.txt", "YDifferenceFromPrevious < .001", "current_frame")
drag/drop it into the graphedit or VirtualDub window and let it play for an hour or two.
Then I get the desired "dups.txt" file with the frame numbers of the duplicate frames. Thanks to you I do not need any longer to step manually through the avi.

Is there a way to trick avisynth to do its magic quicker than while playing the avs in realtime? To let it simply calculate and avoid the realtime replay?

[Gavino]

Is there a way to trick avisynth to do its magic quicker than while playing the avs in realtime? To let it simply calculate and avoid the realtime replay?
Avisynth just delivers frames on request, so it's the client application, not Avisynth, that controls the rendering speed.
In VirtualDub, you can select "Run video analysis pass" from the file menu to get your file written as quickly as possible.

[Cela]

@Gavino
Due to your great help I can better find now where potential duplicates spots are.

Would you please be as kind as to give me more specific help for how to use 'MVFlowFPS'.
Since I am a simple user I best learn by cloning an expert's example.

If you wanted to substitue the duplicate frames for example at these positions (out of a total of 374 detected duplicates)
8639
13526
18876
31076
41898
55427
59958
how would you proceed?
How would the avs read which you would use?
How would you use it in VirtualDub?
And if there were not few but many, e.g. 374, duplicates?

@ all doom9 experts:
What about a plugin which could do these jobs at a mouse click?

> for all frames in the video
>> while not end-of-file get next frame
>>> if duplicate then
>>>>> substitute by interpolation between previous and next frame (if there is a next frame)
>>>>> output the subtitution frame
>>> else directstream copy output the original frame
>>> endif
>> endwhile
> endfor

As a user I can dream about it but never program it myself.

Thanks for your great help! :)

[Gavino]

Would you please be as kind as to give me more specific help for how to use 'MVFlowFPS'.
Actually, I realise now it's MFlowInter that you need to use for this purpose. See this thread for an example which is basically the same as your problem.
And if there were not few but many, e.g. 374, duplicates?
For many replacements, it's more practical to use ConditionalReader (http://avisynth.org/mediawiki/ConditionalReader) and ConditionalFilter (http://avisynth.org/mediawiki/ConditionalFilter) instead of trimming and splicing the different bits together as done in the referenced thread. Putting the two together, you might end up with a script like:
AVISource("c:\test.avi") # or MPEG2Source, DirectShowSource, some previous filter, etc
super = MSuper()
backward_vectors = MAnalyse(super, isb = true, delta=2)
forward_vectors = MAnalyse(super, isb = false, delta=2)
inter = MFlowInter(super, backward_vectors, forward_vectors, time=50, ml=70)
ConditionalFilter(inter, last, "dup", "==", "true")
ConditionalReader("file.txt", "dup")
where file.txt contains a list of the duplicate frames in the form:
Type Bool
Default false

8639 true
13526 true
... etc ...
This file can be easily generated from your previous one (or better still, modify the first script to generate it directly in this form).

Actually, if the duplicate detection is 100% reliable and does not require manual inspection to weed out false positives, you could get rid of the intermediate file step and simply automate the entire process in a single script:
...
inter = MFlowInter(...)
ConditionalFilter(inter, last, "YDifferenceFromPrevious", "<", ".001")

[Cela]

@Gavino
Thank you very much for your quick response and the valuable suggestions and links. :)

I will start with your avs for 100% reliable duplicate detection.

AVISource("c:\test.avi") # or MPEG2Source, DirectShowSource, some previous filter, etc
super = MSuper()
backward_vectors = MAnalyse(super, isb = true, delta=2)
forward_vectors = MAnalyse(super, isb = false, delta=2)
inter = MFlowInter(super, backward_vectors, forward_vectors, time=50, ml=70)
ConditionalFilter(inter, last, "YDifferenceFromPrevious", "<", ".001")


But I may be still missing (or nor really understanding) this:
How should I execute this avs? In VirtualDub with "Run video analysis pass"?
Which command in the avs will write the output? To which file will it be written? Where will I find it?

[Gavino]

How should I execute this avs? In VirtualDub with "Run video analysis pass"?
Which command in the avs will write the output? To which file will it be written? Where will I find it?
Once you have loaded this script into VirtualDub, it is (as far as VDub is concerned) just another video file - that's the magic of Avisynth. So just save the output via VirtualDub in the usual way to a file of your choice. Note that you cannot use Direct Stream Copy when the input is an Avisynth script, as Avisynth serves uncompressed video to its client. So select an appropriate compression (eg xvid) for your output.

The "Run video analysis pass" was used with the previous script simply as a substitute for saving (or playing) the video, in order to get the text output more quickly.

[Cela]

Sorry, another question:
What is 'last' in
ConditionalFilter(inter, last, "YDifferenceFromPrevious", "<", ".001")

Would this result be expected for left person at frame 5 in example (too big for attachment, please see PM).

[Gavino]

What is 'last' in
ConditionalFilter(inter, last, "YDifferenceFromPrevious", "<", ".001")
You need to invest some time in reading up on the basics of Avisynth.
'last' is a special variable in the script language - if the result of a statement is a clip and is not explicitly assigned to a variable, it will be assigned implicitly to 'last'.
In the script above, 'last' is the result of the AVISource call, ie the original video. The script could also have been written as
source = AviSource(...)
...
ConditionalFilter(inter, source, "YDifferenceFromPrevious", "<", ".001")
Would this result be expected for left person at frame 5 in example (too big for attachment, please see PM).
Looking at the script again, I see that it is out by one frame when selecting from the interpolated clip, since frame n of inter corresponds to frame n+1 of the source (interpolated from n and n+2). A fix is to replace the last line by
ConditionalFilter(inter.Loop(2,0,0), last, "YDifferenceFromPrevious", "<", ".001")
(which adds an extra frame at the start of inter to line up with the source).

[Cela]

@Gavino
I am very sorry but I still do not understand what
ConditionalFilter(inter.Loop(2,0,0), last, "YDifferenceFromPrevious", "<", ".001")
is supposed to do exactly.

I read
http://avisynth.org/mediawiki/ScriptClip and
http://avisynth.org.ru/docs/english/corefilters/conditionalfilter.htm

but do not understand what I read. The more I read the more I get confused (I'm just a silly :confused: user):

First, there are more clips in the function definition
ConditionalFilter (clip testclip, clip source1, clip source2, string expression1, string operator, string expression2, bool "show")
as in our usage of that definition.
I cannot really match the entries:
testclip ~ inter.Loop(2,0,0) ?
source1 ~ last ?
source2 ~ ???

or is it more like this:
testclip ~ 'last', clip from DirectShowSource ?
source1 ~ inter.Loop(2,0,0) ?
source2 ~ last

Second, why are there 3 clips in the function definition? Isn't it just a choice (either/or) between to options source1 and source2?

Third, what exactly means 'inter.Loop(2,0,0)' ? I thought inter[/]was just a frame-[i]value returned by the MFlowInter function, is it a clip-vector? Where can I read an explanation of this syntax construct?

Fourth, does last mean the clip from DirectShowSource in our avs ?

Please forgive my ignorance. Do not forget, I am only a user. I know how to eat but not how to cook. But, what you cook tastes well! ;)

Your avs effectively seems to work very well.
My full size test with an almost 3 hour Total Recorder recording looks very good after VirtualDub processing with your avs. :thanks:

P.S.
I did some more reading and tried to figure out what the interpolation avs

DirectshowSource("dupat5.avi",30,true,false,true)
super = MSuper()
backward_vectors = MAnalyse(super, isb = true, delta=2)
forward_vectors = MAnalyse(super, isb = false, delta=2)
inter = MFlowInter(super, backward_vectors, forward_vectors, time=50, ml=70)
ConditionalFilter(inter.Loop(2,0,0), last, "YDifferenceFromPrevious", "<", ".001")

might do:
When I suppose for example, that frames 4 and 5 are duplicates
...aac...
123456789
then "YDifferenceFromPrevious", "<", ".001" is true for frame 5.
At that moment delta=2 back_vectors would point from frame 5 to 3,
forward_vectors from frame 5 to 7?
Thus, clip 'inter' would represent the MFlowInterpolation between frames 3 to 7 ?
But I have the feeling, that my interpretation is wrong and does not make sense.
Especially, it does not lead me to the necessity for inter.Loop(2,0,0). inter alone should do, but obviously inter.Loop(2,0,0) brings better results.

I would be so happy if I could learn to correctly understand this avs.

[Gavino]

or is it more like this:
testclip ~ 'last', clip from DirectShowSource ?
source1 ~ inter.Loop(2,0,0) ?
source2 ~ last
Yes. The other important feature of 'last' in Avisynth is that if a function has a clip as its first parameter, then 'last' will be used as the default value. Since the function requires 3 clips and only 2 are provided, 'last' is used for testclip.
Second, why are there 3 clips in the function definition? Isn't it just a choice (either/or) between to options source1 and source2?
The choice (for each frame) is between source1 and source2, but for maximum flexibility, the choice can be made based on the properties of a third 'test' clip. In many cases (as here), the testclip used is the same as either source1 or source2.
Third, what exactly means 'inter.Loop(2,0,0)' ? I thought inter was just a frame-value returned by the MFlowInter function, is it a clip-vector?
There are no frame values as such in the Avisynth language, only entire clips. Frame n of clip 'inter' contains the results of the interpolation between frames n and n+2 of the source. That's just the way MFlowInter works, when we use time=50 and vectors with delta=2.
Fourth, does last mean the clip from DirectShowSource in our avs ?
Yes.
When I suppose for example, that frames 4 and 5 are duplicates
...aac...
123456789
then "YDifferenceFromPrevious", "<", ".001" is true for frame 5.
Right. So ConditionalFilter will choose frame 5 of source1. We want this to be the frame interpolated from frames 4 and 6 of the source, which means frame 4 of inter. So we set source1 to inter.Loop(...), repeating frame 0 to put frame 4 of inter in frame 5 of source1.

The Avisynth wiki (http://avisynth.org/mediawiki/Main_Page) contains extensive documentation, including a full language reference (http://avisynth.org/mediawiki/AviSynth_Syntax). There is similar documentation (in different languages) installed with Avisynth, usually in C:\Program Files\AviSynth 2.5\Docs.

[Cela]

@Gavino
Thank you for your patience and your comprehensive explanations. Be sure, that in the last days I have read starting from the wiki portal all the avisynth reference information pertaining your avs.
But what I am lacking and did not find is the basic informations needed to understand the reference descriptions.
Your explantions have provided some of the missing basics. :)

Another fundamental basic which I still do not know for sure yet is how and when the command lines in the avs are executed and, more important, when the data processing results of such a statement is available.
I guess it is non-procedural. You seem to be able to assume that each statement will have all data available when needed which is defined in another statement. If, for example, in the DirectShowSource there is a 100,000 frame avi, there will be in your avs all values for backward_vectors, forward_vectors, inter and last available when needed in ConditionalFilter for any of these 100,000 frames. By this scheme MFlowInter gets the correct values of backward_vectors and forward_vectors. Is this so?

Would you please also comment, confirm or reject, my interpretations of the isb explanation in the MVTools2 reference concerning MAnalyse:
isb=false … forward search (motion from the previous frame to current one)
(for example: currrent=5 -> previous=3 when delta =2)
isb=true … backward search (motion from following frame to current one)
(for example: current=5 -> following=7 when delta=2)

When I suppose for example, that frames 4 and 5 are duplicates
...
Thus, clip 'inter' would represent the MFlowInterpolation between frames 3 to 7 ?
Is this correct?

forward 3 --> 5 <-- 7 (backward from 7) ?
For current = 5 all necessary values for frames 3 to 7 are needed and available?
And more general, for any value of current, any value needed by a function would be provided by the other referenced functions?

Then, inter.Loop(2,0,0) can shift inter 3 to 7 by one to inter 4 to 8 and doing this puts frame 4 of inter in frame 5 of source1 because these values are available when needed for current=5 in the example?

Why do you use ml=70 in MFlowInter where the default is ml=100? What is the advantage?

Your avs work great but how to use the results is now an issue. What concerns me is that in order to get the interpolations for the ~1% of duplicates make it necessary to recompress the whole avi, including the 99% of non duplicate frames, which would not need any processing.
I feel that the new avi pays the cleaning from duplicate frames by visible loss of overall quality due to recompression. Its not any easy decision whether to keep the original avi and live with the glitches or to take the new one without the glitches and accept that it looks a litte washed out.

With VirtualDub it is theoretical easy to make manually a 'best-of' the two versions but it is practically feasable only for ~.001% - ~.01% of duplicates. For realistic numbers of duplicates an automated procedure of the manual VirtualDub workflow would be necessary.

Because this is a different issue, I address it in a different thread and would be happy to get your valuable input there, too.

To sum it up, your two avs, to find and make a list of where the duplicate frame glitches are and how to correct them by interpolation, work great and its a real advantage to have them available for use! :)
Your explanations of the Avisynth basics helped me a lot and IMHO would be worth to be published in the Avisynth reference and would help beginners to better understand the reference.

:thanks:

[Gavino]

Another fundamental basic which I still do not know for sure yet is how and when the command lines in the avs are executed and, more important, when the data processing results of such a statement is available.
When starting to understand Avisynth scripting, it's simplest just to regard each filter as producing an entire clip, all of which is then available to later (but not earlier) parts of the script, either explicitly (after assignment to a variable) or implicitly (via 'last', until overwritten). The clever part is that internally Avisynth only creates frames when needed, so if you create a 100,000 frame clip and then only select a small part of it, the remaining frames will never really be created. (Incidentally, that's why the WriteFile script doesn't produce any text until VirtualDub starts pulling frames from the clip to play, analyse or save it.)

The full story is a bit more complicated, especially once run-time filters (http://avisynth.org/mediawiki/Runtime_environment) come into the picture. If/when you want to know the full details, see here (http://avisynth.org/mediawiki/The_script_execution_model) - you will probably need to read this several times before it makes sense, I know I did when I first started.

Your interpretation of isb and the vectors is correct, but their use to derive 'inter' is as follows:
because we have asked MFlowInter for frames 50% between deltas of 2, it will (for frame 4 of inter) interpolate between frames 4 and 6 of the source, using both the forward vector 4->6 and the backward vector 6->4, which are both available in the corresponding vector clips. Note that it does not use vectors derived from the original frame 5 (eg 3->5 or 7->5) here - that's good, since we want to replace that frame!
Why do you use ml=70 in MFlowInter where the default is ml=100? What is the advantage?
I don't know - I just copied that from the other thread. :)

[Cela]

...The full story is a bit more complicated, especially once run-time filters (http://avisynth.org/mediawiki/Runtime_environment) come into the picture. If/when you want to know the full details, see here (http://avisynth.org/mediawiki/The_script_execution_model) - you will probably need to read this several times before it makes sense, I know I did when I first started. ...Yes, several times.... I did it again last night for the MFlowInter command. This morning, at breakfast, I got the inspiration: my fault was, I overread the details in "Motion interpolation function between CURRENT and NEXT (by delta) frame" (which I highlighted here). It produces the NEXT and NOT the CURRENT when executing it for the CURRENT.
I misunderstood it wrongly in this way: that it would use current and next (by delta) frame to produce the current, which did not make sense.

It really takes very careful reading in the struggle for deaper understanding of the wonders af Avisynth.

You gave me a good start. :thanks:

[bcn_246]

@Gavino
Thank you very much for your quick response and the valuable suggestions and links. :)

I will start with your avs for 100% reliable duplicate detection.

AVISource("c:\test.avi") # or MPEG2Source, DirectShowSource, some previous filter, etc
super = MSuper()
backward_vectors = MAnalyse(super, isb = true, delta=2)
forward_vectors = MAnalyse(super, isb = false, delta=2)
inter = MFlowInter(super, backward_vectors, forward_vectors, time=50, ml=70)
ConditionalFilter(inter, last, "YDifferenceFromPrevious", "<", ".001")


But I may be still missing (or nor really understanding) this:
How should I execute this avs? In VirtualDub with "Run video analysis pass"?
Which command in the avs will write the output? To which file will it be written? Where will I find it?

Could anybody re-write this script to use SVPflow instead of MVTools?