How do I find duplicate frames?

[dragon25]

hi folks, is there a specific program that you put the input file into it, for it to report the dupe frames, or do all you guys literally go through the frames by hand and count them.

maybe a silly question but needed to ask.

[Guest]

Fix your title or you'll get struck. Is it so hard to say "How do I find duplicate frames?" Select Edit on your first post and then Go Advanced.

Your title needs to summarize your issue, and not leave people guessing about what your thread is about.

One option is to use MultiDecimate. It produces some statistics files that you can use to detect duplicates. Be warned, blind global decimation of duplicates is not a good idea, as it can ruin audio sync and delete static scenes.

[dragon25]

i only seem to be able to edit the post itself?

[Guest]

Select Edit on your first post and then hit the Go Advanced button. There will be a text field with the thread title that you can modify. Thank you.

[dragon25]

thanks done..

[Guest]

Did you have a look at that thread I pointed you to over at my forum?

[dragon25]

yes but i do suffer a lot with having my brain take in information

all i could really "see" from the page if it was the correct one, was a lot of duscussion and some scripting. maybe as no doubt i have ....completely missed it

thanks for your patience

Ok...........

foud the mulidecimate page and thing to download.

so followed the steps, but got an issue when i try to load the avs

Loadplugin(D:\multidecimate107\MultiDecimate.dll)
AVISource("H:\xxxxx\5_d2v_vfapi.avi")

MultiDecimate()


vdub error

avisythn open failure
script error:expected a, or)
(XXXXX\XXX\xxx\1.avs,line 1, column, column 13)

ok after reading it about dos and not knowing how to do that, with it saying load it into the avisyth script, i also tried this



Loadplugin(D:\multidecimate107\MultiDecimate.dll)
D:\multidecimate107\ProcessMD.exe
AVISOURCE("H:\xxxxx\5_d2v_vfapi.avi")
ConverttoYUY2
MultiDecimate()

[Guest]

In Loadplugin(), you have to put your path in double quotes.

Why is your AVI called 5_d2v_vfapi.avi? Why did you include _vfapi in the filename when you made it?

[dragon25]

becuase i thought i needed to, or do i just add the d2v file

i place the dll file into medainfo to gain the complete adress and thats what i was given, or do i do it from the avisythn plugin folder?

[Guest]

Where did you get the AVI file?

[dragon25]

i used dgindex to gain the d2v from the dvd.

will this not work that way, will it only work on avi's as in pure avi's not vfapi



Loadplugin("C:\Program Files\AviSynth 2.5\plugins\MultiDecimate.dll")

avisource("H:\xxxxx\5_d2v_vfapi.avi")
ConverttoYUY2
MultiDecimate()

this time it loaded and is now playing from what i understand i now leave it playing untill the end then straight away close the applicaiton down, which will automatically do its thing in th avisythn script .

also i was trying to do it in the application it states in the help file, but it kept crashing o im doing it via virtualdubmod 1.5

[Guest]

Why are you using VFAPI frame serving instead of MPEG2Source()? It will work that way but VFAPI is deprecated.

[dragon25]

MPEG2Source()

so with that would the d2v file just be in the brackets

It will work that way but VFAPI is deprecated.

what do you mean by the last word as im thinking you mean no longer works?

ok gained my txt file, opened the mulitidecimate thing and now i have threshold and run length.

now since im the type that has never understood those maths questions such as what is the amoutn from y-z = z to come from z+f

well thats what the explantion of these too looks to me and im baffled.

it says runlength is the number of duplicates ths defined ina row that defines a static scene subject to protection?

so is number 15 enough or what as i just dont get this....most lielly will when the obvious is pointed out.

and if im to look at the txt file dof duplicate amounts, this will be near imposible i think due to the fact the numbers are 1234567 etc, there all over the place.

[Guest]

MPEG2Source()

so with that would the d2v file just be in the brackets Please read the documentation for DGDecode and the Quick Start guide.

It will work that way but VFAPI is deprecated.

what do you mean by the last word as im thinking you mean no longer works? It means it is recommended not to use it because there are better ways available now, i.e., MPEG2Source().

it says runlength is the number of duplicates ths defined ina row that defines a static scene subject to protection? Yes, it says that.

If you are not comfortable with numbers, I don't see how to help you further. Sorry.

[dragon25]

its not that im not comfortable with them, i just need a slight nudge to know what somehtign means at times.

i mean does the 15 mean , frames/secounds/minute
and the treshold aswell.

[Guest]

i mean does the 15 mean , frames/secounds/minute It is referred to as the number of duplicates in a row, so isn't it obvious that it is a frame count?

and the treshold as well. The documentation says this:

Frames with a metric less than threshold will be considered to be duplicates. Examine mfile.txt to examine the frame metrics.
and

To address this, use the "protect static scenes" mode. When you choose this mode from the drop down list, two extra edit boxes appear. Threshold is the metric below which duplicates are declared. The metrics are the same as those used in the mfile.txt file, so you can use that to help you. The fields in mfile.txt are as follows:

FrameNum Metric

Run Length is the number of duplicates (thus defined) in a row that defines a static scene subject to protection. A protected static scene is decimated by the ratio remove/cycle, no more and no less (subject to integer truncation). What part of this do you not understand?

[dragon25]

blimney mate, do you not realise that there are people that may actually not understand somethign as simple as you understand it, you have had experince in this field for quite some time and im only just starting, and isnt it obvious that i dont understand it via the way ive asked for help to understand it?

like i said before, i hve looked at the numbers and for myself can not see any duplicates in simple number counts like 1,1,2,3,4,4

ie duplicates of same numbers.

the numbers i see are different amounts all the time and not in order, which itself i foudn strange as i thought the frame number would be as above.

also the document that i see , i could read but not understand , becuase like before to me its like being told to answer a maths questions like whats the amoutn from z+y = z with the end count being less than z

i was never good at those type of things and never will be, hence why im asking for some help. and if you are nto able to help in this matter, could you point me to somebody that could.

thank you

[Guest]

All I did was ask you what part you didn't understand so that I could explain it better. You're not expecting me to write a "total noob" version of the documentation just for you, are you?

Suppose you see this in your mfile.txt coming out of pass 1 of MultiDecimate():

7 1.390740
8 1.411184
9 1.292269
10 1.507268
11 1.427170
12 1.761919
13 1.698445
14 1.499013
15 1.324455
16 2.109136
17 2.119017
18 1.907831
19 2.119815
20 0.200000
21 2.534512
22 3.269930
23 3.560910
24 3.518068
25 3.933570
26 4.246012
27 4.313966
28 4.490195
29 3.743083
30 2.885745
31 3.148442
32 3.467064
33 3.905692

For example, frame 27 has a metric of ~4.31. It is a number that measures the difference between frame 27 and frame 26. The bigger the number, the more different the frames are. So for a duplicate frame, you expect a small number. Look at frame 20. It has a value of 0.2. It is almost certainly a duplicate.

It's not easy to know what number is small enough to be a duplicate, because the scene could develop slowly, there could be more or less noise, etc. So you have to decide the number which determines what a duplicate is. That is called the "threshold". The way to determine it is to look at the frames and decide which are duplicates. Then look at the numbers in the mfile.txt and work out a threshold that distinguishes the real duplicates.

I hope this helps you.

[dragon25]

thank you for that part, and i do mean from previous to say i didnt really get any of it hence why it was not possible to point out a singl item.


i have noticed in the txt that i have a few 0.0000000 type numbers

so how do i configure that into the threshold? woudl the above number mean the threshold would be 0.0?

and is the run legnth the amount of frames as in the left hand column numbers, between each dupe frame? so it could be like 40

therefore taking dupe frame ever 40 frames?

i might be going way off here but im tryignt o get my head aroudn this.

thanks for you help

surpsied that the help file with this item doesnt explain in "plain" english so mere mortals like myself can understand them

[Guest]

thank you for that part, and i do mean from previous to say i didnt really get any of it hence why it was not possible to point out a singl item. No problem.

i have noticed in the txt that i have a few 0.0000000 type numbers That means the frames are EXACTLY the same. But you can also have duplicates that are not exactly the same, but are a little different due to noise. They would have small numbers.

so how do i configure that into the threshold? woudl the above number mean the threshold would be 0.0? You'd want to pick some small number above 0.0. If all your dups are 0.0, then you could use 0.2 or something like that. But make sure that you don't have other dups that are not exact, so that you can set the right threshold to catch them too.

and is the run legnth the amount of frames as in the left hand column numbers, between each dupe frame? so it could be like 40 Not between each dup frame. The idea is that if you just remove all dups then you'll remove scenes where nothing is happening, like a guy standing still for a few seconds. So you say, well gosh, if there are more than 5 dups in a row, it's really a static scene that I don't want to be removed. So your run length would be 5.

surpsied that the help file with this item doesnt explain in "plain" english so mere mortals like myself can understand them It's a priesthood thing. :) When you master it, you'll be a member of the priesthood.

[Amrit]

Good thread

[dragon25]

what about the run length.

and wih the frames, is there a way to go to the frame number in the item to see if its the dupes you can see.

[Guest]

what about the run length. I described that in my post above. What is your remaining question about it?

and wih the frames, is there a way to go to the frame number in the item to see if its the dupes you can see. Open the script (without MultiDecimate) in VirtualDub. You can scroll through the video and you'll see the frame number in the status area.

[Amrit]

What can be done with finding these Dupe frames?? to remove them? Merge them as one if possible?

[dragon25]

I described that in my post above. What is your remaining question about it?

Open the script (without MultiDecimate) in VirtualDub. You can scroll through the video and you'll see the frame number in the status area.

must have missed the length answer from you.,

well thats what i thought about the frme number, but like i have said before i thought the frame number would go lower to higher numbers, but it doesnt its all mixed so therfore you couldnt locat the frame number according to the file.

[Guest]

well thats what i thought about the frme number, but like i have said before i thought the frame number would go lower to higher numbers, but it doesnt its all mixed so therfore you couldnt locat the frame number according to the file. Huh? If you open your script (without MultiDecimate) in VirtualDub, it will open showing Frame 0. Then single step forward and it will go up by one. What do you mean "it's all mixed up".

[dragon25]

ill show you the txt ina few hour as i messsed up first time, i didnt do the thing via pass 1, so the pass 2 couldnt have gone forward lol.

ah so length will be dupe frame in a row, next to each other

so if the video cam up against a part where it showed the same bit ie if say it was an action scene, the the same frame was still there few frames later.

therfore you cont the dupes in that part to be the length of run.

[Guest]

ill show you the txt ina few hour as i messsed up first time, i didnt do the thing via pass 1, so the pass 2 couldnt have gone forward lol. You don't even need pass 2 for what you are trying to do. Use MultiDecimate(pass=1) to generate your mfile.txt. Then remove MultiDecimate from your script and open the script in VirtualDub. You will now be able to match the frame numbers in the mfile.txt to those you see in VirtualDub. You need pass 2 only if want to actually decimate the source file. Do not try to match the frame numbers in mfile.txt against the video you see with MultiDecimate(pass=2) in your script, because that will decimate frames. And of course don't try to do it with MultiDecimate(pass=1) in your script either, because that will overwrite your mfile.txt! As I said, remove MultiDecimate from your script to do this comparison, after generating mfile.txt.

ah so length will be dupe frame in a row, next to each other Yes.

so if the video cam up against a part where it showed the same bit ie if say it was an action scene If there is action, then it won't be dups! When there is no action, there are dups and we call it a static scene.

therfore you cont the dupes in that part to be the length of run. That's right.

[dragon25]

Yes.

If there is action, then it won't be dups! When there is no action, there are dups and we call it a static scene.

That's right.

no i mean when you come across a active scene and there are dupe frames , say if somebody is running and suddenly the person stops moving when infact they should still be mvoing.

if that makes sense.

[Guest]

There could be dups in such a case if they have been added to adjust the frame rate. Usually, there are only 3 or less dups for that purpose. But in the file I saw from you earlier there was a pathological run of 10 dups for no reason. If you set your run length to 10 so that the bad run of dups is decimated, you'll also decimate all real static scenes of 10 frames in length or shorter. Those longer than 10 frames won't be touched. That may be the price you have to pay to process such screwed up video. Or you can delete such runs manually in VirtualDub. That will allow you to keep a reasonable run length and avoid screwing up audio sync.

[dragon25]

well i know ill have audio sync issues anyways.

ok the txt file is too big for the upload.
so heres part of it

Now are the frame numbers on the left column as i was thinking about this just a moment ago, as i did original think the left column was just the amount of lines there were.

if on the left is in fact the frames, what are the numbers on the right column

0 0.000000
1 0.000000
2 0.000000
3 0.114294
4 0.091695
5 0.095194
6 0.091592
7 0.088515
8 0.087058
9 0.091397
10 0.090773
11 0.089332
12 0.101760
13 0.087837
14 0.089564
15 0.089161
16 0.091020
17 0.123046
18 0.101656
19 0.102208
20 0.114605
21 0.092798
22 0.144581
23 0.107872
24 0.092385
25 0.080956
26 0.100095
27 0.090759
28 0.073367
29 0.103174
30 0.112767
31 0.087889
32 0.147273
33 0.112632
34 0.092913
35 0.100898
36 0.093654
37 0.086066
38 0.108849
39 0.127477
40 0.162655
41 0.121877
42 0.185131
43 0.173662
44 0.154632
45 0.162065
46 0.126799
47 0.187406
48 0.170378
49 0.190465
50 0.192425
51 0.149110
52 0.194884
53 0.191552
54 0.174244
55 0.172637
56 0.159206
57 0.194552
58 0.175283
59 0.191931
60 0.198903
61 0.142514
62 0.245452
63 0.208401
64 0.195551
65 0.190951
66 0.173918
67 0.183951
68 0.189502
69 0.200199
70 0.210938
71 0.169353
72 0.221016
73 0.212599
74 0.218743
75 0.208635
76 0.160189
77 0.247584
78 0.217085
79 0.234327
80 0.241526
81 0.181081
82 0.261626
83 0.248536
84 0.240334
85 0.263389
86 0.200820
87 0.295159
88 0.310723
89 0.358444
90 0.395118
91 0.206345
92 0.434275
93 0.378998
94 0.367546
95 0.355182
96 0.222467
97 0.350964
98 0.372574
99 0.377935
100 0.393309
101 0.242180
102 0.414096
103 0.419288
104 0.420315
105 0.420689
106 0.246041
107 0.474126
108 0.499764
109 0.625426
110 0.725286
111 0.260564
112 0.847457
113 0.957314
114 0.999180
115 1.068842
116 0.297189
117 1.116291
118 1.064448
119 1.059526
120 1.039573
121 0.299059
122 1.065539
123 1.011647
124 0.988037
125 0.935137
126 0.298703
127 1.020251
128 1.033772
129 1.021767
130 1.110957
131 0.303834
132 1.206676
133 1.228204
134 1.352726
135 1.556843
136 0.342809
137 1.512385
138 1.617760
139 1.887899
140 1.827347

[Gavino]

Go back to post#18 and read it again.

In fact, go back and read everything again - sometimes these things take a while to sink in...

[dragon25]

pardon...

why is there somethign that has gone wrong since neuron2 left a comment?

[Gavino]

Post #18 (on the previous page) was where neuron2 explained that the numbers in the right-hand column are the 'metrics'.

Now that you understand a bit more, you may find that going back to the beginning and reading everything again, it will start to make sense. Or maybe not.

[setarip_old]

@dragon25

Hi!

If, as "Gavino" so properly suggests,you RE-read post #18 in this thread (posted by "neuron2"), you'd see that the question you ask aboput the "numbers to the right of the frame numbers" has already been answered:For example, frame 27 has a metric of ~4.31. It is a number that measures the difference between frame 27 and frame 26. The bigger the number, the more different the frames are. So for a duplicate frame, you expect a small number. Look at frame 20. It has a value of 0.2. It is almost certainly a duplicate.

[dragon25]

a so the right hand ciolumn is the metric, ah didnt get that before.

so for a dupe frame i need to locate the same number that is repeated.

but even with this , surely since the threshold and run seems to be of guessed luck, that i may very well cock this up before i even start.

[Guest]

a so the right hand ciolumn is the metric, ah didnt get that before. Gosh I told you this:

"The fields in mfile.txt are as follows:

FrameNum Metric"

and I gave you a simple example. What more do you need? How can I do better than to explain it in simple English and give you an example? Really.

so for a dupe frame i need to locate the same number that is repeated. No, you need to read my simple example again.

but even with this , surely since the threshold and run seems to be of guessed luck, that i may very well cock this up before i even start. No, it's not a matter of luck. I explained how to rationally set those parameters. They may not operate perfectly, because the algorithm is heuristic. But that's always the case when you want to automate a process that normally requires a human mind.

[dragon25]

your slow, blimey what the hell am i then, took me 2 years to figure out how to do 2 pass xvid's and that was with me doign that myself lol.


when there is a lot of information, my brain suffers badly to take it all in even if re read it over and over again, i guess my dyslecixa is part of that reason, which is why im always polite to anyone is helping me as i understand the patience needed to explain somethign that could very well be the most simple and obvious thing in the world.

but thankfully from your patience with me ive gotten further than ever before. so yes im bloody clueless but witha bit of paitence i start to have a clue.

currently googling bad frame removal with vdub thanks for the sujjestion.

but like in the pm, im simple baffled with th metric numbers , becuase if im looking for simlair numbers to be dupes when the frames i have are all dupes, then the numbers im looking at are all different, which i will show you an example

1.886 0.570783
887 0.015137
888 0.313005
889 0.003329
890 0.282401
891 0.279973
892 0.000000
893 0.277962
894 0.020348
895 0.000558

2.1930 0.430989
1931 0.037576
1932 0.244719
1933 0.000000
1934 0.070475
1935 0.123657
1936 0.000000
1937 0.151012
1938 0.121118
1939 0.000000
1940 0.007757

3.2945 0.333341
2946 0.186551
2947 0.000000
2948 0.048572
2949 0.047730
2950 0.000000
2951 0.018525
2952 0.028153
2953 0.000000
2954 0.012266
2955 0.019740

3.3942 3.307272
3943 0.431294
3944 3.172773

4.4913 0.467535
4914 0.337175
4915 0.021481
4916 0.252816
4917 0.248196
4918 0.000000
4919 0.000967
4920 0.131294
4921 0.000000
4922 0.134662

this is where i have spotted the obvious dupe frames and taken the data from the previous multidecimate pass 1

[Guest]

The metrics for duplicates can be different, due to noise and other things. The point is that the numbers will be SMALL. The numbers for frames that are not dups are BIGGER. You have to choose the threshold to be right at the place where any number you see for real duplicates is smaller.

It's not possible to make much sense of your data without seeing the unprocessed source clip that corresponds to it. But please do not post a link to illegitimate material, and I really hope you are not discussing the video for which you have already had a thread closed for rule 6.

All your numbers are low (below 1.0) except for frames 3942 and 3944. Are you sure those are dups? If not then you could set a threshold at 1.0.