Delete Dups

[jmac698]

I have a video where sometimes there's a single dup after a frame. How can I remove this?

scriptclip("""
YDifferenceFromPrevious<.1?deleteframe(current_frame):last
""")

Doesn't work, I'm stuck on how to write it...

[Chikuzen]

DeDup.
http://akuvian.org/src/avisynth/dedup/

[StainlessS]

DeDup as suggested above will probably sort you out, but also, if you have any other weird reasons
for deleting frames (other than duplicates) then the below might come in handy to create a command
file for Prune(), where you list the frames to keep rather than chuck. You would though probably want
to only use it as last resort.


# Edited:-
avisource("d:\avs\avi\1.avi").ConvertToYV12()

Writefileif("d:\avs\avi\1.txt","YDifferenceFromPrevious>=0.1 || current_frame==0",""" "0," ""","current_frame",append=false)

[jmac698]

Thanks, I'm trying dedup now, however the log file is empty except this line:

DeDup 0.17 by Loren Merritt, based on Dup 2.20 beta 1 by Donald Graft/Klaus Post, Copyright 2004

I'm surprised there isn't a simple way to do this in avisynth??

[Chikuzen]

Thanks, I'm trying dedup now, however the log file is empty except this line:


DupMC needs analysis pass.
use VDub's 'Run video analysis pass'

[Gavino]

I have a video where sometimes there's a single dup after a frame. How can I remove this?

scriptclip("""
YDifferenceFromPrevious<.1?deleteframe(current_frame):last
""")

Doesn't work, I'm stuck on how to write it...
Deleting frames with ScriptClip is tricky because to make the deletions cumulative, you need to keep track of what has already been deleted in previous frames. A similar problem was discussed in this thread, where I described the solution, which could be adapted for your case.
The recursive procedure could now be replaced by a GScript 'while' loop (not available at the time).

DeDup is probably a simpler way if it works for your source.

[jmac698]

Ok, fixed the problem with the log file, now I can't get all the frames dedeupped, it's quite clear the threshold. I find the docs confusing. If I could do multiple passes I might get it to work, but it's quite awkward.

I tried Gavino's approach, no luck either, it's taking away too many frames.

function DarkFrames(clip c, int current_frame, float thresh) {
# returns the number of consecutive 'dark' frames at the current frame
c.YDifferenceFromPrevious< thresh ? 0
\ : (current_frame >= c.frameCount-1) ? 1
\ : 1 + DarkFrames(c.Trim(1,0), current_frame, thresh)
}

dir="D:\project001a\dedup fields\"
DirectShowSource(dir+"jvc-with-ext-tbc.mov")
SeparateFields
selectevery(2,0)
converttoyv12

threshold = 2.6 # <- change this to suit requirements
offset = 0
# assume 'last' set earlier in script
ScriptClip("""
subtitle(string(YDifferenceFromPrevious))
offset = offset + DarkFrames(Trim(offset, 0), current_frame, threshold)
Trim(offset, 0)
""")

[jmac698]

Finally I tried Stainless approach. It seems to be working. Stainless, could you add a command to specify the list of deletes?
This type of editing seems tremendously difficult in avisynth.

[Gavino]

I tried Gavino's approach, no luck either, it's taking away too many frames.

function DarkFrames(clip c, int current_frame, float thresh) {
# returns the number of consecutive 'dark' frames at the current frame
c.YDifferenceFromPrevious< thresh ? 0
\ : (current_frame >= c.frameCount-1) ? 1
\ : 1 + DarkFrames(c.Trim(1,0), current_frame, thresh)
}
You've got your condition the wrong way round - the function should return the number of frames you want to delete. Renaming it and changing the comment to suit your purpose, it should be:
function DupFrames(clip c, int current_frame, float thresh) {
# returns the number of consecutive duplicate frames at the current frame
c.YDifferenceFromPrevious > thresh ? 0
\ : (current_frame >= c.frameCount-1) ? 1
\ : 1 + DupFrames(c.Trim(1,0), current_frame, thresh)
}
You also want to comment out the subtitle for the real run, since it will affect the YDifference value.

[Gavino]

This type of editing seems tremendously difficult in avisynth.
Tricky yes, but I doubt any of your fancy NLE's will give you the ability to delete an arbitary set of frames according to some user-defined criterion based on frame contents. The power and flexibility of Avisynth is shown by examples like this.

[StainlessS]

Jmac, eg below should work (or whatever you called the command file.

Prune(cmd="D:\avs\avi\1.txt");

Edit, if you use the Beta version you may want to add "fade=0.0", to switch off the audio fading.

[jmac698]

Yep I used

prune(cmd="keeplist.txt")

[StainlessS]

Hi Jmac, just want to make sure you saw the previous edit.

[jmac698]

Nope missed that, thanks.

[Gavino]

Writefileif("d:\avs\avi\1.txt","YDifferenceFromPrevious>=0.1",""" "0," ""","current_frame",append=false)
Note that YDifferenceFromPrevious() will always return 0 for the first frame (no previous!), so you need a special case for current_frame=0 in the 'if' condition, otherwise frame 0 will always be deleted:
..., "YDifferenceFromPrevious>=0.1 || current_frame==0", ...

A similar change would also apply if using my ScriptClip method.

[sven_x]

Are you sure you want to delete the dups? When you're dealing with VHS grabbings the dups might be a simple replacement to fill the gaps where the capture device could not sync to a frame. When using USB video grabbers I had often series of 1...10 dups at places with problems in tape flow. In that cases the dups are 100% identical copies of the last frame.

In those cases I found that cutting off the dups can disturb the flow of motions, and also it leads to jumps in audio flow. If would be better to interpolate the missing frames from their neighbors using mflow. (If there is no scene transisition -- then interpolation will not make much sense).
The latter is exactly the problem I am working on, but I did not succeed in writing a script that replaces n1, n2 frames at positions x1, x2 ... using a list of duplicate frames from a ConditionalReader file.

[johnmeyer]

Are you sure you want to delete the dups? When you're dealing with VHS grabbings the dups might be a simple replacement to fill the gaps where the capture device could not sync to a frame. When using USB video grabbers I had often series of 1...10 dups at places with problems in tape flow. In that cases the dups are 100% identical copies of the last frame.

In those cases I found that cutting off the dups can disturb the flow of motions, and also it leads to jumps in audio flow. If would be better to interpolate the missing frames from their neighbors using mflow. (If there is no scene transisition -- then interpolation will not make much sense).
The latter is exactly the problem I am working on, but I did not succeed in writing a script that replaces n1, n2 frames at positions x1, x2 ... using a list of duplicate frames from a ConditionalReader file.I had this exact situation and with Didée's help eventually managed to create a script which not only detected the dups (that is relatively easy) but then also found the "jump" that occurred where a frame was missed.

Automatically fix dups followed (eventually) by drops (http://forum.doom9.org/showthread.php?t=161758&highlight=duplicated)

It turns out that in some capture situations, the capture card misses (drops) a frame. So, to get the audio back in sync, it later duplicates a frame (as sven_x says, this is a perfect duplicate). Thus, while you can easily find the dups and interpolate a frame at that point (search this forum for "deletedup"), that usually doesn't give you smooth video. Instead, you have to find the unwanted jump where a frame was not captured, and interpolate a new frame at that point (using mvtools2).

The beauty of Didée's approach, which I modified to get better detection with my particular source, is that it also "automatically" finds and removes the exact duplicates. Thus the duplicates are decimated, and new interpolated frames are inserted at the points which have the biggest "gap" in motion from one frame to the next.

The gap detection isn't perfect, but it was, for my source, pretty darned good (about 98% correct). I'm sure another smart person could take the script as a starting point and, while referring back to Didée's original masktools approach for finding gaps, combine the various ideas and perhaps come up with a detection scheme that is even better. The underlying "plumbing" is all in place, so you can spend you time just on doing better detection of the places where the frame was actually dropped.

Of course if the audio is NOT in sync, and you have to drop these duplicate frame to get back into sync, then all you need is the original "deletedups" function created many years ago. You'll find a link to that if you click on the hyperlink above.

[jmac698]

You're right, that is the underlying problem, but I have an even better approach. I have two copies of video with different dropped *fields*, and yes there is a dupe in place of the dropped field, so it's easy to find. However the dupes were created by a TBC, so it's an analog dupe, and it's not perfect - however still a simple YDifferenceFromPrevious detects them well.

So the reason I'm eliminating dupes is because I do interleave(clip1.separatefields, clip2.separatefields) so normally there's two dupes, if one stream is missing a field there will be 3 dupes. I just remove all dupes, and patch with one of the audio, and I'm done.

Since my detection is reliable, I don't want to rely on detecting jumps. There's no further drops after the dupes; one field is duped (to the next top field) and that's it.

[johnmeyer]

If you are consistently getting dropped frames from new captures, it sure sounds like you need to find a better capture card. I have captured thousands of hours of off-the-air and VHS, Beta, & 8mm videotape, and I never get any dropped frames. The only time I had these problems was when I tried capturing with my ATI Radeon card and those awful Catalyst drivers. I never could get consistent results.

Years ago, I created a "FAQ" for Sony Vegas for someone who was operating a Vegas help site, and in the video capture segment I asked the question: "How many dropped frames is acceptable?" My answer: none. You should never have any.

[jmac698]

I have many capture cards. I have tested them with my Glitch Analyzer, would you like to try it? Sometimes the drops were due to the software or settings I used; for example I found that VirtualDub by default has screwed up my captures for years :( What software do you use?

Anyhow this particular project is to help a board member. I have a different problem myself; there's a duped *frame* followed by a dropped frame.

http://forum.doom9.org/showthread.php?p=1462931

[StainlessS]

Note that YDifferenceFromPrevious() will always return 0 for the first frame (no previous!), so you need a special case for current_frame=0 in the 'if' condition, otherwise frame 0 will always be deleted:


Thanks Gavino, I had spotted it just before deleting a test
command file, but had not put any thought into a fix for it.

Currently, modding Prune() to coalesce adjacent trims from the
same source clip (in the example previously given, Prune would treat every frame as a trim/splice, not optimum and more hungry
on memory for arrays). After coalescing, the number of trims
would be reduced to just a few.

[johnmeyer]

I have many capture cards. I have tested them with my Glitch Analyzer, would you like to try it? Looks interesting. I may give it a go!


Sometimes the drops were due to the software or settings I used; for example I found that VirtualDub by default has screwed up my captures for years :( What software do you use?
I have tried several software programs. I originally used the program that came with my ATI Radeon card, and it worked well when capturing to its proprietary format, but I could never get it to work with formats that are better for editing, such as MJPEG (i.e., any format that compresses each frame individually). I then tried the capture built into VirtualDub, as well as the capture utility buried in my MainConcept MPEG encoder. I could never get any of them to work well (they all dropped frames). In the end, I decided that the problem is that the ATI Radeon card is not a good piece of hardware.

So, I gave up entirely trying to capture using a capture card and instead of have been using the passthrough on my VCR, which has both a TBC and also DV encoder. I also sometimes use one of my camcorders. Yes, I know that I give up the better color space I could get with a better codec than the DV codec (4:1:1 colorspace), and we can debate forever the pros and cons of various codecs. However, any downsides of using the DV codec are offset by the fact that I can, and have, captured thousands of hours of video and never once had a dropped frame. I can simply edit and get on with life. Also, DV is the easiest format ever invented for editing. At the risk of sounding like an old Apple ad: it simply works.

I have a different problem myself; there's a duped *frame* followed by a dropped frame.Well, that's the problem I "solved" (for myself) in the thread that I already linked to above. You are welcome to use that code, or to use the approach (for detecting gaps) that Didée invented. If your dropped frame always follows immediately after your duped frame, then the solution is very, very simple, and was solved with the "filldrops" script written by Mug Funky back in 2005 (and which you can find by clicking on the link I provided in my first post above). It should work perfectly for your situation.

[johnmeyer]

I realized, just after I posted, that Mug Funky's original filldrops script was for progressive and also used the original version of MVTools. I posted an updated version of his script here:

FillDropsI (http://forum.doom9.org/showthread.php?p=1493082#post1493082)

that uses MVTools2 and is designed for interlaced. You might want to read a few of the comments following my post because Gavino had a few suggestions for making my code a little cleaner.