Delete Dups

[jmac698]

I have a video where sometimes there's a single dup after a frame. How can I remove this?

    scriptclip("""
    YDifferenceFromPrevious<.1?deleteframe(current_frame):last
    """)

Doesn't work, I'm stuck on how to write it...

[Chikuzen]

DeDup.
http://akuvian.org/src/avisynth/dedup/

[StainlessS]

DeDup as suggested above will probably sort you out, but also, if you have any other weird reasons
for deleting frames (other than duplicates) then the below might come in handy to create a command
file for Prune(), where you list the frames to keep rather than chuck. You would though probably want
to only use it as last resort.

# Edited:-
avisource("d:\avs\avi\1.avi").ConvertToYV12()

Writefileif("d:\avs\avi\1.txt","YDifferenceFromPrevious>=0.1 || current_frame==0",""" "0," ""","current_frame",append=false)

[jmac698]

Thanks, I'm trying dedup now, however the log file is empty except this line:

DeDup 0.17 by Loren Merritt, based on Dup 2.20 beta 1 by Donald Graft/Klaus Post, Copyright 2004

I'm surprised there isn't a simple way to do this in avisynth??

[Chikuzen]

Originally Posted by jmac698

Thanks, I'm trying dedup now, however the log file is empty except this line:

DupMC needs analysis pass.
use VDub's 'Run video analysis pass'

[Gavino]

Originally Posted by jmac698

I have a video where sometimes there's a single dup after a frame. How can I remove this?

    scriptclip("""
    YDifferenceFromPrevious<.1?deleteframe(current_frame):last
    """)

Doesn't work, I'm stuck on how to write it...

Deleting frames with ScriptClip is tricky because to make the deletions cumulative, you need to keep track of what has already been deleted in previous frames. A similar problem was discussed in this thread, where I described the solution, which could be adapted for your case.
The recursive procedure could now be replaced by a GScript 'while' loop (not available at the time).

DeDup is probably a simpler way if it works for your source.

[jmac698]

Ok, fixed the problem with the log file, now I can't get all the frames dedeupped, it's quite clear the threshold. I find the docs confusing. If I could do multiple passes I might get it to work, but it's quite awkward.

I tried Gavino's approach, no luck either, it's taking away too many frames.

function DarkFrames(clip c, int current_frame, float thresh) {
# returns the number of consecutive 'dark' frames at the current frame
  c.YDifferenceFromPrevious< thresh ? 0
\ : (current_frame >= c.frameCount-1) ? 1
\ : 1 + DarkFrames(c.Trim(1,0), current_frame, thresh)
}

dir="D:\project001a\dedup fields\"
DirectShowSource(dir+"jvc-with-ext-tbc.mov")
SeparateFields
selectevery(2,0)
converttoyv12

threshold = 2.6 # <- change this to suit requirements
offset = 0
# assume 'last' set earlier in script
ScriptClip("""
    subtitle(string(YDifferenceFromPrevious))
  offset = offset + DarkFrames(Trim(offset, 0), current_frame, threshold)
  Trim(offset, 0)
""")

[jmac698]

Finally I tried Stainless approach. It seems to be working. Stainless, could you add a command to specify the list of deletes?
This type of editing seems tremendously difficult in avisynth.

[Gavino]

Originally Posted by jmac698

I tried Gavino's approach, no luck either, it's taking away too many frames.

function DarkFrames(clip c, int current_frame, float thresh) {
# returns the number of consecutive 'dark' frames at the current frame
  c.YDifferenceFromPrevious< thresh ? 0
\ : (current_frame >= c.frameCount-1) ? 1
\ : 1 + DarkFrames(c.Trim(1,0), current_frame, thresh)
}

You've got your condition the wrong way round - the function should return the number of frames you want to delete. Renaming it and changing the comment to suit your purpose, it should be:

function DupFrames(clip c, int current_frame, float thresh) {
# returns the number of consecutive duplicate frames at the current frame
  c.YDifferenceFromPrevious > thresh ? 0
\ : (current_frame >= c.frameCount-1) ? 1
\ : 1 + DupFrames(c.Trim(1,0), current_frame, thresh)
}

You also want to comment out the subtitle for the real run, since it will affect the YDifference value.

[Gavino]

Originally Posted by jmac698

This type of editing seems tremendously difficult in avisynth.

Tricky yes, but I doubt any of your fancy NLE's will give you the ability to delete an arbitary set of frames according to some user-defined criterion based on frame contents. The power and flexibility of Avisynth is shown by examples like this.

[StainlessS]

Jmac, eg below should work (or whatever you called the command file.

Prune(cmd="D:\avs\avi\1.txt");

Edit, if you use the Beta version you may want to add "fade=0.0", to switch off the audio fading.

[jmac698]

Yep I used

prune(cmd="keeplist.txt")

[StainlessS]

Hi Jmac, just want to make sure you saw the previous edit.

[jmac698]

Nope missed that, thanks.

[Gavino]

Originally Posted by StainlessS

Writefileif("d:\avs\avi\1.txt","YDifferenceFromPrevious>=0.1",""" "0," ""","current_frame",append=false)

Note that YDifferenceFromPrevious() will always return 0 for the first frame (no previous!), so you need a special case for current_frame=0 in the 'if' condition, otherwise frame 0 will always be deleted:

..., "YDifferenceFromPrevious>=0.1 || current_frame==0", ...

A similar change would also apply if using my ScriptClip method.

[sven_x]

Are you sure you want to delete the dups? When you're dealing with VHS grabbings the dups might be a simple replacement to fill the gaps where the capture device could not sync to a frame. When using USB video grabbers I had often series of 1...10 dups at places with problems in tape flow. In that cases the dups are 100% identical copies of the last frame.

In those cases I found that cutting off the dups can disturb the flow of motions, and also it leads to jumps in audio flow. If would be better to interpolate the missing frames from their neighbors using mflow. (If there is no scene transisition -- then interpolation will not make much sense).
The latter is exactly the problem I am working on, but I did not succeed in writing a script that replaces n1, n2 frames at positions x1, x2 ... using a list of duplicate frames from a ConditionalReader file.

[johnmeyer]

Originally Posted by sven_x

Are you sure you want to delete the dups? When you're dealing with VHS grabbings the dups might be a simple replacement to fill the gaps where the capture device could not sync to a frame. When using USB video grabbers I had often series of 1...10 dups at places with problems in tape flow. In that cases the dups are 100% identical copies of the last frame.

In those cases I found that cutting off the dups can disturb the flow of motions, and also it leads to jumps in audio flow. If would be better to interpolate the missing frames from their neighbors using mflow. (If there is no scene transisition -- then interpolation will not make much sense).
The latter is exactly the problem I am working on, but I did not succeed in writing a script that replaces n1, n2 frames at positions x1, x2 ... using a list of duplicate frames from a ConditionalReader file.

I had this exact situation and with Didée's help eventually managed to create a script which not only detected the dups (that is relatively easy) but then also found the "jump" that occurred where a frame was missed.

Automatically fix dups followed (eventually) by drops

It turns out that in some capture situations, the capture card misses (drops) a frame. So, to get the audio back in sync, it later duplicates a frame (as sven_x says, this is a perfect duplicate). Thus, while you can easily find the dups and interpolate a frame at that point (search this forum for "deletedup"), that usually doesn't give you smooth video. Instead, you have to find the unwanted jump where a frame was not captured, and interpolate a new frame at that point (using mvtools2).

The beauty of Didée's approach, which I modified to get better detection with my particular source, is that it also "automatically" finds and removes the exact duplicates. Thus the duplicates are decimated, and new interpolated frames are inserted at the points which have the biggest "gap" in motion from one frame to the next.

The gap detection isn't perfect, but it was, for my source, pretty darned good (about 98% correct). I'm sure another smart person could take the script as a starting point and, while referring back to Didée's original masktools approach for finding gaps, combine the various ideas and perhaps come up with a detection scheme that is even better. The underlying "plumbing" is all in place, so you can spend you time just on doing better detection of the places where the frame was actually dropped.

Of course if the audio is NOT in sync, and you have to drop these duplicate frame to get back into sync, then all you need is the original "deletedups" function created many years ago. You'll find a link to that if you click on the hyperlink above.

[jmac698]

You're right, that is the underlying problem, but I have an even better approach. I have two copies of video with different dropped *fields*, and yes there is a dupe in place of the dropped field, so it's easy to find. However the dupes were created by a TBC, so it's an analog dupe, and it's not perfect - however still a simple YDifferenceFromPrevious detects them well.

So the reason I'm eliminating dupes is because I do interleave(clip1.separatefields, clip2.separatefields) so normally there's two dupes, if one stream is missing a field there will be 3 dupes. I just remove all dupes, and patch with one of the audio, and I'm done.

Since my detection is reliable, I don't want to rely on detecting jumps. There's no further drops after the dupes; one field is duped (to the next top field) and that's it.

[johnmeyer]

If you are consistently getting dropped frames from new captures, it sure sounds like you need to find a better capture card. I have captured thousands of hours of off-the-air and VHS, Beta, & 8mm videotape, and I never get any dropped frames. The only time I had these problems was when I tried capturing with my ATI Radeon card and those awful Catalyst drivers. I never could get consistent results.

Years ago, I created a "FAQ" for Sony Vegas for someone who was operating a Vegas help site, and in the video capture segment I asked the question: "How many dropped frames is acceptable?" My answer: none. You should never have any.

[jmac698]

I have many capture cards. I have tested them with my Glitch Analyzer, would you like to try it? Sometimes the drops were due to the software or settings I used; for example I found that VirtualDub by default has screwed up my captures for years What software do you use?

Anyhow this particular project is to help a board member. I have a different problem myself; there's a duped *frame* followed by a dropped frame.

http://forum.doom9.org/showthread.php?p=1462931