How Do I Remove Random Duplicate Frames?

[TheTooleMan]

I have been ripping and encoding video to x264/x265 for years, but I am still new to AviSynth. I recently ran into a bigger obstacle than I can DIY my way around. A kindly soul on the DG support forum pointed me toward the Doom9 Forum, and here I am.

I am attempting to deinterlace and encode episodes of a 1960's TV show from DVD's. The episodes seem to have have been sped up so that they run ~47 minutes instead of the original ~51. The don't seem to have been edited to reduce runtime. The episodes are interlaced. Frame rate is 29.973.

Stepping through the frames of the video in Avidemux, I can see that every 7th frame is a dupe of the 6th.

Here is the AVS command file that produced the best results so far:

loadplugin("C:\Program Files (x86)\dgindexnv\DGDecodeNV.dll")
dgsource("C:\Videos\DGIndex_Out\C1_t00 DGIndexNV.dgi", deinterlace=0)
DGTelecide(mode=1, pthresh=2.0, dthresh=2.0)
DGDecimate(cycle=7, keep=6)

Running these commands with Simple X264 Launcher produces a 25.689 fps video file that runs about 51 minutes. The video is smoother with each 7th frame removed, but there are still duplicate frames in the video that appear randomly.

If you want to look at the video yourself, this folder contains the complete episode and the teaser as ripped by MakeMKV.

The file was ripped with MakeMKV. Take note of the last shot in the teaser before the opening credits, which starts at about 4:15 with a dolly in on a man's face. Then the camera tracks to the left and shows the faces of other soldiers. It's a beautiful sequence, if you have an eye for this sort of thing, but it's riddled with duplicated frames that freeze and stutter after deinterlacing and telecide. Those glitches ruin it for me.

How can I remove the randomly occurring duplicate frames? Thanks for your guidance, or at least for sharing my bewilderment.

[Frank62]

We had such sources more than once in the past. We used TDecimate. But the main trick was, that the amount of too-many or too-less-frames, so also the pattern, changes more than once per episode. You might need a reference, maybe a DVD of another country or even an old VHS source (with 51 minutes length), to see where the differences begin.

[Emulgator]

http://avisynth.nl/index.php/AnimeIVTC
Once you got all the dependencies together, AnimeIVTC(mode=1, omode=1) can be your starting point.

Here it looked usable at first glance, and I did not go into detail any further.
P.S. spending some more minutes framestepping with this primitive setting
occasional frames are missing, but none repeated.
Well, there are >60 parameters, not all of them apply to this use case.

BTW, you may consider Oyster() afterwards, I had fab results with blocky sources.

[johnmeyer]

Originally Posted by TheTooleMan

The episodes seem to have have been sped up so that they run ~47 minutes instead of the original ~51. The don't seem to have been edited to reduce runtime. The episodes are interlaced. Frame rate is 29.973.

Stepping through the frames of the video in Avidemux, I can see that every 7th frame is a dupe of the 6th.

OK, it is film that has been telecined to run at 30 (29.97) fps instead of 24 (23.976) fps. However, it is also sped up.

Let's see, if I wanted to make a film play faster I could simply play the 24 fps film at 30 fps. However, that would be too fast. The other way would be to add fewer duplicate fields (interlaced telecine is usually done by duplicating fields, not frames, in order to make it smoother -- you should check that).

So, if I am understanding what was done, all you have to do is first do the IVTC (Inverse Telecine) correctly and then time it to play at 24 fps.

tfm(display=false)
tdecimate(cycleR=1,cycle=6)
AssumeFPS(23.976)

For normal pulldown, cycle would equal 5, not 6.

I've never used DGDecimate so I don't know how it handles interlaced pulldown. That's why I'm using TDecimate: I'm familiar with how it works.

[Katie Boundary]

You know, I think TDecimate has options for dealing with crap like this.

http://avisynth.nl/index.php/TIVTC/TDecimate

I speculate that this video might have gone through multiple framerate conversions, probably to PAL/SECAM and back. After field-matching, try Assumefps(25).Tdecimate(mode=2) and/or Tdecimate(mode=2,rate=25).Assumefps(23.976) and tell us the results.

[TheTooleMan]

Thanks for your input and suggestions, everyone. I am off to conduct research and perform experiments. I will post back when I have an update.

By the way, this series runs on the H&I channel, which is available over the air and via some streaming apps. I set the app to record the series and will have a look at what they do to the show. Probably will not be pretty...

[poisondeathray]

Another approach is dedup, but assume the constant framerate instead of using the timecodes that it produces for VFR . It will decimate duplicates, or strings of duplicates (triplicates etc..), according to threshold settings . You can run it in debug mode to adjust the settings ( to see which frames are flagged as duplicates) . The benefit is you're not trying to "force" some M in N decimation pattern that could result in duplicates or remove good frames if that was not the actual underlying pattern.

If there was a set pattern, M in N decimation is the correct thing to do. For M in N decimation - sometimes a longer "horizon" can help. e.g instead of 1 in 7, 2 in 14, or 3 in 21 , etc... might work better. For example, I saw a string of 10 good frames in a row (e.g. around 78425 after field matching). 1 in 7 decimation will remove a good frame and it would be "jerky"

[TheTooleMan]

Originally Posted by poisondeathray

Another approach is dedup, but assume the constant framerate instead of using the timecodes that it produces for VFR . It will decimate duplicates, or strings of duplicates (triplicates etc..), according to threshold settings . You can run it in debug mode to adjust the settings ( to see which frames are flagged as duplicates) . The benefit is you're not trying to "force" some M in N decimation pattern that could result in duplicates or remove good frames if that was not the actual underlying pattern.

If there was a set pattern, M in N decimation is the correct thing to do. For M in N decimation - sometimes a longer "horizon" can help. e.g instead of 1 in 7, 2 in 14, or 3 in 21 , etc... might work better. For example, I saw a string of 10 good frames in a row (e.g. around 78425 after field matching). 1 in 7 decimation will remove a good frame and it would be "jerky"

That's a good point, and something I thought about but did not know how to implement. Thanks for the tip!

[TheTooleMan]

Originally Posted by Katie Boundary

You know, I think TDecimate has options for dealing with crap like this.

http://avisynth.nl/index.php/TIVTC/TDecimate

I speculate that this video might have gone through multiple framerate conversions, probably to PAL/SECAM and back. After field-matching, try Assumefps(25).Tdecimate(mode=2) and/or Tdecimate(mode=2,rate=25).Assumefps(23.976) and tell us the results.

I gave your suggestion a try (the second syntax option) and got good results overall, but some unique frames were dropped which caused uneven motion. The length of the teaser clip increased from 5:19 to 5:33.

I tried two-pass TDecimate using mode 4 followed by mode 2. Without "rate=25" in mode 2, the results were jerky and unacceptable. Using "rate=25" gave identical results to the single-step mode 2, including the dropped frame.

I went back to my original message and saw that the best results I could achieve created an output file at 25.689 fps. Using that magic number as the rate parameter, the dropped field went away. The file length extended to 5:41, which is more likely to return the episode to its original length near 51 minutes.

I will do more checking, encode the entire episode, and report back.

Thanks again for the aid!

[TheTooleMan]

Using the framerate 25.689 worked great on the teaser, but the process threw an error when I tried it on the entire video:

[2023-07-31][10:45:21] Simple x264 Launcher (Build #1220), built 2023-06-24
[2023-07-31][10:45:21]
[2023-07-31][10:45:21] Job started at 2023-07-31, 10:45:21.
...
[2023-07-31][10:45:21] error: TDecimate: mode 2 error, unable to achieve a completely synced result!
[2023-07-31][10:45:21] (C:\Videos\DGINDE~1\COMBAT~1\c13fcb~1.avs, line 9)
[2023-07-31][10:45:21]
[2023-07-31][10:45:21] PROCESS EXITED WITH ERROR CODE: 1

I tried 26 fps, which got nearly as good video results. I stretched the audio to the same new length as the video, but it would not sync. Apparently the frame rate varied due to the number of repeated frames removed and other factors.

Your comments and suggestions are appreciated.