Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 26th July 2023, 17:51   #1  |  Link
TheTooleMan
Registered User
 
TheTooleMan's Avatar
 
Join Date: Nov 2004
Location: Texas
Posts: 27
How Do I Remove Random Duplicate Frames?

I have been ripping and encoding video to x264/x265 for years, but I am still new to AviSynth. I recently ran into a bigger obstacle than I can DIY my way around. A kindly soul on the DG support forum pointed me toward the Doom9 Forum, and here I am.

I am attempting to deinterlace and encode episodes of a 1960's TV show from DVD's. The episodes seem to have have been sped up so that they run ~47 minutes instead of the original ~51. The don't seem to have been edited to reduce runtime. The episodes are interlaced. Frame rate is 29.973.

Stepping through the frames of the video in Avidemux, I can see that every 7th frame is a dupe of the 6th.

Here is the AVS command file that produced the best results so far:

loadplugin("C:\Program Files (x86)\dgindexnv\DGDecodeNV.dll")
dgsource("C:\Videos\DGIndex_Out\C1_t00 DGIndexNV.dgi", deinterlace=0)
DGTelecide(mode=1, pthresh=2.0, dthresh=2.0)
DGDecimate(cycle=7, keep=6)

Running these commands with Simple X264 Launcher produces a 25.689 fps video file that runs about 51 minutes. The video is smoother with each 7th frame removed, but there are still duplicate frames in the video that appear randomly.

If you want to look at the video yourself, this folder contains the complete episode and the teaser as ripped by MakeMKV.

The file was ripped with MakeMKV. Take note of the last shot in the teaser before the opening credits, which starts at about 4:15 with a dolly in on a man's face. Then the camera tracks to the left and shows the faces of other soldiers. It's a beautiful sequence, if you have an eye for this sort of thing, but it's riddled with duplicated frames that freeze and stutter after deinterlacing and telecide. Those glitches ruin it for me.

How can I remove the randomly occurring duplicate frames? Thanks for your guidance, or at least for sharing my bewilderment.
__________________
TheTooleMan
TheTooleMan is offline   Reply With Quote
Old 26th July 2023, 18:52   #2  |  Link
Frank62
Registered User
 
Join Date: Mar 2017
Location: Germany
Posts: 240
We had such sources more than once in the past. We used TDecimate. But the main trick was, that the amount of too-many or too-less-frames, so also the pattern, changes more than once per episode. You might need a reference, maybe a DVD of another country or even an old VHS source (with 51 minutes length), to see where the differences begin.
Frank62 is offline   Reply With Quote
Old 26th July 2023, 19:55   #3  |  Link
Emulgator
Big Bit Savings Now !
 
Emulgator's Avatar
 
Join Date: Feb 2007
Location: close to the wall
Posts: 1,590
http://avisynth.nl/index.php/AnimeIVTC
Once you got all the dependencies together, AnimeIVTC(mode=1, omode=1) can be your starting point.

Here it looked usable at first glance, and I did not go into detail any further.
P.S. spending some more minutes framestepping with this primitive setting
occasional frames are missing, but none repeated.
Well, there are >60 parameters, not all of them apply to this use case.

BTW, you may consider Oyster() afterwards, I had fab results with blocky sources.
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain)
"Data reduction ? Yep, Sir. We're that issue working on. Synce invntoin uf lingöage..."

Last edited by Emulgator; 26th July 2023 at 22:13. Reason: I was a bit brief, extended a bit now to avoid ambiguity
Emulgator is offline   Reply With Quote
Old 26th July 2023, 21:16   #4  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,696
Quote:
Originally Posted by TheTooleMan View Post
The episodes seem to have have been sped up so that they run ~47 minutes instead of the original ~51. The don't seem to have been edited to reduce runtime. The episodes are interlaced. Frame rate is 29.973.

Stepping through the frames of the video in Avidemux, I can see that every 7th frame is a dupe of the 6th.
OK, it is film that has been telecined to run at 30 (29.97) fps instead of 24 (23.976) fps. However, it is also sped up.

Let's see, if I wanted to make a film play faster I could simply play the 24 fps film at 30 fps. However, that would be too fast. The other way would be to add fewer duplicate fields (interlaced telecine is usually done by duplicating fields, not frames, in order to make it smoother -- you should check that).

So, if I am understanding what was done, all you have to do is first do the IVTC (Inverse Telecine) correctly and then time it to play at 24 fps.

Code:
tfm(display=false)
tdecimate(cycleR=1,cycle=6)
AssumeFPS(23.976)
For normal pulldown, cycle would equal 5, not 6.

I've never used DGDecimate so I don't know how it handles interlaced pulldown. That's why I'm using TDecimate: I'm familiar with how it works.
johnmeyer is offline   Reply With Quote
Old 26th July 2023, 21:31   #5  |  Link
Katie Boundary
Registered User
 
Katie Boundary's Avatar
 
Join Date: Jan 2015
Posts: 1,070
You know, I think TDecimate has options for dealing with crap like this.

http://avisynth.nl/index.php/TIVTC/TDecimate

I speculate that this video might have gone through multiple framerate conversions, probably to PAL/SECAM and back. After field-matching, try Assumefps(25).Tdecimate(mode=2) and/or Tdecimate(mode=2,rate=25).Assumefps(23.976) and tell us the results.
__________________
I ask unusual questions but always give proper thanks to those who give correct and useful answers.
Katie Boundary is offline   Reply With Quote
Old 27th July 2023, 15:14   #6  |  Link
TheTooleMan
Registered User
 
TheTooleMan's Avatar
 
Join Date: Nov 2004
Location: Texas
Posts: 27
Thanks for your input and suggestions, everyone. I am off to conduct research and perform experiments. I will post back when I have an update.

By the way, this series runs on the H&I channel, which is available over the air and via some streaming apps. I set the app to record the series and will have a look at what they do to the show. Probably will not be pretty...
__________________
TheTooleMan
TheTooleMan is offline   Reply With Quote
Old 27th July 2023, 15:40   #7  |  Link
poisondeathray
Registered User
 
Join Date: Sep 2007
Posts: 5,409
Another approach is dedup, but assume the constant framerate instead of using the timecodes that it produces for VFR . It will decimate duplicates, or strings of duplicates (triplicates etc..), according to threshold settings . You can run it in debug mode to adjust the settings ( to see which frames are flagged as duplicates) . The benefit is you're not trying to "force" some M in N decimation pattern that could result in duplicates or remove good frames if that was not the actual underlying pattern.

If there was a set pattern, M in N decimation is the correct thing to do. For M in N decimation - sometimes a longer "horizon" can help. e.g instead of 1 in 7, 2 in 14, or 3 in 21 , etc... might work better. For example, I saw a string of 10 good frames in a row (e.g. around 78425 after field matching). 1 in 7 decimation will remove a good frame and it would be "jerky"

Last edited by poisondeathray; 27th July 2023 at 15:46.
poisondeathray is offline   Reply With Quote
Old 27th July 2023, 19:52   #8  |  Link
TheTooleMan
Registered User
 
TheTooleMan's Avatar
 
Join Date: Nov 2004
Location: Texas
Posts: 27
Quote:
Originally Posted by poisondeathray View Post
Another approach is dedup, but assume the constant framerate instead of using the timecodes that it produces for VFR . It will decimate duplicates, or strings of duplicates (triplicates etc..), according to threshold settings . You can run it in debug mode to adjust the settings ( to see which frames are flagged as duplicates) . The benefit is you're not trying to "force" some M in N decimation pattern that could result in duplicates or remove good frames if that was not the actual underlying pattern.

If there was a set pattern, M in N decimation is the correct thing to do. For M in N decimation - sometimes a longer "horizon" can help. e.g instead of 1 in 7, 2 in 14, or 3 in 21 , etc... might work better. For example, I saw a string of 10 good frames in a row (e.g. around 78425 after field matching). 1 in 7 decimation will remove a good frame and it would be "jerky"
That's a good point, and something I thought about but did not know how to implement. Thanks for the tip!
__________________
TheTooleMan
TheTooleMan is offline   Reply With Quote
Old 30th July 2023, 22:35   #9  |  Link
TheTooleMan
Registered User
 
TheTooleMan's Avatar
 
Join Date: Nov 2004
Location: Texas
Posts: 27
First Results

Quote:
Originally Posted by Katie Boundary View Post
You know, I think TDecimate has options for dealing with crap like this.

http://avisynth.nl/index.php/TIVTC/TDecimate

I speculate that this video might have gone through multiple framerate conversions, probably to PAL/SECAM and back. After field-matching, try Assumefps(25).Tdecimate(mode=2) and/or Tdecimate(mode=2,rate=25).Assumefps(23.976) and tell us the results.
I gave your suggestion a try (the second syntax option) and got good results overall, but some unique frames were dropped which caused uneven motion. The length of the teaser clip increased from 5:19 to 5:33.

I tried two-pass TDecimate using mode 4 followed by mode 2. Without "rate=25" in mode 2, the results were jerky and unacceptable. Using "rate=25" gave identical results to the single-step mode 2, including the dropped frame.

I went back to my original message and saw that the best results I could achieve created an output file at 25.689 fps. Using that magic number as the rate parameter, the dropped field went away. The file length extended to 5:41, which is more likely to return the episode to its original length near 51 minutes.

I will do more checking, encode the entire episode, and report back.

Thanks again for the aid!
__________________
TheTooleMan

Last edited by TheTooleMan; 30th July 2023 at 23:16. Reason: update
TheTooleMan is offline   Reply With Quote
Old 31st July 2023, 17:55   #10  |  Link
TheTooleMan
Registered User
 
TheTooleMan's Avatar
 
Join Date: Nov 2004
Location: Texas
Posts: 27
Using the framerate 25.689 worked great on the teaser, but the process threw an error when I tried it on the entire video:

[2023-07-31][10:45:21] Simple x264 Launcher (Build #1220), built 2023-06-24
[2023-07-31][10:45:21]
[2023-07-31][10:45:21] Job started at 2023-07-31, 10:45:21.
...
[2023-07-31][10:45:21] error: TDecimate: mode 2 error, unable to achieve a completely synced result!
[2023-07-31][10:45:21] (C:\Videos\DGINDE~1\COMBAT~1\c13fcb~1.avs, line 9)
[2023-07-31][10:45:21]
[2023-07-31][10:45:21] PROCESS EXITED WITH ERROR CODE: 1

I tried 26 fps, which got nearly as good video results. I stretched the audio to the same new length as the video, but it would not sync. Apparently the frame rate varied due to the number of repeated frames removed and other factors.

Your comments and suggestions are appreciated.
__________________
TheTooleMan
TheTooleMan is offline   Reply With Quote
Reply

Tags
deinterlace, dvd, frame rate, mpeg2, telecide

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 17:43.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.