vfr to cfr - how to interpolate several duplicate frames?

[JK1974]

Hi,

I try to "convert" some smartphone videos with variable frame rate to constant frame rate.

Therefore, I had the idea to use the avisynth vfrtocfr plugin first that generates duplicate images whenever there should normally be a new frame in the cfr video, and then, I wanted to interpolate the duplicates.

However, trying to achieve this by testing with filldrops() or the script by johnmeyer at https://forum.doom9.org/showthread.php?t=161758, I recognized that they apparently only work on a single duplicate frame, but my test video - apparently a very bad one - sometimes seems to contain even 2 or 3 consecutive duplicate frames and has a very low average frame rate (I am sure that it normally should be 29.970fps like other videos made at the same day which come closer to this average fps):

MediaInfo:
Frame rate : 24.191 FPS
Minimum frame rate : 14.896 FPS
Maximum frame rate : 30.303 FPS


My current code:
source=ffvideosource( "FJVID_20190719_195709.mp4", timecodes="timecodes.txt" )
source=vfrtocfr(source, timecodes="timecodes.txt", fpsnum=30000, fpsden=1001 )
source=audiodub(source,ffaudiosource("FJVID_20190719_195709.mp4") )

source=filldrops(source)
return source

function filldrops (clip c)
{
super=MSuper(c,pel=2)
vfe=manalyse(super,truemotion=true,isb=false,delta=1)
vbe=manalyse(super,truemotion=true,isb=true,delta=1)
filldrops = mflowinter(c,super,vbe,vfe,time=50)
fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.5")
return fixed
}

And because I want to cut this clip to a movie next to other 29.970 clips, I don´t want to set it to 25fps (and I am convinced that this fps would be wrong as all smartphones I know work with NTSC fps, and also see the maximum fps).

I am not that deep in avisynth programming and especially not the mvtools, so is there any script/link out there that is able to interpolate several consecutive duplicates?

Or any new other best practices on those vfr-to-cfr conversions involving frame interpolation?

Thanks a lot in advance!

[StainlessS]

sometimes seems to contain even 2 or 3 consecutive duplicate frames and has a very low average frame rate
"Low average frame rate", never good when interpolating.

Take a look at Duplicity2/DropDeadGorgeous:-
https://forum.doom9.org/showthread.php?t=175357&highlight=Duplicity2

Duplicity2() to create DBase of dupes.
DropDeadGorgeous() to interpolate the multiple dupes.

EDIT: From https://forum.doom9.org/showthread.php?p=1857865#post1857865
Below rough equivalent for your requirements.
It may be possible to up framerate to a non multiple of source whilst keeping all source frames verbatim, where eg source
is 24 FPS and required = 25 FPS, and using,
Src24.ChangeFPS(25) to produce a new clip at 25 FPS with an exact duplicate every 24 source frames and save new source as eg Src25.avi.
Process new source with eg src25.Duplicity2(ThG=0.1,DBase="D:\Src25_Det.DB"), to produce a DBase (perhaps with MaxDupLen=1, perhaps not).
Then use Src25.DropDeadGorgeous("D:\Src25_Det.DB",ScanAhead=12) to interpolate dupe frames achieving 25.0 and with all original
source frames. With a ScanAhead of eg 12, will locate dupes and then backwards scan up to 12 frames before the dupes looking for
frame of greatest movement, insert an interpolated frame prior to that greatest movement frame and shove all following frames along one
to remove the dupe. Might be worth someone experimenting.


And this followed above quote
I tried above converting [EDIT: true] 25.0 FPS to 30.0FPS, using ChangeFPS(30.0), and Duplicity2(ThG=0.1,MaxDuplen=1),
Then, used DropDeadGorgeous(ScanAhead=5), result produced is 30FPS with all original source frames, and with good fluid motion throughout.
ChangeFPS(30.0) inserts duplicate after every five source frames, and DropDeadGorgeous(ScanAhead=5) introduces
an interpolated frame within 5 frames prior to the duplicate (between frames of greatest motion) and pushes the following frames one position
along to squeeze out the duplicate.
Works pretty good.
EDIT: If source clip was badly decimated from eg 30FPS to 25 and with jerky motion due to decimation, above should produce pretty much the best possible result and repair/tween the jerk frames produced by the decimation.
EDIT: Should also perform well in cases where cadence changes, eg where source was decimated from 29.97 FPS to 25 FPS,
or where source was edited after decimation, neither one should be any problem.

[StainlessS]

Additional bit of info [see "Perhaps this explains better"]
https://forum.doom9.org/showthread.php?p=1873808#post1873808


Duplicity2(), is a dupe Detector, Dupe frame list creator, Dupes replaced with exact dupes, Dupes replaced with interpolated frames + DBase of dupes creator.
DropDeadGorgeous2() uses Duplicity2 DBase to do the almost impossible, smoothing out jerking clips (where source has jerks and dupes, much better than simple interpolation of dupes).
DropDeadGorgeous2 can also [EDIT: help] up framerate (eg 25.0 -> 29.97) keeping all source frames verbatim, and creating smooth result
[squeezes out dupes, where are replaced by interpolated frames between frames of most movement - non dupe frames are relocated between jerk interpolations and dupes to squeeze out dupes (hard to explain it)].

EDIT: Perhaps this explains better

abcXdefgDDhi Src, Two dupe frames preceded (within a set range) by a jerk frame. Jerk frame X, has greater motion to its predecessor than do d, e, f, or g.
abc12Xdefghi DropDeadGorgeous dupe removal and Jerk interpolation

Lower case non dupe non Jerk
X = Frame of greatest motion from predecessor (Jerk)
D = Dupe Frames
Digit = Interpolated frames between X and its predecessor c (Same Count as D dupes)

[johnmeyer]

Multiple dups can only be handled manually, so StainlessS' idea of getting a list of dups is a good one.

The reason multiple dups can't be handled automatically is easy to see if you take it to its logical conclusion: what are ten duplicated frames in a row called?

A still photo.

How do you turn a still photo into a video?

So as you move from a single dup, which can be fixed almost perfectly (in most cases) by using motion estimation, to multiple dups, and you begin to approach having a still photo, you are going to have to make some artistic decisions on how to create some sort of intermediate frames, and you are going to have to create those frames manually.

[StainlessS]

For static scene in Duplicity2 detector/DBase creator.


Function Duplicity2(clip c,"Mode"=0,Float "ThG"=1.0,Float "ThB"=ThG*2.0, Int "ThP"=64,Float "IgP"=0.0,Int "ThK"=7,
\ Int "MaxDupLen"=9,Int "LagMax"=MaxDupLen,Int "MaxInterp"=Min(9,MaxDupLen),
\ int "BlkW"=64,int "BlkH"=BlkW,int "OLapX"=BlkW/2,Int "OLapY=BlkH/2",
\ Int "X"=0,Int "Y"=0,Int"W"=0,Int "H"=0,Float "CW"=0.0,Bool "ChromaI"=False,Bool "AltScan"=False,
\ Int "SPad"=16, Int "SSharp"=1, Int "SRFilter"=4, [* MSuper *]
\ Int "ABlkSize"=16, Int "AOverlap"=4, Int "ASearch"=3,Int "ADct"=0, [* MAnalyse *]
\ Int "RBlkSize"=8, Int "ROverlap"=2, Int "RthSAD"=100, [* MRecalculate *]
\ Float "Iml"=200.0, Bool "IBlend"=True, Int "IthSCD1"=400, Int "IthSCD2"=130, [* MFlowInter *]
\ Int "OthSCD1"=400,Int "OthSCD2"=130, [* Dupe Detect Override @ Scene Change *]
\ String "OverRide"="",String "Frames"="Frames.txt",Bool "Ranges"=True,Bool "FrameAsRange"=False,
\ Bool "Show"=True,Bool "Verb"=True,Bool "ShowBlk"=True,Bool "Dim"=True,Bool "ShowDot"=False,
\ Bool "InterpFast"=True,String "DBase"=""
\ )



MaxDupLen, Default 9. Maximum permitted detection length of duplicate run, If detected dupe run exceeds this setting,
then it is totally ignored and presumed to be a static scene.

MaxInterp, Default Min(MaxDupLen,9), (Range 0->9). For Mode=2(Interpolate/Blend mode) only. Max length of interpolated frames.
Sequences longer than this but less or equal to MaxDupLen will be blended instead.

[JK1974]

Thanks a lot for your hints - haven´t found the scripts when searching for vfr to cfr solutions.

I tested the whole evening and it works really great together with the vfrtocfr plugin in most cases.
However, the video mentioned in my first post seems to be a somehow "damaged" recording because with vfrtocfr, I can see jumps in motion even in the source video file. Using changeFPS seems to give much better results even though I know that then the original vfr timings are ignored. Luckily, at his point, I just identified 2-3 clips of around 80 having this problem.

The other clips just suffer from very few dropped frames here and there that your script handles really very well, but this might also explain why it is vfr and not cfr and why the average frame rate is not 29.970 but slightly lower.