MV interpolation of mixed TV+cinema to 120fps progressive

[markfilipak]

Transcoding a mix of TV and telecined video is the hardest task. It is the last piece in my task list. I present the issues, my nomenclature, some ffmpeg, and some issues.
I would appreciate your thoughts and some clues to overcoming the issues.


"fps" means "frames per second", "sps" means "scans per second", "pps" means "pictures per second".
29.9fps[59.9sps] (read: "29.9 fps of 59.9 sps") is interlaced scans (e.g. NTSC).
29.9fps[2-3[24pps]] (read: "29.9 fps of 2-3 pull-down of 24 pps") is hard telecined (2-3 pull-down) cinema.
29.9fps[3-2[24pps]] (read: "29.9 fps of 3-2 pull-down of 24 pps") is hard telecined (3-2 pull-down) cinema.
29.9fps[3-3-2-2[24pps]] (read: "29.9 fps of 3-3-2-2 pull-down of 24 pps") is hard telecined (3-3-2-2 pull-down) cinema.
29.9fps[2-2-3-3[24pps]] (read: "29.9 fps of 2-2-3-3 pull-down of 24 pps") is hard telecined (2-2-3-3 pull-down) cinema.
'c' denotes combed frame, '-' denotes uncombed frame, time flows left to right.

c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c ...Scans: 29.9fps[59.9sps].
- - c c - - - c c - - - c c - - - c c - - - c c - - - c c - - - c c - - ...Hard Telecine: 29.9fps[2-3[24pps]].
- - - c - - - - c - - - - c - - - - c - - - - c - - - - c - - - - c - - ...Hard Telecine: 29.9fps[2-2-3-3[24pps]].
- c c - - - c c - - - c c - - - c c - - - c c - - - c c - - - c c - - - ...Hard Telecine: 29.9fps[3-2[24pps]].
- c - - - - c - - - - c - - - - c - - - - c - - - - c - - - - c - - - - ...Hard Telecine: 29.9fps[3-3-2-2[24pps]].
- - - - - c c - - - - - c c - - - - - - - - - - - c c - - - - - - - c c ...Hard Telecine: Wacky pull-down (seen in a few videos, e.g. "PASSION FISH");
\____________\___(2 combed frames alternate with 5 / 7.. uncombed frames) -- always odd, always 2 combed frames, seemingly random intervals.
c c c c c c c c c c c c c c c c c c c - - c c - - - c c - - - c c - - - ...Mixed Scans+Hard Telecine: e.g. 29.9fps[[59.9sps]+2-3[24pps]].
\
A well cut, 'Hard Telecine' segment always begins with an uncombed frame.
A poorly cut, 'Hard Telecine' segment may suffer 1 or 2 combed frames coinciding with scene change.

For 29.9fps[59.9sps] segments:

SET UNWEAVE=settb=expr=1001/30000,setpts=expr=N,separatefields,split[1][2]
SET FIELD1=[1]select=eq(mod(n\,2)\,0),minterpolate=fps=120000/1001:mi_mode=mci:scd=fdiff,settb=expr=1/120,setpts=expr=N,split[3][4]
SET FIELD2=[2]select=eq(mod(n\,2)\,1),minterpolate=fps=120000/1001:mi_mode=mci:scd=fdiff,settb=expr=1/120,setpts=expr=N[5]
SET BOB=[3]select=lt(n\,2)[6]
SET AFTERBOB=[5]select=gt(n\,1)[7]
SET MERGE=[6][7]tinterlace=mode=merge[8]
SET NEWFIELD2=%FIELD2%,%BOB%,%AFTERBOB%,%MERGE%
SET WEAVE=[4][8]weave[v]
-filter_complex "%UNWEAVE%,%FIELD1%,%NEWFIELD2%,%WEAVE%"

The 2 fields are separated and independently 4x MV converted to 120fps.
As the 1st field stream passes through unaltered, duplicates of its 1st 2 frames are split off.
The 2nd field stream is delayed by 2 frames and merged with the 2 split off frames from field 1. That process, 1, bobs the 1st 2 frames, and 2, shifts field 2's frames to temporally align them with field 1, thereby making progressive frames out of scan fields with zero cosmetic filtering!
The 2 streams are then woven.

For 29.9fps[2-3[24pps]] segments:

-filter_complex "settb=expr=1001/30000,setpts=expr=N,detelecine,minterpolate=fps=120000/1001:mi_mode=mci:scd=fdiff,settb=expr=1/120,setpts=expr=N[v]"

The result is 5x MV interpolation of the detelecined source.

To pull this off, I need a flag, 'combed', so that, after I combine filter graph sections, I know when to switch between them. Cu Selur suggested I try 'tdm.IsCombed' (https://github.com/HomeOfVapourSynthEvolution/VapourSynth-TDeintMod) but I think it scans an entire clip and can't flag segments.

And I'd also like to replace 'minterpolate' with pipes to/from VapourSynth's superior MV interpolation: Super(), Analyse(), SmoothFps(), but I don't know whether a pipe in the middle of a filter graph is possible -- I suspect not -- or alternatively, whether ffmpeg can support 2 filter graphs with to/from pipes between them -- I've never tried 2 'filter_complex's in one ffmpeg command).

[Selur]

Cu = See you,... Name is just Selur. :)
I think it scans an entire clip and can't flag segments.
Haven't used 'tdm.IsCombed' for ages, but as far as I know it creates a property clip which containes the combed info on a per frame basis,.. but that probably doesn't really help when using ffmpeg.
but I don't know whether a pipe in the middle of a filter graph is possible -- I suspect not -
Nope, at least as I know that there is no filter in ffmpeg which would allow to use a Vapoursynth script for filtering.
-> either ffmpeg or Vapoursynth

Cu Selur

[markfilipak]

Cu = See you,... Name is just Selur. :)

Haven't used 'tdm.IsCombed' for ages, but as far as I know it creates a property clip which containes the combed info on a per frame basis,..

Howdy! I'm not sure what a 'property clip' is. Is it, 1, a scoreboard that can be used as an auxiliary reference source in a 2nd pass, or 2, a text list similar to what ffprobe or MediaInfo produces, or 3, a 'clip' (meaning: a video)?

[Selur]

Since it's passed to FrameEval (http://www.vapoursynth.com/doc/functions/video/frameeval.html#std.FrameEval)
std.FrameEval(vnode clip, func eval[, vnode[] prop_src, vnode[] clip_src])
in the example (https://github.com/HomeOfVapourSynthEvolution/VapourSynth-TDeintMod#example-usage-of-iscombed) it seesm to be an array of video nodes.
My guess is that you best ask one of the authors about further details.
I would have expected it to be just one clip where '_Combed' frame propery (http://www.vapoursynth.com/doc/apireference.html#reserved-frame-properties) are set for the frames. (No clue why it's a vnode[].) :)

Cu Selur

[zorr]

Since it's passed to FrameEval
...
it seesm to be an array of video nodes.
...
I would have expected it to be just one clip where '_Combed' frame propery (http://www.vapoursynth.com/doc/apireference.html#reserved-frame-properties) are set for the frames. (No clue why it's a vnode[].) :)

FrameEval is a general function for any kind of per frame processing. Sometimes you need to access more clips (and their frame properties) than just one, that's why it's a vnode[]. But you don't *have to* input more than one clip if you don't need to. :)

[markfilipak]

Since it's passed to FrameEval (http://www.vapoursynth.com/doc/functions/video/frameeval.html#std.FrameEval) ...
Oh, My! Is that what I think it is? Is that a function to edit frames, fields, etc. (all of what I call "mechanical" video properties) in a way that's analogous to what I do currently in an FFmpeg filter graph?

(I prominantly saved those links.)

Now, for an ignorant question (I have no pride) ...
std.FrameEval(vnode clip, func eval[, vnode[] prop_src, vnode[] clip_src])
in the example (https://github.com/HomeOfVapourSynthEvolution/VapourSynth-TDeintMod#example-usage-of-iscombed) it seesm to be an array of video nodes.
My guess is that you best ask one of the authors about further details.
I would have expected it to be just one clip where '_Combed' frame propery (http://www.vapoursynth.com/doc/apireference.html#reserved-frame-properties) are set for the frames. (No clue why it's a vnode[].)
What is a vnode?

PS: I may just wind up retiring what I know/do in FFmpeg in favor/favour of VS. I'll need some help with it. Are you folks game?

PPS: Just to let you know that I'm not lazy, I searched for "What is a vnode" and got a million irrelevant hits, and for "What is a vapoursynth vnode" and got everything vapoursynth but not vnode; I searched the vapoursynth install directory and the vapoursynth downloaded directory for anything/everything "vnode" and got 4 '.dll's. That means that the vapoursynth documentation '.html' does not contain the string "vnode". ...Oh, and I searched for ANSI, UTF-8, and UTF-16, non-case, non-wholewords.

[richardpl]

Someone finally please help him! He desperately wants to learn avisynth and vapoursynth and forget ffmpeg.

[_Al_]

vnode is just a name, label in general, it means vapoursynth video clip if refering in a new docs.

vapoursynth clip up until recently was always considered always vapoursynth.VideoNode type

import vapoursynth as vs
clip = vs.core.lsmas.LibavSMASHSource('video.mp4')
print(type(clip))
if isinstance(clip, vs.VideoNode):
print('It is python object vapoursynth.VideoNode that is referred as vnode in docs now.')
it prints:
<class 'vapoursynth.VideoNode'>
It is python object vapoursynth.VideoNode that is referred as vnode in docs now.

It just started recently, because audio support was added for vapoursynth, so kind of making sure what docs refer to, video clip or audio clip:

import vapoursynth as vs
my_video = vs.core.lsmas.LibavSMASHSource('video.mp4')
my_audio = vs.core.bas.Source('video.mp4')
if isinstance(my_video, vs.VideoNode):
print('It is vapoursynth.VideoNode object that is called vnode in new docs for R55 and above')
if isinstance(my_audio, vs.AudioNode):
print('It is vapoursynth.AudioNode object that is called anode in new docs for R55 and above')

So yes , it is kind of confusing right now, that link for new docs for API4 (http://www.vapoursynth.com/doc/index.html) , R55 release and above uses vnode reference for video clip and anode for audio clip.