Detelecine VIVTC VFM + QTGMC produce some jagged artifacts

[shph]

Hello. A lot of cartoon DVDs show interlaced frames in dynamic scenes when you simply detelecine them, so sometimes it may be a real problem.

VIVTC VFM detelecine in Hybrid dvelopment version was extended, and in addition to basic "inner" deinterlacer it now provide QTGMC presets (Fast to Very Slow) and NNEDi3.
So i tested some vob examples with new VIVTC VFM detelecine + QTGMC and NNEDi3 and here is what i got:

- QTGMC seems like better than basic "inner" deinterlacer. It 100% removes all interlaced fields that left after detelecine even if used simplest VFM detelecine preset.
- NNEDi3 quality is less accurate than QTGMC.

But here is a problem:

- QTGMC may produce jagged artifacts in straight diagonal lines and some other fine line details. It feels like it attempt to deinterlace progressive frames by mistake.
- Those jagged artifacts are not aliasing and it is impossible to filter them with AA filters.

Here are soeme examples:
https://i.imgur.com/efRTBHX.jpghttps://i.imgur.com/4gnSzJ5.jpg

Source vob file for test:
https://drive.google.com/file/d/1sxFEYQ_pf1x_VFM9VQec8ydZgLGY35Kw/view?usp=sharing

Seems it was a discussion about similar problem here https://forum.doom9.org/showthread.php?t=172825
And the basic problem fix was: "The jaggies are not caused by QTGMC itself, they are caused by TDeint.
Simply using your QTGMC call alone, without TDeint, results in a jaggie-less video.
The problem is you are wrapping QTGMC in TDeint. I suspect it is caused by the fact that by default QTGMC does not leave the original fields unaltered. TDeint does, so the interpolated fields by QTGMC do not match the original ones that TDeint injects, resulting in jaggies."

So here are some questions:
- Is it some bug in QTGMC?
- Is it some bug in relation between QTGMC and VIVTC VFM?
- Is it possible somehow adapt QTGMC script specially for VIVTC VFM detelecine purposes?

Discussion was started here https://forum.selur.net/showthread.php?tid=1495&pid=8887#pid8887 but seems it is more like common problem and not directly related to Hybrid.
Any real life suggestions welcome.

[AdamCarter]

I noticed they mentioned
Instead of using tdeint
https://forum.doom9.org/showthread.php?p=1745735#post1745735
To use a TCombMask+IsCombedTIVTC+ConditionalFilter

So maybe:
We need to use a different tdeint tdeintmod
https://github.com/HomeOfVapourSynthEvolution/VapourSynth-TDeintMod
IsCombedTIVTC Seems to be included in this tdeintmod


It only runs the deinterlacer on combed frames.

They had mentioned a combmask too not sure if its needed but i found this
combmask https://github.com/chikuzen/CombMask/tree/master/vapoursynth

[ChaosKing]

I used this combo for left over combed frames with qtgmc.


def conditionalDeint(n, f, orig, deint):
if f.props._Combed:
return deint#.sub.Subtitle("DEINT", margins=[20,20,70,20])
else:
return orig



clip = core.lsmas.LWLibavSource(source="...")
clip = clip.vivtc.VFM(0, mode=3, cthresh=8, blockx=32, blocky=32)
clip = clip.vivtc.VDecimate()

deint = haf.QTGMC(clip, Preset='very slow', TR2=1, TFF=True, SourceMatch=3 ).std.SelectEvery( cycle=2, offsets=0) #Sharpness=0.7
combProps = core.tdm.IsCombed(clip, metric=1, cthresh=5, blockx=32, blocky=32, mi=90)
clip = core.std.FrameEval(clip, functools.partial(conditionalDeint, orig=clip, deint=deint), combProps)

[shph]

Interesting... This is how VIVTC VFM + QTGMC code currently look in Hybrid:
clip2clip = havsfunc.QTGMC(Input=clip2clip, Preset="Very Slow", opencl=True, TFF=False,FPSDivisor=2)
clip = core.vivtc.VFM(clip=clip, clip2=clip2clip, order=0, field=3)
clip = core.vivtc.VDecimate(clip=clip, clip2=clip2clip)# new fps: 23.976

[Selur]

What Hybrid offers is using QTGMC as alternative to the internal deinterlaced during IVTC.
What ChaosKing does is to identify the still combed frames (using tdm.isComded; tdm is part of the TDeintMod filter) after IVTC and apply QTGMC on those.
-> he purposes an additional step after the normal IVTC (independent of whether QTGMC is used during IVTC or not)
Also note that:
a. Hybrid addresses havsfunc through 'hasfunc' while ChaosKing addresses it through 'haf'
b. Hybrid uses 'FPSDivisor=2' which ChaosKing uses '.std.SelectEvery( cycle=2, offsets=0)'
c. Hybrid uses 'TFF=False' while ChaosKing uses 'TFF=True' in QTGMC
c. note that your script needs 'import functools' which Hybrid doesn't import by default.
in case you try to add his proposal as a custom addition to the Vapoursynth script in Hybrid.

So one could combine those two and use QTGMC for both the deinterlacing during IVTC and for the additional decombing and would have something like:
clip = core.d2v.Source(input="E:/Temp/mpg_2bcf42122044faeeeecd635342234255_853323747.d2v")
# making sure input color matrix is set as 470bg
clip = core.resize.Point(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 29.970
clip = core.std.AssumeFPS(clip, fpsnum=30000, fpsden=1001)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
clip2clip=clip
clip2clip = havsfunc.QTGMC(Input=clip2clip, Preset="Very Slow", opencl=True, TFF=False,FPSDivisor=2)
clip = core.vivtc.VFM(clip=clip, clip2=clip2clip, order=0, mode=1)
clip = core.vivtc.VDecimate(clip=clip, clip2=clip2clip)# new fps: 23.976
# Fix combed frames
# adding helper function
def conditionalDeint(n, f, orig, deint):
if f.props._Combed:
return deint
else:
return orig
clipDeint = havsfunc.QTGMC(Input=clip, Preset="Very Slow", opencl=True, TFF=False, TR2=1, SourceMatch=3, FPSDivisor=2)
clipCombProps = core.tdm.IsCombed(clip=clip, blockx=32, blocky=32)
clip = core.std.FrameEval(clip=clip, eval=functools.partial(conditionalDeint, orig=clip, deint=clipDeint), clipCombProps)
but even when using nnedi3cl instead of nnedi3 or znnedi3 this will probably terribly slow,.... (as to be expected when using QTGMC 'very slow' two times. ;)


Cu Selur

[shph]

By the way, i experimented with QTGMC deinterlace applied to PAL dvds that recorded in unusual but popular format when movie looks progressive inside interlaced VOB container. It was not animation but basic music video movie. QTGMC magically combined those double progressive frames (odd/even frame look near the same), cleans noise and filters tiny pixel level interlace artifacts (sort of combined double exposure effect). But same time i see that qtgmc have the same problem with jagged edges when it goes to straight lines in the movie.
I understand that probably it is not correct to apply QTGMC like this, but i guess it would be really nice if QTGMC was improved somehow and just don't produce jagged edges in any possible situations.

[Selur]

any possible situations
dream on,...