Weird lines with QTGMC

[Reclusive Eagle]

I've been noticing these weird alising issues with QTGMC.
Its not combing. Here (https://imgsli.com/Nzk0NTA) is a comparison between QTGMC and how it should look.

Here are my QTGMC settings:
clip = haf.QTGMC(clip, Preset='Placebo',TR2=3, TFF=True, SourceMatch=2, Lossless=0, EdiMode='NNEDI3', NNSize=4, NNeurons=4)
clip = haf.srestore(clip,frate=29.97)

However even with just clip = haf.QTGMC(clip, Preset='Placebo') it looks like this. Am I missing something?

When I use nnedi by itself it produces clean lines:
clip=core.nnedi3.nnedi3(clip, field=1, dh=True, nsize=4, nns=4, qual=2, etype=0, pscrn=2,exp=2)

However even using the same nnedi settings in QTGMC I can't solve the issue. QTGMC is so much sharper, the only problem is this issue.

I think its interpreting the lines wrong. If I refresh with nnedi it looks like the image shifts 1-2 pixels up and its completely fine.
With QTGMC it shifts down and creates these issues.

[lansing]

QTGMC has a "sharpness" parameter, what if you set it to 0?

[Reclusive Eagle]

QTGMC has a "sharpness" parameter, what if you set it to 0?

Does nothing. Just makes the lines sharper but doesn't fix the issue

[shph]

What is your VapourSynth version?
Could it be the same (or similar) bug as in Vapoursynth R51 https://github.com/vapoursynth/vapoursynth/issues/625 (it was fixed later in R52)

[Reclusive Eagle]

What is your VapourSynth version?
Could it be the same (or similar) bug as in Vapoursynth R51 https://github.com/vapoursynth/vapoursynth/issues/625 (it was fixed later in R52)

VS is R57 with Python 3.9.7
Let me try in R53.

Edit:
Nope still a thing. I guess its just how QTGMC is and why people say reverse telecine instead of it. Yes it adds a lot.
But it also seems to destroy certain details

[shph]

QTGMC applied to progressive frames usually adds artifacts.
"SourceMatch" and "Lossless" other than =0 also may add jagged artifacts if source is low quality. So in addition to Sharpness=0.0 Try to use SourceMatch=0 and see if it makes difference.

Resizing to HD with NNEDI3 usually helps to remove jagged artifacts.

[Jukus]

QTGMC has never spoiled the progressive picture for me

[shph]

Probably you just didn't noticed it. Effect is mostly well visible in animation or sometimes in movies that have fine sharp lines located very close one to each other. Effect is not permanent. It appears randomly in different places that contains fine lines because QTGMC "thinks" that those fine lines are interlaced fields.
It is not happening if you apply QTGMC denoiser only to progressive video.
It it happening if you apply QTGMC deinterlacer to progressive (or to telecined) source.

[Reclusive Eagle]

QTGMC applied to progressive frames usually adds artifacts.
"SourceMatch" and "Lossless" other than =0 also may add jagged artifacts if source is low quality. So in addition to Sharpness=0.0 Try to use SourceMatch=0 and see if it makes difference.

Resizing to HD with NNEDI3 usually helps to remove jagged artifacts.

I've tried all combinations of source match, lossless and the 3 TR values.
The source is interlaced not progressive

[Reclusive Eagle]

Probably you just didn't noticed it. Effect is mostly well visible in animation or sometimes in movies that have fine sharp lines located very close one to each other. Effect is not permanent. It appears randomly in different places that contains fine lines because QTGMC "thinks" that those fine lines are interlaced fields.
It is not happening if you apply QTGMC denoiser only to progressive video.
It it happening if you apply QTGMC deinterlacer to progressive (or to telecined) source.

See I'm not crazy. When it comes to IRL movies and videos the effects may not be notable.

However considering how real video has even smaller and thinner lines than Anime tends to have QTGMC might actually have a real, massive problem.
I've also noticed the "effect" slightly diminishes over a certain amount of frames. Since QTGMC blends frames together this is the reason for the slight diminishing.

However what I have noticed is Double Height NNEDI3 into IVTC fixes the issue entirely.
The problem is you can't use NNEDI3 into QTGMC. Any scaling of the source results in broken deinterlacing with it.
But also last time I check QTGMC IS supposed to double frame height.

That's why its much sharper compared to conventional deinterlace. Because its not throwing away resolution.
But what makes it really weird is despite double height being the solution to this issue QTGMC still has it. Its almost as if it doubles the height at the end after interpolation.
Which then retains all the small issues that persist from a half height 240p field.

[poisondeathray]

The source is interlaced not progressive

No, the content is mostly progressive

Main content animation is rarely interlaced, except for some sections added afterwards like interlaced fades, or titles, or credits

In general - whenever you deinterlace progressive content, you cause artifacts and distortions

In general you want to IVTC (field match + decimate) stuff like anime, to return the original quality progressive frames.

It's especially bad for animation that has duplicate frames when using temporal deinterlacers. e.g. QTGMC. QTGMC will cause new motion in a duplicate frame, so lines can move or be displaced, when they should be otherwise stable


(There are exceptions - sometimes very bad degraded sources can benefit from QTGMC smoothing problems over)





However what I have noticed is Double Height NNEDI3 into IVTC fixes the issue entirely.

You would expect that, because IVTC returns the full quality field matched frames (both fields belonging to the same frame)



The problem is you can't use NNEDI3 into QTGMC. Any scaling of the source results in broken deinterlacing with it.
But also last time I check QTGMC IS supposed to double frame height.



The problem is deinteralcing vs. ivtc (field matching + decimation)

QTGMC is still a deinterlacer at the most basic level . Although it's very good it will still degrade progressive content (all deinterlacers will degrade progressive content)

It will double field height, not frame height. So you're starting with 1/2 the resolution for any deinterlacer. IVTC returns full resolution for progressive content (in general, sometimes you have orphaned fields)

QTGMC is slightly better than traditional deinterlacer in that it takes temporal information into account, compared to traditional spatial deinterlacer . But this can be bad too, because it predisposes to temporal artifacts

[shph]

I mean first deinterlace with QTGMC or VIVTC, then resize to HD with NNEDI3. It should smooth jagged edges. Also if possible share short sample of source video so other people can check it.
https://i.imgur.com/cvCHnPL.jpg
https://i.imgur.com/aoMfhCt.jpg

[Reclusive Eagle]

I mean first deinterlace with QTGMC or VIVTC, then resize to HD with NNEDI3. It should smooth jagged edges. Also if possible share short sample of source video so other people can check it.
https://i.imgur.com/cvCHnPL.jpg
https://i.imgur.com/aoMfhCt.jpg

I thought It was this thread where I posted source, my bad.
Here (https://drive.google.com/file/d/1Ph9pNK4Titguf4rddlHgivaHMKl2K-gk/view?usp=sharing) you go.

Also despite VIVTC and NNEDI3 I still have combing issues when it comes to fast moving content like transitions.

Example (https://imgsli.com/Nzk3Njc)
Despite Selur's help in another thread. I can't figure out how to apply his conditional Combing to double height NNEDI3.
If I try deinterlace with QTGMC then use NNEDI it just compounds the problem.

So currently I am forced into choosing either a lower res file with no interlacing or a higher res file with combing on fast moving frames.
Really not an ideal situation.

[Reclusive Eagle]

Also the whole reason I want to do this is because of how much sharper it is.
If I can get QTGMC +NNEDI3 to work together without the line blurring here is the difference:

Comparison (https://imgsli.com/Nzk3NzI)
See how much sharper QTGMC + NNEDI3 is compared to VIVTC + NNEDI3?
Its not even a comparison.

But even in this perfect scenario frame, if you zoom in you can see issues with QTGMC.
So I have 2 situations I am stuck in here I am forced to choose now.

I can choose the much much sharper QTGMC + NNEDI3 and create artifacts like line blurring and gradient issues.
Or I can choose the much less sharper VIVTC + NNEDI3 that has combing issues on fast moving frames after IVTC

If I Choose VIVTC + NNEDI3, I have to choose either:
*Half res with no combing issues
or
*Double height with combing issues.

So its like a lose lose situation in both choices.

[poisondeathray]

Also the whole reason I want to do this is because of how much sharper it is.
If I can get QTGMC +NNEDI3 to work together without the line blurring here is the difference:

Comparison (https://imgsli.com/Nzk3NzI)
See how much sharper QTGMC + NNEDI3 is compared to VIVTC + NNEDI3?
Its not even a comparison.

But even in this perfect scenario frame, if you zoom in you can see issues with QTGMC.
So I have 2 situations I am stuck in here I am forced to choose now.

I can choose the much much sharper QTGMC + NNEDI3 and create artifacts like line blurring and gradient issues.
Or I can choose the much less sharper VIVTC + NNEDI3 that has combing issues on fast moving frames after IVTC

If I Choose VIVTC + NNEDI3, I have to choose either:
*Half res with no combing issues
or
*Double height with combing issues.

So its like a lose lose situation in both choices.



But QTGMC looks bad - because of the oversharpening halos . Check other frames, you probably have other artifacts too

"sharper" isn't "better" if you have halos

QTGMC sharpens by default. You can apply (over)sharpen filter too to IVTC


How are you getting combing with vivtc ? Is it residual combing ? Are you applying PP ? Where is the source video ?

Option 3,4,5 are to use machine learning scaling . e.g. waifu, esrgan etc... it will be sharper without the halos

[shph]

Just add sharpness in the end (i prefer CAS filter) and VIVTC + NNEDI3 will look sharper and better (less edge ringing effect).
Your source is Variable frame rate with randomly telecined frames. You need to use VIVTC or TIVTC to convert it to progressive. VIVTC or TIVTC have some special modes for sources like these but they are not perfect and can partially remain some tiny amount of interlaced artifacts.
You can also check this thread https://forum.doom9.org/showthread.php?t=183265 and try function provided by PRAGMA for complicated video sources.

There is also windows only AnimeIVTC http://avisynth.nl/index.php/AnimeIVTC

There is also YATTA (for VapourSynth named Wobbly) for very complicated manual work https://forum.doom9.org/showthread.php?t=172496

For sharp and really quality anime upscale you can use VSGAN (Windows only) package weight is about 5GB or so.

This is how TIVTC + NNEDI3 code looks in Hybrid:
# source: '/Volumes/AAA/TMM.mkv'
# current color space: YUV420P8, bit depth: 8, resolution: 720x480, fps: 29.97, color matrix: 470bg, yuv luminance scale: limited, scanorder: telecine (soft)
# Loading /Volumes/AAA/TMM.mkv using LWLibavSource
clip = core.lsmas.LWLibavSource(source="/Volumes/AAA/TMM.mkv", format="YUV420P8", cache=0, prefer_hw=0)
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 29.970
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
clip2clip = clip
clip2clip = havsfunc.QTGMC(Input=clip2clip, Preset="slow", opencl=True, TFF=True,FPSDivisor=2)
# Deinterlacing using TIVTC
clip = core.tivtc.TFM(clip=clip, clip2=clip2clip)
clip = core.tivtc.TDecimate(clip=clip, mode=1)# new fps: 23.976
# make sure content is preceived as frame based
clip = core.std.SetFieldBased(clip, 0)
# DEBUG: vsTIVTC changed scanorder to: progressive
# resizing using NNEDI3CL
# resizing using NNEDI3CL
clip = edi_rpow2.nnedi3cl_rpow2(clip=clip, rfactor=4) # 2880x1920
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1474, h=1080, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: YUV420P16 to YUV420P8 for FFV1Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, range_s="limited")
# set output frame rate to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()

[Reclusive Eagle]

Just add sharpness in the end (i prefer CAS filter) and VIVTC + NNEDI3 will look sharper and better (less edge ringing effect).
Your source is Variable frame rate with randomly telecined frames. You need to use VIVTC or TIVTC to convert it to progressive. VIVTC or TIVTC have some special modes for sources like these but they are not perfect and can partially remain some tiny amount of interlaced artifacts.
You can also check this thread https://forum.doom9.org/showthread.php?t=183265 and try function provided by PRAGMA for complicated video sources.

There is also windows only AnimeIVTC http://avisynth.nl/index.php/AnimeIVTC

There is also YATTA (for VapourSynth named Wobbly) for very complicated manual work https://forum.doom9.org/showthread.php?t=172496

For sharp and really quality anime upscale you can use VSGAN (Windows only) package weight is about 5GB or so.

This is how TIVTC + NNEDI3 code looks in Hybrid:
# source: '/Volumes/AAA/TMM.mkv'
# current color space: YUV420P8, bit depth: 8, resolution: 720x480, fps: 29.97, color matrix: 470bg, yuv luminance scale: limited, scanorder: telecine (soft)
# Loading /Volumes/AAA/TMM.mkv using LWLibavSource
clip = core.lsmas.LWLibavSource(source="/Volumes/AAA/TMM.mkv", format="YUV420P8", cache=0, prefer_hw=0)
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 29.970
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
clip2clip = clip
clip2clip = havsfunc.QTGMC(Input=clip2clip, Preset="slow", opencl=True, TFF=True,FPSDivisor=2)
# Deinterlacing using TIVTC
clip = core.tivtc.TFM(clip=clip, clip2=clip2clip)
clip = core.tivtc.TDecimate(clip=clip, mode=1)# new fps: 23.976
# make sure content is preceived as frame based
clip = core.std.SetFieldBased(clip, 0)
# DEBUG: vsTIVTC changed scanorder to: progressive
# resizing using NNEDI3CL
# resizing using NNEDI3CL
clip = edi_rpow2.nnedi3cl_rpow2(clip=clip, rfactor=4) # 2880x1920
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1474, h=1080, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: YUV420P16 to YUV420P8 for FFV1Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, range_s="limited")
# set output frame rate to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()

The comparison has CAS with sharpness of 1 + LSMOD with additional sharpness and it still doesn't come close to QTGMC. Plus this leads to massive sharpening halos and ringing which is the specific reason why I want QTGMC.

Edit:
Using the comparison of:

clip2clip = clip
clip2clip = haf.QTGMC(Input=clip2clip, Preset="slow", opencl=True, TFF=True,FPSDivisor=2)

clip = core.tivtc.TFM(clip2=clip2clip,clip=clip)
clip = core.tivtc.TDecimate(clip=clip, mode=1)# new fps: 23.976

#Nnedi3
clip=core.nnedi3.nnedi3(clip, field=1, dh=True, nsize=4, nns=4, qual=2, etype=0, pscrn=2,exp=2)

still results in QTGMC blur even with my normal Placebo preset. And on top of that removes the QTGMC sharpness so still struggling

[Reclusive Eagle]

But QTGMC looks bad - because of the oversharpening halos . Check other frames, you probably have other artifacts too

"sharper" isn't "better" if you have halos

QTGMC sharpens by default. You can apply (over)sharpen filter too to IVTC


How are you getting combing with vivtc ? Is it residual combing ? Are you applying PP ? Where is the source video ?

Option 3,4,5 are to use machine learning scaling . e.g. waifu, esrgan etc... it will be sharper without the halos

I mean yes there are issues with QTGMC and bare in mind I can dehalo the sharpness with edge masks so that's not an issue to me.

My problem is to get any comparable sharpness out of VIVTC results in even more sharpening halos and more detrimental effects.

Hence why QTGMC is preferred here because I can work with the trade offs much easier than VIVTC.

Source sample is here
https://drive.google.com/file/d/1Ph9pNK4Titguf4rddlHgivaHMKl2K-gk/view?usp=sharing

In regards to upscaling. I am remastering this footage so I am trying to keep as much of the original line work as possible.
Upscaling will always alter the scene. Regardless of how well trained your models are. Upscaling should be used at the end of the remastering process when you have the cleanest source footage to work with. Also Upscaling for me with my GPU is 20 seconds per frame. i.e 400 hours per episode.

That is not feasible for me. And again, unless I get the source to be extremely clean there is no point in upscaling with a neural network because it modifies the line work even worse than AWarpsharp. Bare in mind this footage is not just for me its for the entire fandom.

In order to meet a minimum quality standard I need to modify the footage as little as physically possible because that's the goal I have.
And how remasters should be treated.

Unless I can get Plum working, which I still haven't been able to. Then I can sharpen through deconvolution and then QTGMC isn't even needed

[shph]

Upscale with NNEDi3 is artifacts free. It is not that sort of toon AI that damage lines. And it is rather fast compare to other more complicated tools. RX580 can do realtime at highest settings. Upscale is not only about changing size. It allow to apply filters in the way that impossible in SD source. Quality upscale also filters a lot of unwanted artifacts.
Here https://forum.doom9.org/showthread.php?p=1907921#post1907921 i shared some tests how NNEDI3 upscale can dramatically improve quality of SD source.

I can only agree with poisondeathray. QTGMC just adds huge amount of basic oversharpening by default.

For anime in most case you need to use DeRing, but not DeHalo.

[poisondeathray]

I mean yes there are issues with QTGMC and bare in mind I can dehalo the sharpness with edge masks so that's not an issue to me.


Wouldn't it be better to not create the halos in the first place ? Dehaloing typically is one of the most damaging operations, even with masks. It's easy to not sharpen as much, the source didn't have bad halos to begin with.


My problem is to get any comparable sharpness out of VIVTC results in even more sharpening halos and more detrimental effects.

Hence why QTGMC is preferred here because I can work with the trade offs much easier than VIVTC.



What were the other detrimental effects ? The IVTC combing ? Were they worse than QTGMC's other detrimental effects ?

I would look at it another way - It's relatively easy to (over)sharpen a good image from a proper IVTC. There are many categories/types of sharpeners and you have lots of control. It's difficult / almost impossible to remove the aliasing and temporal artifacts ,and impossible to recover the lost details caused by QTGMC.

For IVTC comb fallback, you can use TIVTC with pp, or tdm.IsCombed since you have at least some interlaced fades and some issues with combed scene changes. It's similar to TFM pp=x in avisynth, where you can adjust the cthresh and MI. VFM in vapoursynth doesn't have built in pp mode. You can use QTGMC (or something else) selectively for the detected combed frames. Residual fine combing can be treated with vinverse/vinverse2 - it won't cause as much damage as QTGMCing everything



In regards to upscaling. I am remastering this footage so I am trying to keep as much of the original line work as possible.
Upscaling will always alter the scene. Regardless of how well trained your models are. Upscaling should be used at the end of the remastering process when you have the cleanest source footage to work with. Also Upscaling for me with my GPU is 20 seconds per frame. i.e 400 hours per episode.

That is not feasible for me. And again, unless I get the source to be extremely clean there is no point in upscaling with a neural network because it modifies the line work even worse than AWarpsharp. Bare in mind this footage is not just for me its for the entire fandom.



I assumed you were "upscaling" because the images you posted in this thread and other threads were 1536x1152

Not all models are that "extreme". Some are quite conservative. Some have built in NR, and border fixes such as in your sample (oversharpened vertical line by the pillar bars, that one was already in source) . For ESRGAN, you can interpolate between models , or use layer blending to mix. But yes, it's very slow...One option to speed it up is to decimate the duplicates and make it (VFR) using dedup or similar methods. Also, unique frames helps with almost all temporal filtering operations, including temporal denoising . Anyways it sounds like you've ruled it out already...

Weren't you going to use QTGMC? - You could argue that oversharpen halos , aliasing, temporal artifacts , wobbling lines are not part of the "original line work" either...Just playing Devil's advocate here... :D

And weren't you going to BM3D denoise ? That's going to soften some textures and lines even with good line masks. It will "alter the scene" too , to a degree. :D Not exactly a fast filter either. Any denoising will degrade the original textures to some extent, so you have to make some tradeoffs