How do I fix combing issues with Double Height Nnedi3?

[Reclusive Eagle]

Doubling the height of the source with Nnedi3 creates the clearest image by far. However it also has the side effect of massive combing issues.
clip=core.nnedi3.nnedi3(clip, field=1, dh=True, nsize=4, nns=4, qual=2, etype=0, pscrn=2,exp=2)

How do I fix this? Here is a comparison of Double height True vs False
https://imgsli.com/Nzk1Nzc

[videoh]

Maybe your field order is wrong?

[Reclusive Eagle]

Maybe your field order is wrong?

Its tff, but I tried all combinations of fields

[videoh]

Can you post a link to an unprocessed source sample that includes that frame?

[Reclusive Eagle]

Can you post a link to an unprocessed source sample that includes that frame?

Yup, here (https://drive.google.com/file/d/1Ph9pNK4Titguf4rddlHgivaHMKl2K-gk/view?usp=sharing) are the first 20 seconds. Extracted with ffmpeg with -c copy

[videoh]

Did you do IVTC first? If so, how?

If you can restore it to progressive using IVTC you will not get any issues with nnedi3(). But it's hard to get clean progressive for this clip, due to the fades and scenes with just small mouth movements.

So get your IVTC working first.

[Reclusive Eagle]

Did you do IVTC first? If so, how?

If you can restore it to progressive using IVTC you will not get any issues with nnedi3(). But it's hard to get clean progressive for this clip, due to the fades and scenes with just small mouth movements.

So get your IVTC working first.

I do have a lot of success with vivtc:
clip = core.vivtc.VFM(clip, order=1, field=2, mode=3, cthresh=12, blockx=32, blocky=32)

However its still not as clean.
Is IVTC better? Or should I attempt to get AnimeIVTC running in Vapoursynth?

Edit Ok layering Nnedi3 onto VIVTC gives me what I want, however I'd still like to know which is better. VIVTC or IVTC?
Also in regards to Nnedi3 Row, should I have to before deinterlacing, after or after Nnedi3?

[Selur]

You can also try: QTGMC + sRestore + (one or more) DeSpot

[Reclusive Eagle]

You can also try: QTGMC + sRestore + (one or more) DeSpot

I tried QTGMC and it fixes the combing however QTGMC is struggling with close lines. Its blurring them together.

Here (https://imgsli.com/Nzk2MDg)is an example.
This isn't even the worst of it. Do you know of anyway to solve QTGMC blur?
And this is after playing with TR values. It looks worse when they are default. And I don't know much about all of QTGMC's features only basics

[Selur]

I tried QTGMC and it fixes the combing however QTGMC is struggling with close lines. Its blurring them together.
Since your source has tons of dublicates, I would try QTGMC + sRestore and add DeSpot to remove some of the residual artifacts. (adding multiple DeSpot calls can remove more, but might cause problems in some scenes)

This isn't even the worst of it. Do you know of anyway to solve QTGMC blur?
From a screenshot: no clue

[lansing]

Do not use a deinterlacer for anime contents, you need IVTC

[Reclusive Eagle]

Do not use a deinterlacer for anime contents, you need IVTC

I'm actually losing fine detail with all versions of IVTC. Compared to QTGMC with this specific source.
All I need to do is figure out how to solve why QTGMC is blurring lines.
If I can't VIVTC is the next best thing

Edit:
Seems if I sharpen the source before IVTC I can combat the detail loss

[Reclusive Eagle]

Since your source has tons of dublicates, I would try QTGMC + sRestore and add DeSpot to remove some of the residual artifacts. (adding multiple DeSpot calls can remove more, but might cause problems in some scenes)


From a screenshot: no clue

QTGMC and sRestore are all that is needed with this specific source. There aren't any residual artifacts but even sRestore or decimate results in the close line blur.

clip = haf.QTGMC(clip, Preset='Placebo',TR2=3, TFF=True, SourceMatch=3, Lossless=2,EdiMode='NNEDI3',NNSize=4,NNeurons=4)
clip = haf.srestore(clip,frate=29.97)

clip=core.nnedi3.nnedi3(clip, field=1, dh=True, nsize=4, nns=4, qual=2, etype=0, pscrn=2,exp=2)

This is what I have for QTGMC. Even removing all settings, just leaving (clip, Preset='Placebo', TFF=True) results in the same issue.
Even just using QTGMC and displaying the output results in the blur or really close lines.

[Selur]

clip = haf.srestore(clip,frate=29.97)
(I see that something way lower that would get rid of the duplicates is more uncommon to use.)
Why 29.97 and not 23.976?

As a side note:
VIVTC:

clip2clip = clip
clip2clip = core.nnedi3cl.NNEDI3CL(clip=clip2clip, field=1)
clip = core.vivtc.VFM(clip=clip, order=0)
# VDecimate helper function
def postprocess(n, f, clip, deinterlaced):
if f.props['_Combed'] > 0:
return deinterlaced
else:
return clip
clip = core.std.FrameEval(clip=clip, eval=functools.partial(postprocess, clip=clip, deinterlaced=clip2clip), prop_src=clip)
clip = core.vivtc.VDecimate(clip=clip)# new fps: 23.976
seems fine.
No clue why QTGMC does what it does. :)

[Reclusive Eagle]

(I see that something way lower that would get rid of the duplicates is more uncommon to use.)
Why 29.97 and not 23.976?

As a side note:
VIVTC:

clip2clip = clip
clip2clip = core.nnedi3cl.NNEDI3CL(clip=clip2clip, field=1)
clip = core.vivtc.VFM(clip=clip, order=0)
# VDecimate helper function
def postprocess(n, f, clip, deinterlaced):
if f.props['_Combed'] > 0:
return deinterlaced
else:
return clip
clip = core.std.FrameEval(clip=clip, eval=functools.partial(postprocess, clip=clip, deinterlaced=clip2clip), prop_src=clip)
clip = core.vivtc.VDecimate(clip=clip)# new fps: 23.976
seems fine.
No clue why QTGMC does what it does. :)

This definitely fixes the combing issues!
Could you break down the script? This is not how I would have even have thought to write this. (Though I am new to Python).

Also any specific reason why Nnedi3CL instead of Libnnedi3?
Just trying to learn as much as possible.

Also as for 29.97 fps, I really can not find any consensus on what to do with interlaced footage in terms of frame rate.
23.97 is the most suggested however most of it has to do with SD anime on DVDs. My source is a blu-ray with native 29.97 as apposed to 23.97.
Its really hard to tell what to do with anime especially when most of the information is over a decade old.

I also have to at some point get it to 25fps as the source audio for the English version is PAL so that's another story.

[lansing]

Also as for 29.97 fps, I really can not find any consensus on what to do with interlaced footage in terms of frame rate.
23.97 is the most suggested however most of it has to do with SD anime on DVDs. My source is a blu-ray with native 29.97 as apposed to 23.97.
Its really hard to tell what to do with anime especially when most of the information is over a decade old.

I also have to at some point get it to 25fps as the source audio for the English version is PAL so that's another story.

Old anime are guaranteed to be 23.976 fps natively, so 29.97 fps is definitely the wrong frame rate. And what is the region of your blu-ray. I had brought a US blu-ray anime where it only did half of the IVTC job, with field matching but no decimation. So it ended up with 29.97 instead of 23.976. It may also be your case here.

[Reclusive Eagle]

Old anime are guaranteed to be 23.976 fps natively, so 29.97 fps is definitely the wrong frame rate. And what is the region of your blu-ray. I had brought a US blu-ray anime where it only did half of the IVTC job, with field matching but no decimation. So it ended up with 29.97 instead of 23.976. It may also be your case here.

Japan, Idk how they did the interlacing. I believe even the SD dvd is like this. Certain times there are massive question marks when the people releasing content can't even agree on the standards

[Selur]

Also any specific reason why Nnedi3CL instead of Libnnedi3?
NNEDI3 uses the gpu and usually is faster. Also if your gpu can't be used znedi3 (https://github.com/sekrit-twc/znedi3) would be the next best alternative. I don't really see a reason why you would use nnedi3.

Could you break down the script?
Sure.
What the script does is add postprocessing to VIVTC like suggesteg in the VIVTC documentation (https://github.com/vapoursynth/vivtc/blob/master/docs/vivtc.rst), but instead of using eedi3 it uses NNEDI3 (or more precise nnedi3cl).
Here's the script with some comments:
clip2clip = clip
clip2clip = core.nnedi3cl.NNEDI3CL(clip=clip2clip, field=1) # with NNEDI3 deinterlaced clip
clip = core.vivtc.VFM(clip=clip, order=0) # normal field matching

# VDecimate helper function which looks at a frame and returns the deinterlaced frame if the frame is marked as combed and otherwise returns the field matched frame
def postprocess(n, f, clip, deinterlaced):
if f.props['_Combed'] > 0:
return deinterlaced
else:
return clip
# used the helper function to get rid of detected combing
clip = core.std.FrameEval(clip=clip, eval=functools.partial(postprocess, clip=clip, deinterlaced=clip2clip), prop_src=clip)
clip = core.vivtc.VDecimate(clip=clip)# 3:2 decimation

Cu Selur

[Reclusive Eagle]

NNEDI3 uses the gpu and usually is faster. Also if your gpu can't be used znedi3 (https://github.com/sekrit-twc/znedi3) would be the next best alternative. I don't really see a reason why you would use nnedi3.

Ahhh so its like BM3DCuda. I thought libnnedi3 was the GPU version and CL was the CPU version.

Thanks Selur! Will definitely help with my code.
Also is there a way to do this with Double height on NNEDI3?

Doing so results in this error:
"FrameEval: Returned frame has wrong dimensions"

Which makes sense because clip is 720x480 and using DH=True doubles to 720x980.
But downsizing NNEDI3 or changing clip names after Double height and then decombing effectively chancels out the resolution gain with double height.
https://imgsli.com/Nzk3Mjg

Edit apparently changing frame eval to the double height clip crashes vapoursynth too

[_Al_]

placeholder = core.std.BlankClip(clip, height=clip.height*2)
clip = core.std.FrameEval(clip=placeholder, eval= ...a function that doubles height...
try to pass placeholder instead of a clip

[Reclusive Eagle]

For example. Your code works but when I substitute double height in I get frame errors:


clip2clip = clip
#Double height Nnedi3
clip2clip=core.nnedi3.nnedi3(clip, field=1, dh=True, nsize=4, nns=4, qual=2, etype=0, pscrn=2,exp=2)

#resize clip to clip2clip dimensions
clip = core.resize.Lanczos(clip, 720,960)

clip = core.vivtc.VFM((clip=clip, order=0))
def postprocess(n, f, clip, deinterlaced):
if f.props['_Combed'] > 0:
return deinterlaced
else:
return clip
# used the helper function to get rid of detected combing
clip = core.std.FrameEval(clip=clip, eval=functools.partial(postprocess, clip=clip, deinterlaced=clip2clip), prop_src=clip)


But resizing clip to clip2clip height just gives me back the same clip

[Reclusive Eagle]

placeholder = core.std.BlankClip(clip, height=clip.height*2)
clip = core.std.FrameEval(clip=placeholder, eval= ...a function that doubles height...
try to pass placeholder instead of a clip

The video plays but returns "FrameEval: Returned frame has wrong dimensions" when displaying frames that have been decombed.

Basically:
Using Nnedi3 double height results in frame error.
Changing dimensions to match double height before decomb results in the same combed clip.
Changing dimensions to match original height before decomb results in lower resolution as NNEDI3 double height takes into account interweaved frames to increase resolution.
Changing names so FrameEval recognizes the double height NNEDI3 clip crashes Vapoursynth

Anything else results in the same clip with combing issues or lower resolution.
I just want a decombed clip with NNEDI3 double height for the increased resolution