New filter: Fix Telecined Fades

[feisty2]

binary(windows x64, requires msvcr 2017):https://github.com/IFeelBloated/Fix-...eleases/tag/r5
git repo:https://github.com/IFeelBloated/Fix-...ter/Source.cpp

the filter gives a mathematically perfect solution to such(fades were done AFTER telecine which made a picture perfect IVTC pretty much impossible) problem, and it's now time to kiss "vinverse" goodbye cuz "vinverse" is old and low quality.
unlike vinverse which works as a dumb blurring + contra-sharpening combo and very harmful to artifacts-free frames, this filter works by matching the brightness of top and bottom fields with statistical methods, and also harmless to healthy frames.

core.ftf.FixFades(clip, mode=0, threshold=0.002, color=[0.0, 0.0, 0.0], opt=True)

clip: clip to be processed

mode: could be 0(default), 1, or 2
0: adjust the brightness of both fields to match the average brightness of 2 fields.
1: darken the brighter field to match the brightness of the darker field
2: brighten the darker field to match the brightness of the brighter field

threshold: threshold for the average difference per pixel, on a scale of 0.0 - 1.0, but could go beyond 1.0, the frame will remain untouched if the average difference between 2 fields goes below this value

color: base color of the fade, default is [0.0, 0.0, 0.0](black)

opt: call the fastest possible functions if opt=True, else call the C++ functions.

INPUT CLIP MUST BE 32BITS FLOATING POINT FORMAT!!!
apply this filter AFTER field matching!!!

[feisty2]

comparison against vinverse
input

import vapoursynth as vs
core = vs.get_core()
clp = core.lsmas.LWLibavSource("rule6")
clp = core.vivtc.VFM(clp,0)
clp.set_output()

vinverse

import vapoursynth as vs
core = vs.get_core()
clp = core.lsmas.LWLibavSource("rule6")
clp = core.vivtc.VFM(clp,0)

clp = core.vinverse.Vinverse(clp)

clp.set_output()

FTF

import vapoursynth as vs
core = vs.get_core()
clp = core.lsmas.LWLibavSource("rule6")
clp = core.vivtc.VFM(clp,0)
clp = core.fmtc.bitdepth(clp,bits=32,fulls=False,fulld=True)

clp = core.ftf.FixFades(clp)

clp.set_output()

[feisty2]

and vinverse failed miserably at the top-right part of the image, there's clear aliasing around edges, and ftf produced a picture perfect reconstruction.

[Mystery Keeper]

Does it do combing detection? Can you do better combing detection than TDM?

[feisty2]

Originally Posted by Mystery Keeper

Does it do combing detection? Can you do better combing detection than TDM?

No combing detection, it pretty much won't affect the normal frames so no masking required. Technically every pixel will be processed but only the problem frames will be affected.

[ShogoXT]

Does this work in general on IVTC scene transitions? Since using Vapoursynth (through Staxrip) ive been having such a problem with duplicate frames and scthresh with VFM and Vdecimate (no blending???), that ive been forced to stay with mode=0, but scene changes look awful. Ive also been having trouble with dealing with rainbows and dotcrawl in motion, but I guess thats another problem as all the solutions are temporal only.

EDIT: Also scrolling scenes stink...

[feisty2]

Originally Posted by ShogoXT

Does this work in general on IVTC scene transitions? Since using Vapoursynth (through Staxrip) ive been having such a problem with duplicate frames and scthresh with VFM and Vdecimate (no blending???), that ive been forced to stay with mode=0, but scene changes look awful. Ive also been having trouble with dealing with rainbows and dotcrawl in motion, but I guess thats another problem as all the solutions are temporal only.

EDIT: Also scrolling scenes stink...

not sure what you meant by "IVTC scene transitions", it works on any "simple" telecined fades(fade from/to pure color, eg. black), and it won't work on telecined cross-fades(2 or more scenes blending into each other), shit like that is simply beyond repair.

[WolframRhodium]

Great algorithm. The result is fantastic.

Question:
Will the results be even better if the algorithm acting on local rather than global?

[feisty2]

r2:
fixed a stupid memory leak
new parameter "threshold"
new parameter "color"

[feisty2]

Originally Posted by WolframRhodium

Question:
Will the results be even better if the algorithm acting on local rather than global?

done, use the "threshold" parameter

[feisty2]

another demonstration for something other than fading into blackness
input

import vapoursynth as vs
core = vs.get_core()
clp = core.lsmas.LWLibavSource("rule6")
clp = core.vivtc.VFM(clp,0)
clp = core.fmtc.bitdepth(clp,bits=32,fulls=False,fulld=True)
clp = core.std.Expr(clp, ["1.0 x -", "x"]) #invert Y so it would be literally fading into whiteness
clp.set_output()

FTF

import vapoursynth as vs
core = vs.get_core()
clp = core.lsmas.LWLibavSource("rule6")
clp = core.vivtc.VFM(clp,0)
clp = core.fmtc.bitdepth(clp,bits=32,fulls=False,fulld=True)
clp = core.std.Expr(clp, ["1.0 x -", "x"]) #invert Y so it would be literally fading into whiteness

clp = core.ftf.FixFades(clp, color=[1.0, 0.0, 0.0])

clp.set_output()

[jackoneill]

Originally Posted by feisty2

and it won't work on telecined cross-fades(2 or more scenes blending into each other), shit like that is simply beyond repair.

No, no, fades like that can still be saved. By which I mean that you can obtain a 24 fps fade with smooth motion even if field matching is impossible in the fade. The solution involves QTGMC, so the detail preservation isn't awesome, but it's still good, considering the situation.

Steps:
1. Invoke source filter to obtain the untouched 30 fps clip.
2. Extract the fade using Trim.
3. Pass to QTGMC to obtain a 60 fps clip.
4. Stare at the output of QTGMC really carefully. It's been a few months since I did this, so I forget exactly what you're looking for.
5. Extract 4 frames out of a cycle of 10 using SelectEvery.
Tada!

[ShogoXT]

Originally Posted by jackoneill

No, no, fades like that can still be saved. By which I mean that you can obtain a 24 fps fade with smooth motion even if field matching is impossible in the fade. The solution involves QTGMC, so the detail preservation isn't awesome, but it's still good, considering the situation.

Steps:
1. Invoke source filter to obtain the untouched 30 fps clip.
2. Extract the fade using Trim.
3. Pass to QTGMC to obtain a 60 fps clip.
4. Stare at the output of QTGMC really carefully. It's been a few months since I did this, so I forget exactly what you're looking for.
5. Extract 4 frames out of a cycle of 10 using SelectEvery.
Tada!

I was under the impression that if the source is 2:3 telecined that it HAD to be ivtc and not QTGMC. QTGMC is very nice but I thought it was for interlace 121212 only. Vinverse has helped me a little, does the color conversation and fixfade cost much speed?

[fAy01]

Originally Posted by jackoneill

No, no, fades like that can still be saved. By which I mean that you can obtain a 24 fps fade with smooth motion even if field matching is impossible in the fade. The solution involves QTGMC, so the detail preservation isn't awesome, but it's still good, considering the situation.

Steps:
1. Invoke source filter to obtain the untouched 30 fps clip.
2. Extract the fade using Trim.
3. Pass to QTGMC to obtain a 60 fps clip.
4. Stare at the output of QTGMC really carefully. It's been a few months since I did this, so I forget exactly what you're looking for.
5. Extract 4 frames out of a cycle of 10 using SelectEvery.
Tada!

Is there a possibility to write a script/plugin that deinterlaces and matches correctly at the same time without using QTGMC?

[Mystery Keeper]

Originally Posted by ShogoXT

I was under the impression that if the source is 2:3 telecined that it HAD to be ivtc and not QTGMC. QTGMC is very nice but I thought it was for interlace 121212 only. Vinverse has helped me a little, does the color conversation and fixfade cost much speed?

After IVTC you can still find a lot of combed frames of various origins.

[WolframRhodium]

Originally Posted by feisty2

done, use the "threshold" parameter

I mean maybe the calculation of TopFieldSum/BottomFieldSum can be done in a small neighborhood, for example, 16x16 block?

There seems to be some residual fades after filtering, and I think the reason is the calculation is done in global, currently.

[feisty2]

Originally Posted by WolframRhodium

I mean maybe the calculation of TopFieldSum/BottomFieldSum can be done in a small neighborhood, for example, 16x16 block?

There seems to be some residual fades after filtering, and I think the reason is the calculation is done in global, currently.

the statistical info(TopFieldSum/BottomFieldSum) should be no-local according to the weak law of large numbers.
if you got residual combing still even with threshold=0, it's then most likely your video suffers from something other than telecined fades and this filter won't help

[feisty2]

Originally Posted by jackoneill

No, no, fades like that can still be saved. By which I mean that you can obtain a 24 fps fade with smooth motion even if field matching is impossible in the fade. The solution involves QTGMC, so the detail preservation isn't awesome, but it's still good, considering the situation.

Steps:
1. Invoke source filter to obtain the untouched 30 fps clip.
2. Extract the fade using Trim.
3. Pass to QTGMC to obtain a 60 fps clip.
4. Stare at the output of QTGMC really carefully. It's been a few months since I did this, so I forget exactly what you're looking for.
5. Extract 4 frames out of a cycle of 10 using SelectEvery.
Tada!

my definition of "repair" is, to perfectly restore the frame without deinterlacing.

[WolframRhodium]

Originally Posted by feisty2

the statistical info(TopFieldSum/BottomFieldSum) should be no-local according to the weak law of large numbers.
if you got residual combing still even with threshold=0, it's then most likely your video suffers from something other than telecined fades and this filter won't help

Thank you for your reply. I misunderstood the function of this filter before and now I understand it better.

[feisty2]

r3:
AVX and FMA3 optimizations, will probably crash if your CPU is a predecessor of the Haswell microarchitecture (just use r2 in that case)