New filter: Fix Telecined Fades

[feisty2]

binary(windows x64, requires msvcr 2017):https://github.com/IFeelBloated/Fix-Telecined-Fades/releases/tag/r5
git repo:https://github.com/IFeelBloated/Fix-Telecined-Fades/blob/master/Source.cpp

the filter gives a mathematically perfect solution to such(fades were done AFTER telecine which made a picture perfect IVTC pretty much impossible) (http://forum.doom9.org/showthread.php?p=1584763#post1584763) problem, and it's now time to kiss "vinverse" goodbye cuz "vinverse" is old and low quality.
unlike vinverse which works as a dumb blurring + contra-sharpening combo and very harmful to artifacts-free frames, this filter works by matching the brightness of top and bottom fields with statistical methods, and also harmless to healthy frames.


core.ftf.FixFades(clip, mode=0, threshold=0.002, color=[0.0, 0.0, 0.0], opt=True)

clip: clip to be processed

mode: could be 0(default), 1, or 2
0: adjust the brightness of both fields to match the average brightness of 2 fields.
1: darken the brighter field to match the brightness of the darker field
2: brighten the darker field to match the brightness of the brighter field

threshold: threshold for the average difference per pixel, on a scale of 0.0 - 1.0, but could go beyond 1.0, the frame will remain untouched if the average difference between 2 fields goes below this value

color: base color of the fade, default is [0.0, 0.0, 0.0](black)

opt: call the fastest possible functions if opt=True, else call the C++ functions.

INPUT CLIP MUST BE 32BITS FLOATING POINT FORMAT!!!
apply this filter AFTER field matching!!!

[feisty2]

comparison against vinverse
input

import vapoursynth as vs
core = vs.get_core()
clp = core.lsmas.LWLibavSource("rule6")
clp = core.vivtc.VFM(clp,0)
clp.set_output()

http://i.imgur.com/IvBQNka.png
vinverse

import vapoursynth as vs
core = vs.get_core()
clp = core.lsmas.LWLibavSource("rule6")
clp = core.vivtc.VFM(clp,0)

clp = core.vinverse.Vinverse(clp)

clp.set_output()

http://i.imgur.com/3nZsXj5.png
FTF

import vapoursynth as vs
core = vs.get_core()
clp = core.lsmas.LWLibavSource("rule6")
clp = core.vivtc.VFM(clp,0)
clp = core.fmtc.bitdepth(clp,bits=32,fulls=False,fulld=True)

clp = core.ftf.FixFades(clp)

clp.set_output()


http://i.imgur.com/lPcvPBp.png

[feisty2]

and vinverse failed miserably at the top-right part of the image, there's clear aliasing around edges, and ftf produced a picture perfect reconstruction.

[Mystery Keeper]

Does it do combing detection? Can you do better combing detection than TDM?

[feisty2]

Does it do combing detection? Can you do better combing detection than TDM?

No combing detection, it pretty much won't affect the normal frames so no masking required. Technically every pixel will be processed but only the problem frames will be affected.

[ShogoXT]

Does this work in general on IVTC scene transitions? Since using Vapoursynth (through Staxrip) ive been having such a problem with duplicate frames and scthresh with VFM and Vdecimate (no blending???), that ive been forced to stay with mode=0, but scene changes look awful. Ive also been having trouble with dealing with rainbows and dotcrawl in motion, but I guess thats another problem as all the solutions are temporal only.

EDIT: Also scrolling scenes stink...

[feisty2]

Does this work in general on IVTC scene transitions? Since using Vapoursynth (through Staxrip) ive been having such a problem with duplicate frames and scthresh with VFM and Vdecimate (no blending???), that ive been forced to stay with mode=0, but scene changes look awful. Ive also been having trouble with dealing with rainbows and dotcrawl in motion, but I guess thats another problem as all the solutions are temporal only.

EDIT: Also scrolling scenes stink...

not sure what you meant by "IVTC scene transitions", it works on any "simple" telecined fades(fade from/to pure color, eg. black), and it won't work on telecined cross-fades(2 or more scenes blending into each other), shit like that is simply beyond repair.

[WolframRhodium]

Great algorithm. The result is fantastic.

Question:
Will the results be even better if the algorithm acting on local rather than global?

[feisty2]

r2:
fixed a stupid memory leak
new parameter "threshold"
new parameter "color"

[feisty2]

Question:
Will the results be even better if the algorithm acting on local rather than global?

done, use the "threshold" parameter

[feisty2]

another demonstration for something other than fading into blackness
input

import vapoursynth as vs
core = vs.get_core()
clp = core.lsmas.LWLibavSource("rule6")
clp = core.vivtc.VFM(clp,0)
clp = core.fmtc.bitdepth(clp,bits=32,fulls=False,fulld=True)
clp = core.std.Expr(clp, ["1.0 x -", "x"]) #invert Y so it would be literally fading into whiteness
clp.set_output()

http://i.imgur.com/Shra82y.png
FTF

import vapoursynth as vs
core = vs.get_core()
clp = core.lsmas.LWLibavSource("rule6")
clp = core.vivtc.VFM(clp,0)
clp = core.fmtc.bitdepth(clp,bits=32,fulls=False,fulld=True)
clp = core.std.Expr(clp, ["1.0 x -", "x"]) #invert Y so it would be literally fading into whiteness

clp = core.ftf.FixFades(clp, color=[1.0, 0.0, 0.0])

clp.set_output()

http://i.imgur.com/xQMEHes.png

[jackoneill]

and it won't work on telecined cross-fades(2 or more scenes blending into each other), shit like that is simply beyond repair.

No, no, fades like that can still be saved. By which I mean that you can obtain a 24 fps fade with smooth motion even if field matching is impossible in the fade. The solution involves QTGMC, so the detail preservation isn't awesome, but it's still good, considering the situation.

Steps:
1. Invoke source filter to obtain the untouched 30 fps clip.
2. Extract the fade using Trim.
3. Pass to QTGMC to obtain a 60 fps clip.
4. Stare at the output of QTGMC really carefully. It's been a few months since I did this, so I forget exactly what you're looking for.
5. Extract 4 frames out of a cycle of 10 using SelectEvery.
Tada!

[ShogoXT]

No, no, fades like that can still be saved. By which I mean that you can obtain a 24 fps fade with smooth motion even if field matching is impossible in the fade. The solution involves QTGMC, so the detail preservation isn't awesome, but it's still good, considering the situation.

Steps:
1. Invoke source filter to obtain the untouched 30 fps clip.
2. Extract the fade using Trim.
3. Pass to QTGMC to obtain a 60 fps clip.
4. Stare at the output of QTGMC really carefully. It's been a few months since I did this, so I forget exactly what you're looking for.
5. Extract 4 frames out of a cycle of 10 using SelectEvery.
Tada!

I was under the impression that if the source is 2:3 telecined that it HAD to be ivtc and not QTGMC. QTGMC is very nice but I thought it was for interlace 121212 only. Vinverse has helped me a little, does the color conversation and fixfade cost much speed?

[fAy01]

No, no, fades like that can still be saved. By which I mean that you can obtain a 24 fps fade with smooth motion even if field matching is impossible in the fade. The solution involves QTGMC, so the detail preservation isn't awesome, but it's still good, considering the situation.

Steps:
1. Invoke source filter to obtain the untouched 30 fps clip.
2. Extract the fade using Trim.
3. Pass to QTGMC to obtain a 60 fps clip.
4. Stare at the output of QTGMC really carefully. It's been a few months since I did this, so I forget exactly what you're looking for.
5. Extract 4 frames out of a cycle of 10 using SelectEvery.
Tada!

Is there a possibility to write a script/plugin that deinterlaces and matches correctly at the same time without using QTGMC?

[Mystery Keeper]

I was under the impression that if the source is 2:3 telecined that it HAD to be ivtc and not QTGMC. QTGMC is very nice but I thought it was for interlace 121212 only. Vinverse has helped me a little, does the color conversation and fixfade cost much speed?After IVTC you can still find a lot of combed frames of various origins.

[WolframRhodium]

done, use the "threshold" parameter

I mean maybe the calculation of TopFieldSum/BottomFieldSum can be done in a small neighborhood, for example, 16x16 block?

There seems to be some residual fades after filtering, and I think the reason is the calculation is done in global, currently.

[feisty2]

I mean maybe the calculation of TopFieldSum/BottomFieldSum can be done in a small neighborhood, for example, 16x16 block?

There seems to be some residual fades after filtering, and I think the reason is the calculation is done in global, currently.

the statistical info(TopFieldSum/BottomFieldSum) should be no-local according to the weak law of large numbers (https://en.wikipedia.org/wiki/Law_of_large_numbers#Weak_law).
if you got residual combing still even with threshold=0, it's then most likely your video suffers from something other than telecined fades and this filter won't help

[feisty2]

No, no, fades like that can still be saved. By which I mean that you can obtain a 24 fps fade with smooth motion even if field matching is impossible in the fade. The solution involves QTGMC, so the detail preservation isn't awesome, but it's still good, considering the situation.

Steps:
1. Invoke source filter to obtain the untouched 30 fps clip.
2. Extract the fade using Trim.
3. Pass to QTGMC to obtain a 60 fps clip.
4. Stare at the output of QTGMC really carefully. It's been a few months since I did this, so I forget exactly what you're looking for.
5. Extract 4 frames out of a cycle of 10 using SelectEvery.
Tada!

my definition of "repair" is, to perfectly restore the frame without deinterlacing.

[WolframRhodium]

the statistical info(TopFieldSum/BottomFieldSum) should be no-local according to the weak law of large numbers (https://en.wikipedia.org/wiki/Law_of_large_numbers#Weak_law).
if you got residual combing still even with threshold=0, it's then most likely your video suffers from something other than telecined fades and this filter won't help

Thank you for your reply. I misunderstood the function of this filter before and now I understand it better. :thanks:

[feisty2]

r3:
AVX and FMA3 optimizations, will probably crash if your CPU is a predecessor of the Haswell microarchitecture (just use r2 in that case)

[feisty2]

guess I can actually optimize my floating point MVTools now with AVX and FMA, but I don't really want to...

[Mystery Keeper]

guess I can actually optimize my floating point MVTools now with AVX and FMA, but I don't really want to...Pretty please with a cherry on the top?

[Myrsloik]

r3:
AVX and FMA3 optimizations, will probably crash if your CPU is a predecessor of the Haswell microarchitecture (just use r2 in that case)

How much faster is it?

[feisty2]

How much faster is it?

r2 runs at 476.65fps at 1920x1080
r3 runs at 489.65fps at 1920x1080

makes no sense!!!
shouldn't AVX and FMA be at least 8x faster than x87???

I think there's probably something wrong with my compiler (VS2017), can you please compile the source code with your compiler and do a test as well?

to switch avx/fma back to c++

Line 225: FixFadesPrepare_AVX(); -> FixFadesPrepare();

Line 231: FixFadesMode0_AVX_FMA(); -> FixFadesMode0();

Line 234: FixFadesMode1_AVX_FMA(); -> FixFadesMode1();

Line 237: FixFadesMode2_AVX_FMA(); -> FixFadesMode2();

[feisty2]

Pretty please with a cherry on the top?

someday, I'll add that to my to-do list...

[cork_OS]

shouldn't AVX and FMA be at least 8x faster than x87???
1. It's limited by FPU IPC (real number and width of add/mult/etc. units).
2. Why x87? Are you using 80 bits of precision?

[Myrsloik]

r2 runs at 476.65fps at 1920x1080
r3 runs at 489.65fps at 1920x1080

makes no sense!!!
shouldn't AVX and FMA be at least 8x faster than x87???

I think there's probably something wrong with my compiler (VS2017), can you please compile the source code with your compiler and do a test as well?

to switch avx/fma back to c++

Line 225: FixFadesPrepare_AVX(); -> FixFadesPrepare();

Line 231: FixFadesMode0_AVX_FMA(); -> FixFadesMode0();

Line 234: FixFadesMode1_AVX_FMA(); -> FixFadesMode1();

Line 237: FixFadesMode2_AVX_FMA(); -> FixFadesMode2();

YOU ARE NOT USING X87!!! You compiled it as x64 code and the ABI (more or less) requires it to use sse2 instructions to implement it. Obviously at least the scalar float versions. It's even possible that it managed to auto vectorize like half of this code since most of it is just mindless read and sum. Look at the generated code instead of asking us about what you, YOURSELF, told the compiler to do.

Your assumption still wouldn't be true about x87 vs avx. For simple algorithms you run into memory bw limitations long before you see the glory of sse (avx is even more rare to matter). Modern cpus are just too good.

[feisty2]

YOU ARE NOT USING X87!!! You compiled it as x64 code and the ABI (more or less) requires it to use sse2 instructions to implement it. Obviously at least the scalar float versions. It's even possible that it managed to auto vectorize like half of this code since most of it is just mindless read and sum. Look at the generated code instead of asking us about what you, YOURSELF, told the compiler to do.
the "Look at the generated code" part is a bit too hard to me tho... Staring at thousands lines of generated assembly is far beyond my programming skill since I'm not a professionally trained programmer...

Your assumption still wouldn't be true about x87 vs avx. For simple algorithms you run into memory bw limitations long before you see the glory of sse (avx is even more rare to matter). Modern cpus are just too good.
which means it's pretty much pointless to manually optimize simple plugins?

[Myrsloik]

the "Look at the generated code" part is a bit too hard to me tho... Staring at thousands lines of generated assembly is far beyond my programming skill since I'm not a professionally trained programmer...

which means it's pretty much pointless to manually optimize simple plugins?

It's not thousands of lines. The interesting part is about 50 instructions at most. How to do shit properly: set a breakpoint in one of your inner loops, when it's hit simply open the (debug\window\disassembly) window. Look at like 10 instructions and see what it picked. Repeat for all critical loops. Done.

And yes, you're wasting perfectly good internet space by trying to optimize such simple things. Oh, and if you can't read simple disassembled stuff you shouldn't be writing optimizations like these in the first place.

[MonoS]

Ohi feisty, if you want some help with your optimization task fell free to ask to me directly (send me a PM and we can chose a more direct mean of communication)

As Myrsloik said probably your code was just being autovectorized directly by the compiler, on gcc for example if you compile with -O3 and -march=native it will try to vectorize using the best instruction set you have available. Probably vc do the same.

First things first, i'll base all my statement using the intel reference you can find here https://software.intel.com/sites/landingpage/IntrinsicsGuide/ , agner sheet would be better but is not as easy to use, also i'll use the official intel optimization guide http://www.intel.com/content/www/us/en/architecture-and-technology/64-ia-32-architectures-optimization-manual.html.

There are some things you could write better just looking at ProcessLine_AVX_FMA.
First you should avoid all those store, without an assembly listing it's a bit difficult to know, but probably they are using most of the load/store port making it more difficult for the processor to load data (the load port are shared with the store ones), also you are clogging the frontend with useless istruction to decode.
Division are NO when dealing with any kind of core, on haswell they are 21 cycles of latency (and you can issues another one after 13 cycles), you can turn it in a multiplication making the reciprocal of the value, multiplication are WAAAAAY cheaper than division (5 cycles latency, 0.5 througput) and even more you can predivide YMMField and YMMReference and make a single call to fmad.
I'd also suggest to remove the _mm256_set_ps at the start and replace with a _mm256_set1_ps.

If assembly listing are for you hard to reason about, i'll try AICA (always from intel) https://software.intel.com/en-us/articles/intel-architecture-code-analyzer , this software will give you a better understanding where your code is failing, it helped me a lot understand where i was doing things wrong with my optimization in mvtools.

There are other places where you could write better SIMD code, but for now i'll stop :) .

I strongly disagree with myrsloik as this being a waste of time, i find SIMD optimization to be very fun to program with and simple code teach the basics of SIMD programming, and yes, sometimes computer generated assembly is better than handmade one, but people improve over time.

For a tutorial from an "expert", i MUST suggest you to watch some of the first episode of handmade hero in which @cmuratori explayn how to SIMD optimize math heavy code, you can find the playlist here https://www.youtube.com/playlist?list=PLEMXAbCVnmY5qGQB96s7Vysr1nJcX_BW_ what episode from 112 to 121 (337 is a bonus becose i'm pretty behind the series).

With this i go to sleep after have recovered ALL THIS TEXT from ram cause my browser froze just before me finish this message, i hope to have been of some help.

[feisty2]

r4 runs at 626.89 fps at 1920x1080, 150.24 fps faster than the compiler generated code (no idea if optimized or not)
@MonoS
you were right that _mm256_div_ps was the bottleneck there, I removed it and got a noticeable performance boost, thx.

anyone could show me how to do that "opt" parameter thing? like call the avx function if it's supported by the CPU, and call the C++ function if not.. and please tell me I don't have to write asm for it

[TheFluff]

You need to write asm to identify the CPU, but it's trivial. Just steal from the VS source tree:

cpu.asm (https://github.com/vapoursynth/vapoursynth/blob/f65402b242b1eac05de1e485306ea1466d54541e/src/core/asm/x86/cpu.asm)
cpufeatures.c (https://github.com/vapoursynth/vapoursynth/blob/f65402b242b1eac05de1e485306ea1466d54541e/src/core/cpufeatures.c)
cpufeatures.h (https://github.com/vapoursynth/vapoursynth/blob/f65402b242b1eac05de1e485306ea1466d54541e/src/core/cpufeatures.h)

[Myrsloik]

You need to write asm to identify the CPU. From the VS source tree:

cpu.asm (https://github.com/vapoursynth/vapoursynth/blob/f65402b242b1eac05de1e485306ea1466d54541e/src/core/asm/x86/cpu.asm)
cpufeatures.c (https://github.com/vapoursynth/vapoursynth/blob/f65402b242b1eac05de1e485306ea1466d54541e/src/core/cpufeatures.c)
cpufeatures.h (https://github.com/vapoursynth/vapoursynth/blob/f65402b242b1eac05de1e485306ea1466d54541e/src/core/cpufeatures.h)

Just copy those files into your own project. Done.

[feisty2]

r5:
new parameter "opt"
opt: call the fastest possible functions if opt=True, else call the C++ functions.

Just copy those files into your own project. Done.

You need to write asm to identify the CPU, but it's trivial. Just steal from the VS source tree:

cpu.asm (https://github.com/vapoursynth/vapoursynth/blob/f65402b242b1eac05de1e485306ea1466d54541e/src/core/asm/x86/cpu.asm)
cpufeatures.c (https://github.com/vapoursynth/vapoursynth/blob/f65402b242b1eac05de1e485306ea1466d54541e/src/core/cpufeatures.c)
cpufeatures.h (https://github.com/vapoursynth/vapoursynth/blob/f65402b242b1eac05de1e485306ea1466d54541e/src/core/cpufeatures.h)

Tried to use them but got stuck at "cpu.asm", apparently it requires nasm or yasm and these 2 are real pain in the ass, I wasted hours trying to make them work on VS2017 and all I got was millions of errors popping out relentlessly...

I even thought about translating "cpu.asm" to masm, and obviously I would have to translate "x86inc.asm" along with it, and that's a big NO.

so I merged those 3 into one file, "cpufeatures.hpp (https://github.com/IFeelBloated/Fix-Telecined-Fades/blob/master/cpufeatures.hpp)", and wrote my own version of CPUFeatures, with absolutely no trace of (literal) asm.

these 2 functions:

vs_cpu_cpuid()
vs_cpu_xgetbv()

have been integrated into the compiler,

Visual Studio 2017:
vs_cpu_cpuid() -> __cpuid()
GCC:
vs_cpu_cpuid() -> __get_cpuid()

vs_cpu_xgetbv() -> _xgetbv() //defined in immintrin.h


so I think there's no need to write literal asm for them.
and I canceled the "getCPUFeatures()" function, it could be simply integrated into the constructor since I'm using a C++ header.

so instead of

CPUFeatures CPU;
getCPUFeatures(&CPU);
if (CPU.fma3)
xxx

it's now cleaner like

auto CPU = CPUFeatures();
if (CPU.fma3)
xxx

[Are_]

Right now it looks like it's not compiling anymore with GCC.

[feisty2]

Right now it looks like it's not compiling anymore with GCC.

to make it work with GCC

cpufeatures.hpp:
line 1: #include <intrin.h> -> #include <cpuid.h>
line 6: return static_cast<int32_t>(val); -> return static_cast<uint32_t>(val);
line 28: __cpuid(Registers, 1); -> __get_cpuid(1, &eax, &ebx, &ecx, &edx);
line 43:__cpuid(Registers, 7); -> __get_cpuid(7, &eax, &ebx, &ecx, &edx);

[feisty2]

Right now it looks like it's not compiling anymore with GCC.

or just use this version of cpufeatures.hpp (https://github.com/IFeelBloated/Fix-Telecined-Fades/blob/master/cpufeatures_gnu.hpp)

[MonoS]

Glad to be of help :)

[sl1pkn07]

or just use this version of cpufeatures.hpp (https://github.com/IFeelBloated/Fix-Telecined-Fades/blob/master/cpufeatures_gnu.hpp)

nope

https://sl1pkn07.wtf/paste/view/b0cc57fb

[feisty2]

nope

https://sl1pkn07.wtf/paste/view/b0cc57fb

working now?

I'm sure I fixed everything except the "xgetbv" part, that function has been integrated into immintrin.h in Visual Studio 2017 but apparently not in GCC...
You'll have to compile cpu.asm yourself with yasm or nasm and I can't help you with that... I wouldn't have written my own Visual Studio version of CPUFeatures if I could handle them..

[sl1pkn07]

nope
https://sl1pkn07.wtf/paste/view/b2ac319f

let me time to test with yasm, idk how manage it

[feisty2]

that's weird...
please modify the source code manually and report back..
line 139:
replace

_mm256_store_ps(reinterpret_cast<float *>(&YMMField), _mm256_add_ps(reinterpret_cast<const __m256 &>(srcp[y][x]), YMMField));

with

_mm256_store_ps(reinterpret_cast<float *>(&YMMField), _mm256_add_ps(*reinterpret_cast<const __m256 *>(&srcp[y][x]), YMMField));

[sl1pkn07]

https://sl1pkn07.wtf/paste/view/883d7e05

[feisty2]

https://sl1pkn07.wtf/paste/view/883d7e05

I googled that error and apparently the source code was not what caused it.

there's something wrong with your compiling settings
That's the usual message for intrinsics that you haven't told the compiler the target supports.
see: http://stackoverflow.com/questions/35772562/inlining-failed-in-call-to-always-inline-m128i-mm-cvtepu8-epi32-m128i-t

damn, now I get how nice that Visual Studio compiler is (from the user's point of view), I never had any sort of shit like this with Visual Studio...

[sl1pkn07]

maybe is because my processor don't have AVX/FMA3 (Xeon x5650)

[feisty2]

maybe is because my processor dont have AVX/AVX2? (Xeon x5650)

you could try with "-march=haswell" and see if it does the trick

[sl1pkn07]

my processor is westmere/ep (+/- nehalem)

with -msse4.1

https://sl1pkn07.wtf/paste/view/9516661b

EDIT: with -march=haswell build ok. but i think is incompatible with my processor

[feisty2]

my processor is westmere/ep (+/- nehalem)

with -msse4.1

https://sl1pkn07.wtf/paste/view/9516661b

EDIT: with -march=haswell build ok. but i think is incompatible with my processor

doesn't matter, it will call the C++ functions automatically if the AVX functions fail.

[sl1pkn07]

is not better use "if detect, compile it" instead of force compiling?

[feisty2]

is not better use "if detect, compile it" instead of force compiling?

unfortunately, I don't know how to "if detect, compile it"...
is the forcibly-compiled binary running ok on your computer since there's the C++ function backup?