[Neo] f3kdb

[real.finder]

I think you should name the 2.1 as f3kdb2() and flash3kyuu_deband2() if you don't do it like https://forum.doom9.org/showthread.php?p=1882340#post1882340

edit: and f3kdb_dither2() if it still there

and then will be no need for f3kdb_i and f3kdb_legacy

[MeteorRain]

Technically, it's not "2". It's what it *should* be -- doing non hack operations.

[real.finder]

Technically, it's not "2". It's what it *should* be -- doing non hack operations.

Technically yes it's not "2" but that what SAP did (give it the "2" and remove many things from f3kdb and add VS support but even so he don't recommended use it in avs since it was pre/beta) so it's (the part of not "2") half true in this case

so going with "2" in last of functions names is the best option, for both users and scripts editor (don't think there are others than me and I will not update all scripts that has f3kdb with lsb hacks just to make them work with 2.1 for end users! and most of those end users cant even edit simple script to replace with f3kdb_legacy and have 2 of f3kdb dll)

aside form that, I will ask for last time, if you don't like/can't do take the changes from f3kdb_3 then at least add sample_mode=3 (same as sample_mode=1 but horizontal) if not then seems I will not add support for 2.1 in f3kdb_3 since there are no benefit from that

[MeteorRain]

@real.finder
What you said makes sense. I'll take that into consideration.

Regarding to f3kdb_3 and other pending patches, I'll start looking at those after this is completed. I'd like to finish this version before starting another.

[MeteorRain]

I talked to SAP and he suggested having a different name, and suggested the word neo.

So, neo_f3kdb it is. All pending changes will be released as neo_f3kdb r1.

Also would like to know how sample_mode = 3 and 4 works. Thanks!

[real.finder]

I talked to SAP and he suggested having a different name, and suggested the word neo.

So, neo_f3kdb it is. All pending changes will be released as neo_f3kdb r1.

Also would like to know how sample_mode = 3 and 4 works. Thanks!

sample_mode = 3 same as sample_mode = 1 but in horizontal (sample_mode = 1 is vertical)

and sample_mode = 4 is mix of outputs from sample_mode = 1 and 3 https://forum.doom9.org/showthread.php?p=1880098#post1880098

[MeteorRain]

So mode 1 is
O
x
O

mode 2 is
O O
x
O O

mode 3 will be
O x O

and mode 4 will be
O
O x O
O

Correct or not?

[real.finder]

So mode 1 is
O
x
O

mode 2 is
O O
x
O O

mode 3 will be
O x O

and mode 4 will be
O
O x O
O

Correct or not?

mode 1 and mode 3 are correct, don't know how 2 work (I think it's comparing the up, left, down and right pixel with the center pixel)

but the 4 is not, it do up and down comparing, and another comparing for left and right and mix (blend/Average/merge) the result of both

if it can't be added that ok, if there mode 3 in the plugin then mix both output will be faster than mode 1 alone (I did use turn function with mode 1 in f3kdb_3 to simulation mode 3, so it will be slower than mode 3 in the plugin)

[MeteorRain]

https://down.7086.in/AviSynthPlus%20Filters/debug/debug-sample-mode-4-pure-c.7z

test build pure C code for sample mode 3 / 4.

neo_f3kdb(sample_mode=3, opt=0, ...)

neo_f3kdb(sample_mode=4, opt=0, ...)

Let me know if it's what you want.

[Natty]

will it be ever possible to introduce new dithering methods in f3kdb?
not asking for new feature to be added, just asking if its possible in future. example - sierra-2-4a
:thanks:

[real.finder]

https://down.7086.in/AviSynthPlus%20Filters/debug/debug-sample-mode-4-pure-c.7z

test build pure C code for sample mode 3 / 4.

neo_f3kdb(sample_mode=3, opt=0, ...)

neo_f3kdb(sample_mode=4, opt=0, ...)

Let me know if it's what you want.

mode 3 seems ok, but I can't get mode 4 work it show error about in must be between 1 and 3 only

[MeteorRain]

Whoops, please re-download it.

[real.finder]

Whoops, please re-download it.

yes, both seems do it as they should :goodpost:

test code

ImageSource("uonzj6V.png (http://i.imgur.com/uonzj6V.png)")
PointResize(720,480)
ConvertToYV12()
mt_merge(last,FlipHorizontal(),BlankClip(last,color = $FFFFFF).FadeIO(500))
Crop(26, 2, -26, -2)
last.PointResize(Height(),Width()).StackHorizontal(last.fturnright)
#~neo_f3kdb(sample_mode=3, opt=0)
neo_f3kdb(sample_mode=4, opt=0)
luma_histogram() (https://forum.doom9.org/showthread.php?t=161411)

[MeteorRain]

@Natty,

Dithering code is in dither_high.h, and is fairly easy to add new algorithms if someone decides to add that support.

[MeteorRain]

https://down.7086.in/AviSynthPlus%20Filters/debug/debug-sample-mode-34-sse.7z

SSE code for 3 & 4. Please do extensive testing for both pure C and SSE if you can, because I didn't do.

[real.finder]

https://down.7086.in/AviSynthPlus%20Filters/debug/debug-sample-mode-34-sse.7z

SSE code for 3 & 4. Please do extensive testing for both pure C and SSE if you can, because I didn't do.

you mean sse2? with opt=1 both seems ok with this https://forum.doom9.org/showthread.php?p=1884034#post1884034

[MeteorRain]

SSE2 and 3 version will be removed soon to clean up legacy polyfill code.
SSE4.1 is old enough to support bulldozer (2011) and Penryn (Core 2 series, 2007).
And I plan to add AVX2 soon.

Please test with opt=0 (C) and opt=3 (SSE4.1) at the moment.

[Myrsloik]

SSE2 and 3 version will be removed soon to clean up legacy polyfill code.
SSE4.1 is old enough to support bulldozer (2011) and Penryn (Core 2 series, 2007).
And I plan to add AVX2 soon.

Please test with opt=0 (C) and opt=3 (SSE4.1) at the moment.

Did you test so the SSE4.1 code is actually faster than SSE2? A lot of the time developers didn't bother to actually benchmark it...

[MeteorRain]

Blendv only exists in SSE 4.1 and has to be emulated. No way that's faster than native.

[Myrsloik]

Blendv only exists in SSE 4.1 and has to be emulated. No way that's faster than native.

You'll be surprised how small the difference can be sometimes. I also never claimed it to be faster.

[MeteorRain]

Brain farted just wake up please forgive me. I meant definitely slower than native.

Blendv has to be emulated by and and andnot, some mullo also has to be emulated. It might be true that they won't drag down speed much, I also think that see2 and 3 support are no longer necessary.

[ChaosKing]

Thx for supporting both VS and AVS.
I made a quick bench for r1 on a ryzen 2600 1080p source
clip=clip.neo_f3kdb.Deband(preset="veryhigh")
msvc-x64: 145fps
icc-x64: 155fps
(Had to download libmmd.dll for intel build on some random dll site)

[MeteorRain]

Thanks for letting me know about libmmd. I'll try to statically link it and re-upload the release.

[Myrsloik]

Thanks for letting me know about libmmd. I'll try to statically link it and re-upload the release.

If you want to do speed comparisons try clang-cl as well.

[ChaosKing]

Thanks for letting me know about libmmd. I'll try to statically link it and re-upload the release.

neo-f3kdb_r1-fix.zip still needs libmmd.dll (only tested x64)

[DJATOM]

@MeteorRain
Just add libmmds.lib into linking libs and it should work. At least worked for me in the past.

[MeteorRain]

Thanks for the help. I tried to install clang-cl and then ICC toolset blew up. Now having to completely remove and reinstall the freaking ICC. Will let you know after finishing all those craps.


Please re-download fix https://down.7086.in/AviSynthPlus%20Filters/neo-f3kdb_r1-fix.zip

[ChaosKing]

Thanks for the help. I tried to install clang-cl and then ICC toolset blew up. Now having to completely remove and reinstall the freaking ICC. Will let you know after finishing all those craps.


Please re-download fix https://down.7086.in/AviSynthPlus%20Filters/neo-f3kdb_r1-fix.zip
The intel build works now without libmmd.dll
Clang seems to be as fast as the intel build or even some fps faster. :thanks:

[MeteorRain]

Am I reading this wrong? Clang is almost twice speed than VC and Intel builds?
On 1080P bluray with opt=1/2, ICC=45fps and Clang=73fps.
Making me to suspect I didn't properly optimize the other 2 builds.

Or because I'm on AMD?

And 32-bit clang is even faster than 64-bit clang sometimes.

[StvG]

neo_f3kdb(opt=1) - clang x86, clang x64 and icc x64 have almost the same speed (~1% difference). icc x86 have ~4% lower speed.
Intel here.

[real.finder]

sample_mode=4 is the Default, right?

there are other new things that can be added, like chroma bool parameter, and adaptive grain parameter to make grain more or only in dark Part/Scenes

and it's off-topic but what about this? https://forum.doom9.org/showthread.php?p=1883458#post1883458

[DJATOM]

Time elapsed: 0:32.145 - 155.54746162610678084093 FPS # classic f3kdb (x64)
Time elapsed: 0:25.578 - 195.47774818344581149177 FPS # neo f3kdb (x64) (clang build)
Time elapsed: 0:33.242 - 150.41332423316677591174 FPS # neo f3kdb (x64) (icc build)
Time elapsed: 0:31.664 - 157.90663253049507375181 FPS # neo f3kdb (x64) (msvc build)
Measured on 3900X with 2 threads and set affinity for vsedit to 2 cores (1 thread assigned per active core). Anyway I can't utilize all cores with f3kdb (using DGSource), so I believe such comparison shows real difference in optimization.

[StvG]

neo_f3kdb(opt=1) - clang x86, clang x64 and icc x64 have almost the same speed (~1% difference). icc x86 have ~4% lower speed.
Intel here.

Tested with avs+ (avsmeter).
~10% CPU usage.

[MeteorRain]

sample_mode=4 is the Default, right?

there are other new things that can be added, like chroma bool parameter, and adaptive grain parameter to make grain more or only in dark Part/Scenes

and it's off-topic but what about this? https://forum.doom9.org/showthread.php?p=1883458#post1883458
Not yet. This is test build.

Minideen you probably have to ask original porter. I just port the exact code back.

[Natty]

please include a full documentation text file in final build. :thanks:

[MeteorRain]

I need some help on AVX2 version. After mirroring code to AVX2 from SSE4, performance dropped a lot (30-40% slower). I'd really like to know how that could happen.

[Myrsloik]

I need some help on AVX2 version. After mirroring code to AVX2 from SSE4, performance dropped a lot (30-40% slower). I'd really like to know how that could happen.

Did you put the AVX2 code somewhere so we can take a look for ourselves? And is this drop with only one or all the compilers?

[MeteorRain]

Myrsloik, it's on the same repo, avx2 branch. The code somehow can only be compiled by clang-cl (due to the weird template layer that I added) and probably g++.

Running code at opt=3 and opt=4 makes speed difference and I'm wondering what I did wrong.

BTW: Also would like to know if there's a better way to do this SSE/AVX abstraction layer.

[Myrsloik]

Myrsloik, it's on the same repo, avx2 branch. The code somehow can only be compiled by clang-cl (due to the weird template layer that I added) and probably g++.

Running code at opt=3 and opt=4 makes speed difference and I'm wondering what I did wrong.

BTW: Also would like to know if there's a better way to do this SSE/AVX abstraction layer.

Start by looking at the generated code. Since you say you added additional abstraction it could be that functions aren't inlined anymore and you need to liberally sprinkle __forceinline everywhere. For example here (https://github.com/HomeOfAviSynthPlusEvolution/neo_f3kdb/blob/368611ca6dc977033db1ca91fff28b7e479aee70/src/simd/T_AVX2.hpp#L55).
Then there's also the statements of the form static constexpr auto count_zero = _mm_setzero_si128; where you may end up with a function pointer and instead of inlining you end up having a separate function call made every time you call that function.

Better way to do it? Simply write two versions and keep them both clean and readable is one option. Of course the best way would be for you to learn to read the disassembled code so you can spot this kind of thing yourself.

[MeteorRain]

Exact same abstraction layer pointing to SSE is working as fast as it was, making me believing the abstraction layer is doing good.

I did read the disassembled code, and all AVX2 intrinsics were properly inline compiled (I read the debug version so it's filled with useless ymm assignments, but CPU instructions are there, not function calls.)

[Myrsloik]

Exact same abstraction layer pointing to SSE is working as fast as it was, making me believing the abstraction layer is doing good.

I did read the disassembled code, and all AVX2 intrinsics were properly inline compiled (I read the debug version so it's filled with useless ymm assignments, but CPU instructions are there, not function calls.)

You have to compare the optimized versions.

[MeteorRain]

I stopped at some heavily optimized code and got lost.

If I manually inline read_reference_pixels() into _process_plane_simd_impl() (basically just copy pasting code to the caller), the speed of SSE4 will slow down to be worse than AVX2, while AVX2 timing is not affected.
If I leave the code in a __forceinline function, SSE4 version is way faster.
Some magic happened within that function and with tmp_1/2/3/4 arrays. I tried to add volatile on those arrays, that also "corrected" the performance.

I also tried VTune Amplifier, but didn't get meaningful answers as well.

EDIT:

This is both fun and boring. A few hours spent and I finally track down to the place that matters.

Clang completely rewrite the (scalar fetch + _mm_load) sequence into unrolled _mm_insert_epi16, while under avx2 it simply went through the pain loop and a full _mm256_load.

EDIT2:

I rewrote the hotspot using the same style as Clang did. (https://github.com/HomeOfAviSynthPlusEvolution/neo_f3kdb/commit/f59924c92620b9f528045ca93a1016c80898b764) Still not as fast as SSE due to missing intrinsics in AVX2. If you'd like to spend ten or maybe twenty hours on this, please give it a try. Otherwise I'll abandon the AVX2 branch very soon.

[MeteorRain]

I'm preparing for r2 release. Please let me know which should be the default for sample_mode.

[real.finder]

I'm preparing for r2 release. Please let me know which should be the default for sample_mode.

I don't mind if it 2 or 4, so let's see what the other members will says

[Reel.Deel]

I'm preparing for r2 release. Please let me know which should be the default for sample_mode.

Probably the original default.

[MeteorRain]

I keep the default. Released r2.

[Natty]

hi, whats the difference between these two

neo-f3kdb_r2.zip

neo-f3kdb_r2_extra.zip
also in r1, the output mode and depth were same as inputs. will it be the same now?

[MeteorRain]

Compilers. Defaults to clang builds. Extra package comes with msvc and icc builds.

I didn't change output mode and depth code in r2, so should be the same.

[Natty]

using clang x64, had to put neo_f3kdb(...) instead of f3kdb(...)

works absolutely fine :thanks:

[Natty]

ConvertBits(16).neo_f3kdb(output_depth=10)

this is making encode look 1 point darker.

example (https://slowpics.org/comparison/e56cafd8-5676-44c8-a352-52b4999b1fe6)

if i use SmoothTweak to increase brightness by 1 point, it becomes fine (mostly).

plain neo_f3kdb() is also fine.

using 3.2+6-f46aa2b (https://msystem.waw.pl/x265/) x265 10bit with color parameters set to BT.709

sample clip (https://www9.zippyshare.com/v/jj04ufhz/file.html)

edit 1 : i think issue might be in x265 as this never happened before