[Neo] FFT3D / DFTTest

[kedautinh12]

Yep, Intel site is pure shit from that point of view.

Anyway, I uploaded there:

Intel C++ redist 2020.1·216 (https://www.mediafire.com/file/2f576c2qokgen4j/Intel_C++_redist_2020.1%B7216.msi/file)

Intel C++ redist 2020.1·216_x64 (https://www.mediafire.com/file/efea2ncb0r27vew/Intel_C++_redist_2020.1%B7216_x64.msi/file)

Tell me if you need Fortran ones too. :p

Please give me, i need it

[tormento]

Please give me, i need it

Intel Visual Fortran redist 2020.1·216 (https://www.mediafire.com/file/rbhv1cscxpf0rgj/Intel_Visual_Fortran_redist_2020.1%B7216.msi/file)

Intel Visual Fortran redist 2020.1·216_x64 (https://www.mediafire.com/file/3dhr0xnl18xqmmg/Intel_Visual_Fortran_redist_2020.1%B7216_x64.msi/file)

Here they are. :)

Fortran programming is awesome. Which field of activities do you cover? Some AVS+ porting would show the real speed of a CPU ;)

[kedautinh12]

Intel Visual Fortran redist 2020.1·216 (https://www.mediafire.com/file/rbhv1cscxpf0rgj/Intel_Visual_Fortran_redist_2020.1%B7216.msi/file)

Intel Visual Fortran redist 2020.1·216_x64 (https://www.mediafire.com/file/3dhr0xnl18xqmmg/Intel_Visual_Fortran_redist_2020.1%B7216_x64.msi/file)

Here they are. :)

Fortran programming is awesome. Which field of activities do you cover? Some AVS+ porting would show the real speed of a CPU ;)

Thanks

[almosely]

Yep, Intel site is pure shit from that point of view.

Anyway, I uploaded there:

Intel C++ redist 2020.1·216 (https://www.mediafire.com/file/2f576c2qokgen4j/Intel_C++_redist_2020.1%B7216.msi/file)

Intel C++ redist 2020.1·216_x64 (https://www.mediafire.com/file/efea2ncb0r27vew/Intel_C++_redist_2020.1%B7216_x64.msi/file)

Tell me if you need Fortran ones too. :p

Thank you! :-) I will try and bench later today and post the results here. No, I've never used Fortran, but thanks anyway!

[almosely]

Meanwhile I installed the provided newest Intel C++ version 2020 and ran two further tests (https://forum.doom9.org/showthread.php?p=1911115#post1911115) about the different FFTW builds. GCC 9.3 is still the fastest and most efficient build for me.

@ StvG

Maybe this is the reason for your test results with your CPU?

Originally Posted by http://www.fftw.org/release-notes.html
FFTW 3.3.5
Jul 31, 2016

- New SIMD support:
-- Power8 VSX instructions in single and double precision. To use, add --enable-vsx to configure.
-- Support for AVX2 (256-bit FMA instructions). To use, add --enable-avx2 to configure.
-- Experimental support for AVX512 and KCVI. (--enable-avx512, --enable-kcvi) This code is expected to work but the FFTW maintainers do not have hardware to test it.
-- Support for AVX128/FMA (for some AMD machines) (--enable-avx128-fma)
-- Double precision Neon SIMD for aarch64. This code is expected to work but the FFTW maintainers do not have hardware to test it.
-- generic SIMD support using gcc vector intrinsics.
- Add fftw_make_planner_thread_safe() API.
- fix #18 (disable float128 for CUDACC).
- fix #19: missing Fortran interface for fftwq_alloc_real.
- fix #21 (don't use float128 on Portland compilers, which pretend to be gcc).
- fix: Avoid segfaults due to double free in MPI transpose.
- Special note for distribution maintainers: Although FFTW supports a zillion SIMD instruction sets, enabling them all at the same time is a bad idea, because it increases the planning time for minimal gain. We recommend that general-purpose x86 distributions only enable SSE2 and perhaps AVX. Users who care about the last ounce of performance should recompile FFTW themselves.

- edit -

I just uploaded Wolfberry's FFTW 3.3.8 ICC 19.0 build here (http://www.mediafire.com/file/hwu5acm7s60tlt5/FFTW_3.3.8_%25282019-01-29%2529_ICC_19.0.7z/file), if someone needs it. There are multiple files for different SIMD versions (SSE2, AVX, AVX2) for x64 and x86.

[MeteorRain]

Nope, it's likely my code that's broken. In the meantime just use avx2 until I have time to fix it.

[tormento]

Nope, it's likely my code that's broken. In the meantime just use avx2 until I have time to fix it.
No AVX2 here. :(

[tormento]

if someone needs it (will last only 14 days)
Register on MediaFire (https://www.mediafire.com/?weo3yc0) (invitation: +1GB for you and me) and your files will last almost forever. I have ones of years ago.

[StainlessS]

Register on MediaFire
With my free MediaFire acc, think I got something like 52GB storage, I only use a few of hundred MB.
I have ones of years ago.
Me too, 10+ years for some.

[MeteorRain]

No AVX2 here. :(

That's fine. I was talking about StvG's issue and advising downgrading from AVX512 to AVX2. If you don't have AVX2 you don't have to worry about downgrading from AVX512 to AVX2.

[tormento]

I just uploaded Wolfberry's FFTW 3.3.8 ICC 19.0 build here
Did you find any difference with newer version?

[almosely]

Register on MediaFire (https://www.mediafire.com/?weo3yc0) (invitation: +1GB for you and me) and your files will last almost forever. I have ones of years ago.

Done - for the sake of this forum ;-) I usually don't need cloud-space (except for images sharing in forums).

Did you find any difference with newer version?

I only did those speed tests. Huh, this seems to get a neverending story ;-) I just wanted to update AviSynth+ - and then came the rat-tail ;-) ... I think, I am not going to compare the two FFTW versions. If something is going south with my next encodes, I will notice.


Could it be possible that the GCC compile is bypassing the injected Spectre and Meltdown protection microcode somehow, while the ICC and MSVC versions don't?
I have disabled all of them... useless here.

What do you mean with "useless here"? My pc should be protected - it's a workstation and server for just everything and not just for me. The other thread is overloaded by another discussion (dll-linking).

[tormento]

What do you mean with "useless here"? My pc should be protected - it's a workstation and server for just everything and not just for me.
Those kind of vulnerabilities have specific requirements and spots to hit to be effective. With “here” I mean my home, where everything is behind a firewall and I am the only person who touches my workstation, the server and the two NAS. Everything else is not x86 architecture or don’t have access to outside world.

[almosely]

Okay, I understand. But in my case, I just want to let those protections activated. I simply do not know enough about it. I do not have a hardware firewall and a software fw (in my pc) is completely useless in that case; the "hit" would be at a place, where windows or any other OS is not aware of (mainboard; Intel Management Engine afaik). Even me - even if I would have all the necessary knowledge - is not to be trusted every time too. Since I switched from Windows 7 to 10 I noticed a performance plus everywhere. So I have no impulse to turn those precautions off to get more out of my system. But I have to say that Windows 10 is a bitch - it is an absolutely horrible thing to learn everything about it (often the hard way) and to configure it. Windows 7 was waaaa......aaaaay easier to manage.

So my question remains. Is it possible for any program (dll etc) to bypass this IME microcode protection? The difference in performance between GCC and all the other builds of FFTW is almost too big to be true.

- edit -

... "and spots to hit to be effective". Isn't it effective enough to get 100 percent control about the ethernet controller or any other hardware in use (camera, keyboard etc.) regarding only one person - and that unrecognized by the user? That reminds me somewhat of Edward Snowdens reports.

- edit -

The performance of Windows 10 is almost everywhere faster than with Windows 7; SATA is slower with Windows 10 (at least in CrystalDiskMark benchs; tested with two different versions, different bench-tools also and with different chipset-drivers and with different SATA drivers). But the reason for that could be the injected IME microcode. I did not have that patch/update with Windows 7 because it wasn't an automatic update and I was not aware of it (one has to install it manually).

- edit -

Hm, I think I am mixing something up here. While installing and configurating Windows 10, I got aware of the Spectre and Meltdown microcode AND the possible vulnerability of the Intel Management Engine. Till then I thought, I would be protected through regular windows updates alone and never paid attention to the IME at all too. But, after a fresh install of Windows 10, I got serious problems while shutting down or rebooting windows 10 and found out, that the IME was the problem. Some websites advised to downgrade the IME (even the german CHIP magazine), but that would be a) a very stupid thing and b) is not necessary at all. After finding a tool to actually check the functionality of the IME (Intel ME System Tools v8 r3), I realized, that my IME wasn't working properly (after a fresh windows 10 install!). So I uninstalled, updated, changed versions ... till somehow sometime I had the newest version installed (by windows update itself) and the Intel ME System Tools result was free of errors. Since then I got never a problem with being stuck while shutting down windows ever again - and hopefully am protected from that point, but I do not know. While checking that infos about the IME I realized that it had some serious vulnerabilities, especially older versions (thanks CHIP, grrrrr). And such vulnerabilities could be complete control over the hardware as I mentioned above. Spectre and Meltdown are other kinds of threats, but I do want to be protected against them too (at least, as far as it is possible).

- edit -

Man ... That microcode isn't stored within the IME, it's stored within the CPU, injected by the operating system, at every boot. Did I mention that things got very complicated nowadays, sometimes? I think I should resurrect my old Amiga 500, which is resting in the attic ...

[almosely]

I made some new benchmarks (https://forum.doom9.org/showthread.php?p=1911558#post1911558).

[tormento]

So it's fftw? And the old fft3d have those issues as well?
Has anybody tried to use the double or the extended precision versions of the library?

[kedautinh12]

I am already plan for this next update, but I was waiting for more plugins to did MCTD update first since it's not work with HBD at all (many plugins need updates)

Many plugins updated added vp8 interface
https://forum.doom9.org/showthread.php?p=1911708#post1911708

[real.finder]

Many plugins updated added vp8 interface
https://forum.doom9.org/showthread.php?p=1911708#post1911708

not the one I need https://forum.doom9.org/showpost.php?p=1909447&postcount=841

[almosely]

I just noticed that sigma2-4 is not working at all. Do I have to activate it somehow with another parameter?

[kedautinh12]

i see your neo-fft3d missing plane and ncpu for fft3dfilter, can you add it again??

[MeteorRain]

almosely: I figured out the issue. Please stand by for r8.

kedautinh12: Any reason why I should add them back?

[kedautinh12]

I will replace fft3dfilter with neo-fft3d in TemporalDegrain2 use for myself. But TemporalDegrain2 need them to work, can you add them back?

[MeteorRain]

I don't get it. Why TemporalDegrain2 needs them? The function of both parameters have been removed.

FFT3DFilter(plane=degrainPlane, sigma=limitSigma, sigma2=s2, sigma3=s3, sigma4=s4, bt=3, bw=limitBlksz, bh=limitBlksz, ow=overlap, oh=overlap, ncpu=fftThreads)

Change plane=degrainPlane to y=LumaNoise?3:2, u=ChromaNoise?3:2, v=ChromaNoise?3:2, and remove ncpu=fftThreads.

[kedautinh12]

I don't get it. Why TemporalDegrain2 needs them? The function of both parameters have been removed.

FFT3DFilter(plane=degrainPlane, sigma=limitSigma, sigma2=s2, sigma3=s3, sigma4=s4, bt=3, bw=limitBlksz, bh=limitBlksz, ow=overlap, oh=overlap, ncpu=fftThreads)

Change plane=degrainPlane to y=LumaNoise?3:2, u=ChromaNoise?3:2, v=ChromaNoise?3:2, and remove ncpu=fftThreads.

Thanks, sr for my noob requested

[kedautinh12]

Hi, i meet error when replace dfttest with neo_dfttest in TemporalDegrain2, i checked and findout neo_dfttest missing lsb. I was removed lsb but i meet error: "clip should be of same size!", can you add lsb again??

[DJATOM]

In general you don't need lsb stuff per filter, since avs+ provides filters to transform to/from stacked format. (conversion is lossless)
ConvertBits(16).neo_dfttest(...).ConvertToStacked() will be the same as dfttest(..., lsb=true)

[kedautinh12]

In general you don't need lsb stuff per filter, since avs+ provides filters to transform to/from stacked format. (conversion is lossless)
ConvertBits(16).neo_dfttest(...).ConvertToStacked() will be the same as dfttest(..., lsb=true)

it's work, many thanks

[almosely]

almosely: I figured out the issue. Please stand by for r8.

kedautinh12: Any reason why I should add them back?

Thank you :-) I will test and report ...

- edit -

Tested the functionality and it works fine :-) I did not compare the outcome with fft3dfilter.

[Atlantis]

I want to try neo_FFT3D

I don't understand
y, u, v (AviSynth+ only)

Whether a plane is to be filtered.

1 - Do not touch, leaving garbage data
2 - Copy from origin
3 - Process

1 and 2 are the same thing! What's the difference? They both don't touch the plane.

[DJATOM]

Not really the same. 1 option will save time (no copy from src to dst), 2 option will do that copy and 3 option will actually process plane.

[Atlantis]

What would be the use case for option 1? Because even if you want to use different settings for each plane, you use a combination of option 2 and 3.

[StainlessS]

What would be the use case for option 1?
When you dont need any kind of result for that plane and speed is primary factor. [eg perhaps only interested in Luma plane, dont need U or V].
Other filters have the same type of options eg Masktools2.

Unprocessed planes will just contain whatever happened to be in that memory buffer before the filter processed that frame.

[MeteorRain]

I want to try neo_FFT3D

I don't understand
y, u, v (AviSynth+ only)

Whether a plane is to be filtered.

1 - Do not touch, leaving garbage data
2 - Copy from origin
3 - Process

1 and 2 are the same thing! What's the difference? They both don't touch the plane.
Option 1 doesn't guarantee the data is meaningful.

For example, if you only process 1 plane, then it will just use the source memory buffer, and the data in the source memory area remains, and thus you will see the original data untouched.

However if you process 2 planes, the final memory is then copied from 2 process engines, while the 3rd plane has garbage data. It saves you a memory copy every frame.

What would be the use case for option 1? Because even if you want to use different settings for each plane, you use a combination of option 2 and 3.

Because some filters would split luma and chroma, and process then separately. In those cases, you'll end up throwing unused planes away and combining the final results. 1 will save you a tiny bit of time.

In reality, frame copy is not slow, and I have put optimizations to avoid unnecessary copies. It's not a big deal anyway.

[Atlantis]

OK. So here is my use case. I want to use neo_fft3d and apply different sigmas to Y and U V. Please correct me if I'm wrong, I don't think that applying the same amount of noise reduction to all 3 planes is good. So what I have decided for myself is to add half of noise reduction I do on Y to U V.
I write this for example if this is correct

neo_fft3d(bt=5, sigma=4, u=2, v=2)
neo_fft3d(bt=5, sigma=2, y=2)

4 to Y and 2 to U V.
Is this correct and is the philosophy of adding half of Y to U V a good one?

One strange thing I noticed is that in the old FFT3DFilter if I apply the same sigma to all 3 planes, these is a little desaturation. The color of the picture is not as alive as if you apply it only to Y.
It seems in your filter neo_fft3d this doesn't happen. By default yours is processing all 3 planes and I see no desaturation which is good. Any idea why this happened in the old FFT3DFilter ?

[Atlantis]

I am also testing neo_dfttest.

dfttest(U=false, V=false, sigma=96, tbsize=5)
encoded 8496 frames in 6158.00s (1.38 fps), 10137.46 kb/s, Avg QP:24.56

neo_dfttest(sigma=96, tbsize=5, u=2, v=2)
encoded 8496 frames in 5636.64s (1.51 fps), 7649.91 kb/s, Avg QP:24.54

Any idea why there is such a huge difference in the final output? 10137.46 kb/s vs 7649.91 kb/s?

[crystalfunky]

Will this be available in the next Staxrip version?

[Atlantis]

Ask stax76 here to include it
https://forum.doom9.org/showthread.php?p=1917053
I was also going to ask him. I'm going to do more tests.

In real world x265 encoding 4K HDR, neo_fft3d is definitely faster and I'm going to use neo_fft3d.
With neo_dfttest I don't see much improvement in speed.

[Atlantis]

FFT3D is missing scene change detection.
https://github.com/pinterf/fft3dfilter/issues/2

Sometimes you get bad frames with scene change. Like the image is not denoised. With bt=5 you get 4 bad frames and with bt=3 you get 2 bad frames. I noticed DFTTest does the same.

I have no idea how hard it is, can you add this feature? Any workaround? I'm going to open a topic only about this.

[MeteorRain]

I only port filters and optimize existing code. No new feature is implemented due to lack of skills and time.

[real.finder]

i see your neo-fft3d missing plane and ncpu for fft3dfilter, can you add it again??

ncpu should back as DFTTest has threads parameter, as now neo_FFT3D either just single thread or has some setting for multi threads

also, seems I get ignored again https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D/issues/3

maybe I should start put some notice that neo_** plugin are not Recommended and not officially supported as dogway did back then in SMDegrain with cretindesalpes/Firesledge moded mvtools http://web.archive.org/web/20131210170708/http://doom10.org/index.php?topic=2178.0

[MeteorRain]

I was adding back ncpu as I noticed a misunderstanding on fftw multithreading.

Regarding your ticket, yes somehow the repo was not in my watch list. I'll test when I finish my work.

Currently working on soxfilter.

[Atlantis]

I have a suggestion for Neo FFT3D y, u, v values. Things should be intuitive and simple and make sense.
I propose. for Y U V use
0 - Copy from origin
1 - Process
2 - Do not touch, leaving garbage data

for example this
neo_fft3d(y=3, u=2, v=2, sigma=6)
neo_fft3d(y=2, u=3, v=3, sigma=3)

That is hard to follow, this below makes much more sense and easier to read.

neo_fft3d(y=1, u=0, v=0, sigma=6)
neo_fft3d(y=0, u=1, v=1, sigma=3)

You already deleted planes because they were hard to read and YUV are easier names. So the values they get should follow the same rule and not be something like 1 for something that not many people use.

[crystalfunky]

Can somebody give me good values I can use for 4K HDR encoding,
x264 normal movies and also for b/w movies like 12 angry men?
Would appreciate it!

[Atlantis]

Every movie is different and you can't have good values for all. You have to try different things and see what you like.

I personally try to make it as simple as possible, so I just change the sigma value only and see what I like. I go from 1 to even 6.

[crystalfunky]

Are the commands and their values in the neo versions from dftest and fft3d different from the old versions?
If not, I need to understand them, so I can test different values and finally use the filters.

[Dogway]

I get artifacts with this filter (check link (https://www.mediafire.com/file/ej0uaxqnd2wk4kd/Attraction+3+HD4.mkv/file)) and had to roll back to plain dfttest. Probably an issue with HBD but I'm not sure. It gave me headaches for the last few days debugging my script.

This is my function call:
sneo_dfttest(slocation="0.0:5.0 0.2:4.0 1.0:1.0",tbsize=3,u=3,v=3)

[real.finder]

I get artifacts with this filter (check link (https://www.mediafire.com/file/ej0uaxqnd2wk4kd/Attraction+3+HD4.mkv/file)) and had to roll back to plain dfttest. Probably an issue with HBD but I'm not sure. It gave me headaches for the last few days debugging my script.

This is my function call:
sneo_dfttest(slocation="0.0:5.0 0.2:4.0 1.0:1.0",tbsize=3,u=3,v=3)

sneo_dfttest is warper I made to make it use neo_dfttest and if there are no neo_dfttest loaded it will use old dfttest

[Dogway]

sneo_dfttest is warper I made to make it use neo_dfttest and if there are no neo_dfttest loaded it will use old dfttest

Yes I know, I use neo_dfttest through your wrapper (didn't actually check what was doing though), but the filter has issues as depicted by the linked video with random color spots.

I spent 2 days trying different things to find the culprit and it actually was neo_dfttest, so for now I cannot recommend it to anyone.

[ajp_anton]

Your filters neo_dfttest and neo_fft3d don't pass through audio. Is that by design?

[StainlessS]

To not do anything for audio, do not define GetAudio() [in the filter class, think thats what its called] member function, will use default behaviour, ie pass-though.

Mobile: