[Neo] FFT3D / DFTTest [Archive]

View Full Version : [Neo] FFT3D / DFTTest

MeteorRain

15th April 2020, 06:46

This thread serves Neo-FFT3D and Neo-FFTTest.

Neo filters family is not to be confused with AviSynth-Neo. Neo filters family is not affiliated in any way with AviSynth-Neo.

[Neo] FFT3D → GitHub (https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D)

FFT3DFilter is a 3D Frequency Domain filter - strong denoiser and moderate sharpener. It was originally written by Alexander G. Balakhnin aka Fizick, and later modified by martin53 for AviSynth 2.6 and later modified by Ferenc Pintér aka pinterf for further improvement, high bit depth, and more. Kudos to them for creating and improving this fantastic tool.

Difference compared to pinterf/FFT3DFilter

Dual interface, supporting AviSynth+ and VapourSynth
Removed YUY2
Removed multiplane options, added y, u, v options for plane control
Removed existing 3DNow and SSE code
Fixed an incorrect coefficient in Apply2D
Added SSE, AVX and AVX512 routine
Added multi-threading, both internal and AVS-MT prefetch

Result bit-identicalness

Neo-FFT3D C - FFT3DFilter C: identical except for a few situations
Except: bt=2 and sharpen and dehalo where the incorrect coefficient is fixed
Except: bt=4, 5 where the floating point numbers are added in different way ( (fp2r + fpr) + (fcr + fnr) in Neo-FFT3D, (fp2r + fpr + fcr + fnr) in FFT3DFilter ) resulting rounding error
Except: bt=0 and pfactor > 0 causing crash, need your investigation and fixes
===== ===== ===== =====
Neo-FFT3D SSE - C: identical
Neo-FFT3D AVX - C: identical
Neo-FFT3D AVX512 - C: theoretically identical

[Neo] DFTTest → GitHub (https://github.com/HomeOfAviSynthPlusEvolution/neo_DFTTest)

DFTTest is a 2D/3D Frequency Domain denoiser. It was originally written by tritical, and later modified by Ferenc Pintér aka pinterf for further improvement, high bit depth, and more. VapourSynth-DFTTest was ported to VapourSynth interface by HolyWu with further clean ups and efficient SSE2 and AVX2 SIMD routines. Kudos to them for creating and improving this fantastic tool.

Difference compared to pinterf/dfttest

Dual interface, supporting AviSynth+ and VapourSynth
Parameter names and opt levels follow VapourSynth-DFTTest
Removed Y, U, V options, added y, u, v options for plane control
Removed hacked 16 bit I/O
Replaced internal multi-threading with C++ Parallel STL, removed Windows specific APIs
Improved precision of floating point reciprocal using Newton Raphson method
sigma/sigma2/sbsize/sosize/tbsize etc. has different default value

Difference compared to HomeOfVapourSynthEvolution/VapourSynth-DFTTest

Dual interface, supporting AviSynth+ and VapourSynth
Added dither option from AVS counterpart
Added internal multi-threading
Replaced division by Newton Raphson method reciprocal in SSE2
Replaced reciprocal approximation by Newton Raphson method in AVX2

Result bit-identicalness

Neo-DFTTest C - (AVS)DFTTest C: identical except when nlocation is provided, due to order of floating point multiplication
Neo-DFTTest: data = data * (scale * (wscale2 / wscale) * alpha)
(AVS)DFTTest: data = data * scale * (wscale2 / wscale) * alpha
Neo-DFTTest C - (VS)DFTTest C: identical
===== ===== ===== =====
Neo-DFTTest SSE - C: identical except when dither > 2, due to rounding errors in dithering
Neo-DFTTest AVX - C: identical except when dither > 2, due to rounding errors in dithering

feisty2

15th April 2020, 08:10

I took a peek at your dual interface, I think the argument fetching part, particularly the part dealing with array parameters could be improved.
I don't think it's a good idea to assume array parameters are std::vector<T>, and to erase the vector while fetching the arguments.
people might not use vector to store array parameters, it could also be std::array or a pointer to some pre-allocated memory or something else for various reasons.
it's also not a good practice to erase the parameter container, the container might hold more default values than the number of values specified by the user. you lose all default values when you erase the container and the user loses the flexibility to specify a partial array, leaving the rest of default values untouched.

I think it's best if the argument object itself behaves like a container (it does not necessarily to be an actual container) and let the filter developer decide how the arguments should be handled.

// this is bad
args.read("sigma_array", sigma_array);

// this is better, and allows the user to specify a partial array
for (auto x : range{ args["sigma_array"].size() })
sigma_array[x] = args["sigma_array"][x];

// or if you don't need the flexibility to specify a partial array
for (auto sigma : args["sigma_array"])
sigma_array.push_back(sigma);

the dual interface also doesn't yet seem to handle frame properties for vaporsynth, I have completed this functionality and a Rec601ToRGB example is also available. if you plan to add support for frame properties, the syntax should be the same as, or at least very similar to the aforementioned "arguments" thing for the sake of consistency.

MeteorRain

15th April 2020, 08:54

Thanks for taking your time to look at the code. You are absolute right about parameter processing. Part of those code was copied from my previous projects years ago, and there's lots of room for improvement.

At this moment I'm trying to make scaffolding as quick as possible (this wrapper was made after your wrapper, so you know it's still very young). And my hope is if anyone has interests they can send merge requests to improve it. Of course, at this moment I'd aim at C++17 and not 20 or 23. Let's wait until C++20 or 23 becomes mainstream before moving forward.

Your point about array makes a lot sense. I'll need some time to think about it. For now vectors are adequate to my needs, but I'll try to make it better. As you suggested, begin() and end() would be a good start.

MeteorRain

15th April 2020, 12:00

Please test dfttest beta1 (https://down.7086.in/AviSynthPlus%20Filters/neo-dfttest-beta1~20200415.zip).

Example: neo_dfttest(ftype=0, sigma=8.0, sigma2=12.0, pmin=0.0, pmax=500.0, sbsize=4, smode=1, sosize=2, tbsize=1, tmode=0, tosize=0, opt=0, zmean=True, y=3, u=3, v=3, nlocation="0,0,20,40", slocation="0.0:3.0, 0.3 14.0 1.0,10.0", ssystem=1, threads=6)

Syntax is similar to original AVS dfttest except:

Y/U/V=true/false is now y/u/v=3/2/1 where 3=process, 2=copy, 1=garbage.

nstring is now alpha and nlocation.
For example, (nstring="0,0,20,40 a=5.2") is now (nlocation="0,0,20,40", alpha=5.2).

sstring is now ssystem and sstring.
For example, (sstring="$ 0.0:3.0 1.0:10.0") is now (sstring="0.0,3.0,1.0,10.0", ssystem=1).

sstring, ssx, ssy, sst supports any delimiter of "," or " " or ":".
For example, (ssx="0.0:3.0 1.0:10.0") equals (ssx="0.0,3.0,1.0,10.0").

opt is now 0=auto, 1=C, 2=SSE2, 3=AVX2.

No hacked 16bit IO. Only native HBD is supported.

threads=X only applies to SSE2 and AVX2 routine, C code always runs at single thread.

sigma and sigma2 defaults to 8.0. Manually set to 16 to match original result.

C/SSE2/AVX2 now produces bit identical results except for random dithered 8bit, which gets some rounding errors.

On VapourSynth, nlocation, slocation, ssx, ssy, sst are still arrays.

Let me know any issues.

StainlessS

15th April 2020, 12:06

which gets some rounding errors.
Maybe in future use term "rounding differences", allay [lessen] user worries about errors.

MeteorRain

15th April 2020, 12:10

dither > 2 uses Mersenne Twister random number generator to dither data, so outputs would never be the same every time. Whether there are rounding "errors" or "differences" no longer matters, I think.

StainlessS

15th April 2020, 12:16

Its just that some use a translator to read docs, and sometimes translators can change meaning a little and may result in much exaggerated significance.

ChaosKing

15th April 2020, 12:19

Please test dfttest beta1 (https://down.7086.in/AviSynthPlus%20Filters/neo-dfttest-beta1~20200415.zip).

Tested a bit in VS:
- 8-16 + 32 bit depth works fine
- clip.neo_dfttest.DFTTest(ftype=0, sigma=8.0, sigma2=12.0, pmin=0.0, pmax=500.0, sbsize=4, smode=1, sosize=2, tbsize=1, tmode=0, tosize=0, opt=0, zmean=True, nlocation="0,0,20,40", slocation="0.0:3.0, 0.3 14.0 1.0,10.0", ssystem=1, threads=6)
produces
Core freed but 5 filter instance(s) still exist
Core freed but 5 filter instance(s) still exist
Core freed but 537600 bytes still allocated in framebuffers
Core freed but 537600 bytes still allocated in framebuffers
EDIT:
The parameters above triggersValueError: invalid literal for int() with base 10: '0,0,20,40'

- No warnings with a simple neo_dfttest.DFTTest(sigma=8, sigma2=8)
- speed:
-- vapoursynth dfttest 7.9 fps
-- vapoursynth neo dfttest 5.5 fps

MeteorRain

15th April 2020, 12:34

Yea I haven't tested thoroughly on VS.

On VS nlocation and slocation and ss? are still arrays. I should have said those changes only apply to AVS. On VS array inputs are the same as vapoursynth-dfttest.

I'll look on memory leaks later.

Speed difference is probably because I've set it as serialized filter (so single threaded from VS point of view) for debugging purpose. Will change to parallel on release.

ChaosKing

15th April 2020, 12:43

So this would be correct?
neo_dfttest.DFTTest(nlocation=[0,0,20,40])
It crashes immediately :D

MeteorRain

15th April 2020, 22:23

beta2 (https://down.7086.in/AviSynthPlus%20Filters/neo-dfttest-beta2~20200415.zip)
Fixes fmParallel
Fixes nlocation crashes

When nlocation does not crash, there seems no memory leaks found.

ChaosKing

16th April 2020, 00:23

beta2 (https://down.7086.in/AviSynthPlus%20Filters/neo-dfttest-beta2~20200415.zip)
Fixes fmParallel
Fixes nlocation crashes

When nlocation does not crash, there seems no memory leaks found.
Works good now. The only thing I found was a msgCore freed but 1 filter instance(s) still exist
Core freed but 1 filter instance(s) still exist
if you specify an incomplete parameter like this DFTTest(nlocation=[8]). Idk if this counts as an error.

MeteorRain

16th April 2020, 00:31

Supplying incorrect parameters may trigger an error with some temporary data not being freed correctly. I'll check where it leaks, but I don't think it's a big issue.

real.finder

16th April 2020, 00:34

I did some tests

ColorBars(width=640, height=480, pixel_type="yv12")
admfilter(custom_filter="neo_fft3d(Sigma=(adSigma+1.0)/f)")
Prefetch(4)

with https://forum.doom9.org/showthread.php?p=1904672#post1904672

I get
https://i.postimg.cc/23Wgtx5H/Untitled.png (https://postimg.cc/23Wgtx5H)

and with https://forum.doom9.org/showthread.php?p=1906016#post1906016

it's just freeze! and sometimes

https://i.postimg.cc/146wN8Wn/Untitled.png (https://postimg.cc/146wN8Wn)

and sometimes same as the 1st one

MeteorRain

16th April 2020, 01:06

I can't get your script work. I assume I should use "Advanced Denoising and anime bob v1.87"?

I get -- I don't know what 'MotionRampadc' means.

real.finder

16th April 2020, 01:23

I can't get your script work. I assume I should use "Advanced Denoising and anime bob v1.87"?

I get -- I don't know what 'MotionRampadc' means.

you only need https://github.com/realfinder/AVS-Stuff/raw/master/avs%202.5%20and%20up/AdvancedDenoising.avsi and https://github.com/realfinder/AVS-Stuff/raw/master/avs%202.5%20and%20up/Zs_RF_Shared.avsi

and from that error you get seems you didn't use any runtime MT fixed avs+ (both test avs+ builds I post)

edit: also you maybe missed this GRunT update (http://www.mediafire.com/file/77uromd0j0z56ap/GRunT-1.0.2tst.7z/file) pinterf made years ago

MeteorRain

16th April 2020, 01:38

real.finder

16th April 2020, 01:47

Okey grunt it is. Now I can see it crashed. Let me check.

That's easy to find. Filter initialization was executed concurrently, crashed the fftw library. I don't know if initialization should be thread safe or not. I'll add a mutex then.

DFTTest was also crashed, but pinterf fix it, didn't try neo DFTTest yet, but last DFTTest by pinterf work

edit: it fixed since this one https://forum.doom9.org/showthread.php?p=1904777#post1904777

MeteorRain

16th April 2020, 02:07

Yes my solution is similar to that. Usually initialization is single threaded, so you don't have to think about thread safety. In this case multiple instances were initialized so there needs to be a global mutex there. pinterf used static global, and I used a static local, the result would be similar.

beta3 (https://down.7086.in/AviSynthPlus%20Filters/neo-dfttest-beta3~20200415.zip) thread-safe initialization.

real.finder

16th April 2020, 03:07

beta3 (https://down.7086.in/AviSynthPlus%20Filters/neo-dfttest-beta3~20200415.zip) thread-safe initialization.

it's seems work

btw, how you end-up with this case? https://forum.doom9.org/showthread.php?p=1905043#post1905043

and aside from that, do you plan to did the same with all/most/some https://github.com/HomeOfVapourSynthEvolution ? (Dual interface with neo renames)

MeteorRain

16th April 2020, 03:51

real.finder

16th April 2020, 04:16

I believe the change of bit depth should not change a parameter, at least in most cases.
Both vapoursynth-dfttest and dfttest scale parameters to the bit depth so it wasn't a problem.

so, parameter for those now float right? (if they not already), since float more flexible, For example if the parameter 0-255 with float we can set it like 112.5 and so, but with int we have only 256 options and that not suitable for more than 8bit

I haven't decided what to do next, but I may start with a few. mvtools-sf is also a good candidate. Depending on my time availability.

since mvtools already has HBD (still there are no float motion analyzing) and it's complex maybe it's good to start with basic things that has no HBD yet in avs+ like VagueDenoiser, EEDI3 and Dither_resize16/fmtconv and then back to mvtools in the end

MeteorRain

16th April 2020, 05:36

Which parameters are you talking about? DFTTest has all parameters in float I believe (except for those modes, sizes and frame numbers).

And I've uploaded the release (r7).

real.finder

16th April 2020, 05:41

Which parameters are you talking about? DFTTest has all parameters in float I believe (except for those modes, sizes and frame numbers).

if they are, then no need to do anything :) I am just saying if any (Whether here or not)

tormento

16th April 2020, 10:42

And I've uploaded the release (r7).
I saw your readme.md is pointing to official FFTW download page.

Unfortunately they built only 3.3.5 version of dll, while there are many updated ones such as 3.3.7 or 3.3.8. The latest I can find is here (https://forum.doom9.org/showthread.php?p=1907464#post1907464) and a fresh compile from you or pinterf with VS and clang (and perhaps Intel) to test would be nice.

real.finder

16th April 2020, 16:38

is [Neo] FFT3D / DFTTest stable yet? cuz I want to make scripts use them instead of non-neo one

also as I said in #22 more ports from VS will be nice since avs+ still need those updates for HBD, so I will make another wave of scripts updates :)

MeteorRain

16th April 2020, 19:58

real.finder

16th April 2020, 20:28

Whether it is stable depends on your bug report. If you're happy about it then I think it's stable enough.

What are the things that you are looking to replace in your script from VS counterparts? Those 3 you mentioned?

yes and along with SangNomMod those the most important for now, but there are others with kinda less important like TTempSmooth, EEDI2, TCanny (there are TCannymod by chikuzen but I think with no HBD), Yadifmod (there are Yadifmod2 by chikuzen with not completed HBD), IT, DeblockPP7

and after that maybe these VS only like Bwdif, W3FDIF, VMAF, BM3D

StvG

18th April 2020, 11:17

ffvideosource()
neo_ff3d() #r3 crash with no message, r1v4 is ok (x64)

Myrsloik

18th April 2020, 11:40

I believe the change of bit depth should not change a parameter, at least in most cases.
Both vapoursynth-dfttest and dfttest scale parameters to the bit depth so it wasn't a problem.

I haven't decided what to do next, but I may start with a few. mvtools-sf is also a good candidate. Depending on my time availability.

This is one of the cases where you can argue that it should change. We're dealing with transforms so there's a direct connection between input range and transformed range. Just my thoughts. I suppose scaling is easier but then a 0-255 range makes no sense. Either have a proper 0-1 float or make the int range a percentage or whatever.

Don't propagate 0-255 scaled ranges, they make me very unhappy since they make no sense AND lack fine enough steps to for higher bitdepths.

MeteorRain

19th April 2020, 10:05

ffvideosource()
neo_ff3d() #r3 crash with no message, r1v4 is ok (x64)

Can't reproduce here.

Source file resolution? (Tested 1280x720)
Source bit depth? (Tested 8-bit)
FFMS version? (Mine stamped at 2019-8-11)
Anything that I might have missed?

StvG

19th April 2020, 18:00

Can't reproduce here.

Source file resolution? (Tested 1280x720)
Source bit depth? (Tested 8-bit)
FFMS version? (Mine stamped at 2019-8-11)
Anything that I might have missed?
Tested with 1280x720 / 1920x1080 8-bit video, blankclip(pixel_type="yv12").
Added ConvertBits(16) before neo_fft3d() doesn't change the situation.
Tested with this FFMS2 (https://forum.doom9.org/showthread.php?p=1906345#post1906345), latest LSMASH (https://github.com/HolyWu/L-SMASH-Works).
AVSMeter - https://images2.imgbox.com/e6/ba/oz2BaBV8_o.png
AVX-512 CPU.

MeteorRain

19th April 2020, 21:43

Oh avx 512. Yea I don't have any testing environment. Can you try debugging it for me?

(Or I can try to use a CPU emulator or rent a server to debug it. Or I can simply remove it since it's not such a big deal to support it now.)

MeteorRain

19th April 2020, 23:01

Should have fixed it. The issue is more stupid than I thought.

Please try v4.

Also opt parameter is introduced so you can disable AVX512 manually.

StvG

20th April 2020, 11:14

Thanks. It works now.
The output of opt=1 and opt=2 is identical.
The output of opt=3 and opt=4 is identical.
The output of opt=1/2 is a bit different than opt=3/4.

MeteorRain

20th April 2020, 12:39

Do you have a combination of resolution-depth-parameters that I can test?

By the way I did find some issues when running 2 instances together. I'll check that part later.

StvG

20th April 2020, 14:44

ColorBars(pixel_type="yv12") #whatever pixel_type
a=neo_fft3d(opt=3)
neo_fft3d(opt=2)
Compare(a,"yuv")
Live action video would have bigger difference.

MeteorRain

20th April 2020, 23:43

Please try r5 as I fixed the race condition when multiple instances are running together.

StvG

21st April 2020, 12:51

The difference is still there.

kedautinh12

22nd April 2020, 06:35

The difference is still there.

can you comment more detail?

pinterf

22nd April 2020, 07:12

Usually I'm using this comparison at the end of my scripts. It may show if difference has geometric pattern.

...
StackVertical(old, new, Diff(old, new))
return last

Function Diff(clip src1, clip src2)
{
return Subtract(src1.ConvertBits(8),src2.ConvertBits(8)).Levels(120, 1, 255-120, 0, 255, coring=false)
}

MeteorRain

22nd April 2020, 12:54

Thanks for reporting, there's an issue with degrid on. The issue was introduced in r3. I should change for all 3 SIMD files but ended up only changing 1 place, leaving 2 files with incorrect parameters.

I'm uploading r6.

kedautinh12

29th April 2020, 00:43

real.finder

29th April 2020, 02:42

I think it's stable now, can you add your scripts??

I am already plan for this next update, but I was waiting for more plugins to did MCTD update first since it's not work with HBD at all (many plugins need updates)

almosely

2nd May 2020, 18:49

I just found, that there's this NEO-version of FFT3DFilter - very nice! :-) Although a test is waiting ... Question: Is FFTW still needed for this NEO-version? I have FFTW 3.3.8 (AVX) (Wolfberry's compile from 29-01-2019) already installed and in use since then. I just realized that AvsPmod GPo has gotten very slow, since I have updated to the newest AviSynth+ Version 3.5.1 and also updated my Visual C++ Redists 2015-2019 to the latest. Now I am checking everything, I use in combination with AviSynth+ and update all of it - maybe, the slowdown is gone after that.

Ahm, I can vaguely remember, that there was a bug in pinterf's FFT3DFilter (maybe it has been there since the beginning, as there was this luma-bug too): I compared the outcome of RgTools's RemoveGrain vs. FFT3DFilter (with chroma filtering), both on and off, with Histogram("color"/"color2") and there were differences when FFT3DFilter was in the filter chain, although turned off. With RemoveGrain, there was no difference to the unfiltered source. But I noticed that a year ago and thought, that I just use RemoveGrain for chroma filtering and not to bother pinterf a second time ;-) Maybe someone else noticed that too and maybe that misbehaviour is resolved in the NEO-version?

MeteorRain

3rd May 2020, 01:57

almosely:

I'm having difficulties reading your words. Let's split it into parts and you can clarify if I misunderstood.

> Is FFTW still needed for this NEO-version?

Yes. In fact, FFTW is required for all plugins that needs regular FFT transform. FFTW would be the fastest and easiest way to do it.

Of course if it's a GPU plugin than it'll need something else.

> when FFT3DFilter was in the filter chain, although turned off

What do you mean turned off? Does that mean you apply fft3dfilter on a clip but did not use the result? In which part did you notice differences?

almosely

3rd May 2020, 03:01

Ok, then I leave FFTW where it is ;-) Thank you.

I made a new test regarding chroma filtering, although I realized that I had a thinking error (RemoveGrain cannot be turned off, of course). But I can put FFT3DFilter or neo_fft3d into the script and turn the sigma off (setting it to 0.1 = filter is off). When I do that with luma filtering only, then there is absolutely no difference between the original material and the one with either one of the two fft3d filters (in Histogram=luma), as it should be. But when I do that with UV-filtering turned on, then I can see a difference, although there should not be any difference. I made 3 screenshots.

This happens not, when I leave the chroma settings at 2 (y=3,u=2,v=2 neo) or 0 (plane, fft3dfilter). But it happens, when I only filter the chroma (y=2,u=3,v=3 neo / plane=3 fft3dfilter) or all of it (all=3 neo / plane=4 fft3dfilter).

And another question: What is the difference between 1 and 2 (y,u,v)?

And I also installed the newest version of AvsPmod GPo. The first thing, that came up, was an error-message, that Shibatch.dll is installed twice (but it's only within the plugins+ and plugins64+ folders once, each). That one is "btw". But the next one is regarding your neo_fft3d filter. AvsPmod GPo also spit out an error message, that "neo-fft3d" is containing irregular letters. I muted that message for the next times, sadly. But - IIRC - there was a third message, telling me, that the "-" is the villain and it should be a "_" or anything else to be correct. So I renamed your .dll to "neo_fft3d.dll".

My first benchmarks between neo_fft3d and FFT3DFilter resulted in a speed gain of about 9% (only luma filtering) and about 17% with all planes filtered (compression tests of 720p material with other filters turned on, as I usually do). Nice! :-)

And another thing I am wondering about. With opt=0 I got exactly the same results as with opt=2. But with opt=3 the x264 results were a bit different all over the place (bitrate, I-,P-,B-frame sizes and counts etc.). I have an Intel Core i7-3770 (with AVX). Shouldn't the results be all the same?

opt=0

[2020-05-03][02:20:13] y4m [info]: 1280x720p 1:1 @ 30/1 fps (cfr)
[2020-05-03][02:20:13] x264 [info]: using SAR=1/1
[2020-05-03][02:20:13] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-03][02:20:13] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-03][02:22:13] x264 [info]: frame I:23 Avg QP:15.27 size: 60969
[2020-05-03][02:22:13] x264 [info]: frame P:372 Avg QP:19.54 size: 21633
[2020-05-03][02:22:13] x264 [info]: frame B:1125 Avg QP:23.53 size: 7449
[2020-05-03][02:22:13] x264 [info]: consecutive B-frames: 2.2% 2.5% 11.1% 48.9% 21.1% 14.2%
[2020-05-03][02:22:13] x264 [info]: mb I I16..4: 14.4% 69.4% 16.2%
[2020-05-03][02:22:13] x264 [info]: mb P I16..4: 4.9% 20.2% 2.0% P16..4: 33.5% 12.5% 7.4% 0.2% 0.0% skip:19.2%
[2020-05-03][02:22:13] x264 [info]: mb B I16..4: 0.6% 3.1% 0.2% B16..8: 29.6% 8.9% 2.3% direct: 3.8% skip:51.5% L0:45.4% L1:46.8% BI: 7.9%
[2020-05-03][02:22:13] x264 [info]: 8x8 transform intra:74.7% inter:72.8%
[2020-05-03][02:22:13] x264 [info]: direct mvs spatial:99.6% temporal:0.4%
[2020-05-03][02:22:13] x264 [info]: coded y,uvDC,uvAC intra: 66.7% 75.0% 44.9% inter: 11.0% 12.6% 1.9%
[2020-05-03][02:22:13] x264 [info]: i16 v,h,dc,p: 18% 12% 4% 67%
[2020-05-03][02:22:13] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 15% 7% 7% 8% 11% 9% 11% 11%
[2020-05-03][02:22:13] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 13% 3% 8% 14% 14% 13% 8% 8%
[2020-05-03][02:22:13] x264 [info]: i8c dc,h,v,p: 37% 21% 22% 20%
[2020-05-03][02:22:13] x264 [info]: Weighted P-Frames: Y:7.3% UV:5.1%
[2020-05-03][02:22:13] x264 [info]: ref P L0: 59.7% 6.0% 18.4% 7.7% 7.9% 0.2%
[2020-05-03][02:22:13] x264 [info]: ref B L0: 87.6% 8.4% 2.8% 1.2%
[2020-05-03][02:22:13] x264 [info]: ref B L1: 96.6% 3.4%
[2020-05-03][02:22:13] x264 [info]: kb/s:2815.29
[2020-05-03][02:22:13] encoded 1520 frames, 12.13 fps, 2815.42 kb/s

opt=2

[2020-05-03][02:22:20] y4m [info]: 1280x720p 1:1 @ 30/1 fps (cfr)
[2020-05-03][02:22:20] x264 [info]: using SAR=1/1
[2020-05-03][02:22:20] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-03][02:22:20] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-03][02:24:21] x264 [info]: frame I:23 Avg QP:15.27 size: 60969
[2020-05-03][02:24:21] x264 [info]: frame P:372 Avg QP:19.54 size: 21633
[2020-05-03][02:24:21] x264 [info]: frame B:1125 Avg QP:23.53 size: 7449
[2020-05-03][02:24:21] x264 [info]: consecutive B-frames: 2.2% 2.5% 11.1% 48.9% 21.1% 14.2%
[2020-05-03][02:24:21] x264 [info]: mb I I16..4: 14.4% 69.4% 16.2%
[2020-05-03][02:24:21] x264 [info]: mb P I16..4: 4.9% 20.2% 2.0% P16..4: 33.5% 12.5% 7.4% 0.2% 0.0% skip:19.2%
[2020-05-03][02:24:21] x264 [info]: mb B I16..4: 0.6% 3.1% 0.2% B16..8: 29.6% 8.9% 2.3% direct: 3.8% skip:51.5% L0:45.4% L1:46.8% BI: 7.9%
[2020-05-03][02:24:21] x264 [info]: 8x8 transform intra:74.7% inter:72.8%
[2020-05-03][02:24:21] x264 [info]: direct mvs spatial:99.6% temporal:0.4%
[2020-05-03][02:24:21] x264 [info]: coded y,uvDC,uvAC intra: 66.7% 75.0% 44.9% inter: 11.0% 12.6% 1.9%
[2020-05-03][02:24:21] x264 [info]: i16 v,h,dc,p: 18% 12% 4% 67%
[2020-05-03][02:24:21] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 15% 7% 7% 8% 11% 9% 11% 11%
[2020-05-03][02:24:21] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 13% 3% 8% 14% 14% 13% 8% 8%
[2020-05-03][02:24:21] x264 [info]: i8c dc,h,v,p: 37% 21% 22% 20%
[2020-05-03][02:24:21] x264 [info]: Weighted P-Frames: Y:7.3% UV:5.1%
[2020-05-03][02:24:21] x264 [info]: ref P L0: 59.7% 6.0% 18.4% 7.7% 7.9% 0.2%
[2020-05-03][02:24:21] x264 [info]: ref B L0: 87.6% 8.4% 2.8% 1.2%
[2020-05-03][02:24:21] x264 [info]: ref B L1: 96.6% 3.4%
[2020-05-03][02:24:21] x264 [info]: kb/s:2815.29
[2020-05-03][02:24:21] encoded 1520 frames, 12.03 fps, 2815.42 kb/s

opt=3

[2020-05-03][02:24:27] y4m [info]: 1280x720p 1:1 @ 30/1 fps (cfr)
[2020-05-03][02:24:27] x264 [info]: using SAR=1/1
[2020-05-03][02:24:27] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-03][02:24:27] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-03][02:26:28] x264 [info]: frame I:23 Avg QP:15.26 size: 60963
[2020-05-03][02:26:28] x264 [info]: frame P:372 Avg QP:19.56 size: 21631
[2020-05-03][02:26:28] x264 [info]: frame B:1125 Avg QP:23.52 size: 7453
[2020-05-03][02:26:28] x264 [info]: consecutive B-frames: 2.2% 2.5% 11.1% 48.9% 21.1% 14.2%
[2020-05-03][02:26:28] x264 [info]: mb I I16..4: 14.5% 69.3% 16.2%
[2020-05-03][02:26:28] x264 [info]: mb P I16..4: 5.0% 20.2% 2.0% P16..4: 33.6% 12.5% 7.4% 0.2% 0.0% skip:19.2%
[2020-05-03][02:26:28] x264 [info]: mb B I16..4: 0.6% 3.2% 0.2% B16..8: 29.6% 8.8% 2.3% direct: 3.8% skip:51.4% L0:45.5% L1:46.7% BI: 7.9%
[2020-05-03][02:26:28] x264 [info]: 8x8 transform intra:74.7% inter:72.7%
[2020-05-03][02:26:28] x264 [info]: direct mvs spatial:99.6% temporal:0.4%
[2020-05-03][02:26:28] x264 [info]: coded y,uvDC,uvAC intra: 66.7% 75.0% 44.8% inter: 11.0% 12.6% 1.9%
[2020-05-03][02:26:28] x264 [info]: i16 v,h,dc,p: 18% 12% 4% 67%
[2020-05-03][02:26:28] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 15% 7% 7% 8% 11% 9% 11% 11%
[2020-05-03][02:26:28] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 18% 13% 3% 8% 14% 14% 13% 8% 8%
[2020-05-03][02:26:28] x264 [info]: i8c dc,h,v,p: 37% 21% 22% 20%
[2020-05-03][02:26:28] x264 [info]: Weighted P-Frames: Y:7.3% UV:5.1%
[2020-05-03][02:26:28] x264 [info]: ref P L0: 59.6% 6.0% 18.5% 7.7% 8.0% 0.2%
[2020-05-03][02:26:28] x264 [info]: ref B L0: 87.6% 8.4% 2.8% 1.2%
[2020-05-03][02:26:28] x264 [info]: ref B L1: 96.7% 3.3%
[2020-05-03][02:26:28] x264 [info]: kb/s:2815.81
[2020-05-03][02:26:28] encoded 1520 frames, 12.10 fps, 2815.93 kb/s

MeteorRain

3rd May 2020, 03:27

> turn the sigma off (setting it to 0.1 = filter is off)

Bug confirmed. I'll investigate what happened.

> What is the difference between 1 and 2 (y,u,v)?

2 = copy, 1 = garbage data.

> But with opt=3 the x264 results were a bit different

Make sure you have r6 which should have fixed the issue.

almosely

3rd May 2020, 03:34

Nice again! :-) It would be fine to use only one filter for luma and croma, I think (even so RemoveGrain is fast as hell).

I still don't get it. What means "garbage data" or "Do not touch, leaving garbage data". But, all I have to really know is: When I only want to filter luma, I use y=3, u=2, v=2 or shall I use y=3, u=1, v=1? I guess 3-2-2 is the way to go?

Hm, I do have r6 installed :-(

real.finder

3rd May 2020, 03:42

almosely

3rd May 2020, 04:05

A test with RemoveGrain -1,u,v translated the effects to my brain :-) -1 removes the channel completely from the further script progress. So 2 is the way to go for me, when I only want to filter luma (3-2-2).

But it seems since I have updated to the newest AviSynth+ and MSVC++ 15-19, something is not working right anymore. AvsPMod GPu is slow (the old was and the new version is too), even when I switch the values of RemoveGrain, when nothing else is within the script. And sometimes RemoveGrain mode -1 removed the luma channel, and sometimes not - like a bitchy programm. Weird. And I never noticed a red cross in the upper right corner of AvsPMod GPu before the update to the new AviSynth+ and MSVC++ - now I see it almost every time I change a filter value. I know, thats the wrong thread, but I can't help myself to mention that. Sadly I deleted the old avisinth.dlls and devil.dlls from version 3.4.0, which I had installed before.

- edit -

just downloaded the portable version of AviSynth+ 3.4.0 and will test with that.

MeteorRain

3rd May 2020, 04:07

almosely:

Do not attach images like this. You can use a public image hosting service such as imgur. Upload your picture to imgur and paste the image link here.

If you have r6 and still have issues with opt=3 and opt=2 please give me a minimal sample script that you can reproduce the issue.

garbage data means it can return anything and you should not use the data from that plane. For (y=3, u=2, v=1), y plane is filtered, u plane is copied from source, v plane contains garbage and you should not consider v plane as a valid image.
Whether use 2 or 1 depends on your actual script.

almosely

3rd May 2020, 04:43

Ok, now I got the meaning of "-1" - that explains the "bitchy" ;-)

I will use imgur.com the next times.

I made another test, with almost the same script (only the BlackmanResize has different values) and another clip. Now I got 3 different results ...

opt=0

[2020-05-03][05:26:36] y4m [info]: 1280x536p 1:1 @ 24000/1001 fps (cfr)
[2020-05-03][05:26:36] x264 [info]: using SAR=1/1
[2020-05-03][05:26:36] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-03][05:26:36] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-03][05:28:30] x264 [info]: frame I:65 Avg QP:15.61 size:101225
[2020-05-03][05:28:30] x264 [info]: frame P:548 Avg QP:20.40 size: 26703
[2020-05-03][05:28:30] x264 [info]: frame B:1387 Avg QP:23.77 size: 8405
[2020-05-03][05:28:30] x264 [info]: consecutive B-frames: 5.5% 5.6% 14.1% 59.8% 7.2% 7.8%
[2020-05-03][05:28:30] x264 [info]: mb I I16..4: 4.0% 70.9% 25.1%
[2020-05-03][05:28:30] x264 [info]: mb P I16..4: 1.7% 17.6% 3.4% P16..4: 36.0% 16.7% 14.0% 1.0% 0.3% skip: 9.3%
[2020-05-03][05:28:30] x264 [info]: mb B I16..4: 0.3% 3.6% 0.7% B16..8: 36.3% 9.1% 2.5% direct: 3.9% skip:43.6% L0:38.6% L1:43.1% BI:18.3%
[2020-05-03][05:28:30] x264 [info]: 8x8 transform intra:76.4% inter:62.4%
[2020-05-03][05:28:30] x264 [info]: direct mvs spatial:99.6% temporal:0.4%
[2020-05-03][05:28:30] x264 [info]: coded y,uvDC,uvAC intra: 85.4% 86.9% 59.8% inter: 16.9% 17.8% 3.0%
[2020-05-03][05:28:30] x264 [info]: i16 v,h,dc,p: 22% 7% 6% 64%
[2020-05-03][05:28:30] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 9% 7% 7% 9% 12% 9% 12% 11%
[2020-05-03][05:28:30] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 24% 5% 3% 8% 13% 16% 9% 12% 8%
[2020-05-03][05:28:30] x264 [info]: i8c dc,h,v,p: 36% 18% 27% 18%
[2020-05-03][05:28:30] x264 [info]: Weighted P-Frames: Y:9.1% UV:5.1%
[2020-05-03][05:28:30] x264 [info]: ref P L0: 62.4% 14.0% 15.2% 4.9% 3.2% 0.3% 0.0%
[2020-05-03][05:28:30] x264 [info]: ref B L0: 88.5% 8.3% 2.6% 0.6%
[2020-05-03][05:28:30] x264 [info]: ref B L1: 96.5% 3.5%
[2020-05-03][05:28:30] x264 [info]: kb/s:3152.49
[2020-05-03][05:28:30] encoded 2000 frames, 17.02 fps, 3152.57 kb/s

opt=2

[2020-05-03][05:28:35] y4m [info]: 1280x536p 1:1 @ 24000/1001 fps (cfr)
[2020-05-03][05:28:35] x264 [info]: using SAR=1/1
[2020-05-03][05:28:35] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-03][05:28:35] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-03][05:30:29] x264 [info]: frame I:65 Avg QP:15.61 size:101210
[2020-05-03][05:30:29] x264 [info]: frame P:548 Avg QP:20.40 size: 26700
[2020-05-03][05:30:29] x264 [info]: frame B:1387 Avg QP:23.81 size: 8408
[2020-05-03][05:30:29] x264 [info]: consecutive B-frames: 5.5% 5.6% 14.2% 59.4% 7.5% 7.8%
[2020-05-03][05:30:29] x264 [info]: mb I I16..4: 4.0% 70.9% 25.1%
[2020-05-03][05:30:29] x264 [info]: mb P I16..4: 1.7% 17.6% 3.4% P16..4: 36.0% 16.7% 14.0% 1.0% 0.3% skip: 9.3%
[2020-05-03][05:30:29] x264 [info]: mb B I16..4: 0.2% 3.6% 0.7% B16..8: 36.4% 9.2% 2.5% direct: 3.9% skip:43.6% L0:38.6% L1:43.1% BI:18.3%
[2020-05-03][05:30:29] x264 [info]: 8x8 transform intra:76.4% inter:62.3%
[2020-05-03][05:30:29] x264 [info]: direct mvs spatial:99.6% temporal:0.4%
[2020-05-03][05:30:29] x264 [info]: coded y,uvDC,uvAC intra: 85.4% 86.9% 59.8% inter: 16.9% 17.8% 3.0%
[2020-05-03][05:30:29] x264 [info]: i16 v,h,dc,p: 22% 7% 7% 64%
[2020-05-03][05:30:29] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 9% 7% 6% 9% 12% 9% 12% 11%
[2020-05-03][05:30:29] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 24% 5% 4% 8% 13% 16% 9% 12% 8%
[2020-05-03][05:30:29] x264 [info]: i8c dc,h,v,p: 37% 18% 27% 18%
[2020-05-03][05:30:29] x264 [info]: Weighted P-Frames: Y:9.1% UV:5.1%
[2020-05-03][05:30:29] x264 [info]: ref P L0: 62.4% 14.0% 15.3% 4.9% 3.2% 0.3% 0.0%
[2020-05-03][05:30:29] x264 [info]: ref B L0: 88.5% 8.2% 2.6% 0.6%
[2020-05-03][05:30:29] x264 [info]: ref B L1: 96.5% 3.5%
[2020-05-03][05:30:29] x264 [info]: kb/s:3152.61
[2020-05-03][05:30:29] encoded 2000 frames, 17.00 fps, 3152.69 kb/s

opt=3

[2020-05-03][05:30:35] y4m [info]: 1280x536p 1:1 @ 24000/1001 fps (cfr)
[2020-05-03][05:30:35] x264 [info]: using SAR=1/1
[2020-05-03][05:30:35] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-03][05:30:35] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-03][05:32:28] x264 [info]: frame I:65 Avg QP:15.62 size:101210
[2020-05-03][05:32:28] x264 [info]: frame P:548 Avg QP:20.39 size: 26706
[2020-05-03][05:32:28] x264 [info]: frame B:1387 Avg QP:23.78 size: 8408
[2020-05-03][05:32:28] x264 [info]: consecutive B-frames: 5.5% 5.6% 14.1% 59.8% 7.2% 7.8%
[2020-05-03][05:32:28] x264 [info]: mb I I16..4: 4.1% 71.0% 25.0%
[2020-05-03][05:32:28] x264 [info]: mb P I16..4: 1.7% 17.6% 3.4% P16..4: 36.0% 16.7% 14.0% 1.0% 0.3% skip: 9.3%
[2020-05-03][05:32:28] x264 [info]: mb B I16..4: 0.2% 3.6% 0.7% B16..8: 36.3% 9.1% 2.5% direct: 4.0% skip:43.6% L0:38.6% L1:43.2% BI:18.3%
[2020-05-03][05:32:28] x264 [info]: 8x8 transform intra:76.4% inter:62.3%
[2020-05-03][05:32:28] x264 [info]: direct mvs spatial:99.6% temporal:0.4%
[2020-05-03][05:32:28] x264 [info]: coded y,uvDC,uvAC intra: 85.4% 86.8% 59.7% inter: 16.9% 17.9% 3.0%
[2020-05-03][05:32:28] x264 [info]: i16 v,h,dc,p: 22% 7% 7% 64%
[2020-05-03][05:32:28] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 9% 7% 7% 9% 12% 9% 12% 11%
[2020-05-03][05:32:28] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 24% 5% 3% 8% 13% 16% 9% 12% 8%
[2020-05-03][05:32:28] x264 [info]: i8c dc,h,v,p: 36% 18% 27% 18%
[2020-05-03][05:32:28] x264 [info]: Weighted P-Frames: Y:9.1% UV:5.1%
[2020-05-03][05:32:28] x264 [info]: ref P L0: 62.4% 14.0% 15.2% 4.9% 3.2% 0.3% 0.0%
[2020-05-03][05:32:28] x264 [info]: ref B L0: 88.5% 8.3% 2.6% 0.6%
[2020-05-03][05:32:28] x264 [info]: ref B L1: 96.6% 3.4%
[2020-05-03][05:32:28] x264 [info]: kb/s:3152.91
[2020-05-03][05:32:28] encoded 2000 frames, 17.06 fps, 3152.98 kb/s

The script

DGSource("test v2.dgi", crop_l=0, crop_r=0, crop_t=0, crop_b=0, deinterlace=0, use_top_field=true, use_pf=true)

CompTest(5, 40)

RequestLinear(rlim=50, clim=50)

BlackmanResize(1280, 536, taps=9)

neo_fft3d(sigma=1.0, beta=1.0, bw=48, bh=48, sharpen=0.040, scutoff=0.01, y=3, u=2, v=2, bt=5, opt=0)

GradFun3(thr=0.35)

avstp_set_threads(1)

Prefetch(3)

return last

The CompTest-Script (.avsi)

function CompTest(clip c, float percent, float frames)
{
frange = floor((frames/1.)/(percent/100.))
return SelectRangeEvery(c, frange, int(frames))
}

- edit -

Ha! Could it be the mtmodes.avsi? I did not make any new entry for neo_fft3d ... Should be like ...

SetFilterMTMode("neo_fft3d", MT_MULTI_INSTANCE)

.. this could explain it, couldn't it?

MeteorRain

3rd May 2020, 05:09

Don't do set filter mt mode.

Can you please give me a sample script like the one below?

some_source
some_preprocessing

v1 = neo_fft3d(some parameters, opt=1)
v2 = neo_fft3d(some parameters, opt=2)
compare(v1, v2)

Make sure you read non-identical result. Try to give me a minimal script so I can narrow down issue easily.

EDIT:

To narrow down further, please tell me whether you can reproduce it under non-MT mode. Is the problem specific to MT mode only?

almosely

3rd May 2020, 05:45

Is this different (enough)?

https://i.imgur.com/5q09cfQ.png

DGSource("test v3.dgi", crop_l=0, crop_r=0, crop_t=0, crop_b=0, deinterlace=0, use_top_field=true, use_pf=true)

RequestLinear(rlim=50, clim=50)

BlackmanResize(640, 360, taps=16)

v1 = neo_fft3d(sigma=1.0, beta=1.0, bw=48, bh=48, sharpen=0.040, scutoff=0.01, y=3, u=2, v=2, bt=5, opt=1)
v2 = neo_fft3d(sigma=1.0, beta=1.0, bw=48, bh=48, sharpen=0.040, scutoff=0.01, y=3, u=2, v=2, bt=5, opt=2)
compare(v1, v2)

avstp_set_threads(1)

Prefetch(3)

return last

- edit -

There's no difference between ST and MT mode with this (I mean: There are still these differences like in the picture above).

# avstp_set_threads(1)

# Prefetch(3)

MeteorRain

3rd May 2020, 06:10

Thanks.

3 issues were fixed.

1) When input is Y8, the engine was incorrectly skipped.

2) Chroma processing had incorrect negative value float to int rounding.
For positive values, the rounding was correct.
For example, x = 2.1, int(x + 0.5) = 2.
However for a negative value, x = -2.1, int(x + 0.5) -> int(-1.6) -> -1.
Since chroma is processed as signed integer, negative values were incorrectly rounded.

3) bt=5 had a rounding error between C and SIMD code.
Basically in SIMD I was doing result=(a+b)+(c+d)+e while in C code it's result=a+b+c+d+e, resulting a minor rounding error.
Now the C code is the same as SIMD code, and is different to fft3dfilter.

Please try r7.

almosely

3rd May 2020, 06:39

Yeehaa! Thank you very much :-) I can't see any difference with the last script anymore.

... but there's still a minor difference regarding chroma:

https://i.imgur.com/OwlRvJd.png
https://i.imgur.com/kilLcML.png

MeteorRain

3rd May 2020, 06:47

Try sigma=0.001.

almosely

3rd May 2020, 06:59

What a magical mystery! ;-))) ... tried, worked ... set the value-range-change back to 0.1 after that test. Thank you again!

StvG

3rd May 2020, 13:04

v1 = neo_fft3d(sigma=1.0, beta=1.0, bw=48, bh=48, sharpen=0.040, scutoff=0.01, y=3, u=2, v=2, bt=5, opt=4)
v2 = neo_fft3d(sigma=1.0, beta=1.0, bw=48, bh=48, sharpen=0.040, scutoff=0.01, y=3, u=2, v=2, bt=5, opt=2)
compare(v1, v2)
There is difference with r7.

MeteorRain

3rd May 2020, 14:55

StvG: I don't have AVX512 and every time I test with AVX512 I have to rent a server. Can you confirm if the difference only appears on opt=4? What bit depth?

almosely

3rd May 2020, 19:19

I made a test with prefetch(3) and (5) and got different results. Now I am wondering, if that's a normal behaviour (and has always been like that for all my scripts, but I did'nt realize) or if that's because of neo_fft3d.

script(s) for prefetch-test

DGSource("test.dgi", crop_l=0, crop_r=0, crop_t=0, crop_b=0, deinterlace=0, use_top_field=true, use_pf=true)

RequestLinear(rlim=100, clim=100)

neo_fft3d(sigma=1.0, beta=1.0, bw=48, bh=48, sharpen=0.040, scutoff=0.01, y=3, u=2, v=2, bt=5, opt=3)

avstp_set_threads(1)

Prefetch(3)

# Prefetch(5)

return last

x264 result for prefetch(3)

[2020-05-03][18:59:26] y4m [info]: 1280x720p 1:1 @ 30000/1001 fps (cfr)
[2020-05-03][18:59:26] x264 [info]: using SAR=1/1
[2020-05-03][18:59:26] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-03][18:59:26] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-03][19:06:29] x264 [info]: frame I:33 Avg QP:14.82 size:164557
[2020-05-03][19:06:29] x264 [info]: frame P:1785 Avg QP:18.28 size: 28082
[2020-05-03][19:06:29] x264 [info]: frame B:7078 Avg QP:26.95 size: 2907
[2020-05-03][19:06:29] x264 [info]: consecutive B-frames: 1.5% 1.5% 5.1% 6.4% 18.9% 66.6%
[2020-05-03][19:06:29] x264 [info]: mb I I16..4: 3.6% 55.1% 41.3%
[2020-05-03][19:06:29] x264 [info]: mb P I16..4: 0.5% 2.9% 2.3% P16..4: 44.1% 16.1% 16.4% 0.9% 0.4% skip:16.4%
[2020-05-03][19:06:29] x264 [info]: mb B I16..4: 0.0% 0.1% 0.2% B16..8: 34.9% 2.9% 0.7% direct: 0.7% skip:60.5% L0:46.9% L1:44.0% BI: 9.1%
[2020-05-03][19:06:29] x264 [info]: 8x8 transform intra:49.6% inter:31.3%
[2020-05-03][19:06:29] x264 [info]: direct mvs spatial:99.9% temporal:0.1%
[2020-05-03][19:06:29] x264 [info]: coded y,uvDC,uvAC intra: 86.8% 59.0% 33.7% inter: 7.2% 2.9% 0.3%
[2020-05-03][19:06:29] x264 [info]: i16 v,h,dc,p: 7% 53% 12% 28%
[2020-05-03][19:06:29] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 8% 33% 15% 5% 5% 4% 8% 6% 16%
[2020-05-03][19:06:29] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 10% 28% 8% 6% 9% 7% 11% 7% 14%
[2020-05-03][19:06:29] x264 [info]: i8c dc,h,v,p: 40% 38% 14% 9%
[2020-05-03][19:06:29] x264 [info]: Weighted P-Frames: Y:12.2% UV:0.8%
[2020-05-03][19:06:29] x264 [info]: ref P L0: 62.4% 13.9% 18.1% 3.4% 1.9% 0.2% 0.0%
[2020-05-03][19:06:29] x264 [info]: ref B L0: 86.7% 10.4% 2.5% 0.4%
[2020-05-03][19:06:29] x264 [info]: ref B L1: 98.0% 2.0%
[2020-05-03][19:06:29] x264 [info]: kb/s:2051.98
[2020-05-03][19:06:29] encoded 8896 frames, 20.88 fps, 2052.00 kb/s

x264 result for prefetch(5)

[2020-05-03][19:06:33] y4m [info]: 1280x720p 1:1 @ 30000/1001 fps (cfr)
[2020-05-03][19:06:33] x264 [info]: using SAR=1/1
[2020-05-03][19:06:33] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-03][19:06:33] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-03][19:14:10] x264 [info]: frame I:33 Avg QP:14.82 size:164580
[2020-05-03][19:14:10] x264 [info]: frame P:1785 Avg QP:18.28 size: 28080
[2020-05-03][19:14:10] x264 [info]: frame B:7078 Avg QP:26.96 size: 2907
[2020-05-03][19:14:10] x264 [info]: consecutive B-frames: 1.5% 1.5% 5.1% 6.4% 18.9% 66.6%
[2020-05-03][19:14:10] x264 [info]: mb I I16..4: 3.4% 55.2% 41.4%
[2020-05-03][19:14:10] x264 [info]: mb P I16..4: 0.5% 2.9% 2.3% P16..4: 44.1% 16.2% 16.3% 0.9% 0.4% skip:16.4%
[2020-05-03][19:14:10] x264 [info]: mb B I16..4: 0.0% 0.1% 0.2% B16..8: 34.9% 2.9% 0.7% direct: 0.7% skip:60.5% L0:46.9% L1:44.0% BI: 9.1%
[2020-05-03][19:14:10] x264 [info]: 8x8 transform intra:49.8% inter:31.3%
[2020-05-03][19:14:10] x264 [info]: direct mvs spatial:99.9% temporal:0.1%
[2020-05-03][19:14:10] x264 [info]: coded y,uvDC,uvAC intra: 86.8% 59.0% 33.8% inter: 7.2% 2.9% 0.3%
[2020-05-03][19:14:10] x264 [info]: i16 v,h,dc,p: 7% 53% 11% 29%
[2020-05-03][19:14:10] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 7% 33% 15% 5% 5% 4% 8% 6% 16%
[2020-05-03][19:14:10] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 10% 29% 8% 6% 9% 7% 11% 7% 14%
[2020-05-03][19:14:10] x264 [info]: i8c dc,h,v,p: 40% 38% 14% 9%
[2020-05-03][19:14:10] x264 [info]: Weighted P-Frames: Y:12.2% UV:0.8%
[2020-05-03][19:14:10] x264 [info]: ref P L0: 62.4% 13.9% 18.1% 3.4% 1.9% 0.2% 0.0%
[2020-05-03][19:14:10] x264 [info]: ref B L0: 86.7% 10.4% 2.5% 0.4%
[2020-05-03][19:14:10] x264 [info]: ref B L1: 98.0% 2.0%
[2020-05-03][19:14:10] x264 [info]: kb/s:2051.85
[2020-05-03][19:14:10] encoded 8896 frames, 19.35 fps, 2051.87 kb/s

compare-script after that

v1 = DGSource("test pf 3.dgi", crop_l=0, crop_r=0, crop_t=0, crop_b=0, deinterlace=0, use_top_field=true, use_pf=true)

v2 = DGSource("test pf 5.dgi", crop_l=0, crop_r=0, crop_t=0, crop_b=0, deinterlace=0, use_top_field=true, use_pf=true)

Compare(v1, v2)

return last

https://i.imgur.com/PGp9fQz.jpg

- edit -

Maybe my encoding-settings are relevant too:

[2020-05-03][18:59:23] Creating encoder process:
[2020-05-03][18:59:23] "...\Simple x264 Launcher\toolset\x64\x264_x64.exe" --output-depth 8 --crf 18.0 --preset medium --tune film --trellis 2 --direct auto --me umh --partitions all --vbv-maxrate 24000 --vbv-bufsize 30000 --b-adapt 2 --bframes 5 --merange 24 --ref 5 --keyint 300 --subme 10 --aq-mode 1 --sar 1:1 --rc-lookahead 40 --output "...\test pf 3.mkv" --frames 8896 --demuxer y4m --stdin y4m -

- edit -

And another question: Should I use opt=0 or opt=3 with my CPU. Is neo_fft3d switching between different settings/routines while one job is running (opt=0) or is it using the same code for the whole job, once the decision is set at the beginning?

https://i.imgur.com/wQUVssc.png

MeteorRain

3rd May 2020, 21:11

> using the same code for the whole job

This.

opt is only for debugging purpose. Normal users should never need to set this.

> got different results

Spent a few hours on comparing the results.

First thing first, it's not multi-threading related. It's multi-run related, which means it was caused by multiple runs of the script, not because multiple runs have different thread count.

Secondly, I can't figure out the fundamental reason. It seems to me like floating point computing error.

For example,
out[lfp.w][0] = incur[lfp.w][0];
out[lfp.w][1] = incur[lfp.w][1];
This code gives identical result for each run. However,
out[lfp.w][0] = incur[lfp.w][0] * 0.98f;
out[lfp.w][1] = incur[lfp.w][1] * 0.98f;
This code gives different result for each run.

I have no clue what this is. I tried both /fp:precise and /fp:strict if that matters.

Groucho2004

3rd May 2020, 22:10

avstp_set_threads(1)Do you even know what that does? Are you aware that there's nothing in your script that uses avstp?

almosely

4th May 2020, 00:23

@MeteorRain

Okay, I have dropped the parameter "opt" from my call for neo_fft3d.

Do you know that fftw itself is internally multithreaded too? Maybe this is causing it?

@Groucho

Are you 100% sure about that?

avstp is made by the author of dither tools. It handles the internal single-/multi-threading-functionality of the dither tools. Since I regularly use GradFun3, avstp is important to my scripts. If I only use GradFun3, then I turn off multithreading of my script (#prefetch) and don't use "avstp_set_threads" at all (#), so that GradFun3 can work multithreaded internal. If I use some filters which work faster multithreaded in my scripts, then I turn on "my" multithreading (prefetch) and turn off the internal mt of the dither tools (avstp_set_threads(1)), so that they do not interfere each other.

"avstp_set_threads" has to be written at the very end of a script. Therefore I put it straight before the two very last commands of my scripts (prefetch and return) and handle it manually, by myself, to be absolutely in control of "my" mt and the avstp mt - and not within the mtmodes.avsi (I deleted the function within there).

Nobody knows which routine is using avstp.dll for sure, right? Whether dither.avsi nor mt_xxpand_multi.avsi is including "avstp_set_threads" or "avstp". So this setting must be made from the dither.dll - or could; so any other .dll can do this too and make use of avstp.dll. I can check the thread-count with AVSMeter, but do I know 100% if a .dll is working mt or st at everytime? Maybe sometimes the thread-count changes during a job - I simply don't know, how routines are written internally; I can't look inside.

Therefore, setting "avstp_set_threads(1)" is a safety measure everytime I turn on "my" mt with prefetch().

Groucho2004

4th May 2020, 00:33

I was referring to the script you posted in post #62. There's no plugin in that script that uses avstp. A far as I'm aware, only Dither.dll and mvtools2.dll use avstp.
In some cases, such as QTGMC, avstp can be beneficial in combination with Avisynth+'s internal MT, you have to experiment yourself.
avstp initializes a thread pool in its constructor (IIRC) which could make things slower even if there's no plugin using avstp. The statement avstp_set_threads(1) does not disable avstp. I did some experiments with it some time ago and its sheer presence in the plugin directory influences the number of threads and the rendering speed, albeit slightly.

Just some food for thought.

almosely

4th May 2020, 00:43

"could make things slower even if there's no plugin using avstp. The statement avstp_set_threads(1) does not disable avstp. I did some experiments with it some time ago and its sheer presence in the plugin directory influences the number of threads and the rendering speed"

Oha! Thank you for that advice :-) Then I will figure out a new routine of doing my scripting, probably with "LoadPlugin("xyz.dll")" calls, I don't know.

- edit -

@MeteorRain

I am sorry, I am not a programmer (anymore); I used to be one, but that's 22 years ago. Since then I reorganized my brain to function in other ways, less rational (at least different from that time), more emotional, less mathematic, more optical etc. (studied design, now I am going to make some music again, after about 26 years ago, etc.). So I can't help you with C or something related, sadly.

Groucho2004

4th May 2020, 01:02

almosely

4th May 2020, 04:43

So, meanwhile I benched AviSynth+ 3.4.0 vs 3.5.1. The difference was negligibly if not accidental. Perhaps my memory is fooling me or it has something to do with the latest MSVC++ 2015-2019 Update - or my recent RAM upgrade from 2x4 GB to 4x4 GB. I don't know and I don't care anymore. AVS+ 3.5.1 will stay installed.

And I made a little test regarding avstp.dll. At first, I put avstp.dll out of the avs-plugins64+ folder into another folder. From there I can sideload it as necessary.

I used the following scripts ...

ST

DGSource("test 1080p.dgi", crop_l=0, crop_r=0, crop_t=0, crop_b=0, deinterlace=0, use_top_field=true, use_pf=true)

Trim(0, 1000)

RequestLinear(rlim=100, clim=100)

GradFun3(thr=0.35)

return last

ST mit avstp.dll

LoadPlugin("...\AviSynth+ sideload\plugins64+\avstp.dll")

DGSource("test 1080p.dgi", crop_l=0, crop_r=0, crop_t=0, crop_b=0, deinterlace=0, use_top_field=true, use_pf=true)

Trim(0, 1000)

RequestLinear(rlim=100, clim=100)

GradFun3(thr=0.35)

return last

MT (pf 4)

DGSource("test 1080p.dgi", crop_l=0, crop_r=0, crop_t=0, crop_b=0, deinterlace=0, use_top_field=true, use_pf=true)

Trim(0, 1000)

RequestLinear(rlim=100, clim=100)

GradFun3(thr=0.35)

Prefetch(4)

return last

MT (pf 4) mit avstp.dll

LoadPlugin("...\AviSynth+ sideload\plugins64+\avstp.dll")

DGSource("test 1080p.dgi", crop_l=0, crop_r=0, crop_t=0, crop_b=0, deinterlace=0, use_top_field=true, use_pf=true)

Trim(0, 1000)

RequestLinear(rlim=100, clim=100)

GradFun3(thr=0.35)

Prefetch(4)

return last

MT (pf 4) mit avstp.dll mit avstp_set_threads(1)

LoadPlugin("...\AviSynth+ sideload\plugins64+\avstp.dll")

DGSource("test 1080p.dgi", crop_l=0, crop_r=0, crop_t=0, crop_b=0, deinterlace=0, use_top_field=true, use_pf=true)

Trim(0, 1000)

RequestLinear(rlim=100, clim=100)

GradFun3(thr=0.35)

avstp_set_threads(1)

Prefetch(4)

return last

https://i.imgur.com/P6HTlne.png

In that specific case I don't need avstp.dll neither single- nor multi-threaded (it only hurts).

So, thanks for the hint again, Groucho ;-)

- edit -

... sorry for the german language (some words) sneaking in ;-))

MeteorRain

4th May 2020, 05:51

Seems like bw=48 was the problem. Can you try some different numbers like 32/40/56/64/96/128?

I still haven't figured out what went wrong, but this might be the right approach to the problem.

almosely

4th May 2020, 06:54

Hm, I just realized, that the original author set bw=32 as default, after he has set it to 48 at some time. Oddly I remembered that bw=48 would still be the default. I will try some other sizes.

Maybe it has something to do with his statement?

Features and limitations

Filter works only in YV12 or YUY2 color format.
Only specified single color plane may be processed (Y, U or V). Since v1.8.4 it is possible to process several (all) color planes.
Since v1.3 it works with both progressive and interlaced clips.
Tested with Avisynth v2.55, v2.56.
Filter uses fast external FFTW library version 3 (http://www.fftw.org)
as Windows binary DLL (compiled with gcc under MinGW by Alessio Massaro), which support for threads and have AMD K7 (3dNow!) support in addition to SSE/SSE2.
It may be downloaded from ftp://ftp.fftw.org/pub/fftw/fftw3win32mingw.zip
You MUST put FFTW3.DLL file from this package (not fft3dfilter.dll) to some directory in path (for example, C:\WINNT\SYSTEM32).
Filter will NOT work without it!
FFTW works most efficiently for arrays whose size (bw, bh) can be factored into small primes 2, 3, 5, and 7 (and one 11, 13), and otherwise it uses a slower general-purpose routine.
First versions were really slooow! But recent versions are simple slow :-).
Algorithm is improved and assembler 3DNow! and SSE instruction are used for some modes
Algorithm is optimized by speed for forward sequential frames access only.
The sharpening method is experimental, however is quite good since v1.1 (and v1.7).
The noise pattern method is experimental too.
degrid option is not implemented for Kalman filter.
For noisy video it is useful to use median prefilter (for example DeGrainMedian) before FFT3Dfilter.
Multithread mode ncpu>1 is almost not tested (I do not have core duo), try to use ncpu=2 if you have.

- edit -

Now I know why I remember bw=48 as default, grml ...

Function parameters:

first parameter - input clip

sigma - given noise value for all (or highest) frequencies (float>0, default=2.0)
beta - noise margin (float>=1.0, default=1.0 for no noise left):
control noise amount left in the restoration for Wiener filter, so max filter noise attenuation = (beta-1)/beta.
plane - processed color plane: 0 - luma(Y), 1 - chroma U, 2 - chroma V,
3 - chroma planes U and V, 4 - both luma and chroma (default = 0)
bw - block width (integer, default = 48 since v.1.2)
bh - block height (integer, default = 48 since v.1.2)

... but very little at the very bottom of the documentation:

Version changes:
Version 1.9.2 - 11 september 2006 - changed default bw=bh=32; added mode bt=5

pinterf

4th May 2020, 09:28

StvG: I don't have AVX512 and every time I test with AVX512 I have to rent a server. Can you confirm if the difference only appears on opt=4? What bit depth?
I was using Intel Emulator (https://software.intel.com/en-us/articles/intel-software-development-emulator) with the following jolly-joker batch file
"%~dp0/sde" -- cmd.exe
@rem "%~dp0/sde" -- cmd.exe
@rem "c:\sde\sde" -mrm -- cmd.exe
@rem "c:\sde\sde" -wsm -- cmd.exe
@rem "c:\sde\sde" -skx -- c:\virtualdub\VirtualDub64.exe
"c:\sde\sde" -skx -- cmd.exe
@rem "c:\sde\sde" -wsm -- s1.bat
@rem -mrm Set chip-check and CPUID for Merom
@rem -pnr Set chip-check and CPUID for Penryn
@rem -nhm Set chip-check and CPUID for Nehalem
@rem -hsw Set chip-check and CPUID for Haswell
@rem -wsm Set chip-check and CPUID for Westmere, no AVX
@rem -skx Set chip-check and CPUID for SkyLakeX. AVX512F and AVX512BW

Of course, not for speed tests :) but for visual tests this tool was good enough and I could even debug from Visual Studio by setting project properties|debugging to sde

MeteorRain

4th May 2020, 09:33

SDE causes BSOD on my computer 100% time so I gave up after 2 hard CPU reset in 5 minutes.

pinterf

4th May 2020, 09:40

Then I was lucky (i7-7700 - Win10)

almosely

4th May 2020, 18:33

Can you try some different numbers like 32/40/56/64/96/128?

Test-Script (avstp.dll not loaded anymore)

DGSource("test.dgi", crop_l=0, crop_r=0, crop_t=0, crop_b=0, deinterlace=0, use_top_field=true, use_pf=true)

RequestLinear(rlim=50, clim=50)

neo_fft3d(sigma=1.0, beta=1.0, bw=32, bh=32, sharpen=0.040, scutoff=0.01, y=3, u=2, v=2, bt=5)

Prefetch(3)

return last

32 + 3

[2020-05-04][08:09:02] y4m [info]: 1280x720p 1:1 @ 30000/1001 fps (cfr)
[2020-05-04][08:09:02] x264 [info]: using SAR=1/1
[2020-05-04][08:09:02] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-04][08:09:02] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-04][08:16:36] x264 [info]: frame I:33 Avg QP:14.81 size:164393
[2020-05-04][08:16:36] x264 [info]: frame P:1781 Avg QP:18.28 size: 28087
[2020-05-04][08:16:36] x264 [info]: frame B:7082 Avg QP:26.96 size: 2921
[2020-05-04][08:16:36] x264 [info]: consecutive B-frames: 1.5% 1.4% 5.0% 6.5% 18.4% 67.2%
[2020-05-04][08:16:36] x264 [info]: mb I I16..4: 3.6% 55.0% 41.4%
[2020-05-04][08:16:36] x264 [info]: mb P I16..4: 0.5% 3.0% 2.4% P16..4: 44.0% 16.2% 16.3% 0.9% 0.4% skip:16.4%
[2020-05-04][08:16:36] x264 [info]: mb B I16..4: 0.0% 0.1% 0.2% B16..8: 35.1% 2.9% 0.7% direct: 0.7% skip:60.3% L0:47.0% L1:44.0% BI: 9.0%
[2020-05-04][08:16:36] x264 [info]: 8x8 transform intra:49.6% inter:31.2%
[2020-05-04][08:16:36] x264 [info]: direct mvs spatial:99.9% temporal:0.1%
[2020-05-04][08:16:36] x264 [info]: coded y,uvDC,uvAC intra: 86.4% 59.0% 33.8% inter: 7.1% 2.9% 0.3%
[2020-05-04][08:16:36] x264 [info]: i16 v,h,dc,p: 7% 52% 12% 29%
[2020-05-04][08:16:36] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 8% 33% 16% 5% 5% 4% 8% 6% 15%
[2020-05-04][08:16:36] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 10% 29% 8% 6% 9% 7% 11% 7% 14%
[2020-05-04][08:16:36] x264 [info]: i8c dc,h,v,p: 40% 38% 14% 9%
[2020-05-04][08:16:36] x264 [info]: Weighted P-Frames: Y:12.0% UV:0.7%
[2020-05-04][08:16:36] x264 [info]: ref P L0: 62.3% 13.9% 18.2% 3.4% 1.9% 0.2% 0.0%
[2020-05-04][08:16:36] x264 [info]: ref B L0: 86.6% 10.5% 2.5% 0.4%
[2020-05-04][08:16:36] x264 [info]: ref B L1: 98.0% 2.0%
[2020-05-04][08:16:36] x264 [info]: kb/s:2052.01
[2020-05-04][08:16:36] encoded 8896 frames, 19.47 fps, 2052.03 kb/s

32 + 5

[2020-05-04][08:16:41] y4m [info]: 1280x720p 1:1 @ 30000/1001 fps (cfr)
[2020-05-04][08:16:41] x264 [info]: using SAR=1/1
[2020-05-04][08:16:41] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-04][08:16:41] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-04][08:24:11] x264 [info]: frame I:33 Avg QP:14.81 size:164393
[2020-05-04][08:24:11] x264 [info]: frame P:1781 Avg QP:18.28 size: 28087
[2020-05-04][08:24:11] x264 [info]: frame B:7082 Avg QP:26.96 size: 2921
[2020-05-04][08:24:11] x264 [info]: consecutive B-frames: 1.5% 1.4% 5.0% 6.5% 18.4% 67.2%
[2020-05-04][08:24:11] x264 [info]: mb I I16..4: 3.6% 55.0% 41.4%
[2020-05-04][08:24:11] x264 [info]: mb P I16..4: 0.5% 3.0% 2.4% P16..4: 44.0% 16.2% 16.3% 0.9% 0.4% skip:16.4%
[2020-05-04][08:24:11] x264 [info]: mb B I16..4: 0.0% 0.1% 0.2% B16..8: 35.1% 2.9% 0.7% direct: 0.7% skip:60.3% L0:47.0% L1:44.0% BI: 9.0%
[2020-05-04][08:24:11] x264 [info]: 8x8 transform intra:49.6% inter:31.2%
[2020-05-04][08:24:11] x264 [info]: direct mvs spatial:99.9% temporal:0.1%
[2020-05-04][08:24:11] x264 [info]: coded y,uvDC,uvAC intra: 86.4% 59.0% 33.8% inter: 7.1% 2.9% 0.3%
[2020-05-04][08:24:11] x264 [info]: i16 v,h,dc,p: 7% 52% 12% 29%
[2020-05-04][08:24:11] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 8% 33% 16% 5% 5% 4% 8% 6% 15%
[2020-05-04][08:24:11] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 10% 29% 8% 6% 9% 7% 11% 7% 14%
[2020-05-04][08:24:11] x264 [info]: i8c dc,h,v,p: 40% 38% 14% 9%
[2020-05-04][08:24:11] x264 [info]: Weighted P-Frames: Y:12.0% UV:0.7%
[2020-05-04][08:24:11] x264 [info]: ref P L0: 62.3% 13.9% 18.2% 3.4% 1.9% 0.2% 0.0%
[2020-05-04][08:24:11] x264 [info]: ref B L0: 86.6% 10.5% 2.5% 0.4%
[2020-05-04][08:24:11] x264 [info]: ref B L1: 98.0% 2.0%
[2020-05-04][08:24:11] x264 [info]: kb/s:2052.01
[2020-05-04][08:24:11] encoded 8896 frames, 19.63 fps, 2052.03 kb/s

40 + 3

[2020-05-04][08:24:14] y4m [info]: 1280x720p 1:1 @ 30000/1001 fps (cfr)
[2020-05-04][08:24:14] x264 [info]: using SAR=1/1
[2020-05-04][08:24:14] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-04][08:24:14] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-04][08:31:26] x264 [info]: frame I:33 Avg QP:14.82 size:164351
[2020-05-04][08:31:26] x264 [info]: frame P:1781 Avg QP:18.27 size: 28114
[2020-05-04][08:31:26] x264 [info]: frame B:7082 Avg QP:26.98 size: 2912
[2020-05-04][08:31:26] x264 [info]: consecutive B-frames: 1.5% 1.5% 5.0% 6.4% 18.9% 66.7%
[2020-05-04][08:31:26] x264 [info]: mb I I16..4: 3.8% 55.4% 40.8%
[2020-05-04][08:31:26] x264 [info]: mb P I16..4: 0.5% 3.0% 2.3% P16..4: 44.1% 16.1% 16.4% 0.9% 0.4% skip:16.3%
[2020-05-04][08:31:26] x264 [info]: mb B I16..4: 0.0% 0.1% 0.2% B16..8: 35.0% 2.9% 0.7% direct: 0.7% skip:60.4% L0:46.9% L1:44.1% BI: 9.0%
[2020-05-04][08:31:26] x264 [info]: 8x8 transform intra:49.8% inter:31.3%
[2020-05-04][08:31:26] x264 [info]: direct mvs spatial:99.9% temporal:0.1%
[2020-05-04][08:31:26] x264 [info]: coded y,uvDC,uvAC intra: 86.6% 59.1% 33.9% inter: 7.2% 2.9% 0.3%
[2020-05-04][08:31:26] x264 [info]: i16 v,h,dc,p: 7% 53% 12% 28%
[2020-05-04][08:31:26] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 8% 33% 15% 5% 5% 4% 8% 6% 16%
[2020-05-04][08:31:26] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 10% 29% 8% 6% 9% 7% 11% 7% 14%
[2020-05-04][08:31:26] x264 [info]: i8c dc,h,v,p: 40% 38% 14% 9%
[2020-05-04][08:31:26] x264 [info]: Weighted P-Frames: Y:12.2% UV:0.7%
[2020-05-04][08:31:26] x264 [info]: ref P L0: 62.2% 14.0% 18.2% 3.4% 2.0% 0.2% 0.0%
[2020-05-04][08:31:26] x264 [info]: ref B L0: 86.6% 10.6% 2.5% 0.4%
[2020-05-04][08:31:26] x264 [info]: ref B L1: 98.0% 2.0%
[2020-05-04][08:31:26] x264 [info]: kb/s:2051.38
[2020-05-04][08:31:26] encoded 8896 frames, 20.52 fps, 2051.41 kb/s

40 + 5

[2020-05-04][08:31:29] y4m [info]: 1280x720p 1:1 @ 30000/1001 fps (cfr)
[2020-05-04][08:31:29] x264 [info]: using SAR=1/1
[2020-05-04][08:31:29] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-04][08:31:29] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-04][08:38:40] x264 [info]: frame I:33 Avg QP:14.82 size:164351
[2020-05-04][08:38:40] x264 [info]: frame P:1781 Avg QP:18.27 size: 28114
[2020-05-04][08:38:40] x264 [info]: frame B:7082 Avg QP:26.98 size: 2912
[2020-05-04][08:38:40] x264 [info]: consecutive B-frames: 1.5% 1.5% 5.0% 6.4% 18.9% 66.7%
[2020-05-04][08:38:40] x264 [info]: mb I I16..4: 3.8% 55.4% 40.8%
[2020-05-04][08:38:40] x264 [info]: mb P I16..4: 0.5% 3.0% 2.3% P16..4: 44.1% 16.1% 16.4% 0.9% 0.4% skip:16.3%
[2020-05-04][08:38:40] x264 [info]: mb B I16..4: 0.0% 0.1% 0.2% B16..8: 35.0% 2.9% 0.7% direct: 0.7% skip:60.4% L0:46.9% L1:44.1% BI: 9.0%
[2020-05-04][08:38:40] x264 [info]: 8x8 transform intra:49.8% inter:31.3%
[2020-05-04][08:38:40] x264 [info]: direct mvs spatial:99.9% temporal:0.1%
[2020-05-04][08:38:40] x264 [info]: coded y,uvDC,uvAC intra: 86.6% 59.1% 33.9% inter: 7.2% 2.9% 0.3%
[2020-05-04][08:38:40] x264 [info]: i16 v,h,dc,p: 7% 53% 12% 28%
[2020-05-04][08:38:40] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 8% 33% 15% 5% 5% 4% 8% 6% 16%
[2020-05-04][08:38:40] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 10% 29% 8% 6% 9% 7% 11% 7% 14%
[2020-05-04][08:38:40] x264 [info]: i8c dc,h,v,p: 40% 38% 14% 9%
[2020-05-04][08:38:40] x264 [info]: Weighted P-Frames: Y:12.2% UV:0.7%
[2020-05-04][08:38:40] x264 [info]: ref P L0: 62.2% 14.0% 18.2% 3.4% 2.0% 0.2% 0.0%
[2020-05-04][08:38:40] x264 [info]: ref B L0: 86.6% 10.6% 2.5% 0.4%
[2020-05-04][08:38:40] x264 [info]: ref B L1: 98.0% 2.0%
[2020-05-04][08:38:40] x264 [info]: kb/s:2051.38
[2020-05-04][08:38:40] encoded 8896 frames, 20.51 fps, 2051.41 kb/s

56 + 3

[2020-05-04][08:38:44] y4m [info]: 1280x720p 1:1 @ 30000/1001 fps (cfr)
[2020-05-04][08:38:44] x264 [info]: using SAR=1/1
[2020-05-04][08:38:44] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-04][08:38:44] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-04][08:45:29] x264 [info]: frame I:33 Avg QP:14.81 size:165176
[2020-05-04][08:45:29] x264 [info]: frame P:1780 Avg QP:18.27 size: 28208
[2020-05-04][08:45:29] x264 [info]: frame B:7083 Avg QP:26.94 size: 2909
[2020-05-04][08:45:29] x264 [info]: consecutive B-frames: 1.5% 1.6% 4.9% 6.2% 18.6% 67.2%
[2020-05-04][08:45:29] x264 [info]: mb I I16..4: 3.9% 55.1% 41.1%
[2020-05-04][08:45:29] x264 [info]: mb P I16..4: 0.5% 2.9% 2.3% P16..4: 44.2% 16.1% 16.4% 0.9% 0.4% skip:16.3%
[2020-05-04][08:45:29] x264 [info]: mb B I16..4: 0.0% 0.1% 0.2% B16..8: 34.9% 2.8% 0.7% direct: 0.7% skip:60.5% L0:46.9% L1:44.0% BI: 9.1%
[2020-05-04][08:45:29] x264 [info]: 8x8 transform intra:49.7% inter:31.4%
[2020-05-04][08:45:29] x264 [info]: direct mvs spatial:99.9% temporal:0.1%
[2020-05-04][08:45:29] x264 [info]: coded y,uvDC,uvAC intra: 86.9% 59.0% 33.7% inter: 7.2% 2.9% 0.3%
[2020-05-04][08:45:29] x264 [info]: i16 v,h,dc,p: 7% 52% 12% 29%
[2020-05-04][08:45:29] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 8% 33% 15% 5% 5% 4% 8% 6% 16%
[2020-05-04][08:45:29] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 9% 28% 7% 6% 9% 7% 11% 7% 14%
[2020-05-04][08:45:29] x264 [info]: i8c dc,h,v,p: 40% 38% 14% 9%
[2020-05-04][08:45:29] x264 [info]: Weighted P-Frames: Y:12.1% UV:0.7%
[2020-05-04][08:45:29] x264 [info]: ref P L0: 62.4% 14.0% 18.1% 3.4% 1.9% 0.2% 0.0%
[2020-05-04][08:45:29] x264 [info]: ref B L0: 86.6% 10.5% 2.5% 0.3%
[2020-05-04][08:45:29] x264 [info]: ref B L1: 98.0% 2.0%
[2020-05-04][08:45:29] x264 [info]: kb/s:2055.54
[2020-05-04][08:45:29] encoded 8896 frames, 21.85 fps, 2055.56 kb/s

56 + 5

[2020-05-04][08:45:32] y4m [info]: 1280x720p 1:1 @ 30000/1001 fps (cfr)
[2020-05-04][08:45:32] x264 [info]: using SAR=1/1
[2020-05-04][08:45:32] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-04][08:45:32] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-04][08:52:17] x264 [info]: frame I:33 Avg QP:14.81 size:165176
[2020-05-04][08:52:17] x264 [info]: frame P:1780 Avg QP:18.27 size: 28208
[2020-05-04][08:52:17] x264 [info]: frame B:7083 Avg QP:26.93 size: 2909
[2020-05-04][08:52:17] x264 [info]: consecutive B-frames: 1.5% 1.6% 4.9% 6.2% 18.6% 67.2%
[2020-05-04][08:52:17] x264 [info]: mb I I16..4: 3.9% 55.1% 41.1%
[2020-05-04][08:52:17] x264 [info]: mb P I16..4: 0.5% 2.9% 2.3% P16..4: 44.2% 16.1% 16.4% 0.9% 0.4% skip:16.3%
[2020-05-04][08:52:17] x264 [info]: mb B I16..4: 0.0% 0.1% 0.2% B16..8: 34.9% 2.8% 0.7% direct: 0.7% skip:60.5% L0:46.9% L1:44.0% BI: 9.1%
[2020-05-04][08:52:17] x264 [info]: 8x8 transform intra:49.7% inter:31.4%
[2020-05-04][08:52:17] x264 [info]: direct mvs spatial:99.9% temporal:0.1%
[2020-05-04][08:52:17] x264 [info]: coded y,uvDC,uvAC intra: 86.9% 59.0% 33.7% inter: 7.2% 2.9% 0.3%
[2020-05-04][08:52:17] x264 [info]: i16 v,h,dc,p: 7% 52% 12% 29%
[2020-05-04][08:52:17] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 8% 33% 15% 5% 5% 4% 8% 6% 16%
[2020-05-04][08:52:17] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 9% 28% 7% 6% 9% 7% 11% 7% 14%
[2020-05-04][08:52:17] x264 [info]: i8c dc,h,v,p: 40% 38% 14% 9%
[2020-05-04][08:52:17] x264 [info]: Weighted P-Frames: Y:12.1% UV:0.7%
[2020-05-04][08:52:17] x264 [info]: ref P L0: 62.4% 14.0% 18.1% 3.4% 1.9% 0.2% 0.0%
[2020-05-04][08:52:17] x264 [info]: ref B L0: 86.6% 10.5% 2.5% 0.3%
[2020-05-04][08:52:17] x264 [info]: ref B L1: 98.0% 2.0%
[2020-05-04][08:52:17] x264 [info]: kb/s:2055.56
[2020-05-04][08:52:17] encoded 8896 frames, 21.83 fps, 2055.58 kb/s

64 + 3

[2020-05-04][08:52:21] y4m [info]: 1280x720p 1:1 @ 30000/1001 fps (cfr)
[2020-05-04][08:52:21] x264 [info]: using SAR=1/1
[2020-05-04][08:52:21] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-04][08:52:21] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-04][08:59:29] x264 [info]: frame I:33 Avg QP:14.81 size:165277
[2020-05-04][08:59:29] x264 [info]: frame P:1778 Avg QP:18.28 size: 28214
[2020-05-04][08:59:29] x264 [info]: frame B:7085 Avg QP:26.92 size: 2904
[2020-05-04][08:59:29] x264 [info]: consecutive B-frames: 1.4% 1.6% 5.0% 6.4% 18.2% 67.4%
[2020-05-04][08:59:29] x264 [info]: mb I I16..4: 3.8% 55.2% 41.0%
[2020-05-04][08:59:29] x264 [info]: mb P I16..4: 0.5% 2.9% 2.3% P16..4: 44.2% 16.1% 16.3% 0.9% 0.4% skip:16.3%
[2020-05-04][08:59:29] x264 [info]: mb B I16..4: 0.0% 0.1% 0.2% B16..8: 34.9% 2.8% 0.7% direct: 0.7% skip:60.6% L0:46.8% L1:44.2% BI: 9.0%
[2020-05-04][08:59:29] x264 [info]: 8x8 transform intra:49.8% inter:31.5%
[2020-05-04][08:59:29] x264 [info]: direct mvs spatial:99.9% temporal:0.1%
[2020-05-04][08:59:29] x264 [info]: coded y,uvDC,uvAC intra: 87.0% 59.1% 33.8% inter: 7.2% 2.9% 0.3%
[2020-05-04][08:59:29] x264 [info]: i16 v,h,dc,p: 7% 52% 12% 29%
[2020-05-04][08:59:29] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 7% 33% 15% 5% 5% 4% 8% 6% 16%
[2020-05-04][08:59:29] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 9% 28% 7% 6% 9% 7% 11% 7% 14%
[2020-05-04][08:59:29] x264 [info]: i8c dc,h,v,p: 40% 38% 14% 9%
[2020-05-04][08:59:29] x264 [info]: Weighted P-Frames: Y:12.1% UV:0.7%
[2020-05-04][08:59:29] x264 [info]: ref P L0: 62.4% 14.0% 18.1% 3.4% 1.9% 0.2% 0.0%
[2020-05-04][08:59:29] x264 [info]: ref B L0: 86.7% 10.4% 2.5% 0.4%
[2020-05-04][08:59:29] x264 [info]: ref B L1: 98.0% 2.0%
[2020-05-04][08:59:29] x264 [info]: kb/s:2053.48
[2020-05-04][08:59:29] encoded 8896 frames, 20.66 fps, 2053.50 kb/s

64 + 5

[2020-05-04][08:59:33] y4m [info]: 1280x720p 1:1 @ 30000/1001 fps (cfr)
[2020-05-04][08:59:33] x264 [info]: using SAR=1/1
[2020-05-04][08:59:33] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-04][08:59:33] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-04][09:06:42] x264 [info]: frame I:33 Avg QP:14.81 size:165277
[2020-05-04][09:06:42] x264 [info]: frame P:1778 Avg QP:18.28 size: 28214
[2020-05-04][09:06:42] x264 [info]: frame B:7085 Avg QP:26.92 size: 2904
[2020-05-04][09:06:42] x264 [info]: consecutive B-frames: 1.4% 1.6% 5.0% 6.4% 18.2% 67.4%
[2020-05-04][09:06:42] x264 [info]: mb I I16..4: 3.8% 55.2% 41.0%
[2020-05-04][09:06:42] x264 [info]: mb P I16..4: 0.5% 2.9% 2.3% P16..4: 44.2% 16.1% 16.3% 0.9% 0.4% skip:16.3%
[2020-05-04][09:06:42] x264 [info]: mb B I16..4: 0.0% 0.1% 0.2% B16..8: 34.9% 2.8% 0.7% direct: 0.7% skip:60.6% L0:46.8% L1:44.2% BI: 9.0%
[2020-05-04][09:06:42] x264 [info]: 8x8 transform intra:49.8% inter:31.5%
[2020-05-04][09:06:42] x264 [info]: direct mvs spatial:99.9% temporal:0.1%
[2020-05-04][09:06:42] x264 [info]: coded y,uvDC,uvAC intra: 87.0% 59.1% 33.8% inter: 7.2% 2.9% 0.3%
[2020-05-04][09:06:42] x264 [info]: i16 v,h,dc,p: 7% 52% 12% 29%
[2020-05-04][09:06:42] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 7% 33% 15% 5% 5% 4% 8% 6% 16%
[2020-05-04][09:06:42] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 9% 28% 7% 6% 9% 7% 11% 7% 14%
[2020-05-04][09:06:42] x264 [info]: i8c dc,h,v,p: 40% 38% 14% 9%
[2020-05-04][09:06:42] x264 [info]: Weighted P-Frames: Y:12.1% UV:0.7%
[2020-05-04][09:06:42] x264 [info]: ref P L0: 62.4% 14.1% 18.1% 3.4% 1.9% 0.2% 0.0%
[2020-05-04][09:06:42] x264 [info]: ref B L0: 86.7% 10.4% 2.5% 0.4%
[2020-05-04][09:06:42] x264 [info]: ref B L1: 98.0% 2.0%
[2020-05-04][09:06:42] x264 [info]: kb/s:2053.46
[2020-05-04][09:06:42] encoded 8896 frames, 20.63 fps, 2053.48 kb/s

almosely

4th May 2020, 20:30

I just made two further tests with bw=48 and surprisingly they are identical. Now I do some tests with avstp.dll loaded and opt=3, as I did in first place.

MeteorRain

4th May 2020, 22:24

Don't just test two tests. Take more tests. Trim the video short.
In my tests they have certain results, and you may happen to hit the identical one in that 2 attempts.

almosely

5th May 2020, 00:07

Okay, bw=48 with opt=3, with loaded or unloaded or loaded and "deactivated" avstp.dll were all different from each other.

Then I ran the test again for 32, 40, 48, 56, 64, without avstp.dll and without opt=3, for each bw-size 3 times per prefetch (3 or 5). This time trimmed (0-1000 frames).

Only bw=32 was consistent. Alle three pf 3 were the same, all three pf 5 and all six of them together. Additionally only bw=56 with pf 5 was consistent. Everything else had different results.

This time I ran 3 jobs at a time, parallel. All of the last tests were one job at a time, after another.

- edit -

If your results are consistent, could it be that my system is the error? Maybe my undervolted cpu or my newly added second ram-kit? My CPU is undervolted since a year. I never had issues. I tested it that time with Intel Processor Diagnostic Tool and some x264 encodings. From that point where IPDT and/or x264 gave error messages I went up with the offset for 0.030 Volt. Since then I never had any issue with the offset of -0.190 V. And with my RAM: I did runs with Memtest86+ 5.01, for my old 2x 4GB Kit and one for the new constellation of 4x 4GB (two 2-RAM-Kits of the same type from the same vendor). Both tests passed without an error. HWiNFO displays the same adjustment results for each configuration (8/16 GB); the only different thing is the RAM command rate (changed from 1T to 2T). I really hope, that my system is not the reason for that x264 results.

- edit -

I am running the same tests with fft3dfilter right now. Seems that it ends the same way as with neo_fft3d. Till now, all six bw=32 tests gave identical results. All six bw=40 had different results from each other. The rest is running.

- edit -

So, almost the same with fft3dfilter. All six bw=32 tests gave identical results. All other bws gave different results to each other run. And there is a little difference between neo_fft3d and fft3dfilter (both bw=32). Please do not compare the fps, they are not representative in this case.

neo_fft3d

[2020-05-05][00:18:50] y4m [info]: 1280x720p 1:1 @ 30000/1001 fps (cfr)
[2020-05-05][00:18:50] x264 [info]: using SAR=1/1
[2020-05-05][00:18:50] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-05][00:18:50] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-05][00:21:08] x264 [info]: frame I:5 Avg QP:16.06 size:156722
[2020-05-05][00:21:08] x264 [info]: frame P:207 Avg QP:18.58 size: 28493
[2020-05-05][00:21:08] x264 [info]: frame B:789 Avg QP:26.66 size: 3141
[2020-05-05][00:21:08] x264 [info]: consecutive B-frames: 1.6% 1.6% 8.4% 8.0% 17.5% 62.9%
[2020-05-05][00:21:08] x264 [info]: mb I I16..4: 5.2% 48.4% 46.5%
[2020-05-05][00:21:08] x264 [info]: mb P I16..4: 0.5% 2.8% 2.3% P16..4: 44.0% 15.2% 16.4% 1.1% 0.6% skip:17.1%
[2020-05-05][00:21:08] x264 [info]: mb B I16..4: 0.0% 0.1% 0.2% B16..8: 36.4% 3.1% 0.7% direct: 1.0% skip:58.5% L0:44.7% L1:44.3% BI:11.1%
[2020-05-05][00:21:08] x264 [info]: 8x8 transform intra:47.7% inter:34.3%
[2020-05-05][00:21:08] x264 [info]: direct mvs spatial:99.2% temporal:0.8%
[2020-05-05][00:21:08] x264 [info]: coded y,uvDC,uvAC intra: 84.9% 57.5% 31.5% inter: 8.2% 3.2% 0.3%
[2020-05-05][00:21:08] x264 [info]: i16 v,h,dc,p: 7% 51% 12% 30%
[2020-05-05][00:21:08] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 7% 33% 15% 5% 5% 4% 8% 6% 16%
[2020-05-05][00:21:08] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 10% 29% 8% 6% 8% 7% 11% 7% 14%
[2020-05-05][00:21:08] x264 [info]: i8c dc,h,v,p: 41% 37% 13% 8%
[2020-05-05][00:21:08] x264 [info]: Weighted P-Frames: Y:25.1% UV:1.0%
[2020-05-05][00:21:08] x264 [info]: ref P L0: 66.9% 14.3% 14.3% 2.7% 1.6% 0.2% 0.0%
[2020-05-05][00:21:08] x264 [info]: ref B L0: 88.4% 9.2% 2.0% 0.3%
[2020-05-05][00:21:08] x264 [info]: ref B L1: 98.2% 1.8%
[2020-05-05][00:21:08] x264 [info]: kb/s:2194.06
[2020-05-05][00:21:08] encoded 1001 frames, 7.06 fps, 2194.25 kb/s

fft3dfilter

[2020-05-05][02:18:35] y4m [info]: 1280x720p 1:1 @ 30000/1001 fps (cfr)
[2020-05-05][02:18:35] x264 [info]: using SAR=1/1
[2020-05-05][02:18:35] x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[2020-05-05][02:18:35] x264 [info]: profile High, level 4.0, 4:2:0, 8-bit
[2020-05-05][02:20:59] x264 [info]: frame I:5 Avg QP:16.05 size:156878
[2020-05-05][02:20:59] x264 [info]: frame P:207 Avg QP:18.59 size: 28492
[2020-05-05][02:20:59] x264 [info]: frame B:789 Avg QP:26.72 size: 3146
[2020-05-05][02:20:59] x264 [info]: consecutive B-frames: 1.6% 1.6% 8.4% 8.0% 17.5% 62.9%
[2020-05-05][02:20:59] x264 [info]: mb I I16..4: 5.3% 48.2% 46.4%
[2020-05-05][02:20:59] x264 [info]: mb P I16..4: 0.5% 2.9% 2.3% P16..4: 44.2% 15.1% 16.3% 1.1% 0.5% skip:17.1%
[2020-05-05][02:20:59] x264 [info]: mb B I16..4: 0.1% 0.1% 0.2% B16..8: 36.4% 3.1% 0.7% direct: 1.0% skip:58.5% L0:44.8% L1:44.2% BI:11.0%
[2020-05-05][02:20:59] x264 [info]: 8x8 transform intra:47.9% inter:34.6%
[2020-05-05][02:20:59] x264 [info]: direct mvs spatial:99.2% temporal:0.8%
[2020-05-05][02:20:59] x264 [info]: coded y,uvDC,uvAC intra: 84.9% 57.5% 31.6% inter: 8.2% 3.2% 0.3%
[2020-05-05][02:20:59] x264 [info]: i16 v,h,dc,p: 7% 51% 12% 30%
[2020-05-05][02:20:59] x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 7% 33% 15% 5% 5% 4% 8% 6% 16%
[2020-05-05][02:20:59] x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 9% 29% 8% 6% 8% 7% 11% 7% 14%
[2020-05-05][02:20:59] x264 [info]: i8c dc,h,v,p: 41% 37% 13% 8%
[2020-05-05][02:20:59] x264 [info]: Weighted P-Frames: Y:25.1% UV:1.0%
[2020-05-05][02:20:59] x264 [info]: ref P L0: 66.8% 14.3% 14.4% 2.7% 1.6% 0.2% 0.0%
[2020-05-05][02:20:59] x264 [info]: ref B L0: 88.4% 9.3% 2.0% 0.3%
[2020-05-05][02:20:59] x264 [info]: ref B L1: 98.3% 1.7%
[2020-05-05][02:20:59] x264 [info]: kb/s:2195.05
[2020-05-05][02:20:59] encoded 1001 frames, 6.72 fps, 2195.24 kb/s

MeteorRain

5th May 2020, 11:16

A few assumptions / points here:

1) For 1 frame long clip at bw=48 I get 3 possible results after ~10 runs.
x264 [info]: kb/s:6861.36
x264 [info]: kb/s:6863.66
x264 [info]: kb/s:6864.24

So I assume there are certain results possible, and are deterministic. I get one of them randomly. Maybe it's memory bound -- different memory position / alignment gives different result?

2) prefetch should not affect results. If it's deterministic, it is no matter prefetch or single threaded.

3) Both filter didn't use avstp, so you shouldn't need to take it into consideration.

4) I don't think your hardware is at fault.

5) Core code (algorithm) remains the same between fft3dfilter and neo_fft3d, hence they behaved the same on bw=32/40/48.

ChaosKing

5th May 2020, 11:29

pinterf

5th May 2020, 11:43

Isn' that kb/s the bitrate after compression instead of the processing speed? And if they are different for multiple runs that means that the output is also different?

MeteorRain

5th May 2020, 12:39

Have you also tested with only 1 thread in x264?(idk if multiple threads have any impact on 1 frame, but better be sure)

Yes, that was the first thing to test. Also with bw=32 it's always the same result regardless of threads in x264.

Isn' that kb/s the bitrate after compression instead of the processing speed? And if they are different for multiple runs that means that the output is also different?

Yes it's compression result. And yes, the output seems to be different.

I've checked the result, they have very little different at LSB.

TL;DR:
fftwf_execute_dft_r2c() -> deterministic
Apply() -> nondeterministic

Inside Apply(), for example bt=2, inside Apply3D2(), there is code like below
out[w][0] = something * 0.5f;
This multiplication caused nondeterministic. If it's addition instead of multiplication (or *1.0f which is no-op) the result is still deterministic.

I have no clue how this happened. I'm guessing there's some padding issue (because bw=32 always gives deterministic result), but it can also be problems inside fftw (fftw claimed nondeterministic results in certain circumstances).

almosely

5th May 2020, 19:15

In my tests, the results were "almost" consistent, as long as I ran them one-by-one, after each other and while not working at my pc at best. Since (almost) every test had (way more) different results (except bw=32) when running the jobs parallel, I think, it might be correlated with multi-/hyper-threading or something like that. Maybe one task is not giving back the results to the corresponding task, e.g. FFTW is called from task 1 but giving back the result to task 2. But I do not know if something like that is possible at all. But I do know that some plugins do behave wrong (mostly, but sometimes not), if not called sequential in a mt-script. Maybe with a block size greater than 32 there is built up a chain with splitted parts to compute, but put back together in the wrong order. And with a shitload of tasks running parallel these parts are (more often) not sequential anymore, as they would be (by accident/by speed) when running the jobs one-by-one.

MeteorRain

6th May 2020, 22:24

The computing error is very little, so I don't think there's any major issue in the process.
Differences between each run is less than 0.01% (i.e. 1 or 2 bit at LSB in fftw_complex).

As I'm reading articles I noticed that floating points may not get reproducible results for their operations. I wonder if that is part of the reason for this issue.

almosely

7th May 2020, 02:41

Hm, okay. Seems that this is no issue to worry about. Then I will use neo_fft3d from now on (and without restrictions). Thank you again! :-)

pinterf

7th May 2020, 06:15

So it's fftw? And the old fft3d have those issues as well?

almosely

7th May 2020, 14:37

I am running the same tests with fft3dfilter right now. Seems that it ends the same way as with neo_fft3d. Till now, all six bw=32 tests gave identical results. All six bw=40 had different results from each other. The rest is running.

- edit -

So, almost the same with fft3dfilter. All six bw=32 tests gave identical results. All other bws gave different results to each other run.

At least in my last comparison test, it seems to be the same issue with both.

tormento

7th May 2020, 14:41

RTFM:

Question 3.7. FFTW gives results different from my old FFT.
People follow many different conventions for the DFT, and you should be sure to know the ones that we use (described in the FFTW manual). In particular, you should be aware that the FFTW_FORWARD/FFTW_BACKWARD directions correspond to signs of -1/+1 in the exponent of the DFT definition. (Numerical Recipes uses the opposite convention.)
You should also know that we compute an unnormalized transform. In contrast, Matlab is an example of program that computes a normalized transform. See Q3.10 `Why does your inverse transform return a scaled result?'. Finally, note that floating-point arithmetic is not exact, so different FFT algorithms will give slightly different results (on the order of the numerical accuracy; typically a fractional difference of 1e-15 or so in double precision).

Question 3.8. FFTW gives different results between runs
If you use FFTW_MEASURE or FFTW_PATIENT mode, then the algorithm FFTW employs is not deterministic: it depends on runtime performance measurements. This will cause the results to vary slightly from run to run. However, the differences should be slight, on the order of the floating-point precision, and therefore should have no practical impact on most applications.
If you use saved plans (wisdom) or FFTW_ESTIMATE mode, however, then the algorithm is deterministic and the results should be identical between runs.

almosely

7th May 2020, 15:00

Ahhh, there shall be light :-) So, everything is fine then. Thank you!

MeteorRain

7th May 2020, 22:34

tormento: Just so you know that I already read that part during my research.
Regarding to 3.8 it runs under estimate mode, so it shouldn't be nondeterministic.
Regarding to 3.7 it's possible that rounding errors in float caused the issue. It's just that I can't say "for sure" that it "is" the cause of the issue. I can only say it's "likely" the cause.

tormento

8th May 2020, 09:48

tormento: Just so you know that I already read that part during my research.
Aaaaaaand which version is almosely running? :p

MeteorRain

8th May 2020, 11:38

You know I was talking about your "Question 3.7 / Question 3.8" right?

tormento

8th May 2020, 11:54

You know I was talking about your "Question 3.7 / Question 3.8" right?
Definitely.

MeteorRain

8th May 2020, 12:43

I don't know but I used the version from fftw website.

tormento

8th May 2020, 14:36

I don't know but I used the version from fftw website.
If you did not compile on your own, the latest official windows binary you can get from the site is 3.3.5. Try and use the latest versions have been around since some weeks on this forum.

almosely

8th May 2020, 20:46

I have FFTW 3.3.8 (AVX) (Wolfberry's compile from 29-01-2019) already installed and in use since then.

He compiled this version on request by me (here at this forum). I could'nt find any v3.3.8 AVX x64 that time. I don't know in which mode that compile runs. I thought, that would depend on the kind of call, not the compile. I am surprised.

FFTW 3.3.8 built with ICC 19.0 now in my signature.

Have fun :)

Should I use a more recent MSVC build? Clang is for Mac, GCC etc. for what? Nowadays there are so many builds, I don't know which one to use.

StvG

8th May 2020, 21:48

StvG: I don't have AVX512 and every time I test with AVX512 I have to rent a server. Can you confirm if the difference only appears on opt=4? What bit depth?
Sorry for the late reply.
Used script:LoadPlugin(".\NeoFFT3D_r7\clang-x64\neo-fft3d.dll")
ColorBars(1920,1080, pixel_type="rgbp16") #tested rgbp, rgbps, yv12, yuv420p16, yuv420ps, yuv444ps (same behavior always)
n=3
a=neo_fft3d(sigma=1.0, beta=1.0, bw=48, bh=48, sharpen=0.040, scutoff=0.01, y=3, u=2, v=2, bt=5, opt=n)
b=neo_fft3d(sigma=1.0, beta=1.0, bw=48, bh=48, sharpen=0.040, scutoff=0.01, y=3, u=2, v=2, bt=5, opt=1)
Compare(a,b)
Subtitle("neo_fft3d(sigma=1.0, beta=1.0, bw=48, bh=48, sharpen=0.040, scutoff=0.01, y=3, u=2, v=2, bt=5, opt="+string(n)+")"+"\n"+"neo_fft3d(sigma=1.0, beta=1.0, bw=48, bh=48, sharpen=0.040, scutoff=0.01, y=3, u=2, v=2, bt=5, opt=1)", lsp=1, y=1040)

https://thumbs2.imgbox.com/ae/0a/SBZEn8T5_t.png (http://imgbox.com/SBZEn8T5) https://thumbs2.imgbox.com/79/a6/WpKGRVGh_t.png (http://imgbox.com/WpKGRVGh) https://thumbs2.imgbox.com/0a/fc/48odxsGP_t.png (http://imgbox.com/48odxsGP)

tormento

8th May 2020, 22:54

Should I use a more recent MSVC build? Clang is for Mac, GCC etc. for what? Nowadays there are so many builds, I don't know which one to use.
Enjoy (https://forum.doom9.org/showthread.php?p=1908016#post1908016).

ICL, usually, runs best on Intel.

almosely

9th May 2020, 05:03

Thanks! :-)

I tested all FFTW (I guess it's 3.3.8?) versions (from your link above) against each other, including my old 3.3.8 ICC 19.0, and the fastest and least cpu consuming version was GCC 9.3. I have all MS Visual C++ versions from 2005-2019 installed and updated (for this test) my Intel C++ Redist from 15.0.xx (2015) to the latest 19.0.8 Update 8 (2019) version. Sadly, the newest Intel C++ version (2020 19.1) is not downloadable (Intel-Link is broken). The slowest version was MSVC 2019, then Clang 10.0.

Detailed test results are in the corresponding thread, here (https://forum.doom9.org/showthread.php?p=1911115#post1911115).

I rerun my bw-tests also with some of this versions (ICL 19.1, MSVC 2019, GCC 9.3) and the results were the same as with my old FFTW-version: bw=32 was identical, every time; every other block-size got different results from each other run.

tormento

9th May 2020, 09:56

Sadly, the newest Intel C++ version (2020 19.1) is not downloadable (Intel-Link is broken).
Yep, Intel site is pure shit from that point of view.

Anyway, I uploaded there:

Intel C++ redist 2020.1·216 (https://www.mediafire.com/file/2f576c2qokgen4j/Intel_C++_redist_2020.1%B7216.msi/file)

Intel C++ redist 2020.1·216_x64 (https://www.mediafire.com/file/efea2ncb0r27vew/Intel_C++_redist_2020.1%B7216_x64.msi/file)

Tell me if you need Fortran ones too. :p

kedautinh12

9th May 2020, 10:28

Yep, Intel site is pure shit from that point of view.

Anyway, I uploaded there:

Intel C++ redist 2020.1·216 (https://www.mediafire.com/file/2f576c2qokgen4j/Intel_C++_redist_2020.1%B7216.msi/file)

Intel C++ redist 2020.1·216_x64 (https://www.mediafire.com/file/efea2ncb0r27vew/Intel_C++_redist_2020.1%B7216_x64.msi/file)

Tell me if you need Fortran ones too. :p

Please give me, i need it

tormento

9th May 2020, 10:38

Please give me, i need it

Intel Visual Fortran redist 2020.1·216 (https://www.mediafire.com/file/rbhv1cscxpf0rgj/Intel_Visual_Fortran_redist_2020.1%B7216.msi/file)

Intel Visual Fortran redist 2020.1·216_x64 (https://www.mediafire.com/file/3dhr0xnl18xqmmg/Intel_Visual_Fortran_redist_2020.1%B7216_x64.msi/file)

Here they are. :)

Fortran programming is awesome. Which field of activities do you cover? Some AVS+ porting would show the real speed of a CPU ;)

kedautinh12

9th May 2020, 11:18

Intel Visual Fortran redist 2020.1·216 (https://www.mediafire.com/file/rbhv1cscxpf0rgj/Intel_Visual_Fortran_redist_2020.1%B7216.msi/file)

Intel Visual Fortran redist 2020.1·216_x64 (https://www.mediafire.com/file/3dhr0xnl18xqmmg/Intel_Visual_Fortran_redist_2020.1%B7216_x64.msi/file)

Here they are. :)

Fortran programming is awesome. Which field of activities do you cover? Some AVS+ porting would show the real speed of a CPU ;)

Thanks

almosely

9th May 2020, 17:41

almosely

10th May 2020, 00:16

Meanwhile I installed the provided newest Intel C++ version 2020 and ran two further tests (https://forum.doom9.org/showthread.php?p=1911115#post1911115) about the different FFTW builds. GCC 9.3 is still the fastest and most efficient build for me.

@ StvG

Maybe this is the reason for your test results with your CPU?

Originally Posted by http://www.fftw.org/release-notes.html
FFTW 3.3.5
Jul 31, 2016

- New SIMD support:
-- Power8 VSX instructions in single and double precision. To use, add --enable-vsx to configure.
-- Support for AVX2 (256-bit FMA instructions). To use, add --enable-avx2 to configure.
-- Experimental support for AVX512 and KCVI. (--enable-avx512, --enable-kcvi) This code is expected to work but the FFTW maintainers do not have hardware to test it.
-- Support for AVX128/FMA (for some AMD machines) (--enable-avx128-fma)
-- Double precision Neon SIMD for aarch64. This code is expected to work but the FFTW maintainers do not have hardware to test it.
-- generic SIMD support using gcc vector intrinsics.
- Add fftw_make_planner_thread_safe() API.
- fix #18 (disable float128 for CUDACC).
- fix #19: missing Fortran interface for fftwq_alloc_real.
- fix #21 (don't use float128 on Portland compilers, which pretend to be gcc).
- fix: Avoid segfaults due to double free in MPI transpose.
- Special note for distribution maintainers: Although FFTW supports a zillion SIMD instruction sets, enabling them all at the same time is a bad idea, because it increases the planning time for minimal gain. We recommend that general-purpose x86 distributions only enable SSE2 and perhaps AVX. Users who care about the last ounce of performance should recompile FFTW themselves.

- edit -

I just uploaded Wolfberry's FFTW 3.3.8 ICC 19.0 build here (http://www.mediafire.com/file/hwu5acm7s60tlt5/FFTW_3.3.8_%25282019-01-29%2529_ICC_19.0.7z/file), if someone needs it. There are multiple files for different SIMD versions (SSE2, AVX, AVX2) for x64 and x86.

MeteorRain

10th May 2020, 09:24

Nope, it's likely my code that's broken. In the meantime just use avx2 until I have time to fix it.

tormento

10th May 2020, 10:06

Nope, it's likely my code that's broken. In the meantime just use avx2 until I have time to fix it.
No AVX2 here. :(

tormento

10th May 2020, 10:07

if someone needs it (will last only 14 days)
Register on MediaFire (https://www.mediafire.com/?weo3yc0) (invitation: +1GB for you and me) and your files will last almost forever. I have ones of years ago.

StainlessS

10th May 2020, 10:56

Register on MediaFire
With my free MediaFire acc, think I got something like 52GB storage, I only use a few of hundred MB.
I have ones of years ago.
Me too, 10+ years for some.

MeteorRain

10th May 2020, 11:08

No AVX2 here. :(

That's fine. I was talking about StvG's issue and advising downgrading from AVX512 to AVX2. If you don't have AVX2 you don't have to worry about downgrading from AVX512 to AVX2.

tormento

10th May 2020, 12:08

I just uploaded Wolfberry's FFTW 3.3.8 ICC 19.0 build here
Did you find any difference with newer version?

almosely

10th May 2020, 15:44

Register on MediaFire (https://www.mediafire.com/?weo3yc0) (invitation: +1GB for you and me) and your files will last almost forever. I have ones of years ago.

Done - for the sake of this forum ;-) I usually don't need cloud-space (except for images sharing in forums).

Did you find any difference with newer version?

I only did those speed tests. Huh, this seems to get a neverending story ;-) I just wanted to update AviSynth+ - and then came the rat-tail ;-) ... I think, I am not going to compare the two FFTW versions. If something is going south with my next encodes, I will notice.

Could it be possible that the GCC compile is bypassing the injected Spectre and Meltdown protection microcode somehow, while the ICC and MSVC versions don't?
I have disabled all of them... useless here.

What do you mean with "useless here"? My pc should be protected - it's a workstation and server for just everything and not just for me. The other thread is overloaded by another discussion (dll-linking).

tormento

10th May 2020, 17:24

What do you mean with "useless here"? My pc should be protected - it's a workstation and server for just everything and not just for me.
Those kind of vulnerabilities have specific requirements and spots to hit to be effective. With “here” I mean my home, where everything is behind a firewall and I am the only person who touches my workstation, the server and the two NAS. Everything else is not x86 architecture or don’t have access to outside world.

almosely

10th May 2020, 19:31

Okay, I understand. But in my case, I just want to let those protections activated. I simply do not know enough about it. I do not have a hardware firewall and a software fw (in my pc) is completely useless in that case; the "hit" would be at a place, where windows or any other OS is not aware of (mainboard; Intel Management Engine afaik). Even me - even if I would have all the necessary knowledge - is not to be trusted every time too. Since I switched from Windows 7 to 10 I noticed a performance plus everywhere. So I have no impulse to turn those precautions off to get more out of my system. But I have to say that Windows 10 is a bitch - it is an absolutely horrible thing to learn everything about it (often the hard way) and to configure it. Windows 7 was waaaa......aaaaay easier to manage.

So my question remains. Is it possible for any program (dll etc) to bypass this IME microcode protection? The difference in performance between GCC and all the other builds of FFTW is almost too big to be true.

- edit -

... "and spots to hit to be effective". Isn't it effective enough to get 100 percent control about the ethernet controller or any other hardware in use (camera, keyboard etc.) regarding only one person - and that unrecognized by the user? That reminds me somewhat of Edward Snowdens reports.

- edit -

The performance of Windows 10 is almost everywhere faster than with Windows 7; SATA is slower with Windows 10 (at least in CrystalDiskMark benchs; tested with two different versions, different bench-tools also and with different chipset-drivers and with different SATA drivers). But the reason for that could be the injected IME microcode. I did not have that patch/update with Windows 7 because it wasn't an automatic update and I was not aware of it (one has to install it manually).

- edit -

Hm, I think I am mixing something up here. While installing and configurating Windows 10, I got aware of the Spectre and Meltdown microcode AND the possible vulnerability of the Intel Management Engine. Till then I thought, I would be protected through regular windows updates alone and never paid attention to the IME at all too. But, after a fresh install of Windows 10, I got serious problems while shutting down or rebooting windows 10 and found out, that the IME was the problem. Some websites advised to downgrade the IME (even the german CHIP magazine), but that would be a) a very stupid thing and b) is not necessary at all. After finding a tool to actually check the functionality of the IME (Intel ME System Tools v8 r3), I realized, that my IME wasn't working properly (after a fresh windows 10 install!). So I uninstalled, updated, changed versions ... till somehow sometime I had the newest version installed (by windows update itself) and the Intel ME System Tools result was free of errors. Since then I got never a problem with being stuck while shutting down windows ever again - and hopefully am protected from that point, but I do not know. While checking that infos about the IME I realized that it had some serious vulnerabilities, especially older versions (thanks CHIP, grrrrr). And such vulnerabilities could be complete control over the hardware as I mentioned above. Spectre and Meltdown are other kinds of threats, but I do want to be protected against them too (at least, as far as it is possible).

- edit -

Man ... That microcode isn't stored within the IME, it's stored within the CPU, injected by the operating system, at every boot. Did I mention that things got very complicated nowadays, sometimes? I think I should resurrect my old Amiga 500, which is resting in the attic ...

almosely

12th May 2020, 07:15

I made some new benchmarks (https://forum.doom9.org/showthread.php?p=1911558#post1911558).

tormento

12th May 2020, 08:23

So it's fftw? And the old fft3d have those issues as well?
Has anybody tried to use the double or the extended precision versions of the library?

kedautinh12

13th May 2020, 13:21

I am already plan for this next update, but I was waiting for more plugins to did MCTD update first since it's not work with HBD at all (many plugins need updates)

Many plugins updated added vp8 interface
https://forum.doom9.org/showthread.php?p=1911708#post1911708

real.finder

13th May 2020, 22:52

Many plugins updated added vp8 interface
https://forum.doom9.org/showthread.php?p=1911708#post1911708

not the one I need https://forum.doom9.org/showpost.php?p=1909447&postcount=841

almosely

14th May 2020, 04:40

I just noticed that sigma2-4 is not working at all. Do I have to activate it somehow with another parameter?

kedautinh12

3rd June 2020, 02:14

i see your neo-fft3d missing plane and ncpu for fft3dfilter, can you add it again??

MeteorRain

3rd June 2020, 04:33

almosely: I figured out the issue. Please stand by for r8.

kedautinh12: Any reason why I should add them back?

kedautinh12

3rd June 2020, 05:11

I will replace fft3dfilter with neo-fft3d in TemporalDegrain2 use for myself. But TemporalDegrain2 need them to work, can you add them back?

MeteorRain

3rd June 2020, 05:38

I don't get it. Why TemporalDegrain2 needs them? The function of both parameters have been removed.

FFT3DFilter(plane=degrainPlane, sigma=limitSigma, sigma2=s2, sigma3=s3, sigma4=s4, bt=3, bw=limitBlksz, bh=limitBlksz, ow=overlap, oh=overlap, ncpu=fftThreads)

Change plane=degrainPlane to y=LumaNoise?3:2, u=ChromaNoise?3:2, v=ChromaNoise?3:2, and remove ncpu=fftThreads.

kedautinh12

3rd June 2020, 05:40

kedautinh12

3rd June 2020, 08:54

Hi, i meet error when replace dfttest with neo_dfttest in TemporalDegrain2, i checked and findout neo_dfttest missing lsb. I was removed lsb but i meet error: "clip should be of same size!", can you add lsb again??

DJATOM

3rd June 2020, 09:03

In general you don't need lsb stuff per filter, since avs+ provides filters to transform to/from stacked format. (conversion is lossless)
ConvertBits(16).neo_dfttest(...).ConvertToStacked() will be the same as dfttest(..., lsb=true)

kedautinh12

3rd June 2020, 09:31

almosely

6th June 2020, 13:48

almosely: I figured out the issue. Please stand by for r8.

kedautinh12: Any reason why I should add them back?

Thank you :-) I will test and report ...

- edit -

Tested the functionality and it works fine :-) I did not compare the outcome with fft3dfilter.

Atlantis

29th June 2020, 01:36

DJATOM

29th June 2020, 08:46

Not really the same. 1 option will save time (no copy from src to dst), 2 option will do that copy and 3 option will actually process plane.

Atlantis

29th June 2020, 09:50

What would be the use case for option 1? Because even if you want to use different settings for each plane, you use a combination of option 2 and 3.

StainlessS

29th June 2020, 10:09

What would be the use case for option 1?
When you dont need any kind of result for that plane and speed is primary factor. [eg perhaps only interested in Luma plane, dont need U or V].
Other filters have the same type of options eg Masktools2.

Unprocessed planes will just contain whatever happened to be in that memory buffer before the filter processed that frame.

MeteorRain

29th June 2020, 11:06

I want to try neo_FFT3D

I don't understand
y, u, v (AviSynth+ only)

Whether a plane is to be filtered.

1 - Do not touch, leaving garbage data
2 - Copy from origin
3 - Process

1 and 2 are the same thing! What's the difference? They both don't touch the plane.
Option 1 doesn't guarantee the data is meaningful.

For example, if you only process 1 plane, then it will just use the source memory buffer, and the data in the source memory area remains, and thus you will see the original data untouched.

However if you process 2 planes, the final memory is then copied from 2 process engines, while the 3rd plane has garbage data. It saves you a memory copy every frame.

What would be the use case for option 1? Because even if you want to use different settings for each plane, you use a combination of option 2 and 3.

Because some filters would split luma and chroma, and process then separately. In those cases, you'll end up throwing unused planes away and combining the final results. 1 will save you a tiny bit of time.

In reality, frame copy is not slow, and I have put optimizations to avoid unnecessary copies. It's not a big deal anyway.

Atlantis

29th June 2020, 12:36

OK. So here is my use case. I want to use neo_fft3d and apply different sigmas to Y and U V. Please correct me if I'm wrong, I don't think that applying the same amount of noise reduction to all 3 planes is good. So what I have decided for myself is to add half of noise reduction I do on Y to U V.
I write this for example if this is correct

neo_fft3d(bt=5, sigma=4, u=2, v=2)
neo_fft3d(bt=5, sigma=2, y=2)

4 to Y and 2 to U V.
Is this correct and is the philosophy of adding half of Y to U V a good one?

One strange thing I noticed is that in the old FFT3DFilter if I apply the same sigma to all 3 planes, these is a little desaturation. The color of the picture is not as alive as if you apply it only to Y.
It seems in your filter neo_fft3d this doesn't happen. By default yours is processing all 3 planes and I see no desaturation which is good. Any idea why this happened in the old FFT3DFilter ?

Atlantis

29th June 2020, 12:43

I am also testing neo_dfttest.

dfttest(U=false, V=false, sigma=96, tbsize=5)
encoded 8496 frames in 6158.00s (1.38 fps), 10137.46 kb/s, Avg QP:24.56

neo_dfttest(sigma=96, tbsize=5, u=2, v=2)
encoded 8496 frames in 5636.64s (1.51 fps), 7649.91 kb/s, Avg QP:24.54

Any idea why there is such a huge difference in the final output? 10137.46 kb/s vs 7649.91 kb/s?

crystalfunky

29th June 2020, 13:14

Will this be available in the next Staxrip version?

Atlantis

29th June 2020, 14:05

Ask stax76 here to include it
https://forum.doom9.org/showthread.php?p=1917053
I was also going to ask him. I'm going to do more tests.

In real world x265 encoding 4K HDR, neo_fft3d is definitely faster and I'm going to use neo_fft3d.
With neo_dfttest I don't see much improvement in speed.

Atlantis

30th June 2020, 22:27

FFT3D is missing scene change detection.
https://github.com/pinterf/fft3dfilter/issues/2

Sometimes you get bad frames with scene change. Like the image is not denoised. With bt=5 you get 4 bad frames and with bt=3 you get 2 bad frames. I noticed DFTTest does the same.

I have no idea how hard it is, can you add this feature? Any workaround? I'm going to open a topic only about this.

MeteorRain

1st July 2020, 13:42

I only port filters and optimize existing code. No new feature is implemented due to lack of skills and time.

real.finder

3rd July 2020, 14:54

i see your neo-fft3d missing plane and ncpu for fft3dfilter, can you add it again??

ncpu should back as DFTTest has threads parameter, as now neo_FFT3D either just single thread or has some setting for multi threads

also, seems I get ignored again https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D/issues/3

maybe I should start put some notice that neo_** plugin are not Recommended and not officially supported as dogway did back then in SMDegrain with cretindesalpes/Firesledge moded mvtools http://web.archive.org/web/20131210170708/http://doom10.org/index.php?topic=2178.0

MeteorRain

3rd July 2020, 16:31

I was adding back ncpu as I noticed a misunderstanding on fftw multithreading.

Regarding your ticket, yes somehow the repo was not in my watch list. I'll test when I finish my work.

Currently working on soxfilter.

Atlantis

6th July 2020, 01:27

I have a suggestion for Neo FFT3D y, u, v values. Things should be intuitive and simple and make sense.
I propose. for Y U V use
0 - Copy from origin
1 - Process
2 - Do not touch, leaving garbage data

for example this
neo_fft3d(y=3, u=2, v=2, sigma=6)
neo_fft3d(y=2, u=3, v=3, sigma=3)

That is hard to follow, this below makes much more sense and easier to read.

neo_fft3d(y=1, u=0, v=0, sigma=6)
neo_fft3d(y=0, u=1, v=1, sigma=3)

You already deleted planes because they were hard to read and YUV are easier names. So the values they get should follow the same rule and not be something like 1 for something that not many people use.

crystalfunky

10th July 2020, 15:24

Can somebody give me good values I can use for 4K HDR encoding,
x264 normal movies and also for b/w movies like 12 angry men?
Would appreciate it!

Atlantis

11th July 2020, 08:49

Every movie is different and you can't have good values for all. You have to try different things and see what you like.

I personally try to make it as simple as possible, so I just change the sigma value only and see what I like. I go from 1 to even 6.

crystalfunky

14th July 2020, 20:44

Are the commands and their values in the neo versions from dftest and fft3d different from the old versions?
If not, I need to understand them, so I can test different values and finally use the filters.

Dogway

2nd August 2020, 20:48

real.finder

2nd August 2020, 23:37

Dogway

3rd August 2020, 00:58

sneo_dfttest is warper I made to make it use neo_dfttest and if there are no neo_dfttest loaded it will use old dfttest

Yes I know, I use neo_dfttest through your wrapper (didn't actually check what was doing though), but the filter has issues as depicted by the linked video with random color spots.

I spent 2 days trying different things to find the culprit and it actually was neo_dfttest, so for now I cannot recommend it to anyone.

ajp_anton

6th August 2020, 05:07

Your filters neo_dfttest and neo_fft3d don't pass through audio. Is that by design?

StainlessS

6th August 2020, 13:26

To not do anything for audio, do not define GetAudio() [in the filter class, think thats what its called] member function, will use default behaviour, ie pass-though.

Mobile:

l33tmeatwad

15th August 2020, 22:12

This probably won't get any traction, but the way FFTW3 is being loaded by this plugin is causing FFTW3 to hard crash on macOS. I've tried to narrow down what is causing it, it appears to be crashing when calling it a second time (past validating that it exists). Crash will happen when attempting to seek or advance by a single frame. I know Catalina (the only version this will compile on) has caused issues for a lot of similar issues with other plugins, but I'm not sure what to do about fixing it.

kedautinh12

9th September 2020, 18:28

Can anyone compile neo-fft3d with this commit??
https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D/commit/de866338d2de7a2c100aa1ba6a1e098280d5d884

Thanks

real.finder

9th September 2020, 19:14

I already did both http://www.solidfiles.com/v/xZn85B3knMGAy and http://www.solidfiles.com/v/55WKMW38dpX42

kedautinh12

10th September 2020, 00:15

Thanks

LeXXuz

4th October 2020, 13:23

I get artifacts with this filter (check link (https://www.mediafire.com/file/ej0uaxqnd2wk4kd/Attraction+3+HD4.mkv/file))

What kind of artifacts are you referring to? All I see is mosquito noise around the letters (which is probably in the source video too, but masked by noise) and a little bit of banding.

LeXXuz

4th October 2020, 21:55

I get quite different filesizes in direct comparison between dfttest and neo-dfttest with AVS+.
https://abload.de/thumb/screenshot2020-10-041yvjbg.png (https://abload.de/image.php?img=screenshot2020-10-041yvjbg.png)

Have there been any changes to the default filtering method or have the sigma values been reworked?

I used these script lines for dfttest/neo-dfttest:

dfttest(Y=true, U=true, V=true, sigma=64, tbsize=5, dither=0, threads=8, opt=4)

and

neo_dfttest(Y=3, U=3, V=3, sigma=64, tbsize=5, dither=0, threads=8, opt=3)

Shouldn't filesizes roughly be the same with both filters :confused:

Selur

2nd November 2020, 20:52

@realfinder: will the ncpu support make it to the git (https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D) any time soon? because the release there does not support 'ncpu'.

Cu Selur

real.finder

2nd November 2020, 21:07

@realfinder: will the ncpu support make it to the git (https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D) any time soon? because the release there does not support 'ncpu'.

Cu Selur

it's up to MeteorRain :) I am not a plugins Developer nor having access to https://github.com/HomeOfAviSynthPlusEvolution

Selur

2nd November 2020, 21:24

okay, thanks for the info,..

kedautinh12

3rd November 2020, 00:06

@realfinder: will the ncpu support make it to the git (https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D) any time soon? because the release there does not support 'ncpu'.

Cu Selur

In neo_FFT3D compiled by realfinder have ncpu function

Selur

7th November 2020, 13:42

I know, but the source code does not, so no use when on MacOS or Linux. (using Vapoursynth)

real.finder

7th November 2020, 16:10

I know, but the source code does not, so no use when on MacOS or Linux. (using Vapoursynth)

I didn't change the source code, only build from last changes

MeteorRain

2nd January 2021, 05:34

I finally squeeze some time during new year holiday, and upload the latest change, with extra parameter mt that defaults to false to avoid strange issues for FFT3D.

If I get a chance I'm gonna update dfttest as well.

As a side note, if anyone is willing to participate in developing or maintaining any projects in this org, please let me know and I'll invite you.

Nuihc88

4th January 2021, 16:58

I'm seeing neo_FFT3D r10 sources, but no binaries.

StainlessS

4th January 2021, 21:36

Nuihc88,
Yeh I always take a little time to find binaries on current GitHub,
See Releases on right Hand Side, Under Above.
Or here:- https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D/releases

EDIT: Actually its above r9 in small itsy bitsy letters [I downed r9 by mistake].
r10

1537395 zip tar.gz <<<<<<<<<<<< HERE, zip

r9

@msg7086 msg7086 released this on 22 Jun 2020 · 8 commits to master since this release

EDIT: Nah, the zip is only source code, I thought the tar.gz was source.

EDIT: OK, got it now. :)

kedautinh12

5th January 2021, 00:06

He was compile binary r10 and update it but i don't know why he deleted it

MeteorRain

5th January 2021, 01:52

Oh, it was a small glitch. I moved the tag forward to compile a Linux version but forgot to republish the binary on website. Should work now.

FranceBB

6th January 2021, 10:11

Thanks for the update.

Atlantis

6th January 2021, 22:32

Could you please explain more with the new option in V10? Is it bad if we enable mt? How was this working in V9?

kedautinh12

7th January 2021, 00:06

Could you please explain more with the new option in V10? Is it bad if we enable mt? How was this working in V9?

Here: https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D/issues/4

Atlantis

7th January 2021, 00:35

What is libneo-fft3d.dll inside gcc-x64?
Why is it named differently?

kedautinh12

7th January 2021, 02:42

For linux??

pinterf

7th January 2021, 08:13

No, it is gcc build for Windows. For avisynth built with gcc. C++ ABI is different from Msvc.

LeXXuz

7th January 2021, 11:02

Is it possible to set different sigma-strings for luma and chroma planes without having to run neo-dfttest twice?

StainlessS

7th January 2021, 14:32

Just run it twice with eg filter1 Plane=0, and filter2 Plane=3.
1st filter only does Luma, 2nd only chroma.
(you''re still only processing 3 planes, even if it is done in 2 filter instances)

EDIT: Plane

0 Y
1 U
2 V
3 U+V
4 Y+U+V

LeXXuz

7th January 2021, 19:05

I know I can run it twice with only luma and chroma processing. I was just wondering if it wouldn't be faster to have just one filter call instead of two.

StainlessS

7th January 2021, 20:35

Yeah, and you could combine all avisynth filters into a single filter, thus doing away with almost all of them, NOT!

ChaosKing

7th January 2021, 22:16

I know I can run it twice with only luma and chroma processing. I was just wondering if it wouldn't be faster to have just one filter call instead of two.

Benchmark it and post the results :cool:
I don't think there will be much difference.

MeteorRain

8th January 2021, 18:39

What is libneo-fft3d.dll inside gcc-x64?
Why is it named differently?

Didn't intent to include gcc version. I've removed it to avoid confusion for others. No need to re-download.

I know I can run it twice with only luma and chroma processing. I was just wondering if it wouldn't be faster to have just one filter call instead of two.

No. Internally they do the same thing, creating 3 engines and processing 3 planes separately.

LeXXuz

10th January 2021, 09:01

No. Internally they do the same thing, creating 3 engines and processing 3 planes separately.

Ah, that's the info I was looking for. Thank you. :)

real.finder

24th March 2021, 21:07

I get artifacts with this filter (check link (https://www.mediafire.com/file/ej0uaxqnd2wk4kd/Attraction+3+HD4.mkv/file)) and had to roll back to plain dfttest. Probably an issue with HBD but I'm not sure. It gave me headaches for the last few days debugging my script.

This is my function call:
sneo_dfttest(slocation="0.0:5.0 0.2:4.0 1.0:1.0",tbsize=3,u=3,v=3)

this should be fixed on last update in sneo_dfttest in Community tree

I get quite different filesizes in direct comparison between dfttest and neo-dfttest with AVS+.
https://abload.de/thumb/screenshot2020-10-041yvjbg.png (https://abload.de/image.php?img=screenshot2020-10-041yvjbg.png)

Have there been any changes to the default filtering method or have the sigma values been reworked?

I used these script lines for dfttest/neo-dfttest:

dfttest(Y=true, U=true, V=true, sigma=64, tbsize=5, dither=0, threads=8, opt=4)

and

neo_dfttest(Y=3, U=3, V=3, sigma=64, tbsize=5, dither=0, threads=8, opt=3)

Shouldn't filesizes roughly be the same with both filters :confused:

maybe those because neo_dfttest is based on VS dfttest port which has some default settings changes

neo/vs -> original avs dfttest

sigma/sigma2
8.0 -> 16.0

sbsize
16 -> 12

sosize
12 -> 9

tbsize
3 -> 5

so I think these need to be added in

Difference compared to pinterf/dfttest

Dual interface, supporting AviSynth+ and VapourSynth
Parameter names and opt levels follow VapourSynth-DFTTest
Removed Y, U, V options, added y, u, v options for plane control
Removed hacked 16 bit I/O
Replaced internal multi-threading with C++ Parallel STL, removed Windows specific APIs
Improved precision of floating point reciprocal using Newton Raphson method

edit: sigma/sigma2 was mentioned https://forum.doom9.org/showpost.php?p=1907702&postcount=4 but others are not

LeXXuz

9th May 2021, 14:02

I have a general question regarding dfttest. But the original thread I found is rather ancient so maybe this is a better place.

My question is regarding frequencies with the sstring-parameter. I would like to understand this better. :o

From the Wiki:

string sstring, ssx, ssy, sst = ""

Used to specify functions of sigma based on frequency.

If you want sigma to vary based on frequency, then use 'sstring' instead of sigma. sstring allows you to enter values of sigma for different normalized [0.0,1.0] frequency locations.
Values for locations between the ones you explicitly specify are computed via linear interpolation. The frequency range, which is dependent on sbsize/tbsize, is normalized to [0.0,1.0] with 0.0 being the lowest frequency and 1.0 being the highest frequency.

I'd like to understand better what is meant by frequencies in this case. After Fourier transform higher frequencies represent higher/finer structures or detail in the picture while lower frequencies represent low details, right?

Noise is usually somewhat fine detail, so higher frequencies should be filtered maybe more, than lower ones?

I ran some tests on a sample with very fine/ high pitched noise and these sigma strings:
1) sstring="0.0:0 0.7:0 0.8:64 1.0:64" (strong filtering for hf and no filtering for lf)
and vice versa
2) sstring="0.0:64 0.2:64 0.3:0 1.0" (no filtering for hf and strong filtering for lf)

According to the Wiki, the first string should filter that very fine noise much better than the second. But the opposite seems to be the case. So I was wondering if that maybe a typo in the Wiki or I made a mistake.

Can someone test and either confirm or disprove this? :o

real.finder

29th May 2021, 06:45

seems both plugins here have same problem as https://github.com/HomeOfAviSynthPlusEvolution/neo_f3kdb/issues/7 (at least in avs+)

kedautinh12

30th May 2021, 16:57

real.finder updated new commits to fix Update AVS headers and copy frame properties for AVS+ and VS. If anyone interested
https://github.com/HomeOfAviSynthPlusEvolution/neo_DFTTest/commits/master
https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D/pull/6/commits

kedautinh12

31st May 2021, 07:32

After tested with new commits from real.finder. I seen speed was increased

tormento

31st May 2021, 09:48

After tested with new commits from real.finder. I seen speed was increased
Where to find the Windows build?

kedautinh12

31st May 2021, 10:25

Here only x64 built by real.finder, need x86 build by someone, thanks
https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D/files/6566428/neo-x64.zip

GMJCZP

31st May 2021, 14:37

Here only x64 built by real.finder, need x86 build by someone, thanks
https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D/files/6566428/neo-x64.zip

Thanks, we will be waiting for the x86 version to test it.

kedautinh12

17th August 2021, 02:26

Sorry for the late reply.
Used script:LoadPlugin(".\NeoFFT3D_r7\clang-x64\neo-fft3d.dll")
ColorBars(1920,1080, pixel_type="rgbp16") #tested rgbp, rgbps, yv12, yuv420p16, yuv420ps, yuv444ps (same behavior always)
n=3
a=neo_fft3d(sigma=1.0, beta=1.0, bw=48, bh=48, sharpen=0.040, scutoff=0.01, y=3, u=2, v=2, bt=5, opt=n)
b=neo_fft3d(sigma=1.0, beta=1.0, bw=48, bh=48, sharpen=0.040, scutoff=0.01, y=3, u=2, v=2, bt=5, opt=1)
Compare(a,b)
Subtitle("neo_fft3d(sigma=1.0, beta=1.0, bw=48, bh=48, sharpen=0.040, scutoff=0.01, y=3, u=2, v=2, bt=5, opt="+string(n)+")"+"\n"+"neo_fft3d(sigma=1.0, beta=1.0, bw=48, bh=48, sharpen=0.040, scutoff=0.01, y=3, u=2, v=2, bt=5, opt=1)", lsp=1, y=1040)

https://thumbs2.imgbox.com/ae/0a/SBZEn8T5_t.png (http://imgbox.com/SBZEn8T5) https://thumbs2.imgbox.com/79/a6/WpKGRVGh_t.png (http://imgbox.com/WpKGRVGh) https://thumbs2.imgbox.com/0a/fc/48odxsGP_t.png (http://imgbox.com/48odxsGP)

Fixed
https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D/releases

kedautinh12

17th August 2021, 02:27

Thanks, we will be waiting for the x86 version to test it.

Had x86 ver

bruno321

6th March 2022, 06:14

I'm trying to load the plugin but failing. Here's what avsmeter reports:

Cannot load file 'C:/Program Files (x86)/AviSynth+/plugins64/neo-fft3d.dll'. Platform returned code 126:
The specified module could not be found.

Dependencies that could not be loaded:
MSVCP140_ATOMIC_WAIT.dll

Visual C++ Redistributable for Visual Studio 2015 is already installed, is there something else I should be installing? Thanks.

kedautinh12

6th March 2022, 06:43

You can download latest ver here:
https://github.com/abbodi1406/vcredist/releases

kedautinh12

30th August 2022, 05:41

Neo-DFTTest updated r8
https://github.com/HomeOfAviSynthPlusEvolution/neo_DFTTest/releases

FranceBB

31st August 2022, 07:41

Thanks for the heads up.
Github is on drugs this morning, though:

<Code>ServerBusy</Code>
<Message>Egress is over the account limit. RequestId:59449abb-501e-005b-2704-bd1b5e000000 Time:2022-08-31T06:40:34.5336533Z</Message>

EDIT: And now is back.

EDIT2: If you're using Windows XP please note that r8 x86 build will NOT work on Windows XP!