FFT3DFilter 2.6

[pinterf]

Big thanks to martin53 who woke up first an made and update to this plugin recently.

EDIT:
Get latest version from https://github.com/pinterf/fft3dfilter/releases

Then I checked what the source looked like from inside, put it on git, modified one or two things and now it's ready for experimenting.

Experimental.
x86/x64 versions and XP support (I hope)

apply current avs+ headers
10-16 bits and 32 bit float colorspace support in AVS+
Planar RGB support
look for libfftw3f-3.dll first, then fftw3.dll
inline asm ignored on x64 builds
pre-check: if plane to process for greyscale is U and/or V then returns original clip
auto registers MT mode for avs+: MT_SERIALIZED
autoscale sigma and smin/smax parameter from 8 bit scale if colorspace is different


I need help what parameters are affected when bit depth changes. Now I autoscale all the sigma parameters, smin and smax.

Compiled with VS2015

[Groucho2004]

I checked XP support, works fine (x86/x64). By the way, remove the 3DNow code, I'm pretty sure nobody uses this plugin who doesn't have a CPU that supports at least SSE (not even Manolito :D).

[real.finder]

we all love you pinterf, you are our hero :)

[pinterf]

I checked XP support, works fine (x86/x64). By the way, remove the 3DNow code, I'm pretty sure nobody uses this plugin who doesn't have a CPU that supports at least SSE (not even Manolito :D).
I left there four of them. Two had no sse replacement, only c. They'll be kept only until I check them what they are doing and move them to simd intrinsics.

[FranceBB]

Really really appreciated! Big thanks! :D

[tuanden0]

Can't wait to test, thank you pinterf.

[pinterf]

I left there four of them. Two had no sse replacement, only c. They'll be kept only until I check them what they are doing and move them to simd intrinsics.
Converted one of the inline asm function to simd, and yes, worth doing it.
For a simple FFT3DFilter(sigma=3,plane=4)
v2.3 (x86): 10.4 fps (VS2015, inline asm, speed is same as simd) - for comparison
v2.2 (x64): 9.27 fps (ICL build, C)
v2.3 (x64): 9.15 fps (VS2015, C)
test (x64): 11.94 fps (VS2015, simd intrinsics)

[vcmohan]

In short will it be possible to indicate what this filter does? Excepting that it does a 3D FFT I could not infer anything else.

[Groucho2004]

In short will it be possible to indicate what this filter does? Excepting that it does a 3D FFT I could not infer anything else.
Documentation is here (http://avisynth.org.ru/fft3dfilter/fft3dfilter.html).

[pinterf]

New build

Download FFT3DFilter 2.4 (https://github.com/pinterf/fft3dfilter/releases/tag/v2.4)

Some efforts on x64 and Avs+ MT auto-registration changes


FFT3DFilter v2.4 (20170608)
- some inline asm (not all) ported to simd intrisics, helps speedup x64 mode, but some of them faster also on x86.
- intrinsics bt=0
- intrinsics bt=2, degrid=0, pfactor=0
- intrinsics bt=3 sharpen=0/1 dehalo=0/1
- intrinsics bt=3
- Adaptive MT settings for Avisynth+: MT_SERIALIZED for bt==0 (temporal), MT_MULTI_INSTANCE for others
- Copy Alpha plane if exists
- reentrancy checks against bad multithreading usage
Note: for properly operating in MT_SERIALIZED mode in Avisynth+ MT, please use Avs+ r2504 or better.

FFT3DFilter v2.3 (20170221)
- apply current avs+ headers
- 10-16 bits and 32 bit float colorspace support in AVS+
- Planar RGB support
- look for libfftw3f-3.dll first, then fftw3.dll
- inline asm ignored on x64 builds
- pre-check: if plane to process for greyscale is U and/or V return original clip

[stax76]

Could FFT3DFilter filter load the required DLLs if they are not yet loaded and are located in the same folder then FFT3DFilter?

I have the DLLs in the same folder then FFT3DFilter and manually load FFT3DFilter with LoadPlugin, I get the following error:

FFT3DFilter: libfftw3f-3.dll or fftw3.dll not found. Please put in PATH or use LoadDll() plugin

For some tools I just detect issues and show a warning with instruction what to do, if possible I prefer to integrate things in a way that just works without the user having to do something.

[stax76]

I found a plugin called 'LoadDll' which I could use but is there a x64 version?

[raffriff42]

Could FFT3DFilter filter load the required DLLs if they are not yet loaded and are located in the same folder then FFT3DFilter?The DLLs would have to be in the current working directory (which can be set with SetWorkingDir (http://avisynth.nl/index.php/Internal_functions#SetWorkingDir)), on the Path, or in an explicit, specified folder.

(don't know about LoadDll, but I suppose it has the same requirement)

[stax76]

The DLLs would have to be in the current working directory (which can be set with SetWorkingDir (http://avisynth.nl/index.php/Internal_functions#SetWorkingDir)), on the Path, or in an explicit, specified folder.

(don't know about LoadDll, but I suppose it has the same requirement)

This could only work when staxrip loads the script but not x264, I try to make a installer with inno or just run xcopy.

[raffriff42]

Inno (http://www.jrsoftware.org/isdl.php#qsp) supports copying your DLLS to the System directory, which of course is on the Path. Job done.

[stax76]

Inno supports copying your DLLS to the System directory, which of course is on the Path. Job done.

Problem is I'm clueless about inno (just gave up) and I'm apparently not the only one here. :D

Using xcopy now!

[raffriff42]

Try the starter pack (http://www.jrsoftware.org/isdl.php#qsp) and take the optional Inno Studio.

EDIT here's a screenshot - a file to be copied to the 32-bit System directory (click to enlarge)

https://www.dropbox.com/s/j9cml1xnh55h71r/InnoSetup-files-3s.jpg?raw=1 (https://www.dropbox.com/s/sxbgv8zh4ixtw88/InnoSetup-files-3.png?raw=1)

[stax76]

Does that use 'Inno Setup Compiler', I have v5.5.6 installed to use for my custom avs setup, doesn't support High DPI. :(

[stax76]

I have it working, thanks for the screenshot, that has helped. One thing I don't like is the installer is 0.5 MB larger compared to packed with 7zip, this would probably increase my package with 0.5 MB, it's already about 100 MB and I'm very concerned about file size because of the the hoster and users with low bandwidth.

[Groucho2004]

I found a plugin called 'LoadDll' which I could use but is there a x64 version?
I just made a 64 bit version but I can't test it right now. See if it works:
LoadDLL64 (https://www.dropbox.com/s/8wj7ph7mq78c392/LoadDLL64.7z?dl=0)

[stax76]

Thanks, I need automatic loading though for my use case, anything else is suboptimal, now it does xcopy the files to the sys dir which in my opinion is bad practice. I can make a request at pinterf's FFT3DFilter github tracker asking if FFT3DFilter could load the DLLs if they are located in the same dir then FFT3DFilter.

[Groucho2004]

I can make a request at pinterf's FFT3DFilter github tracker asking if FFT3DFilter could load the DLLs if they are located in the same dir then FFT3DFilter.
Not a bad idea. This is pretty easy - Get the full path to fft3dfilter with GetCurrentDirectory() and use a loadlibrary() call with the full path in addition to just "LoadLibrary("libfftw3f-3.dll")".

[Myrsloik]

Not a bad idea. This is pretty easy - Get the full path to fft3dfilter with GetCurrentDirectory() and use a loadlibrary() call with the full path in addition to just "LoadLibrary("libfftw3f-3.dll")".

The proper solution is to use LoadLibraryEx() and altered search paths.

[Groucho2004]

The proper solution is to use LoadLibraryEx() and altered search paths.Sure. I just have some vague memory about a problem with the LOAD_WITH_ALTERED_SEARCH_PATH flag and WinXP but this goes way back and possibly applied to WinXP pre-SP3.

[TheFluff]

Thanks, I need automatic loading though for my use case, anything else is suboptimal, now it does xcopy the files to the sys dir which in my opinion is bad practice. I can make a request at pinterf's FFT3DFilter github tracker asking if FFT3DFilter could load the DLLs if they are located in the same dir then FFT3DFilter.

Fix the underlying problem instead. Avs+ actually already does a half-hearted attempt at fixing the issue (plugins depending on other DLL's not finding their dependencies) by temporarily setting the process-wide DLL search directory (https://msdn.microsoft.com/en-us/library/windows/desktop/ms686203(v=vs.85).aspx) to the folder it is loading the plugin from, when it does the actual loading. That doesn't help in this case though because FFT3DFilter doesn't actually load its dependency when the DLL itself is loaded, it loads it every time the filter is instantiated (which is really weird and I don't understand why). That's the core issue.

Replacing the Avs+ LoadLibrary call (https://github.com/pinterf/AviSynthPlus/blob/2d3cb6c011f520fc9433d6bff61a6d697cb09ecb/avs_core/core/PluginManager.cpp#L514) with LoadLibraryEx using LOAD_WITH_ALTERED_SEARCH_PATH might fix the problem without the silly process-wide loading directory juggling if the search order stuff persists for the loaded DLL after LoadLibrary returns, but I have no idea if that is the case. I think I would expect that it doesn't, but I haven't found a satisfactory answer. For whatever reason, VS went with LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR instead.

Not a bad idea. This is pretty easy - Get the full path to fft3dfilter with GetCurrentDirectory() and use a loadlibrary() call with the full path in addition to just "LoadLibrary("libfftw3f-3.dll")".
There's nothing that says the current directory is the plugins folder. You can actually get a DLL's own path though using some other mildly exotic win32 API gymnastics (I resorted to it in assrender once upon a time) but it's silly. Don't do that, fix the loading side instead.

[Groucho2004]

There's nothing that says the current directory is the plugins folder.You're right, brain fart.

This would work to get the fully qualified path to the plugin when called from the plugin (I assume that's what you meant by "You can actually get a DLL's own path though using some other mildly exotic win32 API gymnastics"):

HMODULE GetCurrentModule()
{
HMODULE hModule = NULL;
GetModuleHandleEx(
GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT,
(LPCTSTR)GetCurrentModule,
&hModule);

return hModule;
}

HMODULE hm = GetCurrentModule();
char szPath[MAX_PATH + 1];
GetModuleFileName(hm, szPath, MAX_PATH);
Note: No error handling whatsoever in this code ;-)

[stax76]

@TheFluff

Not only staxrip loads avs but also the encoders, it might be possible to configure the encoder process, setting working dir or path variable but this would be a great ugly mess I believe, doesn't feel right, I would even consider to examine it.

[TheFluff]

You're right, brain fart.

This would work to get the fully qualified path to the plugin when called from the plugin (I assume that's what you meant by "You can actually get a DLL's own path though using some other mildly exotic win32 API gymnastics"):

Yes that is what I meant and that looks like it could work. When I did it I added a DllMain entry point (Avisynth plugins don't normally have one because it's not needed, but if it exists it'll get called) for the sole purpose of getting the system to pass me a HINSTANCE (= HMODULE) referring to the DLL itself as its first parameter, which I then passed directly to GetModuleFileName. Whatever works. There are probably more clever ways to do it.

@TheFluff

Not only staxrip loads avs but also the encoders, it might be possible to configure the encoder process, setting working dir or path variable but this would be a great ugly mess I believe, doesn't feel right, I would even consider to examine it.

Yes that's what I'm saying, it should be fixed in Avisynth itself. No point in trying to work around it on the client side.

[Groucho2004]

Yes that is what I meant and that looks like it could work.It does work, I tested it.

it should be fixed in Avisynth itself. No point in trying to work around it on the client side.Yes, that makes sense.

[pinterf]

New build
Download FFT3DFilter 2.5 (https://github.com/pinterf/fft3dfilter/releases/tag/v2.5)

Change log
FFT3DFilter v2.5 (20180702)
- 32bit Float YUV: Chroma center to 0.0 instead of 0.5, to match new Avisynth+ r2728-

FFT3DFilter v2.4 (20170608)
- some inline asm (not all) ported to simd intrisics, helps speedup x64 mode, but some of them faster also on x86.
- intrinsics bt=0
- intrinsics bt=2, degrid=0, pfactor=0
- intrinsics bt=3 sharpen=0/1 dehalo=0/1
- intrinsics bt=3
- Adaptive MT settings for Avisynth+: MT_SERIALIZED for bt==0 (temporal), MT_MULTI_INSTANCE for others
- Copy Alpha plane if exists
- reentrancy checks against bad multithreading usage
Note: for properly operating in MT_SERIALIZED mode in Avisynth MT, please use Avs+ r2504 or better.

FFT3DFilter v2.3 (20170221)
- apply current avs+ headers
- 10-16 bits and 32 bit float colorspace support in AVS+
- Planar RGB support
- look for libfftw3f-3.dll first, then fftw3.dll
- inline asm ignored on x64 builds
- pre-check: if plane to process for greyscale is U and/or V return original clip
- auto register MT mode for avs+: MT_SERIALIZED

Previous versions by Fizick and martin53

[FranceBB]

Thank you for this update as well. ^_^

[Zetti]

sharpen - sharpening strength (default=0 - not sharpen)
good values about 0.3 to 1.0 (negative values results in reverse effect)
Is value 1.0 the highest or can it be set higher??

[StainlessS]

Is value 1.0 the highest or can it be set higher??

Perhaps you could try that yourself, and tell us all the result {In ordinary Sharpen(), 1.0 is [I think] the max}.

EDIT: You dont need to bother Pinterf with questions that you could easily verify yourself,
P's time is way more valuable than yours :)

[Wilbert]

Perhaps you could try that yourself, and tell us all the result {In ordinary Sharpen(), 1.0 is [I think] the max}.

EDIT: You dont need to bother Pinterf with questions that you could easily verify yourself,
P's time is way more valuable than yours :)
Please don't tell people what they are allowed to ask or not.

[Kurogane]

In the end how to load libfftw3f-3.dll?

How to install LoadDll?

[Zetti]

A posible problem with fft3dfilter and fftw lib:

https://forum.doom9.org/showthread.php?p=1856944#post1856944

[pinterf]

Perhaps alignment problems. Plus I would check if vzeroupper is properly issued before returning, it won't crash but is performance penalty when returning to an sse2 world. I'll check it.

[Groucho2004]

A posible problem with fft3dfilter and fftw lib:

https://forum.doom9.org/showthread.php?p=1856944#post1856944

I built FFTW with GCC 8.2 and it works fine with fft3dfilter. There are a number of guidelines here (http://www.fftw.org/install/windows.html) about building fftw with MingW (gcc).

For example:
--with-incoming-stack-boundary=2: compile FFTW assuming a 4-byte alignment. On win32, some versions of gcc assume that the stack is 16-byte aligned, but code compiled with other compilers may only guarantee a 4-byte alignment, resulting in mysterious segfaults.

[pinterf]

Checked, no problem on fft3dfilter side, allocations use fftw's own malloc, so this is o.k. Fftw is also checking the alignment and won't use simd code if the alignment is not sufficient.

[tormento]

@pinterf would you try to resurrect a new version of FFT3DGPU?

[pinterf]

@pinterf would you try to resurrect a new version of FFT3DGPU?
The source is not complete.
filtersps.h, filtersps.cpp, dxinput.h, dxinput.cpp is missing.

This is what I found:
Forum: https://forum.doom9.org/showthread.php?t=89941

Known sources of the plugin:

x86:
http://www.avisynth.nl/index.php/External_filters#Spatio-Temporal_Denoisers
http://www.avisynth.nl/users/tsp/fft3dgpu0.8.2.exe
http://avisynth.nl/users/tsp/fft3dgpu0.8.2.7z

And the latest one:
This one is non-existant (it is said to have the missing 4 files: filtersps.h, filtersps.cpp, dxinput.h, dxinput.cpp), tsp provided them (https://forum.doom9.org/showthread.php?p=1102862#post1102862) in an updated version:
http://www.avisynth.org/tsp/fft3dgpu0.8.2a.exe
but it disappeared from the site since then.

x64: (binary only, dll and hlsl)
http://avisynth.nl/index.php/AviSynth%2B_x64_plugins
https://www.mediafire.com/download/2chnt1jkwwm/FFT3DGPU_3-15-2010.rar
Comment:
The HLSL (shader program) file is edited from the original to adhere to pixel shader 3.0 syntax rules.
Please make sure to place the correct file in the same directory as the 64bit plugin. Compiled by Joshy D.

[tormento]

The source is not complete.
filtersps.h, filtersps.cpp, dxinput.h, dxinput.cpp is missing.
:(

In the mean time I have found the x64 you were quoting.

Lot of limitations with Prefetch, works with mode 3 only. Computer interface barely usable when running. Video memory, full all the time.

Anyway, that's it.

[tormento]

The source is not complete.
filtersps.h, filtersps.cpp, dxinput.h, dxinput.cpp is missing.
https://forum.doom9.org/showthread.php?p=1102757#post1102757

[pinterf]

https://forum.doom9.org/showthread.php?p=1102757#post1102757
:) As I wrote the file is not found on the server.

[Reel.Deel]

:) As I wrote the file is not found on the server.

fft3dgpu0.8.2a.exe:
https://www.dropbox.com/s/b25nik2r4qkujf9/fft3dgpu0.8.2a.exe?dl=1

:)


Edit:
Hi there pinterf. Yes, I've not been on on here as I use to but I still lurk :).

[pinterf]

Somehow I knew that you - like a good museum officer - will dig it out and appear soon :), thanks.

[Taurus]

Somehow I knew that you - like a good museum officer - will dig it out and appear soon :), thanks.
Sorry, can not resist.
This gave me a good laugh,
it almost saved my day.
:thanks: Reel.Deel for sharing
and :thanks: pinterf for your humoristic thanksgiving to the digger.

[pinterf]

Download FFT3dGPU 0.8.3 - pfmod (https://github.com/pinterf/FFT3dGPU/releases/tag/0.8.3)
Semi-experimental release - nothing important has been changed in the source. This build is giving me identical results to the old v0.8.2 (x86) but who knows.

At least we have a compilable codebase again, an x64 build which is surely compatible with AVS+ x64 structures, and finally the interface version of the plugin is V6 (for Avisynth 2.6.0.5- and Avisynth+).

You can find _no_ additional colorspaces yet.

An attempt to compile FFT3dGPU under Visual Studio 2017 with current Avisynth+ headers for x86/x64.

Changelog (pinterf)

**v0.8.3 (20181118)**
- project hosted on https://github.com/pinterf/FFT3dGPU
- source and project files updated to build under Visual Studio 2017 (15.9) using DX SDK June 2010
- Moved to Avisynth Interface 6 (v2.6) - using headers from the Avisynth+ project
- added x64 platform

**v0.8.2 (x86: 2006, x64: 2010)**
- Initial source: 0.8.2

[tormento]

Lol. I got in touch just today with TSP.

@pinterf do you still need sources or you found another way?

[pinterf]

Lol. I got in touch just today with TSP.

@pinterf do you still need sources or you found another way?
I've got all I wanted, thanks.

[pinterf]

Download FFT3dGPU 0.8.4 - pfmod (https://github.com/pinterf/FFT3dGPU/releases/tag/0.8.4)
All 8 bit color spaces, except RGB24 and RGB32. For RGB use planar RGB instead.

**v0.8.4 (20181121)**
- New color spaces besides YV12 and YUY2:
Y8, YV16, YV411, YV24 (all 8 bits), 8 bit Planar RGB (Avisynth+)
- FFT3dGPU filter registers itself as MT_SERIALIZED automatically for Avisynth+: no need for SetFilterMTMode.

**v0.8.3 (20181118)**
- project hosted on https://github.com/pinterf/FFT3dGPU
- source and project files updated to build under Visual Studio 2017 (15.9) using DX SDK June 2010
- Moved to Avisynth Interface 6 (v2.6) - using headers from the Avisynth+ project
- added x64 platform

**v0.8.2 (x86: 2006, x64: 2010)**
- Initial source: 0.8.2

[tormento]

Download FFT3dGPU 0.8.4 - pfmod (https://github.com/pinterf/FFT3dGPU/releases/tag/0.8.4)
Much better to split this in its own thread or create a new one.

Or not? :D

[almosely]

Hi,

I just migrated from AVS 2.6.0 x86 MT to AviSynth+ 2772 MT x64 and tested the version 2.5, 2.4 and 2.3 of your FFT3DFilter (with FFTW 3.3.8 AVX).

Do I have to change my colorspace (I am using YUV/YV12) or am I doing anything wrong, because the luma changes dramatically even without setting any value; the same applies for your filter within AVS 2.6.0. Till now I used FFT3DGPU 0.8.2 and there was everything okay; also with 0.8.4 seems everything to be alright.

1. Original
2. FFT3DFilter 2.5 (somewhat dithered, brightened and darkened or something like that; is there a problem with colorspace- or bit-depth-conversion?)
3. FFT3dGPU 0.8.4

These are the corresponding avs-lines:

FFT3DFilter(sigma=0.1, beta=1.0, bw=32, bh=32, sharpen=0.0, scutoff=0.27, plane=0, bt=3)

FFT3DGPU(sigma=0.1, beta=1.0, bw=32, bh=32, sharpen=0.0, scutoff=0.27, wintype=0, plane=0, bt=3, precision=2)

Histogram("luma")

[pinterf]

Hi almosely,

first, I noticed something. What can you see for this clip? On my machine it's like FFT3dGPU is keeping the left half of the picture untouched.
tested with FFT3DGPU 0.8.2 to 0.8.4

EDIT: my source was not mod4 width, seems that output is corrupted for such width values

Function Diff(clip src1, clip src2)
{
return Subtract(src1.ConvertBits(8),src2.ConvertBits(8)).Levels(120, 1, 255-120, 0, 255, coring=false)
}
c = any yv12 source (non-mod4-width)

clip1=c.FFT3DFilter(sigma=0.1, beta=1.0, bw=32, bh=32, sharpen=0.0, scutoff=0.27, plane=0, bt=3)
clip2=c.FFT3DGPU(sigma=0.1, beta=1.0, bw=32, bh=32, sharpen=0.0, scutoff=0.27, wintype=0, plane=0, bt=3, precision=2)
#clip11=clip1.Histogram("luma")
#clip22=clip2.Histogram("luma")
#StackHorizontal(clip1, c)
#Diff(clip1, c) # FFT3dFilter: OK
Diff(clip2, c) # FFT3dGPU: left half is untouched

[almosely]

Save the three images to your local drive. Open one in fullscreen and circle through the three. A difference between the original state and the one from FFT3DGPU is almost not existing, as it should be (no parameters were set). But the difference between FFT3DFilter and the other two screens punches you into your face - it's so big, no one can not see it instantly; and there shouldn't be any difference at all.

[pinterf]

almosely, you were right, you have a human eyesight comparable to an eagle's :), yes, those small dithering-like dots really exist.

Internally all computations are in 32bit float numbers. Integer values are converted to float, making all fft and other stuff, then the result is converted back to e.g. 8 bit integer.
Usually this latter conversion is done with proper rounding, typically: integer_value = truncate(float_value + 0.5).

FFT3dFilter is lacking this rounding, that's why we are seeing the luma difference a bit "busy" instead of a boring flat difference.

Could you please try this build?
https://drive.google.com/open?id=1Q-j7TgvCHYB95KEixDBiEfKn2PrebCVU

EDIT: this fix is also solving an issue - which I never understood - when FFT3DFiltering a simple BlankClip input results in more and more noise over time when bt=0 is used.
orig=blankclip(length=200,width=1920,height=1080,pixel_type="YUV420P8")
new=orig.FFT3DFilter(sigma=3,plane=4,bt=0,degrid=0,pfactor=0.0)
Subtract(orig,new).Levels(120, 1, 255-120, 0, 255, coring=false)

[almosely]

... and there is a very small amount of blurring too, within the created image by FFT3DGPU ;-) But nothing to worry about. BTW, although I see (and hear and smell etc.) extremely good, I encode never higher than 720p, because nobody is able (when really watching and not searching for insufficiency) to see the difference to higher resolutions in a moving image, if the viewing distance is chosen properly to the human eyes abilities (a 20 degrees angle between our face and the left+right borders of a screen); so I did also reduce the screen resolution of my smartphone from 2560x1440 to 1280x720p, because that is absolutely sufficient (although a very little bit of blurring is noticable because of the high contrast of displayed "still images" there) ...

Thank you very much for the explanation - I suspected something like that. But it's not only "busy", there's a variation of the luma values throughout the whole picture (darker areas are getting brighter and vice versa).

And thank you again, for the pre-build of FFT3DFilter. I will test it as soon as possible - I just uninstalled everything regarding my recent migration from AVS 2.6.0 (x86) to AVS+ (x64) in a moment of doubt and have to reinstall ;-) Oh, and I always resize/crop to get mod4 ;-)

[Groucho2004]

I just uninstalled everything regarding my recent migration from AVS 2.6.0 (x86) to AVS+ (x64) in a moment of doubt and have to reinstall ;-)You might want to have a look at the Universal Avisynth Installer from my signature.

[almosely]

I just wanted to add (edit) something to my lines before: If an oculist is attesting ones ability to see as 100% (sharp), than 1280x720p are 111% respectively. That's a fact nearly nobody is aware of (marketing sends regards). Therefore 100% are only 1160x652 (although the human eye is able to see a maximum of 2320x1304 under ideal conditions like still pictures, high contrast, dark environment, young age etc.); normally we just recognize 50%, which would be 596x336 ... And in addition to my earlier posting: Playing games in 720p is mostly satisfying too - therefore it's no problem at all to play triple-A-games with my old GeForce GTX 660 Ti (the last ones were Senua's Sacrifice or Fallout 4) :-)

I noticed your Universal AVS Installer before the very first installation of AVS+, but I decided to make a clean migration; so wether the old 2.6.0 nor AVS+ is installed right now (and no MPC-HC, AvsPmod, AVSmeter, vc_redist2017) :-( Thanks for your hint anyway! :-)

[Groucho2004]

I just wanted to add (edit) something to my lines before: If an oculist is attesting ones ability to see as 100% (sharp), than 1280x720p are 111% respectively. That's a fact nearly nobody is aware of (marketing sends regards). Therefore 100% are only 1160x652 (although the human eye is able to see a maximum of 2320x1304 under ideal conditions like still pictures, high contrast, dark environment, young age etc.); normally we just recognize 50%, which would be 596x336
Do you have links to scientifically sound research that would support your statements?

[almosely]

A few years back from now I purpously read a lot of stuff about human eye abilities and discovered all of that, step by step. But I had to calculate a little bit, to get the actual pendants in screen resolutions. There is a lot of information regarding that (e.g. at wikipedia) (I read within the german edition), like ...

Sehschärfe (https://de.wikipedia.org/wiki/Sehsch%C3%A4rfe)
Auflösungsvermögen (https://de.wikipedia.org/wiki/Aufl%C3%B6sungsverm%C3%B6gen)
Blickfeld (https://de.wikipedia.org/wiki/Blickfeld)
Der Gesichtssinn (https://www.controlling21.de/ergonomie/theorie/grundlagen/gesichtssinn.htm)
Werbung und Wahnsinn (http://www.heise.de/foto/artikel/Werbung-und-Wahnsinn-Wieviel-Pixel-braucht-der-Mensch-226854.html) (that one seems "dead", sadly; but I made a screenshot and saved the article locally): Screenshot (https://www2.zippyshare.com/v/IGao9AYq/file.html)
Fernpunkt (Optic) (https://de.wikipedia.org/wiki/Fernpunkt_(Optik))
Gesichtsfeld (Wahrnehmung) (https://de.wikipedia.org/wiki/Gesichtsfeld_(Wahrnehmung))
Wieviel dpi brauchen Druckdaten wirklich (https://www.safer-print.com/de/faq/In-Aufl%C3%B6sung-begriffen_Wie-viel-dpi-brauchen-Druckdaten-wirklich)
Schärfe (Fotografie) (https://de.wikipedia.org/wiki/Sch%C3%A4rfe_(Fotografie))

Maybe there are links to equally informational english editions of the articles; I hope so.

[Groucho2004]

Maybe there are links to equally informational english editions of the articles; I hope so.Thanks for the links. I was born and raised in Krautland, no problem with the lingo.

[almosely]

You're are welcome :-) But: If you don't believe in what I've said, then just try it for yourself. Calculate the minimun/optimum distance to your screen and watch the same movie(-scene) once in 720p and once in 1080p. I tested it many times with movies and never saw a difference while watching (not searching, nor pausing). To calculate that, measure your screen width and divide it by 2 (this is the opposite leg). Then take 10 degrees as the alpha angle (= half of the human "Gebrauchsblickfeld", maybe something like "used field of vision") and put that into a formula to calculate the adjacent side (=your optimum viewing distance) of a right-angled triangle. We have chosen the size of our livingroom-tv upon that; a bigger tv would be only stressful to ones eyes. Based on that, apple called his displays "retina" btw. (but meanwhile they are "jumping on the train" of resolution madness, too).

-edit-

And ... Therefore the optimal resolution of a flatscreen-TV for watching moving images should be 1280x720p, because it would be sharper than 720p interpolated to 1080p. 1080p is overkill, same as 4k - but three times 720 is, guess what, 4k (so with these displays, 720p will be just fine again, as it was, when flatscreens used to be "only" hd-ready). But, if it is necessary to focus on smaller areas of a screen, then a higher resolution makes sense. That explanes why the national broadcaster in germany do not use more than 720p too. Of course, one can spend money in pay-tv to get 1080p commercial breaks ;-)

-edit-

"4k - Make HD-Ready great again!" :-)

-edit-

The 20 degrees is a tricky one. As far as I remember, it's the area of our commonly used area of eye movement without moving our head. And something around 20-30 degrees is the area, our eyes are able to see all three, or at least two (red+blue) colors. And: Only exactly in the middle we are able to see 100% sharp. But I had to read everything again, to explain it exactly. I just memorized the 20 degrees as the main issue.

[wonkey_monkey]

Calculate the minimun/optimum distance to your screen

How do you do that? Surely any such distance is arbitrary. Most people seem to like to sit a lot closer to their TVs (or have a bigger screen) than I do, for example. That's just their preference.

[almosely]

Maybe they need glasses? ;-) Okay, for real; I don't know why they are doing this. It's not natural, but who cares nowadays? The bigger, the better. Today it seems natural to be unnatural. That explains a lot of drawbacks of our time period. One thing is for sure: One do not get, what's going on (onscreen), not entirely; the closer, the less. But inattention and incomprehension seems to be viral too.

[ChaosKing]

When your tv is bigger than yourself https://www.youtube.com/watch?v=3BJU2drrtCM

[almosely]

Ah, great ;-) So sweet to explain the mechanism of a flip-book (about which nearly everybody was aware of as a eight year old, 35 years back in time) with the help of a modern, interstellar-sized prick-replacement; do the kids nowadays know about flip-books at all?

Something I did just remember regarding apples "retina" displays. While I was researching all of that a few years ago I found out, that apple must have taken the golden selection into consideration. That conclusion came after "reengineering" the correlation between apples screen sizes and resolutions of some of their mobile devices regarding vertical display orientation. They did a great job back than, honestly.

Goldener Schnitt (https://de.wikipedia.org/wiki/Goldener_Schnitt)

[almosely]

Could you please try this build?
https://drive.google.com/open?id=1Q-j7TgvCHYB95KEixDBiEfKn2PrebCVU

EDIT: this fix is also solving an issue - which I never understood - when FFT3DFiltering a simple BlankClip input results in more and more noise over time when bt=0 is used.

Well done! :-) I just reinstalled AVS+ (x64) and corresponding tools and it seems to work perfectly great - I made a quick comparison within AvsPmod GPo and a short testrun through AVSmeter; till now, everything is fine and a few frames faster (0.xx). Thank you!! :-)

[pinterf]

New release.
Download FFT3dFilter v2.6 (20190131) (https://github.com/pinterf/fft3dfilter/releases/tag/v2.6)
almosely, thank you for the report, a bug which was obvious but nobody noticed in the last decade.
Fix: Proper rounding when internal 32 bit float data are converted back to integer pixel values

[StainlessS]

Oh WOW, thank you.
Guess no bodiy noticed cos we is all so easily pleased :)

[FranceBB]

Thank you for the updated version. ^_^

[ajp_anton]

@almosely

So you're claiming that 720p at 20° is the ideal for TV or a smartphone. I disagree, especially with your comment on this being just marketing.
First, you yourself mentioned well over 1080p for still images. Smartphones are very much still images, and so are a lot of parts in a lot of movies.
Second, you don't need to have the whole screen in your perfect vision at the same time. Your peripheral vision also exists, and you can move your point of interest to a different part of the screen.

edit: removed the gigantic quote.

[almosely]

@ajp_anton

Everything you described, I described before. So, what is the extra meaning of your post?

[almosely]

Okay, because it's an affair of my heart, I will try to make that more understandable.

For example, open a .pdf document on your smartphone. I am assuming you own one with a WQHD (1440 x 2560) resolution or higher. Look at what you see: It's crystal clear, but can you actual read it (without your eyes being strained)? Most probably you are going to zoom in a bit. Now do the same with 720p: It is not as crisp and sharp, with a 100% zoom, but that is not relevant at all. You are going to zoom in too. What are you seeing than? Everything without any difference but some little aliasing. If you are paying attention to the content only, you will not pay any attention to that little bit of aliasing at all. You achieve the same result and satisfaction with both resolutions - if you do not push your nose against the display and watch out for fuzziness and tell your buddy next to you about that. It does not have any relevance in real life at all. Same goes for pictures taken by your smartphone camera. There is absolutely no need to be able to see the border of the glasses of the person within the picture 100% crisp because one focuses on the essential content of the image, the composition, the light, the colors, facial expressions and so on. Quickly show any picture to your buddy - he/she will never watch out for sharpness, only pay attention to what's going on substantially. To sit there and compare the capability of one's display againgst another one's is only the "need", that marketing is making up artificially and indoctrinating that into your mind.

The same goes for movies. There are simply no "lot of parts in a lot of movies which are still images". Almost every scene of a moving picture is what?: Moving. Even it is a still life or a quiet landscape scene, there's almost every time a little zooming or panning. Additional to that there is reflection on the screen, surrounding lights or movement next to the screen or even noise or the movie sound track playing - everything is diversion and engaging your mind. You simply pay not that attention to the resolution - normally, but only if someone is telling you to do so (marketing, or marketing speaking through the mouth of your buddy) by disregarding other (essential) things. Movies are telling stories, and that with the help of a lot of different tools - none of them is screen resolution. This is why science (to read within one of my mentioned articles earlier) defines our ability to observe something with our eyes in general somewhere way below 100% (720p=111%!) (exactly at 2 arc minutes, which would be somewhere around 600x340 pixel). It's psychological and physical dependent. The need to focus on the screen resolution is not born within us but by marketing and selling intentions. Actually to focus on that is hurting us badly in a lot of ways. Not only the pollution resulting in constantly buying new devices and throwing the "old ones" away for nothing that matters, as well as the chronological, physical and mental workload put into earning constantly more money to be able to buy those needless things, which is injuring everybody (ourselves, people we should pay more attention to, social relationships etc.), are the results, but the quality of movies itself. Yes, by ridiculously spending our attention to that, movie makers (respectively the big companies, which are only interested in making more money) create their movies in other ways than before. Look at the blockbusters of the last years. They are based upon insane and stressful pace all over the place, nearly in every area of the scene something is moving around, doing something unessentially, the dialogues are dumb as hell, the plot is ... yeah the plot is to show something and mostly not more than that ... and everything is feeding the "need" to look at small areas of the screen instead of the whole picture, a real "scene", telling something essential. When you are talking to somebody, are you paying attention to what he/she is telling you by his/her voice, mimics and gestures or are you constantly looking away, watching out for things going around? Does that imagination ring any bell within yourself? We should pay attention to what really matters, but that has changed a lot. People are getting less focused on essential things, because they are constantly told to do so - by who?: Marketing. And one not negligibly reason for that is this need to pay attention to the resolution of a screen. That's insane. Yourself are telling me to be "not in need to have the whole screen in your perfect vision at the same time, because your peripheral vision exists and you can move your point of interest to different parts of the screen". That is exactly what I define as a big damage/madness of our time, caused by marketing and screen resolution mania. Your point of interest has changed (or is changing constantly) - you are saying that without beeing aware about the consequences followed by that. This is what I mean.

I mentioned earlier that Apples Retina-displays are based upon what I am trying to explain here. Has anybody ever challenged the quality of those displays when they were introduced back in the years? No! Everybody was praising them the hell out of it. Does anybody remember the marketing claim "sharper than reality" which was introduced with full hd? It's simply true and though totally needless. Marketing is audaciously using the plain truth as advertising and makes folks believe nature is wrong or being better than nature! Don't you realize the tremendous mind fuck behind that? People are marketing's helpless sheep, believing what they are told to believe and - to add insult to injury - they are defending those harmful fairy tales and trying to convert those with open eyes and mind to the blind mob. And the very crux of that is, that the ones causing this misery and damage (people who are creating that marketing crap) are hit by the aftermath to the same scale as anybody else.

Maybe it helps to realize/remember that x264 takes that into consideration too and everybody seems to be fine with that (because there's no marketing interest of pointing ones nose straight to the differences between raw and encoded material, which are there (very obvious ones), but not realized while watching and paying not attention to it). Similar goes for mp3/aac vs wav/flac etc and dts is not necessary at all too (btw I am blind testly able to hear differences between mp3 320/v0 and wav and also differences between aac v5 and wav; only aac v6 is transparent to my ears; of course, I am only able to recognize that in some rare situations; with movies, aac v4 is fine and walking outdoors mostly too, but not at night, when its quiet - then I use aac v6).

[FranceBB]

because there's no marketing interest of pointing ones nose straight to the differences between raw and encoded material

No, it's because of the bandwidth; we can't air on an high enough bandwidth, especially 'cause the encode has to be live, so it's not going to be very complex due to the latency required to air the signal with an acceptable delay.
Keep in mind that we have a delay that goes from 1 second to maximum 20 seconds (but it's not used except on very rare occasions).
As to the offline encoding, such as bluray disks delivered to the people, the bandwidth goes up, but we're still limited by the specifications and the capacity of the disk.
Even if you take professional disks (not for the public) to store mezzanine files, they are quad layer and with an higher speed for write/read access capable of supporting higher bitrates, but they are still limited.
For instance, back in the days, Sony XDCAM Disks were 128 GB capable of reaching 50 Mbit/s for the video and other spare mbit/s for the audio up to about 60 Mbit/s.
You basically said that issues are there but nobody cares about compressed sources, but I disagree, as there are people who do notice them and the fact that we're not releasing lossless sources to people is simply because of the bitrate required: there's NO WAY to deliver a lossless file via Satellite (too much bandwidth, too expensive), nor via IPTV (nobody has the bandwidth available to stream such a content), nor via Disks (seriously, even if the episode of a series of few minutes would fit in an 128 quad layer disk, it wouldn't have the required reading speed to access all the data and playback smoothly).
In other words, we don't use lossy codecs because of marketing, but because infrastructures and practical usage.

I am blind testly able to hear differences between mp3 320/v0 and wav and also differences between aac v5 and wav; only aac v6 is transparent to my ears; of course, I am only able to recognize that in some rare situations; with movies, aac v4 is fine and walking outdoors mostly too, but not at night, when its quiet - then I use aac v6).

The filterbank polyphase in the psychoacustic model of mp3 is no match with the AAC, which has a completely different (and better) implementation, anyway audio is treated with a bit more accuracy compared to video due to the technical ability to have rather small files compared to lossless videos: it is actually possible to deliver to users contents with a good fidelity compared to the original master file on Bluray Disks, but not so much on Satellite, IPTV and OTT as we're forced to use lossy codecs like AC3 and AAC with bitrate between 192 kbit/s and 384 kbit/s due to bandwidth limitation (the extra bits saved on the audio are used for the video which is generally more relevant to the user).

Movies are telling stories, and that with the help of a lot of different tools - none of them is screen resolution.

Yes, movies are telling stories and some of them are really moving and last forever.
Saying that a movie is bad because it was shot in the 80s would be like saying that paintings before renaissance were useless due to the lack of perspective.
Anyway, there's nothing wrong in improving and using the technology we have to make things better.

"Yes, but we're not able to see the difference anyway, so why bother?"
Although that it's arguable, I can make you an example that isn't.
There are cameras nowadays capable to shoot at low-resolutions but at a very high framerate, something like the MIT Media ones, capable of recording at 1,000,000,000,000 frames per second through their special technique.
Of course, the human eye is not able to see so many fps, but it's actually useful for science purposes. http://web.media.mit.edu/~raskar/trillionfps/
http://web.media.mit.edu/~raskar/trillionfps/tomato_lightprop_s.gif

Not only the pollution resulting in constantly buying new devices and throwing the "old ones" away for nothing that matters, as well as the chronological, physical and mental workload put into earning constantly more money to be able to buy those needless things, which is injuring everybody (ourselves, people we should pay more attention to, social relationships etc.), are the results, but the quality of movies itself.

What? Nothing stops you from not buying a new device or a new TV. If you wanna stay - for instance - on an SD TV connected via SCART or Composite cables, you are totally free to do so. TV channels not only still air contents in multiple resolutions (including SD), but HD/FULL HD channels can be decoded by an HD/FULL HD decoder and downscaled to SD if your TV only supports that. (For 4K it's a bit different, 'cause decoders don't offer the option to downscale due to the complex conversion that would need to take place in converting an HDR to SDR in case of HDR contents).

every area of the scene something is moving around, doing something unessentially, the dialogues are dumb as hell, the plot is ... yeah the plot is to show something and mostly not more than that ...

There aren't just action movies and silly comedies these days, though.
The fact that the vast majority of movies is focused on special effects or other things 'cause it *has to look good* has nothing to do with resolutions, it's just because the majority of people like, let's say, action movies with no dialogues and just a lot of shootings and bombs and other special effects, but that has nothing to do with resolution, it would have been the same for SD contents.
I do wanna say one thing, though, and it's about HDR.
I remember that when they introduced Dolby, everybody was exasperated to have effects all over the channels, even when it wasn't needed at all and it ended up "disturbing" the view rather than "enhancing" it, but they eventually got it right later on.
Same thing occurred when 3D was first introduced; I remember people trying to make everything getting into your face, it HAD to come toward the user, even when it wasn't needed at all and it was rather disturbing, but they eventually got it right later on.
Nowadays it's the turn of HDR, in which everybody seem prone to exasperate the colors and the contrast, producing encodes very different from the original natural light of the shot, just because "it has to look good and be stunning to people".
Many people seem to be misusing the HDR, by using it to make images "beautiful" rather than as close to the original as possible, but they will eventually get it right.
For instance, we used PQ when we had to record and encode a show at the theatre, 'cause in theatres the show it's all about people sitting there, it's not for TV, so we had a very dark setup overall but we also had lights in the background that were switched on and off at different times. In the old SDR world, setting the camera with the right black level to get the actors on the stage would have led to severe clipping on the lights in the background, but with PQ we were able to get them both right.
I think that THIS is how people should use HDR and what HDR it's meant to and I think people/colorists will eventually get it right in the future.

"not in need to have the whole screen in your perfect vision at the same time, because your peripheral vision exists and you can move your point of interest to different parts of the screen". That is exactly what I define as a big damage/madness of our time

No, it's really up to the content creators, to be fair.
In TV we use 4mm lens that are always on focus on everything and that are capable of keeping everything in focus, regardless of what you are pointing you camera at.
In movies they use different type of lens and it's the director of photography who chooses what has to be on focus and what doesn't.
For instance, in movies, sometimes they choose to recreate things that are "errors" just because it's the feeling that it's meant to be transmitted to people.
One of the common example is whenever you have to shoot a scene in which there's a car approaching you frontally during the night/evening and then stops and is turned off.
Generally, the director of the photography asks you to shoot it the way that it has a thin horizontal line that goes from left to right and then disappears once the car it's switched off.
That line it's actually caused by the lights and although it can be avoided, most people imagine it that way and it's more common to do it this way.
But again, it has nothing to do with resolution.

[almosely]

As to the offline encoding, such as bluray disks delivered to the people, the bandwidth goes up, but we're still limited by the specifications and the capacity of the disk. You basically said that issues are there but nobody cares about compressed sources, but I disagree, as there are people who do notice them ...

1280x720 with average P-quantizers of 18 (x264) are enough for the common end user. Of course, exceptions does make sense everywhere.

Anyway, there's nothing wrong in improving and using the technology we have to make things better.

I agree completely.

Of course, the human eye is not able to see so many fps, but it's actually useful for science purposes.

The end user is not a scientist in general. I was referring to that only.

Nothing stops you from not buying a new device or a new TV.

You severely underestimate the power of marketing's omnipresent influence. There are only a very few people able to resist, if at all (including me, of course). Free will is a hoax, regarding that.

The fact that the vast majority of movies is focused on special effects or other things 'cause it *has to look good* has nothing to do with resolutions ... it would have been the same for SD contents.

Do you seriously believe one would like to sit 1,5 meters away from a 85 inch display in sd resolution, looking what's going around in every corner of the screen? Yes, it has nothing to do with resolution if the screen size would be as small as times ago, because we do not see the differences anyway. But higher resolutions gave birth to tvs in the size of a landscape where everywhere everything is sharp. With that you can easily focus on small parts of the scene, sitting very close to the screen. Look to the upper left side, focus on that, and you will miss, what's going on at the opposite. Movies are'nt meant/built to tell multiple stories at one time; they are'nt landscapes, no movies within movies; they need ones focus on the whole scene, otherwise comprehension is lost. That results in people becoming less interested in "complex" stories - they are simply not able to fully recognize a story anymore. That changed they way, many movies are produced nowadays, especially the blockbusters. The focus shifted from mental content to visual because of that, sometimes up to trivial pure visual springkling. And that mirrors in often found common behaviour, especially of the youth nowadays - they are'nt interested in focussing because they simply can't (it's not their fault) - and marketing keeps on telling them, that everything is alright with that, get even bigger screens and higher resolutions, become paying zombies.

Nowadays it's the turn of HDR, in which everybody seem prone to exasperate the colors and the contrast, producing encodes very different from the original natural light of the shot, just because "it has to look good and be stunning to people".

I know. I used HDR (with DCIMs) a lot while studying design (and marketing, sadly, retrospective) around 12 years ago. I saw every mistake possible with that technique. It's a great thing, but has to be used with care.

[ajp_anton]

Your posts are just so full of things I disagree with, I'm not going to bother answering it all. I'm just going to say that
1280x720 with average P-quantizers of 18 (x264) are enough for the common end user. Of course, exceptions does make sense everywhere.
Who are you to decide what's enough for the common end user? The movie's story doesn't change with resolution, but the experience does change when you see everything clearer. Same with your PDF example earlier, you even said that the aliasing is visible, which kind of proves my point. Why even use 720p, as that's overkill for "being able to read"? Because Higher res, sharper pictures and less aliasing is just more pleasant for the eyes.

[almosely]

I could be anybody, but time is limited and, of course, there's fate. There's plenty to reflect on by now. Thank you, pinterf, again, for spending your time and skills to give us this gift. I am going to use "your" fft3dfilter nearly everytime I encode :-)

[MeteorRain]

Bump for potential bug

https://github.com/pinterf/fft3dfilter/blob/master/fft3dfilter/fft3dfilter_c.cpp#L109

Line 69 and Line 89 are factors for sharpen only and dehalo only.

Sharpen only factor
1 + sharpen*wsharpen[w]*sqrt( psd*sigmaSquaredSharpenMax/((psd + sigmaSquaredSharpenMin)*(psd + sigmaSquaredSharpenMax)) )

Dehalo only factor
(psd + ht2n)/((psd + ht2n) + dehalo*wdehalo[w] * psd )

Line 109

Sharpen and Dehalo factor

WienerFactor *=
1 + sharpen*wsharpen[w]*sqrt( psd*sigmaSquaredSharpenMax/((psd + sigmaSquaredSharpenMin)*(psd + sigmaSquaredSharpenMax)) )
*
(psd + ht2n)/((psd + ht2n) + dehalo*wdehalo[w] * psd );


However in sharpen_C

sfact =
(1 + sharpen*wsharpen[w]*sqrt( psd*sigmaSquaredSharpenMax/((psd + sigmaSquaredSharpenMin)*(psd + sigmaSquaredSharpenMax)) ) )
*
(psd + ht2n) / ((psd + ht2n) + dehalo*wdehalo[w] * psd );


I haven't dig too deep, but this 2 pieces of code look fishy to me. Can you confirm if one of them was wrong?

FYI: I'm refactoring and reorganizing this filter.

[pinterf]

Bump for potential bug

https://github.com/pinterf/fft3dfilter/blob/master/fft3dfilter/fft3dfilter_c.cpp#L109

Line 69 and Line 89 are factors for sharpen only and dehalo only.

Sharpen only factor
1 + sharpen*wsharpen[w]*sqrt( psd*sigmaSquaredSharpenMax/((psd + sigmaSquaredSharpenMin)*(psd + sigmaSquaredSharpenMax)) )

Dehalo only factor
(psd + ht2n)/((psd + ht2n) + dehalo*wdehalo[w] * psd )

...
FYI: I'm refactoring and reorganizing this filter.
Unfortunately I cannot help, did not dig into the algorithms. I noticed during an mmx asm to sse2 intrinsic rewrite that the C code was a bit different (afaik there was a small sigma addition somewhere)

[MeteorRain]

I shot a quick email to the original author and see if I can get something.

I finished organizing all Apply*3D*() family and all their SSE2 counterparts. bt=5 shows a significant speed improvement (85539ms -> 13925ms for a short clip). Might start working on AVX part later.

Hopefully I can get an answer soon.

[pinterf]

Nice. You can check VapourSynth solution, there had been a refactor as well.
https://forum.doom9.org/showthread.php?t=175199

[MeteorRain]

Thanks! That's ... a lot of diversity in this community.

Hopefully I will finish it with most of the fixes in the other repo and make it a dual interface.

[Myrsloik]

Thanks! That's ... a lot of diversity in this community.

Hopefully I will finish it with most of the fixes in the other repo and make it a dual interface.

Insanity! The Avisynth developer code dictates that you must create yet another incompatible fork!

[MeteorRain]

Hi,

It seems, that Sharpen_C code is correct, but

pair of brackets are missed in ApplyWiener2D_C code for non-zero both
sharpen and dehalo.

It is a bug.


So as I expected. Will include that fix also.

[MeteorRain]

(pfactor=1.0, bt=0) seems to be broken. Can you confirm if that's broken on your side?

[MeteorRain]

https://github.com/pinterf/fft3dfilter/blob/master/fft3dfilter/FFT3DFilter.cpp#L2811

Is coverpitch * pixelsize supposed to be coverpitch?

[pinterf]

https://github.com/pinterf/fft3dfilter/blob/master/fft3dfilter/FFT3DFilter.cpp#L2811

Is coverpitch * pixelsize supposed to be coverpitch?
Yes, it worked only for 8 bits! Seems I have not tested all parameter and bit depth combinations.

[pinterf]

(In general) When you make filters that work for both VS and AVS, parameter handling is different when applied on different bit depth.
Conventionally VS filters require to scale parameters to the appropriate bit-depth at the scripting side. In my AVS filters the same parameter magnitudes will work similarly for all bit depths. E.g. sigmas, smin, smax (https://github.com/pinterf/fft3dfilter/blob/master/fft3dfilter/FFT3DFilter.cpp#L585).

[MeteorRain]

Yea I saw that when I refactor those part.

Now I can almost ensure it's very bit identical to the before-refactoring, except for some minor difference (such as different order of float additions and subtractions). Moving from ancient C++ to modern C++ really gives a fresh breeze. I have wiped the original caching system and made a LRU template caching, and refactor bt 2..5 into a single block. Should be much easier to maintain now. Duplicate code in Apply family are all eliminated with a more functional programming structure (lambda callback). This is the first time I'm writing C++ code in Ruby style, so will see how it goes.

Next step is to finish the AVS+VS dual interface, then I'll move back to LSMASH-Works Linux porting work, and hopefully I can get it up and running on Debian very soon.

[real.finder]

maybe parameters scale parameter like paramscale in MaskTools2 will be a good idea, with "none" by Default in vs and Default "i8" in avs

[MeteorRain]

I think the rule of thumb is -- the same clip in different depth with the same parameters should behave similarly.

[real.finder]

I think the rule of thumb is -- the same clip in different depth with the same parameters should behave similarly.

I think that too, but it's too late for vs (or all vs scripts must update for that)

so paramscale will be balanced solution for both, and it's also useful in debugging

also sigmas, smin, smax etc... should be float parameter instead of int (if it not float already) and if the old filter/plugin parameter was int, then it better to have dual filter signatures (https://forum.doom9.org/showthread.php?p=1902809#post1902809)

[MeteorRain]

Since this refactor is going to be a big change, I'm going to ignore full compatibility and try to do things right (at least right according to my mind).

That includes removing of plane / multiplane parameter and use the easier [y/u/v]=[3/2/1] convention, removing YUY2 colorspace support, and a few fixes that might not give bit identical result. I also plan to add a set of window parameters according to my personal need, so only pixels inside window is processed.

[pinterf]

Forget dual filter signatures please, it's a hack, separate the filter dlls properly, it is just conservating the disorder we see sometimes posted as heap of dlls from the last century.

[real.finder]

Since this refactor is going to be a big change, I'm going to ignore full compatibility and try to do things right (at least right according to my mind).

That includes removing of plane / multiplane parameter and use the easier [y/u/v]=[3/2/1] convention, removing YUY2 colorspace support, and a few fixes that might not give bit identical result. I also plan to add a set of window parameters according to my personal need, so only pixels inside window is processed.

Forget dual filter signatures please, it's a hack, separate the filter dlls properly, it is just conservating the disorder we see sometimes posted as heap of dlls from the last century.

then in these cases, function name should be changed, FFT3DFilter2 or Neo_FFT3DFilter

[pinterf]

Yes, if there would be way too different parameters and behaviour then a name separation would help. Like masktools(2), mvtools(2)

[MeteorRain]

I already changed it to Neo_FFT3D since I forgot to mention that. The original filter was written in 2004/2005 where WinXP and YUY2 were mainstream support. I think now it's a good time to move on. Although there are incompatibilities, it's still pretty easy to move to the new filter with minor changes to scripts.

[FranceBB]

I already changed it to Neo_FFT3D since I forgot to mention that. The original filter was written in 2004/2005 where WinXP and YUY2 were mainstream support. I think now it's a good time to move on. Although there are incompatibilities, it's still pretty easy to move to the new filter with minor changes to scripts.

I just popped up out of nowhere to say that although nobody uses 8bit 4:2:2 interleave (YUY2) now (since Avisynth has been having yv16 support for years) and I'm totally in favor of removing it if it's an hassle to support, but the same isn't true for XP as there are still people using XP, so unless it's impossible, I would beg you to keep XP support in the new Neo_FFT3DFilter as well. :)

[StainlessS]

I would beg you to keep XP support in the new Neo_FFT3DFilter as well.
+1 on that.

[MeteorRain]

WinXP has been EOL for long, so I'm not going to keep a copy myself to install it. I have no problem if someone would point out a fix for any potential issue and I can just merge or patch it.

On the other hand, the old filter still works despite some small issues which I pointed out and can be fixed quickly. If XP users would like to use it, they can simply use the original one.

Again, I won't deliberately remove XP support to make it unusable. Its unsupported in the sense of I'm not going to test it during my development.

Also you may notice that the AVX that I'm planning to add is not supported in XP anyway.

[StainlessS]

Again, I won't deliberately remove XP support to make it unusable. Its unsupported in the sense of I'm not going to test it during my development.
Understand you position entirely, thanks very much, [and thanks for not being like that XP hatin' Myrsloik, hangin's too good for that sort :) ]

[FranceBB]

WinXP has been EOL for long
Well, updates were kept up and running for Windows XP Embedded and all the other version like POSReady etc 'till April 2019 and were further extended by Microsoft 'till June 2019 when they dropped support for everyone but companies that were willing to pay 15'000 bucks every year for the Premium Extended Support. I really wished they offered an option to pay for them for private users as well, 'cause I would have definitely bought the Extended Support past July 2019 if it had an affordable price like $50 a year...

I'm not going to keep a copy myself to install it
Fair enough.

I have no problem if someone would point out a fix for any potential issue and I can just merge or patch it.
Oh, that's great. Thank you. :D

I won't deliberately remove XP support to make it unusable. Its unsupported in the sense of I'm not going to test it during my development.
Also you may notice that the AVX that I'm planning to add is not supported in XP anyway.
Ok, that's fine, I guess. Beside, I'm pretty sure there will be people that will test it on their own XP machines for you once you'll release it and they'll eventually point out errors, so it's fine. :)
AVX won't work of course, but I don't think you'll remove SSE code either, so I think XP users are gonna be fine.
The idea would be to have the filter to check for the CPU instructions and use the available assembly optimizations or eventually make them user-available with an opt string parameter; that would work as well.
The only think I'm asking, though, in all this, since it has to be compiled, is to compile it two times (just like Ferenc does): one with whatever you want (either visual studio or Intel Parallel Studio o Clang, whatever) for modern OS and the other one with Visual Studio 2019 targeting v141_xp. If it's not too effort, please just make sure that it does compile successfully. If it does, then it's probably going to run and it's gonna be fine for us.

Sorry for bothering you with this, but I really hope you can understand. :)

Thank you in advance,
Frank.




Side note:

Understand you position entirely, thanks very much, [and thanks for not being like that XP hatin' Myrsloik, hangin's too good for that sort :) ]

Yeah... Myrsloik... I remember when I asked for ffms2 support and other people did too and he then updated the initial post with

TRIGGER WARNING

The posts contained in this thread may be very upsetting for users of old operating systems, old cpus and stupid people

And I was like "Oh, c'mon..." :(

[videoh]

You're a hero for me, Frank, but I agree with Myrsloik. For what it's worth, LOL. We should ask Danila. ;)

[Stereodude]

...We should ask Danila. ;)
No we need to get him and Katie into the same thread and watch the positive feedback loop spiral out of control. :devil:

[MeteorRain]

FranceBB, I'm not particularly sure if anyone would want to continue to use WinXP for this particular work. For embedded environment I can understand that, but for daily use I can't. Win7 however, is much more reasonable to use as a daily drive. But still even Win7 may lack a few modern syscalls (APIs) and could potentially make things harder.

SSE code won't get removed of course. CPU flag based SIMD route selection is already there. My concern is I'm not sure if the xp targeted compilers would complain anything. I may give it a try very soon.

[FranceBB]

FranceBB, I'm not particularly sure if anyone would want to continue to use WinXP for this particular work. For embedded environment I can understand that, but for daily use I can't. Win7 however, is much more reasonable to use as a daily drive. But still even Win7 may lack a few modern syscalls (APIs) and could potentially make things harder.

Yes, I know, which is why I have Fedora (Linux) on my computer and a Windows 10 partition for work-related stuff as well as physical workstations with Win10 at work, however... I do keep a few VMs on my PC: one of them is with my original installation of Windows98SE 'cause it's been my very first operating system, the one I learned to format and re-install and I kinda got attached to it. I never use it, but, once a year, when it's its "birthday", I fire it up, I look at it, I open a few old programs to check that they're still working, I think about the good old days and then I shut it down for the rest of the year 'till the next year comes.
For XP however, I do have a partition with it and I've been part of a group called "Windows XP Forever" for quite some time (and I still am). We're the ones working at the "One Core API (https://github.com/Skulltrail192/One-Core-Api)" together with Samuel (SkullTrail), Dibya, Peter, Henry, Svyatpro, and so on. One Core API (https://github.com/Skulltrail192/One-Core-Api) is basically a set of APIs which have been backported from newer OS to Windows XP. We do it on our spare time and just for fun as we all love our beloved OS, but of course we all have Win10 as well.
If anyone is curious: GitHub Development Link (https://github.com/Skulltrail192/One-Core-Api) - GitHub Binaries Link (https://github.com/Skulltrail192/One-Core-API-Binaries) - Website (outdated) (http://shorthornproject.com/)

If you were wondering what does FranceBB do in his spare time (beside encoding), well... this is pretty much it xD

I think that I am, in some way, for some aspects, a person who's fascinated about the new technologies but also, at the same time, the younger version of Manolito hahahahahaha

No we need to get him and Katie into the same thread and watch the positive feedback loop spiral out of control. :devil:

lol... Out of curiosity, but... do you have the same impression that we're all dudes except for Katie? Like... is Katie Boundary the only woman in the whole Doom9? xD
Like... does encoding only attract dudes? xD


SSE code won't get removed of course. CPU flag based SIMD route selection is already there.

Perfect. :D

My concern is I'm not sure if the xp targeted compilers would complain anything. I may give it a try very soon.

Well, I guess we'll see soon, then. :)

[Stereodude]

FranceBB, I'm not particularly sure if anyone would want to continue to use WinXP for this particular work.
They're just very patient and aren't tempted by the idea of running hardware that's 100x faster.

lol... Out of curiosity, but... do you have the same impression that we're all dudes except for Katie? Like... is Katie Boundary the only woman in the whole Doom9? xD
Like... does encoding only attract dudes? xD
Yes, I expect this place has an extremely heavy bias toward individuals with a Y chromosome. I'm not sure that Katie is actually a girl, but that's gotta be the least significant thing about Katie's unique status on the forum.

Maybe we need to start an initiative?

#learntoencode / #girlswhoencode

[videoh]

the younger version of Manolito Not something to boast about! :p

So it's like collecting antiques. You'd never drive drywall screws manually, but the old tools sure are cool.

[Stereodude]

Not something to boast about! :p

So it's like collecting antiques. You'd never drive drywall screws manually, but the old tools sure are cool.
Drywall?!? They're still using lath and plaster! :sly:

[videoh]

Good point!

[feisty2]

the younger version of Manolito hahahahahaha

whos manolito?

[Atlantis]

Please, where is this Neo_FFT3D to try? I use bt=5 a lot and if there is speed improvements, it will be great.

[MeteorRain]

https://down.7086.in/AviSynthPlus%20Filters/neo-fft3d-alpha~20200328.zip

As I haven't finished all the work, this is considered only alpha / early dev version. Use at your risk.

neo_fft3d(..., y=3, u=3, v=3)

[pinterf]

XP does not support AVX so it is not even reported as valid processor flag in GetCPUFlags.

[tormento]

As I haven't finished all the work, this is considered only alpha / early dev version. Use at your risk.
Any difference in syntax and/or added switches?

[MeteorRain]

Yes, as I said,

neo_fft3d(..., y=3, u=3, v=3)

y u v being the plane to process, 3=process, 2=copy, 1=undefined content. Alpha channel, if there's any, will be copied regardless.
The old "plane=" and "multiplane=" options are removed.
All other options remain the same.

bt = 3 / 4 / 5 should give slightly different result due to rounding errors.
For bt=3, the constant 0.33333333333 will be replaced by 1/3 in current or future version.
For bt=4 / 5, an addition of a,b,c,d,e will be performed as (a+b)+(c+d)+e instead of (((a+b)+c)+d)+e.
In all cases, rounding error should be minimal to observe.

[sl1pkn07]

whos manolito?

https://forum.doom9.org/member.php?u=41597

[StainlessS]

whos manolito?
Bit of a cowboy really.
He got a sister Victoria who is married to a gringo big John Cannon,
and they live together on a fair size ranch with Victoria's stepson {Billy} Blue Boy.

Family Photo (from the right, clockwise):
https://i.postimg.cc/7ff1ZtJg/Mani-Famility-Photo.jpg (https://postimg.cc/7ff1ZtJg)

Manizorgsky, Pictures at An Exhibition:- https://www.google.com/search?q=high+chaparral&tbm=isch&source=iu&ictx=1&fir=qUEJu_9RsyTtjM%253A%252CBAAzhHNPWFEdmM%252C_&vet=1&usg=AI4_-kSAL9PCfwZ7nqTaUahXB3Ytp66KiQ&sa=X&ved=2ahUKEwiKrvLV_L3oAhULUhUIHbMcA00Q_h0wInoECAkQDQ#imgrc=qUEJu_9RsyTtjM:

[MeteorRain]

@Atlantis
Good news for you, Apply3D5 (the main function used by bt=5) is a few times faster than the pure C code.
I've measured Apply3D5 9.3 seconds of a 45 seconds test, down from 90 seconds of a 130 seconds test.

I also added multi-threading to it, so under MT prefetch 4 it's running event faster -- though up to 2x the speed due to locks.

[hello_hello]

They're just very patient and aren't tempted by the idea of running hardware that's 100x faster.

I doubt many people refrain from upgrading their computer because it won't run XP, or because there's no drivers for newer hardware, but I suspect there's many who don't see much point spending money to upgrade Windows on an old PC.

Do most household only have a single PC? There's four used regularly here, one of which is an old Pentium 3.

I've procrastinated replacing my PC because I can't bring myself to install Windows Spyware 10, and switching to Linux would require more effort and a learning curve, although as Linux Avisynth appears to be a thing now it's probably time, but this PC will keep running XP because I use some old Windows-only software, including a program that was abandoned after the upgrade for Windows 2000. There's a fair chance I'll still use this PC to encode video occasionally too.

Old PCs might be slow, but you can still encode faster using them alongside a new computer running the latest hardware than you can if you don't.

[Stereodude]

I doubt many people refrain from upgrading their computer because it won't run XP, or because there's no drivers for newer hardware, but I suspect there's many who don't see much point spending money to upgrade Windows on an old PC.

Do most household only have a single PC? There's four used regularly here, one of which is an old Pentium 3.

I've procrastinated replacing my PC because I can't bring myself to install Windows Spyware 10, and switching to Linux would require more effort and a learning curve, although as Linux Avisynth appears to be a thing now it's probably time, but this PC will keep running XP because I use some old Windows-only software, including a program that was abandoned after the upgrade for Windows 2000. There's a fair chance I'll still use this PC to encode video occasionally too.

Old PCs might be slow, but you can still encode faster using them alongside a new computer running the latest hardware than you can if you don't.
I have 5 PCs that are used "regularly" excluding servers in my house. Two are laptops. One has a 4th gen i5 the other has an 8th gen i7. Desktop 1 has a 2nd gen Xeon and desktop 2 is a Ryzen 9 3950X. My HTPC has a 4th gen i3.

4 of them run Windows 10, the HTPC runs Windows 8.1. Every P3, P4, & Core2 duo/quad system have long been put out to pasture.

With x265 the Ryzen 9 system is ~4x faster than the 2nd E5 Xeon. Everything else is much slower* (technically the 8th gen i7 laptop [Dell XPS 13] is about as fast as the 2nd E5 Xeon in x265 thanks to AVX2 and other core enhancements but it sits in the high 90C's encoding thermally throttling the whole time with the internal fans cranked and I don't think that's good for the system for hours on end). Sure, I could get slightly more throughput using all the systems or even including old retired systems collecting dust, but it doesn't make sense.

1) It's a pain to split encodes if you want to use a qpfile file and have full control of the encoding process. The more pieces the bigger the hassle. Do I want to be messing with getting yet another system encoding it's tiny part of the job for another 5% of encoding throughput? Nope...

2) Even if not splitting single encodes, power efficiency is a thing. Electricity isn't super expensive here and I'm not exactly a treehugger, but doubling my power usage to get another 5% of total encoding throughput by using an old system hardly seems prudent.

[hello_hello]

So you're the person using QPFiles. I figured someone must... but I didn't know who.... :)

I'm OCD about a lot of things, but my enthusiasm for driving myself insane hasn't extended to QPFiles. I did consider it.... as there's software for converting QPFiles from chapters etc, but if you remove the studio promos from the beginning of a movie, which I generally do, they become too much trouble for me. And of course it makes splitting scripts harder, which I do more than I'd like to. Usually to fix a problem I missed, such as cropping or because a word in the hard-coded subs wasn't spelled correctly....

Edit: thinking about it, I could always encode the studio promos and cut them out later if I was using a QPFile created from chapters, although I'd need to split the script or use a QPFile to make sure there's a keyframe in the splitting position.

Actually, I might need to rethink my entire position on QPFiles, now I'm talking about them. In order to edit the crap from the beginning or end of movies and because I'm lazy, I usually encode them separately after splitting the script to ensure there's a keyframe where one needs to be, then I append the split output, add the other streams and tell MKVToolNix to split the output at the same point, so it splits off the audio and subtitles etc too and I don't have to think about it. Instead of splitting the script to ensure there's a keyframe at the split point, I assume I could be using a QPFile. I'll give it a spin to see if it's quicker than copying the script and adding Trim() to the end. Although I would have to remember to add the QPFile to the command line, which I'm sure I'll regularly forget to do for at least the first 100 encodes :(.

[Sharc]

Interlaced source.

script 1:
AVIsource("source").ConverttoYV12(interlaced=true).AssumeTFF()
separatefields()
e=selecteven().fft3Dfilter(interlaced=false)
o=selectodd().fft3Dfilter(interlaced=false)
interleave(e,o)
weave()

script 2:
AVIsource("source").ConverttoYV12(interlaced=true).AssumeTFF()
fft3Dfilter(interlaced=true)

Shouldn't the 2 scripts produce identical interlaced outputs?
When separating the fields only 1 field is the same, the other field is different for the 2 scripts :confused:

[Stereodude]

So you're the person using QPFiles. I figured someone must... but I didn't know who.... :)
Well I used to re-encode for blu-ray which AFAIK required a keyframe where the chapter fell. Even with MKV it works nicer if there's a keyframe at the chapters on some devices.

I made a fairly simple spreadsheet to sort it all out when splitting. It suggests what frames to split the files (length / number of pieces). I open it in VD2 and use the closest scene change to the suggested number, enter those new frame numbers in a different row of the spreadsheet and then based on the original frame numbers of keyframes for chapter stops will tell me what frame number they need to be in each segment with only a little manual manipulation. It's then a copy paste fiesta to put the appropriate trim command into 8 avs files based on the spreadsheet and copy and paste to make 8 .chp files. It's not hard to do, but i'd estimate it still takes 20-30 minutes from start to finish.

[MeteorRain]

I wrote scripts to split AVS into multiple segments by converting chapter files, and then send them to different servers to encode, and later join the GOP files together.
Usually I only have to eac3to the chapters and prepare a universal avsi file to be imported, and all other works are for loop with scripts.

function p(string c, int fstart, int fend)
{
MP_Pipeline("""
lwlibavvideosource(""" + chr(34) + c + ".m2ts" + chr(34) + """, threads=1)

### prefetch: 32, 16
### lock threads to cores 4
### ###

QTGMC(Preset="fast")

### prefetch: 32, 16
### lock threads to cores 6
### ###

left = crop(0,0,976,0).MCTD().crop(0,0,960,0)
crop(944,0,0,0)

### export clip: left
### prefetch: 32, 16
### lock threads to cores 8
### ###

right = MCTD().crop(16,0,0,0)
stackhorizontal(left, right)


### prefetch: 32, 16
### lock threads to cores 10
### ###
""")

trim(fstart,fend)

f3kdb(output_depth=10,y=48,cb=48,cr=48,grainy=0,grainc=0)
}


import("mm.avsi")
p("01", 141600, 157957)


Hope this gives you some idea.

[MeteorRain]

I have uploaded the very first formal version here at GitHub (https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D/releases).

141XP is included. msvc-x86 and -x64 are for daily drive. SSE and AVX routine is included for all core functions.

Let me know how it works for ya.

[ChaosKing]

I guess VS suport is still WIP? The function name is DelogoHD for VS :D

[MeteorRain]

Oops, that was a copy paste mistake. Please re-download and try.
FYI, I don't have VapourSynth environment here so I only made sure it compiles. If it's convenient for you to test on VS please let me know any issue.

[ChaosKing]

hmm, it seems like the filter does nothing. I even tried very high sigma values, but the image is unchanged: clip.neo_fft3d.FFT3D(sigma=90)

[MeteorRain]

I shall find a way to make planes default to [0,1,2]. Also reading some unfreed memory blocks, needs investigating.


Try v3.

[ChaosKing]

v3 clang x64 gets "stuck" in vsedit if more then 1 frame is requested simultaneously. If I hit the play button or benchmark (tested it also with blankclip just to be sure it's not the source)
msvc looks a bit better, it gets only stuck sometimes :D

Tested with R48 / R49 x64, win10, ryzen 2600

oh and sigma works now in v3

Edit:
core.std.SetMaxCPU("none") did not help

[MeteorRain]

SetMaxCPU changes the instruction set to VapourSynth, so I don't think that matters.

I tried clang x64 and msvc x64 with VirtualDub2, both look pretty normal. Does it only break on VSEdit? Mind pasting the full script? Are you on YUV420P8 or some more "advanced" pixel format?

[ChaosKing]

Same thing happens in vdub

import vapoursynth as vs
core = vs.core

clip = core.std.BlankClip(format=vs.YUV420P8)
clip = clip.neo_fft3d.FFT3D()
clip.set_output()

See here what it looks like in vdub: https://i.imgur.com/WBAYLy1.gifv

[MeteorRain]

Reproduced.

When I copy paste to create my VPY script it had set threads to 1 already, which is why I didn't reproduce it.

Multi-threading upstream request needs some special routine and I've fixed it accordingly (hopefully!).


Try v4.

[ChaosKing]

v4 is looking good
A quick speed test (x64) with a ntsc DVD, default values:
- neo fft clang ~ 71.8 fps
- neo fft msvc ~73.9 fps
- neo fft msvc-xp ~74.1 fps
- "old" fft ~ 67.4 fps

run multiple times 1500 frames in vsedit

[FranceBB]

This is weird...
I tried the version from msvc-xp_x86 and Dependency Walker says that everything is fine (https://i.imgur.com/qCVtudI.png), however when I try to use the plugin, it prompts me to an "Access Violation".
Yet, when I run AVSMeter on my plugin folder, it doesn't complain at all, it just shows me neo-fft3d in my plugin+ folder.

FFVideoSource("test.mkv")
neo_FFT3D()


access violation


FFVideoSource("test.mkv")
neo_FFT3D(sigma=2.0, bt=3, y=3, u=3, v=3)


access violation


ColorBars(width=640, height=480, pixel_type="YV12")
neo_FFT3D(sigma=2.0, bt=3, y=3, u=3, v=3)


access violation

https://i.imgur.com/VyOq4VV.png

However... it works fine on Windows 10 (same PC, same hardware, same Avisynth installation, same fftw3.3.8, same AVS Script, same version of AVSPmod)...

The former version made by Ferenc Pinter, however, works like a charm on XP.
But I don't understand why... it should work, every dependency is included, the compiler compiled fine and didn't complain about anything and the CPU - although it supports 'till AVX2 - should be able to use the SSE/SSE2/SSSE3/SSE4.1/SSE4.2 on XP...

Is there a way to force SSE instead of AVX on Windows 10? Just to make sure that nothing is broken in the code and that it has really something (weird and deeply obscure) to deal with XP?

[MeteorRain]

If you can compile it youself, here's the compiling script: https://gist.github.com/msg7086/bb237a503d866d74c1cc8c2c49f90c0e

To remove AVX routine, comment this (https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D/blob/master/src/functions.h#L184) out.

[pinterf]

It's probably not an AVX problem, for XP you have to build with v141_xp platform toolset (Visual Studio 2017 - Windows XP (v141_xp)) _and_ manually adding
/Zc:threadSafeInit-
to the additional options for C/C++ compiler

[MeteorRain]

Thanks, please try NeoFFT3D_r1v5-windows-xp.zip

[FranceBB]

If you can compile it youself, here's the compiling script: https://gist.github.com/msg7086/bb237a503d866d74c1cc8c2c49f90c0e

Ok, sure.

It's probably not an AVX problem, for XP you have to build with v141_xp platform toolset (Visual Studio 2017 - Windows XP (v141_xp)) _and_ manually adding
/Zc:threadSafeInit-
to the additional options for C/C++ compiler

Setting threadsafeinit in the compiler did the trick!
It's working flawlessly now on XP! :)

Thanks, please try NeoFFT3D_r1v5-windows-xp.zip

Your new build works too!

Thank you both! :D

[Groucho2004]

@FranceBB
Time to update AVSMeter. 2.8.5 is 1.5 years old.

[FranceBB]

@FranceBB
Time to update AVSMeter. 2.8.5 is 1.5 years old.

Oops.
How can I have missed that...?
Updated right now! :)
By the way, thank you for AVSMeter, I found it very useful on several occasions, especially when I was walking by some of my colleagues who had any kind of fucked up installation with double/triple plugins, random names etc.
It also helped me to keep my installation and plugin folder perfectly clean with no redundant things. ;)

[MeteorRain]

r2 -- marked as pre-release for now -- has been uploaded to GitHub.

I spend a few hours to completely rewrite the dual interface wrappers and it should be easier to work with now. I'm planning to move some of my filters into the new platform if I have time, including f3kdb (before fixing some raised issue).

Let me know how it works -- and if it is faster than it was. I experimentally used C++ Parallel STL in a few hot code paths and should have improved speed a bit.

[ChaosKing]

v4 is looking good
A quick speed test (x64) with a ntsc DVD, default values:
- neo fft clang ~ 71.8 fps
- neo fft msvc ~73.9 fps
- neo fft msvc-xp ~74.1 fps
- "old" fft ~ 67.4 fps

run multiple times 1500 frames in vsedit
With r2 -- marked as pre-release
clang 109.2 fps
msvc 113.3 fps

Everything is tested on a Ryzen 2600
std.BlankClip(format=vs.YUV420P8, width=1920, height=1080, length=500).neo_fft3d.FFT3D()

x64 r2-pre:
clang:Output 500 frames in 25.57 seconds (19.55 fps)
msvc: Output 500 frames in 24.22 seconds (20.64 fps)

x64 r1
msvc: Output 500 frames in 39.87 seconds (12.54 fps)

x64 original fft3dfilter
Output 500 frames in 42.38 seconds (11.80 fps)


Nearly x2 faster :eek:

And some warning to go :devil:
Core freed but 1866240000 bytes still allocated in framebuffers

In Avisynth r1 complains about the pixel_type for BlankClip() with a correct error msg, but r2 just throws an error: There is no function named neo_fft3d(). It works if I use a valid pixel type BlankClip(pixel_type = "YUV420P8")

EDIT: No wait, I mixed up the dlls: r2 always throws this error: There is no function named neo_fft3d(). I tried all x64+x86 versions

[MeteorRain]

Yea vapoursynth should have some issues. Let me set up some test environment and see how to deal with the issues.

A leaking filter instance cost me nearly an hour or two. This thing is a time killer.

And I put the wrong name on the function name, so...

[MeteorRain]

Uploaded r3. Should have solved all kinds of issues.

[ChaosKing]

The fps gap between clang and msvc is now almost gone.
Both ranged between 31-33 fps.

Output 500 frames in 15.71 seconds (31.82 fps)
Output 500 frames in 14.99 seconds (33.37 fps)
185 fps now on my tested DVD ntsc source.

But this speed up again... What kind of sorcery is this? :devil:

EDIT:
memory warning is gone
avisynth works. I get 28-29fps in avisynth x64 3.4
So now we have a nearly x3 faster fft3d filter. Awesome job!:thanks:

EDIT2:
Added plugin to vsrepo & avsrepo

[tormento]

The fps gap between clang and msvc is now almost gone.
As a programming noob as I am, may I ask why some of AVS contributors started to provide both compiles? I mean, which are differences and advantages? Speed on different CPUs? I am really curious.

[ChaosKing]

It is just something I noticed...

In general different compilers have different optimization strategies implemented. This can lead to plugin A is faster then plugin B with compiler X but plugin C is faster with compiler Y.

Then you could also inspect the generated machine code by compiler X and see why it is faster or slower then the code by compiler Y.

tldr: compiler, please undestand what I want.

[pinterf]

There are so many options and sub-filters that it is impossible to test all cases and choose one "best" compiler.
E.g. there are a tons of possible branches by processor types: SSE2, SSE4, AVX, AVX2, plus 24 modes in RgTools, plus repair option. Many hundred different scenarios.
Then it was a real case that msvc produced 10% faster code in AVX2 for Removegrain mode X while it was 10% slower at mode Y, compared to clang.
Let the user choose, if user is a power-user and have inifinite time to test, he will know why this or that version is chosen. All other user just choose by sympathy (MS rulez, clang inside yeah).

[tormento]

And you forgot Intel Parallel XE too xD

Work colleagues told me miracles of that.

[MeteorRain]

As a programming noob as I am, may I ask why some of AVS contributors started to provide both compiles? I mean, which are differences and advantages? Speed on different CPUs? I am really curious.

A few reasons.

1) clang does lots of optimizations on even SIMD intrinsics code, while MSVC tends to compile as-is. For example, in f3kdb, where there's a selection operation (I can't remember the details), clang directly optimize the whole operation into a completely different instruction sequence, which greatly improved the speed on SSE2, even making it almost as fast or faster than the AVX variant (due to lack of equivalent instruction in AVX). While MSVC simply compiled as-is, with some unroll maybe?

2) clang checks lots of extra warnings and errors. Having a clang compiled version means the code will likely compile on clang @ Linux or MacOS as well.

I personally also compile the code in GCC to make sure it satisfies all common compilers. However GCC uses POSIX ABI so it's not usable under MSVC C++ AVS+ so I didn't include it in the release.

[tormento]

There are so many options and sub-filters that it is impossible to test all cases and choose one "best" compiler.
I just tried a very simple filter case, that uses KNLMeansCL, MaskTools and MVTools only. The only available filter as both clang and VS2019 is MaskTools.

Average on 10x launch of same script:

MaskTools clang: 13.43 fps
MaskTools VS2019: 13.59 fps

The true bottleneck here is MVTools, that I always beg you to release on both compiler versions to see if any difference occurs. :p

[tormento]

A few reasons.
Thanks for the exaustive reply. In the mean time I asked my old friends in University what they use and confirmed me Intel compiler is one of the best to have fast code. On some peculiar cases, mostly scientific simulations, they can see 20x increase vs VS one.

[MeteorRain]

But this speed up again... What kind of sorcery is this? :devil:

I have a pretty smart :sly: LRU caching system and a multi-threading counting system, to make sure I use the least possible memory to work with give multi-threading model. So it should now runs at 6 threads from input.

Then for each plane I let them fly on their own threads, so 18 threads into the engine.

From there, some hot paths like copying window data into blocks operation gets parallel'd using C++ PSTL, which is basically a trouble free thread pool.

Core functions such as Apply and Sharpen families get full SSE/AVX/AVX512 optimization. Because I have a centralized wrapper and put variants into templates, it's pretty easy for me to transform loops into parallel computing. I've tried putting all blocks into their own threads, and it turns out it runs slower. So current design is to run 4 parallel computing batch, and each run a quarter of the data blocks.

I definitely don't want it to occupy too much CPU, because it's usually paired with other filters. So I tweak it to use about 3-4 full cores maximum to maintain a high running efficiency.

[MeteorRain]

Thanks for the exaustive reply. In the mean time I asked my old friends in University what they use and confirmed me Intel compiler is one of the best to have fast code. On some peculiar cases, mostly scientific simulations, they can see 20x increase vs VS one.

Yes and no. Lots of CPU intensive code nowadays are written in assembly / intrinsics anyway, so the compiler is having less and less things to optimize.

Like I said, you must be able to optimize assembly code to further improve the performance. Sometimes you also have to take into consideration what the target CPU is, because, there's a few instructions that runs faster and slower on different CPUs.

For example, if in the SIMD I write code A; B; C, but the compiler thinks my code is dumb because it can be done using X; Y, which is faster, it should delete my code and change it to the faster version right away.

Then there's times when ABC is faster on, for example, haswell, but XY is faster on skylake, now the compiler would ask you, which CPU would you prefer to target. If you say I love haswell then it should leave ABC there.

Optimization is a very interesting topic to study, and I only know a small portion of it. Here's my $0.02.

[ChaosKing]

I have a pretty smart :sly: LRU caching system and a multi-threading counting system, to make sure I use the least possible memory to work with give multi-threading model. So it should now runs at 6 threads from input.

Then for each plane I let them fly on their own threads, so 18 threads into the engine.

From there, some hot paths like copying window data into blocks operation gets parallel'd using C++ PSTL, which is basically a trouble free thread pool.

Core functions such as Apply and Sharpen families get full SSE/AVX/AVX512 optimization. Because I have a centralized wrapper and put variants into templates, it's pretty easy for me to transform loops into parallel computing. I've tried putting all blocks into their own threads, and it turns out it runs slower. So current design is to run 4 parallel computing batch, and each run a quarter of the data blocks.

I definitely don't want it to occupy too much CPU, because it's usually paired with other filters. So I tweak it to use about 3-4 full cores maximum to maintain a high running efficiency.

Now do the same for mvtools :D
Haha just kidding.

[MeteorRain]

Never looked at that code. You tell me how hard it could be for me to get hands on it. If it's not to hard I may actually give it a try.

[ChaosKing]

Never looked at that code. You tell me how hard it could be for me to get hands on it. If it's not to hard I may actually give it a try.

As far as I read on doom9, the code is a mess...

https://github.com/pinterf/mvtools
https://github.com/dubhater/vapoursynth-mvtools

But maybe it would be better and easier to improve the c++ reimplementation of mvtools by feisty2
https://forum.doom9.org/showthread.php?t=172525
https://github.com/IFeelBloated/vapoursynth-mvtools-sf <-- I had some problematic scenes where this version was ok where the other mvtools showed some artifacts. Maybe 32bit precision had something to do with it, idk.

[pinterf]

While fft3dfilter update took some hours for me, I spent months with mvtools.

[StainlessS]

I spent months with mvtools.

You must have found it very interesting :)

[MeteorRain]

How's f2's version working? Is it fully functional in terms of parameters and functions support?

Although he just said he might rewrite it....

[ChaosKing]

How's f2's version working? Is it fully functional in terms of parameters and functions support?

Although he just said he might rewrite it....

Yes everything works as far as I know. It is used for smdegrain (tr >4), TemporalDegrain2, mClean etc here https://github.com/Selur/VapoursynthScriptsInHybrid/blob/master/G41Fun.py#L1929

[MeteorRain]

How fast (slow) it runs? Do you think it is a good idea to forget the original version and just take this as the base?

[Sharc]

While fft3dfilter update took some hours for me, I spent months with mvtools.
FFT3DFilter has the option 'interlaced=true' for the direct filtering of interlaced video. Shouldn't the 2 scripts below produce identical outputs? Actually they don't. Which one is correct? Do I miss something? :confused:

#Script 1
AVISource("source.avi")
separatefields()
even=selecteven().fft3dfilter(plane=4,interlaced=false)
odd=selectodd().fft3dfilter(plane=4,interlaced=false)
v1=interleave(even,odd)

#Script 2
AVISource("source.avi")
fft3dfilter(plane=4,interlaced=true)
v2=separatefields()

return subtract(v1,v2).Levels(127,1,129,0,255) #should give grey picture if v1,v2 are identical

[ChaosKing]

How fast (slow) it runs? Do you think it is a good idea to forget the original version and just take this as the base?

SMDegrain(mvf.Depth(clip, 16), tr=3) # 16bit mvtools "fast version" -> 18 fps
SMDegrain(mvf.Depth(clip, 32), tr=3) # 32bit mvtools-sf -> 3 fps

The only thing I can say is, that in my tests the mvtools-sf version produced better results in some edge cases and the source code is probably a bit easier to read. Maybe dubhater can give a hint which version he would improve ;-)

[tormento]

The only thing I can say is, that in my tests the mvtools-sf version produced better results in some edge cases and the source code is probably a bit easier to read. Maybe dubhater can give a hint which version he would improve ;-)
I have tested SMDegrain under VS too and it's a clear example of added complexity and lower speed. Perhaps VS is much better for other things but not for this specific one.

[Stereodude]

r2 -- marked as pre-release for now -- has been uploaded to GitHub.

I spend a few hours to completely rewrite the dual interface wrappers and it should be easier to work with now. I'm planning to move some of my filters into the new platform if I have time, including f3kdb (before fixing some raised issue).

Let me know how it works -- and if it is faster than it was. I experimentally used C++ Parallel STL in a few hot code paths and should have improved speed a bit.
Why did you remove the plane argument from Neo_FFT3D? Both MCTD and QTGMC make use of it. Unless I'm missing it any speed improvements this Neo version has can't be use for the less aggressive presets because they're not expecting the chroma planes to be processed.

[MeteorRain]

Why did you remove the plane argument from Neo_FFT3D? Both MCTD and QTGMC make use of it. Unless I'm missing it any speed improvements this Neo version has can't be use for the less aggressive presets because they're not expecting the chroma planes to be processed.

Because with y=3, u=2, v=2 I don't see a point keeping plane option there, being redundant and less useful. Let me know what your needs are if this is insufficient.

[Stereodude]

Because with y=3, u=2, v=2 I don't see a point keeping plane option there, being redundant and less useful. Let me know what your needs are if this is insufficient.
Somehow I overlooked those arguments. Let me see if I can modify the scripts to use the new arguments. It looks pretty straightforward at least for MCTD.

[Stereodude]

So MCTD gets a ~13% speed increase using neo_fft3d r3 vs. fft3d 2.6 on HD resolution content on system without AVX2 and a ~10% speed increase on a system with AVX2. This was using a mild NR preset and MPP.

On SD (DVD) content the gains are a lot less.

[tormento]

so mctd gets a ~13% speed increase using neo_fft3d r3 vs. Fft3d 2.6
Post it NOW :D

[Stereodude]

Post it NOW :D
Done: https://forum.doom9.org/showthread.php?p=1907108#post1907108

(pending attachment approval)

[tormento]

Done
:thanks:
(pending attachment approval)
*cough*paste.bin*cough

[Stereodude]

:thanks:

*cough*paste.bin*cough
added...

[tormento]

added...

Thanks.

[pinterf]

What about starting a separate topic for the new plugin?

[DJATOM]

Yeah, it would be nice from mods to move Neo related posts into new topic :)

[StainlessS]

+1 on Neo having own thread.
(helps in avoiding confusion)

[tormento]

+1 on Neo having own thread.
(helps in avoiding confusion)



So should pinterf :)

[StainlessS]

P is still considering it [although one might think 6 months (or is it 12) would be enough time to analyse and make a decision, very careful guy is P, he don't like to jump the gun].

[MeteorRain]

Another option is me converting neo f3kdb to neo family post, and waiting for mod to move replies there. (Is it possible?)

[tebasuna51]

If pinterf (owner of this thread) want we can create a new thread FFT3DFilter-Neo and after move the related post (I need a list of post numbers # involved)

[pinterf]

I do not want but it seems practical.

[poisondeathray]

FFT3DFilter v2.8 pinterf fork plane=3 issue, u,v color shift

colorbars(width=400, height=300, pixel_type="yv12")
histogram("levels")

colorbars(width=400, height=300, pixel_type="yv12")
fft3dfilter(bt=1, sigma=0, plane=3)
histogram("levels")
subtitle("fft3dfilter(bt=1, sigma=0, plane=3), gamma2")

(apng should animate in most browsers)
https://i.postimg.cc/fyfWKYH8/apng1.png (https://postimages.org/)


eg on dark scene (brightened with levels gamma=2 for display purposes) , rgb histogram instead of uv levels
(apng should animate in most browsers)
https://i.postimg.cc/wMgg776L/apng2.png (https://postimages.org/)

sigma= higher value does not alter the shift
plane=4 (all Y,U,V) does not alter the U,V shift

[pinterf]

Interesting. Only negative U and V values are affected. Rounding issue? How does the newer neo version behave?

[poisondeathray]

How does the newer neo version behave?

neo_fft3d r10 works ok (no U,V shift) ; sigma > 0 ok too (or expected results)

neo_fft3d(sigma=0, bt=1, y=2, u=3, v=3)

[pinterf]

Yep, that was fixed in neo_fft3d. Though the original plugin is superseded in most aspects, I'm gonna update it.

[pinterf]

New "yesIknowthereisneo" release, because even old things must be kept clean.
https://github.com/pinterf/fft3dfilter/releases/tag/v2.9

[poisondeathray]

New "yesIknowthereisneo" release, because even old things must be kept clean.
https://github.com/pinterf/fft3dfilter/releases/tag/v2.9

Thanks

Many older scripts use fft3dfilter classic syntax, not updated neo, so this is still useful for some people

[StainlessS]

Yip, I'm still on the antique one, thanks very much.

[GMJCZP]

I was testing version 2.9 and after coding and closing Virtualdub2 or VirtualdubMod it throws me an error message associated with libfftw3f-3. I have already tried several versions of the library and the same error appears, this does not happen with neo_fft3d that works, somewhat slower yes, but correctly.

[pinterf]

I was testing version 2.9 and after coding and closing Virtualdub2 or VirtualdubMod it throws me an error message associated with libfftw3f-3. I have already tried several versions of the library and the same error appears, this does not happen with neo_fft3d that works, somewhat slower yes, but correctly.
Script and how-to steps needed, cannot reproduce.

[GMJCZP]

Script and how-to steps needed, cannot reproduce.

I already found out who is guilty, it's Stab, Stab calls DePan, DePanEstimate and RGTools (DePanEstimate calls libfftw3f-3).
These are the versions I am using:

DePan and DepanEstimate: mvtools-2.7.45
RGTools: 1.2

Stab is the script originally created by g-force (December 10, 2008)

pinterf, if you use a script like this:

SourceFilter(video)
Stab()
fft3dfilter()

It should reproduce the problem. What I find strange is what is the relationship between Stab and fft3dfilter.

Edit: just opening the script with virtualdub and closing the program produces the error.

[pinterf]

Yep, I've got it, debugged it, but do not understand :)

[GMJCZP]

Yep, I've got it, debugged it, but do not understand :)

There seems to be a love triangle between DePanEstimate (Stab) - libfftw3f-3 - fft3dfilter, and while neo_fft3d stays in the final with the girl who is the protagonist of the story. :confused:

[pinterf]

Check please v10 (https://github.com/pinterf/fft3dfilter/releases/tag/v2.10).

[GMJCZP]

All right, thank you pinterf.