KNLMeansCL: OpenCL NLMeans de-noising algorithm [2018-01-29]

[Magik Mark]

Mystery x

What does "true" do in (KNLMeansCl, 2, true)?

I could confirm its faster. I get 25fps in conjunction with Deband, auto adjust, and knlmeans filters


Sent from my iPhone using Tapatalk

[MysteryX]

What does "true" do in (KNLMeansCl, 2, true)?
"true" overrides the definition already set in my AviSynthMT.avsi file :)

In your case, how much faster do you get?

From a single-threaded script, it seems you'll generally get ~30% improvement by running it in MT mode 3, and a further ~3% improvement by instead running it in MT mode 2. Of course these improvements will vary based on the script and computer setup, but it shouldn't be far from these numbers.

[Magik Mark]

"true" overrides the definition already set in my AviSynthMT.avsi file :)

In your case, how much faster do you get?

From a single-threaded script, it seems you'll generally get ~30% improvement by running it in MT mode 3, and a further ~3% improvement by instead running it in MT mode 2. Of course these improvements will vary based on the script and computer setup, but it shouldn't be far from these numbers.



I usually run 4 scripts. FFvideoSource, Flash3kyuudeband, AutoAdjust & KnlMeans. I get around 18fps for the first pass. With the above settings I get around 25fps. This is running in staxrip with prefetch 2. Can you advise further on the optimum MT mode for my other filters?


Sent from my iPhone using Tapatalk

[MysteryX]

In your case you get 38% improvement. Close to the 30%+3% I was talking about.

Someone compiled a list of compatible MT modes for all filters; not sure exactly where it is but you can find it by looking through the AviSynth+ thread. I chose to save that file as AviSynthMT.avsi and tweaked it.

[MysteryX]

I gave another try at running KNLMeans with both devices. The last time, I saw no benefit because I wasn't using MT, and thus the GPUs were working only one at a time and thus there was no benefit whatsoever.

First, regular KNLMeans. MT won't help here because there is nothing else using the CPU. In fact, MT appears to make it "clog" and work in chunks, although that doesn't affect much the overall performance.

colorbars(width = 1920, height = 1080, pixel_type = "yv12").killaudio().assumefps(25, 1)
KNLMeansCL(d = 2, a = 2, device_id=0)

FPS (min | max | average): 1.311 | 1.389 | 1.323
Memory usage (phys | virt): 142 | 183 MiB
Thread count: 12
CPU usage (average): 11%


1 instance of each device.

SetFilterMtMode("KNLMeansCl", 3)
colorbars(width = 1920, height = 1080, pixel_type = "yv12").killaudio().assumefps(25, 1)
c1 = SelectEvery(2, 0).KNLMeansCL(d = 2, a = 2, device_id=0)
c2 = SelectEvery(2, 1).KNLMeansCL(d = 2, a = 2, device_id=1)
Interleave(c1,c2)
Prefetch(4)

FPS (min | max | average): 0.234 | 48718 | 1.531
Memory usage (phys | virt): 197 | 246 MiB
Thread count: 13
CPU usage (average): 15%


2 instances of each device

SetFilterMtMode("KNLMeansCl", 3, true)
colorbars(width = 1920, height = 1080, pixel_type = "yv12").killaudio().assumefps(25, 1)
c1 = SelectEvery(4, 0).KNLMeansCL(d = 2, a = 2, device_id=0)
c2 = SelectEvery(4, 1).KNLMeansCL(d = 2, a = 2, device_id=0)
c3 = SelectEvery(4, 2).KNLMeansCL(d = 2, a = 2, device_id=1)
c4 = SelectEvery(4, 3).KNLMeansCL(d = 2, a = 2, device_id=1)
Interleave(c1,c2,c3,c4)
Prefetch(4)

FPS (min | max | average): 0.286 | 93538 | 1.563
Memory usage (phys | virt): 282 | 352 MiB
Thread count: 14
CPU usage (average): 18%

Note: I had results go up to 2.2 on that last test. There is a "clogging" effect that becomes more serious with Prefetch(8).

Or let's try it this way. MT mode 2 doesn't have the same clogging as MT mode 3

SetFilterMtMode("KNLMeansCl", 2)
colorbars(width = 1920, height = 1080, pixel_type = "yv12").killaudio().assumefps(25, 1)
c1 = SelectEvery(2, 0).KNLMeansCL(d = 2, a = 2, device_id=0)
c2 = SelectEvery(2, 1).KNLMeansCL(d = 2, a = 2, device_id=1)
Interleave(c1,c2)
Prefetch(2)

FPS (min | max | average): 0.368 | 26.44 | 1.963
Memory usage (phys | virt): 336 | 419 MiB
Thread count: 12
CPU usage (average): 15%


Same but 4 threads.

SetFilterMtMode("KNLMeansCl", 2)
colorbars(width = 1920, height = 1080, pixel_type = "yv12").killaudio().assumefps(25, 1)
c1 = SelectEvery(2, 0).KNLMeansCL(d = 2, a = 2, device_id=0)
c2 = SelectEvery(2, 1).KNLMeansCL(d = 2, a = 2, device_id=1)
Interleave(c1,c2)
Prefetch(4)

FPS (min | max | average): 0.357 | 97435 | 2.164
Memory usage (phys | virt): 521 | 641 MiB
Thread count: 17
CPU usage (average): 21%

Now that's what I'm talking about! That's a 40% improvement on top of the 30% improvement for using MT. This can result in a 70% improvement when running a full script.

** Note: the last test sometimes gives 2.2fps, sometimes 1.6fps, I really don't know why.

There might be a thread synchronization issue because one device works faster than the other device. It the device splitting was handled internally by sending the frame to the next available device, it would most likely work a lot smoother.

On a side note, KNLMeans with CPU gives 0.081fps

[Magik Mark]

What does selectevery do?


Sent from my iPhone using Tapatalk

[Magik Mark]

I'm getting selected device not available? How do I correct this?


Sent from my iPhone using Tapatalk

[MysteryX]

This script is for dual-graphics computers. I have a Intel HD 4000 and a Radeon 7670M.

SelectEvery is to send frames 0, 2, 4, 6, 8 to the Intel GPU and frames 1,3,5,7,9 to the AMD GPU.

This won't work if you have a single graphic card.

Some people might even have 2 graphic cards + a dedicated GPU device and all 3 could be working together.

[Magik Mark]

Is it possible to use CPU and GPU in parallel?

[feisty2]

@Khanattila
If you ever thought about writing a new GPU filter or maybe going to do that someday.
could you please consider BM3D (http://forum.doom9.org/showthread.php?t=172172)?
I'm all fucked up by its abysmal performance..

[MysteryX]

On a side note, KNLMeans with CPU gives 0.081fps
Is it possible to use CPU and GPU in parallel?
Sure, but why would you want to do that?

Besides being ridiculously low performance while the CPU could be doing more useful work, doing it the way I said would cause threading synchronization issues because one worker is 20 times faster than the other so the frames can't be split evenly between both.

I'm all fucked up by its abysmal performance..
So you do care for performance :)

[Khanattila]

@Khanattila
If you ever thought about writing a new GPU filter or maybe going to do that someday.
could you please consider BM3D (http://forum.doom9.org/showthread.php?t=172172)?
I'm all fucked up by its abysmal performance..

Hope is the last to die.

[Khanattila]

This script is for dual-graphics computers. I have a Intel HD 4000 and a Radeon 7670M.

SelectEvery is to send frames 0, 2, 4, 6, 8 to the Intel GPU and frames 1,3,5,7,9 to the AMD GPU.

This won't work if you have a single graphic card.

Some people might even have 2 graphic cards + a dedicated GPU device and all 3 could be working together.
Aka alternate frame rendering.

[Khanattila]

A system that allows the use of multiple graphics cards is possible. It takes time.

[feisty2]

So you do care for performance :)

I didn't, till Oyster hit 0.00fps last night, and that's my bottom line

[MysteryX]

A system that allows the use of multiple graphics cards is possible. It takes time.
Doing it the way I said wouldn't be difficult if the code is properly encapsulated.

Just look at the way I have done it here. (https://github.com/mysteryx93/AviSynthShader/blob/master/Src/ExecuteShader.cpp)

My engine is contained within a class. I initialize Engine1, and if running in AviSynth+ MT, initialize Engine2.

GetFrame starts like this

D3D9RenderImpl* render;
mutex_IterateDevice.lock();
if (!isMT || m_IterateDevice == 0) {
m_IterateDevice = 1;
render = render1;
}
else {
m_IterateDevice = 0;
render = render2;
}
mutex_IterateDevice.unlock();


Then I just use 'render' normally.

With the same logic, Engine1 can be initialized on device_id=0 while Engine2 can be initialized on device_id=1; and you can even have 2 of each devices and iterate between all 4. In this case, however, a flag will be required for each engine to report when they are available as the various devices may not work at the same speed.

It's really not difficult, but the key is encapsulation. The entire computation engine must be self-contained so that you can initialize it with a single call.

Here's the code that will split the workload across the various devices.

PVideoFrame __stdcall KNLMeans::GetFrame(int n, IScriptEnvironment* env) {
EngineCl* render = nullptr;
if (!isMT)
render = m_engines[0];
else {
mutex_DeviceSelection.lock();
for (int i=0; i<m_engineCount; i++) {
if (m_engines[i]->IsAvailable) {
render = m_engines[i];
break;
}
}
// If all engines are busy, rotate between devices and the thread will wait on the unique_lock
if (!render) {
render = m_engines[m_IterateDevice++];
if (m_IterateDevice >= m_engineCount)
m_IterateDevice = 0;
}
render->Available = false; // there might be an issue with Available when using rotation, should be easy to fix with testing
mutex_DeviceSelection.unlock();
}

render->DoYourJob();

if (isMT) {
mutex_DeviceSelection.lock();
render.Available = true;
mutex_DeviceSelection.unlock();
}
}

[poisondeathray]

Interesting test results using AFR -

But you wouldn't use that in practice with temporal filters (d=2) , as temporally adjacent frames wouldn't be processed together. Even/odd are processed separately so you are prone to "fluttering"

[MysteryX]

Interesting test results using AFR -

But you wouldn't use that in practice with temporal filters (d=2) , as temporally adjacent frames wouldn't be processed together. Even/odd are processed separately so you are prone to "fluttering"
Engine0 receives GetFrame for frames 0, 4, 8, 12. Engine1 receives GetFrame for frames 1, 5, 9, 13. It can still get the previous and future frames. As for the rest, AviSynth's frame buffer handles it.

In the case of 2 engines running at different speed like I did, I think I get choppy performance because the frame buffer's optimization gets confused.

[poisondeathray]

Engine0 receives GetFrame for frames 0, 4, 8, 12. Engine1 receives GetFrame for frames 1, 5, 9, 13. It can still get the previous and future frames. As for the rest, AviSynth's frame buffer handles it.



So that's a "bad" thing ( quality wise ) when using temporal filters. I'm referring only to quality issues, not performance or programming wise

[MysteryX]

When processing frame 4, it will request frames 3, 4 and 5. No problem there. It doesn't matter which engine or even which thread processes what frame.

With the SelectEvery script I wrote, I'm not exactly sure, but if it's handled within the plugin, then everything is fine for sure.

Edit: SelectEvery won't work because it returns a clip with less frames. It needs to be implemented within KNLMeans to work properly.

[Khanattila]

Doing it the way I said wouldn't be difficult if the code is properly encapsulated.

Just look at the way I have done it here. (https://github.com/mysteryx93/AviSynthShader/blob/master/Src/ExecuteShader.cpp)

My engine is contained within a class. I initialize Engine1, and if running in AviSynth+ MT, initialize Engine2.

GetFrame starts like this

D3D9RenderImpl* render;
mutex_IterateDevice.lock();
if (!isMT || m_IterateDevice == 0) {
m_IterateDevice = 1;
render = render1;
}
else {
m_IterateDevice = 0;
render = render2;
}
mutex_IterateDevice.unlock();


Then I just use 'render' normally.

With the same logic, Engine1 can be initialized on device_id=0 while Engine2 can be initialized on device_id=1; and you can even have 2 of each devices and iterate between all 4. In this case, however, a flag will be required for each engine to report when they are available as the various devices may not work at the same speed.

It's really not difficult, but the key is encapsulation. The entire computation engine must be self-contained so that you can initialize it with a single call.

Here's the code that will split the workload across the various devices.

PVideoFrame __stdcall KNLMeans::GetFrame(int n, IScriptEnvironment* env) {
EngineCl* render = nullptr;
if (!isMT)
render = m_engines[0];
else {
mutex_DeviceSelection.lock();
for (int i=0; i<m_engineCount; i++) {
if (m_engines[i]->IsAvailable) {
render = m_engines[i];
break;
}
}
// If all engines are busy, rotate between devices and the thread will wait on the unique_lock
if (!render) {
render = m_engines[m_IterateDevice++];
if (m_IterateDevice >= m_engineCount)
m_IterateDevice = 0;
}
render->Available = false; // there might be an issue with Available when using rotation, should be easy to fix with testing
mutex_DeviceSelection.unlock();
}

render->DoYourJob();

if (isMT) {
mutex_DeviceSelection.lock();
render.Available = true;
mutex_DeviceSelection.unlock();
}
}

I thank you of the pseudo-code you wrote but I think of a different solution.

[MysteryX]

The code I wrote would require a "QueueLength" counter instead of a "Available" flag as 8 threads might be working with 2 engines and they will never be "available".

I'm wondering whether there's a way to make it work nicely in single-threaded mode, as I have the exact same issues with my own code.

The only way would be for GetFrame to not lock the thread, and the only way for that would be for GetFrame to already have processed the next frame ahead of time, possibly from a different internal thread. There might be a way to get that to work but it would be hacky.

[MysteryX]

Regarding what we discussed here, I've done benchmark tests between MT_SERIALIZED and implementing MT_NICE_FILTER the way I described.

http://forum.doom9.org/showthread.php?p=1773672#post1773672

According to the benchmarks, the penalty of using MT_SERIALIZED is huge.

[Magik Mark]

MysteryX,

Have have these setting in my staxrip. Im geting 21fps. Do you think we can still tweak this?

http://i.imgur.com/H8XYKfJ.png

[MysteryX]

What MT mode is KNLMeans running on? You'll probably get better performance with 4 threads than 2 threads; but then you have to test it on your own system. Test MT mode 2 and 3, with 2, 4, 6 or 8 threads. Results will tell you what works best for you.

[Magik Mark]

Using this sequence, if I input Mt mode in knlmeans, Avisynth crashes


Sent from my iPhone using Tapatalk

[MysteryX]

MT modes are being set AFTER the command is executed! Of course it complains.

[Magik Mark]

MT modes are being set AFTER the command is executed! Of course it complains.

I'm kinda confuse. Can you advise on the sequence of my script? I'm not well verse in avisynth

[MysteryX]

It's kind of off-topic for this thread, but it's easier if you view and edit the full script itself, but it probably appends all the sections one after the other. First you have to define MT modes BEFORE the filters are being called. You need to move that up to the beginning of the script.

[tormento]

Is KNLMeans comparable with V-BM3D?

[Magik Mark]

I have noticed that using vapoursynth, this filter runs a lot faster to the point I could not use opencl for other application like madvr. Is there a way to slow it down a little bit? Can tweaking this helps:

core = vs.get_core(threads=XX)

[aymanalz]

At this point, what is the difference between NLmeansCL2 and KNLmeans? When I use KNLmeans with hybrid, and set the device to GPU, my GPU works at a 100% load, but processor works at 25%, and the encode takes a long time. When I use NLmeansCL2 and set the device to GPU, the CPU runs at 100%, GPU at about 25%, and the encode is much faster.

(I'm encoding a video to x265, with just this denoising filter added.)

What is the possible explanation for this?

[Groucho2004]

At this point, what is the difference between NLmeansCL2 and KNLmeans?
NLmeansCL2 is ancient, doesn't have a temporal mode and uses different default values.

You're comparing apples and oranges.

[Khanattila]

Is KNLMeans comparable with V-BM3D?
You can think about of something like that.
Bilateral --> NLMeans --> V-BM3D

[tormento]

You can think about of something like that.

Bilateral --> NLMeans --> V-BM3D


Is there any chance you compile the VapourSynth version for AVS+OpenCL?

[aymanalz]

NLmeansCL2 is ancient, doesn't have a temporal mode and uses different default values.

You're comparing apples and oranges.

I see, thanks for the info.

So is there a valid reason why KNLmeans is bottlenecking the encode? x265 in extremely CPU intensive, but as I said, the GPU is running full throttle, and the CPU is only running at 25%, when encoding to x265 with KNLmeans filter turned on. Clearly, the x265 encode is being held up by the KNLmeans denoiser.

Is this to be expected? Could the denoising be slower than the x265, even if the former is performed on GPU?

(CPU: Intel quad core 1i7 Haswell
GPU: Nvidia GTX 950M)

[Groucho2004]

So is there a valid reason why KNLmeans is bottlenecking the encode? x265 in extremely CPU intensive, but as I said, the GPU is running full throttle, and the CPU is only running at 25%, when encoding to x265 with KNLmeans filter turned on. Clearly, the x265 encode is being held up by the KNLmeans denoiser.

Is this to be expected? Could the denoising be slower than the x265, even if the former is performed on GPU?
Impossible to tell without knowing a bit about your source and, of course, seeing your script.

[aymanalz]

Impossible to tell without knowing a bit about your source and, of course, seeing your script.

I haven't edited the script, I am just using the default options in Hybrid, except that I've changed the device to GPU. The video is a 2 minute clip from a grainy "Sound of Music" bluray. (It is too big for me to upload.)

So the script is simply:

LoadPlugin("C:\Program Files\Hybrid\avisynthPlugins\LoadDll.dll")
LoadPlugin("C:\Program Files\Hybrid\avisynthPlugins\KNLMeansCL.dll")
LoadDll("C:\Program Files\Hybrid\avisynthPlugins\OpenCL.dll")
LoadCPlugin("C:\Program Files\Hybrid\avisynthPlugins\ffms2.dll")
# loading source: D:\Downloads\Sound of Music clip.mkv
FFVideoSource("D:\DOWNLO~1\SOUNDO~1.MKV",cachefile="D:\Hybrid temp\mkv_42d328cb0f2104fbd18d4e4a004f0124_14604_1_0.ffindex")
# denoising
KNLMeansCL(h=1.80,device_type="gpu")
return last

[Groucho2004]

Please run the script with AVSMeter (http://forum.doom9.org/showthread.php?t=165528) and tell us what speed you get.

Edit: Actually, since you're using Hybrid, the AVSMeter benchmark should be built in.

[Magik Mark]

At this point, what is the difference between NLmeansCL2 and KNLmeans? When I use KNLmeans with hybrid, and set the device to GPU, my GPU works at a 100% load, but processor works at 25%, and the encode takes a long time. When I use NLmeansCL2 and set the device to GPU, the CPU runs at 100%, GPU at about 25%, and the encode is much faster.

(I'm encoding a video to x265, with just this denoising filter added.)

What is the possible explanation for this?

You may want to try staxrip.

[MysteryX]

KNLMeansCL is very intensive and can easily be the bottleneck. If GPU is working at 100%, it *definitely* is the bottleneck and the CPU can't work any faster.

If it is too slow for you, then you have to lower KNLMeans settings.

[Groucho2004]

You may want to try staxrip.He might try mixed vegetables. How would changing the front end that essentially runs the same set of tools improve performance?

[Magik Mark]

Khanattila,

Which settings are good for luma plane only? Is this even possible?

[feisty2]

converttoy8()
knlmeanscl()

[asarian]

This is weird. After yesterday's Windows 10 Anniversary Update, it seems KNLMeansCL no longer registers:


Script evaluation failed:
Python exception: No attribute with the name knlm exists. Did you mistype a plugin namespace?
Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 1491, in vapoursynth.vpy_evaluateScript (src\cython\vapoursynth.c:26897)
File "f:\jobs\thief.vpy", line 10, in <module>
vid = haf.QTGMC (vid, InputType=2, Preset="Very Slow", TR2=3, EdiQual=2, EZDenoise=1.5, NoisePreset="Slower", TFF=True, Denoiser="KNLMeansCL")
File "C:\Program Files\Python35\lib\site-packages\havsfunc.py", line 1072, in QTGMC
dnWindow = core.knlm.KNLMeansCL(noiseWindow, d=NoiseTR, h=Sigma)
File "src\cython\vapoursynth.pyx", line 1105, in vapoursynth.Core.__getattr__ (src\cython\vapoursynth.c:20791)
AttributeError: No attribute with the name knlm exists. Did you mistype a plugin namespace?

[feisty2]

check if OpenCL.dll went missing

[asarian]

check if OpenCL.dll went missing

Sigh. Yes, that was it. Thx. Leave it to MS to do something as insane as to uninstall OpenCL.

Seems the entire old Windows dir (plus subs) got moved to Windows.old; so, now I wonder what else might be missing.

[tormento]

Sigh. Yes, that was it. Thx. Leave it to MS to do something as insane as to uninstall OpenCL.

Seems the entire old Windows dir (plus subs) got moved to Windows.old; so, now I wonder what else might be missing.

I am a "insider". Every build, I had to reinstall video drivers.

[Groucho2004]

I am a "insider". Every build, I had to reinstall video drivers.Insider? Are you privy to information unavailable to us? :D
Maybe "expert" is the word you meant to use?

[feisty2]

Insider? Are you privy to information unavailable to us? :D
Maybe "expert" is the word you meant to use?
insider.windows.com