KNLMeansCL: OpenCL NLMeans de-noising algorithm [2018-01-29]

[Myrsloik]

I'll try, alternatively I could use some SIMD instructions but I do not have much experience.

Are you certain the openmp stuff makes it significantly faster? It's only used on trivial loops so if it's run in a multithreaded way with other filters you'll run out of memory bandwidth anyway.

[Khanattila]

Are you certain the openmp stuff makes it significantly faster? It's only used on trivial loops so if it's run in a multithreaded way with other filters you'll run out of memory bandwidth anyway.

EDIT.

AviSynth. (720p, d=0, a=2, "GPU")

RGB (no buffer) ==> 95.57 fps
YUV (buffer - 1 thread) ==> 69.64 fps
YUV (buffer - 4 thread) ==> 82.02 fps

[Khanattila]

Libswscale? It could be faster.

[Myrsloik]

Why not do the packing and unpacking in the opencl code?

[Khanattila]

Why not do the packing and unpacking in the opencl code?

So, it's like that.

__kernel
void NLM_pack(__read_only image2d_t R, __read_only image2d_t G, __read_only image2d_t B, __write_only image2d out) {

sample_t smp = ...;
int2 coord2 = (int2) (get_global_id(0), get_global_id(1));
float r = read_imagef(R, smp, coord2).x;
float g = read_imagef(G, smp, coord2).x;
float b = read_imagef(B, smp, coord2).x;
float4 val = (float4) (b, g, r, 1.0f);
write_imagef(out, coord2, val);

}

I'll try.

[Khanattila]

720p.

AviSynth YUV420P8 (no buffer): 148 fps
AviSynth YUV420P16 (OpenMP buffer): 111 fps
AviSynth RGB32 (no buffer): 96 fps
AviSynth YUV444P8 (OpenMP buffer): 81 fps
AviSynth YUV444P16 (OpenMP buffer): 57 fps

VapourSynth YUV420P8 (no buffer): 121 fps
VapourSynth YUV420P16 (no buffer): 112 fps
VapourSynth RGB24 (OpenCL buffer): 84 fps
VapourSynth YUV444P8 (OpenCL buffer): 85 fps
VapourSynth YUV444P16 (OpenCL buffer): 70 fps


It seems to work well.

[videoh]

@tormento

Here is a 64-bit test version of DGDecNV built with CUDA 7.5. Please advise your results.

http://rationalqm.us/misc/tormento_64.zip

Note that sm_1x support is dropped by CUDA 7.5.

[YAFU]

@Khanattila, With what parameters in KNLMeansCL you get those fps?
Which is your graphics card?

[Khanattila]

@Khanattila, With what parameters in KNLMeansCL you get those fps?
Which is your graphics card?

vs r27, avs 2.6.0
default... d=0, a=2, with a GTX 760

https://github.com/Khanattila/KNLMeansCL/releases/tag/v0.6.4

[YAFU]

I get about 24-28 fps with "imwri" and also with "ffms2" in movies, 720p. GTX 960. Same version of VS. Using vspipe from terminal.
Supposedly these two graphics cards are similar. Could this be because some component in Linux?

[Khanattila]

I get about 24-28 fps with "imwri" and also with "ffms2" in movies, 720p. GTX 960. Same version of VS. Using vspipe from terminal.
Supposedly these two graphics cards are similar. Could this be because some component in Linux?

I have test with BlankClip.
You must try to reduce the impact of other plugins.

core.std.BlankClip(width=1280, height=720, format=vs.YUV420P8)

[YAFU]

Ok, now I am using this script:
import vapoursynth as vs
core = vs.get_core()
clp = core.std.BlankClip(width=1280, height=720, format=vs.YUV420P8, length=1000)
clp = core.knlm.KNLMeansCL(clp, d=0, a=2, device_type="GPU")
clp.set_output ()


and "vspipe --y4m script.vpy - | ffmpeg -i pipe: -vcodec libx264 -crf 5 encoded.mkv"

That way I get 123 fps. I do not know if that result is with or without that "buffer" you wrote. I guess "no buffer"

[Khanattila]

Ok, now I am using this script:
import vapoursynth as vs
core = vs.get_core()
clp = core.std.BlankClip(width=1280, height=720, format=vs.YUV420P8, length=1000)
clp = core.knlm.KNLMeansCL(clp, d=0, a=2, device_type="GPU")
clp.set_output ()


and "vspipe --y4m script.vpy - | ffmpeg -i pipe: -vcodec libx264 -crf 5 encoded.mkv"

That way I get 123 fps. I do not know if that result is with or without that "buffer" you wrote. I guess "no buffer"

OpenCL support only packed format.

not supported: | BBBBBBBB | GGGGGGGG | RRRRRRRR | AAAAAAAA |
supported: | BGRA | BGRA | BGRA | BGRA | BGRA | BGRA | BGRA | BGRA |

So, a pre-filter is required.

[loneboyz]

KNLMeansCL, this plugin was very wonderful, thank author!

I've tested it with source PAL DVD, that's very noise and result was nice

But I've having a problem while encoding, please give me advice

My source have all 139.512 frames, when run x264 can only encode about below 32.700 frames then appear error

Avisynth error at frame 32792
KNLMeansCL: AvisynthGetFrame error!

If I use Trim in script cut about 32.500 frame then encode, this error don't appear

This is my script I used:
DGDecode_mpeg2source("F:\Setup\Encode\Work\nk.d2v")

y = ConvertToY8().Crop(0,1,0,-1).Padding(0,1,0,1)
u = UToY8().Crop(0,1,0,-1).Padding(0,1,0,1)
v = VToY8().Crop(0,1,0,-1).Padding(0,1,0,1)

YToUV(u,v,y)

#deinterlace
#~ AssumeTFF()
interp = nnedi3(field=1)
tdeint(mode=0,order=1,field=1,edeint=interp,slow=2,emask=TMM(mode=0,order=1,field=1)).Vinverse()

BlindDeHalo3(3,3,125,PPmode=-1) #,PPmode=-3,PPlimit=4
#DeHalo_alpha(ss=2,rx=2.5,ry=2.5,darkstr=0,highsens=100)

KNLMeansCL(D=1, A=1, h=4.5)
#~ f3kdb(dither_algo=3, Y=68, range=15, grainY=0, grainC=0, dynamic_grain=true)

GradFun3 (thr=0.42, smode=2)

FineSharp()

Some info my PC:
- Windows 7 Ultimate 64bit
- Chip intel core i5-4210U @1.7GHz (4CPUs)
- Graphic onboard Intel HD 4400, memory 1696 MB
- Ram 4G
- x264 64b + avs2yuv
- Avisynth 2.6.0.6 32b
- OpenGL 4.0 v9.18.10.3234

Thanks!

[Groucho2004]

But I've having a problem while encoding, please give me advice

My source have all 139.512 frames, when run x264 can only encode about below 32.700 frames then appear error
Run the script through AVSMeter (http://forum.doom9.org/showthread.php?t=165528) with the "-log" switch. Post the log (on pastebin or similar).

[loneboyz]

Result after run AVSMeter

http://8.t.imgbox.com/6AzMLKnF.jpg (http://imgbox.com/6AzMLKnF)

Log file: nk5.log (http://www.mediafire.com/download/5modv8akfa62n5u/nk5.log)

I can't use pasebin because file sỉze larger 2.8 MB. Tks!

[Groucho2004]

Result after run AVSMeter
OK, I just wanted to see if the same happens without the encoder in the chain.
Since it seems to stop at exactly the same frame every time I would say it's related to the source. Remove all filters except the source filter and let it run.
If the problem remains, try another source filter, like LWLibavVideoSource.

[Myrsloik]

Another code question...

Why not use opencl for packing and unpacking in avisynth too?

[loneboyz]

I've tried to split source, use Trim func, 32.000 frames per one and encode many times, everything's fine so reason not related to the source

I've also replaced KNLMeansCL with another denoise plugin (zzz_denoise), it's seem working well when encode whole source's frames

I'll try again as you guide. Tks!

[Khanattila]

Another code question...

Why not use opencl for packing and unpacking in avisynth too?

Version 0.7.0, it should not be urgent.

[Khanattila]

I've tried to split source, use Trim func, 32.000 frames per one and encode many times, everything's fine so reason not related to the source

I've also replaced KNLMeansCL with another denoise plugin (zzz_denoise), it's seem working well when encode whole source's frames

I'll try again as you guide. Tks!

It is certainly a problem of OpenCL.
Somewhere sometime somehow, something went wrong.

[loneboyz]

Thanks to Groucho2004, Khanattila!

I'd changed para device_type, value set GPU and processing of encode reached over 60.000 frames but not appear error

Although I only have the onboard video card, I don't know the quality will be changed how

[Khanattila]

Thanks to Groucho2004, Khanattila!

I'd changed para device_type, value set GPU and processing of encode reached over 60.000 frames but not appear error

Although I only have the onboard video card, I don't know the quality will be changed how
The quality is the same. Change only the speed.

[tormento]

@tormento
Here is a 64-bit test version of DGDecNV built with CUDA 7.5. Please advise your results.

Thanks for your work! However a 32bit version is what I need. :thanks:

[videoh]

http://rationalqm.us/misc/tormento_32.zip

[tormento]

http://rationalqm.us/misc/tormento_32.zip (http://rationalqm.us/tormento_32.zip)
Wrong address if clicked, correct one
http://rationalqm.us/misc/tormento_32.zip works

[tormento]

Here is a 64-bit test version of DGDecNV built with CUDA 7.5. Please advise your results.
Tried 32bit version. No way.

I suppose there must be something wrong between the combination of the two plugin together.

Any other OpenCL filter I can apply to test?

[Groucho2004]

Tried 32bit version. No way.

I suppose there must be something wrong between the combination of the two plugin together.

Any other OpenCL filter I can apply to test?
Having had a quick look at the Nvidia Geforce forum, the feedback on the Win10 drivers seems to indicate that they are an utter mess. Maybe that's the problem.

[tormento]

nnedi3ocl (http://forum.doom9.org/showthread.php?t=169766)
Thanks, however it's so old I can't set OpenCL GPU priority.

If I remove Intel GPU from bios and leave Nvidia only, it simply doesn't work.

http://i.imgur.com/MhdGfRNm.jpg (http://imgur.com/MhdGfRN)

Now I'm going to install Nvidia developer drivers (https://developer.nvidia.com/opengl-driver) with Windows 10 and OpenCL extension support.

Edit: neither developer 355.97 does the magic. KNLMeansCL works perfectly without DGDecNV.

[Khanattila]

720p.

AviSynth YUV420P8 (no buffer): 148 fps
AviSynth YUV420P16 (OpenMP buffer): 111 fps
AviSynth RGB32 (no buffer): 96 fps
AviSynth YUV444P8 (OpenMP buffer): 81 fps
AviSynth YUV444P16 (OpenMP buffer): 57 fps

VapourSynth YUV420P8 (no buffer): 121 fps
VapourSynth YUV420P16 (no buffer): 112 fps
VapourSynth RGB24 (OpenCL buffer): 84 fps
VapourSynth YUV444P8 (OpenCL buffer): 85 fps
VapourSynth YUV444P16 (OpenCL buffer): 70 fps


It seems to work well.

Another code question...

Why not use opencl for packing and unpacking in avisynth too?

v0.6.4 -> v0.7.0-beta

AviSynth YUV420P8: 148 fps -> 153 fps. (+3%)
AviSynth YUV420P16: 111 fps -> 132 fps. (+19%)
AviSynth RGB32: 96 fps -> 97 fps. (+1%)
AviSynth YUV444P8: 81 fps -> 96 fps. (+19%)
AviSynth YUV444P16: 57 fps -> 77 fps. (+35%)

VapourSynth YUV420P8: 121 fps -> 146 fps. (+21%)
VapourSynth YUV420P16: 112 fps -> 133 fps. (+19%)
VapourSynth RGB24: 84 fps -> 96 fps. (+14%)
VapourSynth YUV444P8: 85 fps -> 95 fps. (+12%)
VapourSynth YUV444P16: 70 fps -> 78 fps. (+11%)


Maybe there was still something to be exploited.

[YAFU]

@Khanattila, Is it a known issue that imperfections appear when fast movements at the video edges when you use d=0?

You see here which is within the yellow line:
http://www.image-share.com/upload/3056/274.jpg

This is the original video:
https://www.dropbox.com/s/xcz9u11ifk7cvmy/test.mp4?dl=0

This is the script I use:
import vapoursynth as vs
core = vs.get_core()

##
##ffmpeg with ffms2
clp = core.ffms2.Source(source='/media/test.mp4')
##

clp = core.knlm.KNLMeansCL(clp, d=0, a=12, s=4, h=3.2, device_type="GPU")
clp.set_output ()

from terminal:
vspipe --y4m script.vpy - | ffmpeg -i pipe: -vcodec libx264 -crf 10 encoded.mkv

[feisty2]

do some small pads first

[YAFU]

I searched on google and KNLMeansCL manual to try to know what you refer with "small pads". You mean "a" or "s" value? It seems that if d=0 the problem disappears with very low values of "a", but is very little noise reduction. But, what do you mean with "first"? Some kind of two passes? I really do not know how to do that.

[Groucho2004]

I searched on google and KNLMeansCL manual to try to know what you refer with "small pads".
Possibly something like this:
padding = 16
AddBorders(padding, padding, padding, padding)
KNLMeansCL()
Crop(padding, padding, -padding, -padding)

[YAFU]

Thanks, but sorry. I can't find nothing in Google about that in VapourSynth, is this for avisinth?. I only found something "core.std.AddBorders", but I have no idea how to implement it.
Anyway, what does Crop or AddBorders? The problem will be solved this way, or it just means cutting or hide the problematic part of the video where the imperfections occur?

[Khanattila]

do some small pads first

Possibly something like this:
padding = 16
AddBorders(padding, padding, padding, padding)
KNLMeansCL()
Crop(padding, padding, -padding, -padding)


Out-of-range image coordinates will return a border color, if it works I add it.

[Are_]

Hi YAFU, give it a try: knlm.py (https://gist.github.com/4re/4788aa29cee72ac78361)

import knlm
...
clip = knlm.knlm_yuv(clip)


This works the same regular knlmeanscl with some small differences: d, a, s and h arguments now accept arrays, one value for each plane, and there is an extra option planes, it behaves the same as regular vapoursynth filters. It also adds correct padding duplicating last line and crops it afterwards.

Beware it by default filters all three planes, so if you only want to filter luma you should call it with "planes=0"

[Khanattila]

Thanks, but sorry. I can't find nothing in Google about that in VapourSynth, is this for avisinth?. I only found something "core.std.AddBorders", but I have no idea how to implement it.
Anyway, what does Crop or AddBorders? The problem will be solved this way, or it just means cutting or hide the problematic part of the video where the imperfections occur?

padding = 4 #It should be equal to 'a'
clip = core.std.AddBorders(clip, padding, padding, padding, padding)
clip = core.knlm.KNLMeansCL(clip)
clip = core.std.CropRel(clip, padding, padding, padding, padding)

[Groucho2004]

padding = 4 #It should be equal to 'a'
What about this:
int a Set the radius of the search window. A=0 uses 1 pixel, while D=1
use 9 pixels and son on. Usually, larger it the better the result
of the denoising. Spatial size = (2 * A + 1)^2.
Total search window size = temporal size * spatial size.

[Khanattila]

What about this:
A=0 uses 1 pixel, while A=1 use 9 pixels and son on, my typo.

To clarify padding should be 'a' + 's'. But use only 'a' should be enough.

Take pixel p0(x, y). If 'a'=2 pixel p0 uses for example pixel p1(x+2, y+2).
There are two similarity windows: first centered in p0(x, y) and second centered in p1(x+2, y+2).
If 's'=4 first similarity windows uses p(x-4, y-4), ..., p(x+4, y+4).
Second similarity uses p(x+2-4, y+2-4), ..., p(x+2+4, y+2+4).

EDIT. Good examples here (http://www.3dgep.com/texturing-lighting-directx-11/#Address_Mode).

[YAFU]

Thank you very much to all!

@Are_, I have no idea about programming/scripting, you give me a time to try to learn about this script and then experiment a little. Thanks.

@Khanattila, Ok, if I work with this script all is working fine now:
import vapoursynth as vs
core = vs.get_core()

##
##ffmpeg with ffms2
clp = core.ffms2.Source(source='/media/test.mp4')
##
padding = 24
clp = core.std.AddBorders(clp, padding, padding, padding, padding)
clp = core.knlm.KNLMeansCL(clp, d=0, a=24, s=4, h=3.2, device_type="GPU")
clp = core.std.CropRel(clp, padding, padding, padding, padding)
clp.set_output ()

Now, I need to implement that you say in this script I have taken from blenderartists forum:
https://dl.dropboxusercontent.com/u/34973756/distfiles/1-hard-denoise.vpy

I tried:
http://www.pasteall.org/61211

but I think I'm doing wrong because the resulting video is bigger, 1328 x 768. That does not happen with the first script above applied directly to video with ffms2 (it remains in 1280x720). Surely I am confused with "ret" and "double"

I would appreciate if someone can check where the error is.

[feisty2]

@Khanattila
it works, I been doing it for a long time
@Groucho2004 and YAFU
don't "addborders", do some "padding"

def padding (src, left=0, right=0, top=0, bottom=0):
core = vs.get_core ()
w = src.width
h = src.height
clip = core.fmtc.resample (src, w+left+right, h+top+bottom, -left, -top, w+left+right, h+top+bottom, kernel="point", fulls=True, fulld=True)
return clip

clp = padding (clp,a,a,a,a)
clp = core.knlm.KNLMeansCL (clp, xxx)
clp = core.std.CropRel (clp, a, a, a, a)

[feisty2]

@YAFU
http://www.pasteall.org/61212

[feisty2]

Hi YAFU, give it a try: knlm.py (https://gist.github.com/4re/4788aa29cee72ac78361)

import knlm
...
clip = knlm.knlm_yuv(clip)


This works the same regular knlmeanscl with some small differences: d, a, s and h arguments now accept arrays, one value for each plane, and there is an extra option planes, it behaves the same as regular vapoursynth filters. It also adds correct padding duplicating last line and crops it afterwards.

Beware it by default filters all three planes, so if you only want to filter luma you should call it with "planes=0"

Expr ("x 0.5 +") is required when u shift chroma planes to Y, filters might clamp pixels to [0.0, 1.0] somewhere and the <0 part of the chroma will be lost
yeah, I'm talking about float formats...

[Khanattila]

Thank you very much to all!

@Are_, I have no idea about programming/scripting, you give me a time to try to learn about this script and then experiment a little. Thanks.

@Khanattila, Ok, if I work with this script all is working fine now:
import vapoursynth as vs
core = vs.get_core()

##
##ffmpeg with ffms2
clp = core.ffms2.Source(source='/media/test.mp4')
##
padding = 24
clp = core.std.AddBorders(clp, padding, padding, padding, padding)
clp = core.knlm.KNLMeansCL(clp, d=0, a=24, s=4, h=3.2, device_type="GPU")
clp = core.std.CropRel(clp, padding, padding, padding, padding)
clp.set_output ()

Now, I need to implement that you say in this script I have taken from blenderartists forum:
https://dl.dropboxusercontent.com/u/34973756/distfiles/1-hard-denoise.vpy

I tried:
http://www.pasteall.org/61211

but I think I'm doing wrong because the resulting video is bigger, 1328 x 768. That does not happen with the first script above applied directly to video with ffms2 (it remains in 1280x720). Surely I am confused with "ret" and "double"

I would appreciate if someone can check where the error is.

ret = core.knlm.KNLMeansCL(ret, d=0, a=24, s=4, h=3.2 , device_type="GPU")

Or use feisty2 code.

[Khanattila]

@Khanattila
it works, I been doing it for a long time
@Groucho2004 and YAFU
don't "addborders", do some "padding"

def padding (src, left=0, right=0, top=0, bottom=0):
core = vs.get_core ()
w = src.width
h = src.height
clip = core.fmtc.resample (src, w+left+right, h+top+bottom, -left, -top, w+left+right, h+top+bottom, kernel="point", fulls=True, fulld=True)
return clip

clp = padding (clp,a,a,a,a)
clp = core.knlm.KNLMeansCL (clp, xxx)
clp = core.std.CropRel (clp, a, a, a, a)

I thought that edge of the image was better than using the border color. I was wrong.

However, it is very simple things to code.

[~ VEGETA ~]

I would like to ask the following:

1- Is it faster than TNLMeans? Dfttest? because the normal tnlmeans is slow as hell!

2- I have a core2due laptop (and a more powerful server)... Is it supported? I don't see a suitable driver.

[Khanattila]

I would like to ask the following:

1- Is it faster than TNLMeans? Dfttest? because the normal tnlmeans is slow as hell!

2- I have a core2due laptop (and a more powerful server)... Is it supported? I don't see a suitable driver.

1) With the same arguments it is ~6/7 time faster. But I have a quad core and original TNLMeans is single thread.
KNLMeansCL(device_type="CPU") - 1.910 fps
TNLMeans(2, 2, 4, 4, 0, 0) - 0.305 fps

The main advantage is the ability to use the GPU.

2) If Intel Core 2 is not more supported by intel you can use the AMD APP SDK v2.9 http://developer.amd.com/tools-and-sdks/opencl-zone/amd-accelerated-parallel-processing-app-sdk/

[YAFU]

Thanks feisty2, Khanattila.
So, This padding thing could be implemented within KNLMeans as an option?

@feisty2, I get the following error message with the script:
Failed to evaluate the script:
Python exception: expected an indented block (script.vpy, line 32)
Traceback (most recent call last):
File "vapoursynth.pyx", line 1468, in vapoursynth.vpy_evaluateScript (src/cython/vapoursynth.c:23689)
File "/media/script.vpy", line 32
core = vs.get_core ()
^
IndentationError: expected an indented block

[Are_]

Because messages in the forum don't respect white spaces, except when you use "code" tags, and python relies on them to format the code, add them as needed in that function.

def padding (src, left=0, right=0, top=0, bottom=0):
core = vs.get_core ()
w = src.width
h = src.height
clip = core.fmtc.resample (src, w+left+right, h+top+bottom, -left, -top, w+left+right, h+top+bottom, kernel="point", fulls=True, fulld=True)
return clip

a = 32
clp = padding (clp,a,a,a,a)
clp = core.knlm.KNLMeansCL (clp, a=a)
clp = core.std.CropRel (clp, a, a, a, a)