Deathray - OpenCL GPU accelerated spatial/temporal non-local means de-noising

[Jawed]

Yes, HD 4000 series. It doesn't support a workgroup size of 256. There's a way round this so there's a decent chance I'll make an additional path specifically for that series, when I do other changes I'm planning.

[aNToK]

I'll get around to upgrading the card one of these days but since I don't game much, haven't had the need.

Thanks for the reply and the possible future support.

[bus_labi]

Have a collection of DV home videos - quite a lot of it shot in low light conditions thus noisy. Over last few days extensively tested FFT3Dfilter with various setting to improve the videos.

Found about Deathray and was eager to test it, because had read promising things about NLM denoising.

For testing i wrote simple script that loops 100+ frames thru filter at different settings and outputs filtered video + video with original minus filtered image (contrast enlarged). This gives very good visual of what filter is doing without much pixel peeking. If filter removes only white noise, then only white noise appears in differential image. If there is some structure visible then either that area is not denoised or denoised more or some structure degradation is happening. I am testing only luma channel for now.

Win7 64bit, ATI Radeon HD57xx 1Gb RAM latest versions of Catalyst and Stream SDK installed yesterday. Today experimented with Deathray for an hour or so and not a single crash/problem.

Some observations (based on my noisy dark video sampe).

In low temporal settings - ty = 1,2 there are substantial areas which are not denoised at all. Going up to tY=5 improves, but there are still areas which are less denoised. FFT3DFilter is better in this respect, as there is much less structure in the noise removed. I tested deathray up to ty=5, hy=2 and s = 1.
I could not see difference between hy 1.5 and 2. Will go up from there later.
Deathray was removing about same amount of noise as FFT3Dfilter, but much differently - FFT removed noise was mostly high contrast pixels, where Deathray removed less contrasted pixels and left high contrast noise.
As result FFT3Dfilter produced smoother picture. Higher contrast gives effect of seemingly more detail in deathray'ed picture, but it is less pleasing to look at because of high contrast noise.
Neither of filters at reasonable settings take away any structure.
I noticed that FFT3 tends to denoise more areas which Deathray tends to avoid and tested FFT3 on deathrayed video. This produced most pleasing result as FFT3 removed high contrast pixels. Both filters seem somehow to complement each other. As of now i would not use Deathray alone because of the high contrast noise. On neither of filters i used extreme settings as i prefer some grain to complete smootheness.

I upload some screenshots, but static images do not really give a good impression - moving image is much better and informative. Here is simple script i use for testing.

fr = 100           # number of frames to be looped
sigma_start  = 0.5 # start value for sigma range
sigma_incr   = 0.1 # sigma increment
sigma_finish = 1.0 # end value for sigma range
hY_start     = 2.0   # denoise strength luma
hY_finish       = 2.0   # denoise strength luma
tY           = 5   # temporal radius luma

c = DirectShowSource ("2001 03 229.avi", audio=false).viewfields.crop(0,0,-0,-288).trim (0,fr).grayscale.converttoyv12
gray = BlankClip(c, length=fr, color = $808080)
cf = BlankClip(c, 0)
co = BlankClip(c, length=0)
co = Stackvertical (co,co)

GScript ("""
          while (hY_start <= hY_finish)
                    {
                    ssigma = sigma_start
                    while (ssigma <= sigma_finish)
                                        {
                                        if (FrameCount(cf) > 0) {NOP} else {cr = gray}
                                        cf  = deathray (c, hy = hY_start, huv = 0.0, ty = tY, s =ssigma)
                                        cd  = subtract (c, cf)
                                        Stackvertical ( \
						cf.Subtitle("sigma = " + String(ssigma) + "\n hY =  " + String(hY_start)+ "\n tY =  " + String(tY), align=6, lsp=0), \
						cd.histogram(mode="luma").Subtitle("Difference original - filtered", align=9, lsp=0))
                                        co = co + last
                                        ssigma = ssigma + sigma_incr
                                        }     
          hY_start = HY_start + 1 }
          """)
return (co)

[Jawed]

Originally Posted by bus_labi

For testing i wrote simple script that loops 100+ frames thru filter at different settings and outputs filtered video + video with original minus filtered image (contrast enlarged).

Your automated parameters technique is nice.

This gives very good visual of what filter is doing without much pixel peeking. If filter removes only white noise, then only white noise appears in differential image. If there is some structure visible then either that area is not denoised or denoised more or some structure degradation is happening. I am testing only luma channel for now.

I've been doing something very similar, testing with the histogram("luma") technique or with a simple levels call to magnify the subtraction.

Though I've been doing it by hand, single stepping through clips and tweaking parameters. My reference filter, for what it's worth, is FizzKiller.

Have you evaluated motion-compensated MDegrain (from MVTools 2)? It's slow, but extremely good.

Obviously I also make encodes and watch them, too.

Win7 64bit, ATI Radeon HD57xx 1Gb RAM latest versions of Catalyst and Stream SDK installed yesterday. Today experimented with Deathray for an hour or so and not a single crash/problem.

There is a way to crash Deathray. I'm surprised your script hasn't caused it to crash, to be honest. I don't understand the reason for the crash and I can't reliably reproduce it. But repeatedly hand-tweaking Deathray parameters and single-frame stepping tends to crash it after a while, sometimes.

Some observations (based on my noisy dark video sampe).

In low temporal settings - ty = 1,2 there are substantial areas which are not denoised at all. Going up to tY=5 improves, but there are still areas which are less denoised. FFT3DFilter is better in this respect, as there is much less structure in the noise removed. I tested deathray up to ty=5, hy=2 and s = 1.

I find the "naked" NLM, which is what version 1.00 is, inadequate. This is the original algorithm, first described by Buades, Coll and Morel - with the caveat that the sampling region is limited, instead of being the entire image. (It turns out there's very little to be gained by sampling the entire image). Actually this algorithm is tweaked with a re-weighting for the target pixel that they described later.

It's too weak at settings that are required not to destroy fine detail.

The most recent "NVidia Debug" version contains a crude averaging with the original image. This is considerably better, but the averaging I'm using is not the averaging that is described here:

http://hal.archives-ouvertes.fr/docs...le_revised.pdf

I'm currently getting my head round that.

The NVidia Debug version (110120004) is preferable to 1.00.

I could not see difference between hy 1.5 and 2. Will go up from there later.

Ultimately, one has to balance the 3 key parameters, h, sigma and temporal-radius.

The sample-radius factor, x, is also useful, but has to be used with care (turn h up to 1000 and set x=3 to see what I'm talking about). A planned change to the filter incidentally makes x safer, though truthfully with sane values of h and s, x=2 or 3 is quite safe (and slow...).

I think temporal 5 is too strong.

I hope the regression correction I'm working on is worth it. It should require re-evaluation of settings...

Deathray was removing about same amount of noise as FFT3Dfilter, but much differently - FFT removed noise was mostly high contrast pixels, where Deathray removed less contrasted pixels and left high contrast noise.
As result FFT3Dfilter produced smoother picture. Higher contrast gives effect of seemingly more detail in deathray'ed picture, but it is less pleasing to look at because of high contrast noise.
Neither of filters at reasonable settings take away any structure.
I noticed that FFT3 tends to denoise more areas which Deathray tends to avoid and tested FFT3 on deathrayed video. This produced most pleasing result as FFT3 removed high contrast pixels. Both filters seem somehow to complement each other. As of now i would not use Deathray alone because of the high contrast noise. On neither of filters i used extreme settings as i prefer some grain to complete smootheness.

Make sure you try MVTools 2 techniques. You can also try motion-compensated FFT3DFilter and of course there's FFT3DGPU, which can also be motion-compensated.

Generally I would say don't rush and use lots of test clips.

---

A little sidenote on one aspect of the NVidia "problems". The problem with the max() function turns out to be a difference between OpenCL 1.0 and OpenCL 1.1.

[jclampy]

Hi, Would really like to try this filter out but the links are dead.

[ChaosKing]

Hi, Would really like to try this filter out but the links are dead

DL-Link

[jclampy]

Thankyou very much ChaosKing.

[Jawed]

Does it work? I just made a long post that got lost

[Jawed]

Rather than try to recreate the post I just lost, I'll just briefly link this, which is DeathrayNV110120004.zip:

http://www.mediafire.com/?yyiaqbqp77h236p

This is just the DLL. It is a slightly different algorithm (which I think is better) and it solves the original problem with failure on NVidia.

This version of the DLL always creates a deathray.log file. If there are problems this file should help me help you.

I have a caveat: when I wrote this OpenCL code AMD's and NVidia's compilers were a little bit loose. I think they are stricter now and there's a chance this will not compile with recent drivers.

I didn't fix this issue due to a general lack of interest, so this may be more of an experiment than you wanted. Fingers-crossed!

EDIT: oh yeah, my website is down because my host took it down for going over the 250MB daily limit

[xiulet]

any news about this plugin? :O

thanks

[Jawed]

No news, dead due to lack of interest (mostly my lack of interest, it seems).

My lack of interest is due to the fact that I prefer the results from MDegrain.

My website is available. There are 3 files:

Deathray_1_00.zip - see post 1
Deathray_1_00_DLL.zip - see post 1
DeathrayNV110120004.zip - see post 36

Please note that the source code I've provided in post number 1 does not match the source code for post 36.

Please read the entire thread to learn how best to use this.

For the best quality I recommend DeathrayNV110120004.zip. This contains a tweaked algorithm (hence the improved quality) as well as a fix to ensure it works on NVidia. This DLL works with AMD, too. The downside with this version is that it creates a deathray.log file in the same folder as the Avisynth script.

I have no idea whether Deathray works with recent AMD or NVidia drivers. In theory the newest drivers will reject my code because they are more strict.

Obviously, anyone is welcome to play with the code.

[xiulet]

Originally Posted by Jawed

No news, dead due to lack of interest (mostly my lack of interest, it seems).

My lack of interest is due to the fact that I prefer the results from MDegrain.

My website is available. There are 3 files:

Deathray_1_00.zip - see post 1
Deathray_1_00_DLL.zip - see post 1
DeathrayNV110120004.zip - see post 36

Please note that the source code I've provided in post number 1 does not match the source code for post 36.

Please read the entire thread to learn how best to use this.

For the best quality I recommend DeathrayNV110120004.zip. This contains a tweaked algorithm (hence the improved quality) as well as a fix to ensure it works on NVidia. This DLL works with AMD, too. The downside with this version is that it creates a deathray.log file in the same folder as the Avisynth script.

I have no idea whether Deathray works with recent AMD or NVidia drivers. In theory the newest drivers will reject my code because they are more strict.

Obviously, anyone is welcome to play with the code.

ok thanks you for the info my friend

you mean MVTools+MDegrain...¿?

[Keiyakusha]

Originally Posted by Jawed

My lack of interest is due to the fact that I prefer the results from MDegrain.

So how about porting Mdegrain to gpu then? Porting motion search proved to be possible and if i'm not mistaken its opensource (since mvtools is opensource). But they had different purpose and ported only mflowfps(and renderer part of that is closed source). However they claim their motion search is better than original, which I so far can't see, unfortunately. So maybe it requires some tweaks.

[Jawed]

Originally Posted by Keiyakusha

So how about porting Mdegrain to gpu then?

One of those things I just didn't get round to, and now behind other things.

I logged on to this forum today for the first time in quite a while and discovered VapourSynth.

Previous time I logged in, I discovered people are doing 10-bit encodes.

I'm out of touch.

Porting motion search proved to be possible and if i'm not mistaken its opensource (since mvtools is opensource). But they had different purpose and ported only mflowfps(and renderer part of that is closed source). However they claim their motion search is better than original, which I so far can't see, unfortunately. So maybe it requires some tweaks.

Even just downloading the code and setting up a development project is more effort than I want to get involved in, right now. Sorry.

[Jawed]

I've just realised I could make a git version of this:

Deathray on github.com

This is v1.00 without any of the bug fixes or improvements that I created during the debugging work done for NVidia users.

It's also for an ancient SDK.

It will definitely require work to get it working with latest drivers and/or SDKs.

---

And for those of you who understand Git branches, the branch called FindFixes is my attempt to identify the code that makes the DeathrayNV110120004 version.

[lansing]

i have a question.

I just brought a new nvidia graphic for encoding purposes, and when I did test on both deathray and nlmeanscl with avsmeter, the default nlmeanscl setting gave me 252fps with 70% gpu loaded, while deathray in default setting gave me 47fps and over 90% gpu loaded. It makes me wonder what's the difference between the two to have a huge different in speed like this?

[Jawed]

Originally Posted by lansing

i have a question.

I just brought a new nvidia graphic for encoding purposes, and when I did test on both deathray and nlmeanscl with avsmeter, the default nlmeanscl setting gave me 252fps with 70% gpu loaded, while deathray in default setting gave me 47fps and over 90% gpu loaded.

Wow, it still works? I was expecting newer drivers to stop it working.

It makes me wonder what's the difference between the two to have a huge different in speed like this?

Looking at my notes, I see for 1920x798 video I had about 32fps on HD5870. This is "compute-limited", i.e. memory accesses have no impact on performance. For standard definition I had 122fps theoretical, but memory accesses meant that this was only about 25fps in practice, so I needed to watch 5 videos in parallel (in Media Player Classic) to get 100% GPU usage.

Alternatively a temporal radius of 2 would produce 100% GPU usage (tY=2, tUV=2) on standard definition clips.

Because I coded for AMD, it uses on-chip memory in a way that's potentially "unfriendly" towards NVidia. In other words I took advantage of the very large register capacity on AMD and I wouldn't be surprised if NVidia cards ran out of registers. The newest NVidia cards (Kepler) have much better register capacity, so in theory even if this was a problem a few years ago, it shouldn't be a problem for the latest NVidia cards.

Due to lack of interest, no performance comparisons were made between AMD and Nvidia, so I don't know if this is really true or not.

I don't know the size of the window in nlmeanscl. Deathray uses a 7x7 size, and the larger it is the more effort involved.

Also I don't know the search area in nlmeanscl. Deathray does not search the entire frame to get other 7x7 windows, it only searches near the pixel being filtered (it looks at 48 other 7x7 windows). Again, searching more of the frame is more effort. In fact searching the entire frame is counter-productive, it is better to search a limited area near the pixel being filtered.

I don't think there's much point in the nlmeans algorithm. It is only mildly useful when used with temporal settings and pretty much a waste of time without temporal settings. I prefer MDegrain, which is why I don't use Deathray.

Do you like the results of either Deathray or nlmeanscl?

Do you prefer it when you use temporal settings?

Have you used MDegrain?

[lansing]

mdegrain didn't fit my taste in which it manipulated the image too much. It's going to be either a hit or miss. On moving scene, sometime it gives a good result, and sometime it smooths out the original shape.

I'd did some test with this sample, i uploaded here:
sample

Both deathray and nlmeanscl pretty much gave the same result, except in some low light area with thin lines, deathray at default desaturated the color by a little.

This sample also gives the reason why i don't use mdegrain, notice the panning scene from 2-6 second, there're some spider webs across the screen. When i used smdegrain, it pretty much wiped out half of them. While deathray and nlmeanscl still preserved them.

well on the temporal side, smdegrain did a better job at calming the grains.

[Jawed]

I trimmed off the first frame, which is broken:

http://www.mediafire.com/download/0w..._sample_JA.mp4

I have to say I'm horrified that apparently recent anime can be such a disaster zone.

[Jawed]

Updated the first post with information on version 1.01. Please note that www.cupidity.f9.co.uk is not being maintained with source or DLL. All mirrors in this thread should be considered defunct.