Smooth in time and space?

[bb]

Well, I don't really know if there is already a filter available covering this idea:

The smoothing/denoising filters I know are either spatial or temporal filters. What about smoothing spatially AND temporally at the same time?

E.g. you could average a 3x3x3 block of 3 consecutive frames, i.e. the middle pixel would be averaged by its neighbours in the same frame and the neighbour pixels of the neighbour frames in the time dimension.

I hope the explanation wasn't too complicated to understand, but I think using existing averaging algorithms on a three-dimensional cube instead of just looking at a two-dimensional plane would not be too complicated to implement.

What do you think?

bb

[Acaila]

AFAIK that is exactly what the General Convolution 3D filter does. However it's made for VDub, not Avisynth.

Something like this can easily be achieve by just chaining two filters, but the problem is that they don't work intelligently to determine edges and sharp lines etc.

[poptones]

AVISynth has generalconvolution. Richard added it himself to the last couple releases. And you can chain them pretty easily, as well as add as much intelligence as you like. In addition to using edge detection filters to create masks, you can use lighten and darken via layer().

There are so many ways to do this now it's become difficult to decide which one is best for an application. There are the included filters and the plugins - between those and being able to layer clips using a variety of operations you have pretty much unlimited possibilities. My biggest gripe right now is with finding a way to remove MPEG artifacts without screwing things up too badly. And I'm working on that one, too.

[bb]

Hmm, the 3D averaging I was thinking of is not the same as chaining a spatial and a temporal filter. But the General Convolution 3D filter seem to be the solution (don't know if it has scene change detection, which would be nice). I'll try that one.

Is there a General Convolution 3D filter available for AviSynth, too?

If not, maybe someone feels challenged to implement it. There's also much room for tweaking, like edge detection, etc.

I think this could be a superior method of getting rid of noise in complicated sources like DV.

bb

[avih]

i think a specific 3d convolution filter should be written since it would be much faster, especially since it using 27 different values. also, it should be adaptive.

it can be adaptive in 2 ways:
1. apply the convolution result if it's not 'further' than 'x' from the original pixel value.

2. take into account only pixels that are not 'further' in value from the priginal pixel by 'x'

the 2nd method gives better results imo, but is considerably slower because of the 'if's during the convolution process itself.

it SHOULD be a great filter though.

[Didée]

Dreaming of SmartSmootherHiQ in 3D ...

[manono]

And then instead of taking twice the time to encode, it can take four times the amount.:)

[sh0dan]

Not entirely so, because the temporal check would not be a complete 3D-check, it would only check the current pixel against the same pixel in the previous frame (one check more per pixel).

The idea is quite neat - the algorithm could work like this:
*Calculate the blurred pixel as usual.
*Compare the original pixel to the previous frame. The bigger the difference, the less will the pixel get blurred.

That will make low-freqency noise even easier to detect.

Might actually work!

I've also recived a great suggestion for adaptive threshold, that adapts the threshold to the luma value of the current pixel - so there are lots of improvements in the works. First priority is still an MMX-optimized YUV version though.

[manono]

Hi Klaus-

SSHiQ is far and away the best spatial (and someday spatial-temporal?) smoother out there, and I use it frequently. Please don't get me wrong-I understand what you're saying and was just joking around. Much respect and thanks is due to you and your program, and the optimized version is eagerly awaited.

[sh0dan]

If I get the time, I'll try to throw together a test-version in C - I've been thinking about using temporal data for some time, and this just triggered a good idea. :)

Maybe I should do it tonight, instead of watching a movie ;)

[poptones]

I don't get it. Both multiplication (convolution) and addition (averaging) are associative and commutative. You can specify thresholds within layer or by making a mask from the original (unfiltered) source. 5+3 = 3+5 every time...

The problem with temporal filters is they are so doggone slow. It seems like every filter that adds another frame fetch from a file doubles the encode time.

[dividee]

Sorry I don't follow you. Adaptive operations are usually not commutative nor associative. If for example, the operator @ represents the following function:

a @ b = (a+b)/2 if |b-a| <= treshold
= a otherwise

then
a @ b != b @ a (not commutative)
a @ (b @ c) != (a @ b) @ c (not associative)

For instance, let a=10, b=25, c=15 and treshold=10:
a @ b = 10
b @ a = 25
a @ (b @ c) = 15
(a @ b) @ c = 12.5


The problem with temporal filters is they are so doggone slow. It seems like every filter that adds another frame fetch from a file doubles the encode time.

Not true! temporal filters usually works in 1D while spatial filters usually in 2D (such as convolution), so temporal filters are usually faster. You just have to be careful with implementation of such filters, using some kind of buffering to avoid fetching too much frames at once and trashing the cache with a lot of parallel pointers.

[Koepi]

Isn't it possible to adopt such pixel averaging with toms TomsMoComp filter? I mean that would be the best filter ever.

Apply some spatial smoothing if someone wants that and look where the pixel went an average that out as well ;)

Just to bring in this idea again...

Regards,
Koepi

[Marc FD]

The problem with temporal filters is they are so doggone slow. It seems like every filter that adds another frame fetch from a file doubles the encode time.

I agree with dividee. cnr2 is real-time capable on my AthlonXP 1.4 Ghz
in 640x480. And if richard finish the BitBlt optimisations, it could go damn faster (40-60 fps).

Isn't it possible to adopt such pixel averaging with toms TomsMoComp filter? I mean that would be the best filter ever.

Tom said is algo couldn't be used in a temporal filter :(

PS : I like fast filter and 3D-filter sounds like slow as hell for me.
Like koepi said, the best would be a MC filter, but i would be damn slow too. but will give a much bigger noise-removing effect.
(and would be much much harder to code :D )

[vlad59]

I just coded an avisynth filter that does the 3*3*3 average with a simple matrix till now :


Prec frame : Cur frame : Next frame :
1 2 1 2 4 2 1 2 1
2 4 2 4 8 4 2 4 2
1 2 1 2 4 2 1 2 1


I've also added a luma an chroma treshold like on the example of Dividee to keep edges and take care of scene change.

I work quite good but it's damn slow (in fact it's the slowest filter I've ever seen).

Next to do is MMX a little (the matrix is done for that) and add a little buffering.

If someone already want this filter (it's very very very slow, I warm you), I can post it throught this forum.

[Marc FD]

Slow, it was already know, but does it really worth it ??
I think that the real question.
I think people who chain several filters to gain quality like
sshq with tempsoften are far away from searching fast filters :)

[avih]

I've also added a luma an chroma treshold like on the example of Dividee to keep I've also added a luma an chroma treshold like on the example of Dividee to keep I've also added a luma an chroma treshold like on the example of Dividee to keep edges and take care of scene change.

how did u implement that? with respect to the final result only? or with respect to each of the 26 pixels?

it's also possible to provide different thresholds for the temporal vs 2d operations.

cheers for the work ;)

[vlad59]

@MarcFD

You can not imagine how slow it is ...........
I've attached a zip with the filter. it has only been tested on a full size vob with no other filter.
With a very noisy material, i use this :

convolution3D (8, 16)
first parameter : luma treshold
second parameter : chroma treshold


@avih

In fact, I've made a slow method (don't know if it's a good method).

For each pixel surrounding the center pixel :
if the absolute différence with the center pixel > Treshold
then the corresponding matrix value become 0 so I don't take this pixel into account.

This method will be hard to optimize so I'm corrently trying something else.

[avih]

yup. imo your method should produce very good results, but it's indeed hard to optimize.

i had really rough times trying to optimize this method on a 2d (3x3) deringing filter.. ;)

the other thresholding technique u can use is calc the convolution anyway, and apply it only if the final result is inside |thresh| from the original pixel. that's how deringing is usually implemented.

good luck anyway

[vlad59]

Can a moderator say me if I have attached a zip file with my last post ????

Sorry for this 0-use post

[Swede]

No attachment there... :rolleyes:

[vlad59]

Ooops, I always forget it :mad:

So you can test the slowest filter of earth

EDIT : removed this old zip file

[dividee]

Indeed checking the treshold for each pixel is much better, IMHO. Else, as poptones said, you can do it with existing filters.
And in fact in this case you could probably optimize it a lot by doing three separate [1 2 1] passes, one on each axis (horizontal, vertical and temporal). It would only need to process 9 pixels/output pixel instead of 27 (and 21 for a 7x7x7 instead of 343 :) ), altough temporary stores would be needed, it should be faster.

The intermediary solution is just using something like SmoothHQ followed by TemporalSoften...

What I mean is that for your filter not to be redundant, it has to work like it is working now. (this sentence feels somehow wrong, sorry for bad english)

[tenebrenz]

And in fact in this case you could probably optimize it a lot by doing three separate [1 2 1] passes, one on each axis (horizontal, vertical and temporal). It would only need to process 9 pixels/output pixel instead of 27 (and 21 for a 7x7x7 instead of 343 ), altough temporary stores would be needed, it should be faster.
This sounds like common sense, i think this kind of filter would be a good "one stop" solution to general denoising. I would use it.

[vlad59]

My first goal when I began to optimize this filter was to remove all multiplications so I had to change the way I use tresholds :

Now I do this way :
if the absolute difference is > treshold
the values of the working pixel is remplaced by the values of the center pixel.

So my coefficient matrix is always the same and my MMX/SSE (I use pextrw, don't know if it's a good idea) code only use add and shift (till now 2 times faster : only very very slow ;) ).

From the test I made it seems to respect also the edges and the scene change. I've not done a PSNR to compare the two methods.

I kept my original code so it could become an optionnal C fallback code.

Now my method is to compute the weight of each 3*3 matrix (prec, curr, next), to add them and to divide by 64 (i.e. << 6).

I've also tried to buffer the weight of each frame but as there is a treshold check for every pixel it could produce some ghosting at scene change (I've seen that).

So I you have some theory to help me to optimize or make the algo better, feel free to post here.

[vlad59]

@Dividee

What you propose is to have a matrix like that :


Prec frame : Cur frame : Next frame :
0 0 0 0 1 0 0 0 0
0 1 0 1 6 1 0 1 0
0 0 0 0 1 0 0 0 0


??

Of course it should be faster ... I make a test

[avih]

but that would sort of eliminate the advantage of 3x3x3... don't u think?

[vlad59]

I agree, that's why I asked dividee if I understood him well.

[vlad59]

Another dumb question : how do you debug a avisynth filter with VC6 ???

EDIT : Perhaps wrong section, ooops

[dividee]

No that wasn't what I meant.
What I was talking about is just a standard 3x3x3 convolution without pixel tresholding (although you could treshold the final result), using exactly the same matrix as yours:

1 1 2 1
2 x 1 2 1 = 2 4 2
1 1 2 1

and then

1 2 1 [ 1 2 1 2 4 2 1 2 1 ]
2 4 2 x [1 2 1] = [ 2 4 2 4 8 4 2 4 2 ]
1 2 1 [ 1 2 1 2 4 2 1 2 1 ]

where [ ] represents the third dimension

Sorry I'm not used to manipulate 3D matrixes but i hope you get idea: doing three 1-dimensional passes instead of one 3D pass.

But don't bother too much, it's just a particular cas of 3D convolution symmetrical along each plane.

[dividee]

for debugging filters, I usually create an avs that reference the dll directly in the Debug folder (or change the target of the Debug build to the folder where the avs is). I put VirtualDub as executable for debug session, then press F5 (after adding breakpoints if you want) and load the avs in virtualdub. Hope it helps

[vlad59]

Thanks Dividee

I'll try that. But I just make a post in developpement section for this question, can you remove it ???

Thanks again.

[dividee]

No I can't. But if you have additionnal questions about it, post them there.

[Marc FD]

Hmm, the thread gave me a good filter idea.
I'm going to create a MAD (motion adaptive denoiser) filter.
And maybe modifying it i could create a Noise Killer
( a filter who (almost) ONLY remove noise, like sshq )
But i will make some tests before. And i've my benchmark
project. I'll send a beta to this thread as soon as i've
made the first beta. It should lead to very good results :cool:

PS : It's far away from a 3D convolution but i will be a sort of
temporal+spatial filter ....

[bb]

Just wanted to add that I'm glad this discussion seems to lead to real improvements in denoising techniques. It's exciting to test the upcoming filters!

@vlad59:
Thanks for your effort. I'll test it this evening on analogue capture and on DV material.

@Marc FD:
That sounds very promising, too. I'm curious how it will compete to a 3D solution.


bb

[Marc FD]

That sounds very promising, too. I'm curious how it will compete to a 3D solution.
hehe :devil:
in fact (if it works) it will keep all the pros of temporal and spatial without the cons. The thing is that i'll use a motion detection engine (the same than in CopySame) and choose the best denoising solution for each pel of the video :)
And if i add pattern detection, maybe it could make a nk

I'm going to work. Would be better when you could test :cool:

[bb]

Originally posted by Marc FD
Would be better when you could test :cool:

I'll test it as soon as you have something up and running (and post it, of course).

bb

[Marc FD]

I've made a first "sample". It's not very fast (about 15 fps)
It's not a 3D filter and in fact i only implemented temporalfiltering in it because i don't know how to make a good spatial denoiser.
I've compared it with TempSoften2 and it seems to ghost less on movements scenes...
I didn't have the time to test it alot. i prefer try to create a new type of spatial filter with pattern noise detection.
I prefer create new algos than improving old ones.
When i would have a good spatial filter, MAD would be a real temporal and spatial filter. (in this version only the movement detection is a bit "spatial" :) )

@Dividee

I didn't exactly understood how do you debug avs filters...
do you have a special VDub version ??

I attached MAD.zip
just wait...

[vlad59]

In the alpha version I attached in this thread, i probably swapped the chroma and luma treshold so If you see something strange try to swap those two parameters.

I'm too lazy to check that now, in my next release it'll be clear.

[Koepi]

MarcFD,

you're great, finally a motion compensated temporal smoother, I love you (erm, not in that gay way of course ;) ).

I'm eager to test it on some TV footage as I have plenty problems with denoising such stuff. I think it'll turn out you're making yourself an unforgettable hero :)

Swede, come on, enable the attachment already! I need to test that stuff :)

Marc, how about first finishing the temporal smoother stuff until it works as you want it to and then you look into the sources for spatial filters and adopt that code? (should be the same code as the temporal smoother, just not in the z (time-)dimension, but in the x and y directions of a single picture, or am I mistaken here?).

Anhow, thanks for that (hopefully) amazing filter!

Regards,
Koepi

[Marc FD]

Whow,Whow,Whow !!!
NO RUSH !!!
IT'S NOT A MOTION COMPENSATED FILTER !!!!
IT'S A MOTION ADAPTIVE FILTER !!!!
it would not compensate motion it would adapts to
it !! Nothing to do with ME in codecs like XviD !!!!
And my filter is far from doing amazing results !!
I'm afraid you're overestimating this flter ;)

Currently it just look a _bit_ better in fast scenes
than ie tempsoften2 because it wouldn'nt smooth
what's moving. that's all :(

[Koepi]

Erm, sorry that's what I meant ;)

I think when you can detect motion (maybe the direction?) you can use that information to smooth "corresponding" regions. But I think it's easier to just detect motion areas and then apply the spatial soften, as proposed. Well, take your imte, I didn't want to rush things, I wanted to test it :)

Ok, back to add automaisation support to OggMux...

regards,
Koepi

[Marc FD]

Marc, how about first finishing the temporal smoother stuff until it works as you want it to and then you look into the sources for spatial filters and adopt that code? (should be the same code as the temporal smoother, just not in the z (time-)dimension, but in the x and y directions of a single picture, or am I mistaken here?).

In fact,like i said, i don't like to copy very used algos.
Peoples who have coded them are better than me for that.
I want to try a new way of spatial filtering (and maybe it wouldn't work...)
MAD is not a motion compensated filter :
It's an "hybrid" denoiser :
- In movement areas, he would use spatial filtering, because it leads to better results when there is motion, and if there is blending, it wouldn't be seen.
- In still areas, he would use very aggressive temporal filtering, because there is no ghosting, it wouldn't be seen too.

The idea is just to mix the advantages of spatial and temporal filtering.

@Koepi
Sorry, but i'm afraid you will be VERY disappointed if you think my filter will make miracles :( (i would create an MoComp filter if you want when i would be older.. let's say 18 Ok :) after all, i've writed my first C lines in the beginning of THIS month)

PS : when i say temporal/spatial filtering, it's not like a temporal/spatial smoother. i try to destroy High-frequency noise, not to blend low-freq noise.

PS@Koepi : If you have got an heavy chroma noised analog, you should try cnr2. I think would work much better than MAD on chroma denoising.

Back to work now. will try to create a pattern detection spatial noise killer... hope it would work better than normal Spatialsmoothers :D

[bb]

First testing results (I need to do a lot more, of course):

Convolution3D: Still needs some tweaking, but it seems to be superior to a pure spatial or temporal filter. It's slow indeed, but I had an optimizing approach in mind which I didn't want to keep by myself:
What if you cache three frames and interleave them like 3 lines of 1st frame, then 3 lines of second frame, then 3 lines of third frame, then next 3 lines of first frame again, etc. Then you apply the filter; in the end you have to de-interleave again. I think this could greatly improve processor cache hits.

MAD: I had to guess what the parameters mean, so I tested the proposed "Normal", "Hard", and "Soft" settings. "Normal" and "Hard" seemed to be unusable to me because of way too much ghosting. "Soft" simply didn't get rid of the noise - I'm afraid there's still a lot tweaking to be done...
Marc FD, could you please post the meaning of your filter's parameters?

bb

[vlad59]

@bb

Thanks for testing. I think I'll release a new beta today (3 times faster and exactly the same quality), just little bug to track and it will be good.

Till now I've just added some SSE optimizations but no buffering at all, I'll try to buffer when the main algo will be finished (I don't expect a big speedup if I want to keep quality, but I may be wrong).
I'll try to think about your idea tomorrow, but you have to keep in mind that the slowest part of this algo is the way I use the treshold and that's also prevent me from doing to much buffer optimization.
Let's explain with a sample :

I'm computing a Luma sample = 100 and my luma treshold is 10.
My original values are :
Prec frame : Cur frame : Next frame :
101 101 100 090 101 100 189 190 80
099 100 100 010 100 100 190 200 80
090 105 100 010 010 100 140 201 80

I use my treshold to detect edges and scene change and :
Prec frame : Cur frame : Next frame :
101 101 100 090 101 100 100 100 100
099 100 100 100 100 100 100 100 100
090 105 100 100 100 100 100 100 100

Next I calulate the weight of each frame :
Prec frame : Cur frame : Next frame :
1601 1592 1600

Now the new luma value = (1601 + 2* 1592 + 1600) / 64 = 100
New Cur frame (no change) :
090 101 100
010 100 100
010 010 100

No change with this sample


With this sample you see another pb :
The next frame must be a scene change so for my calculation I change all its values to 100.
Advantage : my convoluted luma value won't become 130 so no ghosting
Drawback : my convoluted luma value has a high probability to keep its original value.

I thought of adding a SAD to check for this case but it will slow again the filter.

What do you think about it

[Ivion]

Ok, also here are my first result using the MAD filter.

Tested on: Rurouni Kenshin Tsuiokuhen OVA 2 (I choose this one cuz RK [Rurouni Kenshin] is usually very noisy and I like this OVA :))
Results: Normal And Soft were both not capable of removing all of the noise, altough Normal did a 'decent' job, tough 2D Cleaner by Jaan Kalda was better. Hard didn't work to well, for there was alot of ghosting, and with alot, I mean alot!
Like: there was a scene of two people talking and one moment the 'camera' (it's anime, so there isn't really a camera :D) was pointed at the wall, and the second moment it pointed to one of the faces and then there were still some 'pieces' of the wall on the image.
So I think some tweaking is gonna be needed, good luck Marc FD!

I currently can't provide you guys with screenshot for it's now encoding and the result that's posted here is based on some still images of the avs script.

[bb]

@vlad59:
Unfortunately I haven't programmed filters (yet). Therefore I'm not familiar with all the MMX and SSE jargon (what's SAD?). I was just thinking that you need to process a lot of pixels from different frames in a way which is _very_ awful for a cache. Thus the reordering of the frame lines could be a huge advantage.

I have to think about your way of processing. Just a (maybe stupid) question: How exactly do you calculate the weight?

bb

[Koepi]

Hm.... i have some versions to offer:

Search And Destroy (hehe, sometimes it really does!)
See At Definition )I like that one)

or simply

Summation of Absolute Differences.

Regards,
Koepi

[vlad59]

Originally posted by Koepi
Hm.... i have some versions to offer:

Search And Destroy (hehe, sometimes it really does!)
See At Definition )I like that one)

or simply

Summation of Absolute Differences.

Regards,
Koepi

I prefer Search And Destroy ;) :D

With my previous example :
100 is my computed value.
200 is the same value but a frame later

Absolute difference is abs (100 - 200) = 100
if this AD > Treshold (10 in my sample)
then I don't use the next frame at all (more speed and maybe more quality)

So the new value is (1601 + 2* 1592) / 48.

Back to code ......

[bb]

Sounds like my question was way too stupid...
(You still didn't explain how you calculate the 1601, but never mind, I know by now that your weight matrix was
1 2 1
2 4 2
1 2 1
)

bb