Smooth in time and space?

[bb]

Well, I don't really know if there is already a filter available covering this idea:

The smoothing/denoising filters I know are either spatial or temporal filters. What about smoothing spatially AND temporally at the same time?

E.g. you could average a 3x3x3 block of 3 consecutive frames, i.e. the middle pixel would be averaged by its neighbours in the same frame and the neighbour pixels of the neighbour frames in the time dimension.

I hope the explanation wasn't too complicated to understand, but I think using existing averaging algorithms on a three-dimensional cube instead of just looking at a two-dimensional plane would not be too complicated to implement.

What do you think?

bb

[Acaila]

AFAIK that is exactly what the General Convolution 3D filter does. However it's made for VDub, not Avisynth.

Something like this can easily be achieve by just chaining two filters, but the problem is that they don't work intelligently to determine edges and sharp lines etc.

[poptones]

AVISynth has generalconvolution. Richard added it himself to the last couple releases. And you can chain them pretty easily, as well as add as much intelligence as you like. In addition to using edge detection filters to create masks, you can use lighten and darken via layer().

There are so many ways to do this now it's become difficult to decide which one is best for an application. There are the included filters and the plugins - between those and being able to layer clips using a variety of operations you have pretty much unlimited possibilities. My biggest gripe right now is with finding a way to remove MPEG artifacts without screwing things up too badly. And I'm working on that one, too.

[bb]

Hmm, the 3D averaging I was thinking of is not the same as chaining a spatial and a temporal filter. But the General Convolution 3D filter seem to be the solution (don't know if it has scene change detection, which would be nice). I'll try that one.

Is there a General Convolution 3D filter available for AviSynth, too?

If not, maybe someone feels challenged to implement it. There's also much room for tweaking, like edge detection, etc.

I think this could be a superior method of getting rid of noise in complicated sources like DV.

bb

[avih]

i think a specific 3d convolution filter should be written since it would be much faster, especially since it using 27 different values. also, it should be adaptive.

it can be adaptive in 2 ways:
1. apply the convolution result if it's not 'further' than 'x' from the original pixel value.

2. take into account only pixels that are not 'further' in value from the priginal pixel by 'x'

the 2nd method gives better results imo, but is considerably slower because of the 'if's during the convolution process itself.

it SHOULD be a great filter though.

[Didée]

Dreaming of SmartSmootherHiQ in 3D ...

[manono]

And then instead of taking twice the time to encode, it can take four times the amount.

[sh0dan]

Not entirely so, because the temporal check would not be a complete 3D-check, it would only check the current pixel against the same pixel in the previous frame (one check more per pixel).

The idea is quite neat - the algorithm could work like this:
*Calculate the blurred pixel as usual.
*Compare the original pixel to the previous frame. The bigger the difference, the less will the pixel get blurred.

That will make low-freqency noise even easier to detect.

Might actually work!

I've also recived a great suggestion for adaptive threshold, that adapts the threshold to the luma value of the current pixel - so there are lots of improvements in the works. First priority is still an MMX-optimized YUV version though.

[manono]

Hi Klaus-

SSHiQ is far and away the best spatial (and someday spatial-temporal?) smoother out there, and I use it frequently. Please don't get me wrong-I understand what you're saying and was just joking around. Much respect and thanks is due to you and your program, and the optimized version is eagerly awaited.

[sh0dan]

If I get the time, I'll try to throw together a test-version in C - I've been thinking about using temporal data for some time, and this just triggered a good idea.

Maybe I should do it tonight, instead of watching a movie

[poptones]

I don't get it. Both multiplication (convolution) and addition (averaging) are associative and commutative. You can specify thresholds within layer or by making a mask from the original (unfiltered) source. 5+3 = 3+5 every time...

The problem with temporal filters is they are so doggone slow. It seems like every filter that adds another frame fetch from a file doubles the encode time.

[dividee]

Sorry I don't follow you. Adaptive operations are usually not commutative nor associative. If for example, the operator @ represents the following function:

a @ b = (a+b)/2  if |b-a| <= treshold
      = a        otherwise

then
a @ b != b @ a (not commutative)
a @ (b @ c) != (a @ b) @ c (not associative)

For instance, let a=10, b=25, c=15 and treshold=10:
a @ b = 10
b @ a = 25
a @ (b @ c) = 15
(a @ b) @ c = 12.5

The problem with temporal filters is they are so doggone slow. It seems like every filter that adds another frame fetch from a file doubles the encode time.

Not true! temporal filters usually works in 1D while spatial filters usually in 2D (such as convolution), so temporal filters are usually faster. You just have to be careful with implementation of such filters, using some kind of buffering to avoid fetching too much frames at once and trashing the cache with a lot of parallel pointers.

[Koepi]

Isn't it possible to adopt such pixel averaging with toms TomsMoComp filter? I mean that would be the best filter ever.

Apply some spatial smoothing if someone wants that and look where the pixel went an average that out as well

Just to bring in this idea again...

Regards,
Koepi

[Marc FD]

The problem with temporal filters is they are so doggone slow. It seems like every filter that adds another frame fetch from a file doubles the encode time.

I agree with dividee. cnr2 is real-time capable on my AthlonXP 1.4 Ghz
in 640x480. And if richard finish the BitBlt optimisations, it could go damn faster (40-60 fps).

Isn't it possible to adopt such pixel averaging with toms TomsMoComp filter? I mean that would be the best filter ever.

Tom said is algo couldn't be used in a temporal filter

PS : I like fast filter and 3D-filter sounds like slow as hell for me.
Like koepi said, the best would be a MC filter, but i would be damn slow too. but will give a much bigger noise-removing effect.
(and would be much much harder to code )

[vlad59]

I just coded an avisynth filter that does the 3*3*3 average with a simple matrix till now :

Prec frame :     Cur frame :   Next frame :
1 2 1            2 4 2         1 2 1
2 4 2            4 8 4         2 4 2
1 2 1            2 4 2         1 2 1

I've also added a luma an chroma treshold like on the example of Dividee to keep edges and take care of scene change.

I work quite good but it's damn slow (in fact it's the slowest filter I've ever seen).

Next to do is MMX a little (the matrix is done for that) and add a little buffering.

If someone already want this filter (it's very very very slow, I warm you), I can post it throught this forum.

[Marc FD]

Slow, it was already know, but does it really worth it ??
I think that the real question.
I think people who chain several filters to gain quality like
sshq with tempsoften are far away from searching fast filters

[avih]

I've also added a luma an chroma treshold like on the example of Dividee to keep I've also added a luma an chroma treshold like on the example of Dividee to keep I've also added a luma an chroma treshold like on the example of Dividee to keep edges and take care of scene change.

how did u implement that? with respect to the final result only? or with respect to each of the 26 pixels?

it's also possible to provide different thresholds for the temporal vs 2d operations.

cheers for the work

[vlad59]

@MarcFD

You can not imagine how slow it is ...........
I've attached a zip with the filter. it has only been tested on a full size vob with no other filter.
With a very noisy material, i use this :

convolution3D (8, 16)
first parameter : luma treshold
second parameter : chroma treshold

@avih

In fact, I've made a slow method (don't know if it's a good method).

For each pixel surrounding the center pixel :
if the absolute différence with the center pixel > Treshold
then the corresponding matrix value become 0 so I don't take this pixel into account.

This method will be hard to optimize so I'm corrently trying something else.

[avih]

yup. imo your method should produce very good results, but it's indeed hard to optimize.

i had really rough times trying to optimize this method on a 2d (3x3) deringing filter..

the other thresholding technique u can use is calc the convolution anyway, and apply it only if the final result is inside |thresh| from the original pixel. that's how deringing is usually implemented.

good luck anyway

[vlad59]

Can a moderator say me if I have attached a zip file with my last post ????

Sorry for this 0-use post