Combining plugin-filters for better quality

[E-Male]

First of all: i´m not a programmer, I just try to give some ideas
Just tell me what you think, maybe some ideas are possible (I gues some are not)

My basic idea is that combining to filters can increase quality
(like convolution3d does combine a spacial and a temporal filter and is better than just using this to after each other)

----------------------------------------------------------------------

resizing + noise reduction:
as far as I understand both resizing and noise reduction create the resulting pixel by using the original pixel plus the ones around it.
If there are enough similarities, maybe both can be combined so the pixels don´t have to be redone twice.

----------------------------------------------------------------------

smart sharpening + noise reduction
this idea is based on msharpen and convolution3d
msharpen does this
1. detect edges
2. aply sharpening to ths detected pixels
the new plugin should do that:
1. detect edges
2. aply noise reduction to the non-detected pixekls
3. aply sharpening to the detected pixels

----------------------------------------------------------------------

the last idea is just to put all resizing, noise reduction and sharpening in on ultimate plugin

----------------------------------------------------------------------

waiting for your thoughts
thx in advance
CU
E-Male

[Guest]

>the new plugin should do that:
>1. detect edges
>2. aply noise reduction to the non-detected pixels

My beta of MSmooth tries to do this, but not very successfully. There are still serious issues to be resolved. It is not as easy as it sounds, I assure you. :)

[int 21h]

Resizing in itself, unless you're using an algorithm like Tom's SimpleResize, is noise reduction in itself because of the various estimations being preformed.

Also, I think this new idea, this movement of sorts toward integrating everything is terrible. Integration when it serves a well established purpose is good (i.e. one tool that preforms the entire transcoding process from A format to B format), however, integration when it contributes to bloat-ware is bad. Avisynth should be considered an environment to work and develop in, not a single application to develop for.

Actually, while I am on this rather O/T moment, I am also of the opinion that the current Avisynth development team would be much better off streamlining and improving the core of Avisynth instead of worrying about adding new filters to it (One of the entire purposes of addon filters is to avoid this exact scenario).

That of course is just my opinion though.

[Guest]

I am also of the opinion that the current Avisynth development team would be much better off streamlining and improving the core of Avisynth instead of worrying about adding new filters to it (One of the entire purposes of addon filters is to avoid this exact scenario).Amen, bro'.

[E-Male]

@neuron2
i honestly got no idea how difficult it is
that´s one of the reasons i posted this

@int 21h
ok, ok, i forgot to say that this should only be done if it gives a quality improvement
if not it´s counterproductive because it makes development harder

also the idea with sharpening+noisereduction could work like this:
one plug-in creates a mask (based on edge detection in this case)
then different other plug-ins can use it
what about that?

but i still think that something like convolution3d with additional options for resizing could give a better result than both proccesses done after each other
maybe it´s so less that it is not worth working on it, i don´t know, just sharing my thoughts

hoping for more comments
CU
E-Male

p.s. I gotta learn programming, if i only had the time...[sigh]

[int 21h]

That is the thing. I don't think this would necessarily improve quality depending on the filter chain that you normally use. If you always put your noise filter/processing filter first and then apply a resize, it shouldn't affect quality at all.

(i.e. Your result can only be as good as your original)

If you wanted to see an exact example of this, apply Con3d to a clip then immediately resize with SimpleResize. (Instead of with a filtering resize like bilinear,bicubic,etc)

I think that this functionality you request already exists in the language and filters of Avisynth. Give it a try. Try subtracting or comparing two clips, one with SimpleResize (since you know it has no filtering) and one with Bicubic or Bilinear. That should give you an idea of what the quality would improve to. :)

[E-Male]

ok

maybe at least idea for a shared-mask will work, but i don´t think i´m the first one thinking of that

[ARDA]

quote:
--------------------------------------------------------------------------------
the last idea is just to put all resizing, noise reduction and sharpening in on
ultimate plugin.
--------------------------------------------------------------------------------

Maybe if we say alternatives parameters for a resizing plugin

quote:
--------------------------------------------------------------------------------
>the new plugin should do that:
>1. detect edges
>2. aply noise reduction to the non-detected pixels
--------------------------------------------------------------------------------
@ neuron2
And why not also contrast,bright,sat and hue at the same time before you return
each pixel? As another options built in the same filter.I remember when you
realeased Tweak to say something like "to add bright in the loop was so cheap!"
It is just an example. What I mean; once the filter has already load the frame,
and I need for instance to apply noise reduction,contrast reduction, etc.

quote:
--------------------------------------------------------------------------------
Resizing in itself, unless you're using an algorithm like Tom's SimpleResize,
is noise reduction in itself because of the various estimations being preformed.
--------------------------------------------------------------------------------
quote:
--------------------------------------------------------------------------------
If you always put your noise filter/processing filter first and then apply a resize,
it shouldn't affect quality at all.
--------------------------------------------------------------------------------
@int 21h
And here another question if we do noise-reduction and resize in the same loop
couldn't we at least gain some speed.I don't know exactly how avisynth manage
a filter chain; does it reload the frame for each filter?

quote:
--------------------------------------------------------------------------------
but i still think that something like convolution3d with additional options for
resizing could give a better result than both proccesses done after each other
--------------------------------------------------------------------------------

I don't imagine why it should give better results but maybe it could be faster
Please forgive me if there is not too much sense in my questions.

Thanks in advance.Arda

[Richard Berg]

Actually, while I am on this rather O/T moment, I am also of the opinion that the current Avisynth development team would be much better off streamlining and improving the core of Avisynth instead of worrying about adding new filters to it
I agree. Making major improvements to the core is really hard (for me anyway) + most of us have too much geek pride to release things that aren't "ready" = such development exists, just under the radar at the moment...

[int 21h]

Originally posted by ARDA
@int 21h
And here another question if we do noise-reduction and resize in the same loop
couldn't we at least gain some speed.I don't know exactly how avisynth manage
a filter chain; does it reload the frame for each filter?


Its quite possible it would be faster. Many times different Avisynth developers have questioned some of the frame buffer usage and memory movements of Avisynth, and again this goes with my previous comment of streamlining the current core of Avisynth.

I also agree that there should technically be no qualitative difference.

[Guest]

And why not also contrast,bright,sat and hue at the same time before you return
each pixel? As another options built in the same filter.I remember when you
realeased Tweak to say something like "to add bright in the loop was so cheap!"
It is just an example. What I mean; once the filter has already load the frame,
and I need for instance to apply noise reduction,contrast reduction, etc.

I hope you're kidding. The logical conclusion is that I should combine all my filters into one. I'll get started on it right away. Not!

[sh0dan]

Originally posted by ARDA

And why not also contrast,bright,sat and hue at the same time before you return
each pixel? As another options built in the same filter.I remember when you
realeased Tweak to say something like "to add bright in the loop was so cheap!"
It is just an example. What I mean; once the filter has already load the frame,
and I need for instance to apply noise reduction,contrast reduction, etc.


Combining filters is not a win-win situation. First of all it complicates the filters very much, since it would have to account for a lot more combinations of parameters than it has to do already. This would lead to many more bugs, even for good filter writers.

Second of all, it does not always give a speed gain. In MMX and assember, you much easier risk running out of registers, forcing filter writers to access memory more often, instead of being able to use registers. Many current filters will show 100% usage of MMX-registers. Furthermore you risk store->load forwardning penalties due to different algorithms.

Regarding the internal cache-copying, it could be helped by filters setting optional cache hints, that enables cache copying, if the filter does not require reading previous or forthcoming frames. This could become 2.1 functionality.

The current architecture ensure a VERY flexible enviroment, as opposed to VDub filters, which are much more limited. I don't think we should sacrifice that flexibility to gain (at my estimation) max. 10% performance.

[meleth]

As i haven't looked on the avisynth code nor know how the filters work this might be totally wrong. But wouldn't it increase the speed of everything if there was one single detection filter that you used first in your filter chain that the other filters took advantage of instead of doing their own detection.

Ofcourse this would probably be stupid now that i think about it, since the real output would change for every filter it passes and the underlaying filter wouldn't know about it. But i'm posting this anyways in case someone comes up with any bright ideas, cause i think in some cases this might help speed things up. If nothing else perhaps the other filters could call functions in it and use for the detection they need and if all the edge/noise/motion etc. detection is done in just one place no doubt there will be more improvements on them since more ppl will be working on them and sharing ideas instead of everyone just doing their seperate thing, or so i think. Could be that i'm just being a tard aswell, i really wouldn't put that past me:P

[Belgabor]

This goes a bit into the direction meleth proposed. Why don't we invent a filter crosstalk interface? This would have to be clever designed not to break compatibility, but especially filters which use duplicate detection or edge detection could profit from it.

On the other hand, perhaps including real masking into avisynth would be enough. Though still some kind of information transfer functionality between filters would probably be useful.

Cheers
Belgabor

[sh0dan]

Cross-filter-talk can't exist, the frame is handed from one filter to the next, it is not even remotely posssible to merge two filters into a single pass.
It would require complete rewrite of avisynth, and impose extreme limitations, and even be slow, if you shift filter on a per-pixel basis. It will be slow, because you will get a function call per filter per pixel, that will mess up the code-cache.

So unless there is an algorithmical reason for merging filters (as Color_YUY2 and tweak for instance).

The only thing IMO worth looking at, is optimizing the image-cache, that is put inbetween every filter - if both source and destination filter allows it, the cache should be able to disable itself (tweak, many others) - or only cache a given number of frames (temporalsoften for instance). That way we could optimize memoryusage in general, since it would allow frame references to be passed unchanged from a source to a destination filter.

Regarding quality, there is little gain in combining filters, unless the filters stores the information temporary at a higher precision (16 bit per channel for instance) - but then again, it would be better to store the data at better quality, since the penalties are overwhelming (as implementing 16bit/pixel would be now).

[meleth]

well it should still be possible to use one detection filter if that filter masks the areas where it has detected something and then the filters uses the masks to process the stream. Or so i imagine.

[ARDA]

@All
Thanks for the long explanations, and the time you've spent for that.

quote:
--------------------------------------------------------------------------------
Originally posted by neuron2
I hope you're kidding. The logical conclusion is that I should combine all my
filters into one.I'll get started on it right away. Not!
--------------------------------------------------------------------------------
I shouldn't have expressed myself correctly, as I said just an example.I just was
thinking in the most frequently used filters.From the point view of a final
user;we've been making for long time TemporalSmoother,resize,etc.
I agree with you, if the logical conclusion is to combine all filters ;not!
is the answer.

quote:
--------------------------------------------------------------------------------
Originally posted by int 21h
I am also of the opinion that the current Avisynth development team would be much
better off streamlining and improving the core of Avisynth instead of worrying
about adding new filters to it (One of the entire purposes of addon filters is
to avoid this exact scenario).
--------------------------------------------------------------------------------
In my modest opinion I also agree with that,I don't like to have unfinished
or bugged filters built in avisynth. About the way of improving I 've not
the knowledge to give an opinion; I leave that for developers.

quote:
--------------------------------------------------------------------------------
Originally posted by sh0dan
Second of all, it does not always give a speed gain. In MMX and assembler, you
much easier risk running out of registers, forcing filter writers to access
memory more often, instead of being able to use registers. Many current filters
will show 100% usage of MMX-registers. Furthermore you risk store->load
forwarding penalties due to different algorithms.
--------------------------------------------------------------------------------
I understand you.I've made as an exercise a simple tool to merge lumas based
in your algo, for weight=0.5 (wihtout MMX instructions) ,and it's so simply that
runs almost as fast the one is built in avisynth.Sometimes simple is better than
complex

quote:
--------------------------------------------------------------------------------
Originally posted by sh0dan
The current architecture ensure a VERY flexible enviroment, as opposed to
VDub filters, which are much more limited. I don't think we should sacrifice
that flexibility to gain (at my estimation) max. 10% performance.
--------------------------------------------------------------------------------
I was not thinking in any change in the architecture.Just to combine one or two
filters as I said before the most frequently used.

quote:
--------------------------------------------------------------------------------
Originally posted by sh0dan
The only thing IMO worth looking at, is optimizing the image-cache, that is put
inbetween every filter - if both source and destination filter allows it, the
cache should be able to disable itself (tweak, many others) - or only cache a
given number of frames (temporalsoften for instance). That way we could optimize
memoryusage in general, since it would allow frame references to be passed
unchanged from a source to a destination filter.
--------------------------------------------------------------------------------
With this post you've answered all my questions.Thanks a lot.

And I finally my apoligizes if I'm making you to waste time in answering newbie
questions. Thanks all for the great revolution we're having in avisynth.
I hope this will never stop. Arda

[E-Male]

as i understand the problem is that there is no easy way for the noise reduction and sharpening filters to get the information the mask filter has cerated.

now i would do it like that (i gues it will be very very slow but will look best)
--------------------------------------------------------
a = avisource(clip)
b = mask(a, *more paramters*)
c = denoise(a, b, *more paramters*)
d = sharp(c,b, *more paramters*)
return d
--------------------------------------------------------
clip is our source, placed on variable "a"
the mask showing the edges is placed on variable "b"
"a" gets denoised ignoring the pixels marked in "b" and the result is "c"
"c" gets sharpendd at the pixels marked in "b" and the result is "d"
"d" is the final result
--------------------------------------------------------

i still think that a combination of msharpen and convolution3d into one would give the best results in quality (maybe also in speed, when it´s optimized)
but i know development would take long, so think about my idea with the mask clip, and tell me if you think it works (i gues you´ll have soem good reason why it won´t..)..

cu
e-male

[Guest]

i still think that a combination of msharpen and convolution3d into one would give the best results in qualityOK, I'll bite! Why do you think this? :)

[E-Male]

Ok, let´s say we got a noisy video
Then the avysynth script could first use convolution3d to reduce the noise and then msharpen to bring back some sharper edges
So far so good


But this has 2 big disatvantages:
1. the edge-detection of msharpen has to work on an already blured picture, which makes it les effective
2. the pixels of the edges are first unsharpend and then sharpend, that menas the are proccessed twice and even by totally oposite filters, this can´t be good for image quality and also waistes some time

now if both are combined this new plug-in could:
1. make the mask from the unedited picture --> easier/better edge detection
2. convolution3d could ignore the pixels marked as edges, making the blur less bad
so every pixel would only be touched by one filter


I hope everyone gets what I mean [not so easy for my to descibe it in english]
CU and thx for your time
E-Male

[int 21h]

@meleth
Its easier to just add edge detection to all the filters you want to use it than try to implement some sort of shared memory structure for that particular data. Slinging around data gets expensive very quickly when its multiplied by 140,000 times. Much easier to just use it and throw it away. I haven't looked Mr Graft's particular implementation, but I wouldn't imagine that detection is very expensive (especially if its asm).

@E-male
Get C3d developer to add edge detection to his filter. Simple as that.

[sh0dan]

Originally posted by ARDA
I understand you.I've made as an exercise a simple tool to merge lumas based
in your algo, for weight=0.5 (wihtout MMX instructions) ,and it's so simply that
runs almost as fast the one is built in avisynth.Sometimes simple is better than
complex


Yeah - averaging 50% can be done faster using pix = (pix1+pix2)>>1 (or slower (pix1+pix2)/2)) instead of pix = (pix*32768+pix2*32768)>>16. Integer SSE even has an instruction for averaging two byte values. Just too lazy to implement hardcoded 50% - it seems like a too special case, to put it in.



With this post you've answered all my questions.Thanks a lot.

And I finally my apoligizes if I'm making you to waste time in answering newbie
questions. Thanks all for the great revolution we're having in avisynth.


Sorry if I sounded a bit arrogant - I hope it helped you to get an idea of how things are connected. AviSynth is the best video-processing framework I have ever seen, and I would wish I could devote more time for it, but right now my job is pulling >12h per day. :(

[ARDA]

quote:
--------------------------------------------------------------------------------
Originally posted by sh0dan
Sorry if I sounded a bit arrogant.....
I could devote more time for it, but right now my job is pulling >12h per day
--------------------------------------------------------------------------------
You didn't sound arrogant.I really apreciate your help, I'm still a newbie
in coding,so I couldn't feel offended at all. I also work more than 12 hours
a day and besides of that I've been more than twenty years without writing
any line of code; with that you can imagine my age and how it's hard for me
start again; but these forum and all the great people that share his work
and knowledge here has made me open some old and new books.

Thanks once more. Arda

[vlad59]

Originally posted by int 21h
@E-male
Get C3d developer to add edge detection to his filter. Simple as that.

I'm very short in time. But I promess I'll have a look at Neuron2's code this weekend to tell if
- I understand the code ?
- the modifications are easy or not ?

If the answer for both questions is true (and of course if Donald let me use his code), I will add it to Convolution3D ASAP.

[Guest]

I remind you that I already am developing both the edge-masked smoother and the edge-masked sharpener and have announced my plans to integrate them. Besides, things are not so easy as E-Male would have you think. Believe me, I have already been banging my head against the wall with these issues. Now that I am getting close to an acceptable solution you want to rush something together?

I don't object to the idea of eventually having the edge-masking in C3D, but I think it is premature, because the edge detection code in MSharpen, while fine for a sharpener, is simply not yet ready to pull its weight for a smoother. Can we be patient for a while, please?

On the other hand, if you solve the problem better than I do, and want to feed it back to me, that too would be acceptable. :)

[ARDA]

---------------------------------------------------------------------
Originally posted by sh0dan
Yeah - averaging 50% can be done faster using pix = (pix1+pix2)>>1
(or slower (pix1+pix2)/2))instead of pix =(pix*32768+pix2*32768)>>16.
Integer SSE even has an instruction for averaging two byte values.
Just too lazy to implement hardcoded 50% - it seems like a too special
case, to put it in.
---------------------------------------------------------------------
Oh by the way I imagine it's a type mistake it should be (pix1+pix2+1))>>1)
and ((pix*32768+pix2*32768)+16384)>>16 :)

Arda

[E-Male]

@neuron&vlad:
Great to hear your working on this
Maybe you could work together, as one got the noise reducter and one the masking tool, together you might find the way to connect both

and i know that talking is easier than coding,
my thought was just like before you aply the concolution3d to a pixel check if the mask detected it as edge and if skip it
but i gues it´s not that simpel (well it never is...)

seems liek this thread is about to end up being totally useless, well it was a try
CU
E-Male

[Guest]

@E-Male

No, I think you have a good point. It's just that the edge masked smoothing problem is not fully solved yet. Unless it detects edges perfectly, you get ugly artifacts along the edges. Reducing the threshold low enough to catch most of the edges also masks too much noise. You can see that from the MSmooth thread. It would be great to integrate masked smoothing but it just isn't ready yet, unless someone else has solved the difficulties and I don't know about it.

I'm not saying the problem is insoluble and I have some ideas about it, but I don't think it currently performs well enough to justify inclusion in C3D. But that doesn't negate the validity of your ideas.

[vlad59]

@neuron2
Be sure that my point was not to steal your code (and your idea) without your agreement. My point was only to use someone else code (with his/her agreement) as I have no time to do it now.

So I'll read your code (I hope it's available, I've been away for 3 days) and I'll try to understand it and help you if I got a magic idea (dreams sometimes become true).

[E-Male]

Well, I don´t think i got another thing to add, because i don´t understand how the edge detection works, so i don´t understand teh specific problem

maybe you can make a short imple explination (but i gues teh code is too complicated for that)

cu
e-male

[Guest]

@vlad59

Sir, no such implication was intended. I release my source code with the specific intent that people *will* use it! My only point is that the problem is not yet solved and we will do better by waiting for a solution than by adding something that doesn't yet work well enough. If it did, I would be thrilled to have you use it in your great filter!

@E-Male

Well, I already explained it and fuller details are in the MSmooth thread. The basic point is that the method produces ugly artifacts on the edges because the edge detection is not working well enough to catch all the edge areas without including lots of the noise. I am actively researching and experimenting in this area, so hope is not yet lost.

[vlad59]

@Neuron2

Sorry for my mistake.
I wake up too early this morning and I think my brain was still asleep so I haven't understood correctly your post.

Sorry again.

Let's read your source and try to understand.

Regards