New smart sharpening filter

[Guest]

After concluding that my edge-masked smoother needed to go back to the shop for repairs, I had the sudden realization that the same basic engine would be great for masked sharpening, whereby important edges are sharpened but noise and fine detail are not amplified. And whereas an edge detection failure is very serious for a smoother, it is much less serious for a sharpener! So, here is my new smart sharpener:

http://shelob.mordor.net/dgraft/msharpen/msharpen10b1.zip

It basically implements the method described here:

http://cs.uhh.hawaii.edu/~jeschke/photography/articles/gimp/SmartSharpening/

It is highly effective on anime. For example, here are two images. The original is a direct DVD rip. It has a hazy, blurry cast. The sharpened one is overdone just a tad to emphasize my point. :)

http://shelob.mordor.net/dgraft/msharpen/hina_orig.jpg
http://shelob.mordor.net/dgraft/msharpen/hina_sharp.jpg

Feedback would be appreciated. Remember, this is a first beta and has no optimizations for speed.

[kyousuke]

interesting.

on your exemple, the background is very well.
but on the character, the sharpening seems to be to hight, on the character's edges for exemple.
i'll test your plug with less aggresive settings.

very interesting for a first version.
good luck for the next

+

[soulfx]

I like it. I been using UnFilter to sharpen up the picture a little after running it through the noise filters and had to set UnFilter low (5,5) in order to still get good compression and not re-introduce noise back into the video.

Tested out Msharpen and it works great. I can increase the sharpening more and not re-introduce noise back into the picture. I found Msharpen(strength=24) to achieve the same amount of compressibilty as UnFilter(5,5). The picture using Msharpen is more sharper and doesn't have the noise that is sharpened with UnFilter(5,5).

I haven't noticed any artifacts with it and it seems to be detecting the edges well at it's default threshold. Some tiny bit of edges are not detected, but they are so small that if the threshold was set lower it would probably start sharpening noise as well.

All that could be asked of now is more speed :)

It would be really nice to have somekind of sharpen filter built into the post-processing on the codec decorders. That way we wouldn't have to worry about sharpening and losing compressability in encoding. Just a thought anyways, but that deserves another thread if it doesn't allready exists.

Peace,
SoulFX

[-h]

It would be really nice to have somekind of sharpen filter built into the post-processing on the codec decorders. That way we wouldn't have to worry about sharpening and losing compressability in encoding. Just a thought anyways, but that deserves another thread if it doesn't allready exists.

ffdshow has this functionality. No idea how well it works.

-h

[Guest]

@kyousuke

Thanks for the feedback. You wrote:on your exemple, the background is very well. but on the character, the sharpening seems to be too high, on the character's edges for example.Maybe you missed the part where I said: "The sharpened one is overdone just a tad to emphasize my point." :)

YUY2 support as well as speed improvement is coming.

Who needs WarpSharp? :)

@soulfx

Thank you for the comments and I am happy that you find it useful. Good point about postprocessing!

@all

My last week or two has been very frustrating, with near failures coming from several new filter attempts. My masked temporal and spatial smoother filters are in the intensive care unit and it looks like the patients may not survive. :( I made an attempt at writing a *robust* duplicate detector that is insensitive to frame size, frame overall luminance, and noise, and which doesn't require a threshold. Believe me, that is a holy grail! It's funny though, because our brains can do the job quite easily, and I don't know how. Anyway, given the head-against-wall banging I've been doing recently, it felt good to make something that may actually be useful.

[trbarry]

Donald -

Cheer up. Nothing ever works on the first try, and most things NEVER work completly correct. I throw away more stuff than I ever release, and sometimes wish I'd pitched even more.

And sometimes it is only when I am finally just totally pi$$ed at some problem that I finally get some inspiration.

- Tom

[vidiot]

Originally posted by neuron2

...I made an attempt at writing a *robust* duplicate detector that is insensitive to frame size, frame overall luminance, and noise, and which doesn't require a threshold. Believe me, that is a holy grail! It's funny though, because our brains can do the job quite easily, and I don't know how. Anyway, given the head-against-wall banging I've been doing recently, it felt good to make something that may actually be useful.

So you´re making a new approach to the copysame filter from Marc FD?
Why the hell is it a difficult task to adopt the natural way of thinking (comparing)!(not really a question) - I would like to introduce the old idea of "fuzzy-logic". Just because even our brain and our eyes won´t work real good at these speeds - but good enough.

I hope you´ll have success with it!

;)

Harald

[Didée]

Originally posted by neuron2
[...]

Who needs WarpSharp? :)

[...] [/B]


Me.

---------

But first I have to test this new plaything, of course ...

[ErMaC]

The filter's definately a great concept. I just emailed Donald with some torture test clips from the Region 2 eva discs which have several different kinds of non-digital-related artifacts (all of them have to do with mastering issues and the DVDs themselves are almost a constant 9MBit/sec) that I can't seem to eliminate with the new sharpening filter.

Included is a screenshot of one of the trouble spots - it's like dotcrawl noise that I've seen in Laserdiscs before. The image is unsmoothed or sharpened, however sharpening it doesn't get rid of the problem from what I can tell.

[Blight]

This looks great for animation, the thing about warpsharp is that it worked really nicely for live footage as well.

I also would like to prompt support for ffdshow type post-processing for this filter. A lot of us have 320x240 type animation already encoded and doing real-time bicubic resize to 640x480 and applying such a filter should look amazing on animation.

[Guest]

@trbarry

Thanks for encouraging words. Of course, your perspective on this is imbued with wisdom.

@vidiot

>So you´re making a new approach to the copysame filter from Marc FD?

No. A good robust duplicate detector has all sorts of applications. But specifically, I want it for Decimate().

>Why the hell is it a difficult task to adopt the natural way of
>thinking (comparing)!(not really a question)

Nobody yet understands how our brains do the work that they do!

>I would like to introduce the old idea of "fuzzy-logic".

That's an idea. I'll look into it.

@Didee

I have a problem with the *concept* of WarpSharp. It is just too destructive of video content for my taste. I can show you simple clips where even with low strengths the distortion of features is really bad. But as they say, one man's meat...

@ErMaC

>Included is a screenshot of one of the trouble spots - it's like
>dotcrawl noise that I've seen in Laserdiscs before. The image is
>unsmoothed or sharpened, however sharpening it doesn't get rid of
>the problem from what I can tell.

There's no reason to suppose a sharpening filter will remove artifacts! I'll be happy to look at your clips with a view to helping you, but this filter is not designed for such tasks.

@Blight

This filter works fine on live footage as well.

[Acaila]

Very interesting filter neuron2.

One thing that had me confused a bit is the way the threshold threshold works. Lower it and you'll increase the sharpening effect, increase it and you'll get less sharpening. Most filter's thresholds work the other way around...
I couldn't detect much effect of the strength setting, the threshold setting had a far greater effect in comparison.

I couldn't detect any noise amplification, which is great (and as it should be I know :))

Even for normal movies this filter seems to work great, I applied it to my Matrix rip for the better (threshold=5, strength=50). For a first test it worked marvelous. I don't have any anime to test it on, so..

Great job, keep up the good work!

Ps. It does seem to increase the EE effect though, but I guess that's unavoidable?

[Guest]

@Acaila

No, no, no. You're all confused. :)

The threshold sets the amount of picture that is considered edges. Set mask to true and then play with threshold. You want to have just the important edges shown and not grain or noise. Then turn off mask and the strength will determine how much sharpening is applied to the areas that showed as white in the mask=true view. Strength 255 is a fairly strong sharpen, so if you just played around with values close to 50, you may have thought it didn't have much effect. The sharpening is continuously variable from strength 0 to 255.

You appear to get more sharpening as you lower the threshold because more areas of the picture get included in the sharpening. But don't set it too low or you'll amplify noise again.

You don't need a really low threshold because the eye is going to be drawn to the important edges; those are the ones you want to sharpen. This is why the masking idea works better for sharpening than smoothing. For smoothing an edge detection failure destroys picture content; for sharpening, it leaves it alone. :)

Please, what is the EE effect?

[MaTTeR]

Originally posted by Acaila

One thing that had me confused a bit is the way the threshold threshold works. Lower it and you'll increase the sharpening effect, increase it and you'll get less sharpening. Most filter's thresholds work the other way around...


Correct me if I'm wrong somebody but I think Unsharp Masking in Photoshop and Gimp work exactly as the filter does here. Perhaps it's just the nature of UM but you will notice the same effect when using Photoshop.

Judging by the before/after screenshots that Acaila sent me this filter looks very impressive to say the least. I hardly noticed any added noise to the processed shot:)

[Guest]

No, it's much better than unsharp masking, because it affects only the edges. Go to the web site I linked and you'll see comparisons. Unsharp masking is really nothing special. It's just an old photographer's term for a sharpening convolution kernel. The key for this filter is that it applies the kernel only to the edge areas.

[Acaila]

No, no, no. You're all confused. :)That's ok, happens all the time :)
I think Koepi can attest to that ;)

You get more sharpening as you lower the threshold because more areas of the picture get included in the sharpening.So lower threshold => increased effect, higher threshold => decreased effect as I said. But most filters work lower theshold => decreased effect, higher threshold => increased effect. I did play around with the mask/threshold settings and they supported this (lower theshold => more white).
So I don't really understand which part I'm confused about..

EE = Edge Enhancement
Sharpening done during the DVD creation process, creates halos around contrasting lines when done too strongly.
Almost all DVD's have it (that I have seen), but usually it's hardly noticable. Until you apply a sharpening filter which obviously strengthens this effect. I don't think there's any avoiding this though, as how else are we supposed to sharpen something?

Maybe only sharpening in luma or chroma gives a better output? Just an (crazy ;)) idea.

[Guest]

@Acaila

OK, you're not confused. :) The threshold parameter is for the edge detector. Think of it that way. The lower the threshold, the more edges you see.

When I perfect my masked smoother (rough beta already available), you'll be able to remove the EE's prior to the sharpening stage.

I already plan for Y only when I add the YUV support.

[zettai]

@neuron2

Do you think it could be worthwhile combining the sharpen and smoother into one filter using the same mask or do you envisage a situation where you might want a different edge threshold for the sharpen and for the blur?

I guess it would only matter for simplicity and possibly speed, not sure - just thinking out loud.

[Guest]

@zettai

Yes, it is a fairly obvious final step to combine the two filters. But it is premature for that right now (until the independent functions are robust and valuable on their own). I think it is already clear that the masking will have to be different for the two. It would be a big mistake to prematurely do this without understanding and perfecting to the extent possible each function in isolation. It is the smoothing that needs more work at this stage. I view combining them as an optimization, and not as any kind of wonderful clever new concept. :)

[Zarxrax]

Wow, this is easily the best sharpening filter for anime thats out there! There's one hitch though, perhaps you could help me understand a little better. Sharpening filters always make the video less compressible. On a small test I just did with this filter (default settings, 100% quality in xvid), the sharpened version came out 60% larger. Now that is a HUGE compressibility hit! What I'm thinking, is if one were to use this filter, then it would obviously make the movie less compressible, causing a larger average quantizer. Now by having larger quantizers, that would just smooth the pitcture back out a lot wouldnt it? So in the end, wouldn't you end up with something that was no sharper (possibly even blurrier) than the original, and significantly lower quality than if you would have just encoded without the sharpening?

[Guest]

60% sounds high. I will do some experiments.

[Bulletproof]

That is a known debate, some people prefer to encode using Bilnear filters when using MPEG-4 codecs. The files will not be as sharp, but compression will be better. It is known that MPEG-4 codecs do not like sharp edges, however this does not in any way make sharpening filters useless. XviD has options to use an MPEG quantizer which is sharper than H263 and you should be able to retain the sharp details. There is also the future obviously and there is a ton of possibilities and improvements that can be made in the video encoding world, which makes it so exciting.

[Guest]

My tests showed a 49% increase. I guess Bulletproof is right (for present technology). Do you want sharp or do you want small? It is your choice. :)

[Guest]

How silly of me. MSharpen just *is* a filter for doing (narrow) EE. :) What I was referring to was the ability to remove mosquito noise prior to applying MSharpen.

But it occurred to me (as things are wont to do from time to time) that if you did unsharpening (contrast reduction, not blurring) just in the edge areas (!), you would attenuate exisiting EE. You'd have to dilate the edge map I imagine. Soon to appear then: my AntiEE filter!

I can use MSharpen to make torture test clips for AntiEE. ;)

[Guest]

It also occurred to me that I can replace the standard sharpen kernel with the same kernel used in Xsharpen (or some other rank order kernel), thereby avoiding the EE that some find objectionable. I will experiment with it.

[Defiler]

This is cool. I was compressing the new He-Man movie (animated) the other day, and I thought to myself "What I really need here is the opposite of MSmooth."
Even if it hurts compressibility, and it will, there are times when you have plenty of bits. Example: One hour cartoon, 464x464 resolution, one 700MB CD.
Time for some testing, I think.

[Bulletproof]

I've been actually looking for a filter that darkens the lines to the animation. This would be a sort of sharpening filter which wouldn't cause the codec to use extra bits, in fact, it would probably save some (very minor). The only problem is that current thresholds cannot accurately detect the lines. I do have the coring filter but this causes problems with shadows that are not pure black. In fact all video is missing pure black, its all mixed shades of black which is causing the problem. Often times I get speckled black dots when using the coring filter cause it cannot clamp to all the proper shades. Though maybe if you use a denoiser on the mask?

[stickboy]

Have you tried flaXen's Cartoon Filter (http://shelob.mordor.net/dgraft/flaxen/flaxen.html) for VirtualDub?

[ErMaC]

Jeez neuron, how many filter are you gonna get out of this edge detection algorithm of yours?

Make an edge-constrained coring filter like the person above might be able to use? :D

Maybe an edge-constrained blurring type algorithm to help remove aliasing from deinterlaced-via-interpolated frames?

Or even weirder, and edge-constrained warp sharping filter?

[Belgabor]

Hm, mentioning the algorhythm, instead of completely relying on threshold, perhaps some kind of "filter" to exclude noise could be added to the edge detect algorhythm.

Just a funny idea of mine.

Cheers
Belgabor

[Guest]

MSharpen version 1.0 beta 2 is now available:

http://shelob.mordor.net/dgraft/msharpen/msharpen10b2.zip

This version is improved as follows:

* Supports YUY2 and RGB space -- no conversions required.
* Significantly faster, especially in YUY2 space.
* Sharpens only the Y channel in YUY2 space.
* Default parameters can be defined by the user.
* High/low quality edge detection to allow quality/speed tradeoff.
* Improved and updated documentation.

Feedback will be appreciated.

Future versions will add:

* Additional sharpening algorithms, e.g., rank order.
* Low-level optimizations for speed.
* Release of source code under GPL.

Thank you for your feedback and support.

[Guest]

I had a bad copy of Beta 2 up there for about 15 minutes. If you downloaded it then and think it acts as if the default threshold is 150, you're right! Download it again, please.

Thankfully, I have a policy of always downloading the uploaded filter and testing it as if I am a user. :)

[Guest]

@Belgabor

The edge detection already has some noise reduction. It could be better but we have a competing user requirement: SPEED!

[Guest]

@ErMaC

Now this is why I love this forum so much. You guys emit an unceasing stream of interesting ideas! Thanks. I'll queue those up behind my AntiEE filter.

[Bulletproof]

It's too bad you can't use the Sharpening filter and the filter to blur out jaggies from deinterlacing at the same time, or can you? ;)

Warpsharp might benefit from this as well if it could tell which frames are jagged, cause weird artifacts happen if you use a moderate strength warpsharp on jagged edged frames.

[stickboy]

Originally posted by ErMaC
Jeez neuron, how many filter are you gonna get out of this edge detection algorithm of yours?How about adding general masking operations to avisynth, and then you can create an edge mask and apply whatever filters you want with it.

[Guest]

Sounds good but many things *require* the functions to be integrated.

Anyway, there is the Layer filter, but I've never been able to make it work and it is not general enough.

[OUTPinged_]

@DG: Msharpen is a really great step forward from warpsharp.

Here is an idea for a blur filter of similar kind.

There is a problem with crappy dvds that some edges are overdone already (too sharp comparing to other). Sharpening the rest of the image is an option, but there are cases when the lines are just too sharp and current "smart blur" filters like UnFilter are blurring the smooth edges too (though to a lesser extent than a "dumb" blur filter).

What is needed there is a filter that will find those "too sharp" edges and blur them a bit so we get unified amount of peak sharpness in the encode (edge can be more blurry than the thresh, but not more sharp).



And AntiEE filter will be an absolute killer stuff :-) Enormous amount of old animes will be possible to rip with sharp lines and get rid of that "i smell a vhs source in your encode" approach.

[PrinceAli0]

Ok this may sound silly but I'm just beginning to understand this whole encoding process of DVD (specifically anime) to DivX, XviD, ect. I reviewed your pictures and this sharpening filter looks great. From what I understand it is to be used with the program Avisynth, is there plans to port this to VirtualDub or is there a plugin that I can add to my .avs file that I load up into VirtualDub for encoding that will make use of this excellent sharpening filter. Have I gone completely off track here or do I make some sense, please let me know.

Thanks

PrinceAli0

[Didée]

@PrinceAli0

[Having a big grin on my face]
One of the most-often-arising-question is: "I desperately need the Vdub filter XYZ, could anyone port it to AviSynth, please?"
The other way round is a really rare case.

Big tip: Learn AviSynth!
You may not see the all the advantages by now - but once you got it, you´ll wonder how you could have ever done without it.

[Didée]

Originally posted by OUTPinged_
@DG: Msharpen is a really great step forward from warpsharp.

[First: Donald, I really appreciate your msharpen filter, and all the others, too. Always great work.]

But I want to state - once more - that WarpSharp should NOT be compared with ANY pixel sharpening filters. These are two totally different techniques. Anyone wants to compare "Resize" with "Loop"? Which one is better?

All pixel sharpeners enhance the delta of a pixel to its environment.
WarpSharp modifies the environment.

All pixel sharpeners make the content less compressable.
WarpSharp increases compressability!

Take some over-blurry content.
What can a pixel sharpener do with it? Nothing!
WarpSharp can, to a certain amount.
(If a one-pixel boundary gets so blurred that it extends over more than three pixels, pixel sharpeners won´t bite at all - except for unsharp masking, but that´s another story.)

On the other hand:
- WarpSharp does a bad job with very fine detail
- too strong settings yield to very strange results
- [much more ...]

I´m really looking forward to what might be squeezed out of msharpen, it will be a really great filter. It is yet now, indeed.

But, for sure, it is not the successor to WarpSharp.

Thanks for reading (and thinking about ;) )

Didée

[manono]

Hi-

My first reaction to the Prince's post was the same as yours Didee (sorry-my keyboard doesn't know how to make the accent over the "e"). But then I read it again, and it looks to me like PrinceAli0 is already frameserving with an .avs.

So, Prince, it's the same process as using TextSub in the other post of yours. You load the plugin and add the Msharpen line to your .avs. The neuron2 filters always come with very good documentation, so if you read it, I think you can figure it out. Good Luck.

[OUTPinged_]

didee:

It is a successor to warpsharp ripper-wise(I know it uses different kind of algorithm). it does similar sharpening and it doesnt make straight lines curvy. If group rippers will finally stop using ws+sshiq with high thresholds... :-)

Can warpsharp be enchanced to keep lines straight? :-(

[Belgabor]

Originally posted by neuron2
@Belgabor

The edge detection already has some noise reduction. It could be better but we have a competing user requirement: SPEED!

Well, if we're talking users here, if it isnt too big a hassle, why don't let the user decide? (lets say mode=0 - no added edge noise removal, mode=1 - additional edge noise detection, mode=2 - [new idea] trying to discern between 'pointy' noise and 'liney' edges [which is prolly very very hard to code & slow])

A point would be you could treat a difficult scene with a lower mode (speedwise) and the rest of the movie with mode=0 so the total speed impact would be bearable.

Cheers
Belgabor

[PrinceAli0]

manono,

I think I'll just give all credit to my encoding to you my friend. You have helped me sooooooo much. I'd hug ya if you were here :) I figured it'd be another line in my .avs file as I read in the doom9 guides that it is the instruction file for frame serving or something in that manner, plus what you told me in my other posts and the instructions I read from the guide included in neuron2's work tiped me to that. But i figured I'd check first before I did anything stupid in my learning process.


To better understand this from my perspective, what Didee is saying is that avisynth is another frame server like virtual dub, right ? I'm gonna read the comparison on the two but I think its by the maker of avisynth (might be wrong here), is there something less biased by a 3rd party like Doom9(if that comparision is biased at all, I haven't looked at it yet)

Thanks

PrinceAli0

[MoonWalker]

Hi neuron2,

After knowing that Msmooth10b2 is in your garage for repairs :) I decided to test Msharpen10b2..There wasn't something I didn't like..There was a thin line on the edges..I thought maybe it was the strength..Firstly I set it to 100(default) and then raising and lowering to see what was happening..Raising was main this more strong, lower made less strong but still noticable..I attach an image with strength 20..

My avs :

SetMemoryMax(40)
LoadPlugin("D:\DivX\GORDIA~1\mpeg2dec.dll")
LoadPlugin("J:\Matrix\1cd-xvid\640\New Folder\MSharpen.dll")
mpeg2source("J:\Matrix\matrix.d2v")
MSharpen(threshold=7,strength=20)
Crop(0,78,720,420)
BicubicResize(640,272,0,0.5)

MoonWalker

[Bulletproof]

I'm not sure, but I think that's what the filter is supposed to do. It's raising the contrast on edges and not raising the contrast on noise.

[High Speed Dubb]

Originally posted by neuron2
But it occurred to me (as things are wont to do from time to time) that if you did unsharpening (contrast reduction, not blurring) just in the edge areas (!), you would attenuate exisiting EE.

@Neuron,
How does unsharpening differ from blurring?

[soulfx]

Yeppers. Thats a normal operation of a sharpen filter. DVD's might not be the best source to test out for sharpening as they are allready probably sharpened anyways.

To really figure out how good this filter works get yourself something that just seems like your looking at it through cateracts and give it a shot (either an orig. source or just run a source through some filters to soften it up a little).

Peace,
SoulFX

[kyousuke]

@donald
thx for the YUY2 mode :)
i always work in this mode when a filter can do it

about this :
* Sharpens only the Y channel in YUY2 space.
could you give me some informations ? Could we sharp X channel in future or not ? etc... thanks :)


@moonwalker
donald said : Although designed specifically for anime, it may be useful elsewhere.
it's right but don't really exact for now on non-anime videos.
perhaps it's a sharpening bug on the edge's chroma with the background, don't know really...
tests this : MSharpen(threshold=7,strength=20,mask=true)

i think you should wait new sharpening algorithms.

++