photohop high pass sharpener ported to avisynth

[GilGalaad]

sharpening, in artistic photography, it's a fundamental step in postprocessing, it compensates the natural softness of shots with high aperture, and it's one of the most important valutation parameters for a "good shot". that's why is vastly discussed on forums.
there are many ways to implement it. one of the most easy, fast and reliable one, it's achieved using a small and understimated photoshop filters, known as "high pass filter", or "accentua passaggio" in the obscene italian translation.
in video encoding too, oftwe we complain about lack of detail, excessive softness of images, or excessive grain, and removal of that causes even less detail.
i've seen many shapeners around, and all of them do their job pretty good, but sometimes they are difficult to use, plenty of options, and requires external plugins. moreover i think none of them uses the simple high pass filter method. so i tried to implement it by myself (and maybe i've done something already available...in this case, forgive this post), mostly because i love to use code written by me. it's simple, it uses only internal avisynth function, does not requires dll's.
let's see the script and the syntax:


function HighPassSharpen(clip c, int mode, float radius)
{
v1 = c.binomialblur(radius,radius)
v2 = v1.invert()
v3 = merge(c,v2)
v3 = v3.greyscale()
v4 = overlay(c,v3,mode="hardlight")

v5 = (mode==0) ? v3 : v4
return v5
}

parameters are quite self-explanatory, the first is the clip to process, the second tells to work in "debug" mode (value 0) or "release" mode (1) , the third specifies the strenght of the effect (accepted values from 1 to 9).
let's see with some screenshot how to proceed, beginning from the original vob:

http://img3.imagebanana.com/img/1q8841l6/thumb/1originale.png (http://img3.imagebanana.com/view/1q8841l6/1originale.png)

not much to say, beautiful image, but the source dvd is not so good, bitrate seems too low, especially in aerial views of the sea (water is a killer for mpeg)
let's call the shapening function in preview mode, and we get this:

http://img3.imagebanana.com/img/bee5mgh6/thumb/2tuning.png (http://img3.imagebanana.com/view/bee5mgh6/2tuning.png)

wich is the same that we get in photoshop with the application of the high pass filter. the resulting screen is almost completely grey, with item's edges "traced" more decisely. technically speaking (you can google about it), to achieve a more sharped image we can start with a blurred image, inverted, and overlapped on the first, with 50% opacity. the more the blur, the more the sharpening. gaussian blur has standard deviation as parameter (the famous sigma), photoshop semplify the problem, using the pixel radius as parameter. avisynth does not use exactly the same function, neither the same parameter, at even at the maximum value, the blurring is relatively small (1/3 pixel). happily, gaussian blur is additive (not linearly, tough). so reiterating the call multiple times, we can obtain a good approximation of a wider range gaussian blur.
my script accepts numers from 1 to 9 (0 is valid too, and it return the original image, cosuming cpu times. this can be useful to make a rapid check between processed and unprocessed image, without commenting other part of the script). value 1 corresponds to 1/3 pixel, value 3 (recomended) corresponds to 1 pixel, growing up not linearly.
preview mode is useful to make a fine tuning of the "radius" value, you can evaluate immediately which edges will be traced and how much, you can start from a low setting and rise it step by step up to the desider threshold, enphatizing the detail, but preventing the noise to be taken in account.

so when you have finished you can use mode=1 and look at the final result:

http://img3.imagebanana.com/img/abw1tki7/thumb/3sharpened.png (http://img3.imagebanana.com/view/abw1tki7/3sharpened.png)

of course you already know that, but i'll say it anyway. sharpening an image is a way to "guess" detailt not present in the original, that's why this must be done with caution, setting the value too high will produce a "plastic" effect, oversharpness, aliasing. and will amplify the noise, even create it, sometimes.
you will notice in this screen, some compression artifacts in the left part. we can think to apply a strong denoising, to flatten the blocks, but that will flatten the detail too, and then recover the crispness with the high pass filtering.

following you can see the image denoised with mvdegrain3 (sad 400), and then sharped with my script (wich has to be applied always after denoising). it's necessary to remind that sharpening is the opposite of flattening in terms of compressibility too, a sharpened imaged is always bigger than the denoised one, and often is even bigger of the original unprocessed.

http://img3.imagebanana.com/img/z8ilxve/thumb/4denoised.png (http://img3.imagebanana.com/view/z8ilxve/4denoised.png)

http://img3.imagebanana.com/img/j81egw4m/thumb/5denoisedsharpened.png (http://img3.imagebanana.com/view/j81egw4m/5denoisedsharpened.png)

if you prefer, you can modify the calls of blur(1.58) with blur(1.58, mmx=false). avisynth manual reports the latter as a slower but more precise algorythm. to be hones, i couldn't see the difference with my eyes, so i leaved it this way.
always in manual, it's suggested the value of 1.0, for repeated calls of the function, i used 1.58 because the gaussian blur can be executed in 2-passes, horizontally and vertically, and then mixed together, this give exactly the same result, with lesser computation. the suggested value 1.0 seems to do it in a not symmetrical way, so i avoided it.

some trials on anime material showed that the even in the lighter setting, this seems to be too much, and creates excessive halo and aliasing. i hope you can fine tune the blur value to find an optimal value, but i'm not a master of anime encoding, so i can't really help.

i always appreciate comments, positive and negative ones, and feedback if you want to try this script.
i am the first to say that the code can be writted much much better. there's no check on out-of-range parameters, the flux control sucks, but avisynth is no c++, and my graduee is near, i don't have much time to spend on hobbies.
and, i repeat, i'm not sure this work has not been done by anyone else, so please forgive me if i bothered you with a well known solution.

update to 1.1: changed the blur function, the range parameter now seems to have sense. you need the variableblur plugin. tnx to gargalash, gavino and neuron2.
update to 1.2 changed overlay into merge, corrected the shift to black, global luminance is definitely closer to the original. tnx to didée.

[le_canz]

Hi,

Your script lacks "" in the beginning :

function HighPassSharpen(clip c, int "mode", int "radius")

I'm still quite new with scripting too, but it looks like an interesting method :)

Unfortunately it generates a lot of ghosting with strong settings (I tried it with a simple DV source, uncompressed)

Maybe more skilled people here will be able to make it better :)

[GilGalaad]

"" are needed only if you want to mark a parameter as optional. since there's no control and no defaults, i don't want anything optional :)
as i suggested, you should use it at lower settings. anyway avisynth does not implement blending methods like photoshop does, haloing, aliasing and some chroma differences are the unwanted result of stronger settings.
in the future, i might want to write some code to make a dll, because i know the potential of this method applied to photography. btw thanks for your comment :)

[le_canz]

It's strange, there was an error when I tried this function without ""

Adding it resolved the issue.

Edit : forget this, the problem was from another part of my script....

[Gargalash]

GilGalaad,
I really like this so far, thank you for this. Here is the problem I encountered:
I experience a color shift. It "looks like" the hue is shifted towards yellow (it's what I see, don't know what's going on). The color space I'm working in (in Avsp) is YV12.
I used a .d2v source from a PAL DVD.

The way it works, it seems to sharpen a lot what is already a clear edge (halo, gosthing...) but it does very well at 4. When I put a stronger setting, looking at the highpass, almost no "softer" edges are added, only the more visible ones get even more sharpening So it doesn't sharpen more edges, it only sharpens more what is already there.
I don't know much, but here is an idea: something that can analyse the highpass results and adapt sharpening to sharpen less on the higher contrast edges and more on the lower contrast edges.

I tried HQDering before sharpening as well but it had no effect on the strong edges.

[GilGalaad]

@gargalash: i've noticed the shift.
to be honest, i think it's a problem concerning the "blending mode" used here -> v4 = overlay(c,v3,mode="softlight"), because until the "grey" frame, the result is not distinguishable.
avisynth's softlight is quite different from photoshop's softlight.
i am not an expert avisynth scripter, and the manual is pretty trivial, i am not even sure if ad adaptive approach is doable.

i will experiment, but i think that for a decent result, i have to write a dll. but i don't know if i will have the time and the skill for it.
the point of this thread is mainly to show an alternative method for sharpening wich is clean, fast, and proved to give good results in other applications.

edit: i've modified slyghtly the script. i made the "gray" frame really grey, with greyscale(). the color shifting seems gone. of course the average luma goes down a bit, this is result of a more contrasted image, but the chroma is untouched. verified with coloyuv(analyze=true)

[Gargalash]

Great, I will keep testing it out! Thanks

[Gargalash]

GilGalaad,
At tried something in photoshop with the High Pass filter. By using a big radius value (60) I get an interesting contrasting.
I duplicate a layer, apply high pass with radius 60 on the top layer soft light blend it. You can try with any image, you will see what I mean.

I would like to get this effect with your script. Radius doesn't mean the same thing in your script. Do you think you can modify it to get this effect in avisynth? I tried to understand how to do it, but I have a lack of knowledge that stops me. Thanks!

[GilGalaad]

as i said, there is direct but not LINEAR correspondence between the radius of a gaussian blur (photoshop raius) and avisynth blur. the formula is not known exactly. you can achieve almost any radius in avisynth simply repeating the blur function many times. the radius parameter of my fuction should be intended as "repetitions" of the avisynth blur, and should be coded with a for cycle to allow an arbitrary number (instead of this not elegant form). but avisynth does not support reiteration, and recursion is slow, ram consuming and with a crazy syntax.
my suggestion is to find an optimal value by yourself. do the blurring in photoshop, then open avisynth and repeat the call of blur many times until you get something very close. probably for a radius of 60 pixels this mean repeating the function 200 times. you can figure what this mean in cpu time, but that's the only way avisynth allows.
as i said, i'd like to have the time to implement this in a dll, would be a final workaround for all those limitations.

[Gavino]

recursion is slow, ram consuming and with a crazy syntax.
I have to disagree with you on all three counts.
There is no 'run-time' penalty in time or memory for using recursion. Any overhead is 'compile-time' only, ie during script loading, and is scarcely noticeable for simple functions with reasonable depths of recursion. The resulting filter graph is exactly the same as if you had unwound the recursion 'by hand'.

As for the syntax, it may look strange if you haven't used recursion before, but it becomes quite natural once you understand it. Your function can be written simply using a recursive 'helper' function, eg
function RepeatBlur(clip c, int n) {
n > 0 ? c.Blur(1.58).RepeatBlur(n-1) : c
}
Then the code at the start of your function
v1 = (radius == 1) ? c.blur(1.58) : c
v1 = (radius == 2) ? c.blur(1.58).blur(1.58) : v1
...
v1 = (radius == 9) ? c.blur(1.58).blur(1.58).blur(1.58)... : v1
can be replaced simply by c.RepeatBlur(radius). This might even be faster to load since there is less text to parse, and of course now the radius is no longer limited to 9.

[Guest]

Common misconception:

Repeated blurs do not increase the radius. Repeating blurs with the the Blur() filter gives only a better and better approximation of a Gaussian blur with a fixed width.

For a truly variable radius, you need something like this:

http://neuron2.net/boxblur/boxblur.html

[Gargalash]

I experienced neuron2's clarification before asking how to get a larger radius. I had modified the function and realized that 60 compared to probably 1000 repeated blurs did not change anything. There was no modification starting at something like 12 repetitions.

Thanks all for your comments.
neuron2, I will be playing with your boxblur later today.

[GilGalaad]

@gavino: tnx for the syntax help :) but, forgive me, any syntax that does not allow any flow control except ternary operator...is crazy, imho, and i can say for sure that i have use recursion before. but tnx again for the example, it's clear and simple.

@neuron2: interesting point, and tnx for the clarification. but the misconception comes from this assert in the avisynth help page:
"If you want a large-radius Gaussian blur, I recommend chaining several copies of Blur(1.0) together. (Anybody remember Pascal's triangle?)"

[Guest]

That just shows that the misconception is as common as I said. :)

In a very limited sense it is correct in that the tails of the gaussian are pushed out as it is iterated (but it rapidly converges), but that's not what people usually mean by a larger radius.

[Gavino]

Repeated blurs do not increase the radius. Repeating blurs with the the Blur() filter gives only a better and better approximation of a Gaussian blur with a fixed width.
That's very interesting. I have also been misled by the statement in the docs.

When you say a fixed width, do you mean fixed independent of the parameter to Blur(), or that for a given parameter value, it converges to a Gaussian whose radius depends on the parameter.

In other words if Blur(x).Blur(x).Blur(x)... converges to Gaussian(r), does r depend on x or is it fixed?
If the first case, do you know what is the relationship between r and x, and if the second, what is the fixed radius?

[Guest]

It's related to the size of the blur kernel, which is fixed at 3x3. I suppose the variance of the gaussian is 3 or some function of 3.

[Wilbert]

I will correct the docs when i've time.

[Gavino]

Repeated blurs do not increase the radius. Repeating blurs with the the Blur() filter gives only a better and better approximation of a Gaussian blur with a fixed width.
At first sight, this is a surprising result.

Theory shows that repeated convolution of any blur filter converges to a Gaussian blur, but also states that the radius of the blur keeps growing proportionally to the square root of the number of repetitions n. Or, to put it another way, the 'variance' (=radius^2) is directly proportional to n. So in principle, repeated applications of Blur() should be capable of (eventually) producing a Gaussian blur of any desired radius.

Blur(1.58) [actually, Blur(log(3)/log(2))] has a (1/3, 1/3, 1/3) kernel which can be shown to have a variance of 2/3.
Hence, repeating it 6 times gives a good approximation to a Gaussian of radius 2 [=sqrt(6*2/3)]. In theory, you should be able to create any radius by repeating it enough times, although (because of the square root factor) the number of repetitions required becomes impractical for larger radii. For example, 150 would be required to get a radius of 10, while to get a radius of 30 would require 1350 iterations.

However, even these numbers are beyond the reach of the Blur() filter. Simple tests confirm neuron2's statement, showing that beyond a certain number of repetitions (about 100), there is no further change in the image, putting an upper limit on the radius that can be reached (about 8).

So what is behind this limit? I believe it has to do with the limits of 8-bit pixel precision.
As you add more repetitions of the filter, each one has a smaller and smaller effect, though in theory it never reaches zero. However, once the point is reached where the effect of the next filter application is a difference too small to be seen in 8 bits, any further iterations have no effect.

As neuron2 suggests, the practical way to construct a Gaussian blur is to start with a wider box blur, such as neuron2's own filter, for which only a small number of repetitions is required to give any desired Gaussian.
For example, BoxBlur(r, 1, 1000) has a variance of r*(r+1)/3, so 6 repetitions (or 2 applications of BoxBlur(r, 3, 1000)) approximates a Gaussian of radius sqrt(2r(r+1)).

[Guest]

Thanks for the clarification, Gavino. As you say, the radius does grow slowly but it's impractical for getting wide kernels.

[GilGalaad]

i updated the script in the first post, using variableblur.
now it doesn't require recursion, it's faster, and radius makes sense. binomialblur can be replaced by a true gaussianblur but for the range i'm using for HD material (radius 1-3) i can't see a real difference (and manual says that, in this range, binomial is a a very good approximation of gaussian)
i preferred variableblur to neuron2's boxblur, since boxblur is a virtualdub plugin and a rgb colorspace conversion should be necessary to make it work (or am i wrong?).
to be honest i'm pretty happy of the results, i'm posting a screenshot from Robin Hood, where is cleary visible the detail gain in the skin and in the sweat.
thank you everyone for testing, corrections and suggestions :)

http://img3.imagebanana.com/img/hxrjaqc5/thumb/orig.png (http://img3.imagebanana.com/view/hxrjaqc5/orig.png) http://img3.imagebanana.com/img/5ehvz0jt/thumb/sharpened.png (http://img3.imagebanana.com/view/5ehvz0jt/sharpened.png)

[Didée]

I recommend to try this:

o = colorbars().converttoyv12()

m1 = o.HighPassSharpen(1,1,method="overlay").subtitle("""method = "overlay" """)
m2 = o.HighPassSharpen(1,1,method="merge") .subtitle("""method = "merge" """)

interleave(m1,o.subtitle("source"),m2)
histogram(mode="levels")
return(last)

#------------------------------


# a modification, to show why it is safer to *not* use Overlay() whenever possible ...

function HighPassSharpen(clip c, int mode, float radius, string "method")
{
v1 = c.binomialblur(radius,radius)
v2 = v1.invert()
v3 = (method=="overlay") ? overlay(c,v2,opacity=0.5)
\ : (method=="merge") ? merge(c,v2)
\ : c.subtitle("unknown method")
v3 = v3.greyscale()
v4 = overlay(c,v3,mode="hardlight")

v5 = (mode==0) ? v3 : v4
return v5
}
Personally, I would leave out the something-is-always-wrong "Overlay()"-filter completely, and build it manually with just-do-what-you-want-to-have "mt_lutxy()" instead ... but that's just me. ;)

[GilGalaad]

impressive difference, didée!
this change corrects the "black shift" that was actually present, and that i was erroneously imputing to the global contrast enhancement. it's not only visible histgram-wise, the difference is evident.
thank you very much :D

[Didée]

Playing a l'il more, it seems that Invert() works a bit fishy ... it seems to center around "127.5" for YUV clips. Just tried it - the mapping for Invert() is

in ---> out
...
126 --> 129
127 --> 128
128 --> 127
129 --> 126
...

Centering at 127.5 is highly symmetric for 0..255 range, sure ... but it's a bit unlucky for filters that assume the centerpoint at '128', like Overlay(soft/hardlight), or MaskTools' filters ...

(That's why I prefer to do as much as possible in MaskTools: less dicegame, more control.)

[GilGalaad]

photoshop's inversion is 128->127, like avisynth (just checked).
and imho it is as it has to be. instead would be nice to have more control on the overlay function, but the formulas used in photoshop are only guessed (some with precision) but of course non totally known.
btw, i don't think a distance of 0.5 in luminance can make any visible difference in any filtering.

the real problem of this kind of filtering (even if for my tastes the results are amazing) is that radius of blurring is the only "strength" parameter. the more the detail, the more the noise, the preview can help to find a treshold to avoid flatty (and sometimes noisy) backround colors and to focus on detail, but the two thing are correlated. the more the sharpening of what you want to be sharpened, the more the noise you don't want to be amplified. an adaptive approach is of course doable with a single picture, just select the subject and sharpen, select the rest and blur if necessary (it's a basic technique for every photographer)...this is not in a movie. so in the future maybe it's better to focus on how turn the high pass filtering "smarter", than make it more precise (when the difference is so small).
but to be honest i don't know even where to start, i don't think a frequency approach is doable, cause detail and noise are both high frequency (and that's the point of this sharpener, and high frequency booster after all).

[Didée]

It depends on application. For a "general" task of inversion, that behaviour is OK. But when working with (greyscale) images in the sense of "difference maps", then it is *not* OK. A difference map absolutely must have a "zero element" (aka "do nothing"). And - in this context - the inversion of "do nothing" has to be "do nothing". Which isn't the case when the zero element is located at half-a-pixel-value.

Adding some sort of a "strength" parameter would be rather easy - just modify the diff'map with levels() or mt_lut(). That's one additional line, fairly easy.

The other aspect is that all of this "Photoshop highpass" method is not particularly impressing in any way. Create a blur, merge at 50% with the original input (at this point, it inherently applies a "strength" of 50%), then modify the input with that. The only difference to the "usual" sharpening routines is that the blur difference isn't applied linearly (additive), but non-linear (more in a "multiplicative" or "filter" way, due to 'softlight'.) We've already seen way more sophisticated examples for using non-linear amounts for sharpening. (*cough*)

Seen from this angle, it's just old hats ... a slightly modified gaussian sharpener, that's all.

(Yep, I hereby declare that I am bored.) ;)

[GilGalaad]

The other aspect is that all of this "Photoshop highpass" method is not particularly impressing in any way.
Seen from this angle, it's just old hats ... a slightly modified gaussian sharpener, that's all.

Well, to be honest, compared with other sharp packages, mostly based on unsharp mask theory, i find theese results quite impressive, but this may me just my taste, or my ignorance of the already existent filters.
I never said i've done something new, or revolutionary. I just wanted to reproduce in video what i often use in photos, considering the easy to use <-> results tradeoff (no billion modes, no zillions parameters for less halo, more halo...and so on)


We've already seen way more sophisticated examples for using non-linear amounts for sharpening. (*cough*)

o'really? :D :D :D
btw tnx again for your help.

[Didée]

(no billion modes, no zillions parameters for less halo, more halo...and so on)
Sure. Making a filter sufficiently elemental (aka primitive), it's ensured that it can't be misused ...


Anyhoo, we're talking just Unsharp-Masking, nothing else.

The primitive: (one possibility)
rY = 1.0 # blur radius
str = 100 # sharpening strength

mt_LutXY( c, c.BinomialBlur(radY=rY), "x x y - "+string(str)+" 100 / * +",U=2,V=2)
Ah, the simplicity.

For simulating the [soft]light blending mode, just adapt the LUT string.

Hint: over two years back, some nice LUT strings were posted here (http://forum.doom9.org/showthread.php?t=108292). [:yawns:] ;)

[manolito]

Hi GilGalaad,

the following comments are from a "Joe Average" dummy real world user. My computer is ancient and slow, and I do not need sharpening for comparing stills, I use it for real world tasks. For quite some time my standard approach to improve analog cable TV captures was:

DeGrainMedian(mode=2)
Undot()
Asharp(1,4)

and I am still quite happy with it. Your HighPassSharpen thread made me curious and inspired me to do a couple of tests. I do have LSF installed, but so far the gain in quality does not IMO justify the speed sacrifice. (Didée always gets bored if a script has less than 10 parameters, and he really gets excited if the speed gauge changes from fps to spf...)

Anyways, I did a couple of tests trying to match the parameters to get a comparable sharpening strength.

Speed:
Asharp(1,4) 9.36 fps
MSharpen(highq=false, strength=70) 7.95 fps
PSharpen() 6.18 fps
HighPassSharpen(1,1) 5.90 fps
LimitedSharpenFaster() 4.33 fps
LSFmod(defaults="fast") 4.04 fps


For quality I am not sure... I watch my encodes on an old fashioned 4:3 CRT TV, and all these encodes look pretty good to me. So I probably will stick with my good old Asharp plugin.


Some more remarks:
For the first incarnation of your HighPassSharpener a radius of 2 gave comparable results to LSF() or ASharp(1,4). The latest version seems to be much stronger, now a radius of 1 seems to be a better match to the other sharpeners.

And a little note for using VariableBlur.dll: Contrary to the documentation fftw3.dll is not only needed for GaussianBlur, in fact the dll will not even load at all if fftw3.dll is not present. Took me a while to figure it out....

Cheers
manolito

[Didée]

(Didée always gets bored if a script has less than 10 parameters, and he really gets excited if the speed gauge changes from fps to spf...)

Ah ... a joke! You see me amused. Hahaha.

Since explaining-in-words seems to be not very fruitful, I'll try with some pictures instead.


a) Source:

http://thumbnails2.imagebam.com/3903/aa2d4e39027424.gif (http://www.imagebam.com/image/aa2d4e39027424)
(c = mpeg2source("vt2_01_1.d2v")


b) Sharpening like we did since Adam & Eve:

http://thumbnails12.imagebam.com/3903/30f3aa39027426.gif (http://www.imagebam.com/image/30f3aa39027426)
( Apply the inverse of some blur filter )
( Here: c.mt_adddiff(mt_makediff(c,c.removegrain(11).removegrain(11))) )


c) Now, the novelty of Adobulus Highpassus Sharpulus:

http://thumbnails17.imagebam.com/3903/b7dfc239027430.gif (http://www.imagebam.com/image/b7dfc239027430)
( c.HighPassSharpen(1,1) )


Finally, to return to your joke ...

d) ... the difference between b) and c) :

http://thumbnails3.imagebam.com/3903/98ac9e39027431.gif (http://www.imagebam.com/image/98ac9e39027431)


Result:

We have nothing new here. It's the very same stuff we've been doing ever-since in Avisynth -- just that it's slower, more complicated, and less precise.

Perhaps you now understand my "bored" reaction a little better ...