BlindDeHalo() - removal of Edge Enhancement artefacts

[Didée]

Hello folks,

once more I did a little scripting, and came up with a procedure that seems quite usable to remove those ugly bright and dark halos from edge enhancement.

One important warning beforehand:

Be careful when using other functions of MaskTools before this script,
or when calling it before iiP() !!!

The MaskTools currently are a little tricky to use - I'll explain this further below.


First, the script:

# BlindDeHalo - v0.1
#
# This script removes the light & dark halos from too strong "Edge Enhancement"
# Adaption of the parameters has to be done manually for each source, the function is not adaptive.
#
# Needed Plugins: MaskTools.dll >= 1.4.15
#                 undot.dll
#

function BlindDeHalo( clip clp, int "strength", int "radius_x", int "radius_y", float "darkamount", string "mode", float "maskblur", bool "fullframe" )
       {
         strength =   default( strength,  80)
         radius_x =   default( radius_x,   2)
         radius_y =   default( radius_y,   2)
         darkamount = default( darkamount,1.0)
         mode =       default( mode,  "soft")
         maskblur =   default( maskblur, 1.0)
         fullframe =  default( fullframe, false)
         modus = mode=="hard" ? "hardlight" : "softlight"

         ox = clp.width
         oy = clp.height
         rx = sqrt(radius_x)
         ry = sqrt(radius_y)

         clp
         edge=Dedgemask(0,255,0,255,"6 10 6 10 -64 10 6 10 6",setdivisor=true,divisor=64).levels(2,0.5,6,0,255,false).undot#.coloryuv(analyze=true)#

         blur1 = bicubicresize(int((ox/rx)/4 +.5)*4,int((oy/ry)/4 +.5 )*4,-1.0,1.0).lanczosresize(ox,oy)
         blur2 = bicubicresize(int((ox/rx)/4 +.5)*4,int((oy/ry)/4 +.5 )*4, 1.0,0.0).lanczosresize(ox,oy)

         diff = YV12subtract(blur1,blur2)

         overlay(diff,diff,mode="hardlight").levels(0,1.0,255,88,168,false)

         darkamount == 1.0 ? NOP : overlay(last,last.levels(0,1.0,128,int(128-128*darkamount),128,false),mode="lighten")
         overlay(last,last.levels(0,1.0,255,128+strength,128-strength,false)) # levels reversed: no more "invert" in below overlay()

         overlay(clp,last.greyscale.blur(maskblur),mode=modus)
         fullframe ? NOP : MaskedMerge(clp,last,edge.deflate.expand.blur(1.58).inflate,useMMX=true)

         return(last)
       }
#

The most basic usage is as simple as: BlindDeHalo()

However, there are several parameters to adjust the filtering, and you definetly want to setup those parameters manually for each source.

Basically, the following is what is going on:

- The script does not reckognize any halos in the source! It simply looks for hard edges, and reduces the contrast in their neighborhood.

- A gaussian blurred (well, sort of that) version of the clip is subtracted from a sharpened version. This delivers a mask that is suited for the given purpose.

- Then, the source clip is "Softlighted" by this mask, et voilà ...


Parameter description

strength [int]
The overall strength of contrast reduction around hard edges. Default is 80, usuable range is 0 - 128.

radius_x, radius_y [int] [int]
This is [similar to] the radius of a gaussian blur. These radii must be adjusted to match the "thickness" of the halos in your source. Default is 2, usable range is from 2 to about 5.

Note: A radius of 1 disables the processing in the corresponding direction. E.g. for VHS-caps, setting radius_y=1 and radius_x=3|4|5 should be the right thing. (Didn't test this myself.)

darkamount [float]
With this parameter, the removing of dark halos can be reduced. A value of 1.0 will treat both bright and dark halos equally, 0.5 will process dark halos with only half the strength, 0.0 will disable the processing of dark halos.
Explanation: Theroretically, bright & dark haloing should be always equal. But I encountered some sources where the bright halos seemed to be much stronger than the dark ones, and so I implemented this parameter.
Moreover, generally bright halos are more distracting than dark ones. So it might be a good idea to process the dark ones a little weaker, to retain more overall contrast in the clip.
Default is 1.0 however, as this gives the fastest processing speed

mode [string]
This can be either "soft" or "hard", and simply decides wether to use 'softlight' or 'hardlight' in the final 'overlay()' command.
Default is "soft". You might try "hard" along with some lower values for "strength" - dunno if this may turn out useful somewhere.

maskblur [float]
This simply tells how strong the lightmask will be blurred. Stronger blurring will keep a little more sharpness in the clip, but will reduce the halo removal a little.
Default is 1.0, range is 0.0 - 1.58. Play and see for yourself.

fullframe [bool]
This is mainly a bug-workaround for a specific strangeness in the current MaskTools.
"false" is (or should be) the best setting regarding IQ. But if set to false, the YV12LUT command from MaskTools will work *no more* after calling BlindDeHalo! This is poison for e.g. iiP() ...
Set it to "true" if you need YV12LUT() after calling BlindDeHalo(), or if you're going to use it as pre-processor for iiP. (I will integrate it in the near future into iiP, anyway).


IMHO, this function can be set up to be way more effective against halos than e.g. using unfilter(-,-) -- better removal of halos, less damage to fine details and overall sharpness.

However, my testing so far was not very extensive: Your experiences, suggestions and flames are welcome.

(BTW, this script is *fast*, compared to Didée's usual standard: a 16:9 704*432 frame gets processed at ~10 fps on my XP1800.)

Have fun

- Didée

[Zarxrax]

Awesome work! I never found unfilter terribly effective against halos, so its very nice to see another option.

[DarkNite]

Thank you for including the darkamount parameter. That was a really nice touch followed by an all too true explanation.

I look forward to seeing BlindDeHalo's features included in iiP. Not that it wasn't good enough already.

[Soulhunter]

Thx! ;]

[Mug Funky]

nice stuff. i've got to eat before i fall over (yeah... i use the computer while standing ), but i will check out on some DV footage, and maybe later with r4 Akira (or did i lend that to someone?) which is very good apart from being obviously sharpened.

[digitize]

Perhaps this is just me being an idiot, not sure. I have masktools 1.4.2 (b/c i have an athlon xp), yet I get an error because it doesn't recognize Dedgemask, any idea.

[Manao]

Yep : get the latest version of the masktools :

http://www.geocities.com/manao47/Filters/

[digitize]

Yep I read the required version incorrectly (read it as 1.4.1.5, rather than 1.4.15) thanks for posting the link.

[mf]

Decimal versioning is much more logical, just look at mfToon .

[kilg0r3]

I know this a lame question ... How are the speeds compared to an unfiltered encode?

[Didée]

kilg0r3:

Bad boy, you are -- I could swear to have read this question already before, but a few posts further above /*me waves finger*/

Well, what holds you back from playing 1000 frames once with, once without BlindDeHalo, and compare for yourself?

But okay ... let me put it this way: it's roughly 3.5 times slower than Convolution3D_YV12.

Was that helpful in any way now?

You won't expect realtime performance from a script that uses

- 4 resizers
- a subtract operation
- an EdgeMask
- 3~5 different overlay()'s

won't you.

- Didée

[DeepDVD]

WOW Didée!

Nice work.

In a team with your newest iiP-version (i know, it's not projected to fix animes but it bring me back some details of my old VHS-anime that no filter could do as good before!)

and

mf's Toon-Function (that had force me to seek for a solution for his coronamask-haloing ... somebody have a try bringing him back on his scratchboard programming out his errors )

I get a very good version out of my VHS-Rip-DivX ...

I hope, the preversion of mf's correction ( not a new version ) will be gold soon... let's hope all together


so far

[Didée]

Originally posted by DeepDVD
the preversion of mf's

Hihi ... I know I should read "pre-version", but somehow I'm forced to read "perversion". Language is a funny thing.

Think of LimitedSharpen(), though. Sometimes, on some sources, I prefer it over iiP, or even use it as pre- or postproc for it (with softer iiP settings). Depends on circumstances, as usual.

- Didée

So... and this is probably a lame question...

Is it better to crank up the sharpness at the source, and capture oversharp, then use BlindDeHalo() to remove the edge enhancement, or is it better to capture a bit softer to begin with then use LimitedSharpen() afterwards?

It's something I plan to test out for myself I just wondered what the opinions of the experts were.

[Socio]

Awesome Didee!

So I understand the correct use, it would be better to run this after IIP if you are using it but before LimitedSharpen if you using it?

[Didée]

Ooops ... I found something on my HD:

function BlindDeHalo2( clip clp, float "rx", float "ry", int "strength", bool "interlaced" )
{
rx =         default( rx, 3.00 )
ry =         default( ry, 3.00 )
strength =   default( strength, 125 )
interlaced = default( interlaced, false )

function m4(float x) {return(x<16?16:int(round(x/4.0))*4)}

i  = interlaced==false ? clp : clp.separatefields()
oxi=i.width 
oyi=i.height
mm2=yv12lutxy(i.expand().blur(.8).expand(),i.inpand.blur(.8).inpand(),yexpr="x y - 2 *")
RR=string((rx+ry)/2.0)
ST=string(float(strength)/100.0)
sm1= i.bicubicresize(m4(oxi/rx),m4(oyi/ry)).bicubicresize(oxi,oyi,1.0,.0)
sm1d=yv12lutxy(i,sm1,yexpr="x y - 18 "+ST+" / "+RR+" / / 128 +")
sm1d=overlay(sm1d,sm1d,mode="hardlight")
clean=yv12lutxy(i,sm1d,yexpr="x y 128 - -").mergechroma(i)
clean1=maskedmerge(i,clean,mm2.expand.blur(1.58).expand .levels(32,1.0,255,0,208,false))
interlaced?clean1.weave():clean1 
return last
}

rx,ry [float] - the [assumed] radius at which the EE formerly was done
strength [int] - hmh, I forgot what this is for - but I remember it should be between 0 and 200
interlaced [bool] - either for interlaced sources, or for progressive ones where the EE was applied on a field-basis, not on a frame-basis

A setting of (2.0,2.0,100) will do almost no harm even to a source that doesn't have any EE.
A setting of (3.25,3.25,150) should let vanish even very strong EE that was formerly applied at a radius of 3.

Don't forget the MaskTools v1.5.5, in case anyone doesn't have them yet ...


Synthetic example:

[Socio]

Very Cool!

You would simply call it is like this correct?

BlindDeHalo2(2.0,2.0,100)

Going to test this out later today!

[Soulhunter]

Nice!


Bye

[Socio]

I tried to test the first one and got a script error;
"MaskedMerge does not have a named argument "Use MMX""

I am using Masktools 1.5.5 so I am not sure what is causing it.

Second version seems to be working, doing a test run now.

Edit: I removed the "useMMX=true" from the first script and it seems to be working, running a test now.

[Chainmax]

Soulhunter: maybe BlindDeHalo2(3.25,3.25,150) will spare me of having to use exorcist on that crappy video I sent you a while ago .