Request: HQDering with softmasking?

[Chainmax]

HQDering is a great-performing halo remover, and the added benefit of slightly sharpening the picture came in handy. Unfortunately, it has a problem: it can produce some noticeable aliasing. I was told that this might be caused by its masking, and that its usage of Deen could also lead to some issues. Would anyone be interested in having a look at the function and try to solve these issues?

[*.mp4 guy]

I might be able to do something (if the masking is the issue), but first, where can HQDering be found?

[Dreassica]

######
##
## HQDering v0.1 by mf
##
## Applies derining by using a smart smoother near edges (where ringing occurs) only.
##
## Usage: Import("HQDering-v0.1.avs")
## HQDering()
##
####

function HQDering(clip input, int "strength", int "overall", string "smoother", string "params") {

strength = Default(strength, 255) # strength, 0-255
overall = Default(overall, 0) # overall smoothing, 0-255
smoother = Default(smoother, "Deen") # filter that smooths
#params = default defined below # filter parameters - use Chr(34) for quotes

defaultnull = Default(params, "")
defaultdeen = Default(params, Chr(34)+"a3d"+Chr(34)+", 4, 15, 15, 20")

params = (smoother=="Deen") ? defaultdeen : defaultnull

try {
smoothedwparams = Eval(smoother + "(input, " + params + ")")
}
catch(err_msg) {
smoothedwoparams = Eval(smoother + "(input)")
}

smoothed = (params=="") ? smoothedwoparams : smoothedwparams

input.EdgeMask(3, 255, 255, 255, "sobel", Y=3, V=1, U=1)
normalmask = last

normalmask.Levels(0, 3.3, 90, 0, 255).Blur(1.0)
amplifiedmask = last

normalmask.Inflate().Inflate().Inflate().Levels(0, 3.3, 90, 0, 255).Blur(1.0).Inflate().Inflate().Inflate().Levels(0, 3.3, 255, 0, 255).Inflate().Inflate()
thickmask = last

YV12Layer(amplifiedmask.Invert(), thickmask, "mul", 255, chroma=false, Y=3, V=1, U=1)

Levels(60, 3.0, 140, overall, strength)

ringingmask = last

MaskedMerge(input, smoothed, ringingmask)

}

Here ya go

[Didée]

Ah, it's that time of the year again: Chainmax' annual HQDering request ... :D

Looking at it again, it seems to me that the masking is a lesser issue. It's rather the default settings of deen: a3d with rather high thresholds. Without any masking, it's a big smear everywhere there is motion, with some of the typical temporal artifacts producing median-cut-alike edges/aliasing. Where these artifacts fall in places covered by the spatial edgemask, there you get what you get.

Solution: too lazy to think about ...

Some suggestions to try:
Reduce deen's thrsholds until the artefacts disappear. (though, at the point they disappear, it's probably quite similar to not using HQDering at all...)
Use another smoother in HQDering.
let it be, use it as it is.
Use another method, not HQDering. (Camembert(), also by mf, comes along with a simplified & much speedier variant of HQDering.)
wait for better suggestions.

[Chainmax]

Dreassica: thanks for the link :). There's also a MaskTools v2 version in this thread (http://forum.doom9.org/showthread.php?t=108114).

Didée: what a sharp eye, it has been a bit over a year since those two threads :). The reason I insist so much is that, even though recent filters are more effective at removing halos, up to this point no halo remover has been able to sharpen the picture like a default HQDering() call does.
Now, no other smoother has been able to reproduce the results provided by its default Deen call.
So, with that in mind, I think maybe a combination of a more sophisticated masking process and some supersampling would reduce the aliasing to a point it would't be noticeable.

[*.mp4 guy]

I am getting an error, "no function yv12layer", but looking at the masktools 1 docs, it should be working.

[foxyshadis]

YV12Layer silently disappeared a few versions before the last masktools 1 release. It can be replaced with overlay or a lutxy:

mt_lutxy("x y " + OP + " * " + 256*(256-OP) + " + *")

I think that would do it.

[Chainmax]

This is the cleaned up MaskTools v2.x version made by foxxyshadis and posted on the thread I linked to:

######
##
## HQDering v0.1 by mf
##
## Applies derining by using a smart smoother near edges (where ringing occurs) only.
##
## Usage: Import("HQDering-v0.1.avs")
## HQDering()
##
####

function HQDering(clip input, int "strength", int "overall", clip "smoother") {

strength = Default(strength, 255) # strength, 0-255
overall = Default(overall, 0) # overall smoothing, 0-255
smoothed = defined(smoother) ? smoother : input.Deen("a3d",4,15,15,20) # filter that smooths

normalmask=input.mt_edge(thY1=3, thY2=255, mode="sobel", chroma="none")

amplifiedmask=normalmask.Levels(0, 3.3, 90, 0, 255).Blur(1.0)

thickmask=normalmask.mt_inflate().mt_inflate().mt_inflate().Levels(0, 3.3, 90, 0, 255)
\ .Blur(1.0).mt_inflate().mt_inflate().mt_inflate().Levels(0, 3.3, 255, 0, 255)
\ .mt_inflate().mt_inflate()

ringingmask=mt_lutxy(amplifiedmask.mt_invert(), thickmask, expr="x y * 255 /").Levels(60, 3.0, 140, overall, strength)

mt_merge(input, smoothed, ringingmask)

}

[*.mp4 guy]

Ok, new problem, I'm not seeing any aliasing produced by hqdering, what situations usually cause it.

[edit] I have a very similar mask that I usually use as a part of larger functions, it might fix the problem.

function halomask1(clip c, int "rad", float "ss"){

ss = default(ss, 1.5)
rad = default(rad, 7)
owidth = c.width
oheight = c.height


ssample = spline36resize(c, round((c.width*SS)/4)*4, round((c.height*SS)/4)*4)
mask1 = mt_lutxy(ssample, Xsharpen(ssample, 127, 255), "x y - abs 1 - 63 *", u=1, v=1)
mask2 = (rad >= 1) ? mask1.mt_expand(u=1, v=1) : mask1
mask2 = (rad >= 2) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 3) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 4) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 5) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 6) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 7) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 8) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 9) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 10) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 11) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 12) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 13) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 14) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 15) ? mask2.mt_expand(u=1, v=1) : mask2
mask2 = (rad >= 16) ? mask2.mt_expand(u=1, v=1) : mask2
mask3 = mt_lutXY(mask1, mask2, "y x - 63 - 1.5 *", u=1, v=1).spline36resize(owidth, oheight)

return(mask3)}

######
##
## HQDering v0.1 by mf
##
## Applies derining by using a smart smoother near edges (where ringing occurs) only.
##
## Usage: Import("HQDering-v0.1.avs")
## HQDering()
##
####

function HQDering(clip input, int "strength", int "overall", clip "smoother") {

strength = Default(strength, 255) # strength, 0-255
overall = Default(overall, 0) # overall smoothing, 0-255
smoothed = defined(smoother) ? smoother : input.Deen("a3d",4,15,15,20) # filter that smooths

normalmask=input.mt_edge(thY1=3, thY2=255, mode="sobel", chroma="none")

#amplifiedmask=normalmask.Levels(0, 3.3, 90, 0, 255).Blur(1.0)

#thickmask=normalmask.mt_inflate().mt_inflate().mt_inflate().Levels(0, 3.3, 90, 0, 255)
#\ .Blur(1.0).mt_inflate().mt_inflate().mt_inflate().Levels(0, 3.3, 255, 0, 255)
#\ .mt_inflate().mt_inflate()

#ringingmask=mt_lutxy(amplifiedmask.mt_invert(), thickmask, expr="x y * 255 /").Levels(60, 3.0, 140, overall, strength)
ringingmask = input.halomask1(ss=1.712, rad=6)#.dctlimit(use64dct=false, use32dct=false, use8dct=true)
mt_merge(input, smoothed, ringingmask)

}

[Chainmax]

Soulhunter once posted a screenshot on another forum that clearly showed the issue. Hopefully he will read this and be able to find that screenshot.

[Chainmax]

...
Please stop by my thread (http://forum.doom9.org/showthread.php?t=129774) and post a link to a Matrix sample vob containing the scene in which you discovered the issue and posted about on the mf forums.

Yeah, I remember my post @ mf's forum... But I dont remember the scene n filterchain I used... IIRC the aliasing was rather puny and I got it only here and there... I think it would be much better if someone would code a replacement for Deen in HQDering... It works out nice for Anime but not that much for Film! Adding better edge-detection/soft-masking could be another addition... But pairing HQDering with some HQ dering filter instead of a old denoiser would make it real HQ! ;]


Bye

The only material I really tested it in (some X-Men analogue captures) has long since been deleted from my HDD, so I don't have material to diagnose either. In any case, including an optional edge detection system, supersampling (using NNEDI, maybe?) and softmasking, along with a MaskTools v2.x porting process will most likely eliminate the aliasing issue.

[*.mp4 guy]

What I do to avoid aliasing on masks is run them through frfun7, it is extremely good at removing aliasing (but kills detail, which doesn't matter on masks), and usually runs faster then supersampling.

-no frfun (http://img151.imageshack.us/img151/552/nofrfun7hi9.png)
-frfun(1.01, 256, 256) (http://img223.imageshack.us/img223/962/frfun7fw5.png)

[Chainmax]

That looks great, you should definitely implement frfun7 on the masking process :).

It would be great to find a freely available implementation on the Canny edge detector in order to create the mask, but unfortunately there doesn't seem to be one :(.

[*.mp4 guy]

unfortunately the masking that is used by hqdering is too coarse to benefit much from frfun, here it is for reference (http://img178.imageshack.us/img178/5761/hqderingbc1.png).

[Chainmax]

Is improving HQDering's masking an option or would that be too complicated?