Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
10th September 2007, 00:54 | #1 | Link |
Huh?
Join Date: Sep 2003
Location: Uruguay
Posts: 3,103
|
Request: HQDering with softmasking?
HQDering is a great-performing halo remover, and the added benefit of slightly sharpening the picture came in handy. Unfortunately, it has a problem: it can produce some noticeable aliasing. I was told that this might be caused by its masking, and that its usage of Deen could also lead to some issues. Would anyone be interested in having a look at the function and try to solve these issues?
__________________
Read Decomb's readmes and tutorials, the IVTC tutorial and the capture guide in order to learn about combing and how to deal with it. |
10th September 2007, 19:49 | #3 | Link | |
Registered User
Join Date: May 2002
Posts: 384
|
Quote:
|
|
10th September 2007, 22:16 | #4 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,389
|
Ah, it's that time of the year again: Chainmax' annual HQDering request ...
Looking at it again, it seems to me that the masking is a lesser issue. It's rather the default settings of deen: a3d with rather high thresholds. Without any masking, it's a big smear everywhere there is motion, with some of the typical temporal artifacts producing median-cut-alike edges/aliasing. Where these artifacts fall in places covered by the spatial edgemask, there you get what you get. Solution: too lazy to think about ... Some suggestions to try:
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
10th September 2007, 23:02 | #5 | Link |
Huh?
Join Date: Sep 2003
Location: Uruguay
Posts: 3,103
|
Dreassica: thanks for the link . There's also a MaskTools v2 version in this thread.
Didée: what a sharp eye, it has been a bit over a year since those two threads . The reason I insist so much is that, even though recent filters are more effective at removing halos, up to this point no halo remover has been able to sharpen the picture like a default HQDering() call does. Now, no other smoother has been able to reproduce the results provided by its default Deen call. So, with that in mind, I think maybe a combination of a more sophisticated masking process and some supersampling would reduce the aliasing to a point it would't be noticeable.
__________________
Read Decomb's readmes and tutorials, the IVTC tutorial and the capture guide in order to learn about combing and how to deal with it. |
11th September 2007, 07:04 | #7 | Link |
ангел смерти
Join Date: Nov 2004
Location: Lost
Posts: 9,556
|
YV12Layer silently disappeared a few versions before the last masktools 1 release. It can be replaced with overlay or a lutxy:
mt_lutxy("x y " + OP + " * " + 256*(256-OP) + " + *") I think that would do it. |
11th September 2007, 15:02 | #8 | Link |
Huh?
Join Date: Sep 2003
Location: Uruguay
Posts: 3,103
|
This is the cleaned up MaskTools v2.x version made by foxxyshadis and posted on the thread I linked to:
Code:
###### ## ## HQDering v0.1 by mf ## ## Applies derining by using a smart smoother near edges (where ringing occurs) only. ## ## Usage: Import("HQDering-v0.1.avs") ## HQDering() ## #### function HQDering(clip input, int "strength", int "overall", clip "smoother") { strength = Default(strength, 255) # strength, 0-255 overall = Default(overall, 0) # overall smoothing, 0-255 smoothed = defined(smoother) ? smoother : input.Deen("a3d",4,15,15,20) # filter that smooths normalmask=input.mt_edge(thY1=3, thY2=255, mode="sobel", chroma="none") amplifiedmask=normalmask.Levels(0, 3.3, 90, 0, 255).Blur(1.0) thickmask=normalmask.mt_inflate().mt_inflate().mt_inflate().Levels(0, 3.3, 90, 0, 255) \ .Blur(1.0).mt_inflate().mt_inflate().mt_inflate().Levels(0, 3.3, 255, 0, 255) \ .mt_inflate().mt_inflate() ringingmask=mt_lutxy(amplifiedmask.mt_invert(), thickmask, expr="x y * 255 /").Levels(60, 3.0, 140, overall, strength) mt_merge(input, smoothed, ringingmask) }
__________________
Read Decomb's readmes and tutorials, the IVTC tutorial and the capture guide in order to learn about combing and how to deal with it. |
11th September 2007, 19:12 | #9 | Link |
Registered User
Join Date: Feb 2004
Posts: 1,348
|
Ok, new problem, I'm not seeing any aliasing produced by hqdering, what situations usually cause it.
[edit] I have a very similar mask that I usually use as a part of larger functions, it might fix the problem. Code:
function halomask1(clip c, int "rad", float "ss"){ ss = default(ss, 1.5) rad = default(rad, 7) owidth = c.width oheight = c.height ssample = spline36resize(c, round((c.width*SS)/4)*4, round((c.height*SS)/4)*4) mask1 = mt_lutxy(ssample, Xsharpen(ssample, 127, 255), "x y - abs 1 - 63 *", u=1, v=1) mask2 = (rad >= 1) ? mask1.mt_expand(u=1, v=1) : mask1 mask2 = (rad >= 2) ? mask2.mt_expand(u=1, v=1) : mask2 mask2 = (rad >= 3) ? mask2.mt_expand(u=1, v=1) : mask2 mask2 = (rad >= 4) ? mask2.mt_expand(u=1, v=1) : mask2 mask2 = (rad >= 5) ? mask2.mt_expand(u=1, v=1) : mask2 mask2 = (rad >= 6) ? mask2.mt_expand(u=1, v=1) : mask2 mask2 = (rad >= 7) ? mask2.mt_expand(u=1, v=1) : mask2 mask2 = (rad >= 8) ? mask2.mt_expand(u=1, v=1) : mask2 mask2 = (rad >= 9) ? mask2.mt_expand(u=1, v=1) : mask2 mask2 = (rad >= 10) ? mask2.mt_expand(u=1, v=1) : mask2 mask2 = (rad >= 11) ? mask2.mt_expand(u=1, v=1) : mask2 mask2 = (rad >= 12) ? mask2.mt_expand(u=1, v=1) : mask2 mask2 = (rad >= 13) ? mask2.mt_expand(u=1, v=1) : mask2 mask2 = (rad >= 14) ? mask2.mt_expand(u=1, v=1) : mask2 mask2 = (rad >= 15) ? mask2.mt_expand(u=1, v=1) : mask2 mask2 = (rad >= 16) ? mask2.mt_expand(u=1, v=1) : mask2 mask3 = mt_lutXY(mask1, mask2, "y x - 63 - 1.5 *", u=1, v=1).spline36resize(owidth, oheight) return(mask3)} ###### ## ## HQDering v0.1 by mf ## ## Applies derining by using a smart smoother near edges (where ringing occurs) only. ## ## Usage: Import("HQDering-v0.1.avs") ## HQDering() ## #### function HQDering(clip input, int "strength", int "overall", clip "smoother") { strength = Default(strength, 255) # strength, 0-255 overall = Default(overall, 0) # overall smoothing, 0-255 smoothed = defined(smoother) ? smoother : input.Deen("a3d",4,15,15,20) # filter that smooths normalmask=input.mt_edge(thY1=3, thY2=255, mode="sobel", chroma="none") #amplifiedmask=normalmask.Levels(0, 3.3, 90, 0, 255).Blur(1.0) #thickmask=normalmask.mt_inflate().mt_inflate().mt_inflate().Levels(0, 3.3, 90, 0, 255) #\ .Blur(1.0).mt_inflate().mt_inflate().mt_inflate().Levels(0, 3.3, 255, 0, 255) #\ .mt_inflate().mt_inflate() #ringingmask=mt_lutxy(amplifiedmask.mt_invert(), thickmask, expr="x y * 255 /").Levels(60, 3.0, 140, overall, strength) ringingmask = input.halomask1(ss=1.712, rad=6)#.dctlimit(use64dct=false, use32dct=false, use8dct=true) mt_merge(input, smoothed, ringingmask) } Last edited by *.mp4 guy; 19th September 2007 at 21:15. Reason: changed ss and rad params |
19th September 2007, 03:45 | #10 | Link |
Huh?
Join Date: Sep 2003
Location: Uruguay
Posts: 3,103
|
Soulhunter once posted a screenshot on another forum that clearly showed the issue. Hopefully he will read this and be able to find that screenshot.
__________________
Read Decomb's readmes and tutorials, the IVTC tutorial and the capture guide in order to learn about combing and how to deal with it. |
17th October 2007, 10:08 | #11 | Link | ||
Huh?
Join Date: Sep 2003
Location: Uruguay
Posts: 3,103
|
Quote:
__________________
Read Decomb's readmes and tutorials, the IVTC tutorial and the capture guide in order to learn about combing and how to deal with it. |
||
17th October 2007, 21:45 | #12 | Link |
Registered User
Join Date: Feb 2004
Posts: 1,348
|
What I do to avoid aliasing on masks is run them through frfun7, it is extremely good at removing aliasing (but kills detail, which doesn't matter on masks), and usually runs faster then supersampling.
-no frfun -frfun(1.01, 256, 256) |
17th October 2007, 22:22 | #13 | Link |
Huh?
Join Date: Sep 2003
Location: Uruguay
Posts: 3,103
|
That looks great, you should definitely implement frfun7 on the masking process .
It would be great to find a freely available implementation on the Canny edge detector in order to create the mask, but unfortunately there doesn't seem to be one .
__________________
Read Decomb's readmes and tutorials, the IVTC tutorial and the capture guide in order to learn about combing and how to deal with it. |
17th October 2007, 22:58 | #14 | Link |
Registered User
Join Date: Feb 2004
Posts: 1,348
|
unfortunately the masking that is used by hqdering is too coarse to benefit much from frfun, here it is for reference.
|
19th October 2007, 09:59 | #15 | Link |
Huh?
Join Date: Sep 2003
Location: Uruguay
Posts: 3,103
|
Is improving HQDering's masking an option or would that be too complicated?
__________________
Read Decomb's readmes and tutorials, the IVTC tutorial and the capture guide in order to learn about combing and how to deal with it. |
Thread Tools | Search this Thread |
Display Modes | |
|
|