Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 10th September 2007, 00:54   #1  |  Link
Chainmax
Huh?
 
Chainmax's Avatar
 
Join Date: Sep 2003
Location: Uruguay
Posts: 3,103
Request: HQDering with softmasking?

HQDering is a great-performing halo remover, and the added benefit of slightly sharpening the picture came in handy. Unfortunately, it has a problem: it can produce some noticeable aliasing. I was told that this might be caused by its masking, and that its usage of Deen could also lead to some issues. Would anyone be interested in having a look at the function and try to solve these issues?
__________________
Read Decomb's readmes and tutorials, the IVTC tutorial and the capture guide in order to learn about combing and how to deal with it.
Chainmax is offline   Reply With Quote
Old 10th September 2007, 05:03   #2  |  Link
*.mp4 guy
Registered User
 
*.mp4 guy's Avatar
 
Join Date: Feb 2004
Posts: 1,348
I might be able to do something (if the masking is the issue), but first, where can HQDering be found?
*.mp4 guy is offline   Reply With Quote
Old 10th September 2007, 19:49   #3  |  Link
Dreassica
Registered User
 
Join Date: May 2002
Posts: 384
Quote:
######
##
## HQDering v0.1 by mf
##
## Applies derining by using a smart smoother near edges (where ringing occurs) only.
##
## Usage: Import("HQDering-v0.1.avs")
## HQDering()
##
####

function HQDering(clip input, int "strength", int "overall", string "smoother", string "params") {

strength = Default(strength, 255) # strength, 0-255
overall = Default(overall, 0) # overall smoothing, 0-255
smoother = Default(smoother, "Deen") # filter that smooths
#params = default defined below # filter parameters - use Chr(34) for quotes

defaultnull = Default(params, "")
defaultdeen = Default(params, Chr(34)+"a3d"+Chr(34)+", 4, 15, 15, 20")

params = (smoother=="Deen") ? defaultdeen : defaultnull

try {
smoothedwparams = Eval(smoother + "(input, " + params + ")")
}
catch(err_msg) {
smoothedwoparams = Eval(smoother + "(input)")
}

smoothed = (params=="") ? smoothedwoparams : smoothedwparams

input.EdgeMask(3, 255, 255, 255, "sobel", Y=3, V=1, U=1)
normalmask = last

normalmask.Levels(0, 3.3, 90, 0, 255).Blur(1.0)
amplifiedmask = last

normalmask.Inflate().Inflate().Inflate().Levels(0, 3.3, 90, 0, 255).Blur(1.0).Inflate().Inflate().Inflate().Levels(0, 3.3, 255, 0, 255).Inflate().Inflate()
thickmask = last

YV12Layer(amplifiedmask.Invert(), thickmask, "mul", 255, chroma=false, Y=3, V=1, U=1)

Levels(60, 3.0, 140, overall, strength)

ringingmask = last

MaskedMerge(input, smoothed, ringingmask)

}
Here ya go
Dreassica is offline   Reply With Quote
Old 10th September 2007, 22:16   #4  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,389
Ah, it's that time of the year again: Chainmax' annual HQDering request ...

Looking at it again, it seems to me that the masking is a lesser issue. It's rather the default settings of deen: a3d with rather high thresholds. Without any masking, it's a big smear everywhere there is motion, with some of the typical temporal artifacts producing median-cut-alike edges/aliasing. Where these artifacts fall in places covered by the spatial edgemask, there you get what you get.

Solution: too lazy to think about ...

Some suggestions to try:
  • Reduce deen's thrsholds until the artefacts disappear. (though, at the point they disappear, it's probably quite similar to not using HQDering at all...)
  • Use another smoother in HQDering.
  • let it be, use it as it is.
  • Use another method, not HQDering. (Camembert(), also by mf, comes along with a simplified & much speedier variant of HQDering.)
  • wait for better suggestions.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 10th September 2007, 23:02   #5  |  Link
Chainmax
Huh?
 
Chainmax's Avatar
 
Join Date: Sep 2003
Location: Uruguay
Posts: 3,103
Dreassica: thanks for the link . There's also a MaskTools v2 version in this thread.

Didée: what a sharp eye, it has been a bit over a year since those two threads . The reason I insist so much is that, even though recent filters are more effective at removing halos, up to this point no halo remover has been able to sharpen the picture like a default HQDering() call does.
Now, no other smoother has been able to reproduce the results provided by its default Deen call.
So, with that in mind, I think maybe a combination of a more sophisticated masking process and some supersampling would reduce the aliasing to a point it would't be noticeable.
__________________
Read Decomb's readmes and tutorials, the IVTC tutorial and the capture guide in order to learn about combing and how to deal with it.
Chainmax is offline   Reply With Quote
Old 11th September 2007, 05:40   #6  |  Link
*.mp4 guy
Registered User
 
*.mp4 guy's Avatar
 
Join Date: Feb 2004
Posts: 1,348
I am getting an error, "no function yv12layer", but looking at the masktools 1 docs, it should be working.
*.mp4 guy is offline   Reply With Quote
Old 11th September 2007, 07:04   #7  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,556
YV12Layer silently disappeared a few versions before the last masktools 1 release. It can be replaced with overlay or a lutxy:

mt_lutxy("x y " + OP + " * " + 256*(256-OP) + " + *")

I think that would do it.
foxyshadis is offline   Reply With Quote
Old 11th September 2007, 15:02   #8  |  Link
Chainmax
Huh?
 
Chainmax's Avatar
 
Join Date: Sep 2003
Location: Uruguay
Posts: 3,103
This is the cleaned up MaskTools v2.x version made by foxxyshadis and posted on the thread I linked to:

Code:
######
##
## HQDering v0.1 by mf
##
## Applies derining by using a smart smoother near edges (where ringing occurs) only.
##
## Usage: Import("HQDering-v0.1.avs")
##        HQDering()
##
####

function HQDering(clip input, int "strength", int "overall", clip "smoother") {

 strength = Default(strength, 255)	# strength, 0-255
 overall  = Default(overall, 0)		# overall smoothing, 0-255
 smoothed = defined(smoother) ? smoother : input.Deen("a3d",4,15,15,20)	# filter that smooths

 normalmask=input.mt_edge(thY1=3, thY2=255, mode="sobel", chroma="none")

 amplifiedmask=normalmask.Levels(0, 3.3, 90, 0, 255).Blur(1.0)

 thickmask=normalmask.mt_inflate().mt_inflate().mt_inflate().Levels(0, 3.3, 90, 0, 255)
  \ .Blur(1.0).mt_inflate().mt_inflate().mt_inflate().Levels(0, 3.3, 255, 0, 255)
  \ .mt_inflate().mt_inflate()

 ringingmask=mt_lutxy(amplifiedmask.mt_invert(), thickmask, expr="x y * 255 /").Levels(60, 3.0, 140, overall, strength)

 mt_merge(input, smoothed, ringingmask)

}
__________________
Read Decomb's readmes and tutorials, the IVTC tutorial and the capture guide in order to learn about combing and how to deal with it.
Chainmax is offline   Reply With Quote
Old 11th September 2007, 19:12   #9  |  Link
*.mp4 guy
Registered User
 
*.mp4 guy's Avatar
 
Join Date: Feb 2004
Posts: 1,348
Ok, new problem, I'm not seeing any aliasing produced by hqdering, what situations usually cause it.

[edit] I have a very similar mask that I usually use as a part of larger functions, it might fix the problem.

Code:
function halomask1(clip c, int "rad", float "ss"){
	
	ss = default(ss, 1.5)
	rad = default(rad, 7)
	owidth = c.width
	oheight = c.height
	
	
	ssample = spline36resize(c, round((c.width*SS)/4)*4, round((c.height*SS)/4)*4)
	mask1 = mt_lutxy(ssample, Xsharpen(ssample, 127, 255), "x y - abs 1 - 63 *", u=1, v=1)
	mask2 = (rad >= 1) ? mask1.mt_expand(u=1, v=1) : mask1
	mask2 = (rad >= 2) ? mask2.mt_expand(u=1, v=1) : mask2
	mask2 = (rad >= 3) ? mask2.mt_expand(u=1, v=1) : mask2
	mask2 = (rad >= 4) ? mask2.mt_expand(u=1, v=1) : mask2
	mask2 = (rad >= 5) ? mask2.mt_expand(u=1, v=1) : mask2
	mask2 = (rad >= 6) ? mask2.mt_expand(u=1, v=1) : mask2
	mask2 = (rad >= 7) ? mask2.mt_expand(u=1, v=1) : mask2
	mask2 = (rad >= 8) ? mask2.mt_expand(u=1, v=1) : mask2
	mask2 = (rad >= 9) ? mask2.mt_expand(u=1, v=1) : mask2
	mask2 = (rad >= 10) ? mask2.mt_expand(u=1, v=1) : mask2
	mask2 = (rad >= 11) ? mask2.mt_expand(u=1, v=1) : mask2
	mask2 = (rad >= 12) ? mask2.mt_expand(u=1, v=1) : mask2
	mask2 = (rad >= 13) ? mask2.mt_expand(u=1, v=1) : mask2
	mask2 = (rad >= 14) ? mask2.mt_expand(u=1, v=1) : mask2
	mask2 = (rad >= 15) ? mask2.mt_expand(u=1, v=1) : mask2
	mask2 = (rad >= 16) ? mask2.mt_expand(u=1, v=1) : mask2
	mask3 = mt_lutXY(mask1, mask2, "y x - 63 - 1.5 *", u=1, v=1).spline36resize(owidth, oheight)
	
return(mask3)}
	
######
##
## HQDering v0.1 by mf
##
## Applies derining by using a smart smoother near edges (where ringing occurs) only.
##
## Usage: Import("HQDering-v0.1.avs")
##        HQDering()
##
####

function HQDering(clip input, int "strength", int "overall", clip "smoother") {

 strength = Default(strength, 255)	# strength, 0-255
 overall  = Default(overall, 0)		# overall smoothing, 0-255
 smoothed = defined(smoother) ? smoother : input.Deen("a3d",4,15,15,20)	# filter that smooths

 normalmask=input.mt_edge(thY1=3, thY2=255, mode="sobel", chroma="none")

 #amplifiedmask=normalmask.Levels(0, 3.3, 90, 0, 255).Blur(1.0)

 #thickmask=normalmask.mt_inflate().mt_inflate().mt_inflate().Levels(0, 3.3, 90, 0, 255)
  #\ .Blur(1.0).mt_inflate().mt_inflate().mt_inflate().Levels(0, 3.3, 255, 0, 255)
  #\ .mt_inflate().mt_inflate()

 #ringingmask=mt_lutxy(amplifiedmask.mt_invert(), thickmask, expr="x y * 255 /").Levels(60, 3.0, 140, overall, strength)
 ringingmask = input.halomask1(ss=1.712, rad=6)#.dctlimit(use64dct=false, use32dct=false, use8dct=true)
 mt_merge(input, smoothed, ringingmask)

}

Last edited by *.mp4 guy; 19th September 2007 at 21:15. Reason: changed ss and rad params
*.mp4 guy is offline   Reply With Quote
Old 19th September 2007, 03:45   #10  |  Link
Chainmax
Huh?
 
Chainmax's Avatar
 
Join Date: Sep 2003
Location: Uruguay
Posts: 3,103
Soulhunter once posted a screenshot on another forum that clearly showed the issue. Hopefully he will read this and be able to find that screenshot.
__________________
Read Decomb's readmes and tutorials, the IVTC tutorial and the capture guide in order to learn about combing and how to deal with it.
Chainmax is offline   Reply With Quote
Old 17th October 2007, 10:08   #11  |  Link
Chainmax
Huh?
 
Chainmax's Avatar
 
Join Date: Sep 2003
Location: Uruguay
Posts: 3,103
Quote:
Originally Posted by Soulhunter
Quote:
Originally Posted by Chainmax
...
Please stop by my thread and post a link to a Matrix sample vob containing the scene in which you discovered the issue and posted about on the mf forums.
Yeah, I remember my post @ mf's forum... But I dont remember the scene n filterchain I used... IIRC the aliasing was rather puny and I got it only here and there... I think it would be much better if someone would code a replacement for Deen in HQDering... It works out nice for Anime but not that much for Film! Adding better edge-detection/soft-masking could be another addition... But pairing HQDering with some HQ dering filter instead of a old denoiser would make it real HQ! ;]


Bye
The only material I really tested it in (some X-Men analogue captures) has long since been deleted from my HDD, so I don't have material to diagnose either. In any case, including an optional edge detection system, supersampling (using NNEDI, maybe?) and softmasking, along with a MaskTools v2.x porting process will most likely eliminate the aliasing issue.
__________________
Read Decomb's readmes and tutorials, the IVTC tutorial and the capture guide in order to learn about combing and how to deal with it.
Chainmax is offline   Reply With Quote
Old 17th October 2007, 21:45   #12  |  Link
*.mp4 guy
Registered User
 
*.mp4 guy's Avatar
 
Join Date: Feb 2004
Posts: 1,348
What I do to avoid aliasing on masks is run them through frfun7, it is extremely good at removing aliasing (but kills detail, which doesn't matter on masks), and usually runs faster then supersampling.

-no frfun
-frfun(1.01, 256, 256)
*.mp4 guy is offline   Reply With Quote
Old 17th October 2007, 22:22   #13  |  Link
Chainmax
Huh?
 
Chainmax's Avatar
 
Join Date: Sep 2003
Location: Uruguay
Posts: 3,103
That looks great, you should definitely implement frfun7 on the masking process .

It would be great to find a freely available implementation on the Canny edge detector in order to create the mask, but unfortunately there doesn't seem to be one .
__________________
Read Decomb's readmes and tutorials, the IVTC tutorial and the capture guide in order to learn about combing and how to deal with it.
Chainmax is offline   Reply With Quote
Old 17th October 2007, 22:58   #14  |  Link
*.mp4 guy
Registered User
 
*.mp4 guy's Avatar
 
Join Date: Feb 2004
Posts: 1,348
unfortunately the masking that is used by hqdering is too coarse to benefit much from frfun, here it is for reference.
*.mp4 guy is offline   Reply With Quote
Old 19th October 2007, 09:59   #15  |  Link
Chainmax
Huh?
 
Chainmax's Avatar
 
Join Date: Sep 2003
Location: Uruguay
Posts: 3,103
Is improving HQDering's masking an option or would that be too complicated?
__________________
Read Decomb's readmes and tutorials, the IVTC tutorial and the capture guide in order to learn about combing and how to deal with it.
Chainmax is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 10:44.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.