Log in

View Full Version : HQDegrain


Alek93j
19th June 2011, 21:53
Hi all, I made this little degrain filter. I hope some of you will find it useful. If you have any kind of feedback, please do let me know what you think about it.

######################################################################
#
# ~HQDegrain v0.9.2 by Alek93j~ 29/06/2011
#
# Required MVtools2.dll http://avisynth.org.ru/mvtools/mvtools2.html
# ttempsmooth.dll http://avisynth.org.ru/docs/english/...tempSmooth.htm
# mt_masktools-25.dll http://manao4.free.fr/masktools-v2.0a48.zip
# blockbuster.dll http://kvcd.net/sansgrip/avisynth/
# removegrain.dll http://avisynth.org/mediawiki/Removegrain
#
# ~Syntax~
##################################################################################
#
# # It doesn't work perfectely with sudden luma changes such as fades, so it
# # is best not to filter those parts.
#
# -Example of a Backup of a NTSC DvD-
#
# MPEG2Source("C:\blabla.d2v")
# ivtc() (your prefer ivtc ectr)
# A=hqdegrain() (Clip normally processed)
# B=ttempsmoothf(maxr=7,strength=5,lthresh=3,cthresh=4) (or your prefer denoise for clean fade)
# B.Trim(0,215)+A.Trim(216,31464)
#
###########################################################################

Function HQDegrain(clip clp,int"Degrain",int"limit",string"preset",bool"HQ",\
int"maxr",int"strength",int"lthresh",int"cthresh",\
clip"PreNoise",int"BkSize",bool"noise",bool"lmask",\
int"Blk",int"PEL",int"DCT",int"Overlap",\
int "thSAD", int "thSADC",int "thSCD1", int "thSCD2")
{

#Presets available "Fast" - "Slow"

Preset=Default(Preset,"Slow")
pset=(Preset=="Fast") ? 0 : 1

#MDegrain Strength

Degrain = Default( Degrain , Select(pset,1,2)) # Means how many frame should be analyzed for a better result, 3 doesn't help much.
limit = Default( limit , 255 ) # limit of mvdegrain. (255 mean no limits)

#ttempsmoothf Strength

HQ = Default( HQ , Select(pset,false,true)) # Reduce artefact...
maxr = Default( maxr , Select(pset,3,5))
strength= Default( strength, Select(pset,3,4))
lthresh = Default( lthresh , Select(pset,2,3))
cthresh = Default( cthresh , Select(pset,3,4))

#Noise Filter

Noise = Default( Noise , true )
BkSize = Default( BkSize , 8 )
lmask = Default( lmask , Select(pset,false,true)) #if lmask is true, noise will not be added in the parts of the image with high luma values, because generally there is no need to filter them.

#Motion Vector Analyse

Blk = Default( Blk , 8 ) # Use 16/for 1080p - 8/for 720p - 4/for 480p to avoid unnecessary slowdowns.
PEL = Default( PEL , 2 )
DCT = Default( DCT , 0 )
OPS = Blk/2
Overlap = Default( Overlap , Select(pset,0,OPS))

#Threshold Setting

thSAD = Default( thSAD , 180 ) # Lower values will reduce artefacts for bad motion vectors, but they will also have a lighter denoising effect. For animated sources, 120 seems to be a good setting to start with.
thSADC = Default( thSADC , thSAD )
thSCD1 = Default( thSCD1 , 400 )
thSCD2 = Default( thSCD2 , 102 )

#Processing

input = clp

mask = clp.invert().levels(82,1,155,0,255,coring=false).Removegrain(12)

nse = defined(PreNoise) ? PreNoise
\: (Noise==true) ? clp.blockbuster(method="noise",Block_size=BkSize) : clp

clp = (Noise==false) ? clp : (lmask==true) ? mt_merge(clp,nse,mask) : nse

sup = clp.Msuper(hpad=Blk,vpad=Blk,PEL=PEL,levels=0)

BV1 = MAnalyse(sup,isb=true ,truemotion=true,delta=1,blksize=Blk,blksizeV=Blk,DCT=DCT,overlap=Overlap)
FV1 = MAnalyse(sup,isb=false,truemotion=true,delta=1,blksize=Blk,blksizeV=Blk,DCT=DCT,overlap=Overlap)

BV2 = (Degrain>=2) ? MAnalyse(sup,isb=true ,truemotion=true,delta=2,blksize=Blk,blksizeV=Blk,DCT=DCT,overlap=Overlap) : Blackness ()
FV2 = (Degrain>=2) ? MAnalyse(sup,isb=false,truemotion=true,delta=2,blksize=Blk,blksizeV=Blk,DCT=DCT,overlap=Overlap) : Blackness ()

BV3 = (Degrain==3) ? MAnalyse(sup,isb=true ,truemotion=true,delta=3,blksize=Blk,blksizeV=Blk,DCT=DCT,overlap=Overlap) : Blackness ()
FV3 = (Degrain==3) ? MAnalyse(sup,isb=false,truemotion=true,delta=3,blksize=Blk,blksizeV=Blk,DCT=DCT,overlap=Overlap) : Blackness ()

clp = (degrain==1) ? clp.MDegrain1(sup,BV1,FV1, thSAD=thSAD,thSADC=thSADC,thSCD1=thSCD1,thSCD2=thSCD2,limit=limit):
\ (degrain==2) ? clp.MDegrain2(sup,BV1,FV1,BV2,FV2, thSAD=thSAD,thSADC=thSADC,thSCD1=thSCD1,thSCD2=thSCD2,limit=limit):
\ clp.MDegrain3(sup,BV1,FV1,BV2,FV2,BV3,FV3,thSAD=thSAD,thSADC=thSADC,thSCD1=thSCD1,thSCD2=thSCD2,limit=limit)

Diff1 = mt_makediff(input,clp,U=3,V=3)

clp2 = clp.ttempsmoothf(maxr=maxr,strength=strength,lthresh=lthresh,cthresh=cthresh)

Diff2 = mt_makediff(input,clp2,U=3,V=3)

lutxy = mt_lutxy(Diff1,Diff2,"x 128 - abs y 128 - abs < x y ?")

output = (HQ==true) ? Input.mt_makediff(lutxy,U=3,V=3) : clp2

output }


Have a nice day.

TheProfileth
20th June 2011, 23:38
Will give this a look

Didée
21st June 2011, 11:13
Welcome on D9, Alek93j. :)


# Required MVtools2.dll
# ttempsmooth.dll
# mt_masktools-25.dll
You forgot one dependency: BlockBuster.dll :)



sup = clp.Msuper(hpad=Blk,vpad=Blk,PEL=PEL,levels=0)
sup2= clp.Msuper(hpad=Blk,vpad=Blk,PEL=PEL,levels=1)


Since no additional pre-filtering is done, there is no need to create a 2nd super clip with levels=1. You can simply use the one "sup" clip for everything.


Also, it seems there is some confusion about chroma processing:

Both the MDegrain's and ttempsmoothF *do* process chroma. But when "HQ=true" (default), then all chroma processing is thrown away because of the mt_makediff's and the mt_lutxy's.

You should decide whether chroma shall be processed or not.
If not, then you can get better speed by setting "plane=0" in MDegrainX, and by setting "cthresh=1" in ttempsmooth.
If chroma should be processed, then you need to add ",U=3,V=3" in all mt_makediff and all mt_lutxy.


Seems to be a nice filter for baby grain. For real men's grain, it's a bit on the weak side. ;)

Funny trivia: It's a "degrain" filter, and the very-first internal operation is to add more grain. (Blockbuster) :)


Judging visually, the results seem pretty okay.

Gerry62
21st June 2011, 13:10
Am looking for a Nice script to clean some 16mm Kine's!


LoadPlugin("TTempSmooth.dll")
Import("HQDegrain.avsi")
AVISource("D:\Raw.avi")
converttoYV12()
FieldDeinterlace(full=true, blend=true)
HQDegrain()

Ive got bloody 'loadplugin' errors.
Will let you know If i get further, as Im keen for a script that can clean 16mm Kine's, even if its only a first pass before something else. Sick of pixilated movement!

Didée
21st June 2011, 13:19
What ttempsmoothf errors? The script is technically correct, there should not occur any errors as long as you respect the valid parameter ranges for ttempsmooth's parameters: maxr=[1...7], strength=[1...8], lthresh/cthresh=[1...256]

Gerry62
21st June 2011, 13:24
What ttempsmoothf errors? The script is technically correct, there should not occur any errors as long as you respect the valid parameter ranges for ttempsmooth's parameters: maxr=[1...7], strength=[1...8], lthresh/cthresh=[1...256]

Yes the script is fine, Very fine!
But the user (IE Me) is a little green, but as keen as mustard to get it to work!
Spent all day on Killer() and failed!

Must be where Im placing the ttempsmooth.dll.
And over the last few weeks ive been reading some of your posts Didee, If I could create a , 'I Am not worthy' script, it would be now, you are a god of the synth~! and please disregard my amuter efforts in getting things to work.

Your a champ alek93j, love your efforts, will love it more when Ive seen it work!

Didée
21st June 2011, 13:35
You still didn't state the exact error. If it is "There is no function named 'ttempsmoothf' ", then you need to LoadPlugin() the ttempsmooth.dll ...

Gerry62
21st June 2011, 14:34
Wow, how stupid am I!
I had the x64 DLL in my plugins directory, hence the error's, Have now put the correct 32bit dll and it's working fine, the defult setup is not as 'strong' as id like the 16mm kino's are 'grain city'. which makes any restoration very difficult as when you clean the grains all movement looks very 'gritty' and pixelated to hell. Will spend some time mucking about with the settings and see how this new function works with my dire material!

Alek93j
21st June 2011, 21:46
Welcome on D9, Alek93j. :)
thanks for welcome


You forgot one dependency: BlockBuster.dll :)

thanks for reminding me, I had forgotten about it.



sup = clp.Msuper(hpad=Blk,vpad=Blk,PEL=PEL,levels=0)
sup2= clp.Msuper(hpad=Blk,vpad=Blk,PEL=PEL,levels=1)


Since no additional pre-filtering is done, there is no need to create a 2nd super clip with levels=1. You can simply use the one "sup"

clip for everything.



I put 2 sup clips because I read in the mvtools documentation that only manalyse requires access to all levels, whereas for the other functions 1 level is enough, thus having the other functions work on just 1 level would give a speedup without hinder quality. Or did I just not understand the documentation (I admittedly am not too sure about this)?


Also, it seems there is some confusion about chroma processing:

Both the MDegrain's and ttempsmoothF *do* process chroma. But when "HQ=true" (default), then all chroma processing is thrown away because

of the mt_makediff's and the mt_lutxy's.

You should decide whether chroma shall be processed or not.
If not, then you can get better speed by setting "plane=0" in MDegrainX, and by setting "cthresh=1" in ttempsmooth.
If chroma should be processed, then you need to add ",U=3,V=3" in all mt_makediff and all mt_lutxy.


I'm not very good with the masktool, I added the HQ function for repair the ttempsmooth artifacts, I mean these:

original
http://img541.imageshack.us/img541/5210/58221536.png
hq false
http://img709.imageshack.us/img709/157/91395202.png
hq true
http://img233.imageshack.us/img233/2233/19052853.png

I admit that I copied the masktool part from your script "temporal degrain", beacuse I got the exact results which I wanted, and I was stupid to not specify it from the beginning, and the main reason I posted the filter is to get help for that part, so, can you help me to get those results in the right way, or, if is it possible, also better? I would be grateful to you.


Funny trivia: It's a "degrain" filter, and the very-first internal operation is to add more grain. (Blockbuster) :)
"This method adds normally distributed -- also known as Gaussian -- noise to the clip. Testing has shown that Gaussian noise is far more suitable for this filter's purposes than uniformly distributed noise."
Blockbuster adds noise, doesn't it? I created this function for "restore" the grain affected by macroblocking etc. caused by a bad compression, they aren't really pleasant during the reproduction. This function also removes a little part of grain extra (for example, in the bright parts of the image).My idea was to fill these empty parts with blockbuster and then to work with MDegrain: it would have switched a part of noise for the grain e it would have removed it, but not completely. And, to finish, make everything homogeneous with ttempsmooth, which is my favourite smoother, as you can see in the screens that I posted first.


Your a champ alek93j, love your efforts, will love it more when Ive seen it work!
thanks, I hope to learn more and to be more helpful

tormento
23rd June 2011, 09:18
Hi all, I made this little degrain filter. I hope some of you will find it useful. If you have any kind of feedback, please do let me know what you think about it.
Thanks. Do you plan to release a x64 aware version? Some filters don't even exist in x64 world ;)

Nephilis
23rd June 2011, 09:48
I put 2 sup clips because I read in the mvtools documentation that only manalyse requires access to all levels, whereas for the other functions 1 level is enough, thus having the other functions work on just 1 level would give a speedup without hinder quality. Or did I just not understand the documentation (I admittedly am not too sure about this)?


Yes only manalyse needs all levels. But when you prepare 2 super clip, the process will need more memory and spend more effort. Also as Didée said you should decide if the filter process the Chroma or not..

Thanks. Do you plan to release a x64 aware version? Some filters don't even exist in x64 world ;)

Everybody is boring for your x64 requests. Try to share something useful posts here not only x64 aware filters/plugins requests..

tormento
23rd June 2011, 18:04
Everybody is boring for your x64 requests. Try to share something useful posts here not only x64 aware filters/plugins requests..
Your 79 posts and 2009 subscription are enough to judge my behaviour and address your brain to such a democratic answer?

Welcome, my new forum moderator.

Nephilis
23rd June 2011, 19:35
Your 79 posts and 2009 subscription are enough to judge my behaviour and address your brain to such a democratic answer?
Welcome, my new forum moderator.

Only in last one year (between 23.06.2010-23.06.2011)

http://forum.doom9.org/showthread.php?p=1509718#post1509718
http://forum.doom9.org/showthread.php?p=1509717#post1509717
http://forum.doom9.org/showthread.php?p=1505148#post1505148
http://forum.doom9.org/showthread.php?p=1494770#post1494770
http://forum.doom9.org/showthread.php?p=1473026#post1473026
http://forum.doom9.org/showthread.php?p=1472166#post1472166
http://forum.doom9.org/showthread.php?p=1472163#post1472163
http://forum.doom9.org/showthread.php?p=1460161#post1460161
http://forum.doom9.org/showthread.php?p=1459685#post1459685
http://forum.doom9.org/showthread.php?p=1432839#post1432839
http://forum.doom9.org/showthread.php?p=1431042#post1431042
http://forum.doom9.org/showthread.php?p=1423182#post1423182
http://forum.doom9.org/showthread.php?p=1416675#post1416675
http://forum.doom9.org/showthread.php?p=1415381#post1415381
http://forum.doom9.org/showthread.php?p=1414650#post1414650
http://forum.doom9.org/showthread.php?p=1412869#post1412869
http://forum.doom9.org/showthread.php?p=1411457#post1411457
http://forum.doom9.org/showthread.php?p=1411207#post1411207
etc.
etc.
etc.

There are more of them..

No comment...

Edit: I think 4763 posts and 2002 subscription are enough :D

Didée
23rd June 2011, 19:35
But, speaking frankly, it IS like that ...

Edit: Haha, Nephilis - this search I did a few moments ago, too. ;)


Edit#2: @tormento:

Funnily, this filter here can be run under Avisynth-x64 practically without any problem. The builds of MVTools-x64 and ttempsmooth-x64 are out there. You have to ditch Blockbuster, but that's not essential for "the method" anyway. Just cancel it, or replace with AddGrain or Gradfun2db. And ba-DANG - another filter you can use with Avisynth-x64 until the next script crash.

tormento
23rd June 2011, 19:42
No problem. I'll shut up, at last.

Alek93j
24th June 2011, 15:43
Ok, I updated the filter with the changes that Didée suggested, thanks for your help.


Edit: up new version

steptoe
16th April 2013, 11:08
Just changed your code very slightly to point directly to the extra filters needed so you don't have to hunt them down

#######################################################################################################
#
# ~HQDegrain v0.9.2 by Alek93j~ 29/06/2011
#
# Required MVtools2.dll http://avisynth.org.ru/mvtools/mvtools-v2.5.11.3.zip
# ttempsmooth.dll http://www.missouri.edu/~kes25c/TTempSmoothv094.zip
# mt_masktools-25.dll http://manao4.free.fr/masktools-v2.0a48.zip
# blockbuster.dll http://avisynth.org/warpenterprises/files/blockbuster_20_dll_20021229.zip
# removegrain.dll http://home.arcor.de/kassandro/prerelease/RemoveGrain-1.0.rar
#
# ~Syntax~
#######################################################################################################