Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 16th April 2006, 23:05   #1  |  Link
Heini011
Registered User
 
Join Date: Nov 2003
Posts: 148
script function RemoveNoiseMC

Hi folks,

i wrote a little script function as a better alternative for the old Dust. it is optimized for removing noise with only a little lost of details. it removes larger spots and other dirt also.
this script works in YV12 color mode only. the video-clip must have a mod 8 resolution.

i suggest to insert in the first line of the avisynth-script:

SetMemoryMax(128) # for 512MB RAM
or
SetMemoryMax(256) # for >=1024 MB RAM

if you want to remove only a little bit of noise from a high quality movie, please look at RemoveNoiseMC_HQ

Code:
function RemoveTempGrain(clip input, int _mode)
{
  rg = RemoveGrain(input, mode=_mode)
  return TemporalRepair(rg, input)
}

function RemoveDirt(clip input, int limit, int rgrain, bool "_grey")
{
  _grey = default(_grey, false)
  _dgr1 = 0.35+rgrain*0.3
  _dgr2 = 0.45+rgrain*0.4
  repmode = 1
  clensed = Clense(input, grey=_grey, cache=4)
  restore = input.FluxSmoothST(3+3*rgrain,rgrain)
  restore = Repair(restore, input, mode=repmode, modeU= _grey ? -1 : repmode)
  restore = rgrain==0 ? restore.RemoveGrain(1) : \
    restore.VagueDenoiser(threshold=_dgr1, chromaT=_dgr1, nsteps=7, percent=75).RemoveGrain(1)
  alt = input.VagueDenoiser(threshold=_dgr2, chromaT=_dgr2, nsteps=7, percent=100).RemoveGrain(5)
  return RestoreMotionBlocks(clensed, restore, neighbour=input, alternative=alt, pthreshold=4+2*rgrain, cthreshold=6+2*rgrain, gmthreshold=40, dist=1, dmode=2, debug=false, noise=limit, noisy=12, grey=_grey)
}

global idx_c = 25

function RemoveNoiseMC(clip,int "rdlimit", int "rgrain", int "denoise", bool "sharp", float "csharpen", bool "_grey")
{
  rdlimit = default(rdlimit,11)
  rgrain = default(rgrain,2)
  denoise = default(denoise,8)
  sharp = (rgrain<1) ? default(sharp,true) : default(sharp,false)
  csharpen = (rgrain>2) ? default(csharpen,0.17) : (rgrain>1) ? default(csharpen,0.15) : default(csharpen,0.13)
  csharpen = sharp ? csharpen : csharpen+0.08
  _grey = default(_grey,false)
  _dgr = 0.45+rgrain*0.4
  dummy = clip.BlankClip(length=0)
  global idx_c = idx_c+1
  cbs = 8
  cov = (cbs>4) ? cbs/4 : 0
  ccf = cbs*cbs/64
  cpn = (denoise>12) ? 50*ccf : (denoise>8) ? 58*ccf : (denoise>5) ? 66*ccf : 72*ccf
  csh = sharp ? 1 : 0
  bvec2 = clip.MVAnalyse(isb=false, blksize=cbs, delta=2, pel=2, sharp=csh, overlap=cov, truemotion=true, pnew=cpn, idx=idx_c)
  bvec1 = clip.MVAnalyse(isb=false, blksize=cbs, delta=1, pel=2, sharp=csh, overlap=cov, truemotion=true, pnew=cpn, idx=idx_c)
  fvec1 = clip.MVAnalyse(isb=true, blksize=cbs, delta=1, pel=2, sharp=csh, overlap=cov, truemotion=true, pnew=cpn, idx=idx_c)
  fvec2 = clip.MVAnalyse(isb=true, blksize=cbs, delta=2, pel=2, sharp=csh, overlap=cov, truemotion=true, pnew=cpn, idx=idx_c)
  backw1 = rdlimit>13 ? \
    rdlimit>20 ? \
      clip.MVFlow(bvec1, idx=idx_c).Deblock(quant=22, aOffset=6, bOffset=6) : \
      clip.MVFlow(bvec1, idx=idx_c).Deblock(quant=16, aOffset=4, bOffset=4) : \
    clip.MVFlow(bvec1, idx=idx_c)
  forw1  = rdlimit>13 ? \
    rdlimit>20 ? \
      clip.MVFlow(fvec1, idx=idx_c).Deblock(quant=22, aOffset=6, bOffset=6) : \
      clip.MVFlow(fvec1, idx=idx_c).Deblock(quant=16, aOffset=4, bOffset=4) : \
    clip.MVFlow(fvec1, idx=idx_c)
  clp = interleave(backw1,clip,forw1)
  clp = clp.RemoveDirt(rdlimit,rgrain,_grey)
  dnc = denoise==0 ? clp.RemoveTempGrain(rgrain).SelectEvery(3,1) : dummy
  clp = clp.SelectEvery(3,1)
  dnc = denoise==0 ? dnc : \
    clp.MVDenoise(bvec2, bvec1, fvec1, fvec2, thT=denoise, thSAD=190+15*denoise, thmv=40, thSCD1=230+5*denoise)
  vid_mo = dnc.VagueDenoiser(threshold=_dgr, chromaT=_dgr, nsteps=7, percent=75)
  vid_mo = rgrain==1 ? vid_mo.RemoveGrain(1) : vid_mo.RemoveGrain(5)
  dnc = dnc.ConditionalFilter(dnc, vid_mo, "(YDifferenceFromPrevious()+YDifferenceToNext())/AverageLuma()", "<", "0.3")
  clp = clp.SeeSaw(dnc, Sstr=csharpen, Szp=12, SdampHi=20, bias=40)
  return clp
}
some examples:

for slight noisy video or spot/dirt removing only:

Code:
function hq_filter(clip c)
{
  c=c.RemoveNoiseMC(rdlimit=6,rgrain=1,denoise=0)
  c=c.LimitedSharpenFaster(Smode=4,strength=18,overshoot=0,wide=false,ss_x=1.3,ss_y=1.3)
  return c
}
medium noise, higher grain:

Code:
function mq_filter(clip c)
{
  c=c.RemoveNoiseMC(rdlimit=10,rgrain=2,denoise=8)
  c=c.LimitedSharpenFaster(Smode=4,strength=20,overshoot=1,wide=false,ss_x=1.3,ss_y=1.3)
  return c
}
high noise or analog capture:

Code:
function lq_filter(clip c) {
  c=c.RemoveNoiseMC(rdlimit=18,rgrain=3,denoise=14)
  c=c.VagueDenoiser(threshold=0.6, nsteps=6, chromaT=0.6, percent=75)
  c=c.deen("a3d",rad=3,thrY=3,thrUV=5,min=0.25,tthY=2,tthUV=3,scd=6)
  c=c.LimitedSharpenFaster(Smode=4,strength=24,overshoot=1,wide=false,ss_x=1.3,ss_y=1.3)
  return c
}
the extreme case with a modded function for high denoise values:

Code:
function RemoveHighNoiseMC(clip clip,int "rdlimit", int "rgrain", int "denoise", float "csharp", bool "_grey")
{
  rdlimit = default(rdlimit,24)
  rgrain = default(rgrain,3)
  denoise = default(denoise,15)
  csharp = (rgrain>2) ? default(csharp,0.32) : default(csharp,0.28)
  _grey = default(_grey, false)
  _dgr = 0.7+rgrain*0.5
  global idx_c = idx_c + 1
  cbs = 8
  ccf = cbs*cbs/64
  cpn = (denoise>12) ? 50*ccf : 57*ccf
  bvec3 = clip.MVAnalyse(isb=false, blksize=cbs, delta=3, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
  bvec2 = clip.MVAnalyse(isb=false, blksize=cbs, delta=2, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
  bvec1 = clip.MVAnalyse(isb=false, blksize=cbs, delta=1, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
  fvec1 = clip.MVAnalyse(isb=true, blksize=cbs, delta=1, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
  fvec2 = clip.MVAnalyse(isb=true, blksize=cbs, delta=2, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
  fvec3 = clip.MVAnalyse(isb=true, blksize=cbs, delta=3, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
  backw1 = rdlimit>20 ? \
      clip.MVFlow(bvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
    clip.MVFlow(bvec1).Deblock(quant=16, aOffset=4, bOffset=4)
  forw1  = rdlimit>20 ? \
      clip.MVFlow(fvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
    clip.MVFlow(fvec1).Deblock(quant=16, aOffset=4, bOffset=4)
  clp = interleave(backw1, clip, forw1)
  clp = clp.RemoveDirt(rdlimit,rgrain,_grey)
  clp = clp.SelectEvery(3,1)
  dnc = clp.MVDenoise(bvec3, bvec2, bvec1, fvec1, fvec2, fvec3, thT=denoise, thSAD=160+12*denoise, thmv=40, thSCD1=255+4*denoise)
  vid_mo = dnc.VagueDenoiser(threshold=_dgr, chromaT=_dgr, nsteps=6, percent=75)
  vid_mo = rgrain==1 ? vid_mo.RemoveGrain(1) : vid_mo.RemoveGrain(5)
  dnc = dnc.ConditionalFilter(dnc, vid_mo, "(YDifferenceFromPrevious()+YDifferenceToNext())/AverageLuma()", "<", "0.3")
  return csharp==0 ? dnc : clp.SeeSaw(dnc, Sstr=csharp, Szp=12, SdampHi=20, bias=40)
}

function vlq_filter(clip c) {
  c=c.RemoveHighNoiseMC(rdlimit=24,rgrain=3,denoise=16,csharp=0.3)
  c=c.VagueDenoiser(threshold=1.0, nsteps=6, chromaT=1.2, percent=75)
  c=c.deen("a3d",rad=3,thrY=4,thrUV=7,min=0.25,tthY=2,tthUV=3,scd=7)
  c=c.LimitedSharpenFaster(Smode=4,strength=28,overshoot=1,wide=false,ss_x=1.2,ss_y=1.2)
  return c
}
needed plugins:

Latest MVTools 1.4, DeBlock 1.2 and VagueDenoiser 0.35.1 from Fizick's Page
RemoveGrain pre 1.0
RemoveDirt 0.9
FluxSmooth 1.1a from warpenterprises
SeeSaw 2006.01.02, therefor MaskTools 1.5.8

for my examples
LimitedSharpenFaster, therefor MaskTools v2.0a30
deen 1.0 beta 1 (old version)


suggestions for parameter selection:

- set rdlimit=0, denoise=0 and rgrain according to the examples above
- look for a very bad frame
- increase rdlimit until the big noise is gone
- increase denoise until the remaining noise is minimized

- set sharp=true for a video with high resolution of fine details, which you want to encode using a high bitrate
set sharp=false for older film material or tv captures

revisions:

april 19th:
-RemoveHighNoiseMC clean-up
april 20th:
-deen 1.0 beta 1 attachment added
-hints for parameter selection added
-minor removedirt parameter change
may 10th:
- spatial filtering for moving block areas in RemoveDirt
- minor parameter changes
may 29th: deen parameter in vlq_filter example
june 12th:
- improved scene switch behaviour
june 24th:
- minor script error corrected (quality unchanged), thanks to Alain2
august 15th:
- bicubic subpixel interpolation in mvanalyze. much better sharpness thanks to Fizick new MVTools!
august 16th:
- adjustments for sharp subpixel interpolation
november 28th:
- sharp option for sharp subpixel interpolation added for manual selection
.
Attached Files
File Type: zip deen10-beta1.zip (15.2 KB, 1788 views)

Last edited by Heini011; 28th November 2006 at 22:20.
Heini011 is offline   Reply With Quote
Old 16th April 2006, 23:48   #2  |  Link
Heini011
Registered User
 
Join Date: Nov 2003
Posts: 148
sample picture: vlq_filter
Attached Images
 
Heini011 is offline   Reply With Quote
Old 16th April 2006, 23:51   #3  |  Link
danpos
BDVD Team
 
danpos's Avatar
 
Join Date: Dec 2004
Location: Rio de Janeiro - RJ/Brasil
Posts: 306
@Heini011

Hi there! It seems quite interesting, so thanks for share it with us. I just have a suggestion: posting all filters dependencies in order to get the script working, because of maybe some users can to have some difficulty to know what filters/versions are necessary to get it working properly.

Cheers,
danpos is offline   Reply With Quote
Old 17th April 2006, 01:50   #4  |  Link
danpos
BDVD Team
 
danpos's Avatar
 
Join Date: Dec 2004
Location: Rio de Janeiro - RJ/Brasil
Posts: 306
@Heini011

Well done, thanks!

Cheers,
danpos is offline   Reply With Quote
Old 17th April 2006, 13:18   #5  |  Link
Heini011
Registered User
 
Join Date: Nov 2003
Posts: 148
sample picture: lq_filter
Attached Images
 

Last edited by Heini011; 17th April 2006 at 13:26.
Heini011 is offline   Reply With Quote
Old 17th April 2006, 13:35   #6  |  Link
Heini011
Registered User
 
Join Date: Nov 2003
Posts: 148
sample picture: hq_filter
(from a pre version of this filter, not exactly the same...)
Attached Images
 
Heini011 is offline   Reply With Quote
Old 17th April 2006, 19:13   #7  |  Link
danpos
BDVD Team
 
danpos's Avatar
 
Join Date: Dec 2004
Location: Rio de Janeiro - RJ/Brasil
Posts: 306
@Heini011

Good examples. They show that the propers 'filters' perform a really good job on the 'dusts' in each case ...

Cheers,
danpos is offline   Reply With Quote
Old 18th April 2006, 22:45   #8  |  Link
Alain2
Registered User
 
Join Date: May 2005
Posts: 236
Thanks Heini011 for sharing your script, it proves to be rather good on some materials I have and was struggling on with other methods ; motion compensation seems to be very efficient at keeping details while doing a good job at denoising!

[edit] I just noticed you changed the vlq_filter from the initial one, introducing the RemoveHighNoiseMC, and also you changed denoise=6 to denoise=0 in the hq_filter... If you make other mods in the future, please let us know or we will miss them ^^

Last edited by Alain2; 19th April 2006 at 01:04.
Alain2 is offline   Reply With Quote
Old 19th April 2006, 07:22   #9  |  Link
rfmmars
Registered User
 
Join Date: Feb 2004
Posts: 743
Thank for the great documentaion that is ever so missing in some other posts. It help a dumby like me.

Richard

Last edited by rfmmars; 19th April 2006 at 17:21.
rfmmars is offline   Reply With Quote
Old 19th April 2006, 13:17   #10  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,556
Another thing I can suggest is to package it into a single file, so people can just download it right to their avs folder. You're more than welcome to upload it to avisynth's wiki and link it from there (though the link will change anytime you update your script, except when linking from the wiki), or you can ask richard for a bit of space. I have a video I'd like to test this out on soon!

Last edited by foxyshadis; 20th April 2006 at 00:41.
foxyshadis is offline   Reply With Quote
Old 19th April 2006, 15:08   #11  |  Link
Morpheus_xx
Registered User
 
Join Date: Feb 2002
Posts: 85
Great functions! I tried it for my tv-capture and I really like the results!

During encoding on my AMD X2 dual core cpu, the load goes only up to 58%.

Could somebody tell me, where in the script I could put the "SetMTMode" commands properly, so that it can be faster?

I tried this, but got green dots over the image:
Code:
SetMTMode(2)
mq_filter()
Any suggestions?
Morpheus_xx is offline   Reply With Quote
Old 20th April 2006, 00:19   #12  |  Link
Alain2
Registered User
 
Join Date: May 2005
Posts: 236
I didn't have a problem without the deblock function from fizick, but I realize that i have also dgdecode.dll in avisynth plugins directory, which contains a deblock() function as well ; maybe it could be a good idea to rename one of these functions to avoid confusion between the 2 (as it's likely that a lot of people have dgdecode.dll in their plugins), or maybe change the script for deblock functions, to be called with the syntaxe deblock_deblock() to specifically call fizick'dll deblock function... (This is probably not a real issue as from fizick's and neuron2's changelogs it appears they didn't really touch manao's core deblock function)
Alain2 is offline   Reply With Quote
Old 20th April 2006, 01:12   #13  |  Link
Heini011
Registered User
 
Join Date: Nov 2003
Posts: 148
Hi,

for thresholds > 0.6 fft3dfilter makes a better job than vaguedenoiser. here is an example for an improved vlq_filter:

Code:
function vlq_filter_ultimate(clip c) {
  c=c.RemoveHighNoiseMC(rdlimit=24,rgrain=3,denoise=16)
  c=c.fft3dfilter(sigma=1.2, sharpen=0.4, plane=4)
  c=c.deen("a3d",rad=3,thrY=4,thrUV=7,min=0.25,tthY=2,tthUV=3,scd=7)
  return c
}
needed plugins:
fft3dfilter185.zip, therefor fftw3.dll


@Morpheus_xx:

SetMTMode(2) should, according to avisynth documentation, be in the first line of your avs-script to support multi-threading.

@moderator:

please approve my deen 1.0 beta 1 attachmend in the first post.

greetings.

Last edited by Heini011; 29th May 2006 at 00:21.
Heini011 is offline   Reply With Quote
Old 20th April 2006, 07:13   #14  |  Link
Morpheus_xx
Registered User
 
Join Date: Feb 2002
Posts: 85
Quote:
Originally Posted by Heini011
@Morpheus_xx:

SetMTMode(2) should, according to avisynth documentation, be in the first line of your avs-script to support multi-threading.
yes, I know. But this gives me green dots over the image...

I think one of the many used filters does not work so well with MTMode(2). That why I've asked where to "tweak" these functions.
Morpheus_xx is offline   Reply With Quote
Old 20th April 2006, 13:08   #15  |  Link
Mr. Brown
Step into the Light
 
Mr. Brown's Avatar
 
Join Date: Dec 2005
Location: Germany
Posts: 20
@ Heini011
thank you for RemoveNoiseMC it's good
Mr. Brown is offline   Reply With Quote
Old 20th April 2006, 14:12   #16  |  Link
aBra
Registered User
 
Join Date: Dec 2005
Posts: 5
I can't use the script with the latest MVTools 0.9.9.1.
aBra is offline   Reply With Quote
Old 20th April 2006, 19:41   #17  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,556
That's the latest pre-1.0 version. You need 1.1.
foxyshadis is offline   Reply With Quote
Old 20th April 2006, 23:14   #18  |  Link
Alain2
Registered User
 
Join Date: May 2005
Posts: 236
1.2.2 (with deblock removed from package and left in the separate plugin, as described in the needed plugins in the first post) ; use the links provided by Heini011 to find the correct plugins versions
Alain2 is offline   Reply With Quote
Old 21st April 2006, 08:32   #19  |  Link
Wilbert
Moderator
 
Join Date: Nov 2001
Location: Netherlands
Posts: 6,364
@Heini011,

I approved your attachment. I'm sorry that I didn't notice it.
Wilbert is offline   Reply With Quote
Old 21st April 2006, 16:55   #20  |  Link
Disabled
Registered User
 
Join Date: Aug 2004
Posts: 211
Well it took me some time to get it running, but now I got it and its a fantastic script!
I did a quick test with chapter 29 from Matrix and everything looked great, besides this:

There is a flashlight in red for just one frame and it gets detected as dirt. I'm impressed by the quality, as even the face is cleared from the light. Really, this is no BS, no color conversion between the both and no other tuning, this is just RNMC working!
I'm using the mq_filter preset...

*edit*
some more examples:




Conclusion: This might be a pretty cool script for slow movies, but extreme action and fog is a no no, as it kills major detail
And by the way: I'm using the mq preset as hq didn't remove enough noise. But I'm trying my own settings now.

Last edited by Disabled; 22nd April 2006 at 08:54.
Disabled is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 02:58.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.