Can any filter fight with this noise?

[Q-W-Y]

Hello, i have very badly noise DV
badnoise.zip (5,8 MB) (http://ceskyinterier.cz/temp/badnoise.zip)

I have problems to encode this...
..and after noise filtering with FluxSmooth, the noise is even worse than the source.

Anybody could help please?

Thank you..

[*.mp4 guy]

I used this avs script:
loadplugin("G:\smoothd.dll")
avisource("C:\badnoise.avi")
smoothd(3,8)
#smoothd(A,B): A=strenth, max=31. B=acuracy, max=8.

and got these results:
http://img205.exs.cx/img205/9602/lessnoise6jj.png

This is the filter i typically use to get rid of tough noise, hope it helps.

edit: By the way you should put post like this in the avisynth usage foeum in the future as i dont beleive its directly related to xvid. unless i misunderstood your post that is:D.

[Q-W-Y]

Hmm.. is it possible to move this post to avisynth thread?

[zulu]

is smoothD really suited for noise removal?
i thought it's a blocking-killer?
the image seems a little bit too soft for my taste, maybe a sharpening filter would help.. any recommendations?

my digicam videos suffers from a similar noise if there isn't enough light..

[Manao]

function efficace(clip c, bool backward, int del)
{
v = c.mvanalyse(isb = backward, lambda = 1000, blksize = 16, delta = del, idx = 1, pel = 2)
d = c.mvcompensate(v, mode = 1, thscd1 = 2500, thscd2 = 170).deblock(quant = 30)
return v.mvchangecompensate(d)
}

source = avisource("k:\badnoise.avi").converttoyv12(interlaced = true).tomsmocomp(1, 0, 1)

bw3 = source.efficace(false, 3)
bw2 = source.efficace(false, 2)
bw1 = source.efficace(false, 1)
fw1 = source.efficace(true, 1)
fw2 = source.efficace(true, 2)
fw3 = source.efficace(true, 3)

denoised = source.mvdenoise(bw3, bw2, bw1, fw1, fw2, fw3, thsad = 2500, thmv = 200, tht = 30, thscd1 = 2500, thscd2 = 170)It needs the MVTools and TomsMoComp ( for deinterlacing ). Settings could be better tuned. http://manao4.free.fr/badclip_source_29.png
http://manao4.free.fr/badclip_denoised_29.png

[Q-W-Y]

Thx all!
I have found FFT3dFilter to be the best for it :)

http://forum.doom9.org/showthread.php?s=&postid=607504#post607504

EDIT: It is best together with Manao's noise filter :)

[zulu]

did you try Manaos script above?
i gave it a try and i must admit the result looks great :)

@Manao:
what params should i tweak if i have a less noisy source?

[scharfis_brain]

@manao: Wow! This denoiser is GREAT!

I did some minor changes on that script to process the whole 50fps information of that PAL-DV-clip:

loadplugin("d:\x\tomsmocomp.dll")
loadplugin("d:\x\mvtools0991.dll")

function tmcbob(clip i, int se) #50fps-bobbing
{
e=i.tomsmocomp(-1,se,0)
o=i.doubleweave().selectodd().tomsmocomp(-1,se,0)
interleave(e,o)
}


function reYV12(clip i) #reverting PAL-DV to YUV 4:2:0 using simple line selection
{
yx=i.converttoyv12().greyscale()
x=i.separatefields().separatefields().selectevery(4,0,2).weave()
ux=x.utoy().converttoyv12()
vx=x.vtoy().converttoyv12()
ytouv(ux,vx,yx)
}


function manaodenoise(clip source) #denoising
{
function efficace(clip c, bool backward, int del)
{
v = c.mvanalyse(isb = backward, lambda = 1000, blksize = 16, delta = del, idx = 1, pel = 2)
d = c.mvcompensate(v, mode = 1, thscd1 = 2500, thscd2 = 170).deblock(quant = 30)
return v.mvchangecompensate(d)
}

bw3 = source.efficace(false, 3)
bw2 = source.efficace(false, 2)
bw1 = source.efficace(false, 1)
fw1 = source.efficace(true, 1)
fw2 = source.efficace(true, 2)
fw3 = source.efficace(true, 3)

source.mvdenoise(bw3, bw2, bw1, fw1, fw2, fw3, thsad = 2500, thmv = 200, tht = 30, thscd1 = 2500, thscd2 = 170)
}


input=avisource("badnoise.avi").assumebff()

input.reYV12().tmcbob(5).manaodenoise()

output=converttoyuy2().assumebff().separatefields().selectevery(4,0,3).weave()

return output

[Manao]

zulu : on a clean source, you can use blocksize = 8 or 16, you should only need 4 vectors clips ( bw2, bw1, fw1, fw2 ), you can lower the quant in the deblock() function ( or even bypass that deblocking, returning directly v in the efficace() function ), and you must lower thSAD, thT, and thSCD1 thresholds. thSAD and thSCD1 should be around blocksize * blocksize * std(noise) ( can be up to 2 times this value ), and tht should be std(noise)( there again, can be raised )

Here, i assumed roughly std(noise) ~ 8, but i choosed a higher thT.

For clean movies, i advise thT < 5, thSAD = 400 / 500 for 8x8 blocks.

Now, for a speed / quality tradeoff, i would replace the pel = 2 by a pel = 1 in mvanalyse, use 16x16 blocks, and disable the deblocking.

[Socio]

That is quite impressive going to test it out my self

[zulu]

Manao:
Thanks a lot! :)
Though i made an anonying observation:
with the script you posted above (your first post), i get massive blocking in high motion areas. anything i can do against that?
i can post a sample clip if you want.

TIA,
zulu

[Manao]

Reduce the thresholds, especially thMV : if the motion vector's length is over thMV, the block won't be used for denoising, so if your issues are only in high motion areas, that's the one you want to lower.

[Q-W-Y]

Originally posted by Manao
Reduce the thresholds, especially thMV : if the motion vector's length is over thMV, the block won't be used for denoising, so if your issues are only in high motion areas, that's the one you want to lower.

I have to say again that your filter is really great. ... very slow, but great result. (this is 2nd day of my encode :) at 1fps )

[Paazabel]

This filter caught my attention, and was also referred to me by scharfi's_brain ... so I tried it on a few pieces of content.

For the most part, it works great. However, here was one instance where I had to scratch my head. Note the window on the right side of the frame ...

Before ...
http://sgt.cc/stupid/Before.jpg

And After ...
http://sgt.cc/stupid/After.jpg

This filter is a bit heavy handed for normal camcorder noise [dnr2() works fine on the above clip], but worked great on some 1980's VHS I had been messing with.

[Manao]

It's the fault of the (de)interlacing, and there's nothing you can do about it. Sometimes, it's just impossible to reconstruct the data lost, and it's one of these case.

What happens is the following :


Original picture

xx0000xx0000xx0000
00xx0000xx0000xx00
0000xx0000xx0000xx
xx0000xx0000xx0000
00xx0000xx0000xx00
0000xx0000xx0000xx
xx0000xx0000xx0000

Original picture, one out of two lines

xx0000xx0000xx0000
0000xx0000xx0000xx
00xx0000xx0000xx00
xx0000xx0000xx0000

Reconstructed picture

xx0000xx0000xx0000
x0000xx0000xx0000x
0000xx0000xx0000xx
000xx0000xx0000xx0
00xx0000xx0000xx00
0xx0000xx0000xx000
xx0000xx0000xx0000

[scharfis_brain]

@Paazabel:

try to use the script that I have posted above.
It should give you a much better vertical detail (the patterned window!) while retaining the interlacing!

but change this line:
input.reYV12().tmcbob(5).manaodenoise()

into
input.reinterpolate411().tmcbob(5).converttoyv12().manaodenoise()

to use it properly with NTSC-DV.

you also need to load reinterpolate411.dll.

[Paazabel]

Thanks for the tips on tweaking the filter. I will probably use this quite a bit on my older content. I just thought it was funny that it turned the perspective backwards on this one.

The ceiling fans were also giving it fits ... but lowering thmv fixed that. They were quite the blocky mess.

[EasyStart]

Pardon my english.

I tried scharfis_brain's script on a 3 minutes 720x576 PAL analogue capture using FFDSHOW huffy YV12 codec. The result looks great but there is one problem. It is probably me, being a newbie in Avisynth Scripting. The encoded mpeg2 file only has 1.5 minutes. What did I do wrong ? I made some small changes to the script. These are the changes:

x=i.separatefields().AssumeFieldBased().selectevery(4,0,2).weave()

input=DirectShowSource("capture huffyuv YV12.avi", fps=25, seek=false, audio= true, video=true ).assumetff()

output=converttoyuy2().assumetff().separatefields().AssumeFieldBased().selectevery(4,0,3).weave()


Please, advise what I need to do to the script to get a 3 minutes in and 3 minutes out into my mpeg encoder.

Thank you.

[EasyStart]

If I changed the line
x=i.separatefields().separatefields().selectevery(4,0,2).weave()

to

x=i.separatefields().AssumeFrameBased().SeparateFields().selectevery(4,0,2).weave()

then

ytouv(ux,vx,yx)

throws an error saying "YToUV: Y clip does not have the double height of the UV clipes! (YV12 mode)"

I am currently running Avisynth build AviSynth_256_200705.exe

Please, help. I am stuck.

[scharfis_brain]

try replacing reYV12() by converttoyv12(interlaced=true)

why do you mess with the code given in the functions?

[EasyStart]

Thanks scharfis_brain for your quick reply. Will try it in a minute. The reason why I was messing around with your code is because I am getting an error on line

x=i.separatefields().SeparateFields().selectevery(4,0,2).weave()

The error is SeparateFields: SeparateFields should be applied on frame-based material: use AssumeFrameBased() beforehand

[EasyStart]

scharfis_brain, I am trying it now.

input.ConvertToYV12( interlaced = true ).tmcbob(5).manaodenoise()

It is 5 minutes into the encoding, CCE tells me it will take another 44 minutes to complete. Your original code, reYV12(), only took 24 minutes to encode a 3 minutes footage. Is there anything you can do to your reYV12() function, because ConvertToYV12( interlaced = true ) is taking 49 minutes instead of 24 minutes. I often encode one and half hour video, this is going to take 24 and a half hours.

32 bits Athlon XP 3200+
Windows XP SP2
1.5 GB RAM
2 x 200 GB harddrive
:cool:

[EasyStart]

scharfis_brain, please, accept my apology for my previous post. I realized that I wasn't thinking straight. The 24 minutes encode was for the 1.5 minutes footage (because I removed a SeparateFields() and selectevery(4,0,2) was dropping half of the frames). A 3 minutes footage should take twice as long.

I will try to thing better next time before posting.
:cool:

[Yuri]

What could be possibly wrong with the script or anything if I get a message: "There is no function 'mvanalyse'"

The script:
---------------------------------
function tmcbob(clip i, int se)
{
e=i.tomsmocomp(-1,se,0)
o=i.doubleweave().selectodd().tomsmocomp(-1,se,0)
interleave(e,o)
}


function reYV12(clip i)
{
yx=i.converttoyv12().greyscale()
x=i.separatefields().separatefields().selectevery(4,0,2).weave()
ux=x.utoy().converttoyv12()
vx=x.vtoy().converttoyv12()
ytouv(ux,vx,yx)
}


function manaodenoise(clip source)
{
function efficace(clip c, bool backward, int del)
{
v = c.mvanalyse(isb = backward, lambda = 1000, blksize = 16, delta = del, idx = 1, pel = 2)
d = c.mvcompensate(v, mode = 1, thscd1 = 2500, thscd2 = 170).deblock(quant = 30)
return v.mvchangecompensate(d)
}

bw3 = source.efficace(false, 3)
bw2 = source.efficace(false, 2)
bw1 = source.efficace(false, 1)
fw1 = source.efficace(true, 1)
fw2 = source.efficace(true, 2)
fw3 = source.efficace(true, 3)

source.mvdenoise(bw3, bw2, bw1, fw1, fw2, fw3, thsad = 2500, thmv = 200, tht = 30, thscd1 = 2500, thscd2 = 170)
}


input=avisource("F:\video_3.avi").assumebff()

input.tmcbob(5).manaodenoise()

output=converttoyuy2().assumebff().separatefields().selectevery(4,0,3).weave()

return output
------------------------------

MVTools v.0.9.9.1 autoloads from PLUGIN folder
Avisynth v.2.5.6.0

Thank you.

[freezer]

@ scharfis_brain

I haven't been able to get your script to work (and feel totally stupid not beeing able). I always get the error message that it expects an "{" in front of the first function{} in the script, but it is just a copy and paste of your script and I double checked that I did not miss a single line. And it wouldn't make any sense to set another { in front of the function definition.

Strange, because Manaos script works like a charme. It might be a stupid little thing, but I am totally lost. As I have a programming background myself it makes me look double worse :(

Btw, I am using AviSynth 2.5.5.

[freezer]

OK, now I feel realy stupid %)
Just had to remove the comments after the function lines now it works.

[FredThompson]

How does scharfis_brain's script compare to wavelet noise reduction wrt speed/output.

btw, the sample images on page 1 of this thread are different frames. Not a major issue, though, the character of the filtering is easily seen.

[ArcticWolf]

Hi i've also a bad DVD source... so i'm trying to filter it but it doesn't go well i think...

http://83.246.113.52/unfiltered.avi (ffdshow ffvh codec) (150MB)

http://83.246.113.52/test3.mkv (filtered + h246) (3MB)


X264 settings:
http://members.chello.nl/h.ambtman/x264-set11.JPG
http://members.chello.nl/h.ambtman/x264-set22.JPG
http://members.chello.nl/h.ambtman/x264-set33.JPG


AVS file i used for test3.mkv:

mpeg2source("K:\DVD\Rips\DRZD08017_VTS_01_PGC1\saikano.d2v",cpu=6)
undot()
tfm(d2v="K:\DVD\Rips\DRZD08017_VTS_01_PGC1\saikano.d2v")
tdecimate(mode=1)
crop(4,4,-4,-6)
Convolution3d (preset="movieLQ")
RemoveGrain(mode=12)
limitedsharpen(dest_x=720, dest_y=480)

i want to try manaodenoise... but i can't get it working somehow... i'm still a newb. with avsi and natural movies...

http://members.chello.nl/h.ambtman/manaodenoise.avsi

i used the sample from this thread... but i get this msg when opening in vdubmod:
Expected "{" in line 4
and nothing more...

can somebody help me with this problem ?

[Didée]

Expected "{" in line 4
See all those "function xyz(blabla) #comment text" in the script?

Delete all those comments. Then it'll work.

[Wilbert]

In case people are still confused:

x=i.separatefields().separatefields().selectevery(4,0,2).weave()

should be

x=i.separatefields().assumeframebased().separatefields().selectevery(4,0,2).weave()

[ArcticWolf]

well it works fine now...
but when moving i get huge blocks in the faces etc...
is that because the deblock filter or ?

[unskinnyboy]

Read posts #11 & #12 in this thread.

[Didée]

That's one of the inherit problems of motion compensation. The deblock line, in contrary, tries to help with the problem ... but it's only an "I surrender" action.

Possibilities:

a) lower the thresholds "thXYZ" in mvdenoise. (But "low enough" to avoid *all* possible artefacts means there's no denoising any more...)

b) use mvflow instead of mvcompensate (will change "blocking" artefacts into tearing/smearing artefacts)
.edit - doh, this doesn't apply for mvdenoise(), of course.

c) Search this forum for "RemoveNoiseMC" and "MCNR_simple2"

Using motion compensation with a small handy script is easy. Using the method for hi-quality filtering without sideffects is a task.
RemoveNoiseMC and MCNR_simple2 are much better in this respect, but still, neither of them is fully free of possible unwanted artefacting. So, this brings up -

possibility d) : don't use motion compensated denoising

( dd) : until some one finished his bricolage ;) )

[Mug Funky]

holy crap! they made a live-action movie of that anime? this is going to be interesting... this is the same saikano where the girl gets turned into a kind of flying robot human nuke from hell, yeah?

back on topic, you could try fft3dfilter on it - it works very well on natural video, and a recent japanese film shouldn't have much (if any) grain. if you're encoding with h.264 most of it will be removed during encoding anyway...

[ArcticWolf]

holy crap! they made a live-action movie of that anime? this is going to be interesting... this is the same saikano where the girl gets turned into a kind of flying robot human nuke from hell, yeah?

back on topic, you could try fft3dfilter on it - it works very well on natural video, and a recent japanese film shouldn't have much (if any) grain. if you're encoding with h.264 most of it will be removed during encoding anyway...

*_*

btw, can somebody tell me what all the threshold are ?

thsad =
thmv = fastmotion threshold
tht =
thscd1 =
thscd2 =

but the rest ?

[Mug Funky]

hmm. not sure we can help you any further... you did read the rules (especially rule 6), right?

[ArcticWolf]

hmm. not sure we can help you any further... you did read the rules (especially rule 6), right?


you can now right ?

[Wilbert]

@ArcticWolf, i had the impression you owned a legal version of the dvd. Talking about a downloaded version of it is a borderline case, but ok with me. However if i'm wrong and you don't own a legal version of it, i urge you not to respond to this thread anymore.

[ArcticWolf]

whatever ;_;

[Mug Funky]

this stuff's all available if you search (though the forum search is a tricky beast to tame, especially with three-letter acronyms etc).

thsad = sad = Sum of Absolute Differences. basically how different two blocks are from each other. the higher this threshold, the more chance of artefacts, the lower it is, the less denosing.
thmv = fastmotion threshold
tht = temporal threshold (i think). pretty much the same as what's in temporalsoften
thscd1 = first of two scenechange detection thresholds.
thscd2 = second of two scenechange detection thresholds. the mvtools docs have more info on how to set these (i keep forgetting).

[FredThompson]

Heh, try searching for "dv". Actually, that's a good idea. I wonder how tough it would be to modify the search to allow "dv" as an acceptable search term. (Yes, there's the DV forum but how to search for DV-specific matches in other areas?)

[unskinnyboy]

It is very much possible.

In vBulletin, the default minimum length required for a word is 4 characters (I believe). (set in AdminCP -> vBulletin Options -> Message Searching Options -> Search Index Minimum Word Length). *BUT*, if you have a special word (for e.g: "DV" in this case) which is relevant to the forum and need to be indexed by the search engine, there is an option right below the above mentioned one where you enter it so that it could be indexed by the search engine (AdminCP -> vBulletin Options -> Message Searching Options -> Words to be Included Despite Character Limit).

"DV" is not included in this forum as an indexable word atm. It should be IMO.