Removing edges(?) grain

[fabioseixal]

So, I'm trying to improve the Dragon Ball Z level 1.1 because i don't like to see that great amount of grain and noise "jumping" around. I'm still on tests but I've encountered on problem that I'm not able to resolve...
The problem I'm trying to talk about is on this (http://img804.imageshack.us/img804/3175/encodedc.png) places. I've uploaded a sample and the source video here (http://www.mediafire.com/?s6so9c03ftybajx). You can try here (http://www.fileserve.com/file/zv6CsKw/dbz.rar) too.
The noise filter I've used so far was this one:

super = MSuper(pel=2, sharp=1)
backward_vec3 = MAnalyse(super,isb = true, delta = 3, overlap=4)
backward_vec2 = MAnalyse(super,isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super,isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super,isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super,isb = false, delta = 2, overlap=4)
forward_vec3 = MAnalyse(super,isb = false, delta = 3, overlap=4)
MDegrain3(super,backward_vec1, forward_vec1, backward_vec2, forward_vec2, backward_vec3, forward_vec3, thSAD=500)

After using that I've noticed the problem that I'm trying to resolve, so I've made a search around here for some grain removals and as an example SMDegrain doesn't do anything, Calmgrain thoes a good job, but it washes too many details...

If someone could give me some hints that would be great guys :)

[Bloax]

CalmGrain? Whoah, someone actually used that?
Well, here's an update to it. Might help.

Unfortunately I can't get your sample, because Mediafire loves the times when your net is being retarded.
function CalmGrain(clip clp,int "tstr",int "bias1",int "bias2")
{
# Requirements: RemoveGrain, RedAverage (Just look it up on Google.)
# By yours truly, Bloax.

# ~~~~~ Wuzzis now?
# This 'should' calm grain down, I can now say that it "should" improve compressibility by a little bit.
# It certainly reduces the eye-tearing effect of heavy noise compression artifacts post-encoding though. :-)
# ~~~~~

# ~~~~~ Wuzzis do now?
# "tstr" Is now the strength of ReduceFluctuations, which I found to be just the thing for noise.
# Due to the fact that it just "stupidly" moves pixels along. Usually quite nasty, but good for calming grain! :P

# "bias*" Are the bias values for some of the masks, higher values (lower, actually) removes more "weak" spots.
# I wouldn't recommend going above negative values, and stay below -32. Mess around with it, or leave it. Basically.

# bias1 is for the main edge detection, higher (lower negative) values include more edges.
# And thus more noise might sneak it, ought to reduce artifacts a bit though.

# bias2 is for the noise mask, the lower the value, the less noise ought to be "caught", in theory.
# As said, just leave them, or mess around with them.

tstr = Default(tstr,16)
bias1 = Default(bias1,-64)
bias2 = Default(bias2,-128)

a = clp.IsYV12() ? clp : clp.ConvertToYV12()

msk=RAverageW(a,32,a.RemoveGrain(19,-1),-32,u=0,v=0,sse=8,bias=bias1) # Edges
msk2=Mt_Edge(a,"laplace",0,20,256,256,sse3=true) # Additional edges
msk3=Mt_Logic(msk,msk2,"max",sse3=true).RemoveGrain(1,-1) # Merging detected edges
noisemsk=RAverageW(a,32,a.RemoveGrain(17,-1),-32,u=0,v=0,sse=8,bias=bias2).Mt_Expand(sse3=true) # Finding noise
noisemskfin=Mt_Logic(msk3,noisemsk,"andn").ReduceFluctuations(64).RemoveGrain(2,-1) # Removing edges from noise mask.
RMerge(a,a.ReduceFluctuations(tstr),noisemskfin) # Merging, also trust me, you don't want edges in the mask. :)
}

[fabioseixal]

CalmGrain? Whoah, someone actually used that?
Well, here's an update to it. Might help.

I was just testing scripts to see how it would turn out. I've tested this avsi now and the results are the same, too much details washed away :s
Tried with tstr=10...

[Bloax]

Well that's :S indeed.
Can't really help then, since Mediafire likes to prepare the downloads very, very carefully.

[fabioseixal]

Well that's :S indeed.
Can't really help then, since Mediafire likes to prepare the downloads very, very carefully.

I Will upload to another host :)

EDIT: Done upload (http://www.fileserve.com/file/zv6CsKw/dbz.rar) to another host

[Didée]

As far as MVTools/MDegrain is concerend, you're suffering "thSAD bailout" from the noise itself. Use pre-denoising for the motion search to achieve lower SADs.

# MVTools2
# RemoveGrain
# MedianBlur
# FluxSmooth

DGSource("source_DBZ.dgi")
o = last
b = o.minblur(2).fluxsmootht()

sup0 = b.msuper(pel=2)
sup1 = o.msuper(pel=2,levels=1,sharp=2)

bv3 = sup0.manalyse(isb=true, delta=3,blksize=16,overlap=8,search=5) # ,DCT=5
bv2 = sup0.manalyse(isb=true, delta=2,blksize=16,overlap=8,search=5) # ,DCT=5
bv1 = sup0.manalyse(isb=true, delta=1,blksize=16,overlap=8,search=5) # ,DCT=5
fv1 = sup0.manalyse(isb=false,delta=1,blksize=16,overlap=8,search=5) # ,DCT=5
fv2 = sup0.manalyse(isb=false,delta=2,blksize=16,overlap=8,search=5) # ,DCT=5
fv3 = sup0.manalyse(isb=false,delta=3,blksize=16,overlap=8,search=5) # ,DCT=5

o.mdegrain3(sup1,bv1,fv1,bv2,fv2,bv3,fv3)

# stackhorizontal(o,last)

return(last)

#==================

function MinBlur(clip clp, int r, int "uv")
{
uv = default(uv,3)
uv2 = (uv==2) ? 1 : uv
rg4 = (uv==3) ? 4 : -1
rg11 = (uv==3) ? 11 : -1
rg20 = (uv==3) ? 20 : -1
medf = (uv==3) ? 1 : -200

RG11D = (r==0) ? mt_makediff(clp,clp.sbr(),U=uv2,V=uv2)
\ : (r==1) ? mt_makediff(clp,clp.removegrain(11,rg11),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20).removegrain(20,rg20),U=uv2,V=uv2)
RG4D = (r<=1) ? mt_makediff(clp,clp.removegrain(4,rg4),U=uv2,V=uv2)
\ : (r<=2) ? mt_makediff(clp,clp.medianblur(2,2*medf,2*medf),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.medianblur(3,3*medf,3*medf),U=uv2,V=uv2)
DD = mt_lutxy(RG11D,RG4D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",U=uv2,V=uv2)
clp.mt_makediff(DD,U=uv,V=uv)
return(last)
}

[fabioseixal]

Didée, thank you for looking up to this. I've tryed your script and unfortunately it didn't solved my problem.
I've tryed other denoisers to for the motion search like FFT3DFilter and dfttest and nothing happened, the problem still persists.

[Didée]

Sure, it does. I just posted the wrong script ( sup0=o.msuper(), where it shoud've been b.msuper() ).

Script corrected. Try again.

[fabioseixal]

Didée, as always you're a genius xD
Thank you very much, very good result here :)