Filtering old and dark anime

[AEtH]

Hi guys, I need some advice filtering this anime, the resulting banding and noise is really killing me, I don't now what to do now.

This is my script:

mpeg2source("video.d2v")

tfm()
tdecimate(mode=1)
crop(8,10,-8,-2)
LanczosResize(640,480)

ConvertToYUY2()
PeachSmoother(NoiseReduction=50)
ConvertToYV12()
BiFrost(interlaced=false)
deflicker()
RemoveGrain(mode=1)
UberSmooth(Preset = "AnimeLQ",NoWarnings=True)
l = gradfunkmirror(1.2)
c = gradfunkmirror(2.1)
mt_merge(l,c,c,y=2,chroma="copy second")
Soothe(LimitedSharpenFaster(),last,50)
FastLineDarken(strength=90,thinning=0)

Video source:
http://rapidshare.com/files/114487211/demuxed.m2v.html

Encoded video:
http://rapidshare.com/files/114487981/Fragment.mp4.html

Command line used for encoding:
--crf 20.0 --level 4.1 --ref 16 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --nf --subme 7 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads auto --thread-input --progress --no-psnr --no-ssim --output "output" "input"

[TheRyuu]

When working with a raw like this you have to keep in mind that you want to try and preserve what detail you have and not totally just blur().blur().blur() it. :)

You have to analyze the source and keep in mind what problems you are trying to fix. Pasted below is a quick something I just made up the past 10 minutes. Of course it can be tweaked more to the source. I wasn't sure how much to smooth it out so you can try playing with the sigma in dfttest, between 0.3-0.5, I wouldn't go about 0.5 let alone 0.4.

I also observed some horizontal combing on almost every frame so I threw in some vinverse to clean that up. Other then that and the rainbowing, nothing all that special about the clip. You could deen it to death, but there really is no need. DMFDerainbow didn't get rid of everything, but it got rid of most of it, try tweaking the settings a bit to try and clean it up more, you can also try mfderainbow, but I've found DMFDerainbow much more effective.

The script below is for a multi-threaded processer (quad core), you can change it or just totally get rid of it. Don't forget to change the idx parameters in the mvanalyze steps as well to unmulti-thread it.

Also for your x264 settings, if you are trying to get it to run on a stand alone player then use one of those profiles, if not, use an AE-xxxQuality profile, where xxx=max/standard/whatever megui gives you. Pick one to suit your speed. Please note that the quality difference between them is not that big, just some are a lot slower :P

Mpeg2Dec3_MPEG2Source("X:\Demuxed Dark Old Anime.d2v")
SetPlanarLegacyAlignment(true)

tfm(order=-1,field=-1,chroma=false)
tdecimate(mode=0)

MT("Vinverse(sstr=2.7)",4,8)

Crop(8, 4, -10, -0)
Spline36Resize(640,480)

colormatrix(mode="Rec.709->Rec.601",hints=false,interlaced=false,threads=4)
MT("DFMDeRainbow()",4,8)

raw=last

global idx1 = 10

MT("""
idx1 = idx1 + 1
source=last
vf1=source.mvanalyse(pel=2,blksize=8,isb=false,idx=idx1,overlap=4,sharp=2,truemotion=true)
vf2=source.mvanalyse(pel=2,blksize=8,isb=false,idx=idx1,delta=2,overlap=4,sharp=2,truemotion=true)
vb1=source.mvanalyse(pel=2,blksize=8,isb=true,idx=idx1,overlap=4,sharp=2,truemotion=true)
vb2=source.mvanalyse(pel=2,blksize=8,isb=true,idx=idx1,delta=2,overlap=4,sharp=2,truemotion=true)
interleave(mvcompensate(source,vf2,idx=idx1,thSCD1=800), \
mvcompensate(source,vf1,idx=idx1,thSCD1=800), \
source, mvcompensate(source,vb1,idx=idx1,thSCD1=800), \
mvcompensate(source,vb2,idx=idx1,thSCD1=800))
""",4,8) #I hope I used these slashes right, I never use them :s

dfttest(sigma=0.3,tbsize=5,opt=2,threads=4)

selectevery(5,2)

filtered=last
ContraSharpening(filtered,raw)

MT("
fastlinedarkenmod(thinning=0,strength=20).AA()
",4,8)
#try and clean up the bad lines and get rid of the jaggies on them. Not nessasary step, get rid of it if you don't like it.

[AEtH]

I used that script and it's not bad... but I'm looking for more sharpening without getting lots of banding.
Is there any way to do that??

[TheRyuu]

I used that script and it's not bad... but I'm looking for more sharpening without getting lots of banding.
Is there any way to do that??

More sharpening != better
You can throw LimitedSharpenFaster(strength=50) after the fastlinedarkenmod call, then throw AA after that.

But you'll probably get haloing. Not much you can do about that. It doesn't look like the source has hard edges so trying some unfilter before the whole script but might not be needed really.

Chances are, if you're new to encoding, you're oversharpening it. Chances are, if you think sharpening it more makes it looks good, it isn't ;)

[thetoof]

You can give this a try:
mpeg2source("demuxed.d2v")
#IVTC
tfm()
tdecimate(1)

#lacing and aliasing removal
vinverse()
aaa()

#heavy chroma denoise (also acts as a derainbow) that doesn't take cpu power
fft3dgpu(sigma=17, bw=32, bh=32, bt=4, plane=3, precision=2)

#Color correction that looks good to my eyes... change if you want
coloryuv(off_u=2)
tweak(cont=1.1,bright=5,sat=1.3)

#Chroma debleed
Mergechroma(chromashift(C=-2))

#Deband and deblock (ajust if you feel that you lose too many details)
gradfun2db(2.5)

#Add somewhat intelligent grain to dither
mt_average(AddGrainC(2),AddGrainC(2,seed=10))

#denoise
grainy=dfttest(smode=0,sbsize=5,tbsize=5,sigma=.8)
clean=dfttest(smode=0,sbsize=5,tbsize=5,sigma=3)

#Stabilize that grain even more to have some nice dithering that'll be kept by x264
dull=grainoptimizer(grainy,clean,tdist=99999)

#Sharpen
sharp=dull.limitedsharpenfaster(2,2,strength=150)
soothe(sharp,dull)

#Remove the overscan
crop(6,4,-8,-0)

#Resize to mod16
spline36resize(704,480)

The cropping process altered the aspect ratio, so you have two options:
1 - encode anamorphic by using the settings I gave you and specifying a Display Aspect Ratio of 1.3183940242763772175536881419234 in mkvmerge
2 - use crop(4,6,-8,-2) and spline36resize(640,480)
I prefer #1 because you have optimal cropping settings and you'll have a better encode than a downsized one.

Also, I recommend doing an uncompressed pass before throwing it into megui, because this is slooooooooooow.

[AEtH]

You can give this a try:

killer script!!!!

That solve almost all the problems I've had handling this video.
Thank you so much!..

PD. Anyway.... it really is slooooooooooooooooow (but it's not a matter of time, of course)

Thanks again

[thetoof]

I rechecked the script a little while after and here are some improvements you could do:

-Ajust gradfun2db... the settings I gave you screwed up your black sky.
-Use this instead of the first dfttest:FFT3DFilter(sigma2=x, sigma3=y,sigma4=z,bh=32,bw=32,ow=16,oh=16,plane=4,bt=5)
Where xyz are values you ajust to keep the type of grain you want.
-Use a higher strength for LFS if you want more sharpening

[Blue_MiSfit]

Lagarith rendering pass FTW!!!

~MiSfit

[thetoof]

hahaha yeah, but if he has enough HDD space, I'd even say "go for uncompressed YV12" to reduce the load on the CPU (not that Lagarith consumes a LOT of cpu power, but with a script like that... :p)

[AEtH]

Maybe I am doing something wrong, but using this:

grainy=fft3dfilter(sigma2=3, sigma3=6, sigma4=9,bh=32,bw=32,ow=16,oh=16,plane=4,bt=5)

instead of this:

grainy=dfttest(smode=0,sbsize=5,tbsize=5,sigma=.8)

give me worst results.


PD. Anyway, using gradfun2db(1.5) is better than gradfun2db(2.5), so I 'm combining this one change with dfttest in the grainy pass... results are awesome.

[thetoof]

Your settings are way too high! They are similar to what I used to clean this (http://img329.imageshack.us/img329/4344/origiy7.jpg).
I'll give you some "optimal" settings in a few days (need to go out of town for a while).

[AEtH]

Thanks, it will be so helpful. Anyway I'll be testing some other configurations to see if I can achieve better results.

[TheRyuu]

hahaha yeah, but if he has enough HDD space, I'd even say "go for uncompressed YV12" to reduce the load on the CPU (not that Lagarith consumes a LOT of cpu power, but with a script like that... :p)

Just use huffy that comes with the ffdshow vfw.
Use YV12 and check adaptive huffman tables. Get compression that rivals lags but at a lower cpu usage and also, frames will never get corrupted ;)
Also, make sure to change the fourcc to ffvh.

[AEtH]

Somehow I managed to achieve better results with fft3dfilter. I used these settings:

grainy=fft3dfilter(sigma=0.3,sigma2=0.5,sigma3=0.6,sigma4=0.3,bh=32,bw=32,ow=16,oh=16,plane=4,bt=5)

[thetoof]

I still haven't had the time to run some tests, but what you came up with seems good to me.

[AEtH]

So, after all, maybe my choice is not bad at all :D
Anyway, if you (or anybody) can exactly explain how these sigma values work (band-pass filter??) it will be so helpful. I tried to find some documentation but without luck.

PS. Also... if you run some test will still be so nice!

[MADAJ]

@thetoof

thx for your avs script, but where can I find aaa() plugin?

[Didée]

Somewhere on this forum ... and it's a script function, not a plugin.

Fastest hit - on Avisynth's Wiki: http://avisynth.org/mediawiki/AAA

[AEtH]

Any chance on getting some info about sigma2, sigma3 and sigma4?

[Dark Shikari]

Its in the fft3dfilter documentation. Sigma4=sigma for lowest frequency, sigma2=for second highest (highest just uses sigma). Default is sigma4=sigma3=sigma2=sigma.

[AEtH]

Well... I did read than in the documentantion. Maybe I'm not explaining well, I'd like to know hoy to apply these values. For example, by running some test I achieved a values of sigma=0.3,sigma2=0.5,sigma3=0.6,sigma4=0.3. What is the behavior of these values?? If sigma=0.3 is the highest frequency... what is the meaning of using sigma3=0.6? It does not make sense.
It could be so useful the values needed to filter all, retain some tipe of grain, something like this.
Thanks

[Dark Shikari]

Well... I did read than in the documentantion. Maybe I'm not explaining well, I'd like to know hoy to apply these values. For example, by running some test I achieved a values of sigma=0.3,sigma2=0.5,sigma3=0.6,sigma4=0.3. What is the behavior of these values?? If sigma=0.3 is the highest frequency... what is the meaning of using sigma3=0.6? It does not make sense.
It could be so useful the values needed to filter all, retain some tipe of grain, something like this.
ThanksIf sigma=0.3 and sigma3=0.6, it means you're filtering the frequencies in the range of sigma3 more strongly than the frequencies in the range of sigma.

[Adub]

The sigma(2,3,4) options filter noise from highest to lowest. If you have high frequency noise, then you deal with sigma and maybe sigma2 only. If you have lower frequency noise then you start to deal with sigma2 and sigma3. I never deal with sigma4, but sometimes you need it for low frequency noise removal. Make sense?

[AEtH]

Ok, thanks a lot, this makes sense. After these explanations I'm really to play with fft3dfilter a lot!! :D

[Adub]

Yeah, it is a very powerful filter. I use it on a lot of sources, and because it's very tunable, I love it to death. I do need to get some practice with some other filters however, as I am becoming dependent on this one to much. Bad habit. But yeah, great filter. Personally I love the GPU version however. All of that denoising at very low speed hit.