help with terrible source (dl sample included)

[grumpygamer]

hello,

I am trying to recover a very old source that I have.
I am sure it can come out greatly improved if only I knew what filters I need applied.
Mc_spuds works wonders but it is too slow and also it seems to have some kind of memory leakage, let alone mvtoon will not work after it for some strange reason and crashes virtualdub (where I'm testing the results)

http://img382.imageshack.us/img382/1020/snapshot20080519153619en2.png

I would love to get a very clean and LIKE-NEW output at a reasonable speed (let's say starting from 3fps).
I have a core2duo e6600 with 2gb ram
the source is PAL as you might see

My script for now is:

mpeg2source("xxxxx.d2v",info=3)
colormatrix(interlaced=false,hints=true)
fft3dfilter(sigma=3,sigma2=3,sigma3=12,sigma4=3,bt=3,plane=1)
fft3dfilter(sigma=3,sigma2=3,sigma3=12,sigma4=3,bt=3,plane=2)
tfm(d2v="xxxxx.d2v")
FixChromaBleeding()
mc_spuds(frames=3,strength=4)
FastLineDarkenMOD(thinning=0,strength=120)
dehalo_alpha()
dfttest(sigma=1)
dfttest(sigma=1)
deblock()
awarpsharp(8.0)

LimitedSharpenFaster(ss_x=1.25,ss_y=1.25,Smode=4,strength=150,soft=30)
crop( 12, 12, -20, -12)

but still the results are not what i'd like.
I know dfttest twice is dumb but the source is REALLY nosiy!
Anyway no pointing fingers to my noobness, I know already...
just see if you can help me out (if you will),
thanks a lot
-gg

here are the sample chunks:

http://rapidshare.com/files/115964231/chunkA.m2v.html

http://rapidshare.com/files/115966209/chunkB.m2v.html

[Revgen]

The best results I've had with noisy cartoons is using Neuron2's msooth filter in addition to fft3dfitler.

Msmooth is here. http://www.neuron2.net/msmooth/msmooth.html

The problem with Msmooth is that it can smear too much or do too little, so you might want to separate your source by different scenes and apply the settings for each scene.

You may even have to use a sharpener and line darkener afterward. I haven't used this method in 3 years so I don't know if there are newer filters that can do this better.

[grumpygamer]

Thanks for the tip but I was hoping in something more specific since I already spent countless hours trying loads of filters and various combinations; anyway I'll keep your suggestion in mind.
-gg

[thetoof]

I've produced a script almost right after your first post, but I was interrupted by something. I'll post it when it's done.

[grumpygamer]

thanks, i can't wait to see the result although I'll have to try it tomorrow morning!

[grumpygamer]

... but I was interrupted by something. I'll post it when it's done.

must be something VERY LONG :D
anyway I am very patient ...

[thetoof]

Hehehe
Actually, the "something" was replaced by "many many things" and they are still ongoing. I'm still steadily posting on the forum because it doesn't take a lot of time to write a few lines and it helps preventing a burn-out from overwork :p

I can see by the huge amount of time you waited that you are VERY patient :rolleyes: just kiddin' ;)

Anyways, here's what I had the time to do in 5 mins after seeing your post. It's not optimized and some other filters may be needed, but I'll do all that tweaking later.
mpeg2source("chunkA.d2v")
#I see progressive movement on every frame, so I guessed this is a typical "film to PAL speedup"
assumefps(24)

#BSC improvement + color correction
tweak(cont=0.8,bright=5,sat=1.6)
coloryuv(off_v=2,off_u=6)

#Anti-aliasing to ensure that your dark lines will look good, even on a big screen (I have a 24", so if you don't see any difference, know that I do :p) You can still remove it to speed up.
aaa()

#Stabilizer (thanks to g-force for the script)
orig=last
temp=orig.TemporalSoften(7,255,255,25,2)
rep=Repair(temp,orig,mode=16).TemporalSoften(1,255,255,25,2)
source=Interleave(rep,orig,rep)
mdata=DePanEstimate(source, range=1, trust=0, dxmax=1, dymax=1)
DePan(source, data=mdata, offset=1)
SelectEvery(3,2)

#Denoise
dfttest(smode=0,sbsize=5,tbsize=5,sigma=3)

#Derainbow
fft3dgpu(sigma=7, bw=32, bh=32, bt=4, plane=3, precision=2)

#Attempt to reduce the haloing (this will change for sure in my next script)
awarpsharp(8.5)

#Crop and resize without screwing up the AR
crop(16,16,-24,-16)
spline36resize(640,480)

#Some other things are needed... maybe sharpening, grain optimization and other stuff
It's slower than 3fps (like most of my scripts for messed up sources). For better speed, split it in many parts (run a lossless pass of a part, reload the resulting clip, apply the other filters, etc etc). A good place to do that would be right after the stabilizer, or even before and after it.

What is your target codec for this encode?

edit: if you have any questions that don't imply testing, it'll be a pleasure to answer them

[grumpygamer]

hi toof, thanks for the script, very cool indeed.
I will split the code in half as you suggested and do it in 2/3 rounds

i want to encode in x264 and with no frame resize.
Anyway your script is really good as I expected and I'll probably pair it up with spuds or some more processing in the end.
If you have any other suggestions I might use to get the clip smoother please let me know.

A couple of questions: i have read somewhere fft3dgpu goes before the deinterlacing, which i don't see here.
So
1. Do you deinterlace?
2. If so, where?
3. If you do shouldnt fft3dgpu go just after that
4. I wanted to keep the same framerate thus keeping assumeFPS(25) is that ok?
Thanks a lot for your time
-gg

edit:
The dfttest is WAAY slow, gets me to 1fps on my dualcore, is there any way to speed up that particular process?

[thetoof]

If you have any other suggestions I might use to get the clip smoother please let me know.
If by smoother you mean without grain, use a higher sigma with dfttest. It will introduce banding.
1. Do you deinterlace?
From the script.... #I see progressive movement on every frame, so I guessed this is a typical "film to PAL speedup"
I wanted to keep the same framerate thus keeping assumeFPS(25) is that ok?
If I am right and this really is a PAL speedup, going back to 24fps means running the cartoon at the speed it should be.
The dfttest is WAAY slow That's just how it is... use a cheaper denoiser if you want faster processing.

[grumpygamer]

by smoother I mean newer...
I would like to try and restore the video to make it look brand new
let's say like an old anime modern encode...
-gg

[thetoof]

Hmmm... could you define "newer"? "modern encode" and "brand new" are not helping me understand what you want...

Could you describe what characterises such an encode for you by giving technical terms (like sharper, vivid colors...) or describing some visual characteristics of something "new". Or, even better, post a screenshot of what you define as a "new anime".

[grumpygamer]

hey toof, thanks for the time you're taking to help me get this done.

With newer I mean, less noise, cleaner and stronger lines, the right amount of graininess
no shifted chroma, no oversaturation...
this is what I can think of.

Will you help me complete the script?
Thanks a lot
-gg

[thetoof]

Dang, didn't think I'd take so long to post again... Anyways, here's a better script that should meet your demands. It needs some tweaking so you can get exactly what you want, but this should be the main core.

mpeg2source("chunkA.d2v")
#I see progressive movement on every frame, so I guess this is a typical "film to PAL speedup"
assumefps(24)

#Anti-aliasing - can be called as maa() if you have animeivtc.avsi in your plugins folder
input=last
mmask = input.mt_edge("sobel",7,7,5,5).mt_inflate()
aa_clip=input.spline36Resize(width(input)*2,height(input)*2).TurnLeft().SangNom().TurnRight().SangNom().spline36Resize(width(input),height(input)).MergeChroma(input)
mt_merge(input,aa_clip,mmask)

#Heavy chroma denoise - reduce sigma if you see artifacts
fft3dgpu(sigma=17, bw=32, bh=32, bt=4, plane=3, precision=2)

#Luma denoise - ajust sigma to keep the amount of grain you want
dfttest(true,false,false,smode=0,sbsize=5,tbsize=5,sigma=3)

#Chroma debleed - ajust to get the right amount of warp sharpening (higher = less bleeding, but may create artifacts) - you can also try the other arguments
mergechroma(awarpsharp(16))

#If you want thinner lines, uncomment the next line and ajust to your needs
#awarpsharp()

#Stabilizer
orig=last
temp=orig.TemporalSoften(7,255,255,25,2)
rep=Repair(temp,orig,mode=16).TemporalSoften(1,255,255,25,2)
source=Interleave(rep,orig,rep)
mdata=DePanEstimate(source, range=1, trust=0, dxmax=1, dymax=1)
DePan(source, data=mdata, offset=1)
SelectEvery(3,2)

#BSC improvement - ajust if you don't like it
Tweak(cont=0.8,bright=5,sat=1.25)

#Color correction
coloryuv(off_v=1.5,off_u=1.25)

#Sharpening... 2 options - use ONE of them
#1 - Global sharpening - ajust strength to your needs
soothe(limitedsharpenfaster(strength=150),last)
#2 - Edge sharpening - ajust strength to your needs
#toon(.5)

#Crop the overscan
crop(12,10,-22,-10)

#Resize to mod16 for encoding
spline36resize(688,560)
Not cropping the overscan would be a waste of bitrate and the resize is only to get to mod16 for substantial compressibility improvement and minimal image modification.
Since your encoding to x264, you'll need to specify an Aspect Ratio of 1.3160671462829736211031175059952 in mkvmerge

The math :
720x576 becomes 768x576 (4/3)
720 / 0,9375 = 768
So, after cropping, the display width should be
(720 - 12 - 22) / 0,9375 = 731.73333333333
and height is
576 - 10 - 10 = 556

So, the correct aspect ratio after removing the overscan is
731.73333333333 / 556 = 1.3160671462829736211031175059952

And, to get to mod16, you take the values after cropping and find the closest
686 / 16 = 42.875
Closest = 43
So mod16 width = 43 * 16 = 688
For the height:
556 / 16 = 34.75
Closest = 35
35 * 16 = 560
If you need anything else, feel free to ask!

[radar]

@grumpygamer
could you post the same clip after using thetoof script.thanks

@thetoof
could your script be used for non-animi.(like old fight tapes)

[thetoof]

If you have the exact same issues with it... yeah ;)

[grumpygamer]

wow that's a lot to handle!
As usual i am speechless, nice!
I'll let you know how it goes
thanks again, really appreciate the time you took
-gg

[thetoof]

np
and this is the screenshot radar requested (second awarpsharp is enabled, but not optimized)
http://img398.imageshack.us/img398/9687/thetoof2smalldb1.jpg