anime filtering

[Kumo]

i'm trying to encode kakurenbo anime,from dvd r-1 to x264.here there are some screens from untouched video:

http://img292.imageshack.us/img292/7411/83670482om7.png
http://img337.imageshack.us/img337/6434/34546098tg1.png
http://img530.imageshack.us/img530/5479/91131624uz2.png
http://img179.imageshack.us/img179/7956/18023993qy4.png
http://img525.imageshack.us/img525/2286/55530925oe6.png
http://img257.imageshack.us/img257/300/99171922nm7.png

problems are mpeg2's micro/macro-bloccks ,consistent noise,unbrilliant colors, and most of all unclear edges.
what kind of filters may i try??

[foxyshadis]

macroblocks: mpeg2source's cpu param, or deblock_qed
dot crawl: tcomb or fft3dfilter
noise: fft3dfilter (or fft3dgpu), frfun7, mvdegrain, fluxsmooth, lots of other choices
color/contrast: HDRAGC or just levels/tweak
edges: seesaw, maybe limitedsharpen or warpsharp

Be careful with filtering, because many denoising methods will futher damage edges. And that's actually very clean video, compared to a lot of anime out there, don't go wild on the denoising unless you have to remove detail to fit to some size. (All these newer shows just beg for HD. ;_; )

[SuperDuperDave]

I like VMToon for sharpening. It doesn't sharpen quite as well as LimitedSharpen, but it still looks great, and it can actually improve compression ratio.

[TheRyuu]

I like VMToon for sharpening. It doesn't sharpen quite as well as LimitedSharpen, but it still looks great, and it can actually improve compression ratio.

Have you tried LimitedSharpenFaster+FastLineDarkenMOD?

Great combo, at least I think so.

I normally run them one after another like so, denoisers are before (fft3d, vaguedenoiser, degrainmedian, removegrain, frfun3d, depending on the noise), and color tweaks (if needed) are after:
LimitedSharpenFaster(ss_x=1.5, ss_y=1.5, smode=3, strength=125, overshoot=1, undershoot=1) #I deliberatly used smode=3, I think it works better on anime, change it to smode=4 if you like
FastLineDarkenMOD(thinning=0, strength=50) #use thinning if you want though, I think it's optional

This produces something similar to vmtoon only a lot faster. It only slightly darkens the lines which is what I like, but feel free to adjust the parameters.
Furthermore, after a few tests, it is better to run fastlinedarken after LSF since if you run it before, it will produce line crawling or whatever you call it.

To each his own though and if you like vmtoon then so be it :)
I just like that fastlinedarken is well, faster :)

##############################
# FastLineDarken 1.3x MT MOD #
##############################
#
# Written by Vectrangle
# Didée: - Speed Boost, Updated: 11th May 2007
#
# * requires mt_masktools v2
# * requires yv12 input
#
# Usage is FastLineDarkenMOD(strength, luma_cap, threshold, thinning),
# named parameters are supported eg FastLineDarken(thinning=0)
#
# Note that you must import this avs into your script using import("...\FastLineDarken.avs")
#
# Parameters are:
# strength (integer) - Line darkening amount, 0-256. Default 48. Represents the _maximum_ amount
# that the luma will be reduced by, weaker lines will be reduced by
# proportionately less.
# luma_cap (integer) - value from 0 (black) to 255 (white), used to stop the darkening
# determination from being 'blinded' by bright pixels, and to stop grey
# lines on white backgrounds being darkened. Any pixels brighter than
# luma_cap are treated as only being as bright as luma_cap. Lowering
# luma_cap tends to reduce line darkening. 255 disables capping. Default 191.
# threshold (integer) - any pixels that were going to be darkened by an amount less than
# threshold will not be touched. setting this to 0 will disable it, setting
# it to 4 (default) is recommended, since often a lot of random pixels are
# marked for very slight darkening and a threshold of about 4 should fix
# them. Note if you set threshold too high, some lines will not be darkened
# thinning (integer) - optional line thinning amount, 0-256. Setting this to 0 will disable it,
# which is gives a _big_ speed increase. Note that thinning the lines will
# inherently darken the remaining pixels in each line a little. Default 24.
#
# Changelog:
# 1.3 - added ability to thin lines, now runs much slower unless thinning=0. Changed the defaults (again)
# 1.2 - huge speed increase using yv12lutxy =)
# - weird darkening issues gone (they were caused by yv12layer)
# - show option no longer available due to optimizations. Use subtract() instead
# 1.1 - added luma_cap option
# 1.0 - initial release
#


function FastLineDarkenMOD( clip c, int "strength", int "luma_cap", int "threshold", int "thinning")
{
## parameters ##
str = string(default(strength, 48) /128.)
lum = string(default(luma_cap, 191))
thr = string(default(threshold, 4))
thinning = default(thinning,24)
thn = string(thinning /16.)

## filtering ##
exin = c.mt_expand().mt_inpand()
diff = mt_lutxy(c,exin,yexpr="y "+lum+" < y "+lum+" ? x "+thr+" + > x y "+lum+" < y "+lum+" ? - 0 ? 127 +",uexpr="x",vexpr="x",u=2, v=2)
linemask= mt_lut(diff.mt_inpand(),"x 127 - "+thn+" * 255 +").RemoveGrain(20,-1)
thick = mt_lutxy(c, exin, yexpr="y "+lum+" < y "+lum+" ? x "+thr+" + > x y "+lum+" < y "+lum+" ? - 0 ? "+str+" * x +",uexpr="x",vexpr="x",u=2, v=2)
thin = mt_lutxy(c.mt_expand(),diff,yexpr="x y 127 - "+str+" 1 + * +",u=2, v=2)

return (thinning == 0) ? thick : mt_merge(thin,thick,linemask,y=3,u=2,v=2)
}

[Kumo]

thank you so much, i'll try a bit, than i'll post some results.

[ChaosKing]

Why i get a green picture if i use fastlinedarkenmod( thinning = 32 ). Or can i only use Fastlinedarkenmod with thinning=0 ?

i used this version of mt_masktools: 2.0.30.0

[TheRyuu]

Why i get a green picture if i use fastlinedarkenmod( thinning = 32 ). Or can i only use Fastlinedarkenmod with thinning=0 ?

i used this version of mt_masktools: 2.0.30.0

I get that too for some reason so I just disable thinning since it fixes the problem. And I think as long as the strength is low enough, you probably don't need thinning.