Anime encoding frustration with artifacts

[nixius]

Hello,

I have done alot of searching not only in this forum but around other websites and forums as well as google. I have posted once or twice as well. I have followed quite a few guides and tried many different combinations off my own back of things in AviSynth and VirtualDub.

I am trying to encode from some ripped VOB files from an Anime DVD, and no matter what techniques I employ I am always left frustrated with small to moderate artifacts and/or noise. The source itself is of good quality.

I also find sometimes that I apply filters, or use certain techniques in VirtualDub, and the input window looks better than the output window, which confuses me :confused:. But that is alittle off topic sorry.

I have tried the XVid guide on this website with GK and VD. I have modified the script that is created for you in GK with many adaptations, as well as adding in things I have tried from google searching. I also have tried a few VD filters, combined with AVIsynth scripts and I now have a huge headache (also pelase note that I haven't just tried throwing everything at it, I have tried to be smart with the combinations I have used).

I am encoding with XViD.

So this leads me to asking a large community such as this one, what kind of techniques do you employ to combat problems that are artifacts and noise. If anyone has any advice, avisynth script tips or simple links I would be very grateful.(just pointing me in the right direction is fine, I'm more than willing to put the effort in).

Thanks for your time,
Adam

[Adub]

First of all, post your script.
Second post a sample for us to play with.
Third post your encoding settings.

[nixius]

## Attempt1
#Telecide(guide=0,order=1,post=2,blend=true,vthresh=1)
#UnComb()

##Attempt2
DGDecode_mpeg2source("I:\DVD_RIPPING\DNAngel\ep2\Working DIrectory\VTS_02_1.d2v",info=3)
ColorMatrix(hints=true,interlaced=true)
AssumeTFF().Telecide(guide=1).Decimate(mode=2,threshold=2.0)

#### GK STUFF ####

# IVTC
#Telecide(order=1,guide=1).Decimate()

# DEINTERLACING (1)
#FieldDeinterlace()
#FieldDeinterlace(blend=false)
#TomsMoComp(1,5,1)

# DEINTERLACING (2)
#KernelDeInt(order=1,sharp=true)


----

I played alot with IVTC and DEINTERLACING (1), these are parts of the script that Gk auto-created, I deleted the things I didn't play with myself. Attempt2 gas given me the best results thus far.

I know it looks a little mish-mash, so I'm sorry for that. Just using FieldDeinterlace() or Telecide(order=1,guide=1).Decimate() by far gave the worst results when used on their own, it got better when using it with UnDot() but not as good as Attempt1 or Attempt2.

Here are some pics, if you need a video sample to play with then I'll try and sort something out for you if thats what you meant.

Attempt01 Pic
http://www.jpopulous.org/attempt1_pic01.png

Attempt02 Pics
http://www.jpopulous.org/attempt2_pic01.png
http://www.jpopulous.org/attempt2_pic02.png

[Edit] Attempt02 pics are zoomed in, there are a few worse cases than this in both, but this is the kind of subtle-ish noise/artifacts that occur throughout.


--

My XVid settings, I'm using the AS@L5 profile and I'm using the target size of ~200MB (for a ~22minute episode) with 2 passes. Everything else I left at default except I changed FourCC used to XVid and turned Cartoon Mode on.

I would have thought though, that my XVid settings wouldn't really be relevant because when I open the video in VDub I see the artifcats and noise before I encode.

Thanks,
Adam


[Edit:] took out some typos ^^;

[Guest]

Those artifacts are dot crawl. You need to get a dot crawl removal filter, aka comb filter.

[nixius]

Hmm, so far I have tried Guava, DeDot and TComb and after fiddling with the settings I get some improvement but not quiet as much as I would have liked. The main problem now is the outside of lines when there is movement, it is not very noticeable on the picture but it is more so in the video.

http://www.jpopulous.org/new.png

There is still some 'crawling' inside the hair for example, but it isn't as bad.

Do you know of any other dot crawl filters that might be worth trying out?

Here are the settings I found to be best

GuavaComb(mode = "NTSC", recall=80, maxVariation=25, activation=30)
TComb(2, 5,6,7,5, false,12,false,4)

And also of note when I was trying this I used:
Telecide(guide=0,order=1,post=2,blend=true,vthresh=1)

I tried the combing before and after Telecide and it seemed to work best after Telecide for TComb but didn't make much diference for Guava or DeDot.

[foxyshadis]

fft3dfilter or fft3dgpu is very effective in situations when a dedicated crosstalk removal isn't sufficient. You can use it alone, or combined with the dedicated with lower sigma settings.

fft3dfilter(sigma=.5,sigma2=3,sigma3=3,sigma4=.5,plane=4) would be a good place to start, I can't remember which sigma ranges dot crawl falls into.

[nixius]

Okay, I finally got around to using ff3dfilter and it worked very nicely. Your settings seemed to work fine when I tried them so I haven't changed them :), but I may do so if I have any further problems.

Thanks alot for your help and patience, it is very much appreciated!

[Warpman]

also disable cartoon mode, cause anime =/= cartoon

[TheRyuu]

mpeg2source("file.d2v",info=3)
tfm(d2v="file.d2v").tdecimate()
colormatrix(hints=true)

crop(8,0,-8,0)
Spline36Resize(640,480)
removegrain(mode=1)

o = last
Assumeframebased().AssumeTFF().SeparateFields()
dbl = mt_Average( SelectEven().EEDI2(field=1,maxd=4),
\ SelectOdd() .EEDI2(field=0,maxd=4), U=3,V=3 )
dblD = mt_MakeDiff(o,dbl,U=3,V=3)
shrpD = mt_MakeDiff(dbl,dbl.RemoveGrain(11),U=3,V=3)
DD = shrpD.Repair(dblD,13)

dbl.mt_AddDiff(DD,U=3,V=3)

removegrain(mode=2)
deen("a2d",1,7,9) #lets kill some trees :D
fft3dfilter(sigma=3,sharpen=0.5)

dull=last
sharp=dull.LimitedSharpenFaster(smode=4,strength=150)
Soothe(sharp,dull,30)
fastlinedarkenmod(strength=35,thinning=20)

awarpsharp(depth=8,cm=1)

antialiasing(hqedge=true)


Deen will help to get rid of your noise, but might kill some trees in the process (i.e. destroy too much detail).

Undot isn't going to do much. All it does is get rid of stray pixels (removegrain(mode=1) does the same thing only faster if you use the sse versions, plus it doesn't have a bug I think undot had).

Sharpening is optional, but I'm sure it would make the image look a lot better.
The script above is built based on one of your png images and is something I wrote up quickly. I'm sure it can be tweaked a lot better to destroy less details and still get rid of the noise.

Also, you can try a filter called "Dedot" and apply it before deinterlacing to see if that helps at all. I've personally never tried it.

[nixius]

Thankyou very much for your replies, I have been busy with some school work as of late so I haven't had a chance to test any of this. I will get back to you as soon as I can.

Again I appreciate the help.