Getting terrible results encoding anime.

[Perscitus]

It has been almost 2 years since I was encoding video and I'm trying to make a backup of an anime. I have spent a good amount of time scouring the forums for info and I'm getting awful results. So I'm looking for any suggestions of what I'm doing wrong. My goal is to create a high quality x264 encode of an anime with vfr and I'm trying to fit it to a 175MB(or 233MB) file size.

What I'm using: MeGui 0.3.0.1020, AviSynth 2.57, x264 (928 Jarod's patched build), DGIndex 1.5.0.0, nnedi 1.3.0.0, TIVTC 1.0.5.0, TDeint 1.1.0.0, TFM 1.0.4, MKVMerge 2.2.0

......1st Pass:
AviSynthLoadPlugin("C:\Apps\Encoding Tools\AviSynth 2.5\plugins\TIVTC.dll")

d2vpath = "C:\DVD\Working\test01.d2v"

DGDecode_mpeg2source(d2vpath,info=3)
tfm(d2v=d2vpath,output="c:\DVD\Working\test01-tfm.txt")
tdecimate(mode=4,output="c:\DVD\Working\test01-tdec.txt")


crop( 8, 8, -8, 0)
nnedi(field=1,dh=true).TurnRight().nnedi(field=1,dh=true).TurnLeft()
Lanczos4Resize(640,352)

x264 program --pass 1 --bitrate 905 --stats "C:\DVD\Working\test01.stats" --bframes 5 --b-pyramid --direct auto --filter 1,1 --subme 1 --partitions none --me dia --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output NUL "input" --psy-rd 0

......2nd Pass:
AvisynthLoadPlugin("C:\Apps\Encoding Tools\AviSynth 2.5\plugins\TIVTC.dll")

d2vpath = "C:\DVD\Working\test01.d2v"

DGDecode_mpeg2source(d2vpath,info=3)
tfm(d2v=d2vpath,input="C:\DVD\Working\test01-tfm.txt")
tdecimate(mode=5,hybrid=2,vfrDec=0,input="C:\DVD\Working\test01-tdec.txt",tfmIn="C:\DVD\Working\test01-tfm.txt",mkvOut="C:\DVD\Working\test01-timecodes.txt",batch=true)

crop( 8, 8, -8, 0)
nnedi(field=1,dh=true).TurnRight().nnedi(field=1,dh=true).TurnLeft()
Lanczos4Resize(640,352)

x264program --pass 2 --bitrate 905 --stats "C:\DVD\Working\test01.stats" --ref 16 --mixed-refs --no-fast-pskip --bframes 5 --b-pyramid --b-rdo --bime --direct auto --filter 1,1 --subme 6 --trellis 2 --partitions none --me umh --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "output" "input" --psy-rd 0

My Result: download here (http://wesker.home.comcast.net/test01p2-perscitus.mkv)
My Source: I'm not sure if it violates rules to link the VOB of the source, I will link it(or not) as soon as I find this out.

[DarkZell666]

Any particular reason you used --partitions none ? :eek:
Using that certainly doesn't help, particularly on the second pass. Use the default setting instead.

Also, why use vfr for output ? That's really looking for extra trouble since static frames barely take up any space at all (x264 marks the static parts of the picture as "skip" and doesn't even encode them). You can easily fit a clean 25min anime episode unresized (720x576 anamorphic, or 720x480 anamorphic) on 175MB and still get acceptable quality (though what's "acceptable" is very subjective).

When I say "clean", I mean correctly denoised. Try adding hqdn3d(2,2,2) just before the resizing step to your script (you'll need the hqdn3d plugin), since DVD's all have noise, even anime ones (and the video you posted is full of noise). If that isn't enough, try hqdn3d() on default settings or fft3dfilter() on default settings instead (much slower).

(I'll let you google the pages for those 2 filters, they're easy to find :p).

Also, what you posted isn't as awful as I first imagined. Also, why disable psy-rd ? Was it worse with it enabled ?

[Atak_Snajpera]

You know what? Try again with RipBot264. Enable Denoiser (ANIME profile) and everything should be ok.

[cogman]

Also, it looks like your settings for first pass and second pass are almost too different. For example. Im pretty sure that Psy-rdo should be enabled on both passes.

Upping the bframes to 16 should help somewhat as well (since this is anime). an really, why aren't you using all partitions?

You can try the "Dup" plugin for avi-synth as well, it should offer some savings to you.

[Adub]

Nah, he is disabling Psy-rdo on both passes. Some tests have shown that it doesn't help with anime as it does with live action material.

[Perscitus]

I took some of the suggestions and made a few changes, definite improvement. I don't know how I managed to miss "--partitions none", that shouldn't have happened. Anyway, here are the updated settings and sample.

......1st Pass:
AviSynthLoadPlugin("C:\Apps\Encoding Tools\AviSynth 2.5\plugins\TIVTC.dll")
LoadPlugin("C:\Apps\Encoding Tools\AviSynth 2.5\plugins\nnedi.dll")
LoadPlugin("C:\Apps\Encoding Tools\AviSynth 2.5\plugins\hqdn3d.dll")

d2vpath = "C:\DVD\Working\test01.d2v"

DGDecode_mpeg2source(d2vpath,info=3)
tfm(d2v=d2vpath,output="c:\DVD\Working\test01-tfm.txt")
tdecimate(mode=4,output="c:\DVD\Working\test01-tdec.txt")


crop( 8, 8, -8, 0)
nnedi(field=1,dh=true).TurnRight().nnedi(field=1,dh=true).TurnLeft()
hqdn3d(2,2,2)
Lanczos4Resize(640,352)

x264program --pass 1 --bitrate 905 --stats "C:\DVD\Working\test.stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter 1,1 --subme 6 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output NUL "input"


......2nd Pass:
AviSynthLoadPlugin("C:\Apps\Encoding Tools\AviSynth 2.5\plugins\TIVTC.dll")
LoadPlugin("C:\Apps\Encoding Tools\AviSynth 2.5\plugins\nnedi.dll")
LoadPlugin("C:\Apps\Encoding Tools\AviSynth 2.5\plugins\hqdn3d.dll")

d2vpath = "C:\DVD\Working\test01.d2v"

DGDecode_mpeg2source(d2vpath,info=3)
tfm(d2v=d2vpath,input="C:\DVD\Working\test01-tfm.txt")
tdecimate(mode=5,hybrid=2,vfrDec=0,input="C:\DVD\Working\test01-tdec.txt",tfmIn="C:\DVD\Working\test01-tfm.txt",mkvOut="C:\DVD\Working\test01-timecodes.txt",batch=true)

crop( 8, 8, -8, 0)
nnedi(field=1,dh=true).TurnRight().nnedi(field=1,dh=true).TurnLeft()
hqdn3d(2,2,2)
Lanczos4Resize(640,352)

x264program --pass 2 --bitrate 905 --stats "C:\DVD\Working\test.stats" --ref 5 --mixed-refs --no-fast-pskip --bframes 16 --b-pyramid --b-rdo --bime --weightb --direct auto --filter 1,1 --subme 6 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "output" "input"

Result: download here (http://wesker.home.comcast.net/test02p2-perscitus.mkv)
The most noticeable issue I still have is apparent in the solid color areas, primarily the dark ones. Am I getting too picky or is there a filter that would fix that?

[cogman]

I may be wrong here. But the problem may stem from the AQ settings. Try relaxing the strength a bit.

[Sharktooth]

it may be.
try disabling AQ reducing the strenght to 0. see if the situation gets better and if it does then try to find the sweetspot rising the AQ strenght until you get the desired results.

[desta]

Also, it looks like your settings for first pass and second pass are almost too different. For example. Im pretty sure that Psy-rdo should be enabled on both passes.
I was led to believe that psy-rdo wasn't required for the first pass.

[Sharktooth]

infact it's not.

[Shinigami-Sama]

infact it's not.
+1
useless on first pass

[Comatose]

So does it do anything or does it get disabled?

[Dark Shikari]

So does it do anything or does it get disabled?Its on if you use subme=6/etc... it is potentially useful, but only if you have time to waste.

[foxyshadis]

You know that with VFR, the first (mode 4) and second (mode 5) pass of the avisynth have to be run first, then the first and second encoding pass? To streamline it, you can run the first x264 pass over both avisynth passes, overwriting the first's stats, then run the second x264 pass. If you run the first pass on the first avisynth and the second on the other, you will be destroying rate control, scene-cut detection, and probably getting wildly undersized files as well.

Make sure that bitrate is also based on the 2nd pass avisynth's length, not the original's. They'll be different since you didn't change the FPS.

[Comatose]

Its on if you use subme=6/etc... it is potentially useful, but only if you have time to waste.
Time to do some testing, then! :p
:thanks:

[Perscitus]

You know that with VFR, the first (mode 4) and second (mode 5) pass of the avisynth have to be run first, then the first and second encoding pass? To streamline it, you can run the first x264 pass over both avisynth passes, overwriting the first's stats, then run the second x264 pass. If you run the first pass on the first avisynth and the second on the other, you will be destroying rate control, scene-cut detection, and probably getting wildly undersized files as well.

Make sure that bitrate is also based on the 2nd pass avisynth's length, not the original's. They'll be different since you didn't change the FPS.

So should I be making an analysis pass, followed by a lossless encode with only IVTC filter applied. Then use that as my source for the 1st and 2nd pass with other AviSynth filters applied and compression(x264)? If I were to change the mode=5 to mode=2 for the second pass AviSynth script, I know it wouldn't be vfr(I didn't realize vfr wasn't a big help for x264) anymore, but in that case, I would be OK running just the 1st and 2nd passes, right?

BTW, I turned off AQ and it seemed to help it out a little bit and made it less noticeable.

[ChronoCross]

So should I be making an analysis pass, followed by a lossless encode with only IVTC filter applied. Then use that as my source for the 1st and 2nd pass with other AviSynth filters applied and compression(x264)? If I were to change the mode=5 to mode=2 for the second pass AviSynth script, I know it wouldn't be vfr(I didn't realize vfr wasn't a big help for x264) anymore, but in that case, I would be OK running just the 1st and 2nd passes, right?

BTW, I turned off AQ and it seemed to help it out a little bit and made it less noticeable.

1) Take your 1st pass script and edit out everything except TFM and tdecimate. run an analysis pass with the absolute fastest settings OR load it into vdub and play the video all the way through. This will create the timecodes files.
2) Take your 2nd pass script and run 2 passes of x264 like you normally would as an encode.