distorted anime encode

[desta]

I've just done a couple of test encodes of the anime movie Akira. The results were... well.. not exactly desirable. I'm sure it's gotta be something I'm doing wrong, or just not doing at all.

I'm encoding at full resolution anamorphic, with a [desired] video bitrate of 1174kbps - not exactly huge, but I thought would still give relatively good results. Screenshots and settings are below. As you'll probably notice, the source is very noisey. Usually if I'm going to encode to x264, I prefer to leave the noise as is, because I want to preserve as much detail as possible. This time however, and because it's anime, I decided I'd probably be able to live with the smoothing... until I saw the results.


Source MPEG-2

http://img486.imageshack.us/img486/6923/01originalsource5bx.png



Denoised, deblocked x264

http://img50.imageshack.us/img50/9528/02denoiseddeblockedx2644oe.png



Plain, unfiltered x264

http://img369.imageshack.us/img369/7544/03plainencodex2646zq.png


.. With both of the x264 encodes I used Sharktooth's AE-Max Quality profile (with slight variations to the settings - below). I'm using MeGUI 0.2.3.2132 and the patched x264 Rv.523.


Denoised, deblocked x264

global MeGUI_darx = 16
global MeGUI_dary = 9
mpeg2source("C:\Akira\akira.d2v",cpu=4,info=3)
ColorMatrix(hints=true)
crop(2,2,-2,-2)
LanczosResize(720,576)
FluxSmoothST(7,7)

--pass 2 --bitrate 1174 --stats "akira_test.stats" --ref 10 --mixed-refs --no-fast-pskip --bframes 5 --b-pyramid
--b-rdo --bime --weightb --direct auto --filter 2,1 --subme 6 --trellis 2 --analyse all --8x8dct --qpmin 5 --ratetol 4.0
--me umh --sar 64:45 --progress --no-dct-decimate --no-psnr --output "akira_test.mkv" "akira_test.avs" --aq-strength 0.8
--aq-sensitivity 15


Plain, unfiltered x264

global MeGUI_darx = 16
global MeGUI_dary = 9
mpeg2source("C:\Akira\akira.d2v",cpu=0)
crop( 2, 2, -2, -2)
LanczosResize(720,576)

--pass 2 --bitrate 1174 --stats "akira_test2.stats" --ref 10 --mixed-refs --no-fast-pskip --bframes 5 --b-pyramid
--b-rdo --bime --weightb --direct auto --subme 6 --trellis 2 --analyse all --8x8dct --qpmin 5 --ratetol 4.0 --me umh
--sar 64:45 --progress --no-dct-decimate --no-psnr --output "akira_test2.mkv" "akira_test2.avs" --aq-strength 0.8
--aq-sensitivity 15


Any help much appreciated. If you need any more info, let me know.

[ChronoCross]

Remove AQ, rateol fromt he command line. Update x264 and try again.

[desta]

Plain, unfiltered, no AQ, bitrate variance = 1, x264 (sharktooth 532)

http://img205.imageshack.us/img205/1417/04noaqplain5cg.png

global MeGUI_darx = 16
global MeGUI_dary = 9
mpeg2source("C:\Akira\akira.d2v", cpu=0)
crop( 2, 2, -2, -2)
LanczosResize(720,576)

--pass 2 --bitrate 1174 --stats "akira_test3.stats" --ref 10 --mixed-refs --no-fast-pskip --bframes 5 --b-pyramid
--b-rdo --bime --weightb --direct auto --subme 6 --trellis 2 --analyse all --8x8dct --qpmin 5 --me umh --sar 64:45
--progress --no-dct-decimate --no-psnr --output "akira_test3.mkv" "akira_test3.avs"

[ChronoCross]

at this point it looks like too low a bitrate. Akira was oversharpened if I remember correctly. trying to encode it at that bitrate might not be enough. But there could be another problem any idea what quant it's giving that frame?

[Sharktooth]

The problem seems to come from the noise.
Fluxsmooth in simply not "enough". Try with this:
TemporalSoften(5,2,2,scenechange=15,mode=2)
LRemovedust(2,5)
removegrain(mode=2)
And lower the loop filter settings.
It's quite slow but will kill the noise for sure.

[desta]

Well, this was the logfile for both passes of the last encode....

Starting job job1-1 at 18:54:13
Job is a video job. encoder commandline:
--pass 1 --bitrate 1174 --stats "akira_test3.stats" --bframes 5 --b-pyramid --direct auto --subme 1 --analyse none
--qpmin 5 --me dia --sar 64:45 --progress --no-dct-decimate --no-psnr --output NUL "akira_test3.avs"
successfully started encoding
Processing ended at 18:55:37
----------------------------------------------------------------------------------------------------------

Log for job job1-1

avis [info]: 720x576 @ 25.00 fps (1500 frames)
x264 [info]: using SAR=64/45
x264 [info]: using cpu capabilities MMX MMXEXT SSE 3DNow!
x264 [info]: slice I:20 Avg QP:30.05 size: 20747
x264 [info]: slice P:738 Avg QP:33.06 size: 7660
x264 [info]: slice B:742 Avg QP:32.84 size: 2548
x264 [info]: mb I I16..4: 56.7% 0.0% 43.3%
x264 [info]: mb P I16..4: 27.1% 0.0% 0.0% P16..4: 31.1% 0.0% 0.0% 0.0% 0.0% skip:41.8%
x264 [info]: mb B I16..4: 6.7% 0.0% 0.0% B16..8: 15.3% 0.0% 0.0% direct: 8.0% skip:70.0%
x264 [info]: final ratefactor: 30.13
x264 [info]: direct mvs spatial:99.2% temporal:0.8%
x264 [info]: kb/s:1061.2

Actual bitrate after encoding without container overhead: 1061.41

----------------------------------------------------------------------------------------------------------
Job completed successfully and deletion of intermediate files is activated
job job1-1 has been processed. This job is linked to the next job: job1-2
Starting job job1-2 at 18:55:37
Job is a video job. encoder commandline:
--pass 2 --bitrate 1174 --stats "akira_test3.stats" --ref 10 --mixed-refs --no-fast-pskip --bframes 5 --b-pyramid
--b-rdo --bime --weightb --direct auto --subme 6 --trellis 2 --analyse all --8x8dct --qpmin 5 --me umh --sar 64:45
--progress --no-dct-decimate --no-psnr --output "akira_test3.mkv" "akira_test3.avs"
successfully started encoding
Processing ended at 19:06:58
----------------------------------------------------------------------------------------------------------

Log for job job1-2

avis [info]: 720x576 @ 25.00 fps (1500 frames)
x264 [info]: using SAR=64/45
x264 [info]: using cpu capabilities MMX MMXEXT SSE 3DNow!
x264 [info]: slice I:20 Avg QP:26.05 size: 28203
x264 [info]: slice P:738 Avg QP:29.95 size: 8518
x264 [info]: slice B:742 Avg QP:30.46 size: 2701
x264 [info]: mb I I16..4: 30.5% 44.5% 25.0%
x264 [info]: mb P I16..4: 13.7% 9.4% 4.1% P16..4: 38.0% 8.9% 3.5% 0.2% 0.1% skip:22.0%
x264 [info]: mb B I16..4: 2.3% 1.2% 0.5% B16..8: 29.2% 1.1% 1.9% direct: 3.9% skip:59.9%
x264 [info]: 8x8 transform intra:34.9% inter:62.4%
x264 [info]: direct mvs spatial:92.2% temporal:7.8%
x264 [info]: ref P 61.3% 15.0% 9.0% 4.4% 3.1% 2.3% 1.8% 1.0% 1.0% 1.1%
x264 [info]: ref B 65.5% 13.4% 6.7% 4.2% 2.9% 2.5% 1.7% 0.9% 1.1% 1.2%
x264 [info]: kb/s:1180.6

Actual bitrate after encoding without container overhead: 1180.92
desired video bitrate of this job: 1174 kbit/s - obtained video bitrate: 1183.7016 kbit/s
----------------------------------------------------------------------------------------------------------
Job completed successfully and deletion of intermediate files is activated
Found intermediate output file 'akira_test3.stats', deleting...
Deletion succeeded.

I think I'll have to do just like you said, and increase the bitrate - I was hoping for more compression, but I'll live. :)

I did have a question though... earlier you said to lower the bitrate variance - I thought a setting of 4 would give better visual results, due to it allocating bits more accurately... is that wrong?

Also, is AQ not needed when "No DCT Decimation" is selected?


edit: Sharktooth, sorry.. I don't know how I missed your reply there. I'll give that a try and let you know the results.


LRemoveDust x264

http://img149.imageshack.us/img149/3673/05lremovedustx2647tf.png

import("LRemoveDust.avs")
global MeGUI_darx = 16
global MeGUI_dary = 9
mpeg2source("C:\Akira\akira.d2v")
crop(2,2,-2,-2)
LanczosResize(720,576)
TemporalSoften(5,2,2,scenechange=15,mode=2)
LRemovedust_YV12(2,5)
removegrain(mode=2)


--pass 2 --bitrate 1172 --stats "akira_test5.stats" --ref 10 --mixed-refs --no-fast-pskip --bframes 5 --b-pyramid
--b-rdo --bime --weightb --direct auto --filter -4,-3 --subme 6 --trellis 2 --analyse all --8x8dct --qpmin 5
--me umh --sar 64:45 --progress --no-dct-decimate --no-psnr --output "akira_test5.mkv" "akira_test5.avs"

I think it's time to give up on this one... hehe.

Chronocross, I did try an encode at 1430kbps - came out the same.

[ChronoCross]

try pumping the bitrate to something ridiculous. like 10000kbps. if it doesn't look good with that kind of bitrate it might be x264 doing something wrong.

[Sharktooth]

oh, please use x264 r532A coz r532 had some problems (maybe affecting even non lossless encodings).
however your last image looks better that the previous ones (at least on the bright edges).
try rising settings of those 3 filters.

[foxyshadis]

You should also make sure you compare the script output with the encoded output; if it's getting butchered by a filter there's nothing x264 can do to fix it.

The fact that you got ~q30 in the other pretty much indicates that you'd need a much higher bitrate. You're probably looking for quality in the q24 range, which for something that noisy might not be possible even at 1500. (Akira is an xvid killer without overfiltering, too.)

[desta]

try pumping the bitrate to something ridiculous. like 10000kbps. if it doesn't look good with that kind of bitrate it might be x264 doing something wrong.

8000kbps x264

http://img116.imageshack.us/img116/1779/068000kbpsx2646mp.png

... that actually doubled the encode time too, hehe.



oh, please use x264 r532A coz r532 had some problems (maybe affecting even non lossless encodings).
however your last image looks better that the previous ones (at least on the bright edges).
try rising settings of those 3 filters.
I just downloaded r532 when ChronoCross said to update... I didn't realise there was a r532A.

As for the last picture looking better... you sure? I don't mean to sound argumentative, but of all of them (not including the one in this post) the Plain, unfiltered, no AQ, bitrate variance = 1, x264 (sharktooth 532) encode looks the best, to me anyway.


You should also make sure you compare the script output with the encoded output; if it's getting butchered by a filter there's nothing x264 can do to fix it.

The fact that you got ~q30 in the other pretty much indicates that you'd need a much higher bitrate. You're probably looking for quality in the q24 range, which for something that noisy might not be possible even at 1500. (Akira is an xvid killer without overfiltering, too.)

I always check the script output first to see what effect the filters are having before I encode, and then I check again against the encoded output afterwards.

Tbh, if it's going to mean a bitrate any higher than 3000kbps, I probably won't bother - might aswell just be using mpeg-2 then, and I've got the DVD for that, hehe. Actually, I've gotta admit, you did kinda lose me on the "The fact that you got ~q30..." bit. Is that referring to the final rate factor indicated in the first pass? Does that mean the higher the number, the more bitrate it needs?

[foxyshadis]

From second pass:

x264 [info]: slice P:738 Avg QP:29.95 size: 8518
x264 [info]: slice B:742 Avg QP:30.46 size: 2701

q18 is generally "transparent", q24 is decent all-around quality, and q30 is kind of crappy as you've seen. To quickly estimate the bitrate you'll need for ~q24, try this:

--crf 26 --keyint 400 --ref 4 --mixed-refs --bframes 5 --b-pyramid --direct auto --subme 3 --no-chroma-me --analyse --8x8dct --threads 2 --thread-input --progress --no-psnr

(customize as desired) It won't look great, you'll still have to plug its bitrate back into your hardcore options to see how it looks.

Oh, and there's something I forgot until now; what's the decoder? If ffdshow, some distributions have an issue by default where h264 decoding is incorrectly post-processed as if it's xvid, if pp is on.

[desta]

So basically, if I do a test encode with any settings I want to use, but set to constant quality 26, I should be able to use the bitrate aquired from log to do a new encode with fairly decent results? I'm not sure I followed what you meant properly...

For playback I use CoreAVC. I have got ffdshow installed with pp on, but h.264 is disabled.