Good (sane) x265 settings for animated content?

[GrandAdmiralThrawn]

Hello,

I've just now fetched x265 from Videolan to start playing around with it and comparing it to my x264 encodes.

I would like to ask about good settings for animated content (hand-drawn cartoons and Anime, not CGI), as x265 doesn't have --tune animation like x264. Resolutions are 1280x720 and 1920x1080.

Also, I stuck to the same preset I've been using for x264, "veryslow".

I compiled the code for both x86_64 Linux as well as WinXP x64, 64-bit binaries in both cases. The assembly has been handled by yasm 1.3.0, so I have the optimizations.

Now, I don't really understand most of the options, but what I've been doing is this (transcoding some H.264/AVC to H.265/HEVC), here on Windows:

avconv -r 24000/1001 -i video-input.h264 -f yuv4mpegpipe -pix_fmt yuv420p - 2>NUL | x265 - --wpp --y4m -D 12 -p veryslow^
--bframes 16 --bitrate 2000 --pass 1 --slow-firstpass --stats v.stats --sar 1 --range full -o pass1.h265

avconv -r 24000/1001 -i video-input.h264 -f yuv4mpegpipe -pix_fmt yuv420p - 2>NUL | x265 - --wpp --y4m -D 12 -p veryslow^
--bframes 16 --bitrate 2000 --pass 2 --stats v.stats --sar 1 --range full -o pass2.h265

As you can see I'm using libavs' avconv to decode H.264 and pipe it as raw YUV 4:2:0 into x265...

Some options I picked might be redundant though. And it's 2-pass instead of 1-pass CRF because I am targeting file sizes instead of quality levels.

Ok, now there are some "problems" with this, and I'd like to ask you what I should do about them:

1.) Here I chose 12-bit per pixel. I take it hardware players will only support 10-bit for H.265/HEVC? I remember that 10-bit was a good choice for animated content with x264 to reduce banding artifacts on gradients. Will 12-bit result in a similar boost over 10-bit? Playback devices would be the PC (MPC-HC and mplayer), Android devices and possibly some hardware players like TVs or so. Maybe even iOS devices in the future. So I might risk locking out hardware players if going for the 12-bit route. But it has to pay off as well, otherwise I won't do that. Do you think it might provide a good improvement for animated content?
2.) The encoder is abysmally slow on my machines, an i7 950 quadcore and a Xeon X5690 hexcore (basically an i7 990X). It's about 10 times (!!) as slow as x264 on --preset veryslow --bframes 16. Can I do something to speed it up without losing too much quality? Or should I just switch to the slow(er) or even medium presets, turning most of the knobs back a little? I'm also wondering if it might be because I'm lacking the AVX/AVX2 and FMA instruction set extensions. Could that be? I just don't want to sacrifice too much quality..
3.) Generally, all I did to tune for animated content was --bframes 16, but is there anything else I should do?
Thank you for your time!

[Asmodian]

3) In my testing --ctu 32 --max-tu-size 16 helped noticeably with animated content, it kept details better and has an overall sharper look.

[GrandAdmiralThrawn]

Ah, ok, thanks!

According to the documentation, larger ctu and max tu size should be better (for animated stuff), but I guess that ain't true, given the blurring issues?

In any case, I'll try and use those!

[MeteorRain]

1) Main12 seems to not be as mature as Main10 in x265. Plus I couldn't see how much benefits can you achieve with 12 bits. Ma10p is generally good enough, at least for now.

2) Veryslow really means that. Back to 10 years ago when x264 was just out, veryslow can be very painful. Same rule applies to x265 too.
x265 is heavily optimized with AVX2, and lack of it you'd suffer seriously.

3) One of my friends actually suggested me to not go too high on b-frames, I'd say 6 or 8 is good enough, unless you're willing to put more time for little profit.

I made the following code according to his suggestion, and that be my default anime encoding settings now. Seems to be good, but YMMV.

--merange 44 --no-rect --no-amp --aq-mode 1 --aq-strength 1.0 --rd 4
--psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --rdpenalty 1
--tu-inter-depth 2 --tu-intra-depth 2 --ctu 32 --max-tu-size 16

[GrandAdmiralThrawn]

Thanks a lot for your input! I'll try your options. Also, I've compiled myself 8-, 10- and 12-bit binaries, so I'll just switch back to 10-bit.

My first result (that was without --ctu 32 and --max-tu-size 16 however) at 2Mbit was quite disappointing. Extreme loss of detail / blurring, plus surprisingly bad at encoding uniform areas of red color... I'd say, subjectively worse than x264 at veryslow and at the same bitrate, while x264 does the job 10 times as fast.

I'll retry it with your suggested options now.

[MeteorRain]

Just updated the command line to reflect the latest recommendations.

[GrandAdmiralThrawn]

I just ran one test adding only --ctu 32 --max-tu-size 16 and switching from 12 to 10-bit. This gave significantly higher encoding speed and the visual difference was insane already!

So much more detail!

I'll add your settings to the mix now, MeteorRain! Also, I just noticed that the source code from Videolan is seemingly old? 1.7+512-f8b8ebdc5457 was what I got from there. So I pulled 1.9+15-425b583f25db from MulticoreWare, recompiled for Windows+Linux, and I will continue with that.

Are some of your parameters also for boosting encoding speed (like --me-range) in a "sane way", or is it all for maximizing quality? Just wondering..

Thanks for the help! :)

[MeteorRain]

I just ran one test adding only --ctu 32 --max-tu-size 16 and switching from 12 to 10-bit. This gave significantly higher encoding speed and the visual difference was insane already!

So much more detail!

I'll add your settings to the mix now, MeteorRain! Also, I just noticed that the source code from Videolan is seemingly old? 1.7+512-f8b8ebdc5457 was what I got from there. So I pulled 1.9+15-425b583f25db from MulticoreWare, recompiled for Windows+Linux, and I will continue with that.

Are some of your parameters also for boosting encoding speed (like --me-range) in a "sane way", or is it all for maximizing quality? Just wondering..

Thanks for the help! :)

Thanks. All credit goes to @LittlePox for the parameters chosen.

It's aiming for sane quality at sane encoding speed, while some of them are biased for more details. If you are interested in more information, you can bug him directly ;)

And yes, go for v1.9, it's better.

[GrandAdmiralThrawn]

I've turned up a few knobs a bit, like --tu-[inter|intra]-depth and --me-range as well as playing with --rect and --amp, and I can now encode at about 1fps on my i7 950, which is about the slowest encoding speed I can tolerate. Still playing around with options a bit..

I've started doing some detailed side-by-side comparisons mostly using "Garden of Words" at 2Mbit ABR (source is BD), and there are indeed some scenes (one with heavy rain for instance), where x265 can preserve more detail than what x264 can do with my old settings at the same bitrate.

There are still some parts where I'm thinking x264 is slightly better or the same, like with my test with the ending of "Mai Otome", but overall it looks like a win for x265.

I'll look at 1Mbit and maybe 3Mbit ABR as well, to check out the bitrate scaling differences...

Thanks!

[~ VEGETA ~]

I'd like to ask about 12-bit. Is it better than 10-bit in HEVC encoding? I see in the replies that you seem to have a better 10-bit result than 12-bit which doesn't seem reasonable in the theoretical point of view.

[Blue_MiSfit]

Please don't summon year old threads to ask questions like these :)

Closing, please feel free to open a new thread.