X265 Best Settings for Anime?

[Grinchy]

Hello!

I want to encode some of my older Anime Files into x265 to save some space.

I'm using this settings at the moment (StaxRip), but i'm still not happy with them.

--merange 44 --no-rect --no-amp --aq-mode 1 --aq-strength 1.0 --rd 4
--psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --rdpenalty 1
--tu-inter-depth 2 --tu-intra-depth 2 --ctu 32 --max-tu-size 16

The biggest Problem is, that the colour of the Output Files are different from the Original.

Original:
https://picload.org/view/rplrwcil/x265original.png.html
(https://picload.org/image/rplrwcil/x265original.png)
x265 Encode:
https://picload.org/view/rplrwcia/x265neu.png.html
(https://picload.org/image/rplrwcia/x265neu.png)

Are there some recommended x265 Settings for Anime Content?


Thank you!
Patrick

[stax76]

Hi,

regarding anime and color there are more competent people here then me but you should upload the full log file because there is more involved then x265, have a look in my signature.

[Sagittaire]

sao must really help for anime like high value for deblocking.

--ctu 32 --max-tu-size 16

anime have really flat block. Why limit size block in this case?

--psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1

in anime, you must preserve gradiant and not detail. I don't think that psy help here.

[Heaud]

From my personal tests with low motion anime, I have found these settings to help:
--psy-rd 1 --tu-intra 4 --tu-inter 4 --limit-tu 1
Limit TU can be either 1 or 3 as both look quite good with anime with the possibility of one being a better option than the other on certain sources.

I have tested Psy RD settings and have found that a value of 1 works best with low detail anime. Setting it too low starts to affect any dithering used causing gradients to look worse, while setting it too high can make the image look less clean. Same can also be said with the AQ Strength -- don't set it higher than 1 and don't drop it below 0.7.

Encoding at 10 bit will also help plenty with preserving gradients, so I highly recommend it for anime and cartoons.

Edit: I should also add that deblock with strength of 1 or 2 is also a must for encoding anime. Leave SAO and Strong Intra Smoothing enabled as the benefit of preserving flat colors and gradients outweigh the need of keeping high frequency detail with anime.

[JohnLai]

Saving disk space?

How about using Level 5.1 and set --ref to 6?
After all, new Ultra HD Blu-ray specification actually recommends 10bit L5.1 with max 6 ref frames.
Set --aq-mode 3 too. I got funny result (with --aq-mode 1) on anime character with black hair where it seems to be 'blocky' at middle of black hair?
Next, B-frames, stick with b-frames=16 in conjunction with b-adapt=1. Well....b-adapt=2 is the best, but it is way too slow when one sets 16 b-frames.
Enable --weightb for better bframe efficiency.


Now, about the color....which decoder you use before piping the output to x265?

[Motenai Yoda]

--level-idc 4 -F 1 --lookahead-slices 0 --no-high-tier --limit-modes
--limit-refs 3 --crf 16 --deblock -1:-1 --rc-lookahead 60 --ref 6 --bframes 8
--aq-mode 3 --aq-strength 0.6 --ssim-rd --dynamic-rd 2.0 --aq-motion
--no-sao --no-strong-intra-smoothing --psy-rd 1.5 --rdoq-level 2 --psy-rdoq 1.0
--qcomp 0.6 --qg-size 16 --tu-intra 4 --tu-inter 4 --limit-tu 4 --rect
--cbqpoffs -3 --crqpoffs -3 --videoformat ntsc --range limited --colorprim bt709
--transfer bt709 --colormatrix bt709 --vbv-maxrate 12000 --vbv-bufsize 12000
I found bframes > 5 don't help too much, and high psy values tend to banding too much.
Imho 8 or 10 bit is a matter of taste as more deblock and sao

[Heaud]

Saving disk space?

How about using Level 5.1 and set --ref to 6?
After all, new Ultra HD Blu-ray specification actually recommends 10bit L5.1 with max 6 ref frames.
Set --aq-mode 3 too. I got funny result (with --aq-mode 1) on anime character with black hair where it seems to be 'blocky' at middle of black hair?
Next, B-frames, stick with b-frames=16 in conjunction with b-adapt=1. Well....b-adapt=2 is the best, but it is way too slow when one sets 16 b-frames.
Enable --weightb for better bframe efficiency.


Now, about the color....which decoder you use before piping the output to x265?

How much of a size difference is it from 8 b-frames and 16-bframes with b-adapt set to 2? Not even x265's placebo preset goes past 8, so I was wondering if it is even worth it.

I do agree with aq-mode of 3 to be quite useful with anime, although some consideration is needed with CRF as it may need to be increased a bit if you want to avoid the increase in encode file size.

[RanmaCanada]

This is what I use, and it works for me.

pmode:rd=4:tu-intra-depth=3:rdoq-level=2:early-skip:b-intra:limit-modes:aq-mode=2:qg-size=16:ipratio=1.38:pbratio=1.28:me=3:max-merge=3:weightb:bframes=6:rc-lookahead=50:ref=6:psy-rdoq=1.38:no-sao

it might not be the best of the best, but it does a great job to my eyes.

[ChaosKing]

This modded build of x265 (https://down.7086.in/x265-Yuuki-Asuna/) has 2 anime presets build in + a high and low bitrate variant.

My personal experience is that rd-refine helps alot, but it is also slow...

[benwaggoner]

How much of a size difference is it from 8 b-frames and 16-bframes with b-adapt set to 2? Not even x265's placebo preset goes past 8, so I was wondering if it is even worth it.
I have seem some slight improvements with some content using 16 b-frames and tweaking b-bias a bit. But that's for times when you want slight quality improvements beyond placebo, and don't care if it takes longer than placebo.

More for "how good quality can I get over a 2G network", NOT for UHD HDR :)!

I do agree with aq-mode of 3 to be quite useful with anime, although some consideration is needed with CRF as it may need to be increased a bit if you want to avoid the increase in encode file size.
Of course, you can probably get away with a higher CRF since banding/blocking in blacks become much less of an issue with aq-mode 3.

For anime, I've found positive results also using:

--tskip --tskip-fast --tu-inter 4 --tu-intra 4 --limit-tu 4

But at a speed hit.

[katzenjoghurt]

Hm... I basically used the above setting but... my character here had a pretty bad hair day afterwards...

https://i.imgur.com/FJychbE.png

What can I do? More bitrate the only solution?

Yes, the hair moves a lot in this scene.

Settings:
1) (BluRay)
2) --crf 23 --profile main10 --output-depth 10 --limit-tu 4 --tu-intra-depth 4 --tu-inter-depth 4 --tskip --tskip-fast --aq-mode 3 --no-open-gop
3) --crf 25 --profile main10 --output-depth 10 --limit-tu 4 --tu-intra-depth 4 --tu-inter-depth 4 --tskip --tskip-fast --aq-mode 3 --no-open-gop

[ChaosKing]

You also tried with --preset slower or veryslow ?

[katzenjoghurt]

Veryslow didn't help really. :-/
I had some success with no_cutree though. Still the result is not really great.


https://i.imgur.com/AJH6FOA.png

"My settings": --crf 24 --profile main10 --output-depth 10 --limit-tu 4 --tu-intra-depth 4 --tu-inter-depth 4 --tskip --tskip-fast --aq-mode 3 --no-open-gop

[Phanton_13]

I'm using ffmpeg and I had always positive results in anime in veryslow with this: keyint=250:ref=6:limit-refs=3:rc-lookahead=60:amp=1:subme=7:aq-mode=3:aq-strength=1.25:psy-rd=0.80:psy-rdoq=2.00

I found that aq-mode 3 with combination of aq-strength in the range of 1.25 to 1.5, psy-rd in the range 0.75-0.80 and psy-rdoq 2.00 do wonders in dark scenes at the cost of bitrate, in one case the crf 25 files with my config looked better that a standard crf 23 and both had a similar size.

[Forteen88]

aq-mode=3:aq-strength=1.25@katzenjoghurt. I've read that you should lower your aq-strength to like 0.6 for anime, because the solid/"texture"-areas are usually not complex compared to real life footage solid areas.
Although high aq-strength might help against banding if I recall correctly.

Maybe you should try a x265-build set for anime, like this,
https://forum.doom9.org/showthread.php?p=1846311#post1846311

or maybe of course lower CRF to like 22.

[katzenjoghurt]

Maybe you should try a x265-build set for anime, like this,
https://forum.doom9.org/showthread.php?p=1846311#post1846311


Tried it out... but struggled a bit as there doesn't seem to be any good documentation what the presets do.
Eventually I used the vcb-s++ preset.

Result:
BluRay vs.
x265@CRF23 / 3422 kb/s:
https://imgur.com/a/R1jMyry

I'm using ffmpeg and I had always positive results in anime in veryslow with this: keyint=250:ref=6:limit-refs=3:rc-lookahead=60:amp=1:subme=7:aq-mode=3:aq-strength=1.25:psy-rd=0.80:psy-rdoq=2.00


Tried it out. For this situation it also didn't help much.
Though at crf22 I kinda liked the result.

BluRay vs.
x265@CRF22 / 4116 kb/s vs.
x265@CRF24 / 3140 kb/s:
https://imgur.com/a/DckbQAb

(Didn't try out veryslow though)



Eventually I came up now with a setup the user Meteorrain posted here in the forum 2 years ago. And added no_cuttree and set aq-strength to 0.5.

--crf 24 --profile main10 --output-depth 10 --rd 4 --ctu 32 --max-tu-size 16 --tu-intra-depth 2 --tu-inter-depth 2 --rdoq-level 1 --psy-rdoq 5 --aq-strength 0.5 --qcomp 0.75 --no-cutree --merange 44 --no-sao --psy-rd 1.6 --rdpenalty 1 --no-strong-intra-smoothing

BluRay vs.
x265 @CRF24 / 3258kb/s:
https://imgur.com/a/i3rdeyY

Still not awesome but (for my eyes) the best result so far at that bitrate for that situation.

[MeteorRain]

@LittlePox has recently (3 months ago) revised those tunes.

You may find the commit here (https://github.com/msg7086/x265-Yuuki-Asuna/commit/28213b6dbdb11649a0b5cecd1899bc34f1a43603).

[ChaosKing]

Tried it out... but struggled a bit as there doesn't seem to be any good documentation what the presets do.


Make a very short encode and open your file in mediainfo to see all settings.

[katzenjoghurt]

@LittlePox has recently (3 months ago) revised those tunes.

You may find the commit here (https://github.com/msg7086/x265-Yuuki-Asuna/commit/28213b6dbdb11649a0b5cecd1899bc34f1a43603).

Thanks MeteorRain... did the same test as above and wasn't happy with the result. I think I like your original settings more.

Compressed several movies now with these altered settings above and I like the result in general - except in dark scenes maybe where areas tend to not stay really flat any more near the edges.

Make a very short encode and open your file in mediainfo to see all settings.

Sure... but what do they do?
Especially the new tunes in there.
When to pick which one?

But I think I'm happy with the settings I found now...

[benwaggoner]

Compressed several movies now with these altered settings above and I like the result in general - except in dark scenes maybe where areas tend to not stay really flat any more near the edges.
Are you using --aq-mode 3? That can really help in low luma details.

It might also be your display. Depending on how it's made, lots of displays get weird artifacts in near-black.

[katzenjoghurt]

I'm pretty sure it's not the display as the original didn't have those artifacts.

I will play around with --aq-mode 3 next time I do an animation encode. Thanks for that hint, benwaggoner! :)