x264-Settings for Anime

[Revenant88]

Hi, I'd like to know which settings I could tune within x264 to improve encoding of Anime speed- and qualitywise,

I currently use r1570 and set to CRF 16.0 with preset slower and -tune animation.

But I feel that more than 5 ref-frames really just slow down encoding and don't do that much for compression, also do I need more than 3 b-frames really?
I'm currently prefering 5 ref and 3 b-frames, using b-adapt 1 and b-pyramid off.
Or should I change something here, something that is worth the speed trade-in?
For deblocking I have the 1:1 set by the preset, should I leave it at that or go for -1:-1 for sharpness?

For ME I use Hexagon with a range of 16 and -subme 7.
Anything worth the speed trade-in here? Higher subme or ME-range?
Should I turn of Psy-RD? Or leave it at 0,40?
I have Trellis on 1 final mb.

Variance AQ is set to 0,8.

Any thoughts on how to improve this, am I maybe trading in too much quality here for speed? I don't mind having to use a little more bitrate really.
Just wanna know if someething might be terribly wrong with these settigns for Anime.

[me7]

Take a look at the animation tuning of x264's help: It raises reference frames and b-frames, sets positive deblock values and explicitly sets psy-rd to 0,4. Do you think the devs made this tuning to trick you?

More b-frames and reference frames are great for animation since it usually contains more redundancy then live-action, blocking becomes very obvious on flat surfaces so better leave deblock at 1:1 and psy-rd and aq-strength on the anime tuning are set to preserve edge details AFAIK, don't touch them unless you know what you do.

To sum it up, you take an animation-optimized setting and remove all the animation optimizations...why?

[Revenant88]

Alright, I'll leave those settings set by animation to their default.
Any tips on settings for ME (hexagon vs multi hex and subme)?

[Puncakes]

First of all: This has been discussed over nine thou-- a lot. Use search next time

Second of all: I feel like typing about it.

Refs and B-frames help a lot with animation, even in high amounts(hence why refs are doubled, and b-frames get +2 with --tune animation).
--b-adapt 1 gets faster and better with more B-frames, so max that out if you want to use it(and don't need hardware compatibility) though --b-adapt 2 is much better(and slower). Fun-fact: --b-pyramid actually speeds up encoding with --b-adapt 2 and high b-frames.
Well, whether or not something is "worth" the speed hit is up to you.
Once again, we can't decide how sharp you want your videos for you.

It seems strange that you're using --preset slower, and then lowering a bunch of quality-improving options it enables in exchange for being "slower." In any case, in my opinion, umh is definitely worth the speed hit over hex.
Subme and me-range aren't quite as important, but I'm a quality freak so I keep them turned up to --subme 10 and --me-range 32(24 is more sane, but not entirely necessary)
That depends on whether or not the sharpness and detail(as well as noise) retention it causes. Just as with the others, it depends on you and your source.
--trellis 2 is another option that much like refs and b-frames, benefits animation most.

I'm confused again on why you're settings options like --tune animation and then reversing them. AQ takes bits away from sharp lines and gives them to flat/detailed areas to prevent blocking/blurring. By taking those bits though, it induces ringing and noise around those lines and edges. It is lowered in --tune animation because such material is generally less detailed than normal film, and sharp lines/edges are more pronounced/important. This does of course once again, come down to whether or not you like the effects it has though.

See above thoughts on possible "improvement."
I don't see anything "terribly" wrong.

tl;dr: You're probably fine, but you could try testing some stuff yourself. Or, if you really don't care about adding more bitrate, you could just lower the CRF.

[Revenant88]

Hey thanks, that's the kinda stuff I wanted to read pal.
Will try things and stuff :)

[Daiz]

I usually use the following settings for my SD (480p) anime encodes:
x264 --crf 16 --preset slower --tune animation --psy-rd 1.0:0.2 -o output.mkv in.avs

Personally I don't like how --tune animation drops psy-rd to 0.4. In my experience, it just leads to gradients getting banded and blocky more often.

[julius666]

Personally I don't like how --tune animation drops psy-rd to 0.4. In my experience, it just leads to gradients getting banded and blocky more often.

And prevents edges from heavy blocking and ringing. :rolleyes:
IMHO psy-rd 0.4 is waaaay to strong for animes. At least on lower bitrates.
I use to set it between 0.1 and 0.2.

[Keiyakusha]

My setteigs may warry depending on content complexity but in cases when "everything is flat" I prefer --preset slower --tune ssim --no-psy and --crf adjusted so the ssim will be not less than 0.995 (usually near --crf 16). Maybe also lower AQ.
I prefer psy only for very high compression/low bitrates and live action of course.

[Dark Shikari]

My setteigs may warry depending on content complexity but in cases when "everything is flat" I prefer --preset slower --tune ssim --no-psy and --crf adjusted so the ssim will be not less than 0.995 (usually near --crf 16). Maybe also lower AQ.
I prefer psy only for very high compression/low bitrates and live action of course.You should never use --no-psy. It will do things like decrease quality before keyframes (because those frames won't be referenced much in the future) causing keyframe flashing.

Use --psy-rd 0, not --no-psy, because the latter does other things that you probably don't want. Of course, --tune ssim also implies --no-psy.

[Daiz]

And prevents edges from heavy blocking and ringing. :rolleyes:

Well, I haven't noticed much if any of that in my encodes. But I've certainly noticed the ugly blocking and banding of gradients in all of my low-psy encodes.

[Majin3]

Guess I'll take the opportunity and also ask if my settings are fine.

--profile high --crf 18 --level 4.1 --keyint 240 --ref 4 --bframes 3 --me umh
--subme 10 --b-adapt 2 --direct auto --trellis 2 --rc-lookahead 60
--psy-rd 0.6:0 --aq-strength 0.6 --deblock 1:1

The goal is to encode 1920x1080p anime while maintaining DXVA compability. For 480p I use the same settings, but with "--level 3.1".
Is it even fully DXVA compatible and is there something I could improve without it getting a lot slower?

[LoRd_MuldeR]

Why not use something simple like this instead:
--preset veryslow --tune animation --level 4.1 --crf 18 --keyint 240 --bframes 3

[julius666]

Guess I'll take the opportunity and also ask if my settings are fine.

--profile high --crf 18 --level 4.1 --keyint 240 --ref 4 --bframes 3 --me umh --subme 10 --b-adapt 2 --direct auto --trellis 2 --rc-lookahead 60
--psy-rd 0.6:0 --aq-strength 0.6 --deblock 1:1

The goal is to encode 1920x1080p anime while maintaining DXVA compability. For 480p I use the same settings, but with "--level 3.1".
Is it even fully DXVA compatible and is there something I could improve without it getting a lot slower?

Yes, you could.
First, use --preset veryslow instead of this:
--me umh --subme 10 --b-adapt 2 --direct auto --trellis 2 --rc-lookahead 60
then use --tune anime instead of this:
--psy-rd 0.6:0 --aq-strength 0.6 --deblock 1:1
this won't make the quality of your encoded video any better (well, they are the same settings after all), just makes you commandline more compact.

If you don't want to play your video on special devices (just on computer), then you could raise your b-frames even in 1080p. B-frame number doesn't hurt DXVA-compatibility, it's pretty safe to use more than 3. I would recommend 9, animes use to gain a lot from b-frames (it will slow your encoding down though).

Then, for 480p, you could raise ref frames, even to 11 (this could slow your encode down too, but ref frames are also very useful for animes).

You could play also with psy-settings, I like lower values for animes personally.

a not-so-slower example:

--crf 18 --level 3.1 --preset veryslow --tune anime --keyint 240 --min-keyint 24 --ref 6 --bframes 9

[Majin3]

@LoRd_MuldeR: That would for example increase reference frames to 16 which is like ~40% slower while (after reading in this forum) not really improving much. I also don't know if that's fine with DXVA.
"veryslow" could also be changed some day to be not that effective (if for example some preset is added after it) or to be much slower, so I don't really want to use presets.

[LoRd_MuldeR]

@LoRd_MuldeR: That would for example increase reference frames to 16 which is like ~40% slower while (after reading in this forum) not really improving much. I also don't know if that's fine with DXVA.

You already are using very slow settings, like "--subme 10" and "--trellis 2". So I was assuming "--preset veryslow" should fit for you. If that is too slow, use "slower" or "slow" instead.

Also "--level 3.1" should limit the number of reference frames accordingly to the Level's DPB limitations. Doesn't it ???

"veryslow" could also be changed some day to be not that effective (if for example some preset is added after it) or to be much slower, so I don't really want to use presets.

If that is ever done, then it will be done for good reason. So in that case you'll run into problems with your "hardcoded" options - instead of getting updated with the preset automatically ;)

[Dark Shikari]

And furthermore, there's no guarantee the behavior of, say, --subme 10, won't change in the future ;)

[Majin3]

If you don't want to play your video on special devices (just on computer), then you could raise your b-frames even in 1080p. B-frame number doesn't hurt DXVA-compatibility, it's pretty safe to use more than 3. I would recommend 9, animes use to gain a lot from b-frames (it will slow your encoding down though).
Ah, I see. I couldn't find anything about a bframes limit for DXVA, so I just set it to 3 (the BD limit).
I just want them to play on my TV and to be DXVA compatible. Guess I'll need to check if my TV cares about higher b-frame numbers.

Then, for 480p, you could raise ref frames, even to 11 (this could slow your encode down too, but ref frames are also very useful for animes).
I've read that "no matter the resolution, it is still generally recommended to only use between 1-5 reference frames, usually centering on 3." here (http://www.avsforum.com/avs-vb/showthread.php?t=972503). Or doesn't that matter?

You already are using very slow settings, like "--subme 10" and "--trellis 2". So I was assuming "--preset veryslow" should fit for you. If that is too slow, use "slower" or "slow" instead.
Ah, never mind. I confused b-frames with reference frames. Yes, "veryslow" would be the best I guess.

Also "--level 3.1" should limit the number of reference frames accordingly to the Level's DPB limitations. Doesn't it ???
I'm not sure.
x264 says "x264 [warning]: DPB size (13 frames, 40734720 bytes) > level limit (4 frames, 12
582912 bytes)" if I set ref too high, but stays quiet if it's just in the preset.

And furthermore, there's no guarantee the behavior of, say, --subme 10, won't change in the future ;)
Hm, good point.


OK, you've all convinced me.
That would make my line:
"--profile high --crf 18 --level 4.1 --preset veryslow --tune animation --keyint 240 --ref 4"
...which changes:
bframes: 3 -> 10
merange: 16 -> 24
psy-rd 0.6:0 -> 0.4:0
adds --partitions all
and makes a ~30% speed loss

I don't really know why I didn't raise merange in my line, but I changed the others out of a reason.
- I couldn't find if bframes should be limited for DXVA, so I set them to the BD limit, 3. julius666 said it doesn't matter though, so I guess 10 should be fine.
- I've often read people saying psy-rd shouldn't be set that low (in this thread too, heh). Yes, the developers know about their settings better of course, but I doubt they're encoding anime all the time, so... Should it really be set to 0.4?
- Is "--partitions all" worth the ~7% speed loss?

Should I maybe also include some vbv limits?

[LoRd_MuldeR]

I'm not sure.
x264 says "x264 [warning]: DPB size (13 frames, 40734720 bytes) > level limit (4 frames, 12
582912 bytes)" if I set ref too high, but stays quiet if it's just in the preset.

...because if you only use the preset, the number of ref frames is adjusted to comply with the specified level.

But if you explicitly overwrite the number of ref frames, you'll get what you have request - even if that is contrary to the specified preset.

And thus the warning message ;)

[Majin3]

I see. But I guess I still need to specify them manually so they don't go over the DXVA limit.

[LoRd_MuldeR]

I see. But I guess I still need to specify them manually so they don't go over the DXVA limit.

Not sure about DXVA. But if DXVA claims to supports H.264 Level 3.1, then it should accept the number of ref frames allowed by the DPB size specified for Level 3.1 (i.e. 5 for 720p footage).

[Keiyakusha]

I see. But I guess I still need to specify them manually so they don't go over the DXVA limit.

What is your limit? Different hardware with different drivers and even OS can have different limits. Some setups can play anything through DXVA, including 5.1 level 16 ref frames.

[Majin3]

Not sure about DXVA. But if DXVA claims to supports H.264 Level 3.1, then it should accept the number of ref frames allowed by the DPB size specified for Level 3.1 (i.e. 5 for 720p footage).
There is some formula for calculating the allowed ref frames from the resolution, so it's not just about the level.

What is your limit? Different hardware with different drivers and even OS can have different limits. Some setups can play anything through DXVA, including 5.1 level 16 ref frames.
My hardware doesn't even support DXVA, that's why I'm asking which settings I should include or limit.
I'm following this limitations (http://www.avsforum.com/avs-vb/showthread.php?t=972503) to ensure compability with most hardware and not just the newest.

[creamyhorror]

As mentioned, b-frames don't affect DXVA compatibility, so just use a reasonable number like 6, which is a good number to use for many/most live-action sources, while anime can actually benefit from more. Reference frames should also be kept high, subject to the limits of your frame size and decoder level (e.g. 4 if you're using the biggest 1080p resolution, and 8-9 if using 720p).

No point limiting both b-frames and reference frames to 3, unless your TV is weirdly finicky. 3 b-frames is unnecessarily few IMO, especially for anime.

Also, don't rely purely on that AVSForum thread, it's really old (avoid b-pyramids?) and the recommendations are arguable (3 reference frames?).

[julius666]

OK, you've all convinced me.
That would make my line:
"--profile high --crf 18 --level 4.1 --preset veryslow --tune animation --keyint 240 --ref 4"
...which changes:
bframes: 3 -> 10
merange: 16 -> 24
psy-rd 0.6:0 -> 0.4:0
adds --partitions all
and makes a ~30% speed loss


--preset veryslow sets bframes to 8. edit: --tune animation sets +2 bframes, so --b-frames 10 is the correct value. I'm just dumb :p
And I'm pretty sure that the 30% speed loss worth the effort, so don't worry :).


- I've often read people saying psy-rd shouldn't be set that low (in this thread too, heh). Yes, the developers know about their settings better of course, but I doubt they're encoding anime all the time, so... Should it really be set to 0.4?


The effectiveness of psy-settings are very subjective. It also depends heavily on the source. You have to do test encodes with several psy-settings, if you want perfect results.
I usually do low-bitrate anime encodings, I prefer very low psy-settings (0.1:0-0.2:0 psy-rd and 0.2-0.3 aq-strength) on this kind of content. Although higher psy-settings make details better, heavy ringing occurs frequently as a side effect. Maybe they are more usable on higher bitrates.

Should I maybe also include some vbv limits?

Since I do low-bitrate encodings, there isn't much chance that the encoding would run into limits. But on very high bitrates it is possible.

[LoRd_MuldeR]

There is some formula for calculating the allowed ref frames from the resolution, so it's not just about the level.

The Level defines the size of the DPB (Decoded Picture Buffer). See "Annex A.3" in the H.264 specifications!

Of course it also depends on the resolution of the video. The higher the resolution of the video, the fewer frames fit into the same (Level-dependent) DPB size ;)

For Level 3.1 and 1280x720 video this can be calculated as:
MaxRefFrames = Floor((DPBSize * 1024 * 8) / (Width * Height * 12)) = Floor((6750 * 1024 * 8) / (1280 * 720 * 12)) = Floor(55296000 / 11059200) = 5

(Note that DPB size is specified in units of 1024 bytes and that for YV12 data we need 12 bits per pixel)

[Majin3]

No point limiting both b-frames and reference frames to 3, unless your TV is weirdly finicky. 3 b-frames is unnecessarily few IMO, especially for anime.
I've tested it on my TV and it doesn't seem to care about even 16 bframes on 1080p, so guess I'll stick to those 10 bframes from the presets.

The Level defines the size of the DPB (Decoded Picture Buffer). See "Annex A.3" in the H.264 specifications!

Of course it also depends on the resolution of the video. The higher the resolution of the video, the fewer frames fit into the same (Level-dependent) DPB size ;)

For Level 3.1 and 1280x720 video this can be calculated as:
MaxRefFrames = Floor((DPBSize * 1024 * 8) / (Width * Height * 12)) = Floor((6750 * 1024 * 8) / (1280 * 720 * 12)) = Floor(55296000 / 11059200) = 5

(Note that DPB size is specified in units of 1024 bytes and that for YV12 data we need 12 bits per pixel)
I see, so that limits are actually just the level limits.
Then I can just leave out the "--ref" too and let x264 set it according to the level?
That would make my lines:
--profile high --level 4.1 --crf 18 --preset veryslow --tune animation --keyint 240
&
--profile high --level 3.1 --crf 18 --preset veryslow --tune animation --keyint 240
Kind of boring, heh


Which leaves the only question: Should I set some VBV limits and if yes which ones?
EDIT: Are --vbv-maxrate 62500 --vbv-bufsize 62500 for 4.1 and --vbv-maxrate 17500 --vbv-bufsize 17500 for 3.1 correct?

[LoRd_MuldeR]

I see, so that limits are actually just the level limits.
Then I can just leave out the "--ref" too and let x264 set it according to the level?
That would make my lines:
--profile high --level 4.1 --crf 18 --preset veryslow --tune animation --keyint 240
&
--profile high --level 3.1 --crf 18 --preset veryslow --tune animation --keyint 240
Kind of boring, heh

Agreed.

Which leaves the only question: Should I set some VBV limits and if yes which ones?
EDIT: Are --vbv-maxrate 62500 --vbv-bufsize 62500 for 4.1 and --vbv-maxrate 17500 --vbv-bufsize 17500 for 3.1 correct?

As far as I know, VBV limitations aren't covered by H.264 Levels. It depends on the requirements of the individual device.

The VBV BuffSize is defined by the device's buffer size. The VBV MaxRate is specified by the drive read-speed (local playback) or by the network bandwidth (streaming).

For DXVA I'm unable to find "official" specifications of the VBV requirements. It may even be different between NVidia and AMD/ATI...

[julius666]

Agreed.
As far as I know, VBV limitations aren't covered by H.264 Levels. It depends on the requirements of the individual device.

The VBV BuffSize is defined by the device's buffer size. The VBV MaxRate is specified by the drive read-speed (local playback) or by the network bandwidth (streaming).

For DXVA I'm unable to find "official" specifications of the VBV requirements. It may even be different between NVidia and AMD/ATI...

well, there is a bitrate-limit based on http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
But I haven't encountered any problem so far.

[LoRd_MuldeR]

well, there is a bitrate-limit based on http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels

MaxBitrate Limit != VBV Requirements

[julius666]

MaxBitrate Limit != VBV Requirements

How could you limit max bitrate then? Or does setting level to a specific value solve the problem? :confused:

[LoRd_MuldeR]

How could you limit max bitrate then? Or does setting level to a specific value solve the problem? :confused:

You can't. Even with 1-Pass ABR mode and VBV enabled, the bitrate is still allowed to fluctuate - within the VBV restrictions.

Also with VBV enabled, there still can be (local) bitrate peaks that exceed the VBV MaxRate. This is perfectly valid, as long as the Buffer doesn't underflow. And x264 makes sure it doesn't ;)

So I think your closest option to a "hard" bitrate limit would be using CBR mode, i.e. VBV with MaxRate == Bitrate. But that's something you probably don't want to use in reality...

See also:
http://forum.doom9.org/showpost.php?p=1356950&postcount=8

[Majin3]

Then I'll just leave out the vbv limits if they are hardware-specific anyway.

Thanks to everyone who helped! My case is closed then.