Trying to find the equivalent settings from h264 to HEVC

[turinggirl]

Hi everyone,

This is my first post here and I only started to try and encode my own shows over the last month. I first tried Handbrake and while it was good, I felt it could be better, so here I am. Now on to the the problem I'm having that I hope is solvable.

So I'm primarily working with anime and I found an encode that is as close to perfect as I can see. What's more the settings were documented. The problem is these are for encoding in h264 and I'm hoping to take my raw media I own and convert them to HEVC.

Thankfully I realize that the h264 settings aren't a 1:1 comparison to HEVC. However I am hoping that smarter and more experienced individuals could assist me in understanding what these settings are doing in h264 and locating a way to hopefully duplicate them in h265.

Okay so enough yammering here are the h264 settings:

10-bit depth
Encoding settings:
cabac=1 / ref=16 / deblock=1:-1:-1 / analyse=0x3:0x133 / me=umh / subme=10 / psy=1 / fade_compensate=0.50 / psy_rd=0.93:0.22 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=18,9 / fast_pskip=0 / chroma_qp_offset=-3 /
[br]threads=4 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=0 / interlaced=0 / bluray_compat=0 / stitchable=1 / constrained_intra=0 / fgo=0 / bframes=8 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=480 /
keyint_min=1 / scenecut=60 / intra_refresh=0 / rc_lookahead=60 / rc=crf / mbtree=1 / crf=15.0000 / qcomp=0.80 / qpmin=20:20:20 / qpmax=40:40:40 / qpstep=4 / ip_ratio=1.25 / aq=3:0.84 / aq-sensitivity=10.00 / aq-factor=1.00:1.00:1.00 / aq2=0 / aq3=0

Any suggestions would be incredibly helpful. Thank you again.

[Forteen88]

So I'm primarily working with anime
...I recommend bframes=16 for cartoon/anime.


Animation

--tune animation adjusts encoder settings to optimize the encode quality for animation content without impacting the encode speed. This is done by:

--psy-rd 0.4
--aq-strength 0.4
--deblock 1:1
--bframes Increase by 2

https://x265.readthedocs.io/en/default/presets.html

--deblock 1:1 might be a bit high though, your deblock is OK, the picture get softer with high deblock-value.

ref=6 should be set maximally if you want GPU-decoding support for the GPU that supports H.265/HEVC hardware decoding.

Overall, I'd suggest this x265-line for anime/cartoon,
x265 --crf 18 --preset slower --profile main10 --output-depth 10 --rc-lookahead 60 --aq-mode 1 --aq-strength 0.4 --psy-rd 0.4 --frame-threads 2 --no-sao --deblock -2:-2 --qcomp 0.70 --ref 6 --bframes 16 --tu-intra-depth 4 --tu-inter-depth 4 --limit-tu 4

But you set crf=15 in x264, so maybe --crf 17 should be used instead in x265, for more bitrate but higher quality.

[turinggirl]

I'll give this a shot on my testing footage and get back to you. Thanks!

[turinggirl]

From what I can tell it seems to be working great, the only issue I seem to be having is that the lines, at times look a little soft. I'm not sure if that's just the nature of HEVC and I need to live with it.


I tested it on my dark scene footage and there was a huge amount of blocky colors as well as moire patterns in some places like on the walls. I'm not sure how to correct this.

[Forteen88]

You can also try this x265-build (by Doom9-member MeteorRain) that was made for anime,
https://github.com/msg7086/x265-Yuuki-Asuna

[turinggirl]

You can also try this x265-build (by Doom9-member MeteorRain) that was made for anime,
https://github.com/msg7086/x265-Yuuki-Asuna

haha funny enough that's exactly what I'm using!

Okay so here are the files.

https://drive.google.com/open?id=1I9VnVbYhQTktCxFvMgnXNjMneAnE_o1S

The largest one is the Raw reference file. and the rest are the different encodes with adjusted settings.

If you pay attention to the walls specifically you can see what I think is banding.

[turinggirl]

That's my biggest issue right now is trying to figure out the proper setting for animation to deal with low brightness low contrast scenes and the color banding that's occurring. In bright high contrast shows the setting suggested above work fantastic.

[Boulder]

You could try aq-mode 3, you probably need to lower aq-strength from the default 1.0 a bit though.

[Forteen88]

That's my biggest issue right now is trying to figure out the proper setting for animation to deal with low brightness low contrast scenes and the color banding that's occurring. In bright high contrast shows the setting suggested above work fantastic.Many encoding-people uses --zones at dark scenes, and sets higher bitrate there. Maybe you should lower CRF from 17 to 16 or 15 if you don't want to bother with optimizing the settings by looking at what scene startframe and endframe you need to set --zones on.

I hear that higher --aq-strength helps against banding in x264, but I don't know if it does that in x265 too. As 'Boulder' suggested, you should try --aq-mode 3
Maybe you need to deband more in AviSynth+/VapourSynth if the source got a bit banding.

[turinggirl]

You could try aq-mode 3, you probably need to lower aq-strength from the default 1.0 a bit though.

I'll give that a try tonight, I was wondering if you had any suggestions as to where I should aim for the aq-strength, would it be like .92 or something more like .5? Thanks again for all the help I really appreciate it.

[turinggirl]

Many encoding-people uses --zones at dark scenes, and sets higher bitrate there. Maybe you should lower CRF from 17 to 16 or 15 if you don't want to bother with looking at what scene startframe and endframe you need to set --zones on.

I hear that higher --aq-strength helps against banding in x264, but I don't know if it does that in x265 too. As 'Boulder' suggested, you should try --aq-mode 3
Maybe you need to deband more in AviSynth+/VapourSynth if the source got a bit banding.

I haven't been using Avisynth+/VapourSynth. I assume this is going to be one those 'down the rabbit hole' type past times isn't it?

[Forteen88]

I haven't been using Avisynth+/VapourSynth. I assume this is going to be one those 'down the rabbit hole' type past times isn't it?Yeah, it's like a hobby :)

[turinggirl]

You could try aq-mode 3, you probably need to lower aq-strength from the default 1.0 a bit though.

So I changed --aq-mode to 3 and aq-strength to .6 and from what I can tell it seems to have done the trick.

Raw:
https://drive.google.com/file/d/14n7yhh4GNjsLbXFc7Z8v8syyMN-LFLQ2/view?usp=sharing

Encode: https://drive.google.com/file/d/1mRFLIkZEpbBKb7H2YRypAfRbttqjtSCD/view?usp=sharing

[Natty]

You could try aq-mode 3, you probably need to lower aq-strength from the default 1.0 a bit though.

would u recommend aq mode 1 + aq strength 1 over aq 3 st 0.80? how do they actually work and whats the difference? i recently tested aq mode 1 along with f3kdb and got good results in low bitrates but i would like to know what experts think and how it works:thanks:

[Boulder]

I haven't done any extensive testing myself, I trust a few users who are quite experienced in dealing with different material. Aq-mode 2 is the default mode nowadays, and the only difference between that and mode 3 is that mode 3 has a bigger bias towards dark areas. That's why in theory, it should be better for SDR material and especially anime etc. where banding is easily seen. The downside is that bitrate will be much higher so a lower aq-strength should compensate that.

Currently I use aq-mode 2 and aq-strength 0.6-0.8 depending on the source. 0.6 for noisy and grainy and higher for cleaner material.

[Natty]

Currently I use aq-mode 2 and aq-strength 0.6-0.8 depending on the source. 0.6 for noisy and grainy and higher for cleaner material.

:thanks:

[Forteen88]

...
Okay so enough yammering here are the h264 settings:
keyint=480...
You might also wanna set --keyint 480
in x265. The compression will be slightly better, but the seeking will not be that good then.

[RanmaCanada]

So what is your final command line? I only ask because there are several great posts about anime on here, but each of them are from different versions of x265 with different effects? from those versions. ie the new Lamda tables, the new speed changes, etc. It would be great so we can compare how different settings have changed from older versions of x265. I believe the last set I am using is from Meteor Rain, or maybe Wolfberry, as the results were fantastic. I can't understand all the intricate commands anymore :P Might be time to upgrade haha.