x265: a good compromise between "slow" and "slower" presets?

[Boulder]

SAO had not been touched at all in a long time in code so I would keep it disabled if that's what you did earlier.

[Nico8583]

Ok thank you so I'll keep no sao.
And sorry it was not Dolby Vision but HDR10+ in my previous post, MKV doesn't support Dolby Vision now.

[redbtn]

Originally Posted by Nico8583

Ok thank you so I'll keep no sao.
And sorry it was not Dolby Vision but HDR10+ in my previous post, MKV doesn't support Dolby Vision now.

HDR10+ requires json file if I remember correctly. I didn't any hdr10+ encodes yet.

[redbtn]

I have some wiered green artifacts that appear in various places. I tried different settings (also --aud helps a lot) and even clean Preset Slow, but they are still there. I don't understand where is the problem.
--cbqpoffs -3 --crqpoffs -3 didn't help

Left - source | Right - encode (both tonemapped with MadVR)



My settings

--level-idc 51 --colorprim 9 --colormatrix 9 --transfer 16 --range limited --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(40000000,50)" --max-cll "3241,902" --hdr --hdr-opt --hrd --aud --chromaloc 2 --repeat-headers --no-sao --no-open-gop --vbv-bufsize 160000 --vbv-maxrate 160000

Mediainfo from encode

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main 10@L5.1@High
HDR format : SMPTE ST 2086, HDR10 compatible
Codec ID : V_MPEGH/ISO/HEVC
Duration : 1 min 44 s
Bit rate : 14.2 Mb/s
Width : 3 840 pixels
Height : 1 600 pixels
Display aspect ratio : 2.40:1
Frame rate mode : Constant
Frame rate : 23.976 (24000/1001) FPS
Color space : YUV
Chroma subsampling : 4:2:0 (Type 2)
Bit depth : 10 bits
Bits/(Pixel*Frame) : 0.097
Stream size : 177 MiB (100%)
Writing library : x265 3.2+7-37648fca915b:[Windows][GCC 9.2.0][64 bit] 10bit
Encoding settings : cpuid=1111039 / frame-threads=2 / numa-pools=6 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=3840x1600 / interlace=0 / total-frames=2500 / level-idc=51 / high-tier=1 / uhd-bd=0 / ref=4 / no-allow-non-conformance / repeat-headers / annexb / aud / hrd / info / hash=0 / no-temporal-layers / no-open-gop / min-keyint=5 / keyint=240 / gop-lookahead=0 / bframes=8 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=72 / lookahead-slices=0 / scenecut=40 / radl=0 / no-splice / no-intra-refresh / ctu=64 / min-cu-size=8 / no-rect / no-amp / max-tu-size=32 / tu-inter-depth=4 / tu-intra-depth=4 / limit-tu=1 / rdoq-level=2 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / no-strong-intra-smoothing / max-merge=4 / limit-refs=1 / limit-modes / me=3 / subme=4 / merange=57 / temporal-mvp / no-frame-dup / no-hme / weightp / weightb / no-analyze-src-pics / deblock=-3:-3 / no-sao / no-sao-non-deblock / rd=4 / selective-sao=0 / no-early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=1.20 / no-rd-refine / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=crf / crf=20.0 / qcomp=0.65 / qpstep=4 / stats-write=0 / stats-read=0 / vbv-maxrate=160000 / vbv-bufsize=160000 / vbv-init=0.9 / crf-max=0.0 / crf-min=0.0 / ipratio=1.35 / pbratio=1.25 / aq-mode=2 / aq-strength=0.80 / no-cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=1 / overscan=0 / videoformat=5 / range=0 / colorprim=9 / transfer=16 / colormatrix=9 / chromaloc=1 / chromaloc-top=2 / chromaloc-bottom=2 / display-window=0 / master-display=G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(40000000,50) / cll=3241,902 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / hdr / hdr-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=5 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=1 / refine-ctu-distortion=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-analysis-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei / no-hevc-aq / no-svt / no-field / qp-adaptation-range=1.00
Language : English
Default : Yes
Forced : No
Color range : Limited
Color primaries : BT.2020
Transfer characteristics : PQ
Matrix coefficients : BT.2020 non-constant
Mastering display color primaries : Display P3
Mastering display luminance : min: 0.0050 cd/m2, max: 4000 cd/m2
Maximum Content Light Level : 3241 cd/m2
Maximum Frame-Average Light Level : 902 cd/m2

[Nico8583]

What is your frameserver ? Have you tried to encode lossless ? Or have you tried to encode to x264 ? Just to compare the output.

[redbtn]

Originally Posted by Nico8583

What is your frameserver ? Have you tried to encode lossless ? Or have you tried to encode to x264 ? Just to compare the output.

Vapoursynth. I did some more tests and it seems like there is not enough bitrate for I-frames. Because I compared encoded I-frame vs original P-frame and with bitrate 160000kb/s color is right, but with bitrate 70000kb/s it's wrong. I'm a little confused

[Boulder]

Originally Posted by redbtn

Vapoursynth. I did some more tests and it seems like there is not enough bitrate for I-frames. Because I compared encoded I-frame vs original P-frame and with bitrate 160000kb/s color is right, but with bitrate 70000kb/s it's wrong. I'm a little confused

Do you mean the VBV buffer? 70000 kbps could well be too low. But that value does not mean the same as the bitrate of the encode.
The specs say max rate at 160000 kbps anyway..
https://en.wikipedia.org/wiki/High_E...ers_and_levels

[redbtn]

Originally Posted by Boulder

Do you mean the VBV buffer? 70000 kbps could well be too low. But that value does not mean the same as the bitrate of the encode.
The specs say max rate at 160000 kbps anyway..
https://en.wikipedia.org/wiki/High_E...ers_and_levels

No, I mean Average Bitrate. If I drop CRF to like 10, I-Frames avg bitrate hit 160000 and then it looks good. But if CRF is higher and I-frames avg bitrate only 70000, it still has color artifacts.
VBV buffer = 160000 always. 4K encode

[Boulder]

Have you tried raising the aq-strength substantially, like to 1.5 - 2.0? Or mode 1 at the default?

All in all, I think those issues should also be posted in the x265 issue tracker along with samples.

[redbtn]

Originally Posted by Boulder

Have you tried raising the aq-strength substantially, like to 1.5 - 2.0? Or mode 1 at the default?

All in all, I think those issues should also be posted in the x265 issue tracker along with samples.

Thx. I'll try it in next couple of hours. I'll report later.
I don't know actually, maybe it's normal, cuz this areas is small and it visible only by comparing frames. But I'd like to that 4k encode at 22-24mb bitrate doesn't have colors issues.

[Boulder]

I think those are quite typical "artifacts" of x265 encoding. I was just thinking that AQ can affect such things because it is shifting bits inside the frame. Lowering qg-size to 16 could also change things.
By the way, hdr-opt nullifies your settings regarding the chroma QP offsets so it won't help.

[redbtn]

Originally Posted by Boulder

I think those are quite typical "artifacts" of x265 encoding. I was just thinking that AQ can affect such things because it is shifting bits inside the frame. Lowering qg-size to 16 could also change things.
By the way, hdr-opt nullifies your settings regarding the chroma QP offsets so it won't help.

If I raise aq-strength, what happens? More complex blocks get more bitrate or opposite?
I read the docs and it seems to me that raising aq-strenght give more bits to flat areas.

Adaptive Quantization operating mode. Raise or lower per-block quantization based on complexity analysis of the source image. The more complex the block, the more quantization is used. This offsets the tendency of the encoder to spend too many bits on complex areas and not enough in flat areas.

Adjust the strength of the adaptive quantization offsets. Setting --aq-strength to 0 disables AQ. At aq-modes 2 and 3, high aq-strengths will lead to high QP offsets resulting in a large difference in achieved bitrates.

[Boulder]

A bigger value should shift bits to less complex blocks. Of course, it would be useful to test drastic changes to both directions and see if there is any effect. Then you know if that is the answer to your issue or if it's something else like the psy options.

[benwaggoner]

There was a presentation on optimization of x265 at SMPTE yesterday, as part of the Compression track I chaired.

A couple of specific optimizations available in 3.2 were called out.

First, --selective-sao 3 only applies SAO for I and P frames, but delivers almost all the potential quality benefit of SAO with a ~4% speedup. This follows general literature that SAO isn't helpful with bi-prediction.

I wouldn't be surprised to see that added to --preset slower down the road.

Second, using --rd 4 and --dynamic-rd 3 instead of --rd 6 for CRF with VBV gives full rd 6 quality at peaks, at the cost of slightly higher bitrates (but similar quality) when CRF instead of VBV is the limiting factor. Speedup is estimated at ~9%, but obviously will vary based on how often the VBV cap is the constraint.

[redbtn]

Thank you benwaggoner for interesting information!

But, there is something wrong with HDR encodes. Because, I made a comparison. 8bit SDR (tonemapped with script) versus 10bit HDR (tonemapped after encode with the same script) versus 8bit SDR x264 (same script).

x264 SDR looks very good.
x265 SDR looks even better! No problem at all.
x265 HDR looks bad. Not transparent encode with color issues that I described above.

All encodes at same bitrate.
What I'm doing wrong? I'd like to know how to do good HDR encode.

[Boulder]

Originally Posted by benwaggoner

First, --selective-sao 3 only applies SAO for I and P frames, but delivers almost all the potential quality benefit of SAO with a ~4% speedup. This follows general literature that SAO isn't helpful with bi-prediction.

In layman's terms, does that mean that SAO has actually become usable to us who call it "Smooth All Objects"? B-frames in x265 encodes often seem to be much smoother than in x264 ones, even without SAO enabled.

EDIT: I think you mean --selective-sao 2? Option 3 enables SAO for all reference frames.

[excellentswordfight]

Originally Posted by redbtn

Thank you benwaggoner for interesting information!

But, there is something wrong with HDR encodes. Because, I made a comparison. 8bit SDR (tonemapped with script) versus 10bit HDR (tonemapped after encode with the same script) versus 8bit SDR x264 (same script).

x264 SDR looks very good.
x265 SDR looks even better! No problem at all.
x265 HDR looks bad. Not transparent encode with color issues that I described above.

All encodes at same bitrate.
What I'm doing wrong? I'd like to know how to do good HDR encode.

Please provide samples, and full information of the pipeline and paramenters.

[redbtn]

Originally Posted by excellentswordfight

Please provide samples, and full information of the pipeline and paramenters.

Sample https://gofile.io/?c=1b5LFV
VapourSynth R48 RC1 (pipeline vspipe.exe)
x265 3.2+7 by Barough https://forum.doom9.org/showthread.p...94#post1887694
Simple x264/265 Launcher 2.94 https://github.com/lordmulder/Simple...ncher/releases

Frame № 37
Screens https://gofile.io/?c=FDrVMt (SDR screen is tonemapped source encoded with the same setting exclude HDR setting)
Tonemap.dll https://gofile.io/?c=Pl4Pmj

The best I could do is --aq-mode 1 --aq-strength 0.8 --cbqpoffs -3 --crqpoffs -3

--level-idc 51 --sar 1:1 --colorprim 9 --colormatrix 9 --transfer 16 --range limited --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(40000000,50)" --max-cll "3241,902" --hdr --hdr-opt --hrd --aud --chromaloc 2 --repeat-headers --min-luma 0 --max-luma 1023 --no-cutree --no-sao --no-open-gop --no-strong-intra-smoothing --vbv-bufsize 160000 --vbv-maxrate 160000 --min-keyint 1 --keyint 240 --ipratio 1.4 --pbratio 1.2 --deblock -3:-3 --qcomp 0.65 --aq-mode 1 --aq-strength 0.8 --ctu 64 --merange 57 --me star --limit-refs 1 --no-early-skip --weightb --b-intra --rd 4 --rdoq-level 2 --rc-lookahead 72 --psy-rd 2 --psy-rdoq 1.2 --max-merge 4 --lookahead-slices 0 --bframes 8 --ref 4 --subme 4 --tu-intra-depth 4 --tu-inter-depth 4 --limit-tu 4 --no-rect --no-amp --limit-modes --cbqpoffs -3 --crqpoffs -3

import vapoursynth as vs
core = vs.get_core()
core.max_cache_size = 32768
clip = core.lsmas.LWLibavSource(source="E:\sample.mkv", format="YUV420P10", cache=1)
clip = core.std.AssumeFPS(clip, fpsnum=24000, fpsden=1001)
clip = core.std.CropRel(clip=clip, left=0, right=0, top=280, bottom=280)

# color adjustment using ToneMap
clip = core.resize.Bilinear(clip=clip, format=vs.RGBS, range_in_s="limited", matrix_in_s="2020ncl", primaries_in_s="2020", primaries_s="2020", transfer_in_s="st2084", transfer_s="linear",dither_type="none", nominal_luminance=125)
clip = core.tonemap.Hable(clip=clip, exposure=2.000)
clip = core.resize.Bilinear(clip=clip, format=vs.YUV420P8, matrix_s="709", primaries_s="709",range_s="limited", transfer_s="709", transfer_in_s="linear", range_in_s="limited", dither_type="ordered")

clip.set_output()

[benwaggoner]

Originally Posted by Boulder

In layman's terms, does that mean that SAO has actually become usable to us who call it "Smooth All Objects"? B-frames in x265 encodes often seem to be much smoother than in x264 ones, even without SAO enabled.

I don't know that this changes that. But the SAO detail loss issues have become way less in more recent versions of x265.

EDIT: I think you mean --selective-sao 2? Option 3 enables SAO for all reference frames.

Oh, yes. Thanks for the correction!

[benwaggoner]

Originally Posted by redbtn

Sample https://gofile.io/?c=1b5LFV
The best I could do is --aq-mode 1 --aq-strength 0.8 --cbqpoffs -3 --crqpoffs -3

That's a quite complex command line! I'd start without all the tweaks, and just do something more like:

--preset slower --hdr --hdr-opt

I don't know what value you'd get from a min-keyint of 1 or setting min/max luma.

Level 5.1 has maxrate/bufsize limits of 40000, not 160000.

It appears you've not set CRF OR bitrate, which means it's being encoded at the default CRF, which is quite high. If you're aiming for excellent quality, I'd start with 18 and iterate from there.

--hdr-opt will tweak the chroma QPs better than just setting the offsets. And if you didn't set HDR, you didn't get HDR metadata, and the video is likely playing back as if it had SDR values, which would look terrible!

And don't set --chromaloc unless you've actually changed the chroma location. If you DID change that, let me know what tool you used to do it!

Also, where did you get your MaxCLL and MaxFALL values? A MaxFALL of 900 suggest at least one blindingly bright frame, and no consumer TV available can actually display a MaxCLL of 3200 nits. Not that content doesn't get made that way sometimes, but I'd definitely investigate the content to make sure something didn't go horribly wrong.