x265 settings for re-encoding 4K BD near-losslessly

[coopzr]

Hello brains trust,

I need to re-encode a movie to bake the DV-FEL into the HDR10 base layer (the FEL expands brightness).

I struggle to find the right settings for this situation because, unlike most other use cases, I'm not doing this to save space. I'd like a near lossless quality without blowing up the file size (--lossless expanded my 70GB movie to 700GB).

The settings I have so far:
--preset slower --crf 10 --aq-mode 1 --rd 4 --psy-rdoq 1.0 --psy-rd 2.0 --profile main10 --level-idc 5.1 --high-tier --output-depth 10 --range limited --hdr10 --colorprim bt2020 --colormatrix bt2020nc --transfer smpte2084 --hdr10-opt --repeat-headers --hrd --aud --deblock -1:-1 --max-luma 1023 --no-sao --chromaloc 2 --vbv-maxrate 160000 --vbv-bufsize 160000 --sar 1 --no-open-gop
(Most of these settings are from Dovi.Scripts (https://forum.doom9.org/showthread.php?t=185317))

Do you recommend any changes?

I think I recall seeing benwaggoner recommend --cbqpoffs -1 and --crqpoffs -1 for HDR content. Should I do that here?

Is aq-mode 1 the right choice?

Any help is greatly appreciated. :)

[excellentswordfight]

Hello brains trust,

I need to re-encode a movie to bake the DV-FEL into the HDR10 base layer (the FEL expands brightness).

I struggle to find the right settings for this situation because, unlike most other use cases, I'm not doing this to save space. I'd like a near lossless quality without blowing up the file size (--lossless expanded my 70GB movie to 700GB).

The settings I have so far:
--preset slower --crf 10 --aq-mode 1 --rd 4 --psy-rdoq 1.0 --psy-rd 2.0 --profile main10 --level-idc 5.1 --high-tier --output-depth 10 --range limited --hdr10 --colorprim bt2020 --colormatrix bt2020nc --transfer smpte2084 --hdr10-opt --repeat-headers --hrd --aud --deblock -1:-1 --max-luma 1023 --no-sao --chromaloc 2 --vbv-maxrate 160000 --vbv-bufsize 160000 --sar 1 --no-open-gop
(Most of these settings are from Dovi.Scripts (https://forum.doom9.org/showthread.php?t=185317))

Settings looks fine, only thing that stands out is no-open-gop, in my experience open-gop support in hevc playback is excellent, I rarely see any reason why not to use it (if not for example adaptive streaming).

But just an FYI, at crf10, its likely that a lot of content will end up at a higher bitrate then the source. And I would say that its still pretty much visually lossless at crf15 so I would probably start there (still grainy titles can probably blow the bitrate over the source even at 15).

Is aq-mode 1 the right choice?

mode 1 is great for high quality SDR, but I do not find it as good for HDR, for hdr at that bitrate level im sure the default mode 2 will do just fine.

[microchip8]

mode 1 preserves noise/grain better while the other modes make it bounce a bit, doesn't matter if SDR or HDR.

I'd also increase subme to 7 as I found out that it makes the image a bit sharper while the lesser modes give a bit softer image. The speed penalty of subme 7 is about 2-4%

Sometimes, even in 10-bit mode, x265 can introduce banding. In such cases, increase psy-rd to 4 and psy-rdoq to 15 and use rdoq-level 1

[Z2697]

Forget what he said.
AQ mode 1 doesn't preserve noise/grain better, or at least not specific to noise/grain.
Subme doesn't "alter" the image so to speak, subpixel motion estimation is subpixel motion estimation. (in x264 --subme is more than subpixel motion estimation but still doesn't alter the image)
I'm not sure what is that on the last line.

[hellgauss]

At that crf i would
1) turn off strong-intra-smoothing, which is on by default
2) Lower psy-rd , try at least 1.5 (this save bitrate a little, and avoid buffer saturation)
3) Perhaps, if you have time, try veryslow. It has been fine tuned in last builds and for my experience (anime encoding) it is worth the effort with respect to slower. However i did not test with that low crf.

[microchip8]

Forget what he said.
AQ mode 1 doesn't preserve noise/grain better, or at least not specific to noise/grain.


I beg to differ and so are others on this forum.

Subme doesn't "alter" the image so to speak, subpixel motion estimation is subpixel motion estimation. (in x264 --subme is more than subpixel motion estimation but still doesn't alter the image)

Encoding same video with same params, one with subme 4 and one with 7, the 7 one looks a bit sharper to me. Yes, I have good eyes. I do multiple encodes a day and can confirm what I see. And I speak about x265, not x264.

I'm not sure what is that on the last line.

Maybe get a clue?

[Z2697]

The whole (core) logic of subme is:
https://bitbucket.org/multicoreware/x265_git/src/8f11c33acc267ba3f1d2bde60a6aa906e494cbde/source/encoder/motion.cpp#lines-1489:1594
Needless to say not gonna do anything regarding sharpness.

Many things can "look like banding". I'm not sure which is being mentioned.
"Real banding" in narrow sense is the lack of precision, which "by definition" is not the case here: x265 doesn't alter the bitdepth (in case of 10-bit mode with 10-bit input), so it's impossible to introduce banding.

[Z2697]

I'd recommend AQ mode 3 when using hdr10-opt because it sets a fixed positive QP bias to blocks with average luma value less than 434.
Chroma offsets are also important as well.

Or maybe they are not that important when the crf is 10, whatever.

[microchip8]

I've had multiple cases where x265 introduced banding in the picture that was not there in the input when comparing the encoded frames with the original input ones. It was clearly visible and the only way to get rid of it while preserving all other encoder options was to increase psy-rd/rdoq values.

Why do you think I'm using these options/values? Just for fun? No. It's because they fix a problem I encountered and have the "side effect" of making the picture look stunning. I don't care so much for theory or algorithms, I care for what I see.

As for aq 1, it is not so much that it preserves noise better. It's the fact that it makes it less pulsating. You can do a search on here and you'll see a few topics that confirm it.

[microchip8]

Please, don't hijack threads! Start your own!

[coopzr]

After a few weeks of testing I've arrived at the following settings so far:

--preset slow --crf 12 --rskip 2 --rskip-edge-threshold 1 --aq-mode 3 --cbqpoffs -1 --crqpoffs -1 --keyint 240 --min-keyint 24 --rc-lookahead 96 --no-sao --profile main10 --level-idc 5.1 --hdr10 --hdr10-opt --repeat-headers --hrd --aud
* I just use --rskip for these encoding tests. It will be turned off when I do my final encode as it makes the pnsr, ssim and vmaf numbers bigger.


The results are as follows:
https://i.ibb.co/LdmmN5QW/F5-B58289-D73-F-4-B82-86-F4-3-F5578-C18-FAE.png (https://ibb.co/F422HVsM)


I don't even know what the numbers mean or what a good range for the numbers are. I just know higher = better.

I see that VMAF min is a bit lower than the avg. What settings can I use to bring that up?

My testing and evaluation goes something like this:
1. Make a small change to settings
2. If PSNR, SSIM and VMAF increase, keep it.
3. Repeat.

Is my testing and evaluation procedure flawed?

Have I wasted weeks of encoding for indistinguishable quality gains?

Do I need someone to slap me back to reality and tell me to stop hyperfixating on making result numbers go bigger?

The movies I intend to encode are The Hunger Games movies (excl. the most recent) as, for some reason, the source encode misbehaves when generating new DV metadata. Therefore I have to bake FEL into a new HDR10 base layer.

As a reminder, I'm not doing this to save space. If my encode is the same size as the source or less I am happy.

[GeoffreyA]

I think the settings are fine and that is a sound procedure, making one change, evaluating, and moving to the next. While these metrics are good as a guide, also double check with the eyes.

At CRF 12, other settings won't matter too much, the bitrate being so high and dwarfing everything else.

deblock -1:-1 can be a good companion to no-sao, though at CRF 12, perhaps it doesn't matter. A higher subme, such as 5, can be helpful too. But again, at this point, CRF is running the show.

[coopzr]

I think the settings are fine and that is a sound procedure, making one change, evaluating, and moving to the next. While these metrics are good as a guide, also double check with the eyes.

At CRF 12, other settings won't matter too much, the bitrate being so high and dwarfing everything else.

deblock -1:-1 can be a good companion to no-sao, though at CRF 12, perhaps it doesn't matter. A higher subme, such as 5, can be helpful too. But again, at this point, CRF is running the show.

After trying your suggested settings the results scored lower. As you said, CRF is likely running the show.

https://i.ibb.co/hRtP1LcX/FCC6-CB92-E64-F-421-A-93-DA-74-F8-F36-AD228.png (https://ibb.co/pv9trbQW)

I'd still like to increase the VMAF min, any ideas?

[excellentswordfight]

Is my testing and evaluation procedure flawed?

Yes, I would not recommend using CRF for tests like this, use 2pass VBR so you can get consistent bitrate between setting changes, as CRF cannot be seen as fixed quality level between different settings, and as settings can have a drastic impact on bitrate at the same crf value, making it hard for you to difference between compression gains and bitrate-changes. When you have dialed in your settings, you can move over to CRF to get consistent quality between different titles.

[tormento]

I'd still like to increase the VMAF min, any ideas?
Remember that VMAF, as other synthetic indexes, if I remember correctly, uses luma only.

[x264N00b]

I'd still like to increase the VMAF min, any ideas?

Encoder Settings from the original Post are fine and a VMAF min of 94 is also pretty good. Personal I like to boost the b-frame quality always a bit with decreasing the pb-ratio slightly. I also get no benefit from disabling SAO when high bitrates are used. Preset slow should be minimum. Too get a better VMAF min you need to boost the bitrate for the affected scenes.
In my experience those scenes are often fade in/outs, scenes with low contrast and red light scenes.

Now the work starts ;) It's recommend to detect the scenes with the lowest VMAF und bitrate first. You already got the tool FFmetrics and it's graph shows you exactly the framenumbers. The nice guy (fifonik) who made FFmetrics also made another cool tool, FFbitrateviewer. You can analyze your encodes bitrate in detail (time based view for this is good). Just look into the scenes and find out if there is really a loss of quality.
Personal I don't trust VMAF so much, so I use AvsPmod which is a script editor und previewer for avisynth to A/B compare source and encode frame wise. (important: to properly compare encodes it's highly recommended to look at the b-frames)
To find out if a frame is a I, P or B-frame in AvsPmod the plugin FFmpegSource2 (ffms2) can be used. Frametype will be shown on the overlay.

Now source and encode can be loaded in AvsPmod in seperate tabs and frames can be compared easily with jumping through the tabs with the keyboard numbers.

The avisynth script to use:
Source=FFVIDEOSOURCE("path of the source videofile")
Interleave(Source)
ffinfo(framenum=true,frametype=true,cfrtime=true,vfrtime=false,version=false,cropping=false,colorrange=false,colorspace=false,sar=false)

If the video comes with HDR it must be tonemapped of course. It can be done pretty easy with the libplacebo avisynth plugin and adding some lines to the script above.

ConvertBits(16)
libplacebo_Tonemap(src_csp=1, dst_csp=0, dynamic_peak_detection=true, tone_mapping_function="spline", contrast_recovery=0.3, contrast_smoothness=3.5)

So if you are not satisfied with the quality of a scene, just figure out it's start und endframe with AvsPmod or the Staxrip previewer and define zones in x265 to boost the bitrate.

--zones <zone0>/<zone1>/...
<start frame>,<end frame>,<option>

Example frame 5000 to 10000 with bitrate multipler of 1.8 and frame 6000 to 7000 with qp (not possible to use crf for zones with x265) of 18:
--zones 5000,10000,b=1.8/6000,7000,qp=18


Another way would be to encode the source with a more constant bitrate, like studios like Arrow, Shout or Eagle Pictures do it usually. Unfortunately x265 has no feature for this and I don't know how it's made properly. A workaround is to set a very low constant rate factor and limit the bitrate with vbv-bufsize and maxrate to lets say 100000k.
But I'm not sure if it will hurt the quality, if the encoder permanently runs into this maxrate limit. Maybe someone with more knowledge could tell us? So honestly I don't know how the professional studios get such a nice nearly constant bitrate like on Timecop (https://caps-a-holic.com/c.php?d1=19033&d2=19137&c=1711)

To get the bitrate a bit more constant and to boost the quality in low contrast scenes, it also can help to disable Cutree and lowering pb-ratio to 1.20. Also decreasing Qcomp to 0.5 could have an positive effect, because it makes the bitrate less "jumpy".


Have I wasted weeks of encoding for indistinguishable quality gains?

Yeah pretty much. I think fine tuning thinks like deblock will not increase your min VMAF noticeable, for that a overall slower preset is probably needed or more bitrate in the affected scenes. Maybe try a "no Cutree" encode for fun. When avisynth is used, I would also use a helper like SelectRangeEvery(6000, 140) or something to figure out my settings and get an idea, how the final encode would be. (avg. bitrate)

[benwaggoner]

Remember that VMAF, as other synthetic indexes, if I remember correctly, uses luma only.
That is correct for VMAF. There are some that due include chroma, like x265's own PSNR-YUV you can get via --log-level 1 or 2.

VMAF is also sensitive to around 3 units; people should not assume that 99.9 and 99.2 or even 98 and 96 would have any subjectively apparent difference when playing at 100% speed at 100% zoom.

[benwaggoner]

Now the work starts ;) It's recommend to detect the scenes with the lowest VMAF und bitrate first. You already got the tool FFmetrics and it's graph shows you exactly the framenumbers. The nice guy (fifonik) who made FFmetrics also made another cool tool, FFbitrateviewer. You can analyze your encodes bitrate in detail (time based view for this is good). Just look into the scenes and find out if there is really a loss of quality.
Personal I don't trust VMAF so much, so I use AvsPmod which is a script editor und previewer for avisynth to A/B compare source and encode frame wise. (important: to properly compare encodes it's highly recommended to look at the b-frames)
To find out if a frame is a I, P or B-frame in AvsPmod the plugin FFmpegSource2 (ffms2) can be used. Frametype will be shown on the overlay.
Of course, a b-frame will generally look a lot worse when paused than during playback. reducing pbratio is a good idea when you get "strobing" where you can see the p/b transitions. This is a much bigger problem with grainy/noisy conent, which reducing ipratio and pbratio are almost essential to get good psychovisual compression efficiency. Some of those "grain strobing" like problems can be harder to detect on pause, so it's important to do QA at 100% speed, which can detect a lot of issues that frame-based metrics are blind to.

If the video comes with HDR it must be tonemapped of course. It can be done pretty easy with the libplacebo avisynth plugin and adding some lines to the script above.
Much better to just use a HDR monitor and watch in native HDR, though! I use Adobe After Effects to do this myself, out to a LG C4 in Filmmaker Mode. I put the source and different encodes to compare as different layers which I can flip on/off to see

Another way would be to encode the source with a more constant bitrate, like studios like Arrow, Shout or Eagle Pictures do it usually. Unfortunately x265 has no feature for this and I don't know how it's made properly. A workaround is to set a very low constant rate factor and limit the bitrate with vbv-bufsize and maxrate to lets say 100000k.
But I'm not sure if it will hurt the quality, if the encoder permanently runs into this maxrate limit. Maybe someone with more knowledge could tell us? So honestly I don't know how the professional studios get such a nice nearly constant bitrate like on Timecop (https://caps-a-holic.com/c.php?d1=19033&d2=19137&c=1711)
Setting vbv bufsize/maxrate to higher than CRF would have generated won't impact quality. You can even use --crf and --bitrate together for a "hit this quality target unless it would exceed this average bitrate." Bufsize and Maxrate should really be determined by HW decoder compatibility and network throughput. If they aren's specified, they are set based on Profile @ Level. In this example of Main10 Level 5.1 they would both be set to 40000 Kbps, which is generally fine for 24p filmic content with good compression settings. The peak for 4K Blu-ray is 80 Mbps, and that's with a much more constrained profile that reduced encoding efficiency quite a bit.

Yeah pretty much. I think fine tuning thinks like deblock will not increase your min VMAF noticeable, for that a overall slower preset is probably needed or more bitrate in the affected scenes. Maybe try a "no Cutree" encode for fun. When avisynth is used, I would also use a helper like SelectRangeEvery(6000, 140) or something to figure out my settings and get an idea, how the final encode would be. (avg. bitrate)
Yeah, there is much less to be gained by tuning in-loop deblocking and such compared to H.264; 0, 0 was better calibrated for a variety of scenarios. Garry Sullivan once told me his biggest design regret from H.264 was the default loop filter being too high for moderate-high bitrate scenarios. For almost all encodes, whether or not to use SAO is the bigger question. It's generally better for lower bitrates while worse for higher bitrates.

[Blue_MiSfit]

Much better to just use a HDR monitor and watch in native HDR, though! I use Adobe After Effects to do this myself, out to a LG C4 in Filmmaker Mode.


Similar here, using DaVinci Resolve and a Decklink interface to render the HDMI.

[benwaggoner]

Similar here, using DaVinci Resolve and a Decklink interface to render the HDMI.
Yep. No substitute for looking at the actual pixels displayed as is!

[FranceBB]

Similar here, using DaVinci Resolve and a Decklink interface to render the HDMI.

Almost the same, but in the office the Blackmagic Decklink is outputting via SDI instead.

https://images2.imgbox.com/23/33/vCeqg6qQ_o.pnghttps://images2.imgbox.com/fb/69/9EFN5CdD_o.png

[benwaggoner]

Almost the same, but in the office the Blackmagic Decklink is outputting via SDI instead.

https://images2.imgbox.com/23/33/vCeqg6qQ_o.pnghttps://images2.imgbox.com/fb/69/9EFN5CdD_o.png
I've done it that way too. With HDMI 2.1, I can get all the HDR10+ and DoVi metadata working like it would in the field, which is also useful. My old rig let me do it both ways, and I was surprised to find myself not really going back to SDI almost ever.

[blublub]

CRF=12 - thats crazy. At this setting does it even make sense to encode/re-encode something? - I mean file size and time taken into account?

A few years back I did quite some test and depending on source material I could not see a diference with 4K encodes at CRF 19, so I stuck with 17 or 18 depending on source material.

Thats what I used so far:
no-sao:rd=4:psy-rdoq=4:me=3:subme=2:aq-mode=1:rskip=2: qcomp=0.75:bframes=4:ref=5:psy-rd=2:deblock=-1,-1:level-idc=51:high-tier:rc-lookahead=30:vbv-bufsize=160000:vbv-maxrate=160000:asm=avx512:frame-threads=8:hdr10-opt

[benwaggoner]

CRF=12 - thats crazy. At this setting does it even make sense to encode/re-encode something? - I mean file size and time taken into account?
Almost certainly not. I've used CRF=12 to make a new mezzanine to incorporate some video processing, sure. But it'll be bigger than the source for most examples.

[rwill]

Hello brains trust,

I need to re-encode a movie to bake the DV-FEL into the HDR10 base layer (the FEL expands brightness).


Sounds pretty mezzanine style to me.