Help me understand my encoding test results

[aSquirrel]

I've tried using chatGPT. To say it's dogshit at helping with x265 is an insult to dog shit.

My goals:

Transcode raw blu-ray rips to more space efficient versions (downmix audio to stereo since I don't have 5.1, strip out all non-forced subtitles, transcode video to keep quality with a smaller file)
Maintain as close to rip visual quality as possible.
Encode must be profitable. Meaning I calculate power and storage costs and the encoded file must encode fast enough that the power usage to encode it + the file size reductions must be a net savings.


I have run about 45 tests with ffmpeg libx265. I found that CRF 21, tune grain, preset slow is as near visually lossless as I can get while keeping reasonable savings. If I had to stick with this, it would be ok. But I can't help the nagging feeling that maybe there's something left on the table.

To that end, I ran a bunch of one-off tests with preset slow, CRF 21, so I could investigate the impact various settings had.

19055

That is actually a CSV of my tests.

I have a python script which goes through and extracts specific frames that are known to be problematic for the encoder based on the SSIM score. I also manually extract frame 146733, which is in the Lobby Scene. It's much easier to grade. Lots of subtle shadows, fine detail in the granite pillars, etc. I have 55 MB worth of screen grabs just for frame 146733. I'm willing to share them, I'm just not sure how is best.

On to my actual questions:


crf21, slow, h265_adv_opt_testing ":aq-mode=2:aq-strength=1.5:cutree=0:qg-size=16" || That looks visually BETTER than crf21 tune grain, and encodes faster, and has a smaller result file. But I don't understand why.
Lowering the CRF from 21 to 19, 18, 17 and 16 all made the SSIM score go down?! I don't understand. My understanding was that SSIM was basically a pixel-diff. So the closer to identical the pixels are, the higher the SSIM is. I understand perception is different, SSIM is just used as a tool to sniff out what might be a good spot to check / on a whole how things are going.
CRF 17, no tune, slow looks basically as good as CRF21 tune slow, while encoding (slightly) faster, and being even smaller than aqmode 2.


I guess...what makes more sense, to toss in flags I know make things look better (like rect0, sao0, rskip0) and go with aqmode, or to go with crf 17, those same improving flags...

ugh. The real problem is I have a lot of data but no real understanding of how to put it together. Almost nothing makes the file smaller, but lots makes it bigger. Many settings make the encode slower, but a few speed it up. I don't want to test 100 combinations. Can you look at all this data and help me pick out what I should target for my next tests?

[Z2697]

It means you are probably doing SSIM wrong.

[GeoffreyA]

On a side note regarding the choice of CRF, grain plays a big role. For films without any, one can get away with much lower values. Last week, I encoded Blade Runner 2049 with CRF 18 slow, and got a ridiculously small bitrate of 2183 kbps. If only all films encoded so well.

[excellentswordfight]

Well, on point one. Everyone said in your first thread that tune grain is a flawed and outdated preset and should not be used, and yet you kept using it.

[microchip8]

Don't use tune grain, disable all SAO, lower deblocking to -1 to -3, use AQ mode 1, set subme to 3 or higher, set ctu to 32 and enable cutree. You'll get better visual quality vs encoding time/cost

[Z2697]

Unless you wrote it that way for simplicity, psyrdoq is not a valid argument name. So the "psyrdoq10" result should be completely identical to the "baseline".
But whatever. Good luck reinventing wheels.

(LOL I just realized this is my 777th post and it contains "good luck")

[aSquirrel]

Well, on point one. Everyone said in your first thread that tune grain is a flawed and outdated preset and should not be used, and yet you kept using it.

And yet nobody was able to provide a set of options which look as good or better than it, while also having a faster encode time / smaller result file. I found the aq-mode string from chatGPT. Your insults don't matter when nobody can produce a result that meets or beats.

Don't use tune grain, disable all SAO, lower deblocking to -1 to -3, use AQ mode 1, set subme to 3 or higher, set ctu to 32 and enable cutree. You'll get better visual quality vs encoding time/cost

Those will make it faster, but not look better. The speed comes from smearing vasaline on all the fine details in the background. I can get that with any of a number of options.

Unless you wrote it that way for simplicity, psyrdoq is not a valid argument name. So the "psyrdoq10" result should be completely identical to the "baseline".
But whatever. Good luck reinventing wheels.

(LOL I just realized this is my 777th post and it contains "good luck")

You're on fire today. And by that I mean you're channeling your inner asshole.


#!/bin/bash

early_exit() {
if [ -f /tmp/stopnext ]
then
powerprofilesctl set power-saver
rm /tmp/stopnext
exit 0
fi
}

h265_adv_opt_testing() {
test_arg=$1
sanitized_test_arg=$(echo $test_arg | tr ':' '+')
crf=$2

x265_args="asm=avx512${test_arg}"
outputf="/path/to/encode_test_data/crf${crf}_slow_notune_${test_arg}_input.mkv"
ffmpeg -y -i "/path/to/input.mkv" -map 0:v:0 -c:v libx265 -crf $crf -preset slow -x265-params "$x265_args" -map 0:a:0 -c:a libopus -ac 2 -b:a 160k -metadata encoding_data=slow -metadata encoding_crf=$crf -metadata encoding_x265args="$x265_args" "$outputf" > /path/to/encode_test_data/encode-log_slow_crf${crf}_notune_${test_arg} 2>&1
echo "ffmpeg -i "/path/to/input.mkv" -i \"$outputf\" -lavfi \"ssim=stats_file=/path/to/encode_test_data/ssim_log_crf${crf}_slow_notune_${sanitized_test_arg}\" -f null -" >> /path/to/encode_test_data/ssim_commands.sh

early_exit
}

h265_adv_opt_testing ":aq-mode=2:sao=0:rect=0:psy-rdoq=10:deblock=-1,-1:rskip=0" 19
#h265_adv_opt_testing ":aq-mode=2:aq-strength=1.5:cutree=0:qg-size=16" 21
h265_adv_opt_testing ":aq-mode=2:aq-strength=1.5:qg-size=16:sao=0:rect=0:psy-rdoq=10:rskip=0" 21


Obviously that's not the entire script, but yes, I do actually have the correct commands and name them differently in my file because I'm too lazy to type the special characters. : are special device names in linux, so I change them to +, which is less problematic.

[Z2697]

This is hilarious, what can I say.
I think we've provided enough information that can be easily pieced together, do we need to spoon-feed you the entire command line?
Speed does come with a cost of compression ratio, but the settings in that reply are not particularily some "speed oriented" options.
No-tune-grain, no-sao, lower deblocking and AQ1 creating smear? Maybe you should go visit an optometrist. Or maybe don't smoke weird mushroom.

[GeoffreyA]

And yet nobody was able to provide a set of options which look as good or better than it, while also having a faster encode time / smaller result file. I found the aq-mode string from chatGPT. Your insults don't matter when nobody can produce a result that meets or beats.

Those will make it faster, but not look better. The speed comes from smearing vasaline on all the fine details in the background. I can get that with any of a number of options.

You're on fire today. And by that I mean you're channeling your inner asshole.


Please, let's remain civil.

To answer your questions:

1) AQ2 is going to look better than --tune grain because, as discussed here (https://forum.doom9.org/showthread.php?t=186641), AQ has the effect of giving low-detail macroblocks more bits than high-detail ones. Psychovisually, humans are more sensitive to the former than the latter. Having AQ off, as in --tune grain, causes a constant quantiser across the frame, but the way the maths works, low-frequency information is subject to greater quantisation error. So, AQ corrects this and is psychovisually better. --tune grain is going to shoot up file size, too, with its high --psy-rd and --psy-rdoq values. I think if we want to put the controversy to the test, we should test with and without --tune grain on some extremely grainy material. The Matrix has a bit but not all that much. Criterion's 4K restoration of Lynch's films are a nightmare of grain, as well as Arrow's RoboCop.

2) If there is conflict between CRF and SSIM score, one should defer to lower CRFs being objectively higher quality, simply because they are the direct lever to internal quantisers, the chief lossy part of the compression pipeline.

3) Similarly to (2), lower CRFs, even with faster presets, will tend to beat higher CRFs with slow presets, CRF being the biggest lever in encoding; the smaller dials can't compete once the CRF distance is big enough.

The best advice is to set CRF, preset, and disable SAO, along with the others' recommendations and 10-bit pixel format. Tuning beyond that leads to diminishing returns, or increasingly more energy cost for little gain. A bigger effect can be had through light denoising, either NLMeans or BM3D, and --nr-inter and --nr-intra.

-c:v libx265 -preset slow -crf %crf%
-x265-params level-idc=41:no-sao=1:deblock=-1,-1:aq-mode=1:subme=5:max-merge=5:keyint=240:min-keyint=24:rc-lookahead=240:bframes=8

[GeoffreyA]

Quick test with RoboCop. --tune grain is more consistent but amplifies grain beyond the original. AQ1, whilst varying and having lower-quality regions, feels closer to the original's level of grain.

Files: https://we.tl/t-mObQCczbo8

ffmpeg -i %src% -c:v libx265 -b:v %bitrate% -preset slow -x265-params no-sao=1:deblock=-1,-1:aq-mode=1:pass=1 -f null -
ffmpeg -i %src% -c:v libx265 -b:v %bitrate% -preset slow -x265-params no-sao=1:deblock=-1,-1:aq-mode=1:pass=2 x265_2pass_5000k.mp4

ffmpeg -i %src% -c:v libx265 -b:v %bitrate% -preset slow -tune grain -x265-params pass=1 -f null -
ffmpeg -i %src% -c:v libx265 -b:v %bitrate% -preset slow -tune grain -x265-params pass=2 x265_2pass_tunegrain_5000k.mp4


https://i.imgur.com/l3pDEU3.png
https://i.imgur.com/XFZYGHc.png
https://i.imgur.com/etBYrZL.png

[Z2697]

Please, let's remain civil.

To answer your questions:

1) AQ2 is going to look better than --tune grain because, as discussed here (https://forum.doom9.org/showthread.php?t=186641), AQ has the effect of giving low-detail macroblocks more bits than high-detail ones. Psychovisually, humans are more sensitive to the former than the latter. Having AQ off, as in --tune grain, causes a constant quantiser across the frame, but the way the maths works, low-frequency information is subject to greater quantisation error. So, AQ corrects this and is psychovisually better. --tune grain is going to shoot up file size, too, with its high --psy-rd and --psy-rdoq values. I think if we want to put the controversy to the test, we should test with and without --tune grain on some extremely grainy material. The Matrix has a bit but not all that much. Criterion's 4K restoration of Lynch's films are a nightmare of grain, as well as Arrow's RoboCop.

2) If there is conflict between CRF and SSIM score, one should defer to lower CRFs being objectively higher quality, simply because they are the direct lever to internal quantisers, the chief lossy part of the compression pipeline.

3) Similarly to (2), lower CRFs, even with faster presets, will tend to beat higher CRFs with slow presets, CRF being the biggest lever in encoding; the smaller dials can't compete once the CRF distance is big enough.

The best advice is to set CRF, preset, and disable SAO, along with the others' recommendations and 10-bit pixel format. Tuning beyond that leads to diminishing returns, or increasingly more energy cost for little gain. A bigger effect can be had through light denoising, either NLMeans or BM3D, and --nr-inter and --nr-intra.

-c:v libx265 -preset slow -crf %crf%
-x265-params level-idc=41:no-sao=1:deblock=-1,-1:aq-mode=1:subme=5:max-merge=5:keyint=240:min-keyint=24:rc-lookahead=240:bframes=8

This person is simply doing SSIM wrong, the frames are misaligned.
Part of this is FFmpeg doing its thing with different "timebase's".
With all other things being the same (and without crazy param combos), lower CRF will have both higher subjective and objective quality.

[Z2697]

By the way I don't understand why reinventing wheel is too much an insult for this person.
Maybe this person is ashamed and mad because he/she didn't realize it until I point it out.

[microchip8]

Those will make it faster, but not look better. The speed comes from smearing vasaline on all the fine details in the background. I can get that with any of a number of options.



What smear do you see? These options actually REDUCE smear. Maybe you need to check out your eyesight as many of us are using them and get stunning results!

In fact, it's AQ mode 2 that makes smear in patches of the image because x265 devs decided to deviate from the AQ mode 2 values of x264 and this is the result you get. AQ mode 1 is actually the mode that REDUCES or entirely eliminates smear! Sometimes, the more simple algorithm gives the better visual results!

Also, your attitude is not welcome here. If you can't act civil, go ask somewhere else like on Videohelp or Reddit.

[RanmaCanada]

Ah typical AI user. Everyone else is wrong and AI knows what's best. Your questions have been answered here multiple times throughout the years, and your trust in AI, instead of your fellow humans, who are experts, just shows you have no social skills. Your attitude is not welcome, nor is it warranted.

Please let the door hit you on the way out.

You aren't welcome here.

[aSquirrel]

Ah typical AI user. Everyone else is wrong and AI knows what's best. Your questions have been answered here multiple times throughout the years, and your trust in AI, instead of your fellow humans, who are experts, just shows you have no social skills. Your attitude is not welcome, nor is it warranted.

Please let the door hit you on the way out.

You aren't welcome here.

Yep. People like you drove actually good people away. And let's not forget I'm being asked to be civil while being attacked for stating a standard was not met. Cool.

Nevermind that two long time and prolific posters repeat, word for word, the same (wrong) advice. But I'm, somehow, the AI? Nah dog.

19063
So, that's the master image, straight from the blu-ray with nothing touching it. I will refer to this image as 'master'

19059
This encode uses
x265 [info]: HEVC encoder version 4.1
x265 [info]: build info [Linux][GCC 15.1.1][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 16 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 4 / wpp(17 rows)
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : star / 57 / 3 / 3
x265 [info]: Keyframe min / max / scenecut / bias : 23 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 25 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 0
x265 [info]: References / ref-limit cu / depth : 4 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-21.0 / 0.60
x265 [info]: tools: rect limit-modes rd=4 psy-rd=2.00 rdoq=2 psy-rdoq=1.00
x265 [info]: tools: rskip mode=1 signhide tmvp strong-intra-smoothing lslices=4
x265 [info]: tools: deblock sao dhdr10-info

Or, in simpler terms, -crf 21 -preset slow and that's it. No other flags. This is what I will refer to as 'baseline'.

19060
Now compare the above image to this one, where deblock -1,-1 is the only change.

x265 [info]: HEVC encoder version 4.1
x265 [info]: build info [Linux][GCC 15.1.1][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 16 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 4 / wpp(17 rows)
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : star / 57 / 3 / 3
x265 [info]: Keyframe min / max / scenecut / bias : 23 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 25 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 0
x265 [info]: References / ref-limit cu / depth : 4 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-21.0 / 0.60
x265 [info]: tools: rect limit-modes rd=4 psy-rd=2.00 rdoq=2 psy-rdoq=1.00
x265 [info]: tools: rskip mode=1 signhide tmvp strong-intra-smoothing lslices=4
x265 [info]: tools: deblock(tC=-1:B=-1) sao dhdr10-info

Look at the granite pillars to the left and right of the 'exit' sign. You'll see that, compared to the 'baseline' run, they are blurry. Neo's abs are less well defined. There are a LOT of areas that are noticeably degraded SPECIFICALLY BY DEBLOCK. Because I ran a baseline with all other settings controlled. Like a good little scientist.

And I have screenshots for the other settings I commented on too, but I'm limited to 5 attachments per post, so you're going to have to take my word for it.

19061
I originally thought this was going to be my 'gold standard'. It fixes some of the encoding issues from the master image. For example, the top belt across Neo's thigh. There are block-based artifacts in the master that get smoothed over in the encode and (IMO) look better. But, if you flip between the master and this image, you'll see that the granite texture behind trinity's head is noticeably less well defined. As if it's just slightly out of focus. This is a VERY good encode, to be clear. But not perfect.

x265 [info]: HEVC encoder version 4.1
x265 [info]: build info [Linux][GCC 15.1.1][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 16 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 4 / wpp(17 rows)
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : star / 57 / 3 / 3
x265 [info]: Keyframe min / max / scenecut / bias : 23 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 25 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 0
x265 [info]: References / ref-limit cu / depth : 4 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 2 / 1.5 / 16 / 1
x265 [info]: Rate Control / qCompress : CRF-21.0 / 0.60
x265 [info]: tools: limit-modes rd=4 psy-rd=2.00 rdoq=2 psy-rdoq=10.00 signhide
x265 [info]: tools: tmvp strong-intra-smoothing lslices=4 deblock dhdr10-info

19062
This is what I'm settling on as my 'gold standard' in place of tune grain. Sorry I didn't include a screenshot of grain tune, but trust me, this looks better than grain tuning. CRF goes from 21 to 19, and I drop aq-strength, since that seems to lead to filesize bloat.

The differences are very subtle, but that 2 extra bits of CRF and (maybe) loss of aq-strength seems to be the real ticket to making this look as good as I can. It costs me 1GB of storage compared to using CRF21, but when factoring for power and storage costs that is, worst case, 8 cents on just shy of $2 in savings. So negligable. And it's already 2GB smaller than the grain tunes, on top of being faster to encode, so it ends up being almost 50cents cheaper than tune grain.

x265 [info]: HEVC encoder version 4.1
x265 [info]: build info [Linux][GCC 15.1.1][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
x265 [info]: Main profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 16 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 4 / wpp(17 rows)
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : star / 57 / 3 / 3
x265 [info]: Keyframe min / max / scenecut / bias : 23 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 25 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 0
x265 [info]: References / ref-limit cu / depth : 4 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-19.0 / 0.60
x265 [info]: tools: limit-modes rd=4 psy-rd=2.00 rdoq=2 psy-rdoq=10.00 signhide
x265 [info]: tools: tmvp strong-intra-smoothing lslices=4 deblock dhdr10-info

So, in the end, yes, you were right. Tune grain isn't the best that can be done. That said, none of you recommended these settings, so don't let the door hit y'all on the way out.

[rwill]

It fixes some of the encoding issues from the master image.

Heh...

[Z2697]

Attatchments don't need approval now?

[Z2697]

Good luck inventing triangular wheels. :D

[VoodooFX]

The Toddler Knot solved - CRF 19 is better than CRF 21. (All pros didn't recommended that, shame on you!)

[microchip8]

We told you not to use AQ mode 2 as it can smear patches of the image, yet you insist in using it and complain of "smear". You can't have your cake and eat it at the same time. We also told you other params, yet again you ignore. If this is not 4K, lower CTU to 32. For others, see posts above

your psy-rdoq is too high (10) - it will introduce unwanted noise/distortion/halo'ing. Lower it to something like 2-4. Also rdoq mode 2 is less effective for psy-rdoq. Set it to 1. Disable SAO (strong-intra-smoothing)

This will most likely be my last post in this thread as I'm not willing to help out someone who's stubborn and doesn't listen, yet complains all the time.

[Z2697]

Yeah I also told this person not to use slices, which he of course did listen and then just went ahead and insult me.
If there's no this slices option in the previous thread, I would not think that this person is very dare to sacrifice compression efficiency for speed, thus I focused on the speed optimization.
And now this person is complaining we don't give him a setting that can increase both speed and compression efficiency. (judging from the notes in the "test results")

This is nuts.
https://files.catbox.moe/mvffaz.jpg

[Swede]

Ok, everybody, especially aSquirrel, cool it down a bit or I will start to handle out strikes. :cool:

[benwaggoner]

Don't use tune grain, disable all SAO, lower deblocking to -1 to -3, use AQ mode 1, set subme to 3 or higher, set ctu to 32 and enable cutree. You'll get better visual quality vs encoding time/cost
--tune grain is really a legacy of x265's 1.x era, and really should be removed now. It's not been adjusted for all the new infrastructure.