x264 vs x265

[Bluedraft]

Encoding the same video in x264 10 bits give me better quality than using x265 10 bits, I'm using handbrake.

Original
https://i.ibb.co/b5Z7rDCp/original.png (https://ibb.co/b5Z7rDCp)

x264 crf 18 / tune grain / very slow
https://i.ibb.co/Cdf19MP/264crf18.png (https://ibb.co/Cdf19MP)

x265 crf 18 / tune grain / very slow
https://i.ibb.co/ymnLwkW3/265-crf18.png (https://ibb.co/ymnLwkW3)

Not only is the grain better preserved with x264 but it hardly distorts the image as it does x265 (look at the clothes rack behind the girl) and the x265 output file is not much smaller.

Are there any special settings that I need to modify to get the same quality in x265? I would prefer x265 because x264 10bit doesn't have to much hardware support.

[StainlessS]

Third (x265) image does not zoom (others do).

[VoodooFX]

x264 is better on such high bitrates like CRF18. x265 is better on medium-low bitrates maybe starting from ~CRF23 and up.

Try --no-sao with x265, video should be less softer without sao.

[Bluedraft]

Third (x265) image does not zoom (others do). Hello StainlessS, try now please

x264 is better on such high bitrates like CRF18. x265 is better on medium-low bitrates maybe starting from ~CRF23 and up.
Try --no-sao with x265, video should be less softer without sao.

I want to at least reach the quality of the x264 pic, raising up the crf in x265 will give me better size/quality ratio than x264 at the same bitrate but not the quality I'm looking for.
x264 8 bit (which is super hardware compatible) gives me almost the same quality as the 10 bit version but generates horrible banding in dark and flat scenes.

I've already set --no-sao, also play a little with rd, rdoq, psy-rdoq, aq-strength, aq-mode, q-comp and zones.
Not much difference.

[Sunspark]

Link 1 and Link 3 are the same 265 image.

[Bluedraft]

Link 1 and Link 3 are the same 265 image.Please try now

[Leo 69]

I assume there's something wrong as there shouldn't be any drastic differences between these codecs at these settings. I've been encoding video for many years and experimented with pretty much all codecs, especially with x264 and x265 and all I can say is that at CRF 18 they are both nearly transparent. x264 does keep slightly more details, but the difference is so miniscule that most people won't ever see it. I pixel-peeped and compared these codecs visually at various settings numerous times.

Handbrake is a nice tool, but you know - bugs exist. Did you examine the encoding settings for x265 that they indeed correspond to the preset 'very slow' and tune 'grain' was enabled? Which build of Handbrake have you been using?

Also, links to your images are dead. It would also be helpful if you could provide a short original video sample that you're having these issues with.

[Blue_MiSfit]

The HEVC benefit is smaller with grainy content, certainly, and especially at a higher quality standard. Cut your bitrate in half and you'll prefer the x265 version most likely. AV1 with FGS even more!

Also, still frame comparisons are not always a good way to do things. Given two compressed versions of the same source, one may appear somewhat softer in some still frame comparisons, but it may have better stability in motion. You can abuse this and make video that's soft and smooth, but there has to be a balance :)

In your case, you really want 10 bit. As you pointed out, 10 bit AVC isn't supported in hardware usually, so HEVC brings that to the table. Seems a pretty clear answer, no?

[Bluedraft]

I assume there's something wrong as there shouldn't be any drastic differences between these codecs at these settings. I've been encoding video for many years and experimented with pretty much all codecs, especially with x264 and x265 and all I can say is that at CRF 18 they are both nearly transparent. x264 does keep slightly more details, but the difference is so miniscule that most people won't ever see it. I pixel-peeped and compared these codecs visually at various settings numerous times.

Handbrake is a nice tool, but you know - bugs exist. Did you examine the encoding settings for x265 that they indeed correspond to the preset 'very slow' and tune 'grain' was enabled? Which build of Handbrake have you been using?

Also, links to your images are dead. It would also be helpful if you could provide a short original video sample that you're having these issues with.

Links to the images samples are working fine here, must be some issue in your browser

I'm using Handbrake 1.9.2 (2025022300), the latest

I'm also using Staxrip which give more control over settings and got the same results in terms of quality and filesize


The HEVC benefit is smaller with grainy content, certainly, and especially at a higher quality standard. Cut your bitrate in half and you'll prefer the x265 version most likely. AV1 with FGS even more!

Also, still frame comparisons are not always a good way to do things. Given two compressed versions of the same source, one may appear somewhat softer in some still frame comparisons, but it may have better stability in motion. You can abuse this and make video that's soft and smooth, but there has to be a balance


In your case, you really want 10 bit. As you pointed out, 10 bit AVC isn't supported in hardware usually, so HEVC brings that to the table. Seems a pretty clear answer, no?

As far as x265 gives me the quality picture of x264 10 bit I've no problem, I did many test with x265 from crf 16 to crf 23 although differences appeared between the results of these test in terms of quality non of the output files are free of that distortion and grain lost.

10 bits is always better and like you said due to the widespread support x265 seems to be the right choice, but I don't want those artifacts that produces at any reasonable bitrate, that's why I'm asking for some help / tips / advices

I also tried AV1 at preset 4, 5 and 6 and it directly kills the grain
I did all the test twice, one in handbrake 1.9.2 and other in Staxrip 2.46.0

[Z2697]

tuen grain should never be used

[VoodooFX]

I want to at least reach the quality of the x264 pic, raising up the crf in x265 will give me better size/quality ratio than x264 at the same bitrate but not the quality I'm looking for.
x264 8 bit (which is super hardware compatible) gives me almost the same quality as the 10 bit version but generates horrible banding in dark and flat scenes.
With x264 8 bit try --aq-strength 1.1 or 1.2, if that doesn't help then you should look at Avisynth filters like GradFun3 and similar.



I also tried AV1 at preset 4, 5 and 6 and it directly kills the grain

Yes, it basically kills fine details, imo, AV1 is useless for home users, and looking at what state it's now I think it will never be better than x265, that's for resolutions lower than 4K and no-nonsense bitrates, as I'm not in touch with that realm and dunno how things are there.

[Z2697]

With x264 8 bit try --aq-strength 1.1 or 1.2, if that doesn't help then you should look at Avisynth filters like GradFun3 and similar.



Yes, it basically kills fine details, imo, AV1 is useless for home users, and looking at what state it's now I think it will never be better than x265, that's for resolutions lower than 4K and no-nonsense bitrates, as I'm not in touch with that realm and dunno how things are there.

Current AV1 encoders, when given high (or even mid) bitrate, does not preserve enough detail that deserve the bitrate you gave it.
Other than that, it has, I'd say plenty use cases for at least some home users (me not included, however). It has rather niche use case for me, but not completely useless.

[Z2697]

If you are caring THAT much about grains, then you should probably NOT re-encode it. Unless the source is a "raw footage" (lossless / uncompressed, or mezzanine format).

[excellentswordfight]

tuen grain should never be used
Thank you.

Also, crf is not a "constant quality" setting in that sense that the same value can be compared across different codecs as a metric for quality, heck it cannot even be compared within the same codec using different presets. I would recommend using 2pass avg bitrate when doing test like these, as settings changes can have a drastic impact on overall bitrate which make evaluation very hard. Tuning in an crf that correspond compression level/bitrate can than be used when you are happy with the image quality.

If you want something comparable to x264:s tune film preset, use --no-sao --deblock -1:-1, instead of tune grain. If you then wanna tune even more for grain, you can start increasing --psy-rd and --psy-rdoq (something like --psy-rd 4 and --psy-rdoq 2 should be a good startingpoint).

I also prefer to use --aq-mode 1 for x264 style high quality (i.e. low crf) SDR encodes.

[Stereodude]

FWIW, I found 10-bit HEVC encoding by x265 did not play that nicely with 8-bit source content some years ago if -F was greater than 1 on the encode. However, 10-bit source content was much better in that regard. With the latest versions of x265 and 10-bit source content I don't see an issue with -F greater than 1. 10-bit H.264 works fine for standard def with software decoding. Software decoding is more of a problem with FHD (and higher) 10-bit H.264 content.

Turning 8-bit source content into "real" 10-bit source content is not the most trivial thing. Usually you need some sort of high bit depth processing step(s) that will output appreciable content into the in between luma and chroma steps.

[Z2697]

x264's tune grain is after something actually, but x265's is kinda insane...

--aq-strength 0.5 --no-dct-decimate
--deadzone-inter 6 --deadzone-intra 6
--deblock -2:-2 --ipratio 1.1
--pbratio 1.1 --psy-rd <unset>:0.25
--qcomp 0.8

no-dct-decimate, deadzone=6:6, and deblock=-2:-2, that's generally good for high bitrate image quality, the rest is some psychovisual "enhancement" I guess.


param->rc.ipFactor = 1.1;
param->rc.pbFactor = 1.0;
param->rc.cuTree = 0;
param->rc.aqMode = 0;
param->rc.hevcAq = 0;
param->rc.qpStep = 1;
param->rc.bEnableGrain = 1;
param->recursionSkipMode = 0;
param->psyRd = 4.0;
param->psyRdoq = 10.0;
param->bEnableSAO = 0;
param->rc.bEnableConstVbv = 1;

I don't see any thing generally good except no-sao, which is already the de facto standard for high bitrate x265 custom encoding settings...
Sure enough, grain-rc is a specifically designed rc option, but it's focusing more on "let QP not fluctuate too much". Same as ipb-factors and qpstep.
Disableing rskip can be in the "generally good" category, but it usually not noticeable.
As the documentation said, "The purpose of this option is neither to retain nor eliminate grain, but prevent noticeable artifacts caused by uneven distribution of grain."

[VoodooFX]

I also prefer to use --aq-mode 1 for x264 style high quality (i.e. low crf) SDR encodes.

Oh yeah, I think --aq-mode 1 in x265 is a must for any CRF/resolution.
Years back I was scratching my head why suddenly my x265 HD encodes started to look like shit at some random scenes, culprit was that in some update x265 changed its aq-mode default from 1 to 2...

[Bluedraft]

Thank you.

Also, crf is not a "constant quality" setting in that sense that the same value can be compared across different codecs as a metric for quality, heck it cannot even be compared within the same codec using different presets. I would recommend using 2pass avg bitrate when doing test like these, as settings changes can have a drastic impact on overall bitrate which make evaluation very hard. Tuning in an crf that correspond compression level/bitrate can than be used when you are happy with the image quality.

If you want something comparable to x264:s tune film preset, use --no-sao --deblock -1:-1, instead of tune grain. If you then wanna tune even more for grain, you can start increasing --psy-rd and --psy-rdoq (something like --psy-rd 4 and --psy-rdoq 2 should be a good startingpoint).

I also prefer to use --aq-mode 1 for x264 style high quality (i.e. low crf) SDR encodes.

Thanks for your help excellentswordfight!

I tried aq-mode 1 and surprisingly it did have a notorious positive impact on the picture reducing artifacts and distortion in the part of the image where the coat rack is located.

x265 2pass aq-mode 1
https://i.ibb.co/jPQqLZBZ/x265-crf18-aqm1.png (https://ibb.co/jPQqLZBZ)

[microchip8]

aq-mode 1 is best for both x264 and x265. All other variance-based and bias-based modes make the picture worse while slurping up bitrate

[Z2697]

I'm not sure if you are still using tune grain, but the rc-grain in tune grain is not meant to be used with CUTree or AQ.

[Bluedraft]

I'm not sure if you are still using tune grain, but the rc-grain in tune grain is not meant to be used with CUTree or AQ.

In the last picture sample I didn't. I use 2 pass encoding with these settings

--bitrate 15000 --preset very slow --output-depth 10 --psy-rd 4 --psy-rdoq 2 --aq-mode 1 --colorprim bt709 --colormatrix bt709 --transfer bt709 --range limited --deblock -1:-1 --no-sao

Besides the difference in grain uniformity between x264 and x265 I noticed also that x265 produce some random greenish spots in the grain, any idea what could be causing this?

[Z2697]

In the last picture sample I didn't. I use 2 pass encoding with these settings

--bitrate 15000 --preset very slow --output-depth 10 --psy-rd 4 --psy-rdoq 2 --aq-mode 1 --colorprim bt709 --colormatrix bt709 --transfer bt709 --range limited --deblock -1:-1 --no-sao

Besides the difference in grain uniformity between x264 and x265 I noticed also that x265 produce some random greenish spots in the grain, any idea what could be causing this?

x264 has automated chroma qp offset based on psy-trellis and is at least -2 (or at most? I mean it gets more negative) IIRC, x265 only has "half the chroma quality if it's 4:4:4". (offsets are set to +6 if not specified through command line)

So you probably need to set a negative offset in x265.
With the settings by tune grain, x264 calculates the chroma qp offset as -4. I guess it's reasonable to mimic that in x265. (because as I said earlier, unlike x265's tune grain, x264's tune grain actually means something)
x265 has 2 offset arguments for 2 chroma planes.
--cbqpoffs -4 --crqpoffs -4

[Bluedraft]

Wonderful, I'll try it, thanks a lot Z2697!

[benwaggoner]

I'm not sure if you are still using tune grain, but the rc-grain in tune grain is not meant to be used with CUTree or AQ.
Or used with anything, really. It's a really old hack that hasn't been needed since at least x265 2.3.

[excellentswordfight]

In the last picture sample I didn't. I use 2 pass encoding with these settings

--bitrate 15000 --preset very slow --output-depth 10 --psy-rd 4 --psy-rdoq 2 --aq-mode 1 --colorprim bt709 --colormatrix bt709 --transfer bt709 --range limited --deblock -1:-1 --no-sao

Besides the difference in grain uniformity between x264 and x265 I noticed also that x265 produce some random greenish spots in the grain, any idea what could be causing this?
Out of curiosity, is it still present if you only change preset to slow or slower? Since they changed the presets a long time ago, making veryslow almost a placebo preset its not that widely used, making it more likely that some odd behavior caused by a combination of settings to go unnoticed.

[VoodooFX]

@Bluedraft

As your initial problem is with x264 8 bit, did you tried my suggestions?