Alternative to x265?

[DJ Bobo]

Hi guys

I was wondering if there is a viable alternative to x265 in 2026, as I am not satisfied with its encoding quality in 8-bit mode, and I hate the longer encoding times in 10-bit mode (for nothing really; I only use 8-bit sources)

I have a DivX 10 Pro license, but its HEVC quality is horrible. Aynbody tried DivX 11 Pro?!

And the newer TMPGEnc Video Mastering Works versions use x265 internally, so it is not a real alternative.

The software shouldn't be too expensive (~100€ range).

EDIT: Has to be an HEVC encoder of course. I am not interested in alternative codecs (as many seem to have misunderstood my question)

EDIT2: Suggestions so far:

ks265codec
Kvazaar
Using x265 with an alternative adaptive quantization setting (--aq-mode 1 or 3)

[Columbo]

x264 won't work for you?

[DJ Bobo]

@ Columbo
I am obviously looking for the most efficient compression possible, while keeping wide hardware compatibility.
I use AVC when I need strict AVCHD or Blu-ray compatibility.

[GeoffreyA]

SVT-AV1-HDR is an option if compatibility allows.

[DJ Bobo]

@ GeoffreyA
Not compatible I'm afraid; besides, it has even worse encoding times than x265 in 10-bit mode :-S

[Columbo]

Life is full of disappointments. :(

[Z2697]

Where did you get the 8 bit is worse and 10 bit is worthless?

[DJ Bobo]

@Z2697
8-bit has banding even if the bitrate is high enough (Q18 for example)
10-bit is a waste of time, because it is only needed to avoid banding, as x265 is not well tuned for 8-bit encoding (I thought this was common knowledge! :O )
So basically I need an HEVC encoder that does not tend to banding in 8-bit mode.
The AMD hardware encoder does not have banding in 8-bit mode, but then again it does not have 2-pass, and it does not support B-Frames on my GPU, so not very efficient at the end.

[Z2697]

Looks like AMF doesn't use angle mode in blocks larger than 8x8 :rolleyes: (tested on 780M iGP)
Well, by pure coincidence I do have a solution :)

[rwill]

What kind of banding are we talking about anyway? The worms like artifact due to intra prediction or the normal banding on large flat areas?

*edit*

To answer the original question, from what I know (which is not much), there are not many good HEVC encoders left. And they are all way above 100$.

[DJ Bobo]

@Z2697
You do? :-D

@rwill
Yeah, the "normal" banding on large areas, like shown in the left picture HERE (https://www.vsmimagescreation.fr/effet-de-bande-banding) (normal?? since when is this normal?? lol)

[rwill]

Was this with x265 version 4.2 or newer ? Because there is a fix that could lessen banding by Z2697 in 4.2...

[Z2697]

Was this with x265 version 4.2 or newer ? Because there is a fix that could lessen banding by Z2697 in 4.2...

Which one... the chroma qp offset?
But it's default to 0, and the before fix behavior of small offset doesn't diverge too much.

It can be hard to distinguish between the real banding and intra prediction worms, without tools that can view decoding steps.
Since OP says the bitrate is high and the AMF encoder is showing less problem (and my observation it's not using angles in large blocks), I suspet it's the latter.
(Is it CRF 18? Not really that high IMO...)

But then again if it's intra prediction worms I think 10 bit won't help much... but OP seems to imply it helps...

[Z2697]

@Z2697
You do? :-D

@rwill
Yeah, the "normal" banding on large areas, like shown in the left picture HERE (https://www.vsmimagescreation.fr/effet-de-bande-banding) (normal?? since when is this normal?? lol)

I mean if the remedy is limiting intra prediction, I do have written some experimental fork.

Banding (the real) is to be expected, dithering can help to distribute the error and reduce banding.
Dither pattern get destoried by compression, that's another way 8 bit encoding can create more banding, visually.
It's low variance high detail so AQ should help. But x265 default to a poorly-tweaked AQ2.
Use AQ1 instead.

[GeoffreyA]

@ GeoffreyA
Not compatible I'm afraid; besides, it has even worse encoding times than x265 in 10-bit mode :-S

Its preset 4 is about as fast as x264 veryslow; higher presets are extremely fast. But back to the question, have you tried AQ mode 3? Also, Z2697 has a version that fixes the intra banding that can look like, in many ways, the classic gradient/no-dithering banding.

[DJ Bobo]

Okay, so basically, what you are saying is, the upcoming v4.2 should fix the problems that are in the current stable version 4.1, even without changing the AQ mode?
Indeed I never changed the AQ mode.
So what is it now? Why someone tells me AQ1 and the other AQ3? :-D

[GeoffreyA]

So what is it now? Why someone tells me AQ1 and the other AQ3? :-D

If one is dealing with anime, AQ3 is advisable to defeat banding. If live action, AQ1 would be optimal for high-quality encoding. The default AQ2 has issues. There was useful discussion not too long ago:

https://forum.doom9.net/showthread.php?t=186641

[Z2697]

Okay, so basically, what you are saying is, the upcoming v4.2 should fix the problems that are in the current stable version 4.1, even without changing the AQ mode?
Indeed I never changed the AQ mode.
So what is it now? Why someone tells me AQ1 and the other AQ3? :-D

No, I don't see me sending limit intra prediction patch to mcw.
I don't like this approach.

If you want to test it, clone and build:
https://github.com/Mr-Z-2697/x265-experimental/tree/E-2026-05-25_limited-intra-angles
To mimic AMF behavior, use --limit-intra-angle=1

And 4.2 is released.

If you want to read some AQ stuff:
https://forum.doom9.org/showthread.php?t=186641
But in short x265 tweaked AQ2 constant and have some effect that some of us don't like. AQ3 is based on AQ2.

If one is dealing with anime, AQ3 is advisable to defeat banding. If live action, AQ1 would be optimal for high-quality encoding. The default AQ2 has issues. There was useful discussion not too long ago:

https://forum.doom9.net/showthread.php?t=186641

I'm more of a AQ1 for everything person :)

[Z2697]

https://github.com/ksvc/ks265codec
(no source)
:devil:

[microchip8]

Have you tried Kvazaar? https://github.com/ultravideo/kvazaar

More encoders: https://ultravideo.fi/#encoder

[GeoffreyA]

No, I don't see me sending limit intra prediction patch to mcw.
I don't like this approach.


Why, if I may ask? It's useful having this setting.

[Z2697]

Why, if I may ask? It's useful having this setting.

I have a feeling that the added if-else branches will slow down the encoding slightly, in normal setting.

I did some test and... got no conclusive result. So I guess the difference is below measurable?

[Balling]

x265 has a bug that causes both Intel Arrow Lake and Nvidia HW decoder to break on it. Crazy stuff guys. Patch is there, https://github.com/Multicorewareinc/x265/issues/758

[rwill]

Yes, crazy stuff indeed.

https://forum.doom9.org/showthread.php?t=186393

[Z2697]

Not sure if they don't accept that patch or just missed...
Maybe I should open a PR on github?

[Columbo]

FYI there are 28000+ patches in state New. You have to know one of the elites to get any attention for a patch.

[GeoffreyA]

Not sure if they don't accept that patch or just missed...
Maybe I should open a PR on github?

Go for it.

[jpsdr]

Yes, the patch is not on the standard repo. I have it on my branch, like the other critical patch witch fixes the broken multi-pass, still not on standard repo !!

[Z2697]

Yes, the patch is not on the standard repo. I have it on my branch, like the other critical patch witch fixes the broken multi-pass, still not on standard repo !!

multipass fix was merged as https://github.com/Multicorewareinc/x265/commit/cfee9638c82b655c5887cedbdf1aa856f81b906a

[jpsdr]

It's a fix for something, yes, but not the multi-pass fix.

Edit:
this one (https://github.com/jpsdr/x265/commit/d925c130c2ab5598a672b7b2c26f1ad4ff193289).

[Z2697]

Well, if we mean the same bug, it was caused by the improper fix for this aq-motion.
So a proper fix will not cause the bug, or in other words, will fix the bug.

[Z2697]

MVD PR merged https://github.com/Multicorewareinc/x265/commit/62728cd0c696655f34972b47cff1e1e77d84e426