x265 - awful quality in dark scenes

[THU22]

Recently I started experimenting with x265.

After closer inspection I noticed that it handles dark scenes very poorly, especially when the source is not perfect.
I took some screenshots and included some video samples for comparison. You should look at it in a dark environment with optimal display brightness, so you do not get any black crush. You should also disable any image processing your TV or player might have, I did not notice that stuff at first, as my TV has a very nice "smooth gradation" feature which helps with lower quality video.

https://mega.nz/#!PQpCXIzY!NdyzOs1Zq_19TvGkWiiImSD2H7H68ICH3hSL739iGCs (password: doom9)

The left side of the picture is the most important here - the bottom left corner in the first shot, and the entire left side of the window in the second shot.

As you can see, x265 looks bad even at 10 Mbps, and just absolutely awful at 5 Mbps and CRF 18.

15 Mbps x264 looks very close to the original, almost indistinguishable.
10 Mbps x264 looks worse, but just slightly better than x265 in my opinion, which is crazy.

This is also the reason why I prefer 2-pass encoding over CRF, because CRF always gives lower bitrate to dark scenes.


Anyway, is this how x265 is right now? Or am I doing something wrong?
I use MeGUI for encoding, I used both the bundled 1.9 version of x265 and several of the latest 2.0 builds. It also makes no difference which preset I use.

After analyzing x265 encodes from other people, I noticed the same thing in dark scenes.
At the moment it seems x265 is not worth using over x264, at least for anything other than 4K.

[Leo 69]

Here's my CRF18 encode of your source, which resulted in a ~5 mbps stream.
http://www.mediafire.com/download/ul5v32pa7q9i9bf/1_-_source_new.mkv

The result is certainly not transparent and colors are just slightly "off", but it's far better than what you offered.
You must keep in mind, that the codec is relatively young, is being actively developed and will not reach its full potential soon. Learn its settings and read the forums to squeeze the most out of it.

[benwaggoner]

--aq-mode 3 is exactly for this. It reduces QPs in dark areas, which improves quality on LCD displays which have elevated low luma compared to the classic cathode ray tube which all our basic Rec. 601/709 video math remains based on. Note that --aq-mode 3 with CRF will increase bitrate proportionally to how much complexity you have in dark regions, and also with aq-strength..

[sneaker_ger]

10 bit encoding also helps. We had a lot of these complains on this forum about x264 as well - that's why --aq-mode 3 was created and later also implemented in x265. You might find more tips with a forum search.

[THU22]

Here's my CRF18 encode of your source, which resulted in a ~5 mbps stream.
http://www.mediafire.com/download/ul5v32pa7q9i9bf/1_-_source_new.mkv

The result is certainly not transparent and colors are just slightly "off", but it's far better than what you offered.
You must keep in mind, that the codec is relatively young, is being actively developed and will not reach its full potential soon. Learn its settings and read the forums to squeeze the most out of it.

Thanks for uploading your encode.

It definitely looks better than mine, but still not as good as 15 Mbps x264 in my opinion.
Do you think your encode at 10 Mbps would look comparable to that, or even better?

I tried using the 10-bit encoder, as well as aq-mode 3, but I am not happy with the results, even at 10 Mbps.
MeGUI does not offer any advanced options for x265, only the preset and bitrate, you have to use the command line to set whatever you want, which is not ideal.
What software are you using with x265, by the way?
Also, is there any downside in using 10-bit or 12-bit encoding?

If I could achieve the 15 Mbps x264 result with 10 Mbps x265, I would probably switch to HEVC, but at the moment I do not see that happening.

[trip_let]

Well, x265 is generally more efficient than x264 but not to the point where x265 with 5 Mbps is going to beat x264 at 15 Mbps. At 10 Mbps it should be relatively close or maybe in x265's favor. Depends on the material and what you're looking for.

You could also try processing the input with some filters (think Avisynth, Vapoursynth, or maybe even whatever is in those GUIs). That in addition to --aq-mode 3, maybe increasing --aq-strength, >8 bit, etc. Possibly even bumping up the psy stuff like --psy-rd, psy-rdoq. Some of this could be counterproductive, adding more bits where it doesn't help, reducing bits where it matters if you're doing 2 pass.

Higher bit depth runs slower but usually shouldn't have worse quality. Maybe in some edge cases it causes bits to be wasted in places, lowering perceived quality elsewhere? Practically it's going to be a quality improvement, generally even more so if you have >8 bit input. And even if your actual input is 8 bit, with Avisynth/Vapoursynth you could do 16 bit processing including potentially grain/deband or something along those lines and pass out >8 bit to the encoder. Higher bit depth is just going to be slower on encode and decode and potentially not supported by hardware decode. Though some stuff has 10-bit hardware HEVC decode.

Personally I just use command line x264 and x265. Don't know about others.

[turab]

This is also the reason why I prefer 2-pass encoding over CRF, because CRF always gives lower bitrate to dark scenes.

Really? I was always told 2-pass and crf use the same rate control algorithm. Settings like --aq-mode are independent of the two.

[THU22]

I did another test, this time with 12-bit encoding. And I have to say I am astonished by the results.

Not only is the bitrate lower than for 10- and 8-bit encodes (CRF 20), but the quality is incredible, almost transparent.
In case of that screenshot comparison, the 12-bit encode seems to look even better than the source.

I also included another clip this time. Focus on the second transition, after the Pixar logo. Going frame by frame is recommended. Mind-boggling dfiference.

https://mega.nz/#!bJ4xQbYA!2wrj_suINfunol6U5xORDPOjY7935NN1Sen4jax-BZc (password: doom9)

Default medium preset, only --aq-mode 3 set in the command line for all encodes.
Encoding rate is not that much slower, and software playback is perfectly fine.

I do not care about hardware compatibility, I just want the best quality at a low file size. Does this mean 12-bit is the way to go?

[Leo 69]

I also noticed the bitrate decrease in a 12-bit mode, asked in the main thread about this phenomenon, but never got an answer.

[nakTT]

I also noticed the bitrate decrease in a 12-bit mode, asked in the main thread about this phenomenon, but never got an answer.
Yeah, decrease slightly at the same preset. Wonder why too.

[benwaggoner]

Yeah, decrease slightly at the same preset. Wonder why too.
12-bit encoding has some fundamental differences; it requires a different binary!

I wouldn't expect to get identical bitrates at the same CRF across 8/10/12, particularly with high-depth input. The overhead of dithering when doing 8-bit encode from 10/12 bit source can be significant.

[cojj]

I also noticed the bitrate decrease in a 12-bit mode, asked in the main thread about this phenomenon, but never got an answer.


According to https://en.wikipedia.org/wiki/High_Efficiency_Video_Coding:

Additionally, in the Main 10 profile 8-bit video can be coded with a higher bit depth of 10-bits, which allows improved coding efficiency compared to the Main profile.


If same logic applies, 12bit should have better coding efficiency => better compression?

[LoRd_MuldeR]

According to https://en.wikipedia.org/wiki/High_Efficiency_Video_Coding:

Additionally, in the Main 10 profile 8-bit video can be coded with a higher bit depth of 10-bits, which allows improved coding efficiency compared to the Main profile.


If same logic applies, 12bit should have better coding efficiency => better compression?

"better coding efficiency" is synonymous to "better compression". And it means that you can get either: Better quality at the same average bitrate, or
Same quality at a lower average bitrate

And yes, 10-Bit encoding is supposed to give better compression than 8-Bit. And 12-Bit encoding is supposed to give better compression than 10-Bit.

But: Be aware that CRF mode is not a "constant quality" mode. Yes, the same CRF value roughly results in the same quality across different sources, as long as you don't change any other encoder settings. However, as soon as you change other encoder options (e.g. you use another Preset), you can not expect that a certain CRF x still results in the exactly same quality as it did before. And even more so when switching from 8-Bit to 10-Bit (or form 10-Bit to 12-Bit) encoding!

So, encoding the same source with the same CRF value twice, once with 8-Bit and once with 10-Bit (or with 10-Bit and 12-Bit), and then comparing the bitrates of the resulting files will tell you nothing. That's because you almost certainly would be comparing files of different quality! Certainly, the quality of those encodes is not guaranteed to match as closely as it would need to match in order to make a bitrate comparison meaningful. You would be comparing apples and oranges.

Instead, if you want to do a proper fair/unbiased comparison, both resulting files (e.g. 8-Bit and 10-Bit) need to have the same average bitarte - which is most easily achieved in 2-Pass mode. Then the quality of those files can be compared.

[THU22]

I have done a few 12-bit encodes now and so far I am very happy with the results at CRF 18 and Medium preset (aq mode 1, as 3 is pointless with this bit depth).

Almost transparent quality, averaging 5-7 Mbps at full 1080p, depending on the type of material. CRF 16 does not really improve the image any further.

One low-motion animation encode got 2.5 Mbps and dark scenes look perfect, while 2-pass 8-bit 10 Mbps looks awful.

Thanks for all your help. Seems like I will be sticking with this. :)

[benwaggoner]

Did anyone else try using --aq-mode 3 while sticking with 8-bit? 10 and 12 bit can do great things, but there are lots of 8-bit only HEVC decoders out there...

[Forteen88]

Excuse me for opening an old thread, but aren't --zones a good way to handle dark scenes?
It's much more job though (finding the frames with dark scenes and lower the CRF-value at those frames), compared to just setting --aq-mode 3

EDIT: OK, thanks RainyDog, I thought so too (about internal' encoders). By the way, I remember that years ago, with x264, I used to use --zones to lower bitrate on end-credits of movies :)

[RainyDog]

Excuse me for opening an old thread, but aren't --zones a good way to handle dark scenes?
It's much more job though (finding the frames with dark scenes and lower the CRF-value at those frames), compared to just setting --aq-mode 3

Yes. That's what all the *cough* 'internal' encoders do these days.

AQ Mode 3 really doesn't make all that much difference.

[Blue_MiSfit]

I'm curious which 8 bit HEVC decoders you consider to be relevant, Ben?

[benwaggoner]

Yes. That's what all the *cough* 'internal' encoders do these days.

AQ Mode 3 really doesn't make all that much difference.
It can help significantly but sporadically. I find it makes the worst 1% look better. I wish there was a separate parameter to more finely adjust the luma QP ramp in darks. Sometimes an aq-mode 2.5 or 2.6 would be ideal :).

[benwaggoner]

I'm curious which 8 bit HEVC decoders you consider to be relevant, Ben?
They are all relevant to their given markets. I find what x265, Elemental, and Beamr/Vanguard doing personally very interesting. Elecard, Ateme, and Main Concept all get use, but I'm not personally familiar with their current state.

It's wonderful we have x265 as a very flexible, free, and open source encoder as a reference. It makes it a lot easier to talk about, but less discussion doesn't mean proprietary encoders can't be excellent.

It's great we have a very competitive HEVC encoder market, so they are all improving and can leapfrog each other even, and all can specialize for different verticals and scenarios.

My professional encoder needs are quite specific of course.