Mapping of CRF between x265 and x264?

[benwaggoner]

I know that CRF values between x264 and x265 can't be expected to match, or even have a simple relationship.

That said, has anyone played around with comparisons enough to get a sense of CRF values in x264 and x265 that result in roughly equivalent quality for any particular scenarios?

[gonca]

Presently running an encode. If you want I can post screencaps and settings for comparison.

[gonca]

Comparison of x264 to x265 >>> preset, bitrate, fps. crf, ssim

[benwaggoner]

Relatedly, how about mapping of CRF between the 8-bit and 16-bit encoder modes? It seems (unsurprisingly) that the same CRF value yield a lot higher bitrates when encoding to Main 10 than to Main when using otherwise identical source and settings. Probably something to do with how CRF is an offset to QP, and actual quantization values will be relatively larger when you're starting with a lot more frequency data.

My "the kids are running around crazy and I can't think straight" guesstimate is that the difference should be 12 CRF, since each 6 QP represents a 2x change in the actual quantizer (at least in H.264...), and since 10-bit is 4x 8-bit, that'd be 12 QP steps to get a 4x quantizer difference. But I'm equally sure it's nowhere near that simple in practice...

[gonca]

Working on a different comparison chart with various xx-bit encoders. Do you prefer the chart in pdf or excel format?

[gonca]

Comparison part 2

[benwaggoner]

Comparison part 2
Interesting data. Looks like 8-bit and 10-bit x264 are quite comparable. But going from 8-bit to 16-bit builds of x265 have a huge delta. Rough ballpark around 6 steps? But once you're using the 16-bit build, there's not much difference between 10-bit and 8-bit output.

Good stuff!

[gonca]

Remember, this is one sample from one movie. More samples from different movies averaged out would produce more valid results. I would say that a set of predetermined set of movies, in their entirety, covering a variety of encoding situations would be ideal. However, if you look at the speeds you quickly realize that this is a time consuming effort. For the record I am using an i7 3930K @4.4Ghz.

[benwaggoner]

Remember, this is one sample from one movie. More samples from different movies averaged out would produce more valid results. I would say that a set of predetermined set of movies, in their entirety, covering a variety of encoding situations would be ideal. However, if you look at the speeds you quickly realize that this is a time consuming effort. For the record I am using an i7 3930K @4.4Ghz.
No doubt! The actual differential, if it is even consistent isn't known. But we can see from this data that there definitely is a big one with x265 10-bit versus x265 8-bit and x264.

It would be great if x265 could get whatever correction factor got implemented for the same situation in x264.

[gonca]

If you wish and are willing to be a little patient I could try and repeat the comparison with a different sample from a different movie. In fact you might want to suggest some movies and time frame(s) for comparison. Please bear in mind the speed of x265 at slow preset, or slower, if you wish that tested, on the length of the samples.

[benwaggoner]

If you wish and are willing to be a little patient I could try and repeat the comparison with a different sample from a different movie. In fact you might want to suggest some movies and time frame(s) for comparison. Please bear in mind the speed of x265 at slow preset, or slower, if you wish that tested, on the length of the samples.
I am certainly curious, and I can be patient for favors :).

Tears of Steel is a pretty good source clip for stuff that is readily available. The earlier Elephants Dream is also interesting and has some really challenging bits for CGI.

I don't know that preset should make that much of a difference as long as you're not setting any bitrate. I'd say initial tests using the default preset for both x264 and x265 would be fine. Or perhaps default x265 and a x264 preset that results in similar encoding time.

Generally different speed presets with CRF just result in slightly different file sizes, but similar quality.

However, my supposition is that CRF is ballpark similar between x264 and x265 8-bit, and it would require some serious double-blind visual comparison to determine a real delta. So the real question is the differential between x265 8-bit and x265 16-bit. Comparing 8-bit to 8-bit and 16-bit to 8-bit at the same preset is probably about as apples-to-apples as we'll get, and relatively straightforward. We may find a relatively predictable CRF ratio that gives the same file sizes.

[gonca]

I am presently downloading some 1080p and 4k clips, all legal before anyone asks, so we can get some more comparisons.

[gonca]

Comparison on crowd run 1080p50. Original has bitrate over 1Gbps

[gonca]

Ultrafast preset, same file size, crf comparison

[gonca]

low bitrate ABR comparison

[gonca]

high bitrate ABR comparison

[gonca]

For the double blind test, we can work on a semi official not technically correct test. I can encode the test clips, and if the members of the forum co-operate, not use mediainfo right off the bat, we could run an impromptu semi half double bind test

[easyfab]

or convert the files to x264 lossless ?

[benwaggoner]

It's interesting to note that bitrate goes UP significantly as the preset gets slower at the same CRF with x265. This is the opposite to the behavior that we'd expect, and that we see in x264. At a constant visual quality, more expensive encoding techniques should result in a smaller file size. Clearly CRF is acting quite weirdly in x265 overall, not just in the 16-bit v. 8-bit pipelines. PSNR and CRF are also less correlated with x265. Since x264 defaults to tuning for SSIM, perhaps we should compare both x264 and x265 using SSIM. And it's a better subjectively correlated metric anyway.

Comparing x265 to x265, encoding 16-bit to 10-bit versus 8-bit makes very little difference. In fact, the values are identical, which makes me wonder if a true 10-bit source wasn't used or something.

We continue to see a huge differential between 8-bit to 8-bit versus 16-bit to 8-bit, but it's not predictable by any CRF offset or anything. This may be due to CRF just not being fully tuned for 16-bit processing or something.

[gonca]

I have been using SSIM. The rate control in x265 is still a work in progress, which means that the 16 bit version might have a few issues, since it is a newer development. If you look at the low bitrate comparison x265 8 bit has the better SSIM value most of the time, and the other flavors of x265 have the lowest SSIM.

What do you think of my suggestion for the double blind tests?

[benwaggoner]

I have been using SSIM. The rate control in x265 is still a work in progress, which means that the 16 bit version might have a few issues, since it is a newer development. If you look at the low bitrate comparison x265 8 bit has the better SSIM value most of the time, and the other flavors of x265 have the lowest SSIM.
Yeah, that's probably the best we can do for easily calculated objective metrics right now. But I anticipate that SSIM will tend to relatively underrate x265 versus x264, as x265 (and probably HEVC in general) does a much better job avoiding temporal artifacts, which a single-frame metric like SSIM won't pick up on.

What do you think of my suggestion for the double blind tests?
I love double blind tests! They're the closest thing to something I actually trust :).

If you're worried about MediaInfo being used, you could also convert the clips back to lossless H.264 in 8-bit or 10-bit before distribution. That's how the Hydrogen Audio double-blind tests have worked, by converting back to FLAC IIRC.

[gonca]

If you can recommend a decent and free file sharing site, since Doom9 doesn't allow uploading of video files, I can work on getting the encodes done.
video > x264 > x264 loss less
x265

[gonca]

Let the double blind test begin

crowd_run_1080p50 ABR 1500 Kbps

http://www.mediafire.com/download/pfcyj2p5718du8y/crowd_run_1080p50_1_1.mkv

http://www.mediafire.com/download/f4sy78q8z8ubrkp/crowd_run_1080p50_1_2.mkv

http://www.mediafire.com/download/7t0jqjefn18f4n5/crowd_run_1080p50_1_3.mkv

[gonca]

crowd_run_1080p50 ABR 10000Kbps

http://www.mediafire.com/download/4lpccd3ye55dl5d/crowd_run_1080p50_1_4.mkv

http://www.mediafire.com/download/6gogjbblm9hzp4s/crowd_run_1080p50_1_5.mkv

http://www.mediafire.com/download/sfcb4bx98cf2a74/crowd_run_1080p50_1_6.mkv

[gonca]

crowd_run_1080p50 CRF 18

http://www.mediafire.com/download/94cvl59dmdfcyy9/crowd_run_1080p50_1_7.mkv

http://www.mediafire.com/download/gou6evkgs4zma37/crowd_run_1080p50_1_8.mkv

More samples to come

[gonca]

Second batch

crowd_run_2160p50 ABR 4000Kbps

http://www.mediafire.com/download/updfa7dukqbbu60/crowd_run_2160p50_2_1.mkv

http://www.mediafire.com/download/a0o3l4l0qr4kmew/crowd_run_2160p50_2_3.mkv

[gonca]

crowd_run_2160p50 ABR 20000Kbps

http://www.mediafire.com/download/eljl5vquygsnyb8/crowd_run_2160p50_2_5.mkv

http://www.mediafire.com/download/sm694akzabibpic/crowd_run_2160p50_2_6.mkv

[gonca]

crowd_run_2160p50 CRF 18

http://www.mediafire.com/download/a6sr72qc53cz7ns/crowd_run_2160p50_2_7.mkv

http://www.mediafire.com/download/64i02q8sftqs04a/crowd_run_2160p50_2_8.mkv

Feedback appreciated

[edison]

It look like the quality of x265 lost in the case that in same bitrate compare.

[gonca]

Updated the version of x265 for the double-blind samples. Try those.

[edison]

crowd_run_2160p50 CRF 18

http://www.mediafire.com/download/a6sr72qc53cz7ns/crowd_run_2160p50_2_7.mkv

http://www.mediafire.com/download/64i02q8sftqs04a/crowd_run_2160p50_2_8.mkv

Feedback appreciated

wow, 2800Mbps !

[vivan]

because lossless

[macromizer]

Updated the version of x265 for the double-blind samples. Try those.

Not to nitpick too much, but you do realize that a double-blind test means that the test giver doesn't know which sample is which, right? If you're doing all the encoding yourself than you know which sample is which and that means you're simply doing a single-blind test.

[gonca]

Not to nitpick too much, but you do realize that a double-blind test means that the test giver doesn't know which sample is which, right? If you're doing all the encoding yourself than you know which sample is which and that means you're simply doing a single-blind test.

That's why I previously referred to it as a semi double blind test.
I will not participate in the test, since I know which is which, and I have re-encoded the samples to x264 lossless so the willing participants don't know which is which. Lets not get too hung up on some issues yet since x265 seems to be changing quickly.

My function in these tests is to encode and upload and let other members offer their opinions on the samples.

[Blue_MiSfit]

My poor Q6600 CPU at work is belching smoke and flames trying to play these clips :)

I'll have to try my i7 at home!

[gonca]

I did notice that playback is just a little cpu intensive. I wonder how an i5 would handle it?

[benwaggoner]

I did notice that playback is just a little cpu intensive. I wonder how an i5 would handle it?
With WPP, HEVC decoding should scale to as many threads as the Mod64 height of the frame. And more recent processors certainly will get more pixels-per-clock than older ones.

[gonca]

With WPP, HEVC decoding should scale to as many threads as the Mod64 height of the frame. And more recent processors certainly will get more pixels-per-clock than older ones.

Pardon me for simplifying your statement, but HEVC encoding / decoding isn't meant for legacy hardware.

I'll try to encode some more samples this weekend for comparison, but uploading x264 lossless takes a few minutes.
If anyone needs to know which sample is which, let me know and I will PM you the info.

[gonca]

ducks_take_off_1080p50 ABR 1500 Kbps

http://www.mediafire.com/download/p4t9inwgc49kr24/ducks_take_off_1080p50_3_1.mkv
http://www.mediafire.com/download/xbmf02ebnjc7ddz/ducks_take_off_1080p50_3_2.mkv
http://www.mediafire.com/download/yy3kiq6ezu88ti5/ducks_take_off_1080p50_3_3.mkv

[gonca]

ducks_take_off_1080p50 ABR 10000 Kbps

http://www.mediafire.com/download/lbwfqqmm2i4atvv/ducks_take_off_1080p50_3_4.mkv
http://www.mediafire.com/download/tslgppgpzxzdz6p/ducks_take_off_1080p50_3_5.mkv
http://www.mediafire.com/download/4270sub4bxyht3b/ducks_take_off_1080p50_3_6.mkv

[gonca]

ducks_take_off_1080p50 CRF 18

http://www.mediafire.com/download/383438n0vmiv1ka/ducks_take_off_1080p50_3_7.mkv
http://www.mediafire.com/download/ov1djku9aug9thm/ducks_take_off_1080p50_3_8.mkv

Feedback, and opinions welcome.

[gonca]

ducks_take_off_2160p50 ABR 4000 Kbps

http://www.mediafire.com/download/l1buat5s3rxev12/ducks_take_off_2160p50_4_1.mkv
http://www.mediafire.com/download/24393nrr94s0kp1/ducks_take_off_2160p50_4_2.mkv

[gonca]

ducks_take_off_2160p50 ABR 20000Kbps

http://www.mediafire.com/download/3ja4z7p13icbw9o/ducks_take_off_2160p50_4_3.mkv
http://www.mediafire.com/download/kmbspb6ssamiee5/ducks_take_off_2160p50_4_4.mkv

[gonca]

The CFR 18 files encoded to x264 lossless are huge (4GB). If someone is truly interested in comparing them I will upload them.

[gonca]

Here is the pdf showing which file was encoded with each encoder

[benwaggoner]

This new commit suggests that the 10-bit QP mapping has changed or is changing:

https://bitbucket.org/multicoreware/x265/commits/3a19a9fdb103979e65a9daf15c46c0735e8d743e

The what and why are as mysterious as always though. What I wouldn't give for some more detailed comments along with these commits :)! Probably only one out of twenty gives much of a hint as to the potential real-world impact of the change.

[gonca]

After a while I can always repeat the process. Just figure I'll let things settle in a bit first, allow for some of these changes to be stabilized somewhat, like psy-rd. One thing I've read is that setting CTU=16 for SD and CTU=32 for HD might help with detail retention. Might try that soon.

[benwaggoner]

After a while I can always repeat the process. Just figure I'll let things settle in a bit first, allow for some of these changes to be stabilized somewhat, like psy-rd. One thing I've read is that setting CTU=16 for SD and CTU=32 for HD might help with detail retention. Might try that soon.
I think we can exclude psy-rd for the moment. But yes, a lot of stuff does seem in flight right now.

I think comparing just --preset medium with stock settings using the same 8-bit source and comparing Main and Main 10 in 8-bit is a reasonable test. But if they're already trying to fix this, no need to sweat it until they say it's done.

[gonca]

A quick test comparing CTU=32 and CTU=64 on a high def source might be interesting just to compare the effects on detail retention. The problem right now is that we are thinking of the x265 options in terms of x264, and realistically, they probably behave differently.

[benwaggoner]

A quick test comparing CTU=32 and CTU=64 on a high def source might be interesting just to compare the effects on detail retention. The problem right now is that we are thinking of the x265 options in terms of x264, and realistically, they probably behave differently.

--rdpenalty might also be a relevant parameter to impact this behavior. --rdpenalty 1 uses rate distortion in choosing 32x32 TU.

Docs:
--rdpenalty <0..2>
Penalty for 32x32 intra TU in non-I slices. Default 0

Values: 0:disabled 1:RD-penalty 2:maximum
I've wondered why the non-RD mode is the default. Almost by definition, RD is going to be better than non-RD.