How to compare video encoders

[x265_Project]

I thought this might be useful, so I wrote up a blog post about it. Constructive feedback is welcomed.
http://x265.org/compare-video-encoders/

[littlepox]

I thought this might be useful, so I wrote up a blog post about it. Constructive feedback is welcomed.
http://x265.org/compare-video-encoders/

A few points I would like to address:

1. If possible don't use bit-rates too high or too low. Otherwise you may end up getting to the conclusion that two encoders are equally perfect or bad.

2. Use proper video player or editor and avoid pre-processing/post-processing, INCLUDING resizing. I've seen people use lousy players with the player itself tries to "optimize" the video, and the player use poor precision so that a 10-bit well encoded video is full of banding. madVR with post-processing/subjective adjusting disabled(like default settings) is ideal.

3. If you make screenshots do NOT save them in lossy format, like jpeg. PNGs and BMPs are fine.

4. Comparing screenshots is not a idea too bad coz it allows you to compare details carefully. But you need to prepare quite a lot so that they cover different scenes, including high-motion scenes, dark scenes, etc. If you want to see it in larger size please use nearest-point up-size with integer scaling factor. Any other ways to upscale effectively becomes a post-processing.

5. comparing crf values and conclude the smaller one has the better quality is a stupid idea. A more stupid one would be comparing crf values across x264/x265 encodes.

[Asmodian]

4. Comparing screenshots is not a idea too bad coz it allows you to compare details carefully. But you need to prepare quite a lot so that they cover different scenes, including high-motion scenes, dark scenes, etc. If you want to see it in larger size please use nearest-point up-size with integer scaling factor. Any other ways to upscale effectively becomes a post-processing.

Comparing screen shots is a bad idea because different encodes and codecs make different frame type decisions, comparing one encode's P frame to another's I frame or B frame is counter productive. Even comparing frames of the same type can be unhelpful due to different rate control decisions.

[x265_Project]

Great suggestions... except that I can't agree with #4 above.

While it's extremely tempting to compare still frames, other than using a still frame to point out an artifact that is clearly visible when you play the video at full frame rate, we need to avoid this temptation (at least when it comes to comparing quality). As Asmodian points out, with 2 different encoders you may end up comparing an I frame to a B frame that is 1/10 the bit rate (not a fair comparison at all).

But again... things that are visible in still frames aren't alway visible at full speed. Video encoders are optimized to make video look great, and so artifacts that aren't visible at full playback frame rate are not so important. Still frames will never show you motion inaccuracy that might be obvious at full speed. In other words, everything may look great, but some objects may actually be out of position.

[littlepox]

Great suggestions... except that I can't agree with #4 above.

While it's extremely tempting to compare still frames, other than using a still frame to point out an artifact that is clearly visible when you play the video at full frame rate, we need to avoid this temptation (at least when it comes to comparing quality). As Asmodian points out, with 2 different encoders you may end up comparing an I frame to a B frame that is 1/10 the bit rate (not a fair comparison at all).

But again... things that are visible in still frames aren't alway visible at full speed. Video encoders are optimized to make video look great, and so artifacts that aren't visible at full playback frame rate are not so important. Still frames will never show you motion inaccuracy that might be obvious at full speed. In other words, everything may look great, but some objects may actually be out of position.

That's why I emphasize the number and coverage. If you make dozens of screen-shots that cover nearly all scenes, and those from sample A CONSISTENTLY outperform, you know you have something creditable to say.

When you are forced to compare videos at high bit-rates, or the two encoders performs very close (like comparing slightly different parameters/versions of one encoder), this is almost the only way to go unless you trust the benchmark values.

[birdie]

Video must be evaluated as video, not still frames. It’s relatively easy to compare the visual quality of two decoded frames, but that’s not a valid comparison. Video encoders are designed to encode moving images. Things that are obvious when you are examining a single frame may be completely invisible to any observer when viewed as part of a sequence of images at 24 frames per second or faster. Similarly, you’ll never spot motion inaccuracy or other temporal issues such as pulsing grain or textures if you’re only comparing still frames.

This one I will totally disagree with. There is a dozen of reasons why this item is completely wrong but I'll just give one of them: archival quality does matter unless you encode for streaming.

Also, you know, many people love to pause video and scrutinize all the little details. They don't care what type of frame they're looking at, be it I or P or B - a good codec must encode such a way there is zero difference in quality between different frame types. Otherwise I can create a codec which will beat any existing codec by properly encoding only I frames (e.g. using JPEG2000) and outputting actual garbage for the rest of the frames.

[Asmodian]

If it looked better in motion your theoretical codec encoding only I frames and outputting actual garbage for the rest of the frames would be better, wouldn't it? Of course it wouldn't look better watched at normal speed, but if it did it would actually be a better lossy video codec.

If you compare still frames, even ignoring the frame type issue for now, you can think a codec with in-motion artifacts is perfect while a codec with absolutely no motion artifacts but a little less detail or a few more artifacts in a still frame looks much worse. Humans are very good at seeing motion artifacts so in motion the first codec might be the one that looks worse.

None of our modern lossy codecs encode video such that there is zero difference in quality between frame types because it has been obvious since MPEG2 that this is not optimal. Using a lower quality for B frames has improved total quality per bit a lot.

The point is that a codec optimized for those people who pause the video to inspect individual frames is not as good for those people who are simply watching the video. A lot of people do inspect frames, I admit I still do it too when picking encode settings, but it isn't actually a good way to judge video codecs. The problem is that when using transparent bitrates it is so hard to feel confident about judging video quality and images are so much easier. Sadly easier does not necessarily mean better. :(

[littlepox]

you can think a codec with in-motion artifacts is perfect while a codec with absolutely no motion artifacts but a little less detail or a few more artifacts in a still frame looks much worse. :(

This can be easily avoided if you have the source as the reference, and in most cases you do.

[nevcairiel]

The key to take away from that is you need to judge both, but most importantly not judge still frames alone as the primary metric, like so many people like to do.
Motion is extremely important of a factor, and that is something that is entirely lost on your still frames. Even if you would judge every single frame individually, you're still not judging motion, because that requires watching scenes as a whole, not any amount of individual frames.

As Asmodian already points out, while in a perfect world all frames and frame types would be equal, in reality at the same bitrate, thats not beneficial for the overall quality. Obviously you don't want to go down to a level where there is obvious "I-Frame pumping", however once you go down to full pixel-by-pixel comparisons, there are differences to be found.
If this is not acceptable, you can typically override the qp offsets for i/p/b, and/or generally increase the bitrate.

If you require perfect constant quality for archival purposes, you are probably better served with codecs designed for video archival. Its always a good idea to pick the best tool for a specific job. :)

[birdie]

I can choose x265 codec parameters to reach archival quality with a required trade-off between the size and the quality of the result. That doesn't make this codec any less preferable than special codecs meant for archival and nothing else. Also, there's CRF=0 which enables lossless compression and I doubt those specialized codecs produce much smaller files, however I can admit they might be quite faster.

Also, you, guys, totally forget that one can choose to insert I frames at a different pace than what the standard dictates (every second), which makes the point of comparing or not comparing frames based on their types extremely moot. A random chosen frame should contain enough details to match your liking otherwise it's a cheating codec/a codec which is not thoroughly optimized for video.

So, nothing will dissuade me from comparing random frames produced by different codecs.

Also, there's a huge problem with human's perception. For instance it's impossible to see all the details in the entire frame - you usually point your vision at a specific part of the scene - what are the chances that your future audience, including your future self, will be looking at the same parts of the screen and pay close attention to them?

Ten years ago I could tolerate various artifacts introduced by video compression, nowadays I'm cringing looking at my past encodes.

So, it's not about achieving archival quality compression level. It's primary about your audience and being future-proof.

[Asmodian]

So basically you are going with a less-optimal comparison method because you do not feel like you can accurately judge quality when watching full speed video. Welcome to the club, however please understand that this is not truly optimal.

I don't understand this belief that frame types don't (or shouldn't?) matter. Adding extra I frames changes nothing in this regard, why would it?

Yes, it is very hard to judge quality differences between high quality encodes when watching the video once at full speed, but this does not mean that you can accurately judge quality simply by comparing random frames either. Always make sure to watch each encode at full speed and weigh any quality differences more than differences noted when comparing still frames.

I am not saying never look at still frames, especially for high quality encodes, simply that it should never be the most important, or even worse the only, way encodes are compared. The argument about being future-proof is exactly the point! If you simply use still frames to do your comparison you might be saving the lower quality video because you missed some artifact that was only visually apparent during motion because you were too focused on eliminating artifacts that are only visually apparent in still frames. Maybe you miss it now, given that you cannot notice a difference between the two during normal playback, but in the future it might be obvious and you picked the wrong one.

This is true at every quality level, the intensity of the artifacts goes down as quality goes up so it is harder and harder to notice any differences but optimal for still images and optimal for video are still not necessarily the same. If you cannot see a difference between the two in motion then, sure, start comparing frames as the only way to judge any potential differences. I also agree that these differences might become more obvious when watching the video on another display, another time, or under different conditions; there is a reason to target quality improvements you cannot notice during a few play throughs. Personally, I always target a CRF value below that necessary for "transparent" quality. However, this does not mean that judging still frames is a good way to judge differences, only that it is the only way we have in some cases. :p

[hello_hello]

I'm one for comparing single frames, but I learned a long time ago it's almost pointless comparing frames from different encodes/encoders, at least at sensible bitrates. Often I've preferred one encode, and then a few frames later I've preferred the other, but that generally translates to encoding differences rather than quality differences as such, so I always add the AVS output to the comparison. Then you can at least nitpick over how accurately frames are encoded, relative to the source.

Speaking of comparing still frames and nitpicking, the last time I did just that to compare x264 (tune film) and x265 (defaults) at standard definition, x265 didn't really thrill me. It was a while ago, but should x265 be reserved for high resolutions, leaving the lower resolution stuff for x264?

These screenshots were taken Nov 2016, so if I'm doing x265 a disservice, let me know and I'll delete the link. They were typical for the encode. I didn't cherry pick a well encoded frame to make x264 look good, or to make x265 look bad.
https://forum.videohelp.com/threads/381376-Where-is-the-new-super-codec-%28the-Alliance-for-Open-Media%29#post2467576

[burfadel]

x265 has improved. In your comparison I'm presuming you used the default settings? There are a lot of tweak settings you can do which can make a huge difference. That aside, you are comparing a (rounded) 3.5 MBPS picture with a 5.0 MBPS picture. If your x265 was at 5.0 MBPS it would more than make up for the quality difference. Actually I would think with tweaked settings the 3.5 MBPS x265 encode would probably look better than the 5.0 MBPS x264 encode.

[hello_hello]

The bit rates I mentioned initially were the result of using CRF18 for each encoder. I ran the x265 encode again using two passes so the bitrate would be the same as the x264 CRF18 encode (no tuning). 4960kbps. That's the encode I used for the screenshot. The x264 screenshot was taken from the encode where I'd set Tune Film, but it didn't increase the bitrate much and looked pretty much the same as without Tune Film, so when I realised I used the tune film screenshot for x264 I didn't bother changing it.

That's the reason it was disappointing the x264 encode looked a fair bit better, because the bitrates were the same.

Maybe I'm wrong, but I've found when playing around with x264's settings over the years there's no free lunch. If you adjust a setting that improves the encoding quality in some way, there's a corresponding bit rate increase (for CRF encoding) and for 2 pass encoding it might distribute the bits differently, but if the over-all bitrate doesn't change you're probably robbing Peter to pay Paul to a certain extent.