AV1, VP9, x265, x264 low bitrate comparison

[karasu]

I made some test with the new AV1 codec from aomedia and I made some captures for comparison :

https://vincenttschanz.github.io/1ko-videocodecs/codec-comparison/

The goal was to use a very low bitrate to see which encoder will give better result and how the picture will be degraded.

Don't hesitate to tell me if I made mistakes or if I can refine the comparison, thanks.

[sneaker_ger]

Thx.
VP9 > AV1 > x265 > x264

[ChaosKing]

Nice comparison!

I think VP9 wins.
And I'm a bit disappointed by x265. Worm like artefacts in frame 226 and look at the street lamp (bottom left) in frame 399

[Taurus]

Thanks!
VP9 > AV1 > x265 > x264
Really amazing VP9 @ this low bitrate.

[hajj_3]

not that helpful as people won't use bitrates this low with resolutions this high. A better test would be a low bitrate 840x480 or 640x360.

[bstrobl]

Most of the improvements in AV1 over VP9/HEVC are still kept as experiments that need to be enabled during compile time, might be interesting to try a couple of those out.

Still good to see VP9 beating out HEVC, would have thought the performance was slightly worse. Encoder improvements maybe?

[karasu]

Most of the improvements in AV1 over VP9/HEVC are still kept as experiments that need to be enabled during compile time, might be interesting to try a couple of those out.


I'd be glad to try, do you know where I can find informations about that?

[bstrobl]

I'd be glad to try, do you know where I can find informations about that?

https://www.youtube.com/watch?v=lzPaldsmJbk talks more about this (roughly 16 minutes in). A couple of experiments are listed at 36:17, but am not sure where to get a complete list. Maybe others here can pitch in for that.

Edit: the slide at 40:20 gives a list of enabled experiments to show the progress over the default build.

[easyfab]

here the complete list:

https://aomedia.googlesource.com/aom/+/master/configure#246

[easyfab]

and here the experiments that are adopted and now default:

https://aomedia.googlesource.com/aom/+/master/configure#471

[Atak_Snajpera]

To be honest all look terrible. Besides who on earth encodes blu-ray collection using such ridiculously low bitrate? Less than 600kbps for 1920x800? Audio E-AC3/DTS has bigger bitrate than this! Increase bitrate to some more sane levels (1.5Mbps-2Mbps) and you will see that competition is better in retention of fine details.

[Blue_MiSfit]

Why are the bitrate targets different here?

[karasu]

To be honest all look terrible. Besides who on earth encodes blu-ray collection using such ridiculously low bitrate?

They look terrible indeed, but who talked about ripping blu-ray? In this case I wanted to see the visual degradation. Beside, there are fare worse quality in production everywhere on the Web.

Also, current encoding speed is also a problem IMO.

[karasu]

Why are the bitrate targets different here?

My goal was to aim for the same resulting file size for fair comparison. AV1 gave me the first size and I had to adapt settings for the other encoders to match as closely as possible.

[ChaosKing]

If think it would be also nice to see all 4 encodes with a 1000kb/s and 1500kb/s bitrate. I wonder if vp9 will still be the winner...

[Tommy Carrot]

VP9 and AV1 use golden frames, which means every Nth frame has much higher quality than the rest of them. So looking at individual frames can be misleading, because if you're looking at golden frames, or a few frames after them, the quality is much higher than the average quality of the video. x264 and x265 doesn't cheat like that, not to mention golden frames introduce a rather annoying effect during normal playback, i wish codec developers would abandon this technique (but they won't, because it improves the metrics...).

[karasu]

VP9 and AV1 use golden frames

Isn't that the same as I-frames? Do you know a way to check the type of frame when decoding ?

[ChaosKing]

Try this:

http://stackoverflow.com/questions/37554724/get-frame-type-for-specific-frame-using-ffmpeg

[Quikee]

VP9 and AV1 use golden frames, which means every Nth frame has much higher quality than the rest of them. So looking at individual frames can be misleading, because if you're looking at golden frames, or a few frames after them, the quality is much higher than the average quality of the video. x264 and x265 doesn't cheat like that, not to mention golden frames introduce a rather annoying effect during normal playback, i wish codec developers would abandon this technique (but they won't, because it improves the metrics...).

AFAIK golden frame is just a VP9 terminology for a reference frame from the past (a P frame would have a golden frame) so from that VP9 and AV1 don't have anything special than other codecs have. Boosting the quality of a golden frame makes sense if it serves as a reference frame for more other frames and boosting its quality improves the prediction. I would be surprised that x264 and x265 doesn't do something similar. What you describe is when the frame quality is boosted too much and this is just a fail of the rate control. Many would agree that the rate control in libvpx (the reference VP9 implementation) is not good. In libaom there is a rather new experiment available - "xiphrc", which is the theora / daala rate control adapted for AV1 and it shows promising results already, but it needs a lot more work and tuning.

Generally, I don't think it makes sense to test AV1 without enabling many of the experiments (I expect most of them will be part of AV1). Which one to enable is also not known and won't until they'll evaluate them and either drop or accept them as default. So that would be at the time near bitstream freeze.

[Tommy Carrot]

Isn't that the same as I-frames? Do you know a way to check the type of frame when decoding ?
No, they are regular p-frames, just with boosted quality.
AFAIK golden frame is just a VP9 terminology for a reference frame from the past (a P frame would have a golden frame) so from that VP9 and AV1 don't have anything special than other codecs have. Boosting the quality of a golden frame makes sense if it serves as a reference frame for more other frames and boosting its quality improves the prediction. I would be surprised that x264 and x265 doesn't do something similar. What you describe is when the frame quality is boosted too much and this is just a fail of the rate control. Many would agree that the rate control in libvpx (the reference VP9 implementation) is not good.
You're probably right, it could be improved with some tuning, they are boosted way too much in vp9, and in AV1 last time i checked a few months ago. In some cases the periodic sharpening effect is very noticeable and annoying. X264 and x265 don't have anything like that, they have very consistent quality. You could get similar effect with them if you set I-frame distance to 1 second or so, but generally you don't do that.

[sneaker_ger]

x264 and x265 have --ipratio and --pbratio to set qp ratio between IP and PB. By default 1.4 and 1.3.

[Tommy Carrot]

x264 and x265 have --ipratio and --pbratio to set qp ratio between IP and PB. By default 1.4 and 1.3.

Yep, the problem with VP9 and probably with AV1 is that the golden frames are boosted way more than that. The quality fluctuation in x264 and x265 is not really noticeable, while i can't say that about vp9. Daala had the same problem in the later iterations, so using its rate control will not fix this issue.

[IgorC]

They look terrible indeed, but who talked about ripping blu-ray? In this case I wanted to see the visual degradation. Beside, there are fare worse quality in production everywhere on the Web.

Also, current encoding speed is also a problem IMO.

Interesting test. Wow, I didn't expect VP9 would that better than x265 at low bitrates.

It would be great to see how they perform at Youtube bitrate ~1.5-2 Mbps 1080p or Netflix's ~2-4 Mbps 1080p.

[karasu]

I updated the page with links to actual video files. AV1 is kinda useless as it can only be decoded by aomdec. Can someone tell whatI messed up by muxing the x264 flux to mp4 ? (ffmpeg -i x264.x264 -c:v copy -movflags +faststart x264.mp4) The video is stuttering in the browser (VLC playback is fine).

Also I added frame 260, where I believe x265 does a better job to keep details behind the smoke.

http://video.1ko.ch/codec-comparison/

[sneaker_ger]

ffmpeg is not writing the ctts atom for re-ordering. Use mp4box or l-smash muxer.

[karasu]

ffmpeg is not writing the ctts atom for re-ordering. Use mp4box or l-smash muxer.

Thanks, the file has been replaced.

[colinhunt]

I thought it might be fun to resurrect this thread a bit, and to see how codecs have improved since karasu created the comparison. I re-created his source and used either the same or equivalent cmdlines to encode new x265 and vp9 files, then ran some 3ssim comparisons.

(post-re-doing karasu's encodes)

So, yeah, I'm a dumbass. Nevcairiel spotted a problem with the earlier metric graphs. The problem was that I used karasu's VP9 and x265 encodes that he had made from his source file. I did not have his source file so I created one from the same Tears of Steel PNGs, using the same method, and thinking that of course the result is an identical source.y4m. Well, nope.

I tried to find the exact same encoder exes karasu used and did find vp9 v1.6.1 and x265 v2.3, although not exactly the same build numbers. I'm re-encoding "old-vp9" and "old-x265" files from my source file.

Here's the 3ssim metric for x265 v2.3 and x265 v2.7 versus source:

https://saitti.kuvat.fi/kuvat/VQMetrics/ToS400f--x265v2.3_vs_x265v2.7--3ssim.png/_medium.jpg

Average score (dotted line) is a little bit higher for v2.7.

Here's the 3ssim for VP9 1.6.1 and VP 1.7.0 versus source:

https://saitti.kuvat.fi/kuvat/VQMetrics/ToS400f--vp9v161_vs_vp9170--3ssim.png/_medium.jpg

1.7.0 has a very slightly higher average score.

And here's the VP 1.7.0 and x265 v2.7 versus source 3ssim graph:

https://saitti.kuvat.fi/kuvat/VQMetrics/ToS400f_vp9_v1.7.0-177_vs_x265_v2.714_3ssim.png/_medium.jpg

VP9 has a higher average score than x265 in this metric.

[nevcairiel]

It seems curious that old-VP9 and old-x265 have basically the same shape and practically identical values in the graph, only extremely minor differences. That seems somewhat unlikely, considering the "new" graphs look quite a bit different.

[colinhunt]

It seems curious that old-VP9 and old-x265 have basically the same shape and practically identical values in the graph, only extremely minor differences. That seems somewhat unlikely, considering the "new" graphs look quite a bit different.
Yup, I done goofed. Thanks for pointing it out!

[colinhunt]

Here's an interesting one. 2-pass 545kbps VP9 1.7.0 versus 233kbps 1-pass xvc. The outcome's not in doubt for one second, but xvc puts up a good fight between frames 200 and 300.

https://saitti.kuvat.fi/kuvat/VQMetrics/ToS400f--xvc233kbps_vs_VP9_170.png/_medium.jpg

[Asmodian]

That is interesting, that section is also where x265 does noticeably better than VP9. I wonder if it is simply a difference in bit rate distribution.

[colinhunt]

I re-encoded the xvc sample with qp33 instead of qp40, resulting in a bitrate of 587 kbps. Still single pass, still using speed-mode 2 (i.e. Fast):

https://saitti.kuvat.fi/kuvat/VQMetrics/ToS400f--vp9v170_vs_xvc587kbps.png/_medium.jpg

xvc got a higher average score than vp9 in this metric, although with a slightly higher bitrate than vp9 encode.

[colinhunt]

The 587kbps xvc encode looks pretty good, all things considered, so I piped it from xvc decoder to ffmpeg and encoded the output in lossless h.264.

190 MB / 94 Mbps, take a peek: https://drive.google.com/open?id=1LgNinJWST4I8zAoZBzfi9uYqLnYQ9Aw8