Benwaggoner HEVC encoding challenge

[benwaggoner]

With all the talk about VP9 and AV1 and VVC and different encoder implementations for all of them, I thought it might be fun to set up an open challenge for folks to deliver the best possible quality for each.

To that end, in a semi-inebriated late-night conversation at IBC, we defined a scenario relevant to some important real-world scenarios that reasonably stress current encoders.

• Tears of Steel
• 1.0, 1.5, and 2.0 Mbps ABR
• 4 Mbps peak bitrate
• 12 Mbps VBV
• Max 5 sec (120 frame) GOP
• No preprocessing

Here's the source I used (.y4m.7z): https://1drv.ms/u/s!AlvIQZWsyeO-kKpoG5f3PHAFhWcGig

And my current best HEVC x265 encodes:
1.0: https://1drv.ms/v/s!AlvIQZWsyeO-kKpmDQlxoY-00rk3AQ
1.5: https://1drv.ms/v/s!AlvIQZWsyeO-kKplp2EQ8-Q4bCNVZw
2.0: https://1drv.ms/v/s!AlvIQZWsyeO-kKpZkpw-WQKF1dSF_g

I didn't do any content specific encoding in these. I just did the slowest, highest quality encode 2-pass I had the patience for. Basically

• --preset placebo
• --cu-lossless
• --tskip
• -F 1
• --ref 6
• --bframes 16
• --aq-mode 3
• --rd-refine

If I had infinite patience I'd add --me sea and --subme 7 for PlusUltraPlacebo.

Update 6/3/2021
To evaluate anime/line art encoding I'm adding Netflix's Sol Levante as a second source for testing. It's an interesting compression challenge with a lot of titles/credits that can be shrunk down enormously, and some very complex hybrid line art/CGI sequences that x265 can't make look great at even 12 Mbps. It should be very interesting for rate control!

Sol Levante's 4K HDR source is available under Creative Commons, from which I derived an 8-bit SDR 1080p .y4m, as 8-bit SDR playback and testing is still a lot more available.

I've kicked off a 1000 Kbps test encode with x265 for reference that I should post tomorrow. The challenge follows the same constraints for bitrate, VBV, etcetera.

[Forteen88]

Thanks. But why not set this? --profile main10
10-bits might help a little with banding maybe.
Is there any reason not to set --profile main10 in x265? Most of latest years GPU:s supports 10-bit H.265-decoding partially or fully.

EDIT: OK, thanks Blue_MiSfit, although watching video at this high resolution on an Android phone (small screen) is just wasteful.

[Blue_MiSfit]

Older devices with early HEVC decoders?

I know some FireTV devices and Android phones are 8 bit only

[smok3]

I like the chalenge, but the source is something i don't want to watch, any other possible candidates?

[benwaggoner]

Originally Posted by smok3

I like the chalenge, but the source is something i don't want to watch, any other possible candidates?

Sure. Anything that's publicly available and has an appropriate license.

I like Tears of Steel as something that's reasonably short that has a variety of appropriate characteristics. Live action, CGI, film grain, motion graphics in credits.

Any test is going to be somewhat arbitrary and focused on a particular scenario. The one here is very much about encoder features at least as much as bitstreams. It stresses psychovisual tuning and rate control, and ignores encoding performance.

An obvious followup test would be 1-pass CBR with fixed 2 second Closed GOP.

Something incorporating quality @ perf would be interesting, but it's way harder to compare and verify performance.

[benwaggoner]

Originally Posted by Forteen88

Thanks. But why not set this? --profile main10
10-bits might help a little with banding maybe.
Is there any reason not to set --profile main10 in x265? Most of latest years GPU:s supports 10-bit H.265-decoding partially or fully.

One big reason for me is that comparing 10-bit encodes is a lot harder, since different setups can use very different conversion methods. Something that gets a nice dither on one platform might get banding due to truncation on another. Sticking to 8-bit reduces

Also, I wanted to be able to compare 8-bit only codecs.

I'd love to do a HDR-10 test as a followup. I've not found good HDR test content with an appropriate license yet, though.

EDIT: OK, thanks Blue_MiSfit, although watching video at this high resolution on an Android phone (small screen) is just wasteful.

Yeah, this is really meant for bigger screens. A 250-500 Kbps SD version would an interesting followup test. Among other things, it would make getting an AV1 encode a lot more feasible.

[K.i.N.G]

Originally Posted by benwaggoner

I'd love to do a HDR-10 test as a followup. I've not found good HDR test content with an appropriate license yet, though.

You can get the source files (which include 32 bit exr files) from the Tears of Steel project.
It's all open source...

https://media.xiph.org/tearsofsteel/

[Sparktank]

Originally Posted by K.i.N.G

You can get the source files (which include 32 bit exr files) from the Tears of Steel project.
It's all open source...

https://media.xiph.org/tearsofsteel/

AFAIK, TOS hasn't been upgraded for HDR.
It's all SDR.

You can find custom edits on youtube where users graded it themselves.

[K.i.N.G]

Originally Posted by Sparktank

AFAIK, TOS hasn't been upgraded for HDR.
It's all SDR.

Thats why I mention the exr files and not the actual exported movie...
The source files are exr which are 32bit (or 16bit at worst, but standard exr is 32bit) thus you can safely grade/master them to 10 HDR, even 12bit if you want. There's plenty of dynamic range/headroom.

I'm not sure in which format the actual filmed/video parts are available though... I'm quite busy so i haven't checked.
I suppose they are in raw format with sufficient dynamic range aswell... but that would be a guess

[benwaggoner]

Originally Posted by K.i.N.G

Thats why I mention the exr files and not the actual exported movie...
The source files are exr which are 32bit (or 16bit at worst, but standard exr is 32bit) thus you can safely grade/master them to 10 HDR, even 12bit if you want. There's plenty of dynamic range/headroom.

I looked into this a few years back. While we have lots of precision in the source, it was all graded and rendered for 709 SDR. One could probably use those assets to remaster it to HDR, but that would require regrading, rerendering the CGI stuff, and recompositing. Feasible, certainly, and a lot less work than doing a project from scratch. But still a sizable effort to make it look like "Real" HDR, and that's even assuming they did the RAW acquisition preserving the highlights and such.

I've been involved in a number of HDR remastering projects, and doing it well is quite a lot of effort. Particularly for something as CGI-centric as Tears of Steel.

[K.i.N.G]

Originally Posted by benwaggoner

I looked into this a few years back. While we have lots of precision in the source, it was all graded and rendered for 709 SDR. One could probably use those assets to remaster it to HDR, but that would require regrading, rerendering the CGI stuff, and recompositing. Feasible, certainly, and a lot less work than doing a project from scratch. But still a sizable effort to make it look like "Real" HDR, and that's even assuming they did the RAW acquisition preserving the highlights and such.

The 32bit exr's that are available seem to have the 709 grading baked in indeed (which is weird) but its not that big of a deal since they kept the dynamic range and full exposure info in there (they aren't clipped). So, no problems there.
Those exr's seem to be saved in 'only' 1080p though...

Originally Posted by benwaggoner

I've been involved in a number of HDR remastering projects, and doing it well is quite a lot of effort. Particularly for something as CGI-centric as Tears of Steel.

CGI data is pretty much ideal. It doesn't get any better than that.
Of course you always have those people who render everything clipped and bake weird color corrections right into the renders, that's another story. But that also happens with video footage, so no difference in that regard.
CGI with the rec.709 profile baked in is still miles easier to work with than video footage with the camera's color response + some custom curve baked in though.

[benwaggoner]

[QUOTE=K.i.N.G;1854914]The 32bit exr's that are available seem to have the 709 grading baked in indeed (which is weird) but its not that big of a deal since they kept the dynamic range and full exposure info in there (they aren't clipped). So, no problems there.
Those exr's seem to be saved in 'only' 1080p though...

I am sure there are UHD ones as well. I think I downloaded them once some years ago. Maybe off an FTP?

My .y4m was generated from the 16-bit PNG sequence.

CGI data is pretty much ideal. It doesn't get any better than that.

The final has film grain added to the CGI, but if the EXR is clean plates, awesome!

Of course you always have those people who render everything clipped and bake weird color corrections right into the renders, that's another story. But that also happens with video footage, so no difference in that regard.
CGI with the rec.709 profile baked in is still miles easier to work with than video footage with the camera's color response + some custom curve baked in though.

My understanding is they didn't do anything specific to prevent >>709 things from happening. But they did all the work in 709, so wouldn't have known or worried about if anything in particular was clipped or whatever.

It would be awesome if someone put in the time to do a HDR master! From what you've found so far, it might not even be that big a project. Maybe a day or two for someone with the right skills, software, and hardware. Two days to remastered an hour show to HDR with decent sources is pretty luxurious, and this is only 11 minutes.

But anyway, I'd love to see anyone else's best encodes with the SDR clip, in x265, different HEVC, or even other codecs. In particular, I'd love to see a VP9 expert's best VP9 effort.

[jonatans]

Hi Ben, great challenge that you have arranged!

Here are encodes with the xvc codec at three different rates:

There is no strict rate control in xvc yet so the margins to the target rates are quite large.

851 kbps: https://drive.google.com/file/d/1xrM...ew?usp=sharing (15% under target rate)
1361 kbps: https://drive.google.com/file/d/1cRs...ew?usp=sharing (9% under target rate)
1924 kbps: https://drive.google.com/file/d/1K46...ew?usp=sharing (4% under target rate)

To play the files you need a player with xvc support such as this build of ffplay: https://drive.google.com/file/d/1gyj...ew?usp=sharing

The zip file also contains the build of ffmpeg that was used to encode the files using the following command:
ffmpeg.exe -i d:/ToS_1920x800_xdither.y4m -c:v libxvc -speed-mode 2 -qp X -max-keypic-distance 120 -threads 16 enc.mp4
with X being 28, 25 and 23, respectively.

It can be noted that these encodes are single pass, with no lookahead (beyond the 16 picture sub-GOP structure), and that reference pictures are stored with 10-bit precision internally (there is no 8 bit profile of xvc).

I would be happy to hear what people think of it! Thanks!

[smok3]

jonatans; Tryed with the latest build of mpv (Debian) and I guess that is not yet supported out of the box?

mpv xvc_tos_851kbps.mp4 
Playing: xvc_tos_851kbps.mp4
 (+) Video --vid=1 (*) ( 1920x800 24.000fps)
Failed to initialize a decoder for codec ''.
Video: no video
No video or audio streams selected.


Exiting... (Errors when loading file)

mpv --version
mpv 0.29.0-75-gda1073c247 Copyright © 2000-2018 mpv/MPlayer/mplayer2 projects
 built on Wed Oct 24 09:29:52 CEST 2018
ffmpeg library versions:
   libavutil       56.19.101
   libavcodec      58.33.102
   libavformat     58.19.102
   libswscale      5.2.100
   libavfilter     7.37.100
   libswresample   3.2.100
ffmpeg version: N-92246-gc2ac3b8e6

[jonatans]

Originally Posted by smok3

jonatans; Tryed with the latest build of mpv (Debian) and I guess that is not yet supported out of the box?

Correct.

In order to play the xvc files you need to use a player with xvc support such as the version of ffplay provided in the zip file I uploaded (https://drive.google.com/file/d/1gyj...ew?usp=sharing)

[Blue_MiSfit]

WOW.

That is quite impressive at 851 Kbps. Like, dramatically better than HEVC.

[jonatans]

Originally Posted by Blue_MiSfit

WOW.

That is quite impressive at 851 Kbps. Like, dramatically better than HEVC.

Thanks!

[ShortKatz]

Originally Posted by jonatans

The zip file also contains the build of ffmpeg that was used to encode the files using the following command:
ffmpeg.exe -i d:/ToS_1920x800_xdither.y4m -c:v libxvc -speed-mode 2 -qp X -max-keypic-distance 120 -threads 16 enc.mp4

Hi jonatans, how can I build a version of ffmpeg that supports xvc? I would need it for Mac and would like to include this ffmpeg in my version of HandBrake than. Thanks.

[abaxas]

Increase me-range (or whatever it's called in x265) for a ultra small increase in quality for a massive increase in encoding time.

Not sure if it applies to x265 but there is an ultra small quality bump for single thread encoding. Noting 4 encodes running on 1 core each will complete much faster than 4x single encodes running on all 4 cores.

[Emulgator]

Originally Posted by Blue_MiSfit
WOW.
That is quite impressive at 851 Kbps. Like, dramatically better than HEVC.

Second that. That 851kbit/s file blew my socks off. 0.024bit/pixel and perfectly watchable.
Single Hair visible and stays through motion, a little mosquito around titles
the girl's moving robotic arm pulls only minimum morphing artefacts through the leaves.
Giving bitrate of 1921 even that arm's morphing wake is almost gone, titles mosquito improve,
not missing any detail from a good blu-ray encode...
How was encoding time if I may ask ?

BTW, back in 2009 DS had test-encoded Big Buck Bunny with x264 Build 999+1 at around 0.03bpp,
yielding good results, but BBB is not as challenging as TOS motionwise, good parts of the picture stay static in BBB..
https://forum.doom9.org/showthread.p...ow#post1247874