Benwaggoner HEVC encoding challenge

[benwaggoner]

With all the talk about VP9 and AV1 and VVC and different encoder implementations for all of them, I thought it might be fun to set up an open challenge for folks to deliver the best possible quality for each.

To that end, in a semi-inebriated late-night conversation at IBC, we defined a scenario relevant to some important real-world scenarios that reasonably stress current encoders.

• Tears of Steel
• 1.0, 1.5, and 2.0 Mbps ABR
• 4 Mbps peak bitrate
• 12 Mbps VBV
• Max 5 sec (120 frame) GOP
• No preprocessing

Here's the source I used (.y4m.7z): https://1drv.ms/u/s!AlvIQZWsyeO-kKpoG5f3PHAFhWcGig

And my current best HEVC x265 encodes:
1.0: https://1drv.ms/v/s!AlvIQZWsyeO-kKpmDQlxoY-00rk3AQ
1.5: https://1drv.ms/v/s!AlvIQZWsyeO-kKplp2EQ8-Q4bCNVZw
2.0: https://1drv.ms/v/s!AlvIQZWsyeO-kKpZkpw-WQKF1dSF_g

I didn't do any content specific encoding in these. I just did the slowest, highest quality encode 2-pass I had the patience for. Basically

• --preset placebo
• --cu-lossless
• --tskip
• -F 1
• --ref 6
• --bframes 16
• --aq-mode 3
• --rd-refine

If I had infinite patience I'd add --me sea and --subme 7 for PlusUltraPlacebo.

Update 6/3/2021
To evaluate anime/line art encoding I'm adding Netflix's Sol Levante as a second source for testing. It's an interesting compression challenge with a lot of titles/credits that can be shrunk down enormously, and some very complex hybrid line art/CGI sequences that x265 can't make look great at even 12 Mbps. It should be very interesting for rate control!

Sol Levante's 4K HDR source is available under Creative Commons, from which I derived an 8-bit SDR 1080p .y4m, as 8-bit SDR playback and testing is still a lot more available.

I've kicked off a 1000 Kbps test encode with x265 for reference that I should post tomorrow. The challenge follows the same constraints for bitrate, VBV, etcetera.

[Forteen88]

Thanks. But why not set this? --profile main10
10-bits might help a little with banding maybe.
Is there any reason not to set --profile main10 in x265? Most of latest years GPU:s supports 10-bit H.265-decoding partially or fully.

EDIT: OK, thanks Blue_MiSfit, although watching video at this high resolution on an Android phone (small screen) is just wasteful.

[benwaggoner]

Originally Posted by Forteen88

Thanks. But why not set this? --profile main10
10-bits might help a little with banding maybe.
Is there any reason not to set --profile main10 in x265? Most of latest years GPU:s supports 10-bit H.265-decoding partially or fully.

One big reason for me is that comparing 10-bit encodes is a lot harder, since different setups can use very different conversion methods. Something that gets a nice dither on one platform might get banding due to truncation on another. Sticking to 8-bit reduces

Also, I wanted to be able to compare 8-bit only codecs.

I'd love to do a HDR-10 test as a followup. I've not found good HDR test content with an appropriate license yet, though.

EDIT: OK, thanks Blue_MiSfit, although watching video at this high resolution on an Android phone (small screen) is just wasteful.

Yeah, this is really meant for bigger screens. A 250-500 Kbps SD version would an interesting followup test. Among other things, it would make getting an AV1 encode a lot more feasible.

[K.i.N.G]

Originally Posted by benwaggoner

I'd love to do a HDR-10 test as a followup. I've not found good HDR test content with an appropriate license yet, though.

You can get the source files (which include 32 bit exr files) from the Tears of Steel project.
It's all open source...

https://media.xiph.org/tearsofsteel/

[Sparktank]

Originally Posted by K.i.N.G

You can get the source files (which include 32 bit exr files) from the Tears of Steel project.
It's all open source...

https://media.xiph.org/tearsofsteel/

AFAIK, TOS hasn't been upgraded for HDR.
It's all SDR.

You can find custom edits on youtube where users graded it themselves.

[K.i.N.G]

Originally Posted by Sparktank

AFAIK, TOS hasn't been upgraded for HDR.
It's all SDR.

Thats why I mention the exr files and not the actual exported movie...
The source files are exr which are 32bit (or 16bit at worst, but standard exr is 32bit) thus you can safely grade/master them to 10 HDR, even 12bit if you want. There's plenty of dynamic range/headroom.

I'm not sure in which format the actual filmed/video parts are available though... I'm quite busy so i haven't checked.
I suppose they are in raw format with sufficient dynamic range aswell... but that would be a guess

[Blue_MiSfit]

Older devices with early HEVC decoders?

I know some FireTV devices and Android phones are 8 bit only

[smok3]

I like the chalenge, but the source is something i don't want to watch, any other possible candidates?

[benwaggoner]

Originally Posted by smok3

I like the chalenge, but the source is something i don't want to watch, any other possible candidates?

Sure. Anything that's publicly available and has an appropriate license.

I like Tears of Steel as something that's reasonably short that has a variety of appropriate characteristics. Live action, CGI, film grain, motion graphics in credits.

Any test is going to be somewhat arbitrary and focused on a particular scenario. The one here is very much about encoder features at least as much as bitstreams. It stresses psychovisual tuning and rate control, and ignores encoding performance.

An obvious followup test would be 1-pass CBR with fixed 2 second Closed GOP.

Something incorporating quality @ perf would be interesting, but it's way harder to compare and verify performance.

[Blue_MiSfit]

WOW.

That is quite impressive at 851 Kbps. Like, dramatically better than HEVC.

[jonatans]

Originally Posted by Blue_MiSfit

WOW.

That is quite impressive at 851 Kbps. Like, dramatically better than HEVC.

Thanks!

[abaxas]

Increase me-range (or whatever it's called in x265) for a ultra small increase in quality for a massive increase in encoding time.

Not sure if it applies to x265 but there is an ultra small quality bump for single thread encoding. Noting 4 encodes running on 1 core each will complete much faster than 4x single encodes running on all 4 cores.

[Emulgator]

Originally Posted by Blue_MiSfit
WOW.
That is quite impressive at 851 Kbps. Like, dramatically better than HEVC.

Second that. That 851kbit/s file blew my socks off. 0.024bit/pixel and perfectly watchable.
Single Hair visible and stays through motion, a little mosquito around titles
the girl's moving robotic arm pulls only minimum morphing artefacts through the leaves.
Giving bitrate of 1921 even that arm's morphing wake is almost gone, titles mosquito improve,
not missing any detail from a good blu-ray encode...
How was encoding time if I may ask ?

BTW, back in 2009 DS had test-encoded Big Buck Bunny with x264 Build 999+1 at around 0.03bpp,
yielding good results, but BBB is not as challenging as TOS motionwise, good parts of the picture stay static in BBB..
https://forum.doom9.org/showthread.p...ow#post1247874

[jonatans]

Originally Posted by Emulgator

Second that. That 851kbit/s file blew my socks off.

Thanks!

Originally Posted by Emulgator

How was encoding time if I may ask ?

I didn't capture the exact encoding time but if I recall correctly it was in the order of 24 hours per encode on my 4-core laptop (which would correspond to an average encoding speed of 0.2 fps).

[jonatans]

The xvc repo has been updated with the latest changes related to integration with ffmpeg: https://github.com/divideon/xvc/commits/master

As mentioned in an earlier post, the ffmpeg version with xvc support is available here: https://github.com/perher/FFmpeg/tree/release/4.0-xvc

That ffmpeg version can be built using the normal process (e.g. as described at https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu) with the additional steps of checking out and building xvc:

cd ~/ffmpeg_sources && \
git -C xvc pull 2> /dev/null || git clone --depth 1 https://github.com/divideon/xvc && \
mkdir xvc_build && \
cd xvc_build && \
PATH="$HOME/bin:$PATH" cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX="$HOME/ffmpeg_build" -DBUILD_TESTS=OFF ../xvc && \
PATH="$HOME/bin:$PATH" make && \
make install

And then add --enable-libxvc when building ffmpeg.

We have not tested integration with HandBrake which as far as I understand would require creation of a new module definition for xvc in HandBrake's contrib folder.

[benwaggoner]

Originally Posted by jonatans

Thanks!
I didn't capture the exact encoding time but if I recall correctly it was in the order of 24 hours per encode on my 4-core laptop (which would correspond to an average encoding speed of 0.2 fps).

That's about the encoding time I got on a 6-core system using --preset placebo --subme 7 --cu-lossless --tskip --me sea

I'll do a test encode to the same ABR this weekend with all the whistles on and see what x265 can do. Probably not as good, but it'd be nice to have an apples-to-apples comparison.

Still hoping someone will provide their best-effort AV1 and/or VP9 encodes!

[SmilingWolf]

Originally Posted by benwaggoner

Still hoping someone will provide their best-effort AV1 and/or VP9 encodes!

I could try a VP9 encode, but I have absolutely no idea what a best effort vpxenc cmdline would look like. Even the MSU preset from the 2017 report doesn't help, since most of the knobs they set are defaults anyway, while some others just go on blindly favoring blurring. Plus, they set --cpu-used to 1, so I don't really know what to think about those settings.

In a recent set of tests I run I used this:

vpxenc --codec=vp9 --frame-parallel=0 --tile-columns=2 --good --cpu-used=0 --tune=psnr --passes=2 --threads=2 --end-usage=q --cq-level={} --test-decode=fatal --ivf -o test.vp9.cq{}.ivf orig.i420.y4m

Perhaps I could set --tile-columns to 0 and call it a day.

EDIT: just to make sure, I'm downloading the ToS_1920x800_xdither.7z file. Is this good for the test?

[Atak_Snajpera]

I didn't capture the exact encoding time but if I recall correctly it was in the order of 24 hours per encode on my 4-core laptop (which would correspond to an average encoding speed of 0.2 fps).

Cool now we just need dual EPYC (ROME) 128C/256T and we can start thinking about encoding our movie collections with amazing ~7 fps ...

[benwaggoner]

Originally Posted by Atak_Snajpera

Cool now we just need dual EPYC (ROME) 128C/256T and we can start thinking about encoding our movie collections with amazing ~7 fps ...

THAT would be an amazing degree of parallelization!

[Atak_Snajpera]

Originally Posted by benwaggoner

THAT would be an amazing degree of parallelization!

Easy if you divide movie in chunks...