Versatile Video Coding (VVC) / H.266: HEVC successor

[pandy]

https://jvet.hhi.fraunhofer.de/

Up to 64 more computationally complex than H.265 for encoding and perhaps even 16x times for decoding...

Any thoughts?

[burfadel]

Needs to be more efficient than that time wise.

[Jamaika]

I know for sure. HEVC codec discouraged me from Fraunhofer. There was a long decoding time in JCT-VC codecs used in BPG images.
http://hevc.kw.bbc.co.uk/git/w/jctvc-hm.git

For me another new codec

[hajj_3]

https://jvet.hhi.fraunhofer.de/

Up to 64 more computationally complex than H.265 for encoding and perhaps even 16x times for decoding...

Any thoughts?

where does it say that?

[sneaker_ger]

The increase of computational complexity compared to HEVC is reflected by the fact that
the encoder software run time increases by factors of approximately 12 and 60 in RA and
AI configurations, respectively. Correspondingly, the decoder run time increases by factors
of approximately 10 and 2.5 in RA and AI, respectively. These are average numbers over
the entire set of sequences from [3]. The worst case complexity may even be more dra-
matically higher compared to HEVC.
https://show.ibc.org/__media/Technical-Papers-2017/JR-Ohm---STATUS-AND-PERSPECTIVES-OF-VIDEO-CODING.pdf

[hajj_3]

https://show.ibc.org/__media/Technical-Papers-2017/JR-Ohm---STATUS-AND-PERSPECTIVES-OF-VIDEO-CODING.pdf

• Submission deadline: February 2018.
• Evaluation of responses: April 2018.
• First test model: October 2018.
• First version of new video compression standard: October 2020.

does this mean that october 2020 is when it would be ratified or does this happen at some point after the "first version"?

[benwaggoner]

https://jvet.hhi.fraunhofer.de/

Up to 64 more computationally complex than H.265 for encoding and perhaps even 16x times for decoding...

Any thoughts?
Encoding time in a reference encoder doesn't really matter. 64x generally means it has 64x more ways to do things than HEVC, but real-world encoders aren't going to do an exhaustive search of all those modes! They'll use heuristics and early exits to deliver as good quality as possible within available time. As encoders have always done since the beginning of time. x265 is >>100x faster than the HEVC HM reference encoder even with --preset slower, for example.

Decoder time of 16x would be a huge problem. HEVC was carefully designed to have no more than 2x the complexity of H.264 even with all the options on. No one would ever come out with a video codec standard that requires 16x the silicon area or clock speed or memory or anything. Even MPEG-2 -> HEVC was only about 4x the decoder complexity per pixel at a given quality.

[iwod]

What sort of bitrate reduction are they expecting?

[sneaker_ger]

At this point they don't seem to have set any public goal and instead say it's a "study" to see what may be possible with new tools.
As of the current status, the bit rate reduction of JEM6, when compared to an HM16 soft-
ware encoder implementing the HEVC Main 10 Profile, is around 30 % in Random Access
(RA) configuration using motion compensation in a hierarchical B picture structure, and
around 20% in All Intra (AI) configuration (without motion compensation). This result was
obtained when averaging the bit rate reduction comparing at same PSNR by the so-called
Bjøntegaard Delta criterion
[...]
Measuring PSNR can be misleading as a criterion for judging quality. In order to assess
the subjective visual benefit, expert viewing tests were performed during the 7 th JVET
meeting in the context of evaluating responses to the Call for Evidence[...]Typically, bit
rate savings at same visual quality of between approximately 35 and 60% were observed.

Just read the doc from post #5 ...

[iwod]

At this point they don't seem to have set any public goal and instead say it's a "study" to see what may be possible with new tools.


Just read the doc from post #5 ...

Great, the Web site were...ugh.

It is good they recognized they have an Open Source and Free Codec competition. Since they have gobbled up enough money with H.265 I hope they sort out their license fees and structure first BEFORE moving on.

Edit: I know that is not the purpose of the study. But they surely need to keep this in mind.

I am on the HEVC side for now, this is simply because it is the only choice. I dont think AV1 will go anywhere. But AV2, I hope they get it right.

[pandy]

Encoding time in a reference encoder doesn't really matter. 64x generally means it has 64x more ways to do things than HEVC, but real-world encoders aren't going to do an exhaustive search of all those modes! They'll use heuristics and early exits to deliver as good quality as possible within available time. As encoders have always done since the beginning of time. x265 is >>100x faster than the HEVC HM reference encoder even with --preset slower, for example.

Decoder time of 16x would be a huge problem. HEVC was carefully designed to have no more than 2x the complexity of H.264 even with all the options on. No one would ever come out with a video codec standard that requires 16x the silicon area or clock speed or memory or anything. Even MPEG-2 -> HEVC was only about 4x the decoder complexity per pixel at a given quality.

But when we compare x264 and x265 then increasing complexity of the JET looks quite unpleasant... I know that overall trend is to use more threads but still - increasing computational complexity (exponentially) quickly become serious issue.
Those 64x and 16x are just worse case scenario a bit exaggerated figures.
In 80 when MPEG-1 (and later 2) was born every 2 - 3 years CPU computational power was almost doubled, now we observe severe stagnation on this - rarely new CPU's are providing more than 20% processing gain...

[bstrobl]

But when we compare x264 and x265 then increasing complexity of the JET looks quite unpleasant... I know that overall trend is to use more threads but still - increasing computational complexity (exponentially) quickly become serious issue.
Those 64x and 16x are just worse case scenario a bit exaggerated figures.
In 80 when MPEG-1 (and later 2) was born every 2 - 3 years CPU computational power was almost doubled, now we observe severe stagnation on this - rarely new CPU's are providing more than 20% processing gain...

We are slowly romping up against physical limits of chip production, and codec development is doing pretty much the same. Many of the changes are now an attempt at trying to discern an improvement or if its just noise due to the huge amounts of coding tools and their tiny contribution. At some stage it simply won't be worth upgrading codecs due to insane encode/development costs.

The JET folks really need to sort out their licensing mess though since AV1 can encode similar quality when compared to HEVC at 75% of the bitrate. Granted, AV1 seems a bit rushed but as long as most of the nagging issues from VP9 are fixed more people will hop onto it and stay if patent costs keep increasing with the h.26x codecs.

[iwod]

But when we compare x264 and x265 then increasing complexity of the JET looks quite unpleasant... I know that overall trend is to use more threads but still - increasing computational complexity (exponentially) quickly become serious issue.
Those 64x and 16x are just worse case scenario a bit exaggerated figures.
In 80 when MPEG-1 (and later 2) was born every 2 - 3 years CPU computational power was almost doubled, now we observe severe stagnation on this - rarely new CPU's are providing more than 20% processing gain...

Yes, and it also happens the world have moved to Mobile, and we dont use CPU to do decoding anymore, in most cases we are doing Hardware accelerated decoding with Specific Hardware or DSP, and it also happen they are 10x more efficient at doing so.

But i do understand the concern. If I remember x265 started off being 5x to 10x slower then x264. And the reference encoder were very slow.

I am mainly concerned about two things,

1. Real Time Encoding, that is if you are encoding a live sports like Football Match, I think we are running into some limitation there in regards to Speed / Bitrate and Quality.

2. Decoding: As benwaggoner mentioned, the decoding complexity hasn't increased much, MPEG-2 -> HEVC is only 4x per pixel, which is very little increase in that space when we had 10 - 20x computation power. What has increased though is we moved from 480P to 4K, that is ~20x Pixels.

Unless 3D, VR, or some other Killer Apps comes in, JET or AV2 will properly be the last video codec improvement we see. Pretty much like Audio Codec, bandwidth increase in 5G will means most of those bitrate saving being less of a concern.

[benwaggoner]

Unless 3D, VR, or some other Killer Apps comes in, JET or AV2 will properly be the last video codec improvement we see. Pretty much like Audio Codec, bandwidth increase in 5G will means most of those bitrate saving being less of a concern.
You know, I thought the same thing back in early 1997. We could almost deliver (analog 480i) broadcast quality on a CD-ROM! What was there left to do? I was actively planning my career pivot to film restoration.

Then Peter Jacobsen called up and asked if we can use this new RealVideo beta thing to embed some golf clips into his web site without buffering.

xHE-AAC is a big material improvement for important markets. We could have 10x the efficiency of HEVC today and deliver pristine UHD HDR over 4G networks.

Or deliver a great full-screen experience over a 2G network to people riding a bus in rural India. 720p @ 20 Kbps would still be a worthwhile improvement over 720p @ 30 Kbps.

I am confident that incremental improvements in compression efficiency will remain a multi-billion dollar market through my retirement. The main thing that could cause things to slow down is if we really hit a wall in terms of silicon process nodes, and stop getting big year-on-year MIPS per watt gains. If we were stuck on the same node, we might start running out of exciting new things to do after a decade or so.

There is still a huge market for improved MPEG-2 encoders...

[benwaggoner]

But when we compare x264 and x265 then increasing complexity of the JET looks quite unpleasant... I know that overall trend is to use more threads but still - increasing computational complexity (exponentially) quickly become serious issue.
Those 64x and 16x are just worse case scenario a bit exaggerated figures.
The real comparison should be quality at equal perf. If a codec offers a 100% improvement at 64x encoding time, but a 50% improvement at the same encoding time, it's still a big win. I doubt anyone is doing HEVC with the same coverage of theoretical options as they do with H.264, but HEVC still delivers some big real-world efficiency improvements at practical speeds.

These theoretical encoding times are really JM versus HM; encoders which are far too slow for any practical use, and really not very good psychovisually. And for decoder complexity, that's what Profiles and Levels are for. If there is a 16x worst case, probably 85% of the practical efficiency gains can be found in the first 2x of decoder complexity. Hitting the right balance between efficiency gains and decoder complexity is a huge part of designing a major bitstream format, and one of the reasons that the MPEG/ITU codecs are so good. There are just so many eyeballs from so many different industry sectors pounding on it to squeeze every last bit and MIPS out.

In 80 when MPEG-1 (and later 2) was born every 2 - 3 years CPU computational power was almost doubled, now we observe severe stagnation on this - rarely new CPU's are providing more than 20% processing gain...
Per core and for general compute, perhaps. But whenever a chip is "5-40%" faster, video encoding is the 40%. Because we can go widely parallel, and use SIMD instructions like AVX2 harder than about anything. And the good implementations get so much hand-tuned assembly optimizations.

The new 8th gen Core processors look to offer ~2x encoding throughput per dollar and per watt. More cores, less thermal throttling using AVX/AVX2, microarchitectural improvements, and hopefully some value in AVX512.

This is a bigger leap than most generations, but a slowdown in Moore's law will hit video the least and latest of almost technology.

[LigH]

At the moment, the sources appear to be available with building solutions for some MSVC versions and a makefile for Linux; would it be hard to adapt the latter for MSYS/MinGW? I would ask that in their repo as well...

[nevcairiel]

Linux makefiles is what MinGW uses.

[LigH]

I just don't know if they require some quirks in details; Multicoreware has separate build files for msys and linux, that may be related to CMake though...

[foxyshadis]

It seems like hikvision is the only one who's shown any interest this year, and I actually know a few people there. I should reach out and see what they're working on; aside from that, the project is essentially dead. I guess improving HEVC is still the only priority for all the big research companies.

[qyot27]

I just don't know if they require some quirks in details; Multicoreware has separate build files for msys and linux, that may be related to CMake though...
They do that for 'convenience' (although whose convenience is beyond me), not because it's anywhere near necessary. I've never used those build scripts for either Linux or cross-compiled MinGW builds - there's virtually no difference in how to configure CMake, which is all those scripts even do anyway - they're not actual Makefiles or MSVC solutions, only .sh and .bat scripts to tell CMake to use the Makefiles or Visual Studio generators.

The only thing that presents an actual need to configure them differently is that MinGW requires specifying a cross-compiling toolchain file - and that's only necessary when actually cross-compiling (read: Linux, OS X, or Cygwin, not MSys2).

[LigH]

So, chances are good that the makefile will work with the native compiler set of either MinGW32 or MinGW64 ... maybe I try to compile it then. Despite the last change being half a year old.

[colinhunt]

^ I'd be interested in testing the binary if you manage to compile one.

[LigH]

Unfortunately I did not. There are some errors (several "enum constant in boolean context" warnings treated as errors in extlib/Eigen/src/Core/Assign.h). I may get further if I knew how to disable this strictness?!

Paste: JVET JEM rev. 595 in MinGW64 (https://pastebin.com/Z3JJ2bft)

[LigH]

I commented out the CPPFLAGS option "-Werror" in build/linux/common/makefile.base to get the software built anyway; so expect it to break later... :o

JVET (https://www.mediafire.com/?pdekmpyk4e61m) rev. 595 (HM-16.6-JEM-7.1) — small archive for the minimum software selection, large only for the curious

[Tommy Carrot]

Thanks for the build, so far it's working without any crashes. I cant say anything about the quality yet, but the encoding speed is not that bad. I mean it's very slow, but not AV1 slow. :D

Edit: nevermind, on a higher resolution (720p) test encoding, it's much slower in comparison, the encoding speed is around halfway between AV1 --cpu=used=0 and --cpu-used=1. The quality at low bitrate is VERY impressive though. Easily better than AV1 or XVC.

[iwod]

It seems like hikvision is the only one who's shown any interest this year, and I actually know a few people there. I should reach out and see what they're working on; aside from that, the project is essentially dead. I guess improving HEVC is still the only priority for all the big research companies.

I thought they have some Demo in NAB?

[LigH]

I do not even know what "NAB" means; but yes, it was reported in the thread about AOM.

[nevcairiel]

"National Association of Broadcasters", they just had their tradeshow last week where content creators and codec people like to show off.

[LigH]

Could a moderator please fix the typos in the thread title, as I reported a few days ago?

[Tommy Carrot]

I've done some more tests with this codec. It's unfortunate that even if they finish the development it's probably doomed because of patents and licensing issues, because from a technical standpoint, it seems to be significantly better than AV1. It has consistently better (and more consistent) quality, in some cases by a pretty big margin.

[hajj_3]

I've done some more tests with this codec. It's unfortunate that even if they finish the development it's probably doomed because of patents and licensing issues, because from a technical standpoint, it seems to be significantly better than AV1. It has consistently better (and more consistent) quality, in some cases by a pretty big margin.

The other big problem is that it will be ratified 2.5yrs after AV1. There will be huge numbers of phones, tablets, laptops, desktops, tvs etc with hardware decoders shipping by the time h.266 is ratified. I reckon within 1yr of h.266 is ratified that AV2 will be ratified, which will likely have 0-15% better compression than h.266.

[colinhunt]

JVET (https://www.mediafire.com/?pdekmpyk4e61m) rev. 595 (HM-16.6-JEM-7.1) — small archive for the minimum software selection, large only for the curious
Thanks!

[iwod]

The other big problem is that it will be ratified 2.5yrs after AV1. There will be huge numbers of phones, tablets, laptops, desktops, tvs etc with hardware decoders shipping by the time h.266 is ratified. I reckon within 1yr of h.266 is ratified that AV2 will be ratified, which will likely have 0-15% better compression than h.266.

Nearly 1.4 Billion of Smartphone, 150M Tablet, 250M Notebook and Desktop. 200M of TV, Camera, STB, and other consumer electronics combined per year

That is 2 Billion unit every year from Consumer Market. Excluding the professional uses.

If they only take 50 Cents / unit, that is annual $1B revenue stream. If you expect the codec to last 10 - 20 years, that is total $20 Billion return.

With No Cap. The more unit you sell the more you pay.

The software implementation should be free. I mean, the codec would now be so complex it is literally impossible to play without a top range general hardware or dedicated decoder. Along with free image format.

[augman000]

Thanks for the build, so far it's working without any crashes. I cant say anything about the quality yet, but the encoding speed is not that bad. I mean it's very slow, but not AV1 slow. :D

Edit: nevermind, on a higher resolution (720p) test encoding, it's much slower in comparison, the encoding speed is around halfway between AV1 --cpu=used=0 and --cpu-used=1. The quality at low bitrate is VERY impressive though. Easily better than AV1 or XVC.

I'm going to start doing some comparative testing of my oen today.

When you say "low bitrate", are you talking 512kbps or less? Also, have you gone below 480p in your low bitrate tests?

[Tommy Carrot]

I'm going to start doing some comparative testing of my oen today.

When you say "low bitrate", are you talking 512kbps or less? Also, have you gone below 480p in your low bitrate tests?

On low bitrate, generally i mean around or under the size of x264 crf 30. But i tested it waaaay lower and in normal range as well, on different clips. Generally my finding is that the lower the bitrate, the larger JVET's advantage is (especially on cartoon stuff).

However, be prepared that the encoder is very slow, and cannot read y4m, only raw yuv. I only had the patience to test it on few seconds long 720p and lower res videos. But as the encoder is single-threaded, you can run a couple encodings in parallel.

[augman000]

On low bitrate, generally i mean around or under the size of x264 crf 30. But i tested it waaaay lower and in normal range as well, on different clips. Generally my finding is that the lower the bitrate, the larger JVET's advantage is (especially on cartoon stuff).

However, be prepared that the encoder is very slow, and cannot read y4m, only raw yuv. I only had the patience to test it on few seconds long 720p and lower res videos. But as the encoder is single-threaded, you can run a couple encodings in parallel.

Sure, I've been playing around with AV1 for quite some time now, so I've learned to be very patient.

Tried to run a quick test on lunch with no luck. Could you give me a sample vommand line syntax? I obviously overlooked or misunderstood something when I looked over the usage info.

Thanks.

[Tommy Carrot]

Here is an example:
TAppEncoderstatic.exe -c config-samples\encoder_randomaccess_jvet10.cfg -q 25 -fr 25 -wdt 720 -hgt 288 -f 50 -i test.yuv -b jvet.bin

I encoded a 720x288 video at 25 fps, and encoded 50 frames in total. You have to set every parameter manually, the encoder is kinda dumb.

In the config file, i set the intra period to 256 frames, i think the default is too low.

[augman000]

Here is an example:
TAppEncoderstatic.exe -c config-samples\encoder_randomaccess_jvet10.cfg -q 25 -fr 25 -wdt 720 -hgt 288 -f 50 -i test.yuv -b jvet.bin

I encoded a 720x288 video at 25 fps, and encoded 50 frames in total. You have to set every parameter manually, the encoder is kinda dumb.

In the config file, i set the intra period to 256 frames, i think the default is too low.

Ah, I think I already see my mistake.

Yhanks for the help.

[hajj_3]

Could a moderator rename this thread please as this codec has an official name now 'Versatile Video Coding' (VVC): http://news.itu.int/versatile-video-coding-project-starts-strongly/

[LigH]

I wonder if a new source repo will be announced. The Fraunhofer JVEM SVN had a few updates to r599 to prepare a "transition to VTM".

[TomV]

Could a moderator rename this thread please as this codec has an official name now 'Versatile Video Codec' (VVC): http://news.itu.int/versatile-video-coding-project-starts-strongly/

Versatile Video Coding

[AntiqueRefrigerator]

so is the war about codecs changing from quality to cpu cycles? av1 seems to be barely better than hevc and takes 1000x longer. sure you can have more quality, but what about creation time? do we still care about that?

[iwod]

so is the war about codecs changing from quality to cpu cycles? av1 seems to be barely better than hevc and takes 1000x longer. sure you can have more quality, but what about creation time? do we still care about that?

If you look back at video codec history, CPU cycle questions were always on the table. The only difference is 20 years we didn't even know where the limits are, we thought we could infinitely scale, 10Ghz+ processors, or 64 Core CPU and parallelism would work out fine.

Now that we have reached a plateau, these questions becomes much more important.

[iwod]

http://blog.chiariglione.org/the-mpeg-machine-is-reasy-to-start-again/

Sounds like VVC is coming along nicely. I wish they could aim a little higher then the 50% improvement though.

There is another pieces about the problem of IP.

http://blog.chiariglione.org/ip-counting-or-revenue-counting/

[benwaggoner]

so is the war about codecs changing from quality to cpu cycles? av1 seems to be barely better than hevc and takes 1000x longer. sure you can have more quality, but what about creation time? do we still care about that?
No one cares about exhaustive search, which is where the orders of magnitude speed increases come for. Every tool has to get tested every way, so there's a combinatorial explosion.

The real metric is efficiency @ perf. If a new codec is able to deliver better quality in the same encoding time, that's a win. And as Moore's Law marches on, the number of MIPS/pixel we are willing to spend keeps going up up up.

x265 beats x264 for most scenarios with the same encoding time, AND it can provide much more efficient encoding for the patient.

Any new codec has to show that it can provide significant efficient gains in reasonable encoding time. And quality @ perf gets way better in the initial years after a new bitstream is defined as performance and psychovisual tuning gets done.

Increases in decoder complexity are a bigger barrier, since that goes to the cost of all those decoders in all those devices. If something takes 4x the silicon and milliwatts to decode versus an existing standard, it would have to be REALLY more efficient.

MPEG-2 -> H.264 -> HEVC were about 2x more complex per pixel, for about 2x better efficiency. Generally more complexity or lower efficiency gains don't really move the market (see MPEG-4 part 2, Theora, VP6-9)

[benwaggoner]

The other big problem is that it will be ratified 2.5yrs after AV1. There will be huge numbers of phones, tablets, laptops, desktops, tvs etc with hardware decoders shipping by the time h.266 is ratified. I reckon within 1yr of h.266 is ratified that AV2 will be ratified, which will likely have 0-15% better compression than h.266.
The delay of the bitstream freeze has pushed hardware AV1 decode back a year and we'll probably only see significant numbers of AV1 HW decode devices shipping in 2020. So if H.266 is looking promising by then, or if AV1 hasn't surpassed HEVC in quality @ perf, the big drivers of adoption might not happen. AV1's compression efficiency improvements are initially promising, but it needs to demonstrate significant gains versus HEVC in double-blind visual testing at similar encoding complexity.

Does anyone have any sense of the increased HW decoder complexity of AV1 versus HEVC, and how much is additive (versus the same decoder using the same functional blocks for multiple codecs)?

I worry that the current AV1 encoder is so slow that there really isn't a big enough corpus of encoded content to figure out what quality/efficiency delta could be anticipated in the real world in a given time frame.

The cost to add AV1 decode depends on the delta in silicon area and milliwatts. I've not heard any estimates for that delta yet. Anyone else? Obviously the cheaper the cost to add decode, and the bigger the relative value, are both going to be big factors in adoption.

[foxyshadis]

MPEG-2 -> H.264 -> HEVC were about 2x more complex per pixel, for about 2x better efficiency. Generally more complexity or lower efficiency gains don't really move the market (see MPEG-4 part 2, Theora, VP6-9)

I really believe that it's been closer to 1.4x quality for 2x complexity, at some "midrange" (when you insert ASP between MPEG-2 and AVC). Each generation gets 2x or more better at the low-end, and barely budges the needle at the top-end, but some magical middling rate that's barely objectionable is where codecs are really put to the test. HEVC easily qualifies for "not worth the hassle" except for its day-1 promise of built-in hardware-accelerated 10-bit and HDR; except at the absolute lowest bitrates it never comes close to living up to its half-rate promise. If AVC RExt had found its way into hardware, HEVC's use case would have been squeezed to nothing, considering the ongoing licensing spats, while the world again waited for the next world heavyweight champion to appear.

A quadratic increase in complexity for a linear increase in efficiency sucks, but that's what we have the remnants of Moore's law for.

[hajj_3]

https://streaminglearningcenter.com/codecs/impressive-early-results-versatile-video-coding-vvc.html

[LigH]

I am not able to build the "VTM Repository" (trunk) in MSYS2 / MinGW: If you try to create the Makefile with the CMake generator "MinGW Makefiles", it complains that sh.exe must not be in the path; if you try the generator "MSYS Makefiles", BBuildEnv.cmake warns that it is not fully supported, and make fails with:

jvet-svn/source/Lib/CommonLib/x86/CommonDefX86.h:158:71: error: SSE vector return without SSE enabled changes the ABI [-Werror=psabi]

It seems to be mostly optimized for Visual C++.

[Blue_MiSfit]

Does anyone have any sense of the increased HW decoder complexity of AV1 versus HEVC, and how much is additive (versus the same decoder using the same functional blocks for multiple codecs)?

I worry that the current AV1 encoder is so slow that there really isn't a big enough corpus of encoded content to figure out what quality/efficiency delta could be anticipated in the real world in a given time frame.


Well said. I'd like to see a lot more of this.