Any news about a potential successor to VVC / H.266?

[benwaggoner]

Originally Posted by ksec

VVC 1.0 was standardised in June 2020 at the time it was VTM 9. But in reality it was 2.0 in 2022 April with all compliance and standard test ready. Which at the time it was VTM 16.

I think FVC / ECM 11 right now is much closer to VTM 9 / VVC 1.0 stage. And in terms of Next Gen Codec, this is by far the earliest reach by historic standard. Part of me think that is due to COVID and researcher got insane productivity while being stuck at home.

I still remember I was impressed by VVC during VTM development, well turns out we squeeze another ~30% BD-Rate out of it within 2-3 years time.

Codec developers have been worried we're running out of ideas for decades. But it never quite seems to happen, thank goodness. And thank Moore's Law allowing us to keep on increasing decode compute requirements some and encode compute requirements a bunch.

But on the other hand it will likely be the first codec that requires hardware decoder to work.

You think? Not having a software decoder for testing would add a lot of friction. Hopefully at least it could be done in a GPU accelerated implementation so we can get work done before HW decoders are available.

[Jamaika]

Intra Block Copy VVC errors have been corrected. Videos can be played.
Converting VTM videos requires adding framerate (-r). Video still stutters bit.
https://github.com/ffvvc/FFmpeg/pull/198

In function 'prepare_intra_edge_params_8',
    inlined from 'intra_pred_8' at extra/vvc_intra_template.c:627:5:
extra/vvc_intra_template.c:535:21: warning: writing 16 bytes into a region of size 0 [-Wstringop-overflow=]
  535 |             left[i] = top[i] = left[0];
      |             ~~~~~~~~^~~~~~~~~~~~~~~~~~
extra/vvc_intra_template.c: In function 'intra_pred_8':
extra/vvc_intra_template.c:625:21: note: at offset -105 into destination object 'edge' of size 3136
  625 |     IntraEdgeParams edge;
      |                     ^~~~

https://www.sendspace.com/file/f2zj4b

[ksec]

Originally Posted by benwaggoner

You think? Not having a software decoder for testing would add a lot of friction. Hopefully at least it could be done in a GPU accelerated implementation so we can get work done before HW decoders are available.

It is definitely the largest increase in terms of decoder complexity. At least not a viable options for vast majority of machines, especially Mobile Phones or Laptops. I dont think we will ever get a dav1d level decoder for VVC or FVC / H.267 ( FVC not being an official name ). And even if we assume we do get someone to write the insane amount of hand written assembly of dav1d. If we consider VVC being similar level of complexity as AV1. And we are looking at 8-10x decoding complexity of AV1 / VVC to FVC. You will need 8x more powerful computer to decode an FVC file with dav1d level of optimisation.

We are looking at an Apple M3 / or current Top Tier CPU from x86 to barely decode 1080P FVC files at 30fps with 100% CPU usage with an dav1d level of decoder ( Which we likely wont get ). I dont expect this level of CPU performance to filter through to the low end any time soon or even within next 10 years. And we haven't even talked about 4K.

So realistically any usage of FVC would requires dedicated hardware acceleration.

But Of course I hope I am utterly wrong.

[ksec]

From April's Meetings.

The rate reduction for natural sequences over VTM 11 in RA configuration for {Y, U, V} increased from ECM-11.0’s {-22.56%, -31.91%, -33.67%} to ECM-12.0’s {-24.01%, -33.20%, -35.34%}.

[Tommy Carrot]

I've been curious about ECM / H.267 for a while, and i could not find a working windows build of this codec anywhere, and i dont want to beg for builds either, so with some reluctance, i set up visual studio on my work PC (btw, it's cool and all, but i feel ~40 GB for it is a bit excessive.). To my surprise, i could compile ECM encoder without a hitch, no crashes, everything works as it should.

So i could finally try it myself. It is very slow, even slower than the AV2 encoder, so much so that testing it is kinda difficult. (a 100 frames long 720p clip took about 14 hours to finish). But the quality/efficiency is very impressive indeed. Nothing really comes close to it, especially at lower bitrates. It easily outclasses AV2 in it's current stage.

[birdie]

Originally Posted by Tommy Carrot

I've been curious about ECM / H.267 for a while, and i could not find a working windows build of this codec anywhere, and i dont want to beg for builds either, so with some reluctance, i set up visual studio on my work PC (btw, it's cool and all, but i feel ~40 GB for it is a bit excessive.). To my surprise, i could compile ECM encoder without a hitch, no crashes, everything works as it should.

So i could finally try it myself. It is very slow, even slower than the AV2 encoder, so much so that testing it is kinda difficult. (a 100 frames long 720p clip took about 14 hours to finish). But the quality/efficiency is very impressive indeed. Nothing really comes close to it, especially at lower bitrates. It easily outclasses AV2 in it's current stage.

Sounds great except VVenc loses badly to x265 under certain conditions and x266 is nowhere to be found and you're already talking about H.267.

https://github.com/fraunhoferhhi/vvenc/discussions/389

I'm not a fan of how VVC has been deployed so far.

[GeoffreyA]

Originally Posted by birdie

Sounds great except VVenc loses badly x265 under certain conditions and x266 is nowhere to be found and you're already talking about H.267.

https://github.com/fraunhoferhhi/vvenc/discussions/389

I'm not a fan of how VVC has been deployed so far.

I think the problem is that VVenC denoises the video during encoding, leading to a softer picture. Recently, working with anime carrying artifacts and quantisation noise, I saw evidence of this. I found that libaom wiped the video clean, leading to an acceptable if soft picture. With VVC, I saw the same effect: the picture was cleaned up, though AV1 did a better job. (At low bitrates, too, it seems to inherit HEVC's, or x265's, characteristic artifacts along lines.) I don't know if VVenC exposes the option to disable denoising, but if it did, video might be sharper.

[birdie]

Originally Posted by GeoffreyA

I think the problem is that VVenC denoises the video during encoding, leading to a softer picture. Recently, working with anime carrying artifacts and quantisation noise, I saw evidence of this. I found that libaom wiped the video clean, leading to an acceptable if soft picture. With VVC, I saw the same effect: the picture was cleaned up, though AV1 did a better job. (At low bitrates, too, it seems to inherit HEVC's, or x265's, characteristic artifacts along lines.) I don't know if VVenC exposes the option to disable denoising, but if it did, video might be sharper.

There are options for that but they invariably blow up the bitrate:

https://github.com/fraunhoferhhi/vvenc/discussions/388

[benwaggoner]

Originally Posted by GeoffreyA

I think the problem is that VVenC denoises the video during encoding, leading to a softer picture. Recently, working with anime carrying artifacts and quantisation noise, I saw evidence of this. I found that libaom wiped the video clean, leading to an acceptable if soft picture. With VVC, I saw the same effect: the picture was cleaned up, though AV1 did a better job. (At low bitrates, too, it seems to inherit HEVC's, or x265's, characteristic artifacts along lines.) I don't know if VVenC exposes the option to disable denoising, but if it did, video might be sharper.

I don't think it is denoising specifically, versus getting some higher QPs but with a codec that does a much better job of concealing block and particularly inter block artifacts.

Are you comparing AV1 and VVC at the same bitrates? I'd expect VVC to do somewhat better than AV1 at this in general. Of course, AV1 encoders are a lot more mature at this point.

Reencoding from source that already has video encoding artifacts is always a tricky challenge. Generally the more simlilar the codecs are, the better; reencoding from H.264 to HEVC is generally cleaner than from H.264 to VP9 as HEVC can fall back to pretty much symmetrically encoding the input pixels. Similarly, I'd expect VP9 to reencode better to AV1 than VVC, as they share so much common architecture (but I've not tested that).

As a best practice, cleaning up artifacts before reencoding is preferred. Garbage in is always at least as much garbage out, and often worse than that. The bitrates required to not come out worse than a source already encoded for distribution are often as high or higher than the original bitrate anyway. Reference and enterprise encoders are always tested and tuned on uncompressed sources, as that's what premium content has. Rest assured CrunchyRoll's sources don't have those kind of artifacts!

Given Google's heavy involvement in AV1 encoder development and use via YouTube, it would make absolute sense that libaom was tuned to handle artifacts typical in user-generated content, not just professional mezzanines or uncompressed test sources.

[GeoffreyA]

Memory wasn't the best, so I did a quick test again. I would say that, at low bitrates, AV1 and VVC are on the same footing, but could be trading one artifact for another, and that will be subjective. What's certain, though, is that the bad source was "cleaned up," leading to better picture. At higher bitrates, both are preserving the artifacts of the source, leading to worse picture. So, this supports your suggestion that higher QPs, along with concealment, are saving the day. (It could also be that denoising, if present, is being cut down with more bitrate.)

The source is x264-encoded, 5 Mbps, 480p anime supposedly taken straight from the Blu-ray; it appears to be from the same master used for the DVD releases. It is not in the best shape: soft, along with subtle ringing, I'd say. I agree that one should clean up artifacts before encoding, but in this case, AV1 killed two birds with one stone: more compression and better picture (to a limit)!

Generally, though, I find that VVC is slightly ahead of AV1 and a touch sharper.

[GeoffreyA]

Looks like there is some sort of denoising after all: MCTF. See birdie's link above as well as the following, p. 16:

https://ieeexplore.ieee.org/stamp/st...number=9503377

[FranceBB]

Originally Posted by benwaggoner

Rest assured CrunchyRoll's sources don't have those kind of artifacts!

Yep, I worked there in 2013 and we used to get Apple ProRes HQ files at 23,976p. Despite not being lossless, they were high quality mezzanine files. The whole licensing thing for Japanese Anime was a very big issue, though, which meant that according to which agreement you had, you could end up either getting a proper master or getting the same "TX Ready" master sent to Japanese Broadcasters and that was unfortunately XDCAM-50 (i.e a 50 Mbit/s Long GOP MPEG-2 stream at 29.970 interlaced TFF with 3:2 pulldown). Luckily it was still 1920x1080, albeit 8bit and with lots of banding. With anime being anime you also had the hard question of "what the heck do I do know" when some shows were clearly 23,976 with 3:2 pulldown BUT had either the opening or the ending 29.970 progressive. Do I deinterlace everything to 29.970p to preserve the opening/ending? Do I just IVTC everything to 23,976p (thus decimating the opening/ending)? The guideline back then was to always encode at 23,976p in those situations which is what we did. One of the worst mezzanine files we used to receive was Naruto Shippuden (I also always found odd that they kept using the same numbering as if the first series and the shippuden were the same thing, so it was like 220 + whatever episode number it was. Like episode 392 of the shippuden was 612). Why I'm mentioning Naruto in particular? Well, because we didn't have access to the TX file it was coming from and the guys at TV Tokyo were creating an high bitrate H.264 progressive file for all the streaming platforms licensing it for simulcast, but they were so not used to it that they didn't quite always encode it right, so you could end up with a 23,976fps progressive file in which some scenes had repeated frames as they were inverse telecined incorrectly. If anything, I think they were deinterlacing to 29,970 and then decimating to 23,976p so sometimes, if the pattern was right after they trimmed the clock, the colorbars etc, it matched and some other times it didn't if it was off by a few frames.



Anyway, from there our own mezzanine file was created (an H.264 1920x1080 level 4.1 4:2:0 8bit + AAC in mp4 with very very very high bitrate and almost always 23,976p) and then sent to the distribution encoder (along with the .ass) that would create the renditions automatically for the various resolutions and bitrates to populate the CDN. At that point, the publishing team would test those on the various devices to make sure everything was fine before the content was "unlocked" automatically at the scheduled time.


It feels like a lifetime ago, considering that I've abandoned the streaming sector a long time ago to dedicate to linear broadcasting and I've been working at Sky for 8 and a half years.

Originally Posted by benwaggoner

Given Google's heavy involvement in AV1 encoder development and use via YouTube, it would make absolute sense that libaom was tuned to handle artifacts typical in user-generated content, not just professional mezzanines or uncompressed test sources.

Yep and we can see this a lot in YouTube videos encoded AV1. While VP9 often presents plenty of artifacts, AV1 streams are generally much softer. This is because the AV1 approach is to blur and average out details rather than showing blocking while being bit-starved, so if the source already had artifacts it's very much possible that those ended up being "cleaned" by the same averaging out process. Whether that was intentional or not I'm not sure, but something tells me that AV1 was probably just trying to average out / smooth out problematic uncorrelated high frequency components and due to more luck than anything compression artifacts from low bitrate sources actually usually end up in that category.

Originally Posted by GeoffreyA

Generally, though, I find that VVC is slightly ahead of AV1 and a touch sharper.

I would be surprised if it wasn't.
The goal by MPEG for H.266 VVC was to at the very least be 35% more efficient than H.265 HEVC (with 40% being the target) and in general VVEnc tests show it to be around 13% more efficient than AV1. Mind you, VVEnc is just 4 years old.

[GeoffreyA]

Originally Posted by FranceBB

Yep, I worked there in 2013 and we used to get Apple ProRes HQ files at 23,976p.

It's interesting hearing about your experience in the industry. Most properly-encoded anime is almost always 23.976. Regarding that poor source I mentioned, namely "Sorcerer Hunters," there were other issues too. Inverse telecining in FFmpeg, with fieldmatch, decimate, and others, worked well for the main 26 episodes; but the OVA proved troublesome and I was left with combing in at least one scene. Same story in VapourSynth. So I've left it for now till I learn more.

Regarding AV1 and softness, I think future codecs will increasingly take this path because that is what is viewed, seemingly by many, as high quality these days. For my part, I find it regrettable.

[nevcairiel]

Originally Posted by GeoffreyA

Regarding AV1 and softness, I think future codecs will increasingly take this path because that is what is viewed, seemingly by many, as high quality these days. For my part, I find it regrettable.

This is not new, it started with HEVC already, in contrast to H264.
Of course the alternative to softness is blocking artifacts, like H264 was famous for. I rather have softness then blocking artifacts.
Thats why in many cases people instantly preferred HEVC at very low bitrates, since a soft image is more watchable then a blocky H264 encode

The real solution is more bitrate, but we aren't getting that.

Of course thats the same reason new codecs may appear a bit sharper again - at the same bitrate you get a bit more quality, less need to reduce details, eg. less soft.

[ksec]

Originally Posted by Tommy Carrot

I've been curious about ECM / H.267 for a while, and i could not find a working windows build of this codec anywhere, and i dont want to beg for builds either, so with some reluctance, i set up visual studio on my work PC (btw, it's cool and all, but i feel ~40 GB for it is a bit excessive.). To my surprise, i could compile ECM encoder without a hitch, no crashes, everything works as it should.

So i could finally try it myself. It is very slow, even slower than the AV2 encoder, so much so that testing it is kinda difficult. (a 100 frames long 720p clip took about 14 hours to finish). But the quality/efficiency is very impressive indeed. Nothing really comes close to it, especially at lower bitrates. It easily outclasses AV2 in it's current stage.

H.267 ECM ( I believe the original name for it was Future Video Codec aka FVC but I dont see it being used anywhere any more) has roughly 8 - 10x the encoding complexity compared to VVC's VTM.

Right now ECM is showing about 25% BD-R compared to VTM. And up to 50% for Text and Graphics with Motions.

It also increase decoding complexity by 8x, the highest we have seen in any codec generation. I am just not entirely sure how this will work on Mobile. May be we could use it with LCEVC at 720P to reconstruct 1080P files?

Using VVC + LCEVC, which is what the Brazil TV 3.0 are going to be using in 2025, is proving to be high quality and extremely resource efficient. I remember earlier this year Brazil Government and University published a paper showing anywhere from -10% ( using more Bit Rate ) to 60% reduction [1] compared to VVC alone. And this is with early stage VVC and LCEVC encoder.

Hopefully we will have other encoder to play with soon. I hope there will be a Beamr for VVC.





[1] Somewhat interesting is that LCEVC is developed by V-Nova, a British Company and being true to British fashion they absolutely down play the potential of the tech. Which is quite amusing to me. XD

[GeoffreyA]

Originally Posted by nevcairiel

This is not new, it started with HEVC already, in contrast to H264.
Of course the alternative to softness is blocking artifacts, like H264 was famous for. I rather have softness then blocking artifacts.
Thats why in many cases people instantly preferred HEVC at very low bitrates, since a soft image is more watchable then a blocky H264 encode

The real solution is more bitrate, but we aren't getting that.

Of course thats the same reason new codecs may appear a bit sharper again - at the same bitrate you get a bit more quality, less need to reduce details, eg. less soft.

I fully agee---there's no contest between the soft image and blocky one---and this has been happening since HEVC. But one does get the feeling, if strictly incorrect, that transparency is harder to achieve than before.

[benwaggoner]

Originally Posted by nevcairiel

This is not new, it started with HEVC already, in contrast to H264.
Of course the alternative to softness is blocking artifacts, like H264 was famous for. I rather have softness then blocking artifacts.
Thats why in many cases people instantly preferred HEVC at very low bitrates, since a soft image is more watchable then a blocky H264 encode

The real solution is more bitrate, but we aren't getting that.

Of course thats the same reason new codecs may appear a bit sharper again - at the same bitrate you get a bit more quality, less need to reduce details, eg. less soft.

I don't think there's anything keeping modern codecs from having as much detail as older ones, and in some cases they have tools that allow even greater detail than before. HEVC's transform skip and lossless CU options available in all profiles mean complex graphics with very sharp edges can be encoded perfectly, which isn't always possible in any frequency transform only codec.

It's more that the more advanced the codec, the better in-loop artifact reduction tools it has. Before in-loop deblocking, once a reference frame hit too high a QP, quality was trashed for the rest of the GOP, and you had shorter max GOP durations. Once you could get soft instead of blocky, future frames that referenced a high QP frame could spend bits adding detail instead of trying to erase erroneous detail of blocks.

HEVC added SAO, which does similar stuff for ringing artifacts. VVC is able to do much better with motion vector artifacts than prior codecs, allowing high QP prediction not look as artificial.

These sorts of tools all shift codecs towards having high QP result in just loss of detail instead of detail loss with introduction of wrong detail (ala MPEG-2, where 8x8 block patterns often became painfully visible). With a good encoder, that means it takes fewer bits to hit a certain level of high quality. For noisy content, it can also mean that it's not feasible to save all that many bits over prior codecs without losing some detail.

It also means that bitrates can be pushed much lower without introducing distracting artifacts. If a broadcaster is using fixed RF bandwidth, adding more, softer channels makes compelling economic sense, as you can get away with a lot more compression before customer start complaining. And even for IP streaming, bandwidth costs are a big part of the total cost of the business, so reducing them can help the bottom line a lot.

Still, it's not like we were getting artifact free high detail from those sectors before; it's pretty much premium content delivered over IP where the economics made sense for using enough bits to look consistently good. I'll take softer over soft-with-blocks any day. Although the psychovisual factors there can get complex; the added high frequencies of DCT artifacts in MPEG-2 offered a certain "sizzle" that some customers actually preferred over the uncompressed source.

Another reason we can see early-development encoders tend towards softness is that PSNR is intrinsically biased in that direction with per-frame QP. Encoders need psychovisually optimized adaptive quantization to lower QP in flatter areas to provide a more subjectively balanced encode. SSIM and VMAF are only a little better at properly accounting for the value of preserving detail more in low-detail areas.