Alliance for Open Media codecs

[marcomsousa]

I think there are no HW encoders or decoders for AV1 yet.
Realtek have one HW decoder for av1 - https://www.realtek.com/en/press-room/news-releases/item/realtek-launches-worldwide-first-4k-uhd-set-top-box-soc-rtd1311-integrating-av1-video-decoder-and-multiple-cas-functions

Android 10 is released with the support for Opus audio support and AV1 video codec support- https://www.phoronix.com/scan.php?page=news_item&px=Android-10-Released

Av1 August news - https://www.singhkays.com/blog/av1-ecosystem-update-august-2019/

[NikosD]

Realtek have one HW decoder for av1 - https://www.realtek.com/en/press-room/news-releases/item/realtek-launches-worldwide-first-4k-uhd-set-top-box-soc-rtd1311-integrating-av1-video-decoder-and-multiple-cas-functions Right!
I had something in mind that I had recently read, so I found it also here, in this thread: First SoC launched by Realtek: https://www.realtek.com/en/press-room/news-releases/item/realtek-launches-worldwide-first-4k-uhd-set-top-box-soc-rtd1311-integrating-av1-video-decoder-and-multiple-cas-functions Moreover, I took the chance and found out this recorded video of the chipset in action, decoding in real-time a 4K60fps AV1 clip from YouTube without dropping a frame: https://www.youtube.com/watch?v=IGtjBwMwTtE
Lastly, I even found an announcement of Realtek regarding an integrated circuit RTD2893, capable of 8K AV1 decoding: https://www.realtek.com/en/press-room/news-releases/item/realtek-wins-three-best-choice-awards-at-computex-taipei-2019-including-2-best-choice-golden-awards-copy.

Realtek is first and fast!

[Blue_MiSfit]

...recorded video of the chipset in action, decoding in real-time a 4K60fps AV1 clip from YouTube without dropping a frame: https://www.youtube.com/watch?v=IGtjBwMwTtE


That's awesome!

[soresu]

Working AV1 decoder chips are good, I'm guessing they might make it into 2020 TV's.

Of course there are also the forthcoming Rockchip SoC's coming next year - I'm pretty sure one of them (RK3530 I think) was aimed at HDTV's too, with the other (RK3588) likely to make it into cheap Android TV boxes with a bit more oomph than previously seen in that segment, given most non smartphone ARM based kit tops out at low clocked A72/A73 CPU cores.

[soresu]

On a side note, the experimental branch of AOM has a patch for 'NN based entropy coding' which presumably is a bit more involved than mere rate decision or motion estimation optimisations.

I wonder how it compares to the AV1 entropy coding for quality/complexity, isn't that a derivative of the Daala technique?

[marcomsousa]

Paint.NET adding AV1 (*.avif) decoding in v4.2.2

[vidschlub]

It's kinda strange we have projects like x264/x265 for patent encumbered H.264/H.265 codecs, yet nothing like that for VP9/AV1.

I am unsure if I previously responded to this but this is more info I wasn't aware of.

Isn't there 3x AV1 encoders?
Are all 3 of them closed source or only designed for mass scale installs on cloud hardware or something?

Why are the 3 AV1 encoders inferior (in some ways?) to the x264 and x265 projects?

[LigH]

How much time has already been spent to develop x264 and x265? And since when are "final" AV1 specifications available? Compare and ask again. :sly:

[hajj_3]

Possible updates about AV1 adoption over the next week: https://aomedia.org/aomedia-members-demo-av1-at-ibc2019-demos-spotlight-av1s-royalty-free-ultra-high-definition-uhd-web-video-capabilities/

[soresu]

I am unsure if I previously responded to this but this is more info I wasn't aware of.

Isn't there 3x AV1 encoders?
Are all 3 of them closed source or only designed for mass scale installs on cloud hardware or something?

Why are the 3 AV1 encoders inferior (in some ways?) to the x264 and x265 projects?

There are 3 open source AV1 encoders that I know of - LIBAOM (AOM reference), SVT-AV1 (Intel) and RAV1E (Xiph/Mozilla), in various stages of speed and quality optimisation.

There are other closed source, proprietary encoders like Aurora (ML focused optimisations galore), EVE-AV1 (Two Orioles/Ronald Bultje), and likely the usual customers like Ateme, Cisco and such have their own encoders in various stages at the moment.

The recent Big Apple Video conference gave a run down on most of them save RAV1E, likely due to Vimeo's conflict of interest there as a conference sponsor and a RAV1E contributor/sponsor.

The main point to take away is that x265 has 6-7 years of development, and x264 has closer to 15 years of development. They are both fairly mature, if not necessarilly the fastest options today (SVT-HEVC may change the game on x265 on speed).

Meanwhile AV1 was only standardised last year, it takes time to get these new codecs ship shape for production purposes, let alone significant maturity.

[marcomsousa]

Adapt to multi-codec world or die, warns Bitmovin as AV1 escalates

Observing the video ecosystem from the video developer perspective is often overlooked, overshadowed somewhat by those higher up, so it was refreshing to read a report from encoding expert Bitmovin – giving the devs a well-deserved voice. Results from Bitmovin’s third annual developer survey showed an expectantly overwhelming reliance on H.264, although interestingly one-in-five developers plan to implement AV1 in 2020 – with big ramifications for the wider video industry. Device manufacturers, browser vendors, and content distributers like Cisco, Mozilla, and YouTube have already started implementing AV1 on larger scales, leading Bitmovin to conclude that AV1 is well positioned to compete with H.265/HEVC and to succeed VP9 for open-source use cases in 2020. This goes against the majority of conversations…



https://rethinkresearch.biz/articles/adapt-to-multi-codec-world-or-die-warns-bitmovin-as-av1-escalates/

[LigH]

It's all "conservative politics", even in technology :sly:

[soresu]

What's conservative about adapting, or am I missing something in the full article?

[stax76]

If x265 and nvenc are the last encoders relevant for home users then I will miss building encoder GUIs, the first generations were painful but the last two generations were quite fun. :)

[benwaggoner]

If x265 and nvenc are the last encoders relevant for home users then I will miss building encoder GUIs, the first generations were painful but the last two generations were quite fun. :)
There isn't anything codec-specific about nvenc. NVidia supports HW decode for multiple codecs, and can add more.

Something like 85% of new phones have HEVC HW decode, and we don't even have a release date for the first phone SoC that does hardware AV1. UHD Blu-ray and ATSC 3.0 are HEVC. There is going to be a substantial market for HEVC for a decade or more, just like there is still quite a lot of MPEG-2 still being encoded and delivered, and just like there will be a lot of H.264 for years too.

Heck, there was still Windows Media PlaysForSure content being published as of 2-3 years ago.

[LigH]

The usual threshold of hardware encoders is the limited temporal complexity. This hurts more for more advanced codecs taking more advantage of temporal redundancies. NVEnc may be a useful realtime encoder for AVC and HEVC. But it will be beaten easily if you can encode "offline" and spend more efforts into looking ahead.

[marcomsousa]

There is lot of interest right now for HEVC 8k real time for the next sport events..

[Nintendo Maniac 64]

Phoronix has more benchmark numbers of SVT-AV1 v0.5 and dav1d v0.3, this time with the 48-core EPYC 7642:

https://www.phoronix.com/scan.php?page=article&item=amd-epyc-7642&num=3

[excellentswordfight]

There is lot of interest right now for HEVC 8k real time for the next sport events..
Which is crazy to me, I work with alot of UHD sport events, and we dont use enough bandwith already to do it justice, both on contribution and distribution side. The move to 8k is just beyond me, it must be tons of money from the monitor manufacturers cause I cant see any broadcasters or productions to see benefits in it. The price/performance ratio for UHD is already awful. I dont get the res craze, if you dont wanna spend the bits, dont increase it. Heck XDCAM50 (mpeg2 1080i) can still look better then what google is doing with 4k.

I would rather see a push for 1080p50/60, full 10bit pipline and rec2020 with a modern codec and a decent bitrate to replace the norm of 1080i/720p h264, cause it still looks amazing and takes way less effort and money to upgrade to. Cause most viewers wouldnt wanna finance an multi million dollar upgrade for some tech dreams, but I guess that we can still make them.

This is from an live/broadcast perspective mind you, VOD is a different scenario.

[benwaggoner]

I would rather see a push for 1080p50/60, full 10bit pipline and rec2020 with a modern codec and a decent bitrate to replace the norm of 1080i/720p h264, cause it still looks amazing and takes way less effort and money to upgrade to. Cause most viewers wouldnt wanna finance an multi million dollar upgrade for some tech dreams, but I guess that we can still make them.

This is from an live/broadcast perspective mind you, VOD is a different scenario.
And that is certainly what North American sports broadcasters are focused on for the next big thing: 1080p60 10-bit HEVC HDR.

Interest in broadcast 8K is mainly in countries like Japan. South Korea, and China where there is a lot more available RF to use and where TV production is material to the national economy.

It is an interesting question for how many bits with what codec where a higher resolution pays off. At 6 Mbps CBR, 1080p60 is obviously better than 4K. But going from 1080 H.264 to 1080 HEVC worked at similar bitrates. The next gen VVC codec looks like it might offer the efficiency to do 8K VVC at the same bitrates as 4K HEVC. But we're some years out from having practical real-time VVC 8K encoders. Even 8K HEVC is still emerging tech, although certainly being done in trails and such.

[Nintendo Maniac 64]

Even 8K HEVC is still emerging tech, although certainly being done in trails and such.

A 64core Zen2 Epyc processor can already do realtime 8k HEVC 10bit encoding, and at 79fps to boot:

https://www.techspot.com/news/81905-amd-epyc-rome-cpu-performs-real-time-8k.html


So if you only need 30fps or 24fps, or perhaps 50fps or 25fps for 50Hz territories, then you could get away with a considerably lower CPU core count.

Heck even for 60fps you could probably get away with a 48 core Epyc since, if the multi-threaded encoding scaling was 100%, you'd be seeing 59.25fps on a 48core Epyc. However, since multi-threaded encode scaling almost never perfectly scales with core-count, and since CPUs with fewer cores tend to also have higher base clocks, it'd be quite likely that you could even see slightly above the required 60fps from a "mere" 48core Zen2 Epyc processor.

[excellentswordfight]

And that is certainly what North American sports broadcasters are focused on for the next big thing: 1080p60 10-bit HEVC HDR.
So i noticed, "3G" seems to be much more a thing there than here in europe. Even if a broadcaster would like to switch to 1080p/3G most productions over here dont offer that contribution anyway. I visited a brand new station/studio in the states not that long ago, everything built on 1080p60, nothing for UHD.

I'm still not sold on HDR for live content tbh, I have played around with both PQ and HLG and both comes with backwards comparability to SDR issues and complexity, and to be frank the live productions I've seen had issues even i HDR. I'm all for "HDR" as a tech, but when it comes to real world live productions it creates a lot of headache, maybe to much for it to ever become mainstream. Something that just going 10bit (which productions in most cases already are) and just increasing the colorspace doesnt (rec2020 specc), and imo is good enough, especially on a high contrast tv-set like an oled.

Interest in broadcast 8K is mainly in countries like Japan. South Korea, and China where there is a lot more available RF to use and where TV production is material to the national economy.
Yeah there is no suprice that it's the same countries that wanna push the consumer hw...

A 64core Zen2 Epyc processor can already do realtime 8k HEVC 10bit encoding, and at 79fps to boot:

https://www.techspot.com/news/81905-amd-epyc-rome-cpu-performs-real-time-8k.html


So if you only need 30fps or 24fps, or perhaps 50fps or 25fps for 50Hz territories, then you could get away with a considerably lower CPU core count.

Heck even for 60fps you could probably get away with a 48 core Epyc since, if the multi-threaded encoding scaling was 100%, you'd be seeing 59.25fps on a 48core Epyc. However, since multi-threaded encode scaling almost never perfectly scales with core-count, and since CPUs with fewer cores tend to also have higher base clocks, it'd be quite likely that you could even see slightly above the required 60fps from a "mere" 48core Zen2 Epyc processor.
Well, although it is impressive, without knowing the actual image quality and bitrate of the produced encode, it doesnt mean that much to me.

[Nintendo Maniac 64]

without knowing the actual image quality and bitrate of the produced encode, it doesnt mean that much to me.

If broadcasters were that concerned with quality and bitrate, then they wouldn't be pushing to broadcast 8k in the first place. :p

[marcomsousa]

Broadcom BCM7218X STB SoC Comes With AV1 Hardware Decoding
https://www.cnx-software.com/2019/09/25/broadcom-bcm7218x-stb-soc-av1-hardware-decoding-wifi-6/

[marcomsousa]

AV1 codec encoder/decoder implementation SVT-AV1 releases version 0.7.0, with more SIMD optimizations for performance, improved multithreading support, and more video features.

[ChaosKing]

AV1 codec encoder/decoder implementation SVT-AV1 releases version 0.7.0, with more SIMD optimizations for performance, improved multithreading support, and more video features.
I get 11fps with a 1080p BD source. CPU ryzen 2600. Not bad.

[soresu]

New Phoronix tests with SVT-AV1 (0.6 and 0.7)

Link here (https://www.phoronix.com/scan.php?page=news_item&px=EPYC-7742-Xeon-8280-Video-Enc).

[Nintendo Maniac 64]

New Phoronix tests with SVT-AV1 (0.6 and 0.7)

Link here: https://www.phoronix.com/scan.php?page=news_item&px=EPYC-7742-Xeon-8280-Video-Enc

Don't forget they included dav1d v0.4.0 results as well, and they're not only measuring performance by FPS now but are even including the max and min FPS to boot.

And they're even testing 10bit AV1 decoding in dav1d! This makes me very happy since, from previous testing, I actually found dav1d's 10bit decode performance on typical consumer CPU thread counts (2 to 8 threads) to be slower than the reference AV1 decoder, so I'll be keeping an eye out for any future performance gains for 10bit AV1 decoding.


(for reference, their most recent review (https://www.phoronix.com/scan.php?page=article&item=amd-epyc-7642&num=3) had used dav1d v0.3.0 and measured performance by seconds with 8bit only)

[soresu]

And they're even testing 10bit AV1 decoding in dav1d! This makes me very happy since, from previous testing, I actually found dav1d's 10bit decode performance on typical consumer CPU thread counts (2 to 8 threads) to be slower than the reference AV1 decoder, so I'll be keeping an eye out for any future performance gains for 10bit AV1 decoding.


There will be a lot to gain, the various SIMD issue/bugs on gitlab show basically nothing accelerated for 10 bit as yet, as they have been concentrating on 8 bit for all ISA currently:

AVX2 (https://code.videolan.org/videolan/dav1d/issues/78).
SSSE3 (https://code.videolan.org/videolan/dav1d/issues/216).
NEON (https://code.videolan.org/videolan/dav1d/issues/215).

Nevcariel mentioned something about 10 bit content not being available at the moment (broadcast, not test content ala Chimera), so I doubt it's a priority while there are still missing gaps in the 8 bit SIMD code, which there still is on NEON at the very least.

[Nintendo Maniac 64]

Nevcariel mentioned something about 10 bit content not being available at the moment

Well, non yar-har-fiddle-dee-dee content anyway. :p

[benwaggoner]

A 64core Zen2 Epyc processor can already do realtime 8k HEVC 10bit encoding, and at 79fps to boot:

https://www.techspot.com/news/81905-amd-epyc-rome-cpu-performs-real-time-8k.html

So if you only need 30fps or 24fps, or perhaps 50fps or 25fps for 50Hz territories, then you could get away with a considerably lower CPU core count.
Making the raw bitstream is becoming feasible, yes. But having it work at industrial scale with broadcast reliability, with good 8K monitoring solutions, end to end distribution and all that? Still at the experimental stage.

An open question would be at what bitrate an 8K signal would look better than a 4K signal. Being able to spend 4x the MIPS per pixel in 4K can be material, and it takes pretty low QPs to preserve detail in an 8K frame that wouldn't be in a 4K downconvert.

Practical 8K could well wait for the VVC codec. I've not seen much AV1 in 8K, but only 20% better than HEVC isn't likely to be sufficient. "8K in 4K bandwidth" is a good story, just like HEVC gave "4K in 1080p bandwidth" versus H.264.

[benwaggoner]

I'm still not sold on HDR for live content tbh, I have played around with both PQ and HLG and both comes with backwards comparability to SDR issues and complexity, and to be frank the live productions I've seen had issues even i HDR. I'm all for "HDR" as a tech, but when it comes to real world live productions it creates a lot of headache, maybe to much for it to ever become mainstream. Something that just going 10bit (which productions in most cases already are) and just increasing the colorspace doesnt (rec2020 specc), and imo is good enough, especially on a high contrast tv-set like an oled.
HLG has an impossible goal: provide a single signal that does HDR on HDR TVs and SDR on SDR TVs. HLG winds up being tuned to optimize one or the other, or just be mediocre for both. Good live HDR today is produced and distributed in PQ, and SDR versions are made in parallel or derived from the PQ. It's a big challenge since it's a rare production where ALL feeds can be PQ, so SDR needs to get upconverted for some stuff still.

But when you've got a good HDR PQ source, HDR PQ output can look great; much better than SDR at the same bitrate.

[soresu]

"8K in 4K bandwidth" is a good story, just like HEVC gave "4K in 1080p bandwidth" versus H.264.

Isn't that 8K at 2x 4K bandwidth considering the 50% improvement target of VVC?

Likewise HEVC is at best hitting 70-75% reduction vs AVC for 4K, and even then x265 does not seem to be quite living up to that currently given the recent MSU test results (https://www.compression.ru/video/codec_comparison/hevc_2018/#4k_report), though maybe I read them wrong?

[Nintendo Maniac 64]

Even more SVT-AV1 v0.7 benchmarks from Phononix, this time all on the i9-7980XE but testing performance between different OSes and distros (including Windows 10):

https://www.phoronix.com/scan.php?page=article&item=windows-linux-creators&num=4

[benwaggoner]

Isn't that 8K at 2x 4K bandwidth considering the 50% improvement target of VVC?

Likewise HEVC is at best hitting 70-75% reduction vs AVC for 4K, and even then x265 does not seem to be quite living up to that currently given the recent MSU test results (https://www.compression.ru/video/codec_comparison/hevc_2018/#4k_report), though maybe I read them wrong?
In general you need fewer bits per pixel as the number of pixels goes up, as any given artifact is a lot smaller. This is probably extra true for 8K; a completely messed up 4x4 block at 8K takes up as much visual field as 1 bad pixel at 1080p. The classic rule of thumb is that bitrate should go up proportionately to the 3/4ths power of the change in frame size. Thus:

new bitrate=old bitrate * (width/oldwidth * height/oldheight)^0.75.

But with modern codecs that scale better to high resolutions, the factor is going to be lower/ maybe 2/3rds? That works out to about 2x more bits, which gets covered by the 2x efficiency improvement.

Also, newer codecs have less objectionable distortions, so PSNR underestimates subjective improvements due to error-suppression features like in-loop deblocking. AV1 has a ton of those, which is presumably why we are seeing its greatest strengths at lower bitrates.

This is all ballpark. But H.264 gave 720p at roughly 480p MPEG-2 bandwidth and HEVC gave 4K at roughly 1080p H.264 bandwidth.

[marcomsousa]

libgav1 new AV1 decoder from Google. Focus on android OS.

https://chromium.googlesource.com/codecs/libgav1/

[Nintendo Maniac 64]

libgav1 new AV1 decoder from Google. Focus on android OS.

https://chromium.googlesource.com/codecs/libgav1/


And Phoronix has already managed to benchmark it against dav1d v0.4.0 on a bunch of different AMD and Intel CPUs:

https://www.phoronix.com/scan.php?page=news_item&px=dav1d-libgav1-AV1-performance

[soresu]

Yeah seems libgav1 has a long way to go for x86 at least.

ARM NEON is almost fully accelerated for 8 bit video in dav1d, so until Phoronix does some tests with ARM cores I'd assume a similar result there too.

On a different note, I've been keeping an eye on the experimental AOM/AV2 code branch for a while now.

Seems like a fair amount of work has gone in to it already - though judging any current cumulative improvement is difficult without an obvious AWCY link that compares the AOM master to the experimental code.

Can anyone point me to something here?

[Tommy Carrot]

On a different note, I've been keeping an eye on the experimental AOM/AV2 code branch for a while now.

Seems like a fair amount of work has gone in to it already - though judging any current cumulative improvement is difficult without an obvious AWCY link that compares the AOM master to the experimental code.

Can anyone point me to something here?

According to this (https://aomedia.googlesource.com/aom/+/613bf43dfe7153973a4cd07b5f053ec6da1c6ff6), the gains are currently at around 0.55%. Not much currently, but i guess they are only starting to add new tools.

[marcomsousa]

AV1 is ready for prime time: SVT-AV1 beats x265 and libvpx in quality, bitrate and speed
https://medium.com/@ewoutterhoeven/av1-is-ready-for-prime-time-svt-av1-beats-x265-and-libvpx-in-quality-bitrate-and-speed-31c1960703db

[benwaggoner]

AV1 is ready for prime time: SVT-AV1 beats x265 and libvpx in quality, bitrate and speed
https://medium.com/@ewoutterhoeven/av1-is-ready-for-prime-time-svt-av1-beats-x265-and-libvpx-in-quality-bitrate-and-speed-31c1960703dbThe VMAF scores are too close to indicate any real subjective difference. And things get confusing given some encoders, and AV1 itself, are tuned against VMAF. The more a metric is explicitly targeted, the lower its correlation to subjective quality measurements becomes.

It does suggest quality @ bitrate are becoming ballpark similar, which itself is quite an accomplishment versus encoders that have been developed for far longer.

Sent from my SM-T837V using Tapatalk

[Tommy Carrot]

I'd say for visual quality SVT-AV1 isn't quite there yet. There still are some motion artifacts that are not really harmful for the metrics, but visually are quite unpleasant. The slowest preset mostly gets rid of them, that's where SVT-AV1 really starting to get the upper hand over x265, but that's just way too slow.

But still, SVT-AV1 is improving quite steadily. Aomenc is still better for maximum quality per bitrate, but it's just too slow, and the encoding speed is improving at a snail's pace.

[soresu]

I'm more interested in that ML augmented Aurora AV1 encoder, sounds like it's got some gains over libaom's quality, while putting on some considerable speed, though my memory of their BAV talk is a little hazy now.

[RanmaCanada]

AV1 is ready for prime time: SVT-AV1 beats x265 and libvpx in quality, bitrate and speed
https://medium.com/@ewoutterhoeven/av1-is-ready-for-prime-time-svt-av1-beats-x265-and-libvpx-in-quality-bitrate-and-speed-31c1960703db
If it really was ready for primetime, they would have posted their sources instead of graphs that can easily be inconsequential. We've all seen encodes that score perfectly, but look like utter garbage.

There are no pictures, no videos, nothing visual for people to look at. Which when you are dealing with encodes, is the most important thing.

[soresu]

If it really was ready for primetime, they would have posted their sources instead of graphs that can easily be inconsequential. We've all seen encodes that score perfectly, but look like utter garbage.

There are no pictures, no videos, nothing visual for people to look at. Which when you are dealing with encodes, is the most important thing.

The graphs are AreWeCompressedYet, the linked Medium post lists its sources at the bottom, and the AWCY source lists its test videos.

[RanmaCanada]

The graphs are AreWeCompressedYet, the linked Medium post lists its sources at the bottom, and the AWCY source lists its test videos.

I don't think you understood what I meant. There are are no visuals on the page. No frame grabs, no side by side video comparisons, etc. When dealing with a visual medium like codecs, you need to actually see what the end result looks like. Graphs mean nothing as you can program a codec to cheat and give every single signal that a testing program wants. It's what we do here almost every single time someone fights over what codec is better, and what implementations work best.

As I said, we've all seen encodes that score incredibly high, if not perfect, but visually look like garbage.

[dapperdan]

As I said, we've all seen encodes that score incredibly high, if not perfect, but visually look like garbage.

Have we? Can you point to one such example on arewecompressedyet?

I generally believe the numbers, but for the same reason that people give for not believing the numbers.

Arewecompressedyet has been churning out stats for years now, using standard test clips, standard metric code, documented command lines, all available on GitHub and publicised all the results.

Basically it's the gold standard for objective metrics.

And despite regularly reading complaints here about how it doesn't reflect subjective reality, I've never once seen a link to a specific result, video or even screenshot to back these claims up.

Not that I would put particular weight on one example even if found. It would be like reading a study about a new cancer drug and saying "What about Bob? Bob died! We should use healing crystals instead."

I have seen subjective study results that generally back up the objective results within and between codecs (and this is how they're initially tested and built, so it shouldn't really be a surprise, but it shows they've not been too gamed since)

So something odd with human psychology is going on, but I don't think it's the metrics that are the problem.

edit: I should note the one stat were I do thibk arewecompressedyet is fallible.Id not take a single encoding time number seriously, I'd check there was a consistent pattern, and double check it for multithreaded behaviour if that was relevant.

[Tommy Carrot]

Dapperdan, there is a simple example. X264 (and probably x265) gives much better results for nearly all metrics with --tune psnr or ssim, compared to the default settings, but visually it's almost always considerably worse.

Also, as i said, SVT-AV1 has pretty good metrics with -enc-mode 4, but visually it's nowhere near x265 for example.

[nevcairiel]

Metrics have a simple problem that we're seeing here - and that is that none of them truely quantify human perception. If it would accurately reflect that, then there would be no problem, not even as newer metrics come closer, they are still not there.

If you create a new metric, its usually "good" at first, because the metric compares encoders, and encoders don't do anything to cheat.
But as a metric gets more popular, encoders will try to cheat and optimize for those metrics. That means they score high in those metrics - it does not mean the image got better. For a metric to remain relevant for a long time, you would have to make sure of two things, (a) that the metric is somewhat related to human vision, and (b) that encoders are forbidden to optimize for it. Unfortunately (b) is an impossible goal.

This is why purely "objective" metric-driven comparisons are always flawed, if they are not accompanied by subjective human tests as well, and both objective and subjective tests and results are properly aligned.
VMAF got worse as a reference as encoders like SVT and x265 started to optimize for that particular metric, instead of just optimizing for a better visual quality.

[dapperdan]

Do you happen to have a link to subjective study on naive users (i.e. not people who know what classic artifacts look like) showing that is really the case?

This an extreme example and it may well be true, but even then how do we know for a fact that the person doing the tuning wasn't paying too much attention to one specific artifact and not subtly ruining other areas they weren't focussed on? If the answer is just "well before there was an artifact here and now it's gone then that's not really convincing to me, compared with multiple objective stats that generally correlate with user reports in lab settings.

If all the metrics across all the different test clips show the same improvement in that case, then I'd be suspicious that the metric is actually correct and the tuning is wrong.

I'll try and locate two such encodes on AreWeconpressedYet and compare them.

Found a couple of examples, x264 very slow on beta.areweconpressedyet. not a perfect match as they were done about 6 months apart but, if we ignore that:

As you might expect tune PSNR wins on PSNR Y and APNSR Y, though not on the Cb and Cr variants where they're neck and neck.

The gap similarly closes massively on PSNR HVS, with the standard tuning winning at higher bitrates. Generally most metrics don't really change other than PSNR for Y and SSIM variants.

I'd guess from looking at the results that tune PSNR just means switching off anything they did to tune for SSIM ( the Fast SSIM variant specifically) but that effected PSNR negatively. Those two metrics move in opposite directions, where other metrics (even other PSNR and SSIm variants) show no real difference. So it's somewhat ironic (assuming I'm right about that) to raise this as an obvious case of subjective being better than tuning to metrics.

I'll try to find a tune ssim example, which if I'm correct will do better than default on fast SSIm especially, while losing on PSNR and not really affecting most other metrics.