Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > VP9 and AV1

Reply
 
Thread Tools Search this Thread Display Modes
Old 8th October 2019, 17:28   #1841  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,750
Quote:
Originally Posted by marcomsousa View Post
AV1 is ready for prime time: SVT-AV1 beats x265 and libvpx in quality, bitrate and speed
https://medium.com/@ewoutterhoeven/a...d-31c1960703db
The VMAF scores are too close to indicate any real subjective difference. And things get confusing given some encoders, and AV1 itself, are tuned against VMAF. The more a metric is explicitly targeted, the lower its correlation to subjective quality measurements becomes.

It does suggest quality @ bitrate are becoming ballpark similar, which itself is quite an accomplishment versus encoders that have been developed for far longer.

Sent from my SM-T837V using Tapatalk
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 8th October 2019, 20:57   #1842  |  Link
Tommy Carrot
Registered User
 
Tommy Carrot's Avatar
 
Join Date: Mar 2002
Posts: 863
I'd say for visual quality SVT-AV1 isn't quite there yet. There still are some motion artifacts that are not really harmful for the metrics, but visually are quite unpleasant. The slowest preset mostly gets rid of them, that's where SVT-AV1 really starting to get the upper hand over x265, but that's just way too slow.

But still, SVT-AV1 is improving quite steadily. Aomenc is still better for maximum quality per bitrate, but it's just too slow, and the encoding speed is improving at a snail's pace.
Tommy Carrot is offline   Reply With Quote
Old 9th October 2019, 01:24   #1843  |  Link
soresu
Registered User
 
Join Date: May 2005
Location: Swansea, Wales, UK
Posts: 196
I'm more interested in that ML augmented Aurora AV1 encoder, sounds like it's got some gains over libaom's quality, while putting on some considerable speed, though my memory of their BAV talk is a little hazy now.
soresu is offline   Reply With Quote
Old 9th October 2019, 02:24   #1844  |  Link
RanmaCanada
Registered User
 
Join Date: May 2009
Posts: 328
Quote:
Originally Posted by marcomsousa View Post
AV1 is ready for prime time: SVT-AV1 beats x265 and libvpx in quality, bitrate and speed
https://medium.com/@ewoutterhoeven/a...d-31c1960703db
If it really was ready for primetime, they would have posted their sources instead of graphs that can easily be inconsequential. We've all seen encodes that score perfectly, but look like utter garbage.

There are no pictures, no videos, nothing visual for people to look at. Which when you are dealing with encodes, is the most important thing.
RanmaCanada is offline   Reply With Quote
Old 9th October 2019, 02:46   #1845  |  Link
soresu
Registered User
 
Join Date: May 2005
Location: Swansea, Wales, UK
Posts: 196
Quote:
Originally Posted by RanmaCanada View Post
If it really was ready for primetime, they would have posted their sources instead of graphs that can easily be inconsequential. We've all seen encodes that score perfectly, but look like utter garbage.

There are no pictures, no videos, nothing visual for people to look at. Which when you are dealing with encodes, is the most important thing.
The graphs are AreWeCompressedYet, the linked Medium post lists its sources at the bottom, and the AWCY source lists its test videos.
soresu is offline   Reply With Quote
Old 9th October 2019, 03:06   #1846  |  Link
RanmaCanada
Registered User
 
Join Date: May 2009
Posts: 328
Quote:
Originally Posted by soresu View Post
The graphs are AreWeCompressedYet, the linked Medium post lists its sources at the bottom, and the AWCY source lists its test videos.
I don't think you understood what I meant. There are are no visuals on the page. No frame grabs, no side by side video comparisons, etc. When dealing with a visual medium like codecs, you need to actually see what the end result looks like. Graphs mean nothing as you can program a codec to cheat and give every single signal that a testing program wants. It's what we do here almost every single time someone fights over what codec is better, and what implementations work best.

As I said, we've all seen encodes that score incredibly high, if not perfect, but visually look like garbage.
RanmaCanada is offline   Reply With Quote
Old 9th October 2019, 07:42   #1847  |  Link
dapperdan
Registered User
 
Join Date: Aug 2009
Posts: 201
Quote:
Originally Posted by RanmaCanada View Post
As I said, we've all seen encodes that score incredibly high, if not perfect, but visually look like garbage.
Have we? Can you point to one such example on arewecompressedyet?

I generally believe the numbers, but for the same reason that people give for not believing the numbers.

Arewecompressedyet has been churning out stats for years now, using standard test clips, standard metric code, documented command lines, all available on GitHub and publicised all the results.

Basically it's the gold standard for objective metrics.

And despite regularly reading complaints here about how it doesn't reflect subjective reality, I've never once seen a link to a specific result, video or even screenshot to back these claims up.

Not that I would put particular weight on one example even if found. It would be like reading a study about a new cancer drug and saying "What about Bob? Bob died! We should use healing crystals instead."

I have seen subjective study results that generally back up the objective results within and between codecs (and this is how they're initially tested and built, so it shouldn't really be a surprise, but it shows they've not been too gamed since)

So something odd with human psychology is going on, but I don't think it's the metrics that are the problem.

edit: I should note the one stat were I do thibk arewecompressedyet is fallible.Id not take a single encoding time number seriously, I'd check there was a consistent pattern, and double check it for multithreaded behaviour if that was relevant.

Last edited by dapperdan; 9th October 2019 at 07:46.
dapperdan is offline   Reply With Quote
Old 9th October 2019, 12:12   #1848  |  Link
Tommy Carrot
Registered User
 
Tommy Carrot's Avatar
 
Join Date: Mar 2002
Posts: 863
Dapperdan, there is a simple example. X264 (and probably x265) gives much better results for nearly all metrics with --tune psnr or ssim, compared to the default settings, but visually it's almost always considerably worse.

Also, as i said, SVT-AV1 has pretty good metrics with -enc-mode 4, but visually it's nowhere near x265 for example.
Tommy Carrot is offline   Reply With Quote
Old 9th October 2019, 12:41   #1849  |  Link
nevcairiel
Registered Developer
 
Join Date: Mar 2010
Location: Hamburg/Germany
Posts: 10,344
Metrics have a simple problem that we're seeing here - and that is that none of them truely quantify human perception. If it would accurately reflect that, then there would be no problem, not even as newer metrics come closer, they are still not there.

If you create a new metric, its usually "good" at first, because the metric compares encoders, and encoders don't do anything to cheat.
But as a metric gets more popular, encoders will try to cheat and optimize for those metrics. That means they score high in those metrics - it does not mean the image got better. For a metric to remain relevant for a long time, you would have to make sure of two things, (a) that the metric is somewhat related to human vision, and (b) that encoders are forbidden to optimize for it. Unfortunately (b) is an impossible goal.

This is why purely "objective" metric-driven comparisons are always flawed, if they are not accompanied by subjective human tests as well, and both objective and subjective tests and results are properly aligned.
VMAF got worse as a reference as encoders like SVT and x265 started to optimize for that particular metric, instead of just optimizing for a better visual quality.
__________________
LAV Filters - open source ffmpeg based media splitter and decoders
nevcairiel is offline   Reply With Quote
Old 9th October 2019, 12:55   #1850  |  Link
dapperdan
Registered User
 
Join Date: Aug 2009
Posts: 201
Do you happen to have a link to subjective study on naive users (i.e. not people who know what classic artifacts look like) showing that is really the case?

This an extreme example and it may well be true, but even then how do we know for a fact that the person doing the tuning wasn't paying too much attention to one specific artifact and not subtly ruining other areas they weren't focussed on? If the answer is just "well before there was an artifact here and now it's gone then that's not really convincing to me, compared with multiple objective stats that generally correlate with user reports in lab settings.

If all the metrics across all the different test clips show the same improvement in that case, then I'd be suspicious that the metric is actually correct and the tuning is wrong.

I'll try and locate two such encodes on AreWeconpressedYet and compare them.

Found a couple of examples, x264 very slow on beta.areweconpressedyet. not a perfect match as they were done about 6 months apart but, if we ignore that:

As you might expect tune PSNR wins on PSNR Y and APNSR Y, though not on the Cb and Cr variants where they're neck and neck.

The gap similarly closes massively on PSNR HVS, with the standard tuning winning at higher bitrates. Generally most metrics don't really change other than PSNR for Y and SSIM variants.

I'd guess from looking at the results that tune PSNR just means switching off anything they did to tune for SSIM ( the Fast SSIM variant specifically) but that effected PSNR negatively. Those two metrics move in opposite directions, where other metrics (even other PSNR and SSIm variants) show no real difference. So it's somewhat ironic (assuming I'm right about that) to raise this as an obvious case of subjective being better than tuning to metrics.

I'll try to find a tune ssim example, which if I'm correct will do better than default on fast SSIm especially, while losing on PSNR and not really affecting most other metrics.

Last edited by dapperdan; 9th October 2019 at 13:28.
dapperdan is offline   Reply With Quote
Old 9th October 2019, 13:56   #1851  |  Link
dapperdan
Registered User
 
Join Date: Aug 2009
Posts: 201
I couldn't find any x264 tune ssim encodes, but I did find some x265 tune PSNR that show basically the same pattern. Tuning for PSNR at the expense of Fast SSIM, while other metrics don't really change.

The major difference is that the PSNR tuning seems to help with CB and Cr for x265, not just the Y channel.

Fascinating. So it appears x264 and x265 have their own version of VMAF which is just the combination of PSNR and FastSSIM metrics.
dapperdan is offline   Reply With Quote
Old 9th October 2019, 15:03   #1852  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,753
Quote:
Originally Posted by dapperdan View Post
So it appears x264 and x265 have their own version of VMAF which is just the combination of PSNR and FastSSIM metrics.
That sounds confusing to me. PSNR and SSIM are mainly frame-to-frame metrics, whereas VFAM should also consider motion, thus have a temporal window.

The main metric used in x264 and x265 is the "rate factor", if I'm not completely wrong...
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 9th October 2019, 15:36   #1853  |  Link
dapperdan
Registered User
 
Join Date: Aug 2009
Posts: 201
I just meant in the sense that VMAF is a fusion of other metrics, including PSNR.

VMAF use fancy statistics to combine them all based how they agreed with subjective measurements but it looks like x264 just combined Fast SSIM and PSNR when they didn't move in opposite directions.
dapperdan is offline   Reply With Quote
Old 9th October 2019, 16:27   #1854  |  Link
mzso
Registered User
 
Join Date: Oct 2009
Posts: 930
Will SVT-AV1 and rav1e be added to ffmpeg as encoder libraries?
Or is that not possible because of licensing?
mzso is offline   Reply With Quote
Old 9th October 2019, 18:55   #1855  |  Link
sneaker_ger
Registered User
 
Join Date: Dec 2002
Posts: 5,565
AFAIK there are no licensing issues. Both rav1e and SVT-AV1 have open licences (BSD 2-Clause and BSD+patent). Some patches already seem to exist but it seems aren't fully ready yet (i.e. developers still need to do some work or want to wait a bit more for the respective projects to mature).
sneaker_ger is offline   Reply With Quote
Old 9th October 2019, 19:12   #1856  |  Link
quietvoid
Registered User
 
Join Date: Jan 2019
Location: Canada
Posts: 570
rav1e is still missing a stable C API for ffmpeg, IIRC.
quietvoid is offline   Reply With Quote
Old 10th October 2019, 01:51   #1857  |  Link
TD-Linux
Registered User
 
Join Date: Aug 2015
Posts: 34
Quote:
Originally Posted by mzso View Post
Will SVT-AV1 and rav1e be added to ffmpeg as encoder libraries?
Or is that not possible because of licensing?
For rav1e we are waiting on releasing 0.1. We're down to one blocker bug, so it should be out soon: https://github.com/xiph/rav1e/issues/1636
TD-Linux is offline   Reply With Quote
Old 10th October 2019, 01:54   #1858  |  Link
TD-Linux
Registered User
 
Join Date: Aug 2015
Posts: 34
Quote:
Originally Posted by dapperdan View Post
Fascinating. So it appears x264 and x265 have their own version of VMAF which is just the combination of PSNR and FastSSIM metrics.
Unfortunately it's impossible to use VMAF directly for RDO because an encoder has to evaluate distortion on very small blocks, down to 8x8 pixels. So the best you can do is come up with something that approximates it.
TD-Linux is offline   Reply With Quote
Old 10th October 2019, 07:43   #1859  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 6,753
Just that you mention rav1e ... this week they broke compilation for the x86 target, possibly in an attempt to add assembler optimizations. The developers seem to lack of a cross-compilation environment for x86 (where I am not sure if that specifically means 32 bit code) for automated testing. I discovered that issue during my usual sporadic runs of the media-autobuild suite.
__________________

New German Gleitz board
MediaFire: x264 | x265 | VPx | AOM | Xvid
LigH is offline   Reply With Quote
Old 14th October 2019, 07:42   #1860  |  Link
Kirakishou
Registered User
 
Join Date: Sep 2019
Posts: 1
Hi guys. I’m not good at English, sorry for that. Got file [Xrip][Nekopara][OVA_Extra][GB][1080P][AV1_10bit].mp4 from nyaa. Last and several dozen previous build of mpv playback it choppy. x86 and x64 version of mpv.
But mpc-hc 1.8.4.x86 playback it smooth. Only x86 build of mpc-hc 1.8.4, x64 also playback it choppy. And next builds of mpc-hc also playback it choppy. As far as I understand there were some changes in LAV Filters which led to the worst result. Maybe this information will be useful to someone.
Kirakishou is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 19:24.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.