Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Announcements and Chat > General Discussion

Reply
 
Thread Tools Search this Thread Display Modes
Old 21st June 2010, 14:16   #21  |  Link
CruNcher
Registered User
 
CruNcher's Avatar
 
Join Date: Apr 2002
Location: Germany
Posts: 4,928
Emanem_
if you would really compare it on a psy level VP8 currently has only 1 real Psy like option and that's a very basic deadzone change done via --sharpness=x (which could have been usefull in this case)
also posting non correct frame comparisons with frames that are different between source and encodes doesn't help much (also not informing what frames we actually see their and how far they away from the keyframe isn't nice)

And yeah it's obvious you didn't hit bitrate with ffmpeg and posting this then as valid comparison is blatantly wrong you should know that

read this http://pastebin.org/346660 understand it live it
__________________
all my compares are riddles so please try to decipher them yourselves :)

It is about Time

Join the Revolution NOW before it is to Late !

http://forum.doom9.org/showthread.php?t=168004

Last edited by CruNcher; 21st June 2010 at 14:26.
CruNcher is offline   Reply With Quote
Old 21st June 2010, 14:56   #22  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,667
Quote:
Originally Posted by CruNcher View Post
Emanem_
if you would really compare it on a psy level VP8 currently has only 1 real Psy like option and that's a very basic deadzone change done via --sharpness=x (which could have been usefull in this case)
also posting non correct frame comparisons with frames that are different between source and encodes doesn't help much (also not informing what frames we actually see their and how far they away from the keyframe isn't nice)

And yeah it's obvious you didn't hit bitrate with ffmpeg and posting this then as valid comparison is blatantly wrong you should know that

read this http://pastebin.org/346660 understand it live it
Come on, don't post my drafts, post the link to the actual blog post
Dark Shikari is offline   Reply With Quote
Old 21st June 2010, 15:15   #23  |  Link
CruNcher
Registered User
 
CruNcher's Avatar
 
Join Date: Apr 2002
Location: Germany
Posts: 4,928
Yeah sorry credits go to Dark and all the other contributors in #x264
__________________
all my compares are riddles so please try to decipher them yourselves :)

It is about Time

Join the Revolution NOW before it is to Late !

http://forum.doom9.org/showthread.php?t=168004
CruNcher is offline   Reply With Quote
Old 21st June 2010, 15:59   #24  |  Link
Emanem_
Registered User
 
Join Date: Feb 2010
Posts: 15
Quote:
Originally Posted by Dark Shikari View Post
Come on, don't post my drafts, post the link to the actual blog post
Thanks for your ideas and reccomendations to properly test videos.
How would you objectively test codecs with psy on? Which metric would you reccommend?

OMG, I posted that table with 'real' bitrate and thought that encoders would have used the requested bitrate!

As soon as I have time I'll reupdate and republish it.

Regarding objective measures, do you have any metric to suggest other than PSNR and SSIM?
Cheers,
Emanem_ is offline   Reply With Quote
Old 21st June 2010, 16:16   #25  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,667
Quote:
Originally Posted by Emanem_ View Post
Thanks for your ideas and reccomendations to properly test videos.
How would you objectively test codecs with psy on? Which metric would you reccommend?
That's inherently not possible -- I'd use screenshots and my eyes.
Quote:
Originally Posted by Emanem_ View Post
Regarding objective measures, do you have any metric to suggest other than PSNR and SSIM?
Cheers,
Not that I know of. There's some fancy ones, but I don't know of any implementations. Or any proof that they're any good.

Here's what I'd do:

1. Pick a metric (SSIM or PSNR).

2. Tune all tested encoders, wherever possible, for these metrics (e.g. --tune psnr in x264). Use the slowest settings and so forth -- get the most out of each encoder.

3. Do an objective chart accordingly.

4. Tune the encoders for psy instead, and use that to generate screenshots.

The downsides of this method are that:

a) The graphs will bias against encoders that have good psy optimizations, like x264. This is because they don't take into account the fact that psy optimizations make the difference between encoders greater than one would measure via raw objective metrics.

b) The graphs will bias even more against encoders that can't turn off their psy optimizations (e.g. if you're optimizing for PSNR, Theora Ptalarbvorm is by-default-and-you-can't-turn-it-off optimized for SSIM). This is to the point where if you can't disable psy opts, objective testing is basically unfair and shouldn't be done.

Last edited by Dark Shikari; 21st June 2010 at 16:21.
Dark Shikari is offline   Reply With Quote
Old 21st June 2010, 20:32   #26  |  Link
Emanem_
Registered User
 
Join Date: Feb 2010
Posts: 15
Quote:
Originally Posted by Dark Shikari View Post
That's inherently not possible -- I'd use screenshots and my eyes.Not that I know of. There's some fancy ones, but I don't know of any implementations. Or any proof that they're any good.
Thanks again!

I do understand your points.
But how must be hard to develop a codec and test/verify that what you've done is an improvement if there is no such automated way to do it? Do you authors of x264 have at your disposition a group of experts to help you with psy measurements?

Anyway, about x264 and VP8, I've always used the slowest settings (think during encoding that VP8 in the Shark scene goes 1.5 FPS while x264 30+ on my Phenom II X4 965 B.E.).

Apart the stupid mistake (not taking a look at the table with file size and real bitrates), I've always tried to compare apples to apples (i.e. 4 MBps bitrate) and I tried to do a visual comparison as well.
And, with surprise, now that I look at that table again, VP8 bitrate for 4 MBps is lower than x264! So could it have used more quality? Perhaps those 40 KBps more in the first Avatar scene could decrease distance with x264 HP?

Indeed my comparison didn't pretend to be serious (it's in the title).

But at the end of the day I've found out that VP8 is very good quality wise.
Have you ever give it a shot, you know just out of curiosity, to take a peek at the competition? Have you ever encoded a HD video with VP8 and seen it? I'm just curious...

I mean, clearly in the eyes of an expert my comparison lacks of maybe ppm (or png) frames (btw should I be authorized to publish them if I don't own the images?) and the final encodings, plus maintaining a strict filesize between different encodings.
But you see, I've always tried to be more fair than possible.

At the end of the day I look to the better codec as the codec which given the same bitrate will provide more quality or if same quality least disadvantages (i.e. no software patents and/or speed).

Thanks again for your hints,
Regards,

Ps. I'm not against x264. I don't like the fact that H.264 is patented (and generally software patents do exist). Nevertheless I still think x264 is the best and stuff encoded with it works on my PS3 beautifully
I have never taken a look at x264 sources, but I bet the quality is as the software itself, really and truly exceptional. Too bad for patents.
Emanem_ is offline   Reply With Quote
Old 21st June 2010, 20:48   #27  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,667
Quote:
Originally Posted by Emanem_ View Post
Thanks again!

I do understand your points.
But how must be hard to develop a codec and test/verify that what you've done is an improvement if there is no such automated way to do it? Do you authors of x264 have at your disposition a group of experts to help you with psy measurements?
We have our eyes and intuitions But of course I don't claim to have "golden eyes" or any of that type of BS that audiophiles like to spout; I'm as limited as anyone else.

Quote:
Originally Posted by Emanem_ View Post
Anyway, about x264 and VP8, I've always used the slowest settings (think during encoding that VP8 in the Shark scene goes 1.5 FPS while x264 30+ on my Phenom II X4 965 B.E.).
30+ FPS isn't the slowest . Try placebo.

Quote:
Originally Posted by Emanem_ View Post
But at the end of the day I've found out that VP8 is very good quality wise.
Have you ever give it a shot, you know just out of curiosity, to take a peek at the competition? Have you ever encoded a HD video with VP8 and seen it? I'm just curious...
Yes, I tested it as part of my analysis of VP8. It came out rather lacking, losing by about 20-30% PSNR-wise and even more visually (due to lack of psy opts). It beat x264 Baseline profile by about 8% though, PSNR-wise. Of course, in a visual comparison, even x264 Baseline beat VP8, but that's not surprising.

It's even worse if you pick a non-super-slow mode though; VP8 does incredibly badly on its "fast" modes, which is what really matters for practical use (try encoding a 1080p movie with --best!).

Of course, speed issues are problems with the encoder, not the format. The format itself is a bit better compression-wise than H.264 Baseline, but greatly inferior to Main or High. Speed-wise, it's pretty similar to Main or High, maybe a bit slower.

Last edited by Dark Shikari; 21st June 2010 at 20:50.
Dark Shikari is offline   Reply With Quote
Old 21st June 2010, 21:05   #28  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 13,239
Quote:
Originally Posted by Dark Shikari
Quote:
Originally Posted by Emanem_
Regarding objective measures, do you have any metric to suggest other than PSNR and SSIM?
Cheers,
Not that I know of. There's some fancy ones, but I don't know of any implementations. Or any proof that they're any good.
Well, there is the MSU Quality Measurement Tool available, which in includes an implementation of VQM along with some others.

Be aware: I have no experience how useful VQM is or how it compares to SSIM. Just mentioned it for completeness...
__________________
Go to https://standforukraine.com/ to find legitimate Ukrainian Charities 🇺🇦✊

Last edited by LoRd_MuldeR; 21st June 2010 at 21:08.
LoRd_MuldeR is offline   Reply With Quote
Reply

Tags
comparison, psnr, ssim, vp8, x264

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 18:16.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.