Small codec comparison: VP7b5 vs RV10

[Sirber]

Avalible here:

http://www.detritus.qc.ca/forum/viewtopic.php?t=95

I also added xvid and x264.

[Sharktooth]

This time VP7 hit the target.
Now it's time to concentrate on the quality.

[Sagittaire]

IMO this comparison (VP7 vs RV10) is useless ... VP7 or NDAVC are very better than RV10 (Movie and Anime) for very high quant encoding ...
http://multimediacom.free.fr/Video/NDAVC-600.mp4

Make VP7 with better quant interval (like [16-56] or smaller interval) and perhabs lower variability but VP7 is very better than RV10 even for animate source ...

IMO the real fight is VP7 vs H264 ...

[dragongodz]

IMO this comparison (VP7 vs RV10) is useless
thats funny considering many have said the same about your metrics tests. maybe you should remember that before being insulting.

if you think there is some problem with the test then why not just point out what you think it is ?

[Sagittaire]

In fact I think that only for title with this particular comparison "RV10 vs VP7" is useless because I think that RV10 can't figth with VP7 or H264 ... "RV10 vs other" for example could be a better title for example ... lol

thats funny considering many have said the same about your metrics tests.

Not for codec developper like One2Tech or Bobololo but only for XviD fan (like you perhabs). And Metric confirm always that my eyes see (like you perhabs) -> H264 or VP7 are very better than MPEG4 ASP or RV10 ...

[dragongodz]

only for XviD fan (like you perhabs)
and now you want to try and insult me ? tell you what, why dont you find a post or 2 where i say xvid, or any other codec for that matter, is the best thing since sliced bread. cant ? so dont try and call me an xvid fanboy without something to back it up.

And Metric confirm always that my eyes see (like you perhabs)
i will always believe what i see before believing a metric. if you want an example of a metric not being the same as what people see then look in the rather huge Rejig thread. Makira made a change at one point to increase PSNR and it worked. guess what though, the result actually looked worse to people. Makira checked by actually comparing clips visually and indeed saw the higher PSNR results looked worse so changed it back. so no not like me perhaps.

Not for codec developper like One2Tech or Bobololo
oh really ? funny but this thread
http://forum.doom9.org/showthread.php?threadid=92532
seems to suggest On2Tech has some missgivings about the bugs and flaws in SSIM and therefore brings in to question the consistancy of its results.

H264 or VP7 are very better than MPEG4 ASP or RV10
well thats your opinion and many may agree but that doesnt give you any right to just dismiss Sirbers tests. as i said ,if you think there is a problem with his test then just say what it is and leave it at that. there is no need for you to call it useless.

[Sagittaire]

insult for siber ... where ... ???
-> say "it's bad title" is an insult ... lol

insult for you ... where ... ???
-> say "you are perhabs XviD fan" is an insult ... lol



I will always believe what i see before believing a metric. if you want an example of a metric not being the same as what people see then look in the rather huge Rejig thread. Makira made a change at one point to increase PSNR and it worked. guess what though, the result actually looked worse to people. Makira checked by actually comparing clips visually and indeed saw the higher PSNR results looked worse so changed it back. so no not like me perhaps.


With Overall PSNR ... interessing ... where ... ???



oh really ? funny but this thread
http://forum.doom9.org/showthread.php?threadid=92532
seems to suggest On2Tech has some missgivings about the bugs and flaws in SSIM and therefore brings in to question the consistancy of its results.


it's an particular test with only very dark frame ... an particular "bug" in very particular conditions ... lol

[Sirber]

The main goal of this comparaison was to test the new beta 5, and it was a success.

Quote from my private mail from On2Tech:
Right. We all have more work to do that is for sure.

Actually I have a good regard for the Real codec as it usually has quite low levels of artifact (even if it sometimes looses detail) so I was surprised.

Certainly I prefer Real to WM9 on most clips.On the logo scene I think B-frames probably help quite a bit (not supported in Vp7). We have various tricks in Vp7 to try and make up for this but I suspect they are not working quite right on that section. Something for us to work on and improve in future releases

[On2Tech]

I think that none of you will be surprised to hear me say that to my eyes VP7 does do better (overall) on this clip than RV10 and I suspect it would do better in a metric comparison. In particular I think that both the RV10 samples show significant motion artifacts and staleness in the background during the earlier sections. But that does not mean that VP7 is perfect either and the time we stop accepting constructive criticism, be it from Sirber, Sagittaire or anybody else is the time we stop improving our codecs.

I support Sirber’s (or anybody else’s) right to make their own subjective judgements based upon the sorts of artifact that they find more or less intrusive. He has put up the clips so people are free to make up their own minds.

Just to confuse the issue further I would say that as a means of judging codecs all metrics are somewhat flawed (see below) but so are subjective tests because they are so, how can I put this, "subjective". We all have different eyes and different preferences. There is even a considerable amount of evidence that subjective preferences are cultural, so a focus group in one country will give completely different results than one in another.

Regarding Objective measures:-

Average PSNR is a useful guide in some situations but tends to be dominated by the best quality frames and a few very good frames can make a big difference to the clip average.

Overall PSNR is inclined to go slightly too far the other way and is dominated by the worst few frames. This does not always make sense subjectively. For example a blocking artifact in a complex high motion scene may be nearly invisible, but in a low motion close up of a face it is very distracting. A codec tuned entirely based on overall psnr may tend to spend too many bits in the hardest sections.

SSIM. We have not played much with this before now but it also has strengths and weaknesses. Certainly we have shown that it is capable of throwing up silly results in some situations and I have some concerns over the weightings it gives based on brightness. These seem based on the assumption that darker parts of the images are always less important. This may be true when they are juxtaposed with very bright images sections that tend to draw the eye, but where the whole image is dark I think the opposite is true and quite small errors can be very obvious.

So what is the answer? Metrics are useful but there will always be legitimate differences regarding which ones are best. Subjective tests are also useful but no two people will ever agree :)

On2Tech

[dragongodz]

Subjective tests are also useful but no two people will ever agree
agree, which is why most of us say blind tests done with a group of people(the more the better) will show you more real world results about what people do and do not find good etc. :)

and a big apology to Sirber for taking this thread off track. i just did not think his work needed to be called useless no matter if people agree or disagree with his results. as On2Tech said, the clips are there for people to download and make up their own minds.

[IgorC]

Can anybody to do some videosamples of VP7b5 and H.264 using the film source? Anime is almost spotless for me.

[Sirber]

My ADSL broke again. :angry: