2004 codec shoot-out

[bobololo]

Hehe I'm sorry, this won't probably help to get back to the original topic but I couldn't resist to quote these statments that GomGom expressed on #xvid while I was reading this thread :)


<GomGom> yo
-:- SignOff war-: #xvid (Remote closed the connection)
-:- war- [war@209.81.41.149] has joined #xvid
<GomGom> raaaa koepi is not there
<GomGom> he's put online a build with a shitty RC... fuck
<GomGom> i told him not to do so
<GomGom> but it always give inferior PSNR
<GomGom> and lot of time much worse worst PSNR


Obviously this shouldn't be taken seriously, it was just for relaxing the atmosphere :)

[Koepi]

In sharktooth tests, the PSNR and SSIM of strict scaling always was higher than with loose scaling... or did i misread something? Anyhow, as we mention that here - those values don't mean too much. I'll ask GomGom to do some tests with other samples, with this build everything seems to be fine.

Enough for off-topic now, let's discuss that over at the xvid thread.

Regards
Koepi

[Ghim]

Originally posted by Gabriel_Bouvigne
On audio encoders we are using tools like PEAQ, which are trying to take into consideration human audition, instead of pure raw differences like psnr. But given today's state of video encoders regarding psychovisuals, I thing that using psnr, even if not optimal, is still a good indication.
You just have to know how it works, and specially its limitations.


There are already subjective models like PEAQ (I personnaly think PEAQ is quite average for a model but that's not the point here) but for video. I mentionned it earlier in my previous posts (the J.144 models are a good example) and the concerned models really give good results... but nobody here seems to take interest in this kind of tool. I don't know if it's possible to get those models easily, but at least it shows it's possible...

I'm really not convinced by the SSIM metric and I've never found any information proving its performances... If anyone has a link with some good information on SSIM results, I'm interested...

[Sagittaire]

Obviously this shouldn't be taken seriously, it was just for relaxing the atmosphere

To make peace with Koepi:

I prefer a little worse Average PSNR (-0.1 dB or -0.2dB) if Min PSNR is very higher (+ 1dB). Average PSNR isn't really good tools for testing Rate Control quality ... ;)

[mgh]

From my experiments xvid is good only at high bit rates. I tried a fairly static clip at 384x288 showing a water pitcher at 300 kbps-
results-
xvid heavy blockiness
divx-jagged edges for the pitcher
while both vp6 and RV10 represented the pitcher perfectly.
My conclusion-with due respect to all gurus here-best codec depends on clip chosen-resolution-and kbps chosen for testing.
In short-tests for best codecs are meaningless-and good only for adding fuel to the fire on forums!!:D
Anyway, that's my humble opinion.

[Doom9]

@Soulhunter, babayaga: good idea about outside help except for a few killer drawbacks: first of all, I often get my hands on codecs that are not to be publicly released.. I can't just share those, and there are very few people I trust with software I'm not supposed to give out.

Then, the trust issue comes up again when it comes to sticking to settings and knowing the tests were done the way I want it done.. since it's published under my name I need to be sure and triple check that everything was done properly.. and I can't be sure of that unless a very good friend or myself did it.

Last but not least, the people encoding would have to offer GBs of encoded video for me to download.. while many people these days have pretty decent downlinks, uplink is another story entirely.

[SeeMoreDigital]

Doom9,

How proactive are some of these codec manufactures at dealing with requests?

Okay, I can't imagine the likes of Micro$oft would be willing to generate a full movie encode from a "reference DVD" or even the odd chapter or two but it would be nice to think the guys at DivX, 3ivx, XviD, VP6, Real, Ateme etc would. Especially as many of their representatives are active on this forum.


Cheers

[Doom9]

Okay, I can't imagine the likes of Micro$oft would be willing to generate a full movie encode from a "reference DVD" or even the odd chapter or two but it would be nice to think the guys at DivX, 3ivx, XviD, VP6, Real, Ateme etc would. Especially as many of their representatives are active on this forum.I had this option for two codecs last time, but I had to decline. How can I be sure that whatever encoded material I'm getting is encoded the way I want it to be, and that there's no foulplay anywhere along the way?

Only when I encode on my own do I know that the results are beyond any doubts.

[SeeMoreDigital]

Originally posted by Doom9
I had this option for two codecs last time, but I had to decline. How can I be sure that whatever encoded material I'm getting is encoded the way I want it to be, and that there's no foulplay anywhere along the way?

Only when I encode on my own do I know that the results are beyond any doubts. I guess you would have to outline what sort of "foul play" you're talking about.

That said, if you asked some of these companies to generate encodes (from a nominated source) to a certain file size or at a certain bitrate and stated what bells and whistles you wanted then to use... I can't see the harm in asking them to do this!

Or why not say to some of these companies... Okay, I'd like you to encode "this" 96 minute movie, with audio at "96Kbps" using "xxx by yyy" pixels and stick it onto a 702MB CD~R. Then post it to me with details of how you did it?

At the end of the day it would still be up to you to decide if what they've done is above board.


Just a thought

[Soulhunter]

Originally posted by SeeMoreDigital

That said, if you asked some of these companies to generate encodes (from a nominated source) to a certain file size or at a certain bitrate and stated what bells and whistles you wanted then to use... I can't see the harm in asking them to do this!

Maybe different/stronger filtering, or a "source tweaked" encoder !!!

Who knows... :rolleyes:


Bye

[SeeMoreDigital]

Originally posted by Soulhunter
Maybe different/stronger filtering, or a "source tweaked" encoder !!! Maybe...

If filters are going to be used, then lets know about them... so we can all use them... As for tweaked encoders... lets have these too!


Cheers

[Soulhunter]

With "source tweaked" encoders I meant...

Tweaked to give better results @ this specific source !!!

Could mean that you get worse results with a different source... ;)


Bye

[SeeMoreDigital]

Originally posted by Soulhunter
With "source tweaked" encoders I meant...

Tweaked to give better results @ this specific source !!! I see...

At least by asking the codec developers to generate and submit their own "comp test" encodes, Doom9 could use them as a base line... Meaning, if he can't generate his own encodes to come anywhere close to theirs, he would be able to ask them why... and report such.

There has to be an element of trust. And should that trust be broken, the codec manufacturer should deserve to be exposed as cheats!


Cheers

[Doom9]

Meaning, if he can't generate his own encodes to come anywhere close to theirs, he would be able to ask them why... and report such.That would mean I'd have to encode everything on my own as well.. there goes the time advantage. Plus a limited amount of filtering might not be detectable, thus offering a lot of possibilities for foul play. And, of codec makers do the encoding, you can forget about reproducability.. who says I got all the settings that matter?

I see this idea the same way as having a CPU maker having his own benchmarks published in a CPU test, rather than have the site that publishes the article do their own tests.. there should be a good amount of scepticism towards sources that are clearly biased. I wouldn't trust results that I cannot reproduce - something which was drilled into me at college.. in science unless you can reproduce it, there's no reason to believe in it.

[SeeMoreDigital]

While I understand what you're saying, I think it's a shame you don't trust (some) of the codec manufacturers and ask them to get more involved.

Like I say, at the end of the day they will only have themselves to blame if they cheat...

Or... how about this one. Why not start a new "Codec Shoot-Out" solely for the manufacturers to get involved in... a sort of "Put-up or Shut-up" shoot-out!

Still give them boundaries to work within (similar to the ones I mentioned before) but let them use any method of encoding available to them... It might prove rather interesting all the same!


Cheers anyway.

[Sagittaire]

Tweaked to give better results @ this specific source !!!

For example with DivX 5.2.1:

The best mode with VFW interface is -pq 64 (slow mode) but with register tweak the best mode is -pq 255 (slowest mode) ...

For better result DivX EKG could be used ...

[SeeMoreDigital]

I wonder how many people have actually used DivX's EKG application on a full movie encode?

I dare say they might have tried it a couple of times and abandoned the tool ;)


Cheers

[Doom9]

Like I say, at the end of the day they will only have themselves to blame if they cheat...How can you possibly prove cheating? It might not be that apparent. It is rather naive to think that given the opportunity to make your own product look better that nobody would actually do it. And your 2nd idea once again comes to "no reproducibility". What good is a test if you cannot reproduce the results? Since it's my name under the test, I'd rather have nothing than something I consider to be unworthy of publishing.

And I'm sure MS would go on record saying they ripped that and that movie and then mailed it to a 3rd party they haven't even met.

Last but not least.. try making a comparison with 10 codecs and 8 sources.. only the comparing part will make you want to run away screaming.. and it's the part where you absolutely must not screw up. And then the screenshot taking part.. I suggest you try it once.. it makes you want to kill yourself.. it really sucks beyond imagination.. go to in MPC sure is nice, but not every codec reacts the same, so you end up with image-flicking to get the right shot and that's definitely not my favorite past time.

[Sagittaire]

Good subjectif test for me:

Full movie encode:
- easy encoding (90 min in 1 CDR and low quant ~q2-3 with XviD for example)
- medium encoding (120 min in 1 CDR and medium quant ~q4 with XviD for example)
- hard encoding (150 min in 1 CDR and high quant ~q6-8 with XviD for example)

Littles samples (30 sec for example): Capture are useless and doesn't prove anything ...

With these codec:
+ RV10
+ VP6
+ WMV9
+ MPEG4 ASP (XviD and DivX5)
+ MPEG4 AVC (H264 Ateme)
- DivX 3.11 (useless for me), x264 and SNOW (in alpha build) ...

-> 3*6 = 18 full encoding ... :devil:

[plonk420]

how about a "double-blind" abx test a la HA? you can beat on these differences that some people can see and not others all day, but in the end shouldn't it be about sample1a vs sample1b .. and which one just plain looks better and why (with all the switch setting madness being hidden behind the scenes) .. and the average of all the responses where no one's vote counts for more than anyone else's?

*shrug*

[Nil Einne]

Would one option be to give the source to the codec manufacturers and the bitrate and tell them to work outhow they want you to encode it. Then they send you back the settings or perhaps even a AVS script or whatever with the settings. Of course, you will still need to do all the encoding yourself but it will only be a minimal amount of your time that you have to spend. You just load up the all the scripts and start compressing. It'll still take a lot of your computer time, unless you have a load of computers but it'll probably be a lot easier then trying to decide for yourself what settings to use with the various bitrates and sources.

The trouble with double blind tests is to be really reliable for the general population you a large sample of different people. Doing it online is not really that reliable since you tend to get a small sample size, you can't be sure how often a person votes and you usually get a skewed population. Specifically, the people who take part are likely to be people with a strong interest in these matters. It is easily possible, perhaps likely that people have grown adjusted to the natural 'flaws' that on codec generates but they are maybe no to used to the flaws of another codec and therefore consider it poorer quality even though the general population would not*. Also, what do you want to test? There is a good chance a number of people are going to be viewing it close up on their monitors. This is fine if this is what you want to test but if you want to test it at a distance on larger TVs, the results are probably going to be different.

*Off topic but this is frequently a problem IMHO with consumer double blind tests. For example, audiophiles frequently claim a extremely high end hifi with their extremely expensive speakers are better then some cheaper stuff. And they and maybe also the general population can tell the difference between the two. The audiophiles in a double blind test may rate the expensive stuff better sounding but if the general population does not, then it's really not relevant to the general population. Of course, if you want to be an audiophile and decide the expensive stuff is better that's up to you, but the vast majority of people do not. Of course, if you're not an audiophile you're not likely to 'waste' the money. I suppose what I'm trying to say is that it's important to remember just because a segment of the population think something is better, doesn't necessarily mean it is better. The 'uneducated' who either don't notice the different or more likely if they pay enough attention do notice a difference but can't decide which one is better aren't wrong or stupid.

[plonk420]

i'm sure there's a HUGE interest in a shoot out, but now that i think about it, i'm not so sure as to how many would be interested in *doing* the double blind test...

i for one would be glad to if the software is availible (a la ABCHR) and ... bug free, at least to the point of stable funtionality. i could test on a Flat CRT, LCD, and with an extra effort, a 27" TV. the only problem is that the LCD is on my PowerMac.

[Joe Fenton]

The problem I see with double-blind tests on video is in the amount of data to download and compare. If you use 30 sec clips, that limits how much the person has to download and watch, but really screws up the test since short clips don't give you a real sense of how the codec works on longer encodes.

However, (almost) no one is going to download dozens of hour long encodes and watch then all with an eye towards comparing quality. Most the people I know would lose interest after the second viewing. ;)

"Let's see... we now move on to our eighth codec. Two more hours of "Ishtar" in H.264 format! Don't forget to mark how you think it compares to RV10, which you saw the day before yesterday." :D

[Doom9]

"Let's see... we now move on to our eighth codec. Two more hours of "Ishtar" in H.264 format! Don't forget to mark how you think it compares to RV10, which you saw the day before yesterday." ;)

I also tend to think that comparing video is more complex than comparing audio because there are more dimensions.

But the main problem is size (you know how big a losslessly compressed video is) and copyright.. can you honestly imagine a movie studio giving anyone permission to offer 3 chapters of one of their hit titles for download in 10 different formats?

[pogo stick]

So it will be almost impossible to organize double-blind tests using internet. But how about making such test as a scientific project in some University or College? It would be also very hard task, but I am sure that there will be some volunteer. When I was a student I was ready to test almost anything for couple of beer. :)

[stephanV]

Originally posted by Joe Fenton
The problem I see with double-blind tests on video is in the amount of data to download and compare. If you use 30 sec clips, that limits how much the person has to download and watch, but really screws up the test since short clips don't give you a real sense of how the codec works on longer encodes.
For proper testing you use 10 second clips, 30 seconds is already way too long. But of course, screenshots are an excellent way of determining how a codec works on longer encodes... right? Right.

"Let's see... we now move on to our eighth codec. Two more hours of "Ishtar" in H.264 format! Don't forget to mark how you think it compares to RV10, which you saw the day before yesterday."
What are you talking about???

Anyway, do a search for VQEG and find out about subjective testing...

[Archimedes]

To the subjective codec shoot-out test i’ve added an "objective" test with Fritz Framalyzer.

I’ve compared all pair of images (original frame and encoded frame) with Fritz Framalyzer and noticed the results (euclidean distances in RGB). This causes an encoder order for every scene and therefore for the complete film.

(Edit.)

Josef

[Koepi]

Originally posted by Archimedes
To the subjective codec shoot-out test i’ve added an "objective" test with Fritz Framalyzer.

I’ve compared all pair of images (original frame and encoded frame) with Fritz Framalyzer and noticed the results (euclidean distances in RGB). This causes an encoder order for every scene and therefore for the complete film.


You didn't read the code comparison properly it seems. You _cannot_ judge a codec from still frames. In one codec there might be a higher quantized b-frame where another codec uses an I- or P-frame.

Also you don't get the overall impression of the codec at running the movie at normal speed.

Thus doing a codec comparison just by comparing screenshots is completely invalid. (No, I didn't look at your comparison; dunno if xvid lost or won or anything [i couldn't care less because the test is flawed big time anyways], just "to take the wind out of some people's complains" which will sure arise.)

Cheers
Koepi

[JnZ]

Originally posted by Archimedes
I’ve compared all pair of images...

Koepi have right. Codec comparsion based on frame comparsion only is bad. Just notify, that if you encode movie by different codecs,you've got different I,P,B frames distribution. So if you want compare images from eg. frame 1113, one codec have there I frame, other B frame etc.

And other thing is, how movie looks when you play it (this is real, not comparing static images).

Bye

[subliminal]

yes that would be nice

[Archimedes]

You’ve right, sure, i allready know that. One can not compare i. e. I frames with B or P frames. Normally the best would be to analyze the complete videos (frame by frame). As i only have access to the images i could’nt compare the videos itself.

So, it was’nt really a test. It’s only an additional information to the subjective image viewing (comparing) process from the codec shoot-out sides. And the question, how objective values cooperated with subjective impressions. No more.

Josef