[New Test] XviD vs Other Codec ... ;-)

[Sagittaire]

My test is Here (http://jfl1974.free.fr/HTM/32_Test_Codec_Video.htm)

With popular source : Matrix Reloaded Chapter 20 to 26 PAL 16/9 MPEG2 MP@ML ~6 Mbps

With usual (best?) setting for XviD 1.0.1: VHQ4, Ultra high, trelli, all interval quant 2-31, HVS-Better, GMC, QPel, AQ, Bframe 2/1.50/1.00

With usual resize: 720*304 in 2.35 (~640*352 in 16/9)

With Usual Compressibility: Very Low compressibility (streaming bitrate) and Very High compressibility (~97% or q2 encoding)

And MPEG4 ASP isn't the best against the WMV9 VCM, VP6, or RV10. Test in french but graph, sample, capture are an universal language ...

[Sharktooth]

IMHO VP6 is the best codec around.
MPEG4 ASP doesnt stand a chance, however MPEG4 AVC (with all its features) could fight against those more advanced codecs like RV10 and VP6.

[Didée]

Yes, Sagittaire. It is good now.

You told us those convincing facts so many times by now, that even I have learned my lesson.

Mpeg-4 ASP is indeed totally crap. It is probably the most inferior codec one can use at all for video encoding!

Oh, BTW, you should ultimatively give CorePNG a try!!

That one is supposely the very very best ;) for encoding computer generated images. - And since you are so deep in staring at graphs and charts, this one should be your preferred codec!

I myself am still of old school: I usually watch video of natural image sources. And that is the only needed argument why I still use this sort of l00ser-codecs every day.


Sarcasm? Me?? Never!


- Didée

[Tommy Carrot]

Originally posted by Sharktooth
IMHO VP6 is the best codec around.


That might be true, but as far as i see VP6 is practically useless because of the quality fluctuations. It has that bothering phenomenon that during the slow scenes the details are slowly going away then suddenly everything is sharp again (not because of the keyframes), and this is repeated in every sec or so.

[Sagittaire]

That one is supposely the very very best for encoding computer generated images

Now it's hard to find a movie without computer generated images ...

Sarcasm? Me?? Never!

[Sharktooth]

Originally posted by Tommy Carrot
That might be true, but as far as i see VP6 is practically useless because of the quality fluctuations. It has that bothering phenomenon that during the slow scenes the details are slowly going away then suddenly everything is sharp again (not because of the keyframes), and this is repeated in every sec or so.
Well i made a lot of VP6.2 encodes for my backups and i didnt experience all those quality fluctuations. However after i bought a DVD burner i preferred XVID Q2 + AAC encodes in MP4 container for archivial purpouse.That grants me MPEG4 compliance for future standalones playback and an almost perfect quality.
But i still use VP6.2+AAC in AVI (thanks to avimuxgui) for 1CD backups (sorry but i hate realmedia stuff...).

EDIT: I'm also doing some experiments with theora+vorbis in OGG container (im fashinated by the patent free thing...), but video quality is not as good as i wish.

[Sagittaire]

For low and medium compressibility RV10, VP6 or WMV9 are unbeatable ...

For very high compressibilty all codec are good ... DivX3 ffdshow will give also very good results ...

[communist]

Originally posted by Sagittaire
MPEG4 ASP isn't the best against the WMV9 VCM, VP6, or RV10.
It amuses me to see that you already gave away the obvious answer away :rolleyes:

Really what exactly is the point of this thread in this forum? Annoy the XviD users (and fans)? Well I certainly dont care but I think a mod should move this into the New A/V Formats - Codecs forum since its not just about XviD :o

[Sharktooth]

Well i think it's here cause it's Xvid vs Other codecs...
I'm xvid user too and i'm not annoyed by the fact there are better codecs around.
Xvid 1.x is the BEST MPEG4 ASP codec ever made and there is still room for improvement (for example VHQ in bframes...), its free and sources are available.
It is MPEG4 compliant (1.1 will also include VBV), that means it is and will be widely supported by hardware manufacturers (read standalone players).
Maybe 2.0 will also feature MPEG4 AVC (h.264) with a major improvement in video quality.

[Tommy Carrot]

It's amusing that Mpeg4 ASP is how inferior in these _artificial_ tests, however in the practical usage these "better" codecs couldn't convince me at all about their superiority, and believe me, i've extensively tested all of them.

I can easily accept those are better at low bitrates, but it has no significance for me.

Edit: i gotta agree, i also believe h.264 will bring mayor quality improvements.

[Sagittaire]

Xvid 1.x is the BEST MPEG4 ASP codec ever made

it's true, in order for metric Test XviD 1.0.1, DivX 5.1.1, ND MPEG4 and 3ivX ...

It's amusing that Mpeg4 ASP is how inferior in these _artificial_ tests

Capture it's useless... there is too much difference between the frames (local IPB, motion, luminance ...). Sample in 500, 1000 and 1500 for 25, 50 and 75% compressibility for each codec: if you don't like metric there is nothing better than a sample to compare visual quality ...

[Sharktooth]

Originally posted by Sagittaire
if you don't like metric there is nothing better than a sample to compare visual quality ...
The samples you provided are quite explicative.
Just one remark: next time dont use AQ for the XVID encodes at mid-high bitrates.

[stephanV]

I'm getting a bit tired by all those "private" codec tests for several reasons:

1. Their statistical significance is virtually 0. They usually only test one source and from that source only a few fragments. Compare that with an audio-codec test 'by Roberto. Also, i believe capping people are still using DivX 3.11/NanDub. Are they all insane (as far as capping people are sane anyway ;) )? Or is testing DVD-sources something different as testing capped sources? And what about DV-stuff?

2. The fragments presented are usually screen-shots. Use of screen shots is not completely irrelevant (they are a good way of showing artifacts) but it is not comparable with viewing a video in real-time.

3. The conclusion drawn from them go way to far considering the amount of sources used and fragments presented. Again I refer to the audio-codec test. (http://www.rjamorim.com/test/multiformat128/results.html)
One codec can be said to rated better than another codec with 95% confidence if the bottom of its line segment is at or above the top of the competing codec's line segment. For example, in the chanchan plot below, Lame is rated better than Atrac3 with 95% confidence. And iTunes is rated better than Lame with greater than 95% confidence.
Compare such a conclusion with:
For low and medium compressibility RV10, VP6 or WMV9 are unbeatable ...
To what test are you refering there?

4. The intentions of the test are usually wrong. Lots of the private tests seem to be there to prove codec A is better than codec B. Well it might very well be that codec A is actually better, but performing a test to prove that is not necessary as you already think so anyway.

I'm not saying people should stop testing, but that people should learn to interpret their results. Using one source to find a definite answer to which codec is best is just ridiculous. Small tests are good for showing developers artifacts/bugs and the like (you can say: "Hey, your codec does something funny here that another codec doesn't do") but much more is just not possible. You've only proven then that on that particular source and that particular bit rate you think that codec is the best.

One of the few codec testers I've seen who was fair about his results was doom9. (although he still gets mis-interpret a lot, but that's not his fault)


You all can continue now. :)

[Sharktooth]

Your point is very valid stephanV, infact i didnt say VP6 is the best codec. I just said IMHO VP6 is the best codec.
However when i'm about to do a 1CD movie encode i usually do some tests to see what's the better one for the material i'm encoding.
I usually choose some sequencies, estimate the final bitrate and encode them with Xvid, VP6 and WMV9 (both). Then i decide what codec to use for the full backup.
I use VP6 on 9 out of 10 encodes. This is enaugh for me to state that IMHO VP6 is the best codec.
But as i said in one of my previous posts i use Xvid Q2 to encode on dvd-r for archival purpouse.

[Sagittaire]

my test: sample ~20 min chapter 20 to 26

- Visual Test: "blind test" with 3 differents observer
- Metric Test: the metric are relatively good for comparaison and are used by all the codec developpers. They use them himself to compare their codec vs other codecs ...
- Capture test: it's useless and that doesn't prove anything
- Sample test if you don't like Visual, Metric and Capture: 60 sec for low, 40 sec for high and 20 sec for high compressibility it's largely enough because with RC regulation normally quality is very constant (VBR encoding)


Else make the test with Matrix Reloaded or other source ... but results and conclusions will be the same: VP6 or RV10 are best, then WMV9 and finally MPEG4 ASP ...

[Andrey]

"Extreme" 1er 1er 1er 1er 5eme
Where: XviD RV10 VP6 WMV9 DivX3
Hmmm. Strange. Do the samples messed up ?
See Extreme high motion & low motion scenes.
What I can tell from images RV10 result is simply annoying, to be honest...
VP6 lack sharpness.
WM9 is only one who is acceptable along with Xvid.
If I put those 1-5 by hand, RV10 is definitely 2, not 5...
EDIT: For whaterver reason Divx3 image is not showing for me :(
EDIT2: WM9 on meduim is ugly on still shots :D
EDIT3: Yeah, RV10 is crap on Morpheus face on Low - details are not only smooth, but that sun blick (or whatever it is in English) is not on it's original place :) VP6 is REALLY good on Low matrix shot...

[Sagittaire]

it's very hard to compare codec with high compressibility encoding ... IMHO RV10 is best with Sharpest image but the differences are too small and it's too subjective ...

For high compressibilty all the codecs are very good: all the choice are good and XviD is the best choice because it's open source ...

[Andrey]

IMHO RV10 is best with Sharpest image
Hmmm.
It seems to me that I have a bit different eyes :D
(BTW, I fail badly when I need to distinquish colors, but on brightness my eyes worked very well last time I checked)
From still pictures you provided I like most: VP6, XVID, WM9 (a bit blurry sometimes) but the most annoying is RV10 IMHO.
On low it is simply a crap for me.
And WM9 badly failed on meduim low motion shot for my taste...
Xvid is too distorted on low to see it on monitor(MPEG quant ?). TV is needed.
(I do not count divx3, because I have a half of it's samples loaded)
I think, we need double blind test to figure it all up ! :D

[Sagittaire]

screenshots test: it's useless and that doesn't prove anything

with a "good choice" for the frames "I prove" that DivX3 is better than XviD: for exemple bframe for XviD and Iframe for DivX3 ...

Video quality evaluation is impossible with screenshots: conclusion for frame N is not true for frame N+1 especially with XviD/RV10 and Bframe ...

The only good visual evaluation for a video codec is a sample in a "blind test" ... download sample and compare ...

[Soulhunter]

Compared the screenshots...

IMO, only WMV9 is able to reach XviD's quality @ high bitrates (c+d) !!!

Btw, wrong sample @ c) Comp. High / RV10 Low Motion... ;)


Bye

[Sharktooth]

WMV9? Uhm... It smooths out too much details...
It's good for low bitrates but not pleasing at high bitrates.
IMHO the best shots @ high/extreme are XVID and VP6.

[Sagittaire]

IMHO the best shots @ high/extreme are XVID and VP6.

Perhabs for N frame but not or N+1 Frame ...

Exemple in PSNR sequence for RV10 (and XviD)


TR Q am PSNR Bits MiPSNR AvPSNR AvBits Fr.type
=======================================================
352080 5 2152 47.09 88664 40.04 44.78 87567 100
352040 10 180 45.18 17832 40.04 44.78 87560 d 100
352160 9 13728 42.01 398776 40.04 44.78 87595 Key 100
352120 12 5512 38.70 199520 38.70 44.78 87608 d 100
352240 7 5696 42.30 334496 38.70 44.78 87636 100
352200 12 3472 38.88 190008 38.70 44.78 87647 d 100
352320 8 5628 41.76 238992 38.70 44.78 87664 100
352280 12 4144 39.11 170016 38.70 44.78 87674 d 100
352400 8 5456 42.19 187776 38.70 44.78 87685 100
352360 12 4400 39.77 110568 38.70 44.78 87688 d 100
352480 8 5276 42.28 198952 38.70 44.78 87700 100
352440 12 3016 40.27 87112 38.70 44.78 87700 d 100
352560 8 1028 42.57 115368 38.70 44.78 87703 100
352520 11 284 41.37 31976 38.70 44.78 87697 d 100


PSNR = 38.88 dB for 352200 ms
PSNR = 42.30 dB for 352240 ms

42.30 - 38.88 = 3.42 dB

the bframe causes very large local disparity of quality for RV10 and XviD. ScreenShoot for Visual Comparatif Codec is impossible: Bframe for XviD, Pframe for RV10 or Pframe for XviD, Bframe for RV10 and the conclusion will be not the same ... :devil:

[Soulhunter]

Originally posted by Sharktooth
WMV9? Uhm... It smooths out too much details...
It's good for low bitrates but not pleasing at high bitrates.
IMHO the best shots @ high/extreme are XVID and VP6.
True, WMV9 usually smooths the background more than VP6 (also -> extreme) !!!

But if you watch the foreground (skin pores) WMV9 is closer to the original IMO... :rolleyes:

Nah, think its a matter of personal preferences !!!


Bye

[guada]

Hello sagittaire, :)

I probably go to ask you a silly question:
Does exist videos test to feel the codecs, I speak about it because to test my installation high fidelity, I call on CD test audio.

Thanks for your answer

[gnr]

nice test. i want to try vp6. already installed the codec but didnt find the target file size option as i can see in xvid. what should i use for encoding? plz help.

[Koepi]

lol - and newbies fall for your pseudo-scientifical tests, amazing :)

Could you PLEASE add a huge big warning to your comparisons and tests, that they're scientifically not correct (we discussed that to death already) and represent only your, and your oppionion alone?

Thank you.

[Andrey]

Soulhunter: only WMV9 is able to reach XviD's quality @ high bitrates
That what I ment. This is why I asked, if resulting still pics are not swapped occassionally...
Koepi: and newbies fall for your pseudo-scientifical tests
He-he... :p
BTW, the fact that Xvid and VP6 wins on still shots does not mean that they automatically wins when comparing videos...
As I already said, double blind test needed to check it...
EDIT: grammar

[Sagittaire]

Originally posted by Koepi
lol - and newbies fall for your pseudo-scientifical tests, amazing :)

Could you PLEASE add a huge big warning to your comparisons and tests, that they're scientifically not correct (we discussed that to death already) and represent only your, and your oppionion alone?

Thank you.

my test: sample ~20 min chapter 20 to 26 in 720*304

- Visual Test : "blind test" with 3 differents observer ...

- Metric Test : the metric are relatively good for comparaison and are used by all the codec developpers. They use them himself to compare their codec vs other codecs ...
http://www.on2.com/pdf/vp6_white_paper.pdf

- Capture test: it's useless and that doesn't prove anything because local specification for exemple ...

- Sample test: 60 sec for low, 40 sec for high and 20 sec for high compressibility it's largely enough because with RC regulation normally quality is very constant (VBR encoding) ...


Else make the test with Matrix Reloaded or other source ... but results and conclusions will be the same:

- Low Compressibility 25%: RV10, WMV9 or VP6 are very better than XviD or DivX3. when there is a difference of 5 point to the SSIM to discuss is useless ...

- Medium Compressibility 50%: VP6 and RV10, then WMV9 and XviD but differences are small except for DivX3 ...

- Medium Compressibility 75%: Really difficult to compare and it's really subjective: RV10, VP6, XviD, WMV9 then DivX3

- High Compressibility 100%: Too much difficult to compare ...


@ Koepi

IMHO MPEG4 ASP isn't the best codec: RV10 or VP6 are very better ... but if there are exemple which prove the opposite I would like to see them ... and not with ScreenShoot but with sample ...

[unmei]

i could not agree more with StephanV.
I know pretty well which codecs i like for encodes and which ones i like to test a bit, there is not much to influence :) only thing such tests can do for me is convincing me to test another one once more, but that is usually just for fun - i know pretty well it is not going to be a test that changes my mind.

However if someone were to take the trouble to coordinate a not too large blind test i'd gladly participate. It were interesting to see how much my opinion differs from what i actually do prefer when i not know what codec goes with what output, and also what characteristics other people look for/at.

[stephanV]

Originally posted by Sagittaire
my test: sample ~20 min chapter 20 to 26 in 720*304

again, statistacilly insignificant.


- Visual Test : "blind test" with 3 differents observer ...


again, statistacilly insignificant. nowhere is stated how the test is performed (my french is not that good though), therefor testing conditions are uncontrolable and therefor everybody should refute these results. also, the error margin caused by only three persons and only one source would cause all codecs to be tied. even Indeo or Cinepack would probably tie with any of these codecs in such a small test.


- Metric Test : the metric are relatively good for comparaison and are used by all the codec developpers. They use them himself to compare their codec vs other codecs ...
http://www.on2.com/pdf/vp6_white_paper.pdf

ALL codec developers? I see koepie object a little to this test. Moreover, your source is particulary bi-ased. you dont expect us to accept PSNR metrics because a codec developer that scores good in that area uses this in his own tests. (which again, only included a very small amount of samples


- Sample test: 60 sec for low, 40 sec for high and 20 sec for high compressibility it's largely enough because with RC regulation normally quality is very constant (VBR encoding) ...

again, its only relevant for that source.


Else make the test with Matrix Reloaded or other source ... but results and conclusions will be the same:

this is ridiculous, thinking you can extrapolate your results on ANY source by having tested only one.

im sorry, but scientifically your explanation of the results is completely off.

[Sagittaire]

this is ridiculous, thinking you can extrapolate your results on ANY source by having tested only one.

Tests PSNR/SSIM with multiple source/resolution and never MPEG ASP was better than VP6 or RV10 ...

but if there are exemple which prove the opposite I would like to see them ... and not with ScreenShoot but with sample ... :D

I think that if the conclusion were "XviD is the best" everyone would have said "very good test" ... but conclusion is "XviD isn't the best" then everyone would have said "not very good test" ... lol

[stephanV]

Originally posted by Sagittaire
Tests PSNR/SSIM with multiple source/resolution and never MPEG ASP was better than VP6 or RV10 ...


first you have to prove that those metrics are accurate enough to replace human eyes. if you cant, any of those results can just as easily be refuted.

[Sagittaire]

I repeat ... if there are exemple which prove the opposite I would like to see them ... and not with ScreenShoot but with sample ... with these information if possible ... :D

- Source
- Script avs used
- Sample with XviD, WMV9, RV10 and VP6

[stephanV]

Originally posted by Sagittaire
I think that if the conclusion were "XviD is the best" everyone would have said "very good test" ... but conclusion is "XviD isn't the best" then everyone would have said "not very good test" ... lol

so it was my point 4 after all. you were out on causing a fight.

i don't care if XviD finishes last, i don't even use XviD. what i do care about is that when such a test is performed, that it is done in the right way and not with a crappy non-scientifical method. (sorry to be so blunt) also, the interpretation of the results should be according with the test and you're way of there again. the certainty with which you present your conclusions is just as ridiculous as is your stubbornness to defend them.

i will shut up now, and let every one else decide for himself.

[Koepi]

This is a good closing post StephenV, thank you.

Since this is all about sagittaire's personal oppinion it's no use for anyone.

My last statement:
I wonder why you keep on repeating "ne-na-ne-na-na, show me show me show me" if you don't even answer (and falsify) the points we put in question. You simply can't so you try to distract from them in a very childish manner.

I for sure won't waste my time by installing yet some more codecs, _properly_ setup some test sources, and so on and so on (you didn't take the time either, so why should I? Doom9 already did that.)

<flame>
Unless you do a proper comparison don't ever start a thread like this again here, I'll remove them immediatly. (Or let the people have some fun like in this thread about your serious lack of understanding the in-depth meaning of quality metrics and "blind tests" obviously [no, blind test doesn't mean you should let someone judge who's blind ;) ] and close it a few hours later.)
</flame>

Read the forum rules again. Or do we lack a rule against trolling / intentionally start flame threads?

Koepi