2004 codec shoot-out

[vlada]

Does anybody plan a new codec comparision? This time I would add Theora, h.264 (x264, Ateme) and compare it to XviD, DivX, RV10, WMV, VP6 and maybe even MPEG-2. I would also add a new source, not only DVD rips. It would be interesting how codecs perform in compressing DV material or in HDTV resolution with high bitrates. I now it takes a lot of time to do it.
Vlada

[deXtoRious]

I'm unable to find Ateme's h264 version anywhere :(

[Tommy Carrot]

Originally posted by deXtoRious
I'm unable to find Ateme's h264 version anywhere :( This is no wonder, it has not yet been released. :D

I wouldn't include theora, if the official theora encoder has any resemblance to the theora encoder in ffdshow, because then it's VERY inferior to the other codecs, not worth to waste any time.

[deXtoRious]

Originally posted by Tommy Carrot
[B]This is no wonder, it has not yet been released. :D

Thanks for clarifying :)

[Sagittaire]

Make codec comparison with capture is very useless:
- VP6 and "Shapeness": Sharpeness effect, conclusion for n frames isn't the same for n+1 frame
- DivX5, XviD, RV10, H264: P and Bframe difference, conclusion for n frames isn't the same for n+1 frame.
- For All codecs: Rate Control difference

http://jfl1974.free.fr/Test/Comparatif/XviDvsWMV9-2000.jpg

the only solution to compare visually codecs is to make samples with the same size. For me make comparison with capture is very useless.


Matrix II 720*304

Source DVD PAL 720*576 MPEG2 MP@ML, chapter 20 to 26, ~35 000 frames

Test in french (http://jfl1974.free.fr/HTM/32_Test_Codec_Video.htm) ... but Capture, Sample & Graphs are universal language

Codecs
- H264 Ateme
- XviD 1.0.1
- RV10 Elysean
- VP6 6.2
- DivX 5.2.1
- WMV9 VCM
- DivX 3 ffvfw


Tests
- 500-2000 Kbps and 25-100% compressibility (/quant2 XviD H263)
- SSIM, Average PSNR, Overall PSNR and Frames PSNR test
- Blind test
- Samples
- Diaporama

[IgorC]

If I'm not mistake Sagitariie has compared ffdshow x264 with other codecs before . And result wasn't bad at all. ffdshow x264's quality was between a little better than WMV 9, but worse than RV10.

And now ffdshow is in active development , just after beta testing of Ateme H.264 was over ;) . So it will be good to see ffdshow x264 in comparation test.

[McoreD]

Thanks for the link for your comparison test Saggitare.

Now you have very much convinced me that the following ranking is reasonable.

1- H264 Ateme
2- VP6 or RV10
3- WMV9 or XviD
4- DivX3 ffdshow

Cheers,
McoreD

[dragongodz]

i think vlada is saying he would like to see a new Doom9 codec comparison with these new codecs etc.

not much point really until they are really released though. :)

[Koepi]

LOL, yet another user is believing in Saggitaire's _scientifically incorrect_ comparisons.

What a waste.

Congrats though, this made my evening *rotfl*

Regards
Koepi


EDIT: removed potentially insulting "someone is dumb enough", wasn't meant insulting.

[Sirber]

@Koepi

Why are you mad at Sagitaire? Is it because XviD comes 3rd in his tests?

[PiXuS]

@Sirber

Just read past posts (http://forum.doom9.org/search.php?s=&action=finduser&userid=33350) from Sagitaire...

[RadicalEd]

Originally posted by PiXuS
@Sirber

Just read past posts (http://forum.doom9.org/search.php?s=&action=finduser&userid=33350) from Sagitaire...

17 pages of non-contextual posts isn't very helpful in determining why what should be an objective representation from the administration isn't. I'd like some more insightful details about this, because I haven't found flaw with Sag's tests. Largely because my own results are rather consistent with his and his testing methodology is pretty expansive.
So, anyone care to explain?

[Koepi]

Encode a movie to 1 or 2 cds, once with vp6, and once with xvid.

Compare the result. You'll be surprised (...how saggitaire can mess up with codec settings to make them produce worse output than default settings).

I mean, it's hard to ignore the "pumping" quality of VP6 which is very annoying for my taste.

@sirber:
nope, that's not the reason. The reason is that he doesn't use sane settings for the codecs. And then some background knowledge which you obviously don't have or already forgot about his scientifical background - but this doesn't belong in the public, I'd recommand a good google search on french video fora.

Regards
Koepi

[Doom9]

some of you might have noticed during this past week that I was not to be seen a lot.. in fact I was offline from Tuesday until Saturday. 8 hours from now I'll be offline until Saturday again, and the same thing goes for the week thereafter. Being away from any computer obviously doesn't go well with codec comparisons ;)

I think the end-of-the-year period is a pretty good one for codec comparisons.. it's cold so people stay inside and warm themselves up with high powered PCs ;)

Codecs included will be those that have been improved since the last time, so DivX, XviD, NeroDigital (the upcoming H.264 part), VP6 and RV10. Other codecs depend on a bunch of factors, like time available (WMV9 has been tested before and hasn't really been improved, perhaps I'll find a suitable libavcodec implementation and can get the proper suggestion as to settings), corporate policy decisions (I hope I'll be allowed to publish ND H.264 tests, and I'm sitting on some other unreleased beta software).

There are no plans for sources yet, but one thing is certain: I don't have time to do a reasonable DVD rip test which imho means at least 3 sources, and include other sources like DVB, DV and analogue captures. Do not forget that this is supposed to be a (non paying) hobby, and most of those asking for codec comparisons would quickly give up the endeavour when they realize how much effort is required.

[Ghim]

In order to do a proper Codec shoot-out, wouldn't it be better to try organising a proper subjective test session ?
There are already standardised protocols developped for video quality assessment like DSCQS or ACR methods (I would recommend DSCQS for mid-high bitrate codecs testing but ACR is very good to test more conditions in a short time).
Of course, there would be a need for at least 6 differents source videos (8-10 seconds).
Then, for each codec included, different bitrates could be tested to test the codecs behavior for different bitrates.

The problem would be to find enough subjects (at least 20) and to find the proper sources for the tests (VQEG sources could be used freely by example). It also represent a large work to produce all the data and prepare the tests but the result will probably be much more scientific than the codecs comparisons done in the past.

For reference, you can try going on www.vqeg.org and get some good documentations on video subjective testing.

[Manao]

Ghim : everwicked is trying to organize such a test. The hard part, for the moment, seems to get the majors / tv channels to allow him to use / distribute samples from their movies / shows.

And, if such a test allows to compare the raw performance of the codecs ( coding efficiency ), it doesn't say anything about the two pass mode - which need longer samples - which is also a foundamental part of a codec.

[Ghim]

Originally posted by Manao
Ghim : everwicked is trying to organize such a test. The hard part, for the moment, seems to get the majors / tv channels to allow him to use / distribute samples from their movies / shows.

And, if such a test allows to compare the raw performance of the codecs ( coding efficiency ), it doesn't say anything about the two pass mode - which need longer samples - which is also a foundamental part of a codec.

I knew that everwicked had developped a DSCQS software but I didn't know he was planning to organise a subjective test. It would be great if it was done. I know how hard it is to get content free to use/distribute and I don't know if he's going to be able to get them for free... There are already quite a few high quality contents used by VQEG that could be reused, even if these contents are not necessary very interesting.

For the 2-pass mode, I agree with you that 8-10 seconds is too short... But this kind of test is still much better than tests done by 1 person and not based on any standard protocol...
There is however a way to include a bit of the 2 pass testing in subjective tests... It would require to encode a very long sequence using the 2-pass mode and then cut 8-10 seconds blocks to create correct-size samples.
An other possible way to do it would be to append all the samples of the test in 1 long file, encode it and then cut the sequences before using them in the test.

[Manao]

It would require to encode a very long sequence using the 2-pass mode and then cut 8-10 seconds blocks to create correct-size samples. That wouldn't work because two pass algorithms would not distribute bits in the same way, some would favorize these 10 seconds, while other would not.An other possible way to do it would be to append all the samples of the test in 1 long file, encode it and then cut the sequences before using them in the test.That'd be better.

[Sagittaire]

Originally posted by Koepi
Encode a movie to 1 or 2 cds, once with vp6, and once with xvid.

Compare the result. You'll be surprised (...how saggitaire can mess up with codec settings to make them produce worse output than default settings).


I use the best setting for XviD and for other codec ... lol

XviD 1.0.1 setting with VDM: 2 pass + Ultra high + VHQ4 + H263 Matrix (good matrix visually, for me best matrix with PP4, best Matrix with metric) + chroma motion + bframe 2/1.50/1.00 + GMC + QPel + Adaptative Quant (psy mode better for eyes and for SSIM) + all quant 2-31 + Post-Process PP4

But if you think that setting aren't the best ... you could make encoding yourself ... source is very accessible. Matrix Reloaded Chapter 20 to 26 with this very usual and simple script:


Video=Mpeg2Source("G:\Mes dossiers\B.A\Matrix\azerty.d2v")
Video=Trim(Video,0,0)
Video=Crop(Video,0,76,-0,-76)
Video=Convolution3D(Video,0,3,4,3,4,3,0)
Video=BicubicResize(Video,720,304,0,0.5)
Return(Video)


My test use bitrate in ~q8 ~q2 interval for XviD with 23 min and 35 000 frames to Matrix Reloaded. All the possible type of ripps are tested:
- Streaming Quality -> 25% for compressibility (~q8)
- 1CDR or 2CDR Ripp -> 50% or 75% for compressibility (~q4 and ~q3)
- DVDR Ripp -> 100% for compressibility (~q2)


My samples are here:
DivX3 500 Kbps (http://jfl1974.free.fr/Video/Sample-DivX3-500.avi)
DivX5 500 Kbps (http://jfl1974.free.fr/Video/Sample-DivX5-500.avi)
XviD 500 Kbps (http://jfl1974.free.fr/Video/Sample-XviD-500.avi)
WMV9 500 Kbps (http://jfl1974.free.fr/Video/Sample-WMV9-500.avi)
RV10 500 Kbps (http://jfl1974.free.fr/Video/Sample-RV10-500.rmvb)
VP6 500 Kbps (http://jfl1974.free.fr/Video/Sample-VP6-500.avi)
H264 500 Kbps (http://jfl1974.free.fr/Video/Sample-H264-500.mp4)

DivX3 1000 Kbps (http://jfl1974.free.fr/Video/Sample-DivX3-1000.avi)
DivX5 1000 Kbps (http://jfl1974.free.fr/Video/Sample-DivX5-1000.avi)
XviD 1000 Kbps (http://jfl1974.free.fr/Video/Sample-XviD-1000.avi)
WMV9 1000 Kbps (http://jfl1974.free.fr/Video/Sample-WMV9-1000.avi)
RV10 1000 Kbps (http://jfl1974.free.fr/Video/Sample-RV10-1000.rmvb)
VP6 1000 Kbps (http://jfl1974.free.fr/Video/Sample-VP6-1000.avi)
H264 1000 Kbps (http://jfl1974.free.fr/Video/Sample-H264-1000.mp4)

If somebody (Koepi perhabs ...???) is able to make better sample with XviD (or other codec) with same script ... then it's very simple he can show their samples ...

Perhaps that Koepi thinks that XivD is better than H264 Ateme or VP6 or RV10 ... but not my eyes and metrics !!!



Originally posted by Doom9
Codecs included will be those that have been improved since the last time, so DivX, XviD, NeroDigital (the upcoming H.264 part), VP6 and RV10. Other codecs depend on a bunch of factors, like time available (WMV9 has been tested before and hasn't really been improved, perhaps I'll find a suitable libavcodec implementation and can get the proper suggestion as to settings), corporate policy decisions (I hope I'll be allowed to publish ND H.264 tests, and I'm sitting on some other unreleased beta software).


1) VP6, RV10, XviD 1.0.2, DivX 5.2.1, H264 Ateme
+ perhabs WMV9 Advanced Profil (new codec) with WM9Encodeur
- perhabs DivX 3.11

2) If possible not capture comparison but little sample comparison ... I repeat but for example VP6 PPP sequence can't be compared with XviD PBP in capture test



VP6 1000 Kbps
32000 1.0637 +0.0771 19 -22 43.3425 P
32001 1.0996 +0.1098 20 -22 43.1607 P
32002 0.9917 +0.0707 14 -15 44.2863 P Sharpeness?
32003 1.1154 +0.0983 20 -22 42.9298 P
32004 1.1319 +0.1286 18 -27 42.9571 P
32005 1.1748 +0.0597 19 -24 42.3986 P
32006 1.1912 +0.0714 26 -27 42.3078 P
32007 1.1160 +0.0881 17 -23 43.2668 P Sharpeness?
32008 1.2040 +0.0560 24 -26 42.0997 P
32009 1.1663 +0.0521 27 -23 42.3630 P
32010 1.2204 +0.0870 24 -26 41.7919 P



XviD 1000 Kbps
32000 1.0800 +0.0510 21 -19 43.2779 P
32001 1.3259 +0.0704 38 -43 40.5481 B
32002 1.1041 +0.0995 16 -19 43.2003 P
32003 1.3393 +0.0357 32 -34 40.4564 B
32004 1.1372 +0.1027 17 -29 43.0339 P
32005 1.3630 -0.0232 39 -38 40.3129 B
32006 1.1313 +0.0603 18 -22 43.0729 P
32007 1.4107 +0.0197 33 -40 40.1519 B
32008 1.1154 -0.0027 17 -22 43.1401 P
32009 1.3603 -0.0379 34 -43 40.2671 B
32010 1.0825 +0.0630 19 -26 43.2931 P


Very better frame for XviD:
32006(+0.8dB), 32008(+1dB), 32010(+2.5dB)

Very better frame for VP6:
32001(+2.5dB), 32002(+1dB), 32003(+2.5dB), 32005(+2dB), 32007(+3dB), 32009(+2dB)


DivX5 BPB sequence can't be compared with PBP XviD in shoot test. RV10 or H264 PBBBP can't be compared with PBBPB XviD in shoot test. Rate Control specification (H264, RV10, WMV9, XviD or DivX) can't be compared in capture test ...

[dragongodz]

well i happen to like Doom9's tests and find them an interesting read every time he does a new one. check them out some time if you havent.

using PSNR for results should be done with a great deal of caution aswell. it is not 100% reliable. rely on your own eyes before believing anything else.

in the end its up to each person to decide what they prefer so do some tests for yourself and then decide. :)

[Sagittaire]

Originally posted by dragongodz
using PSNR for results should be done with a great deal of caution aswell. it is not 100% reliable. rely on your own eyes before believing anything else.


PSNR can't show difference with 0.1 dB or 0.5 dB but can with 2 dB or 3 dB


Very better frame for XviD:
32006(+0.8dB), 32008(+1dB), 32010(+2.5dB)

Very better frame for VP6:
32001(+2.5dB), 32002(+1dB), 32003(+2.5dB), 32005(+2dB), 32007(+3dB), 32009(+2dB)


in this sequence make capture comparison is very useless and doesn't prove anything ... !!!

[Koepi]

Dragongodz made the proper point here.

Sagittaire, your settings _are_ messed up. AQ is meant for low bitrate scenarios. GMC will give a little better (subjective) quality, but shouldn't be used either - no hardware supports 3 warp point gmc.
Look in my previous post, I didn't say h264 is not superiour.

PSNR can tell you that _something_ changed. It doesn't mean better quality (remember XviD's TOO_SMALL_LIMIT: better PSNR, but visible quality was much degraded, the image was too smooth.). We had that very often, but please feel free to continue making a fool out of yourself.

Koepi

[virus]

Originally posted by Koepi
but please feel free to continue making a fool out of yourself.
I know I'm going to get a strike for that, because it's your usual way of dealing with those who don't share your views, but I don't care, being banned from Doom9 is better than being here as dummy, coward "yesmen" :-\

Sagittaire opened a thread to discuss his settings and used most of Teegedeck's suggestions for his tests. Quite in fact, his settings are very standard. My own results are pretty consistent with Sagittaire's (and RadicalEd said the same, if I understood him correctly).
Also, PSNR has been used for years by educated scientists around the world as a way of getting objective measurements (look at all the H.264 vs MPEG-4 ASP vs MPEG-2 graphs out there).

And I do believe that you're only criticizing because XviD didn't perform well on those tests, too. If XviD came up on 1st place, you'd never complain. Your harsh posts against Sirber prove that, too. IMHO you need to understand that there is life beyond XviD. You can go on insulting everyone which likes different codecs, you're a mod here so you can get away with it :-\

Still, at the cost of being banned from Doom9, I want to let my voice be heard, loud and clear: this is not right. Sagittaire did a great job and I hope he can go on despite the attacks. He gives objective measurements as well as visual impressions and sample clips from the sources. I can't imagine a better way to run a comparison.

Koepi, why don't you allow him to say that he likes VP6/RV10 more than XviD? Tell me why. It's Sagittaire's personal opinion. Could you call him a moron for expressing his personal opinions?

virus

[Koepi]

virus,

i don't quite get it where you have the impression from that i'm such a dick.

I just voice my oppinion, too. Of course sagittaire and anyone else is allowed to have his own oppinion.

Please re-read my last post - this time without that "what a dickhead" stereotype, and you'll see that I have a very valid point there.

I don't care if codec a or codec b or c wins any comparison. But if the base is flawed and not at all scientific but claims to be that (your example is wrong too; those scientific works you are talking about use the measurement PSNR on certain samples and usually talk about drawbacks, too. I don't see that Sagittaire somewhere mentions that he knows about the validity(?) of PSNR tests.) I want to be able to write that without such a weird "you'll strike me, you'll ban me, hehehe."

You certainly need to check the cases where I strike, you have mistaken something for sure.

Could we return on-topic please? Such things belong into PMs.

Koepi

[Sagittaire]

Originally posted by Koepi
Sagittaire, your settings _are_ messed up. AQ is meant for low bitrate scenarios. GMC will give a little better (subjective) quality, but shouldn't be used either - no hardware supports 3 warp point gmc.
Look in my previous post, I didn't say h264 is not superiour.

PSNR can tell you that _something_ changed. It doesn't mean better quality (remember XviD's TOO_SMALL_LIMIT: better PSNR, but visible quality was much degraded, the image was too smooth.). We had that very often, but please feel free to continue making a fool out of yourself.


My test isn't hardware compatibility test:
Qpel and GMC for XviD if it's better for quality. For high quality (q2 and q3) all codec are very good: it's very hard to see if X is better than Y: best possible setting for low/medium quality and psy mode (A.Q.) actived ...


My test isn't only Average PSNR test:
- Metric test: Average PSNR, Overall PSNR and the best metric for me SSIM. For example SSIM is able to detect visual subjectif gain from psy mode for H264, DivX5 and XviD ... try it Koepi
- Capture test
- blind test
- Sample in download

[Ghim]

I just want to continue on the objective measurement. I think people should stop using metrics as PSNR.
There are currently numerous new objective assessment models that are being normalised inside the ITU-T (see www.vqeg.org for more details and competition results) and that give much better results than the PSNR. The first published standard is J.144 for broadcast video quality measurement. The PSNR was used as a reference during the vqeg competition and even if it doesn't perform too bad, it's still quite far from being representative of the opinion scale. I'm not too sure either about the SSIM metrics and how it really compares to the subjective opinion.

[IgorC]

Koepi
if you can do better than these samples it will proove your opinion
http://jfl1974.free.fr/Video/Sample-XviD-500.avi
http://jfl1974.free.fr/Video/Sample-XviD-1000.avi

[PiXuS]

Originally posted by Sagittaire
I use the best setting for XviD and for other codec ... lol

XviD 1.0.1 setting with VDM: 2 pass + Ultra high + VHQ4 + H263 Matrix (good matrix visually, for me best matrix with PP4, best Matrix with metric) + chroma motion + bframe 2/1.50/1.00 + GMC + QPel + Adaptative Quant (psy mode better for eyes and for SSIM) + all quant 2-31 + Post-Process PP4

[snip]


XviD 1.0.2 is just a minor bugfix compared to 1.0.1. But still, why use 1.0.1 when 1.0.2 has been available for months now?

Also, you say you use the "best settings" for XviD. What's best for you? If I refer to your first post, I think you forced XviD into encoding to a target bitrate of 2000kbps, right? Why not to a target size or a target quant? Out of the three possible targets type you could use, I think you used the worst for XviD.

If you wanted best quality, you can't use H.263 matrix. Even the flawed PSNR measure will show you it isn't up to par with MPEG or even better matrices like SixOfNine (do you know about SixOfNine BTW?).

If you wanted the best quality, you wouldn't use b-frames. You wouldn't use AQ. You wouldn't post-process.

Strike 1.

So.... what are you aiming for? To me, when someones says "I want the best settings for XviD", I need to clarify the meaning. Best settings for a 1-CD encode? Best settings for a 2-CD.. 3-CD.. 1-DVD? For Anime material, for good quality Hollywood products, for shitty DVDs.....??? You didn't specify.

Strike 2.

And, no... I ain't going to tell you how to configure XviD properly (there is a TON of information about this subject here on forum.doom9.org and it is the actual point of this post).


PiXuS

[virus]

Originally posted by Koepi
i don't quite get it where you have the impression from that i'm such a dick.
I don't think you're "a dick", but I do think that you find difficult to accept other people's "truth" (which may differ from yours, of course).

But if the base is flawed and not at all scientific but claims to be that
Personally, I don't recall Sagittaire claiming that his results are "scientific" - maybe because I don't understand French very well. But I'd like to know what a person should do in order to run a "scientific" test, according to your definition. Every approach can be criticized.

I don't see that Sagittaire somewhere mentions that he knows about the validity(?) of PSNR tests.
True, but there are a lot of tech papers out there where the authors only give PSNR results without further comments. It's because they assume that the reader already knows about the drawbacks of objective measurements. Anyway Sagittaire provides sample clips - and this for sure shows that he understands their importance, too.

You certainly need to check the cases where I strike, you have mistaken something for sure.
Maybe, because the striker's name is not always public, but still I think I've seen enough, in the last 10 months here; otherwise I'd have kept my mouth shut.

Could we return on-topic please?
I think we're perfectly on topic. We are talking about codec comparisons here, and Sagittaire runs codec comparisons (doing the best he can do - a single person cannot run a complete blind test, unfortunately).

virus

[dragongodz]

when requant was being constantly worked on a change was made that increased PSNR. it was released and people complained. why ? because it actually looked worse. yes PSNR increased but visually it went down. if you want proof go and find it in that mammoth Rejig thread.
so again i say, believe your own eyes first.

Qpel has also been shown to sometimes lower PSNR by the way but i guess these things dont matter since they are suposed to be "the best" settings.

wasnt there an occasional problem with VHQ4 and B frames occasionally smearing detail at 1 point ? probably fixed but which version, i cant remember. Koepi should know what i am talking about. :)

Sagittaire - what are you trying to say by shoot test by the way ???

i don't quite get it where you have the impression from that i'm such a dick.
Koepi please dont give me lines like that, i may be tempted to answer it. :D

[stephanV]

Originally posted by PiXuS
Also, you say you use the "best settings" for XviD. What's best for you? If I refer to your first post, I think you forced XviD into encoding to a target bitrate of 2000kbps, right? Why not to a target size or a target quant? Out of the three possible targets type you could use, I think you used the worst for XviD.
LOL... whats the principle difference between a encode which uses a target bitrate or a target filesize? answer: none. A target quant? why? that wouldnt be fair to the other codecs.


If you wanted best quality, you can't use H.263 matrix. Even the flawed PSNR measure will show you it isn't up to par with MPEG or even better matrices like SixOfNine (do you know about SixOfNine BTW?).
There is exactly no hard evidence that any custom quant matrix is better than H.263... at best you could say they perform better at different ranges.


If you wanted the best quality, you wouldn't use b-frames.
Right... b-frames really stink... errrrr... NOT!


So.... what are you aiming for? To me, when someones says "I want the best settings for XviD", I need to clarify the meaning. Best settings for a 1-CD encode? Best settings for a 2-CD.. 3-CD.. 1-DVD? For Anime material, for good quality Hollywood products, for shitty DVDs.....??? You didn't specify.

He does not have to specify it, i think he chose what he found what worked best on that source (or he took advice as it seems). Targeting a certain filesize is not always a good idea as you have to account for audio too then. Some prefer too keep AC3, other want it MP3 at this and that bitrate and so on... It is allowed to ignore such parameters for testing i think. After all we're testing quality here, not someones need to pack a movie on a CD.

Don't get me wrong... the only part i can appreciate about saggitaire's test are his samples (which is the ONLY way to make a good comparison). The metrics and his blind test (blind test? what? where? how?) are completely worthless IMO.

[Sagittaire]

Originally posted by PiXuS
XviD 1.0.2 is just a minor bugfix compared to 1.0.1. But still, why use 1.0.1 when 1.0.2 has been available for months now?


XviD 1.0.2 is just a very little decoder bugfix



Also, you say you use the "best settings" for XviD. What's best for you? If I refer to your first post, I think you forced XviD into encoding to a target bitrate of 2000kbps, right? Why not to a target size or a target quant? Out of the three possible targets type you could use, I think you used the worst for XviD.


With Matrix reloaded chapter 20 to 26 and with my avs script (in 720*304) bitrate for q2 encoding is 1970 Kbit/s. I use 2000 Kbit/s for 100% compressibility, 1500 Kbit/s for 75%, 1000 Kbit/s for 50% and 500 Kbit/s for 25% in my test for all codec. To simplified 100% is ~q2 encoding, 75% ~q3, 50% ~q4 and 25% ~q8 for average I,P frame with XviD ...



If you wanted best quality, you can't use H.263 matrix. Even the flawed PSNR measure will show you it isn't up to par with MPEG or even better matrices like SixOfNine (do you know about SixOfNine BTW?).


H263 is a very good HVS quantisation (see Soulhunter Test). For me HVS-best is the best matrix with PP0 (less blocking but more ringing) but with PP4 for me the best quantisation is H263 and by far (PP4 is necessary with quant higher than 4)

Strike 1 ... lol



So.... what are you aiming for? To me, when someones says "I want the best settings for XviD", I need to clarify the meaning. Best settings for a 1-CD encode? Best settings for a 2-CD.. 3-CD.. 1-DVD? For Anime material, for good quality Hollywood products, for shitty DVDs.....??? You didn't specify.


My test use bitrate in ~q8 ~q2 interval for XviD with 23 min and 35 000 frames to Matrix Reloaded. All the possible type of ripps are tested:
- Streaming Quality -> 25% for compressibility (~q8)
- 1CDR or 2CDR Ripp -> 50% or 75% for compressibility (~q4 and ~q3)
- DVDR Ripp -> 100% for compressibility (~q2)

For animate H264 and RV10 are very very very better than MPEG4 ASP

Strike 2 ... lol



If you wanted the best quality, you wouldn't use b-frames. You wouldn't use AQ. You wouldn't post-process.


No bframe with XviD ... !!?
No A.Q. ... it's a very good HVS function ... !!?
No Post-Process ... with ~q4 or ~q8 encoding ... !!?

No Comment: strike 3 ... lol


If you want i can make my test with "your best setting" ...
XviD 1.0.2 setting with VDM: 2 pass + Ultra high + VHQ4 + SixOfNine Matrix + chroma motion + no bframe + GMC + QPel + no Post-Process PP0

... but DivX5 must be really better than XviD in this case ... lol

[Rrrough]

by Koepi: Removed all offtopic stuff from the post. Unfortunately that means there's nothign left.

by Rrrough:
Well, virus was right, you can get a strike for criticizing insulting posts from moderators, even if it is off-topic.
ON-TOPIC again, I think it's great when someone is working hard for the community doing
CODEC COMPARISONS, even if you don't agree with its scientific base.
Everyone is up to himself to draw conclusions out of it, so no need for insulting such a work. Everyone has the possibilty to conduct a more scientific comparison and present it here.
Thank you Sagittaire for your work.

[SeeMoreDigital]

Personally, I would like to see some comp tests generated at 1:1, instead of cropped and resized.

That said I have nothing against cropping the mattes away, as long as the vertical resolution is preserved (or as near as damn) to the 16th pixel.

Don't get me wrong, resizing encodes has its place and is very useful. But in my opinion it does give an accurate representation of a codecs true performance characteristics!

But I guess I am a majority of one... again?


Cheers

[babayaga]

Originally posted by Doom9

There are no plans for sources yet, but one thing is certain: I don't have time to do a reasonable DVD rip test which imho means at least 3 sources, and include other sources like DVB, DV and analogue captures. Do not forget that this is supposed to be a (non paying) hobby, and most of those asking for codec comparisons would quickly give up the endeavour when they realize how much effort is required.

Why don't you simply request companies/groups to do the encode themself (it's very time consuming for you I guess) ??
It should not be so unfair since in your previous tests, the encode config was given by the codec authors.

[virus]

Originally posted by Koepi
You certainly need to check the cases where I strike, you have mistaken something for sure.
Yeah, after a quick review, especially at Rrrough's post, I did mistake something indeed. I said the strike is your usual way of dealing with those who don't share your views and this is not entirely true.

I simply forgot to add the other way: deleting criticism to your (bad, imho) behaviour, labelling it as "off-topic". Why don't you edit your own posts where you insulted Sagittaire without a reason instead? Why don't you avoid calling people "dumb" because they read Sagittaire's comparisons? He does this job for us all, and he does it for free. How can you overstep the forum rules this way and delete people's complaints? Is this a fair behaviour?

Again, I want to show all my support to Sagittaire and please, Sagittaire, keep on with it, it's very appreciated.

virus

[Trahald]

Originally posted by dragongodz
when requant was being constantly worked on a change was made that increased PSNR. it was released and people complained. why ? because it actually looked worse. yes PSNR increased but visually it went down. if you want proof go and find it in that mammoth Rejig thread.
so again i say, believe your own eyes first.


Agreed. since a good part of compression benefit (mpeg2/4/etc) is based on psychovisual tricks (example - using lower bitrate in dark are next to a lighted area.. eyes have a hard time seing the detail in the dark area in that case, why waste bits on it) . most codecs have options to crank up the tricks but even with the options off, most/all lossy codecs are about psychovisual tricks. PSNR does not allow for that. it just says.. does this picture look like the origional. if not.. how much different. this may lead to a superior visually picture getting a worse score than another that scores better. psnr != human eyes.

and the point is.. if you want to make an opinion about what codec is the best. great! wmv9 looks better to you.. excellent. its just that taking it from opinion to fact with a test that only suitable if the movie is viewed by a robot is the problem area.

[virus]

Originally posted by Trahald
Agreed. since a good part of compression benefit (mpeg2/4/etc) is based on psychovisual tricks (example - using lower bitrate in dark are next to a lighted area.. eyes have a hard time seing the detail in the dark area in that case, why waste bits on it)
This is an oversimplification, I think.
Most of the compression efficiency does not come from psychovisual tricks, but from advanced encoding tools, like a better entropy coder (e.g. CABAC), a more precise motion estimation (e.g. QPel) and a number of similar state-of-the-art techniques.

And the performance of these advanced encoding tools can be measured by synthetic benchmarks, too (bitrate reduction @ fixed quantizer, or PSNR/SSIM increase). Of course other compression tools "speculate" on the HVS trying to "fool" the eye, while gaining compression efficiency: these are not well suited for objective measurements. So it's a bit of a mix between "objectively measurable" and "subjectively measurable" stuff.

Personally, I prefer to have both synthetic scores (which are just a cheap approximation of subjective ones - but hey, not everyone can organize a blind test with 2000 people everyday ;)) and actual samples that I can watch on my own. So I can compare and see under what situations/settings/sources PSNR/SSIM is a good indicator (and what PSNR/SSIM margin is needed to make a difference) and when instead it doesn't reflect my subjective opinion.

Sometimes and for some codecs/settings/bitrates PSNR works OK, other times it fails. Still, it's a very useful tool, provided that you know how to deal with it.

[Sagittaire]

Once again it's just a small codec test:
- Metric Test
- Capture Test
- Blind Test
- Sample Test

if you don't like the metric test, read the blind test
if you don't like my blind test, read the capture test
if you don't like my capture test, download my sample
if you don't like my sample, make sample yourself

if you think that the codec X can be better with your setting ... make test and post your sample (I wait your sample Koepi).

It's not my fault if codec X is better than codec Y with metric. It's very simple to make metric test (even for "fool" like me ... lol): I read and report result in graph. If you think that Average PSNR, Overall PSNR or SSIM aren't good for codec comparison, it's not my fault. I use metric test because metric test are used by the developers of codec for developping but too, to compare with the other codec (Babayaga, Bobololo, One2Tech, C0mPr355
, karl_lillevold, Digital56K, Gej and certainely sysKin ... )



example - using lower bitrate in dark are next to a lighted area.. eyes have a hard time seing the detail in the dark area in that case, why waste bits on it

Yes it's true ... PSNR can't detect that but SSIM can: SSIM is better with XviD and DivX5 psy mode. SSIM is better with H264 psy mode too ...


and the point is.. if you want to make an opinion about what codec is the best. great! wmv9 looks better to you.. excellent. its just that taking it from opinion to fact with a test that only suitable if the movie is viewed by a robot is the problem area.

How much looks at the WMV9 with the best Pos-Process ... perhabs 10%. WMV9 use deblock and dering with highest level by default.
http://jfl1974.free.fr/Video/WMVPostpross.exe

But for SSIM WMV9 is just better than XviD for low/medium bitrate (~q8 ~q4 interval in my test)

http://jfl1974.free.fr/Test/Comparatif/SSIM.gif

[Tommy Carrot]

I don't want to participate in this debate, just i share that view that the synthetic quality measuring tools are hardly reliable for practical use. There are too many cases when they are contradicting to the reality. Qpel increases the perceived quality, but usually degrades the PSNR result. Or the famous example: if you increase the brightness a bit, you won't notice the difference, but the PSNR result will be very low.

[SeeMoreDigital]

Originally posted by Tommy Carrot
I don't want to participate in this debate, just i share that view that the synthetic quality measuring tools are hardly reliable for practical use. There are too many cases when they are contradicting to the reality. Qpel increases the perceived quality, but usually degrades the PSNR result. Or the famous example: if you increase the brightness a bit, you won't notice the difference, but the PSNR result will be very low. Agreed!

And talking about Qpel. If every encoded pixel contains more detail because during it's analysis, it's been divided into quarters. I'll be damned if I know why it requires "switching on" during playback/decoding!


Cheers

[Sirber]

In my tests (not scientifical), at 500kbps, H264 and RV10 are very simillar. VP6 comes second with some artefacts.

Note: All that is IMHO :D

[Gabriel_Bouvigne]

Just my small personnal (an biased) opinion on synthetic tests:

They have a big interest during codec developement. In audio we (at least me and Ivan) are using them during developement as a quick tools to give rough estimations. Usually, they give some right indications for changes made into a codec.

But this is only usually. In no way they are the absolute truth. They can give totally wrong results, especially when comparing different encoders (in the audio case, at least).

I strongly believe that PSNR is far from beeing good, in an absolute way, as encoders can try to fool the HVS. But on the other hand, current video encoder do not rely that much on psychovisuals, at least when compared to audio encoders using psychoacoustics.

On audio encoders we are using tools like PEAQ, which are trying to take into consideration human audition, instead of pure raw differences like psnr. But given today's state of video encoders regarding psychovisuals, I thing that using psnr, even if not optimal, is still a good indication.
You just have to know how it works, and specially its limitations.

I hope that one day video compression will have evolved enough so that psnr will be a totally useless tool, as in audio.

In the meantime, you can still have a quick rough estimation with psnr.

Btw, there are some other tools that can discern changes in a more subtle way than psnr, which is just telling you if there is a difference, but not in which way.
As an example, you can check this:
http://gabriel.mp3-tech.org/work/comp_color_online/

(Slides are in french, but it should not be hard to grab the overall concept)

[Sharktooth]

Well the only remark i have for sagittaire is he used h.263 for low, mid and high bitrates.
Specific CQMs could be used to "help" PSNR and/or SSIM.
However there are settings and matrices that improves perceived quality but lower synthetic tests numbers...
VP6.2, for example, ralays a lot in its postprocessing "image reconstruction". On2 gives 2 "best" settings: Best PSNR - Best Quality.
That will make you think...
IMHO numbers give a great indication of a codec efficiency, but the final word is upon your eyes.
IMHO (and for my eyes) vp6 is not ABSOLUTELY better than xvid... but xvid is ABSOLUTELY better than divx (expecially in the near future when xvid 1.1 will see the light).
I prefer xvid to WMV9 even if tests say the contrary. But thats a personal preference... as I dont like RV10 and prefer VP6...

[virus]

Originally posted by SeeMoreDigital
And talking about Qpel. If every encoded pixel contains more detail because during it's analysis, it's been divided into quarters. I'll be damned if I know why it requires "switching on" during playback/decoding!
You're misunderstanding QPel a bit ;)

Pixels don't contain more detail, nor they get divided (you cannot divide a pixel, it's an "atomic" unit). Instead, visual info gets interpolated: from the original 16x16 block, you can derive a 64x64 block (four times bigger in each direction - hence the so called "quarter-pixel precision") based on the original smaller one. Then the encoder take its decisions based on the 64x64 interpolated version; thus, the decoder needs to do the same thing, in order to reconstruct properly the output.

(actually, I'm simplyfing things a bit. But I think you get the idea)

hope this makes things a bit clearer :)
virus

EDIT: there was an incorrect statement - now removed! sorry... :p

[Soulhunter]

@ Doom9

About the "codec shoot-out with more samples" idea...

I like the idea to use more different samples (and maybe even MPEG2 re/trans-coding) !?!

If you dont have enough time to do all this stuff, I would like to help you... :)

The only stuff I need is a detailed (fool proof) instruction for the setup !!!

-> And the sources of coz... ;)


Bye

[Audionut]

Originally posted by IgorC
Koepi
if you can do better than these samples it will proove your opinion
http://jfl1974.free.fr/Video/Sample-XviD-500.avi

Koepi is a busy man.
Your's are the 720x304 frames.
Test at 500kbps.

http://www.queenslander.net/users/lioneldavey/doom9/avi1.jpg
http://www.queenslander.net/users/lioneldavey/doom9/mkv1.jpg
http://www.queenslander.net/users/lioneldavey/doom9/avi2.jpg
http://www.queenslander.net/users/lioneldavey/doom9/mkv2.jpg

edit: also the bitrate does not drop below 550kbps in your source.

[Sharktooth]

Keep the same RESOLUTION and the same AVS plz.

[SeeMoreDigital]

Thanks virus...

This is what DivX has to say about Qpel: -

http://img86.exs.cx/img86/2646/What_is_Qpel_by_DivX.png


Cheers

[Sagittaire]

@ Audionut

Here my script ... crop left/right is not necessary ... with this source (Matrix Reloaded PAL).


Video=Mpeg2Source("G:\Mes dossiers\B.A\Matrix\azerty.d2v")
Video=Trim(Video,0,0)
Video=Crop(Video,0,76,-0,-76)
Video=Convolution3D(Video,0,3,4,3,4,3,0)
Video=BicubicResize(Video,720,304,0,0.5)
Return(Video)


My setting with XviD 1.0.1 (or XviD 1.0.2 if you want): 2 pass + Ultra high + VHQ4 + H263 Matrix + chroma motion + bframe 2/1.50/1.00 + GMC + QPel + trelli + adaptative quant + all quant 2-31 + Post-Process PP4

Bitrate for sample: 625 Kbps (extrat to complete encoding chapter 20 to 26 with 500 Kbps)

Capture is useless and doesn't prove nothing: perhabs RC for your sample is little different and quant for your frame localy higher. Perhabs your resize/pre-process is different. With sample I can check script and the setting used for XviD. PLEASE Post Sample with same bitrate and setting used


But here if you want my capture with PP4

my capture with PP4
http://jfl1974.free.fr/Video/Capture1.jpg

your capture in 704*288
http://www.queenslander.net/users/lioneldavey/doom9/mkv1.jpg

and not with PP0
http://www.queenslander.net/users/lioneldavey/doom9/avi1.jpg


my capture with PP4
http://jfl1974.free.fr/Video/Capture2.jpg

your capture in 704*288
http://www.queenslander.net/users/lioneldavey/doom9/mkv2.jpg

and not with PP0
http://www.queenslander.net/users/lioneldavey/doom9/avi2.jpg


You choose PP0 and perhabs custum matrix (I think really aggressive with high frequencies) ... it's your choice but make capture with bframe (not P or I frame) or in high motion scene ... :devil: