2004 codec shoot-out [Archive]

Sagittaire

16th October 2004, 22:34

Make codec comparison with capture is very useless:
- VP6 and "Shapeness": Sharpeness effect, conclusion for n frames isn't the same for n+1 frame
- DivX5, XviD, RV10, H264: P and Bframe difference, conclusion for n frames isn't the same for n+1 frame.
- For All codecs: Rate Control difference

http://jfl1974.free.fr/Test/Comparatif/XviDvsWMV9-2000.jpg

the only solution to compare visually codecs is to make samples with the same size. For me make comparison with capture is very useless.

Matrix II 720*304

Source DVD PAL 720*576 MPEG2 MP@ML, chapter 20 to 26, ~35 000 frames

Test in french (http://jfl1974.free.fr/HTM/32_Test_Codec_Video.htm) ... but Capture, Sample & Graphs are universal language

Codecs
- H264 Ateme
- XviD 1.0.1
- RV10 Elysean
- VP6 6.2
- DivX 5.2.1
- WMV9 VCM
- DivX 3 ffvfw

Tests
- 500-2000 Kbps and 25-100% compressibility (/quant2 XviD H263)
- SSIM, Average PSNR, Overall PSNR and Frames PSNR test
- Blind test
- Samples
- Diaporama

IgorC

17th October 2004, 00:02

If I'm not mistake Sagitariie has compared ffdshow x264 with other codecs before . And result wasn't bad at all. ffdshow x264's quality was between a little better than WMV 9, but worse than RV10.

And now ffdshow is in active development , just after beta testing of Ateme H.264 was over ;) . So it will be good to see ffdshow x264 in comparation test.

McoreD

17th October 2004, 00:26

Thanks for the link for your comparison test Saggitare.

Now you have very much convinced me that the following ranking is reasonable.

1- H264 Ateme
2- VP6 or RV10
3- WMV9 or XviD
4- DivX3 ffdshow

Cheers,
McoreD

dragongodz

17th October 2004, 01:47

i think vlada is saying he would like to see a new Doom9 codec comparison with these new codecs etc.

not much point really until they are really released though. :)

Koepi

17th October 2004, 04:11

LOL, yet another user is believing in Saggitaire's _scientifically incorrect_ comparisons.

What a waste.

Congrats though, this made my evening *rotfl*

Regards
Koepi

EDIT: removed potentially insulting "someone is dumb enough", wasn't meant insulting.

Sirber

17th October 2004, 04:26

@Koepi

Why are you mad at Sagitaire? Is it because XviD comes 3rd in his tests?

PiXuS

17th October 2004, 05:54

@Sirber

Just read past posts (http://forum.doom9.org/search.php?s=&action=finduser&userid=33350) from Sagitaire...

RadicalEd

17th October 2004, 07:53

Originally posted by PiXuS
@Sirber

Just read past posts (http://forum.doom9.org/search.php?s=&action=finduser&userid=33350) from Sagitaire...

17 pages of non-contextual posts isn't very helpful in determining why what should be an objective representation from the administration isn't. I'd like some more insightful details about this, because I haven't found flaw with Sag's tests. Largely because my own results are rather consistent with his and his testing methodology is pretty expansive.
So, anyone care to explain?

Koepi

17th October 2004, 09:55

Encode a movie to 1 or 2 cds, once with vp6, and once with xvid.

Compare the result. You'll be surprised (...how saggitaire can mess up with codec settings to make them produce worse output than default settings).

I mean, it's hard to ignore the "pumping" quality of VP6 which is very annoying for my taste.

@sirber:
nope, that's not the reason. The reason is that he doesn't use sane settings for the codecs. And then some background knowledge which you obviously don't have or already forgot about his scientifical background - but this doesn't belong in the public, I'd recommand a good google search on french video fora.

Regards
Koepi

Doom9

17th October 2004, 10:59

some of you might have noticed during this past week that I was not to be seen a lot.. in fact I was offline from Tuesday until Saturday. 8 hours from now I'll be offline until Saturday again, and the same thing goes for the week thereafter. Being away from any computer obviously doesn't go well with codec comparisons ;)

I think the end-of-the-year period is a pretty good one for codec comparisons.. it's cold so people stay inside and warm themselves up with high powered PCs ;)

Codecs included will be those that have been improved since the last time, so DivX, XviD, NeroDigital (the upcoming H.264 part), VP6 and RV10. Other codecs depend on a bunch of factors, like time available (WMV9 has been tested before and hasn't really been improved, perhaps I'll find a suitable libavcodec implementation and can get the proper suggestion as to settings), corporate policy decisions (I hope I'll be allowed to publish ND H.264 tests, and I'm sitting on some other unreleased beta software).

There are no plans for sources yet, but one thing is certain: I don't have time to do a reasonable DVD rip test which imho means at least 3 sources, and include other sources like DVB, DV and analogue captures. Do not forget that this is supposed to be a (non paying) hobby, and most of those asking for codec comparisons would quickly give up the endeavour when they realize how much effort is required.

Ghim

17th October 2004, 11:20

In order to do a proper Codec shoot-out, wouldn't it be better to try organising a proper subjective test session ?
There are already standardised protocols developped for video quality assessment like DSCQS or ACR methods (I would recommend DSCQS for mid-high bitrate codecs testing but ACR is very good to test more conditions in a short time).
Of course, there would be a need for at least 6 differents source videos (8-10 seconds).
Then, for each codec included, different bitrates could be tested to test the codecs behavior for different bitrates.

The problem would be to find enough subjects (at least 20) and to find the proper sources for the tests (VQEG sources could be used freely by example). It also represent a large work to produce all the data and prepare the tests but the result will probably be much more scientific than the codecs comparisons done in the past.

For reference, you can try going on www.vqeg.org and get some good documentations on video subjective testing.

Manao

17th October 2004, 11:55

Ghim : everwicked is trying to organize such a test. The hard part, for the moment, seems to get the majors / tv channels to allow him to use / distribute samples from their movies / shows.

And, if such a test allows to compare the raw performance of the codecs ( coding efficiency ), it doesn't say anything about the two pass mode - which need longer samples - which is also a foundamental part of a codec.

Ghim

17th October 2004, 12:13

Originally posted by Manao
Ghim : everwicked is trying to organize such a test. The hard part, for the moment, seems to get the majors / tv channels to allow him to use / distribute samples from their movies / shows.

And, if such a test allows to compare the raw performance of the codecs ( coding efficiency ), it doesn't say anything about the two pass mode - which need longer samples - which is also a foundamental part of a codec.

I knew that everwicked had developped a DSCQS software but I didn't know he was planning to organise a subjective test. It would be great if it was done. I know how hard it is to get content free to use/distribute and I don't know if he's going to be able to get them for free... There are already quite a few high quality contents used by VQEG that could be reused, even if these contents are not necessary very interesting.

For the 2-pass mode, I agree with you that 8-10 seconds is too short... But this kind of test is still much better than tests done by 1 person and not based on any standard protocol...
There is however a way to include a bit of the 2 pass testing in subjective tests... It would require to encode a very long sequence using the 2-pass mode and then cut 8-10 seconds blocks to create correct-size samples.
An other possible way to do it would be to append all the samples of the test in 1 long file, encode it and then cut the sequences before using them in the test.

Manao

17th October 2004, 12:17

It would require to encode a very long sequence using the 2-pass mode and then cut 8-10 seconds blocks to create correct-size samples. That wouldn't work because two pass algorithms would not distribute bits in the same way, some would favorize these 10 seconds, while other would not.An other possible way to do it would be to append all the samples of the test in 1 long file, encode it and then cut the sequences before using them in the test.That'd be better.

Sagittaire

17th October 2004, 12:50

Originally posted by Koepi
Encode a movie to 1 or 2 cds, once with vp6, and once with xvid.

Compare the result. You'll be surprised (...how saggitaire can mess up with codec settings to make them produce worse output than default settings).

I use the best setting for XviD and for other codec ... lol

XviD 1.0.1 setting with VDM: 2 pass + Ultra high + VHQ4 + H263 Matrix (good matrix visually, for me best matrix with PP4, best Matrix with metric) + chroma motion + bframe 2/1.50/1.00 + GMC + QPel + Adaptative Quant (psy mode better for eyes and for SSIM) + all quant 2-31 + Post-Process PP4

But if you think that setting aren't the best ... you could make encoding yourself ... source is very accessible. Matrix Reloaded Chapter 20 to 26 with this very usual and simple script:

Video=Mpeg2Source("G:\Mes dossiers\B.A\Matrix\azerty.d2v")
Video=Trim(Video,0,0)
Video=Crop(Video,0,76,-0,-76)
Video=Convolution3D(Video,0,3,4,3,4,3,0)
Video=BicubicResize(Video,720,304,0,0.5)
Return(Video)

My test use bitrate in ~q8 ~q2 interval for XviD with 23 min and 35 000 frames to Matrix Reloaded. All the possible type of ripps are tested:
- Streaming Quality -> 25% for compressibility (~q8)
- 1CDR or 2CDR Ripp -> 50% or 75% for compressibility (~q4 and ~q3)
- DVDR Ripp -> 100% for compressibility (~q2)

My samples are here:
DivX3 500 Kbps (http://jfl1974.free.fr/Video/Sample-DivX3-500.avi)
DivX5 500 Kbps (http://jfl1974.free.fr/Video/Sample-DivX5-500.avi)
XviD 500 Kbps (http://jfl1974.free.fr/Video/Sample-XviD-500.avi)
WMV9 500 Kbps (http://jfl1974.free.fr/Video/Sample-WMV9-500.avi)
RV10 500 Kbps (http://jfl1974.free.fr/Video/Sample-RV10-500.rmvb)
VP6 500 Kbps (http://jfl1974.free.fr/Video/Sample-VP6-500.avi)
H264 500 Kbps (http://jfl1974.free.fr/Video/Sample-H264-500.mp4)

DivX3 1000 Kbps (http://jfl1974.free.fr/Video/Sample-DivX3-1000.avi)
DivX5 1000 Kbps (http://jfl1974.free.fr/Video/Sample-DivX5-1000.avi)
XviD 1000 Kbps (http://jfl1974.free.fr/Video/Sample-XviD-1000.avi)
WMV9 1000 Kbps (http://jfl1974.free.fr/Video/Sample-WMV9-1000.avi)
RV10 1000 Kbps (http://jfl1974.free.fr/Video/Sample-RV10-1000.rmvb)
VP6 1000 Kbps (http://jfl1974.free.fr/Video/Sample-VP6-1000.avi)
H264 1000 Kbps (http://jfl1974.free.fr/Video/Sample-H264-1000.mp4)

If somebody (Koepi perhabs ...???) is able to make better sample with XviD (or other codec) with same script ... then it's very simple he can show their samples ...

Perhaps that Koepi thinks that XivD is better than H264 Ateme or VP6 or RV10 ... but not my eyes and metrics !!!

Originally posted by Doom9
Codecs included will be those that have been improved since the last time, so DivX, XviD, NeroDigital (the upcoming H.264 part), VP6 and RV10. Other codecs depend on a bunch of factors, like time available (WMV9 has been tested before and hasn't really been improved, perhaps I'll find a suitable libavcodec implementation and can get the proper suggestion as to settings), corporate policy decisions (I hope I'll be allowed to publish ND H.264 tests, and I'm sitting on some other unreleased beta software).

1) VP6, RV10, XviD 1.0.2, DivX 5.2.1, H264 Ateme
+ perhabs WMV9 Advanced Profil (new codec) with WM9Encodeur
- perhabs DivX 3.11

2) If possible not capture comparison but little sample comparison ... I repeat but for example VP6 PPP sequence can't be compared with XviD PBP in capture test

VP6 1000 Kbps
32000 1.0637 +0.0771 19 -22 43.3425 P
32001 1.0996 +0.1098 20 -22 43.1607 P
32002 0.9917 +0.0707 14 -15 44.2863 P Sharpeness?
32003 1.1154 +0.0983 20 -22 42.9298 P
32004 1.1319 +0.1286 18 -27 42.9571 P
32005 1.1748 +0.0597 19 -24 42.3986 P
32006 1.1912 +0.0714 26 -27 42.3078 P
32007 1.1160 +0.0881 17 -23 43.2668 P Sharpeness?
32008 1.2040 +0.0560 24 -26 42.0997 P
32009 1.1663 +0.0521 27 -23 42.3630 P
32010 1.2204 +0.0870 24 -26 41.7919 P

XviD 1000 Kbps
32000 1.0800 +0.0510 21 -19 43.2779 P
32001 1.3259 +0.0704 38 -43 40.5481 B
32002 1.1041 +0.0995 16 -19 43.2003 P
32003 1.3393 +0.0357 32 -34 40.4564 B
32004 1.1372 +0.1027 17 -29 43.0339 P
32005 1.3630 -0.0232 39 -38 40.3129 B
32006 1.1313 +0.0603 18 -22 43.0729 P
32007 1.4107 +0.0197 33 -40 40.1519 B
32008 1.1154 -0.0027 17 -22 43.1401 P
32009 1.3603 -0.0379 34 -43 40.2671 B
32010 1.0825 +0.0630 19 -26 43.2931 P

Very better frame for XviD:
32006(+0.8dB), 32008(+1dB), 32010(+2.5dB)

Very better frame for VP6:
32001(+2.5dB), 32002(+1dB), 32003(+2.5dB), 32005(+2dB), 32007(+3dB), 32009(+2dB)

DivX5 BPB sequence can't be compared with PBP XviD in shoot test. RV10 or H264 PBBBP can't be compared with PBBPB XviD in shoot test. Rate Control specification (H264, RV10, WMV9, XviD or DivX) can't be compared in capture test ...

dragongodz

17th October 2004, 13:06

well i happen to like Doom9's tests and find them an interesting read every time he does a new one. check them out some time if you havent.

using PSNR for results should be done with a great deal of caution aswell. it is not 100% reliable. rely on your own eyes before believing anything else.

in the end its up to each person to decide what they prefer so do some tests for yourself and then decide. :)

Sagittaire

17th October 2004, 13:30

Originally posted by dragongodz
using PSNR for results should be done with a great deal of caution aswell. it is not 100% reliable. rely on your own eyes before believing anything else.

PSNR can't show difference with 0.1 dB or 0.5 dB but can with 2 dB or 3 dB

Very better frame for XviD:
32006(+0.8dB), 32008(+1dB), 32010(+2.5dB)

Very better frame for VP6:
32001(+2.5dB), 32002(+1dB), 32003(+2.5dB), 32005(+2dB), 32007(+3dB), 32009(+2dB)

in this sequence make capture comparison is very useless and doesn't prove anything ... !!!

Koepi

17th October 2004, 13:32

Dragongodz made the proper point here.

Sagittaire, your settings _are_ messed up. AQ is meant for low bitrate scenarios. GMC will give a little better (subjective) quality, but shouldn't be used either - no hardware supports 3 warp point gmc.
Look in my previous post, I didn't say h264 is not superiour.

PSNR can tell you that _something_ changed. It doesn't mean better quality (remember XviD's TOO_SMALL_LIMIT: better PSNR, but visible quality was much degraded, the image was too smooth.). We had that very often, but please feel free to continue making a fool out of yourself.

Koepi

virus

17th October 2004, 14:10

Originally posted by Koepi
but please feel free to continue making a fool out of yourself.
I know I'm going to get a strike for that, because it's your usual way of dealing with those who don't share your views, but I don't care, being banned from Doom9 is better than being here as dummy, coward "yesmen" :-\

Sagittaire opened a thread to discuss his settings and used most of Teegedeck's suggestions for his tests. Quite in fact, his settings are very standard. My own results are pretty consistent with Sagittaire's (and RadicalEd said the same, if I understood him correctly).
Also, PSNR has been used for years by educated scientists around the world as a way of getting objective measurements (look at all the H.264 vs MPEG-4 ASP vs MPEG-2 graphs out there).

And I do believe that you're only criticizing because XviD didn't perform well on those tests, too. If XviD came up on 1st place, you'd never complain. Your harsh posts against Sirber prove that, too. IMHO you need to understand that there is life beyond XviD. You can go on insulting everyone which likes different codecs, you're a mod here so you can get away with it :-\

Still, at the cost of being banned from Doom9, I want to let my voice be heard, loud and clear: this is not right. Sagittaire did a great job and I hope he can go on despite the attacks. He gives objective measurements as well as visual impressions and sample clips from the sources. I can't imagine a better way to run a comparison.

Koepi, why don't you allow him to say that he likes VP6/RV10 more than XviD? Tell me why. It's Sagittaire's personal opinion. Could you call him a moron for expressing his personal opinions?

virus

Koepi

17th October 2004, 14:18

virus,

i don't quite get it where you have the impression from that i'm such a dick.

I just voice my oppinion, too. Of course sagittaire and anyone else is allowed to have his own oppinion.

Please re-read my last post - this time without that "what a dickhead" stereotype, and you'll see that I have a very valid point there.

I don't care if codec a or codec b or c wins any comparison. But if the base is flawed and not at all scientific but claims to be that (your example is wrong too; those scientific works you are talking about use the measurement PSNR on certain samples and usually talk about drawbacks, too. I don't see that Sagittaire somewhere mentions that he knows about the validity(?) of PSNR tests.) I want to be able to write that without such a weird "you'll strike me, you'll ban me, hehehe."

You certainly need to check the cases where I strike, you have mistaken something for sure.

Could we return on-topic please? Such things belong into PMs.

Koepi

Sagittaire

17th October 2004, 14:21

Originally posted by Koepi
Sagittaire, your settings _are_ messed up. AQ is meant for low bitrate scenarios. GMC will give a little better (subjective) quality, but shouldn't be used either - no hardware supports 3 warp point gmc.
Look in my previous post, I didn't say h264 is not superiour.

PSNR can tell you that _something_ changed. It doesn't mean better quality (remember XviD's TOO_SMALL_LIMIT: better PSNR, but visible quality was much degraded, the image was too smooth.). We had that very often, but please feel free to continue making a fool out of yourself.

My test isn't hardware compatibility test:
Qpel and GMC for XviD if it's better for quality. For high quality (q2 and q3) all codec are very good: it's very hard to see if X is better than Y: best possible setting for low/medium quality and psy mode (A.Q.) actived ...

My test isn't only Average PSNR test:
- Metric test: Average PSNR, Overall PSNR and the best metric for me SSIM. For example SSIM is able to detect visual subjectif gain from psy mode for H264, DivX5 and XviD ... try it Koepi
- Capture test
- blind test
- Sample in download

Ghim

17th October 2004, 14:25

I just want to continue on the objective measurement. I think people should stop using metrics as PSNR.
There are currently numerous new objective assessment models that are being normalised inside the ITU-T (see www.vqeg.org for more details and competition results) and that give much better results than the PSNR. The first published standard is J.144 for broadcast video quality measurement. The PSNR was used as a reference during the vqeg competition and even if it doesn't perform too bad, it's still quite far from being representative of the opinion scale. I'm not too sure either about the SSIM metrics and how it really compares to the subjective opinion.

IgorC

17th October 2004, 14:29

Koepi
if you can do better than these samples it will proove your opinion
http://jfl1974.free.fr/Video/Sample-XviD-500.avi
http://jfl1974.free.fr/Video/Sample-XviD-1000.avi

PiXuS

17th October 2004, 14:52

Originally posted by Sagittaire
I use the best setting for XviD and for other codec ... lol

XviD 1.0.1 setting with VDM: 2 pass + Ultra high + VHQ4 + H263 Matrix (good matrix visually, for me best matrix with PP4, best Matrix with metric) + chroma motion + bframe 2/1.50/1.00 + GMC + QPel + Adaptative Quant (psy mode better for eyes and for SSIM) + all quant 2-31 + Post-Process PP4

[snip]

XviD 1.0.2 is just a minor bugfix compared to 1.0.1. But still, why use 1.0.1 when 1.0.2 has been available for months now?

Also, you say you use the "best settings" for XviD. What's best for you? If I refer to your first post, I think you forced XviD into encoding to a target bitrate of 2000kbps, right? Why not to a target size or a target quant? Out of the three possible targets type you could use, I think you used the worst for XviD.

If you wanted best quality, you can't use H.263 matrix. Even the flawed PSNR measure will show you it isn't up to par with MPEG or even better matrices like SixOfNine (do you know about SixOfNine BTW?).

If you wanted the best quality, you wouldn't use b-frames. You wouldn't use AQ. You wouldn't post-process.

Strike 1.

So.... what are you aiming for? To me, when someones says "I want the best settings for XviD", I need to clarify the meaning. Best settings for a 1-CD encode? Best settings for a 2-CD.. 3-CD.. 1-DVD? For Anime material, for good quality Hollywood products, for shitty DVDs.....??? You didn't specify.

Strike 2.

And, no... I ain't going to tell you how to configure XviD properly (there is a TON of information about this subject here on forum.doom9.org and it is the actual point of this post).

PiXuS

virus

17th October 2004, 14:54

Originally posted by Koepi
i don't quite get it where you have the impression from that i'm such a dick.
I don't think you're "a dick", but I do think that you find difficult to accept other people's "truth" (which may differ from yours, of course).

But if the base is flawed and not at all scientific but claims to be that
Personally, I don't recall Sagittaire claiming that his results are "scientific" - maybe because I don't understand French very well. But I'd like to know what a person should do in order to run a "scientific" test, according to your definition. Every approach can be criticized.

I don't see that Sagittaire somewhere mentions that he knows about the validity(?) of PSNR tests.
True, but there are a lot of tech papers out there where the authors only give PSNR results without further comments. It's because they assume that the reader already knows about the drawbacks of objective measurements. Anyway Sagittaire provides sample clips - and this for sure shows that he understands their importance, too.

You certainly need to check the cases where I strike, you have mistaken something for sure.
Maybe, because the striker's name is not always public, but still I think I've seen enough, in the last 10 months here; otherwise I'd have kept my mouth shut.

Could we return on-topic please?
I think we're perfectly on topic. We are talking about codec comparisons here, and Sagittaire runs codec comparisons (doing the best he can do - a single person cannot run a complete blind test, unfortunately).

virus

dragongodz

17th October 2004, 15:19

when requant was being constantly worked on a change was made that increased PSNR. it was released and people complained. why ? because it actually looked worse. yes PSNR increased but visually it went down. if you want proof go and find it in that mammoth Rejig thread.
so again i say, believe your own eyes first.

Qpel has also been shown to sometimes lower PSNR by the way but i guess these things dont matter since they are suposed to be "the best" settings.

wasnt there an occasional problem with VHQ4 and B frames occasionally smearing detail at 1 point ? probably fixed but which version, i cant remember. Koepi should know what i am talking about. :)

Sagittaire - what are you trying to say by shoot test by the way ???

i don't quite get it where you have the impression from that i'm such a dick.
Koepi please dont give me lines like that, i may be tempted to answer it. :D

stephanV

17th October 2004, 15:29

Originally posted by PiXuS
Also, you say you use the "best settings" for XviD. What's best for you? If I refer to your first post, I think you forced XviD into encoding to a target bitrate of 2000kbps, right? Why not to a target size or a target quant? Out of the three possible targets type you could use, I think you used the worst for XviD.
LOL... whats the principle difference between a encode which uses a target bitrate or a target filesize? answer: none. A target quant? why? that wouldnt be fair to the other codecs.

If you wanted best quality, you can't use H.263 matrix. Even the flawed PSNR measure will show you it isn't up to par with MPEG or even better matrices like SixOfNine (do you know about SixOfNine BTW?).
There is exactly no hard evidence that any custom quant matrix is better than H.263... at best you could say they perform better at different ranges.

If you wanted the best quality, you wouldn't use b-frames.
Right... b-frames really stink... errrrr... NOT!

So.... what are you aiming for? To me, when someones says "I want the best settings for XviD", I need to clarify the meaning. Best settings for a 1-CD encode? Best settings for a 2-CD.. 3-CD.. 1-DVD? For Anime material, for good quality Hollywood products, for shitty DVDs.....??? You didn't specify.

He does not have to specify it, i think he chose what he found what worked best on that source (or he took advice as it seems). Targeting a certain filesize is not always a good idea as you have to account for audio too then. Some prefer too keep AC3, other want it MP3 at this and that bitrate and so on... It is allowed to ignore such parameters for testing i think. After all we're testing quality here, not someones need to pack a movie on a CD.

Don't get me wrong... the only part i can appreciate about saggitaire's test are his samples (which is the ONLY way to make a good comparison). The metrics and his blind test (blind test? what? where? how?) are completely worthless IMO.

Sagittaire

17th October 2004, 15:30

Originally posted by PiXuS
XviD 1.0.2 is just a minor bugfix compared to 1.0.1. But still, why use 1.0.1 when 1.0.2 has been available for months now?

XviD 1.0.2 is just a very little decoder bugfix

Also, you say you use the "best settings" for XviD. What's best for you? If I refer to your first post, I think you forced XviD into encoding to a target bitrate of 2000kbps, right? Why not to a target size or a target quant? Out of the three possible targets type you could use, I think you used the worst for XviD.

With Matrix reloaded chapter 20 to 26 and with my avs script (in 720*304) bitrate for q2 encoding is 1970 Kbit/s. I use 2000 Kbit/s for 100% compressibility, 1500 Kbit/s for 75%, 1000 Kbit/s for 50% and 500 Kbit/s for 25% in my test for all codec. To simplified 100% is ~q2 encoding, 75% ~q3, 50% ~q4 and 25% ~q8 for average I,P frame with XviD ...

If you wanted best quality, you can't use H.263 matrix. Even the flawed PSNR measure will show you it isn't up to par with MPEG or even better matrices like SixOfNine (do you know about SixOfNine BTW?).

H263 is a very good HVS quantisation (see Soulhunter Test). For me HVS-best is the best matrix with PP0 (less blocking but more ringing) but with PP4 for me the best quantisation is H263 and by far (PP4 is necessary with quant higher than 4)

Strike 1 ... lol

So.... what are you aiming for? To me, when someones says "I want the best settings for XviD", I need to clarify the meaning. Best settings for a 1-CD encode? Best settings for a 2-CD.. 3-CD.. 1-DVD? For Anime material, for good quality Hollywood products, for shitty DVDs.....??? You didn't specify.

My test use bitrate in ~q8 ~q2 interval for XviD with 23 min and 35 000 frames to Matrix Reloaded. All the possible type of ripps are tested:
- Streaming Quality -> 25% for compressibility (~q8)
- 1CDR or 2CDR Ripp -> 50% or 75% for compressibility (~q4 and ~q3)
- DVDR Ripp -> 100% for compressibility (~q2)

For animate H264 and RV10 are very very very better than MPEG4 ASP

Strike 2 ... lol

If you wanted the best quality, you wouldn't use b-frames. You wouldn't use AQ. You wouldn't post-process.

No bframe with XviD ... !!?
No A.Q. ... it's a very good HVS function ... !!?
No Post-Process ... with ~q4 or ~q8 encoding ... !!?

No Comment: strike 3 ... lol

If you want i can make my test with "your best setting" ...
XviD 1.0.2 setting with VDM: 2 pass + Ultra high + VHQ4 + SixOfNine Matrix + chroma motion + no bframe + GMC + QPel + no Post-Process PP0

... but DivX5 must be really better than XviD in this case ... lol

Rrrough

17th October 2004, 15:32

by Koepi: Removed all offtopic stuff from the post. Unfortunately that means there's nothign left.

by Rrrough:
Well, virus was right, you can get a strike for criticizing insulting posts from moderators, even if it is off-topic.
ON-TOPIC again, I think it's great when someone is working hard for the community doing
CODEC COMPARISONS, even if you don't agree with its scientific base.
Everyone is up to himself to draw conclusions out of it, so no need for insulting such a work. Everyone has the possibilty to conduct a more scientific comparison and present it here.
Thank you Sagittaire for your work.

SeeMoreDigital

18th October 2004, 13:16

Personally, I would like to see some comp tests generated at 1:1, instead of cropped and resized.

That said I have nothing against cropping the mattes away, as long as the vertical resolution is preserved (or as near as damn) to the 16th pixel.

Don't get me wrong, resizing encodes has its place and is very useful. But in my opinion it does give an accurate representation of a codecs true performance characteristics!

But I guess I am a majority of one... again?

Cheers

babayaga

18th October 2004, 13:20

Originally posted by Doom9

There are no plans for sources yet, but one thing is certain: I don't have time to do a reasonable DVD rip test which imho means at least 3 sources, and include other sources like DVB, DV and analogue captures. Do not forget that this is supposed to be a (non paying) hobby, and most of those asking for codec comparisons would quickly give up the endeavour when they realize how much effort is required.

Why don't you simply request companies/groups to do the encode themself (it's very time consuming for you I guess) ??
It should not be so unfair since in your previous tests, the encode config was given by the codec authors.

virus

18th October 2004, 14:21

Originally posted by Koepi
You certainly need to check the cases where I strike, you have mistaken something for sure.
Yeah, after a quick review, especially at Rrrough's post, I did mistake something indeed. I said the strike is your usual way of dealing with those who don't share your views and this is not entirely true.

I simply forgot to add the other way: deleting criticism to your (bad, imho) behaviour, labelling it as "off-topic". Why don't you edit your own posts where you insulted Sagittaire without a reason instead? Why don't you avoid calling people "dumb" because they read Sagittaire's comparisons? He does this job for us all, and he does it for free. How can you overstep the forum rules this way and delete people's complaints? Is this a fair behaviour?

Again, I want to show all my support to Sagittaire and please, Sagittaire, keep on with it, it's very appreciated.

virus

Trahald

18th October 2004, 14:27

Originally posted by dragongodz
when requant was being constantly worked on a change was made that increased PSNR. it was released and people complained. why ? because it actually looked worse. yes PSNR increased but visually it went down. if you want proof go and find it in that mammoth Rejig thread.
so again i say, believe your own eyes first.

Agreed. since a good part of compression benefit (mpeg2/4/etc) is based on psychovisual tricks (example - using lower bitrate in dark are next to a lighted area.. eyes have a hard time seing the detail in the dark area in that case, why waste bits on it) . most codecs have options to crank up the tricks but even with the options off, most/all lossy codecs are about psychovisual tricks. PSNR does not allow for that. it just says.. does this picture look like the origional. if not.. how much different. this may lead to a superior visually picture getting a worse score than another that scores better. psnr != human eyes.

and the point is.. if you want to make an opinion about what codec is the best. great! wmv9 looks better to you.. excellent. its just that taking it from opinion to fact with a test that only suitable if the movie is viewed by a robot is the problem area.

virus

18th October 2004, 14:56

Originally posted by Trahald
Agreed. since a good part of compression benefit (mpeg2/4/etc) is based on psychovisual tricks (example - using lower bitrate in dark are next to a lighted area.. eyes have a hard time seing the detail in the dark area in that case, why waste bits on it)
This is an oversimplification, I think.
Most of the compression efficiency does not come from psychovisual tricks, but from advanced encoding tools, like a better entropy coder (e.g. CABAC), a more precise motion estimation (e.g. QPel) and a number of similar state-of-the-art techniques.

And the performance of these advanced encoding tools can be measured by synthetic benchmarks, too (bitrate reduction @ fixed quantizer, or PSNR/SSIM increase). Of course other compression tools "speculate" on the HVS trying to "fool" the eye, while gaining compression efficiency: these are not well suited for objective measurements. So it's a bit of a mix between "objectively measurable" and "subjectively measurable" stuff.

Personally, I prefer to have both synthetic scores (which are just a cheap approximation of subjective ones - but hey, not everyone can organize a blind test with 2000 people everyday ;)) and actual samples that I can watch on my own. So I can compare and see under what situations/settings/sources PSNR/SSIM is a good indicator (and what PSNR/SSIM margin is needed to make a difference) and when instead it doesn't reflect my subjective opinion.

Sometimes and for some codecs/settings/bitrates PSNR works OK, other times it fails. Still, it's a very useful tool, provided that you know how to deal with it.

Sagittaire

18th October 2004, 15:03

Once again it's just a small codec test:
- Metric Test
- Capture Test
- Blind Test
- Sample Test

if you don't like the metric test, read the blind test
if you don't like my blind test, read the capture test
if you don't like my capture test, download my sample
if you don't like my sample, make sample yourself

if you think that the codec X can be better with your setting ... make test and post your sample (I wait your sample Koepi).

It's not my fault if codec X is better than codec Y with metric. It's very simple to make metric test (even for "fool" like me ... lol): I read and report result in graph. If you think that Average PSNR, Overall PSNR or SSIM aren't good for codec comparison, it's not my fault. I use metric test because metric test are used by the developers of codec for developping but too, to compare with the other codec (Babayaga, Bobololo, One2Tech, C0mPr355
, karl_lillevold, Digital56K, Gej and certainely sysKin ... )

example - using lower bitrate in dark are next to a lighted area.. eyes have a hard time seing the detail in the dark area in that case, why waste bits on it

Yes it's true ... PSNR can't detect that but SSIM can: SSIM is better with XviD and DivX5 psy mode. SSIM is better with H264 psy mode too ...

and the point is.. if you want to make an opinion about what codec is the best. great! wmv9 looks better to you.. excellent. its just that taking it from opinion to fact with a test that only suitable if the movie is viewed by a robot is the problem area.

How much looks at the WMV9 with the best Pos-Process ... perhabs 10%. WMV9 use deblock and dering with highest level by default.
http://jfl1974.free.fr/Video/WMVPostpross.exe

But for SSIM WMV9 is just better than XviD for low/medium bitrate (~q8 ~q4 interval in my test)

http://jfl1974.free.fr/Test/Comparatif/SSIM.gif

Tommy Carrot

18th October 2004, 15:08

I don't want to participate in this debate, just i share that view that the synthetic quality measuring tools are hardly reliable for practical use. There are too many cases when they are contradicting to the reality. Qpel increases the perceived quality, but usually degrades the PSNR result. Or the famous example: if you increase the brightness a bit, you won't notice the difference, but the PSNR result will be very low.

SeeMoreDigital

18th October 2004, 15:50

Originally posted by Tommy Carrot
I don't want to participate in this debate, just i share that view that the synthetic quality measuring tools are hardly reliable for practical use. There are too many cases when they are contradicting to the reality. Qpel increases the perceived quality, but usually degrades the PSNR result. Or the famous example: if you increase the brightness a bit, you won't notice the difference, but the PSNR result will be very low. Agreed!

And talking about Qpel. If every encoded pixel contains more detail because during it's analysis, it's been divided into quarters. I'll be damned if I know why it requires "switching on" during playback/decoding!

Cheers

Sirber

18th October 2004, 15:52

In my tests (not scientifical), at 500kbps, H264 and RV10 are very simillar. VP6 comes second with some artefacts.

Note: All that is IMHO :D

Gabriel_Bouvigne

18th October 2004, 16:00

Just my small personnal (an biased) opinion on synthetic tests:

They have a big interest during codec developement. In audio we (at least me and Ivan) are using them during developement as a quick tools to give rough estimations. Usually, they give some right indications for changes made into a codec.

But this is only usually. In no way they are the absolute truth. They can give totally wrong results, especially when comparing different encoders (in the audio case, at least).

I strongly believe that PSNR is far from beeing good, in an absolute way, as encoders can try to fool the HVS. But on the other hand, current video encoder do not rely that much on psychovisuals, at least when compared to audio encoders using psychoacoustics.

On audio encoders we are using tools like PEAQ, which are trying to take into consideration human audition, instead of pure raw differences like psnr. But given today's state of video encoders regarding psychovisuals, I thing that using psnr, even if not optimal, is still a good indication.
You just have to know how it works, and specially its limitations.

I hope that one day video compression will have evolved enough so that psnr will be a totally useless tool, as in audio.

In the meantime, you can still have a quick rough estimation with psnr.

Btw, there are some other tools that can discern changes in a more subtle way than psnr, which is just telling you if there is a difference, but not in which way.
As an example, you can check this:
http://gabriel.mp3-tech.org/work/comp_color_online/

(Slides are in french, but it should not be hard to grab the overall concept)

Sharktooth

18th October 2004, 16:24

Well the only remark i have for sagittaire is he used h.263 for low, mid and high bitrates.
Specific CQMs could be used to "help" PSNR and/or SSIM.
However there are settings and matrices that improves perceived quality but lower synthetic tests numbers...
VP6.2, for example, ralays a lot in its postprocessing "image reconstruction". On2 gives 2 "best" settings: Best PSNR - Best Quality.
That will make you think...
IMHO numbers give a great indication of a codec efficiency, but the final word is upon your eyes.
IMHO (and for my eyes) vp6 is not ABSOLUTELY better than xvid... but xvid is ABSOLUTELY better than divx (expecially in the near future when xvid 1.1 will see the light).
I prefer xvid to WMV9 even if tests say the contrary. But thats a personal preference... as I dont like RV10 and prefer VP6...

virus

18th October 2004, 16:30

Originally posted by SeeMoreDigital
And talking about Qpel. If every encoded pixel contains more detail because during it's analysis, it's been divided into quarters. I'll be damned if I know why it requires "switching on" during playback/decoding!
You're misunderstanding QPel a bit ;)

Pixels don't contain more detail, nor they get divided (you cannot divide a pixel, it's an "atomic" unit). Instead, visual info gets interpolated: from the original 16x16 block, you can derive a 64x64 block (four times bigger in each direction - hence the so called "quarter-pixel precision") based on the original smaller one. Then the encoder take its decisions based on the 64x64 interpolated version; thus, the decoder needs to do the same thing, in order to reconstruct properly the output.

(actually, I'm simplyfing things a bit. But I think you get the idea)

hope this makes things a bit clearer :)
virus

EDIT: there was an incorrect statement - now removed! sorry... :p

Soulhunter

18th October 2004, 17:23

@ Doom9

About the "codec shoot-out with more samples" idea...

I like the idea to use more different samples (and maybe even MPEG2 re/trans-coding) !?!

If you dont have enough time to do all this stuff, I would like to help you... :)

The only stuff I need is a detailed (fool proof) instruction for the setup !!!

-> And the sources of coz... ;)

Bye

Audionut

18th October 2004, 17:39

Originally posted by IgorC
Koepi
if you can do better than these samples it will proove your opinion
http://jfl1974.free.fr/Video/Sample-XviD-500.avi

Koepi is a busy man.
Your's are the 720x304 frames.
Test at 500kbps.

http://www.queenslander.net/users/lioneldavey/doom9/avi1.jpg
http://www.queenslander.net/users/lioneldavey/doom9/mkv1.jpg
http://www.queenslander.net/users/lioneldavey/doom9/avi2.jpg
http://www.queenslander.net/users/lioneldavey/doom9/mkv2.jpg

edit: also the bitrate does not drop below 550kbps in your source.

Sharktooth

18th October 2004, 18:01

Keep the same RESOLUTION and the same AVS plz.

SeeMoreDigital

18th October 2004, 19:05

Thanks virus...

This is what DivX has to say about Qpel: -

http://img86.exs.cx/img86/2646/What_is_Qpel_by_DivX.png

Cheers

Sagittaire

18th October 2004, 19:20

@ Audionut

Here my script ... crop left/right is not necessary ... with this source (Matrix Reloaded PAL).

Video=Mpeg2Source("G:\Mes dossiers\B.A\Matrix\azerty.d2v")
Video=Trim(Video,0,0)
Video=Crop(Video,0,76,-0,-76)
Video=Convolution3D(Video,0,3,4,3,4,3,0)
Video=BicubicResize(Video,720,304,0,0.5)
Return(Video)

My setting with XviD 1.0.1 (or XviD 1.0.2 if you want): 2 pass + Ultra high + VHQ4 + H263 Matrix + chroma motion + bframe 2/1.50/1.00 + GMC + QPel + trelli + adaptative quant + all quant 2-31 + Post-Process PP4

Bitrate for sample: 625 Kbps (extrat to complete encoding chapter 20 to 26 with 500 Kbps)

Capture is useless and doesn't prove nothing: perhabs RC for your sample is little different and quant for your frame localy higher. Perhabs your resize/pre-process is different. With sample I can check script and the setting used for XviD. PLEASE Post Sample with same bitrate and setting used

But here if you want my capture with PP4

my capture with PP4
http://jfl1974.free.fr/Video/Capture1.jpg

your capture in 704*288
http://www.queenslander.net/users/lioneldavey/doom9/mkv1.jpg

and not with PP0
http://www.queenslander.net/users/lioneldavey/doom9/avi1.jpg

my capture with PP4
http://jfl1974.free.fr/Video/Capture2.jpg

your capture in 704*288
http://www.queenslander.net/users/lioneldavey/doom9/mkv2.jpg

and not with PP0
http://www.queenslander.net/users/lioneldavey/doom9/avi2.jpg

You choose PP0 and perhabs custum matrix (I think really aggressive with high frequencies) ... it's your choice but make capture with bframe (not P or I frame) or in high motion scene ... :devil:

bobololo

18th October 2004, 20:47

Hehe I'm sorry, this won't probably help to get back to the original topic but I couldn't resist to quote these statments that GomGom expressed on #xvid while I was reading this thread :)

<GomGom> yo
-:- SignOff war-: #xvid (Remote closed the connection)
-:- war- [war@209.81.41.149] has joined #xvid
<GomGom> raaaa koepi is not there
<GomGom> he's put online a build with a shitty RC... fuck
<GomGom> i told him not to do so
<GomGom> but it always give inferior PSNR
<GomGom> and lot of time much worse worst PSNR

Obviously this shouldn't be taken seriously, it was just for relaxing the atmosphere :)

Koepi

18th October 2004, 21:02

In sharktooth tests, the PSNR and SSIM of strict scaling always was higher than with loose scaling... or did i misread something? Anyhow, as we mention that here - those values don't mean too much. I'll ask GomGom to do some tests with other samples, with this build everything seems to be fine.

Enough for off-topic now, let's discuss that over at the xvid thread.

Regards
Koepi

Ghim

18th October 2004, 22:24

Originally posted by Gabriel_Bouvigne
On audio encoders we are using tools like PEAQ, which are trying to take into consideration human audition, instead of pure raw differences like psnr. But given today's state of video encoders regarding psychovisuals, I thing that using psnr, even if not optimal, is still a good indication.
You just have to know how it works, and specially its limitations.

There are already subjective models like PEAQ (I personnaly think PEAQ is quite average for a model but that's not the point here) but for video. I mentionned it earlier in my previous posts (the J.144 models are a good example) and the concerned models really give good results... but nobody here seems to take interest in this kind of tool. I don't know if it's possible to get those models easily, but at least it shows it's possible...

I'm really not convinced by the SSIM metric and I've never found any information proving its performances... If anyone has a link with some good information on SSIM results, I'm interested...

Sagittaire

18th October 2004, 22:41

Obviously this shouldn't be taken seriously, it was just for relaxing the atmosphere

To make peace with Koepi:

I prefer a little worse Average PSNR (-0.1 dB or -0.2dB) if Min PSNR is very higher (+ 1dB). Average PSNR isn't really good tools for testing Rate Control quality ... ;)

mgh

1st November 2004, 18:41

From my experiments xvid is good only at high bit rates. I tried a fairly static clip at 384x288 showing a water pitcher at 300 kbps-
results-
xvid heavy blockiness
divx-jagged edges for the pitcher
while both vp6 and RV10 represented the pitcher perfectly.
My conclusion-with due respect to all gurus here-best codec depends on clip chosen-resolution-and kbps chosen for testing.
In short-tests for best codecs are meaningless-and good only for adding fuel to the fire on forums!!:D
Anyway, that's my humble opinion.

Doom9

1st November 2004, 18:52

@Soulhunter, babayaga: good idea about outside help except for a few killer drawbacks: first of all, I often get my hands on codecs that are not to be publicly released.. I can't just share those, and there are very few people I trust with software I'm not supposed to give out.

Then, the trust issue comes up again when it comes to sticking to settings and knowing the tests were done the way I want it done.. since it's published under my name I need to be sure and triple check that everything was done properly.. and I can't be sure of that unless a very good friend or myself did it.

Last but not least, the people encoding would have to offer GBs of encoded video for me to download.. while many people these days have pretty decent downlinks, uplink is another story entirely.

SeeMoreDigital

1st November 2004, 19:41

Doom9,

How proactive are some of these codec manufactures at dealing with requests?

Okay, I can't imagine the likes of Micro$oft would be willing to generate a full movie encode from a "reference DVD" or even the odd chapter or two but it would be nice to think the guys at DivX, 3ivx, XviD, VP6, Real, Ateme etc would. Especially as many of their representatives are active on this forum.

Cheers

Doom9

1st November 2004, 22:08

Okay, I can't imagine the likes of Micro$oft would be willing to generate a full movie encode from a "reference DVD" or even the odd chapter or two but it would be nice to think the guys at DivX, 3ivx, XviD, VP6, Real, Ateme etc would. Especially as many of their representatives are active on this forum.I had this option for two codecs last time, but I had to decline. How can I be sure that whatever encoded material I'm getting is encoded the way I want it to be, and that there's no foulplay anywhere along the way?

Only when I encode on my own do I know that the results are beyond any doubts.

SeeMoreDigital

1st November 2004, 23:32

Originally posted by Doom9
I had this option for two codecs last time, but I had to decline. How can I be sure that whatever encoded material I'm getting is encoded the way I want it to be, and that there's no foulplay anywhere along the way?

Only when I encode on my own do I know that the results are beyond any doubts. I guess you would have to outline what sort of "foul play" you're talking about.

That said, if you asked some of these companies to generate encodes (from a nominated source) to a certain file size or at a certain bitrate and stated what bells and whistles you wanted then to use... I can't see the harm in asking them to do this!

Or why not say to some of these companies... Okay, I'd like you to encode "this" 96 minute movie, with audio at "96Kbps" using "xxx by yyy" pixels and stick it onto a 702MB CD~R. Then post it to me with details of how you did it?

At the end of the day it would still be up to you to decide if what they've done is above board.

Just a thought

Soulhunter

1st November 2004, 23:48

Originally posted by SeeMoreDigital

That said, if you asked some of these companies to generate encodes (from a nominated source) to a certain file size or at a certain bitrate and stated what bells and whistles you wanted then to use... I can't see the harm in asking them to do this!

Maybe different/stronger filtering, or a "source tweaked" encoder !!!

Who knows... :rolleyes:

Bye

SeeMoreDigital

2nd November 2004, 00:34

Originally posted by Soulhunter
Maybe different/stronger filtering, or a "source tweaked" encoder !!! Maybe...

If filters are going to be used, then lets know about them... so we can all use them... As for tweaked encoders... lets have these too!

Cheers

Soulhunter

2nd November 2004, 01:00

With "source tweaked" encoders I meant...

Tweaked to give better results @ this specific source !!!

Could mean that you get worse results with a different source... ;)

Bye

SeeMoreDigital

2nd November 2004, 01:21

Originally posted by Soulhunter
With "source tweaked" encoders I meant...

Tweaked to give better results @ this specific source !!! I see...

At least by asking the codec developers to generate and submit their own "comp test" encodes, Doom9 could use them as a base line... Meaning, if he can't generate his own encodes to come anywhere close to theirs, he would be able to ask them why... and report such.

There has to be an element of trust. And should that trust be broken, the codec manufacturer should deserve to be exposed as cheats!

Cheers

Doom9

2nd November 2004, 08:19

Meaning, if he can't generate his own encodes to come anywhere close to theirs, he would be able to ask them why... and report such.That would mean I'd have to encode everything on my own as well.. there goes the time advantage. Plus a limited amount of filtering might not be detectable, thus offering a lot of possibilities for foul play. And, of codec makers do the encoding, you can forget about reproducability.. who says I got all the settings that matter?

I see this idea the same way as having a CPU maker having his own benchmarks published in a CPU test, rather than have the site that publishes the article do their own tests.. there should be a good amount of scepticism towards sources that are clearly biased. I wouldn't trust results that I cannot reproduce - something which was drilled into me at college.. in science unless you can reproduce it, there's no reason to believe in it.

SeeMoreDigital

2nd November 2004, 10:09

While I understand what you're saying, I think it's a shame you don't trust (some) of the codec manufacturers and ask them to get more involved.

Like I say, at the end of the day they will only have themselves to blame if they cheat...

Or... how about this one. Why not start a new "Codec Shoot-Out" solely for the manufacturers to get involved in... a sort of "Put-up or Shut-up" shoot-out!

Still give them boundaries to work within (similar to the ones I mentioned before) but let them use any method of encoding available to them... It might prove rather interesting all the same!

Cheers anyway.

Sagittaire

2nd November 2004, 12:21

Tweaked to give better results @ this specific source !!!

For example with DivX 5.2.1:

The best mode with VFW interface is -pq 64 (slow mode) but with register tweak the best mode is -pq 255 (slowest mode) ...

For better result DivX EKG could be used ...

SeeMoreDigital

2nd November 2004, 13:03

I wonder how many people have actually used DivX's EKG application on a full movie encode?

I dare say they might have tried it a couple of times and abandoned the tool ;)

Cheers

Doom9

2nd November 2004, 13:06

Like I say, at the end of the day they will only have themselves to blame if they cheat...How can you possibly prove cheating? It might not be that apparent. It is rather naive to think that given the opportunity to make your own product look better that nobody would actually do it. And your 2nd idea once again comes to "no reproducibility". What good is a test if you cannot reproduce the results? Since it's my name under the test, I'd rather have nothing than something I consider to be unworthy of publishing.

And I'm sure MS would go on record saying they ripped that and that movie and then mailed it to a 3rd party they haven't even met.

Last but not least.. try making a comparison with 10 codecs and 8 sources.. only the comparing part will make you want to run away screaming.. and it's the part where you absolutely must not screw up. And then the screenshot taking part.. I suggest you try it once.. it makes you want to kill yourself.. it really sucks beyond imagination.. go to in MPC sure is nice, but not every codec reacts the same, so you end up with image-flicking to get the right shot and that's definitely not my favorite past time.

Sagittaire

2nd November 2004, 13:55

Good subjectif test for me:

Full movie encode:
- easy encoding (90 min in 1 CDR and low quant ~q2-3 with XviD for example)
- medium encoding (120 min in 1 CDR and medium quant ~q4 with XviD for example)
- hard encoding (150 min in 1 CDR and high quant ~q6-8 with XviD for example)

Littles samples (30 sec for example): Capture are useless and doesn't prove anything ...

With these codec:
+ RV10
+ VP6
+ WMV9
+ MPEG4 ASP (XviD and DivX5)
+ MPEG4 AVC (H264 Ateme)
- DivX 3.11 (useless for me), x264 and SNOW (in alpha build) ...

-> 3*6 = 18 full encoding ... :devil:

plonk420

14th November 2004, 06:26

how about a "double-blind" abx test a la HA? you can beat on these differences that some people can see and not others all day, but in the end shouldn't it be about sample1a vs sample1b .. and which one just plain looks better and why (with all the switch setting madness being hidden behind the scenes) .. and the average of all the responses where no one's vote counts for more than anyone else's?

*shrug*

Nil Einne

15th November 2004, 17:17

Would one option be to give the source to the codec manufacturers and the bitrate and tell them to work outhow they want you to encode it. Then they send you back the settings or perhaps even a AVS script or whatever with the settings. Of course, you will still need to do all the encoding yourself but it will only be a minimal amount of your time that you have to spend. You just load up the all the scripts and start compressing. It'll still take a lot of your computer time, unless you have a load of computers but it'll probably be a lot easier then trying to decide for yourself what settings to use with the various bitrates and sources.

The trouble with double blind tests is to be really reliable for the general population you a large sample of different people. Doing it online is not really that reliable since you tend to get a small sample size, you can't be sure how often a person votes and you usually get a skewed population. Specifically, the people who take part are likely to be people with a strong interest in these matters. It is easily possible, perhaps likely that people have grown adjusted to the natural 'flaws' that on codec generates but they are maybe no to used to the flaws of another codec and therefore consider it poorer quality even though the general population would not*. Also, what do you want to test? There is a good chance a number of people are going to be viewing it close up on their monitors. This is fine if this is what you want to test but if you want to test it at a distance on larger TVs, the results are probably going to be different.

*Off topic but this is frequently a problem IMHO with consumer double blind tests. For example, audiophiles frequently claim a extremely high end hifi with their extremely expensive speakers are better then some cheaper stuff. And they and maybe also the general population can tell the difference between the two. The audiophiles in a double blind test may rate the expensive stuff better sounding but if the general population does not, then it's really not relevant to the general population. Of course, if you want to be an audiophile and decide the expensive stuff is better that's up to you, but the vast majority of people do not. Of course, if you're not an audiophile you're not likely to 'waste' the money. I suppose what I'm trying to say is that it's important to remember just because a segment of the population think something is better, doesn't necessarily mean it is better. The 'uneducated' who either don't notice the different or more likely if they pay enough attention do notice a difference but can't decide which one is better aren't wrong or stupid.

plonk420

15th November 2004, 22:58

i'm sure there's a HUGE interest in a shoot out, but now that i think about it, i'm not so sure as to how many would be interested in *doing* the double blind test...

i for one would be glad to if the software is availible (a la ABCHR) and ... bug free, at least to the point of stable funtionality. i could test on a Flat CRT, LCD, and with an extra effort, a 27" TV. the only problem is that the LCD is on my PowerMac.

Joe Fenton

16th November 2004, 00:21

The problem I see with double-blind tests on video is in the amount of data to download and compare. If you use 30 sec clips, that limits how much the person has to download and watch, but really screws up the test since short clips don't give you a real sense of how the codec works on longer encodes.

However, (almost) no one is going to download dozens of hour long encodes and watch then all with an eye towards comparing quality. Most the people I know would lose interest after the second viewing. ;)

"Let's see... we now move on to our eighth codec. Two more hours of "Ishtar" in H.264 format! Don't forget to mark how you think it compares to RV10, which you saw the day before yesterday." :D

Doom9

16th November 2004, 09:03

"Let's see... we now move on to our eighth codec. Two more hours of "Ishtar" in H.264 format! Don't forget to mark how you think it compares to RV10, which you saw the day before yesterday." ;)

I also tend to think that comparing video is more complex than comparing audio because there are more dimensions.

But the main problem is size (you know how big a losslessly compressed video is) and copyright.. can you honestly imagine a movie studio giving anyone permission to offer 3 chapters of one of their hit titles for download in 10 different formats?

pogo stick

16th November 2004, 09:56

So it will be almost impossible to organize double-blind tests using internet. But how about making such test as a scientific project in some University or College? It would be also very hard task, but I am sure that there will be some volunteer. When I was a student I was ready to test almost anything for couple of beer. :)

stephanV

16th November 2004, 10:17

Originally posted by Joe Fenton
The problem I see with double-blind tests on video is in the amount of data to download and compare. If you use 30 sec clips, that limits how much the person has to download and watch, but really screws up the test since short clips don't give you a real sense of how the codec works on longer encodes.
For proper testing you use 10 second clips, 30 seconds is already way too long. But of course, screenshots are an excellent way of determining how a codec works on longer encodes... right? Right.

"Let's see... we now move on to our eighth codec. Two more hours of "Ishtar" in H.264 format! Don't forget to mark how you think it compares to RV10, which you saw the day before yesterday."
What are you talking about???

Anyway, do a search for VQEG and find out about subjective testing...

Archimedes

27th May 2005, 00:51

To the subjective codec shoot-out test i’ve added an "objective" test with Fritz Framalyzer.

I’ve compared all pair of images (original frame and encoded frame) with Fritz Framalyzer and noticed the results (euclidean distances in RGB). This causes an encoder order for every scene and therefore for the complete film.

(Edit.)

Josef

Koepi

27th May 2005, 07:07

Originally posted by Archimedes
To the subjective codec shoot-out test i’ve added an "objective" test with Fritz Framalyzer.

I’ve compared all pair of images (original frame and encoded frame) with Fritz Framalyzer and noticed the results (euclidean distances in RGB). This causes an encoder order for every scene and therefore for the complete film.

You didn't read the code comparison properly it seems. You _cannot_ judge a codec from still frames. In one codec there might be a higher quantized b-frame where another codec uses an I- or P-frame.

Also you don't get the overall impression of the codec at running the movie at normal speed.

Thus doing a codec comparison just by comparing screenshots is completely invalid. (No, I didn't look at your comparison; dunno if xvid lost or won or anything [i couldn't care less because the test is flawed big time anyways], just "to take the wind out of some people's complains" which will sure arise.)

Cheers
Koepi