Custom matrix comparison - V3 - Round 1 - Finished

[Alxemi]

One more vote here :D

[CruNcher]

Bump

[Teegedeck]

Soulhunter, I wouldn't mind if you tried to draw some more attention to any comparison currently running by including a capital "OPEN" in the subject title of the thread.

[Soulhunter]

Originally posted by Teegedeck

Soulhunter, I wouldn't mind if you tried to draw some more attention to any comparison currently running by including a capital "OPEN" in the subject title of the thread.

Or even a "small" link in my sign... :D


Bye

[Sharktooth]

bump!

[*.mp4 guy]

and another bump

[Audionut]

I really don't understand the point of this 900kbps test.

You say that you don't want to remove grain etc, etc.
Yet the quality of your encodes is crap.
And I think it's obvious that nobody would be happy if there results were the same as this test.

ie: they'd delete the file and start again.

I understand where you're coming from, but at the end of the day when the quality of the encode is shit house, and nobody would watch it, what's the point.

Here is my effort, with my new matrix.
I've removed grain, resized etc, etc.
But it's watchable.

http://pic.integrityp2p.com/Audionut/test.avi

[Teegedeck]

I don't think you really wanted to sound that rude. But I have to warn you anyway not to try to advertise your matrix at the expense of other forum members.

If you don't change that tone the resulting comments could take on a similar tone which would lead to the result of me issueing strikes and editing and closing the whole thread - something I don't want to do just because of your matrix; which BTW has nothing to do with this thread.

Thus, please add that link to the discussion-thread about your matrix and remove it from this thread, it would prevent things from going more off-topic here.

Edit: It is too late to submit your matrix for this round of the test, but maybe for the upcoming ones?

[CruNcher]

I agree with Teegedeck here you could have opened a new topic like "Let´s compare Penises" or something and not try to hijack this one ;) but be warned what you show their is semi impressive i also have some impressive things in my pants :D :rolleyes:

[Soulhunter]

@ Audionut

This is not a "which CQM produces DVD quality @ 900kbps" test...

Its more a "which CQM works best under such hard conditions" test !!!


And yes, I fully agree with Teegedeck and CruNcher @ this...

If you wanna "advertise" this CQM, put a link in your sign like I do it !!!


Bye

[Sharktooth]

@Audionut: The test encodes are full anamorphic encodes (cropped black bars) with AR signaling (704x432 AR 16:9) and only the mosquito noise was removed...
That was because this test was meant to stress the codec and the CQMs to produce a distinguishable difference between matrices.
Producing a watchable 704x288 encode with a blurred and filtered source is easy... but in that case an untrained eye won't spot any difference from matrix X and matrix Y, and i wish to remember you this comparison is open to everyone...

BTW here's a couple of SS...
This is clip1 CMQ11... (704x432)
http://ebola.gamersrevolt.it/temp/st.jpg

and this is your encode stretched to 704x432 for a comparison...
http://ebola.gamersrevolt.it/temp/an.jpg

The SS were taken from the same frame. note how your filtering and different resolution have distorted the image.
Now tell me what's best...

[Didée]

Originally posted by Sharktooth
with a blurred and filtered source ... an untrained eye won't spot any difference from matrix X and matrix Y
Same applies to the current situation. To an untrained eye, everything looks just the same.

BTW, remember that under such hard "high compression" conditions, you are not testing the performance of a matrix solely. What you are really testing is the interaction of matrix and codec's ME. (Already have had a look on the motion vectors?)

Furtheron, I think we agree that we would not do a serious encoding in the way it was done in this 900kbps test, since the results are obviously not good. We would either use a higher bitrate, or stronger filtering. Or both. But *then* the codec would make other decisions in ME again ... probably "more natural" ones. And it's a problem getting obvious especially with strong compression, when the "mathematically ideal" motion vectors don't align with the natural motion.

Also, in this test many matrices look rather similar because they actually *do* *become* very similar. Look at the used quantizer ranges (wohoo!) - at these quants the biggest part of available DCT values gets quantized away anyways :) - And if the matrices actually are only working with a short handful of leftover values from the upper-left low-frq range, they all become very similar indeed ...

[Sharktooth]

Didée... come on... look at the first clip and at the last clip in the zip. Only a blind cant distinguish the differences...

[Soulhunter]

@ Sharktooth

I think there went something wrong @ the second picture...

Have you maybe used nearest neighbor for the resizing ???


Bye

[Didée]

You won't believe how visually insensitive most people are! (though the relative quantity of "most" might be somewhat more graceful amongst this' forum members, compared to the rest of "real world" - hopefully.)

Of course there are differences. But I repeat that in this test, personally I tend to differentiate between two groups only, perhaps three with some good will.
... Except of course for the matrix no. *censored*, which I'd rate the highest of all, for obvious reasons ;)


edit:

I think there went something wrong @ the second picture...
Have you maybe used nearest neighbor for the resizing ???
I'd say no, you would see that on the hi-contrast edges. But that effect is only on the smoother gradient transition. Smells a little like DCTFilter overdone. Or it's a characteristica of that matrix he used, can't tell that.

[Soulhunter]

Originally posted by Didée

I'd say no, you would see that on the hi-contrast edges. But that effect is only on the smoother gradient transition. Smells a little like DCTFilter overdone. Or it's a characteristica of that matrix he used, can't tell that.

Audionut's source:
http://img37.exs.cx/img37/2770/Source.jpg

Bilinear resized:
http://img37.exs.cx/img37/7768/Bilinear.jpg

Nearest neighbor:
http://img37.exs.cx/img37/7968/NearestN.jpg

So, Sharktooth's sample was resized with nearest neighbor !!!

Btw, for some reason his pic shows a block @ the left side...

I guess there went something wrong while decoding !?!


Bye

[Sharktooth]

Uhm. Maybe there's something wrong... i'll check.
EDIT: Video card driver problem... now the image is correctly (bicubic) resized.

[Soulhunter]

Btw, I think its time to finish this round !?!

I gonna post the results @ Monday... ;)


Bye

[Audionut]

Com'on, there are over 90 words in the mentioned post, and you are all getting concerned over,,
Originally posted by Audionut
Here is my effort, with my new matrix.


My point is that the encode that I posted is watchable

Originally posted by Soulhunter
Its more a "which CQM works best under such hard conditions" test !!!

Which confirms the point that I am "trying" to make.
Who cares how well these matrices perform "at such conditions", when nobody is going to put these matrices "through such conditions" in "real life", because it simply provides "useless" results.
IE: crap quality encodes.

Please tell me you understand WTF i'm talking about!

[Teegedeck]

The problem is that you mixed two things in your post that you shouldn't have mixed.

Your criticism about the merits of this kind of test is valid and has been voiced before. So far so good. Then you set against this your own matrix and your own encoding. This has nothing to do with this test, as the encode hasn't been performed under the same circumstances. But by doing this you set the results your own matrix produced against the results of the other matrices used in the test.

Thus you should under not have let it sound as if the avi you linked to could prove that your matrix was better in any way than the matrices used in this test.

The two points you mentioned would have been OK if they stood just isolatedly for themsevles; one of them belonged into the thread about your matrix, as I've said before. Please try to avert giving wrong impressions in the future.

[Sharktooth]

Originally posted by Audionut
Who cares how well these matrices perform "at such conditions", when nobody is going to put these matrices "through such conditions" in "real life", because it simply provides "useless" results.
Well, if a matrix performs better than the other matrices in those conditions i think that the winning matrix could be "elected" as the best matrix for low bitrates and not just 900... (maybe even lower!!!).
Why? coz a 704x288 encode with all filterings and stuff @ 700-800kbit (or less) could be a goal for someone...

[Soulhunter]

Sharktooth is right...

Testing stuff under "harder than normal" conditions, exposes the weak points better !!!


In theory this means...

The CQM that scores best @ 900kbps, should be the best for low bitrates in general !!!


Bye

[Teegedeck]

Audionut, the place to criticize the test-setup would have been the pre-test discussion-thread, it's too late now. Don't PM me about it.

And BTW, I didn't like the set-up, either, but not liking something is no reason to get worked-up about it. Keep your comments clean and don't use rude language.

[Audionut]

Originally posted by Teegedeck
Audionut, the place to criticize the test-setup would have been the pre-test discussion-thread, it's too late now. Don't PM me about it.
Here I was thinking that I was taking this thread way off topic, and that the best option would be a PM.
And you simply quote rules and regs back at me.

Originally posted by Teegedeck
And BTW, I didn't like the set-up, either, but not liking something is no reason to get worked-up about it. Keep your comments clean and don't use rude language.

Soulhunter and Sharktooth provided helpful answers as to why they run the test the way that they have. To which I am Gratefull.
They also throughly critised my encode, pointing out all the problems with it.

You simply, singled out the fact that I had used my own matrix, and pounced on that fact.

[Teegedeck]

I am not attacking you in any way but only watching that no-one here gets abusive - please re-read your first post in this thread and you may understand why I reacted to it. Let it go now.

It is understandable that you feel singled-out but please try to be objective and you might see that this is nothing personal. Take your time to reply, perhaps after a good night's sleep. I'm gonna do just that. Good night. :)

[Soulhunter]

@ Audionut

The encodes for the 1600/2200kbps round are already done n' uploaded !!!

But if you want, I will use ya matrix for my CQM comparison V4 then... ;)


Bye

[Soulhunter]

Matrices and appropriate numbers


CQM_NO_ 001 = YACQM
CQM_NO_ 002 = Soulhunters v6
CQM_NO_ 003 = SixOfNine HVS
CQM_NO_ 004 = EQM V3 LR
CQM_NO_ 005 = HVS Best
CQM_NO_ 006 = HVS Good
CQM_NO_ 007 = Standard MPEG
CQM_NO_ 008 = Bach1
CQM_NO_ 009 = H.263
CQM_NO_ 010 = Jawors 1CD
CQM_NO_ 011 = EQM v3 ULR



Results from best to worst


CLIP-01:

4,60 = EQM v3 ULR
4,15 = H.263
3,90 = Jawors 1CD
3,55 = Bach1
3,50 = Standard MPEG
2,70 = HVS Best
2,40 = HVS Good
2,25 = EQM V3 LR
2,15 = Soulhunters v6
1,85 = SixOfNine HVS
1,35 = YACQM


CLIP-02:

4,25 = EQM v3 ULR
4,05 = H.263
4,05 = Jawors 1CD
3,75 = Bach1
3,20 = Standard MPEG
2,80 = HVS Best
2,65 = HVS Good
2,40 = EQM V3 LR
2,35 = Soulhunters v6
2.35 = SixOfNine HVS
1,85 = YACQM


Comments...


- EQM v3 ULR, H.263 and Jawors 1CD matrix are superior @ very low bitrates !!!

- The bad results for my CQM's proofs than I dont manipulate the votes... :D

- The standard MPEG matrix is much better than I thought !!!

- Thanks @ all 9 participants for taking the time... :)


Bye

[Sharktooth]

Wow, H.263 scored a great 2nd place. I thought Jawor's matrix could easily beat it...
It's time for round 2 :)

[len0x]

Originally posted by Sharktooth
Wow, H.263 scored a great 2nd place. I thought Jawor's matrix could easily beat it...

Well, having a look at best 3 samples now - I would say that its still obvious that Jawor's matrix is doing much better than H263 and even V3ULR. IMHO higher amount of details cannot compensate for larger blockiness... And during playback blocks are more visible than small details...

[Alxemi]

Very interesting results!!!
Thank you soulhunter for the work in the comparison and Sharktootk & Jawor for your matrices.
Next round please!:D

[Soulhunter]

Think I gonna start the 2nd round tomorrow... ;)


Bye

[Chainmax]

My eyesight isn't trained enough to be of use in these tests, so I want to extend my gratitude to Soulhunter for creating these tests and to all the people that submitted their votes :).

[pwh04]

Another big thanks to Soul for the testing and to all who have provided the matrices.. I use them and it's much appreciated!

[Teegedeck]

Another big thank-you, and I guess we all hope that the next round will be even more interesting!

The big surprise for me was that I had expected Javor's matrix to win it all, and also Bach's scored better than I expected. The good marks for the MPEG-quant didn't surprise me that much.

In the 1st clip, EQM v3 ULR clearly wins, while Javor's matrix and H.263 are so close that I'd say the error margin makes it a tie for 2nd place with Bach and MPEG-standard as a close runner-up in 3rd place. The rest of the field falls behind.

In the 2nd clip, EQM v3 ULR, Javor's matrix and H.263 are so close together that I'd say they tie, Bach following in 2nd.

[Chainmax]

May I suggest a future round with anime/cartoon sources?

[Soulhunter]

Originally posted by Chainmax

May I suggest a future round with anime/cartoon sources?


I already planed to include a chapter of Animatrix in custom matrix comparison V4... ;)


Bye

[Teegedeck]

BTW, was denoising used for the 2nd round, also (/hoping that the answer is 'no')?

[Soulhunter]

Originally posted by Teegedeck

BTW, was denoising used for the 2nd round, also (/hoping that the answer is 'no')?

Nope, no filtering for the 2nd and 3rd round... ;)


Bye

[Sharktooth]

Originally posted by Soulhunter
I already planed to include a chapter of Animatrix in custom matrix comparison V4... ;)
I've done TONS of animatrix encodes for testing... :D
However,if you're interested, i should have a 720p animatrix trailer somewhere in my HD...

[Soulhunter]

Originally posted by Sharktooth

However,if you're interested, i should have a 720p animatrix trailer somewhere in my HD...

Sure, would be nice... :)

But is it the same trailer that was used on DVDs ???

Coz, iirc its mixed with much cgi and mainly fast motion... :o


Bye

[Sharktooth]

Yep, same trailer.

[Soulhunter]

Well, then Im not sure if it fits the "which CQM is best for anime" purpose... :rolleyes:

But lets discuss this in the preparation thread for CQM comparison V4 then !!!


Bye

[Soulhunter]

Ok, round 2 (http://forum.doom9.org/showthread.php?s=&threadid=85455) started... :)


Bye