Custom matrix comparison - V3 - Round 1 - Finished

[Soulhunter]

Once again...


1. How is it supposed to work ???


We did a encode of this source...

Gladiator (PAL/R2) / Chapter No. 19 (3:57 min.)


With this AVS...

LoadPlugin("C:\SOFTS\AVSFILES\MPEG2DEC3.DLL")
LoadPlugin("C:\SOFTS\AVSFILES\TEMPORALCLEANER.DLL")

Mpeg2Source("gladiator.d2v", idct=2)

Crop(6,76,704,432)

TemporalCleaner(5,10)
UnDot()


And this settings...

XviD v.1.0.2 (Koepis) / Defaults / 2Pass @ 900kbps (VHQ4) / Trellis


With 11 different matrices !!!


We took 2 samples of each encode, and named them like this:

CQM_NO_006_CLIP-01.avi


The red text indicates the matrix that was used...

CQM_NO_006_CLIP-01.avi


The blue text indicates the scene that was used...

CQM_NO_006_CLIP-01.avi



2. But how to vote ???


Simple, watch the samples under your "usually" playback conditions...

When you got a good impression, write down a number between 1-5 !!!

1 = Crap
2 = Bad
3 = Tolerable
4 = Good
5 = Perfect


Then send me your votes like this...

CLIP-01:

CQM_NO_001 = 2.0

CQM_NO_002 = 4.5

CQM_NO_003 = 5.0

etc...


And after some time (maybe 4 weeks), I post the results !!!



3. How to get the samples ???


1st. Click here... (http://service.gmx.net/mc/wTqkmJdHCNtdUjFvrxssJnzFwaAzPB)

2nd. Press this button -> http://img60.exs.cx/img60/6854/bt_gmxmediacenterstarten.gif

3rd. Download the standard or broad-band zip file !!!


Hope this explains it more or less... :rolleyes:


EDIT: Comparison ended, you can find the results here (http://forum.doom9.org/showthread.php?s=&postid=570152#post570152) !!!


Tia n' Bye

[Teegedeck]

Awww, they all look like '2' to me... :(

I don't know if I really can rate this but I'm gonna try.

Thanks for giving us that opportunity, hard as it may be. ;)

[Soulhunter]

I would like to use some longer samples (10 seconds or so), but this would raise the download time... :(

The 2200kbps round is already 25MB huge, now imagine I would double the length of the samples !!!

Dont know how much people would download 50MB to participate @ this test... :rolleyes:

I could add a additional zip with longer samples for broad-band users !?!


Bye

[708145]

A special broadband version would be nice.

:sly: The upper limit is given by GMX I guess (1GB).

bis besser,
T0B1A5

[Soulhunter]

Ok, guess Ive done the cutting + uploading in two hours or so... ;)

Btw, thanks again for the GMX hint !!!


Bye

[ChronoCross]

the quality makes me kinda sad. I wouldn't give these more than a 1 out of 10. there is no real difference between the clips. Sorry.

[Soulhunter]

Ok, have the broad-band samples up now !!!

@ ChronoCross

Yeah the quality is indeed very low... :(

But what have you expected @ 900kbps !?!


Bye

[ChronoCross]

yes indeed I knew what the quality would be at 900kbps. I don't think that a CQM comparison can be done because the difference like I said is so neglegable.

[Sharktooth]

i can clearly see the differences, however look at the artifacts (ringing and blocking) and details (both on still and moving objects) in a frame-by-frame comparison.
Hint: There are 2 or 3 clips that are clearly better than the others.

[Soulhunter]

Originally posted by Sharktooth

Hint: There are 2 or 3 clips that are clearly better than the others.

Really ???

Deny myself to make a comment @ this... :D :rolleyes: ;)


Bye

[Sharktooth]

Oh well... i was generous saying "2 or 3"... :D

[*.mp4 guy]

Yes they are all very similar, but there are some that i think were noticibly better and a few that were a bit worse(some were less blocky on the sword and the arm in the begining of sample 1 in my opinion). There was practically no detail in any of them. well thats my 1/2 cent.

[Audionut]

Sorry I haven't been around lately, and haven't had a chance to reply to the other thread.

IMHO you should have used a chapter either side of chapter 19.
Chapter 19 requires a HIGH bitrate for decent quality. Chapters either side require substantially less bitrate.
Would have made xvid's job alot easier at encoding the samples at 900kbps. while maintaing good quality.


Removegrain does the same shit as undot, just better.
Get a sample clip, and undot it at a const. quant.
Now change undot for removegrain() and const. quant encode it.
Note the difference in filesize!

AFAI Concerned this sample at 900kbps needs to be resized.

My script recommendation, although to late is this.

LoadPlugin("C:\DGDecode.dll")
LoadPlugin("C:\Unfilter.dll")
Loadplugin("C:\removegrain.dll")
LoadPlugin("C:\FluxSmooth.dll")

Mpeg2Source("C:\gladiator.d2v")

removegrain()
fluxsmootht()
Crop(8,76,-8,-70)

Lanczos4resize(704,288)
removegrain()
unfilter(-5,-5)


Note:the reason for the largeish crop left & right is this.
1.Removegrain and fluxsmooth do not process the very edges of frames.
2.maintains mod16 width

[Teegedeck]

Originally posted by Sharktooth
i can clearly see the differences, however look at the artifacts (ringing and blocking) and details (both on still and moving objects) in a frame-by-frame comparison.
Frame-by-frame comparisons are VERBOTEN this once! :D

The idea to compare sample clips instead of frames was precisely about having a more realistic comparison. Usually, we don't watch a movie frame-by-frame. So if a viewer cannot tell the difference by viewing the samples, then there simply is no difference worth mentioning.

Also I doubt whether it makes sense trying to judge which artifact looks better than the other (blockfest beauty competition?). Or whether ringing looks better than flickering blocks or something similar. Both are annoying and should lead to a bad mark IMHO. :o

@Soulhunter: I'm having problems with the relative rating system. I cannot attain a 'worst', 'average' or 'best' mark if all I can relate to are the samples themselves because I cannot tell enough differences to spread the marks across the whole span of marks from 1 to 10. I take it you actually meant something in a manner like this: 1= blockfest, 2=unbearable, 3=hardly bearable 4=very annoying, 5=annoying, 6=mildly annoying, 7=not true to the original but not annoying, 8=close to the original, 9=hard to tell from the original, 10=transparent. Hm, I'd need the original to compare it to for that. Tough.
How about: 1= blockfest, 2=unbearable, 3=hardly bearable 4=very annoying, 5=annoying, 6=mildly annoying, 7=not pleasing but not annoying, 8=almost pleasing to watch, 9=almost entirely pleasing to watch, 10=pleasing to the eye.

:-/

[Sharktooth]

Originally posted by Audionut
Removegrain does the same shit as undot, just better.
Filtering was discussed long time ago but we wanted to remove noise and preserve film grain and details. The only alternative was "No filtering at all".
FluxSmoothT and RemoveGrain were tested and did not produced the expecet results.
Unfilter(-5,-5) is out of place because blurring = removing details.
Resizing was also dropped in favour of AR signaling.

Originally posted by Teegedeck
Frame-by-frame comparisons are VERBOTEN this once! :D

The idea to compare sample clips instead of frames was precisely about having a more realistic comparison. Usually, we don't watch a movie frame-by-frame. So if a viewer cannot tell the difference by viewing the samples, then there simply is no difference worth mentioning.
Maybe a Frame-by-frame comparison is not necessary but evaluating quality or spotting details/defects in a moving clip is quite difficult while it's easy to spot moving artifacts...

Originally posted by Teegedeck
Also I doubt whether it makes sense trying to judge which artifact looks better than the other (blockfest beauty competition?). Or whether ringing looks better than flickering blocks or something similar. Both are annoying and should lead to a bad mark IMHO. :o
I meant to look at the quantity of artifacts and how much they annoy your eyes. There are samples with more annoying artifacts than others...

[Teegedeck]

Results sent.

[Sharktooth]

I voted too...

[Teegedeck]

BTW, the second clip is a really high-motion one. I casted separate votes for the clip at normal speed and at 1/4 speed. And I oriented them by the rating system I suggested above. I'd say if we were strict about reproducing 'normal' viewing-conditions we'd have to question whether comparing such clips makes sense.

@Sharktooth: Of course I understand what you mean but the comparison gets more problematic with rising compression.
First we haven't got an original to compare the clips to - which is a technical problem and can't be helped - then we have the clips getting farther and farther away from that absent original's look as compression gets higher. And with this, verdicts about quality become more and more verdicts about which form of encoding errors one prefers over others. What is worse, more smoothing or some more blocks? A matter of common or individual taste. A form of taste which is 'artificial' or 'synthetic' IMHO in a sense that we normaly perceive a video-image as good if it looks life-like - i.e. close to the thing it depicts.
Such a quality of similarity is the thing we get very far detached from with such strong compresson. And 'similarity to an (absent) original' becomes a very vague notion by then. That is why I find it difficult to make big distinctions between the samples.

[Sharktooth]

Oh, well, i have it, but it's 180Mbytes...

[Teegedeck]

The more important aspect is the copies becoming too dissimilar from the original, anyway. Not that it stopped me from expressing _my_ taste, with all its vagueness.

[Soulhunter]

Wow, already 3 ppl voted... :)

Big thanks !!!

[708145]

Wow, already 3 ppl voted...

Big thanks !!!

Make that 4.

bis besser,
Tobias

[kurt]

kurt voted too :D

cya

[JarrettH]

I'm working on it....:p
Thanks so much for the effort. I remember encoding Alien 3 SE using a bitrate of 910kbps with AC3 audio (2CD) and the movie was 2hrs 25min. This should be an interesting comparison then (Gladiator is 10min shorter).;)

[Soulhunter]

Nice, 7 votes so far... :cool:


Bye

[((( atom )))]

i am currently downloading the files but as i read from the thread the bitrate used on the clips was rather low..

ok, there are some matrices designed for low bitrates and these will look rather fine i expect, but what about the high br ones?
i always encode at a compressibility above 80%, targeting for either 1400mb or 2gb, if it isn't enough. bytes _are_ cheap meanwhile..

i also have a lare collection of mostly old, meaning very noisy, movies.

so all in all - and i guess i am not alone here - seing the matrices perform at higher bitrates would be very interesting.

comments, please..

[((( atom )))]

hmm, i just wiewed the clips shortly and find it very hard to build an opinion..

i have the feeling the comparision actually is "which matrix produces the least artifacts" wich also is interesting for the mentioned lower bitrate targets.

"wich matrix preserves the most detail" for instance looks like a different story for me.

personally i would suggest to offer more test aiming at different questions: least artifacts and most detail.

the ideal matrix will always be a compromise between thes two, right? that is then where taste comes into play...

it was mentioned above that comparing single frames would be verboten, i am sorry but i disagree :(
when i watch a scene, i can not se everything at once and i will see different things every time again, especially when the scene is of high motion nature.
it is extremely difficult judging a running movie, that is why so many people are satisfied with low bitrates, i believe.
also i experienced many times, that a movie, wich i thought was of decent quality on a given playout set, suddenly looked worse lets say on a projector, due to the fact that i could follow small details more easily even through motion because they were a lot bigger and easier to keep up with.

if the frame is good, surely the running film will be.

[Soulhunter]

Originally posted by ((( atom )))


so all in all - and i guess i am not alone here - seing the matrices perform at higher bitrates would be very interesting.

comments, please...

As discussed in the preparation thread (http://forum.doom9.org/showthread.php?s=&threadid=82022), there will be a 1600kbps and 2200kbps round... ;)


Originally posted by ((( atom )))

hmm, i just wiewed the clips shortly and find it very hard to build an opinion...


My suggestion...

- First, try to separate between the "very good" and the "very bad" ones

- Now give the best sample 10 points and the worst sample 1 point

- Finally, try to judge the rest in contrast to the best/worst one


Bye

[708145]

- First, try to separate between the "very good" and the "very bad" ones
- Now give the best sample 10 points and the worst sample 1 point
- Finally, try to judge the rest in contrast to the best/worst one


Oops! :eek: Then you should scale all my results. I used a different scale.

bis besser,
T0B1A5

[Soulhunter]

Originally posted by 708145

Oops! :eek: Then you should scale all my results. I used a different scale.

bis besser,
T0B1A5

Yap, Ive scaled them... ;)

Simply used multiplication to get the min/max = 1/10 values !!!


EDIT1:

Think next time I should include this under "how to vote" !?!

Thought the "1=worst/10=best" would be enough... :rolleyes:


EDIT2:

Ok, Ive added the suggestion to my first post !!!


Bye

[((( atom )))]

Originally posted by Soulhunter
[B]As discussed in the preparation thread (http://forum.doom9.org/showthread.php?s=&threadid=82022), there will be a 1600kbps and 2200kbps round... ;)ok, didn't know that thread. i will look it up.

[Soulhunter]

Originally posted by ((( atom )))

ok, didn't know that thread. i will look it up.

Ehm, what I remember you have posted in this thread yesterday !!!

I know, sometimes the huge amount of threads is confusing... :D


Bye

[Teegedeck]

I'd like to voice some protest against this 'upscaling'. :o

That method artificially creates clear winners and loosers where actually the participants in this test found it very hard to find differences at all, let alone judge them.

While your method will even overemphasize a 'clarity' which wasn't there, subjective tests actually don't give clear results; on the contrary such tests leave a broad room for statistical error and uncertainty. Which is why the statistical analysis of a test is perhaps even more of a task than its preparation. If you look at the presentation of the results of Roberto's listening tests, as an example of good practice, only if two test samples' scores differ enough to exclude statistical error as a reason for that difference, one of the test samples can be said to have scored better than the other with an acceptable level of certainty (in Roberto's listening tests that means with 95% confidence). See here (http://www.rjamorim.com/test/32kbps/results.html).

Here's a little link on statistical analysis (http://davidmlane.com/hyperstat/index.html).

I don't ask you to perform an ANOVA analyis (there probably won't be enough results to justify performing one, anyway) or anything like that but I urge you to give up stretching those results over the 1-10 spectrum. If test participants mean to say 'those samples all look that bad, there hardly is a difference', then you shouldn't try to create the image that there were big differences.

[Soulhunter]

Originally posted by Teegedeck

I'd like to voice some protest against this 'upscaling'. :o

That method artificially creates clear winners and losers where actually the participants in this test found it very hard to find differences at all, let alone judge them.

While your method will even overemphasize a 'clarity' which wasn't there, subjective tests actually don't give clear results; on the contrary such tests leave a broad room for statistical error and uncertainty. Which is why the statistical analysis of a test is perhaps even more of a task than its preparation. If you look at the presentation of the results of Roberto's listening tests, as an example of good practice, only if two test samples' scores differ enough to exclude statistical error as a reason for that difference, one of the test samples can be said to have scored better than the other with an acceptable level of certainty (in Roberto's listening tests that means with 95% confidence). See here (http://www.rjamorim.com/test/32kbps/results.html).

Here's a little link on statistical analysis (http://davidmlane.com/hyperstat/index.html).

I don't ask you to perform an ANOVA analysis (there probably won't be enough results to justify performing one, anyway) or anything like that but I urge you to give up stretching those results over the 1-10 spectrum. If test participants mean to say 'those samples all look that bad, there hardly is a difference', then you shouldn't try to create the image that there were big differences.

Generally I agree with you... :)

But 708145 "authorized" me in the pm to scale his results !!!

Anyhow, I will study the info you gave me before resuming discussion... ;)


Thanks n' Bye

[Teegedeck]

Well, that's all I could ask; it's your test! :)

[Soulhunter]

@ Teegedeck

PM is on the way !!!

Btw, Ive discovered that this GMX hosting has a traffic limit of 2GB... :scared:


Bye

[Soulhunter]

Ok, we (actually Teegedeck) found a easy solution...

If you upscale for example my results to your relative scale, this would express my judgment of the matrices' quality correctly, though with large overemphasis. If you now downscale all results again to a range of 1-5, instead of 1-10, (still on a relative scale!) you at least deemphasize those differences again, especially if you express the margin of error in a similar manner as Roberto when you present the results.

[Teegedeck]

Just noticed that it doesn't make much sense the way I expressed it. Must of course read: scale up all results to 1-10, then divide by 2.

[Soulhunter]

Originally posted by Teegedeck

Just noticed that it doesn't make much sense the way I expressed it. Must of course read: scale up all results to 1-10, then divide by 2.

Hmm, but thats the way I understood it... :confused:

Scale the results -> 1-10 -> 1-5 !!!

Thats a division of 2, no...


Bye

[CruNcher]

Ok rated those 2 samples, but i have to say im not happy with the Content that was chosen the 2nd clip was ok but for me the 1st clip is nothing else it shows high or ok maybe middle motion but it´s definatly not low motion like a direct conversation shot, i´m a little upset about this :(

[Soulhunter]

Originally posted by CruNcher

Ok rated those 2 samples, but i have to say im not happy with the Content that was chosen the 2nd clip was ok but for me the 1st clip is nothing else it shows high or ok maybe middle motion but it´s definatly not low motion like a direct conversation shot, i´m a little upset about this :(

Imo its mainly mid/low motion !!!

Its hard to make everyone happy... :(


Bye

[Teegedeck]

Originally posted by Soulhunter
Hmm, but thats the way I understood it... :confused:

Scale the results -> 1-10 -> 1-5 !!!

Thats a division of 2, no...


Bye
Yeees, you understood what I meant. After all it seems I didn't express it so badly after all. :p

[Arcon]

it would be nice if you could also include the quality factor in bits per pixel for the final results. i think its much easier to find the right matrix based on the bpp instead of the bitrate/resolution/framerate given (assuming that we won't have one matrix that wins all bitrate contests ;)).

[len0x]

Originally posted by Arcon
i think its much easier to find the right matrix based on the bpp instead of the bitrate/resolution/framerate given

Do ppl still believe that? Both ways are not appropriate to decide on matrix...

[Arcon]

Originally posted by len0x
Do ppl still believe that? Both ways are not appropriate to decide on matrix...
but i have to choose one way :( i can't wait for an inspiration particle to solve this problem and a bpp seems to me more general than a bitrate. if there is an error in reasoning please tell me.

[Teegedeck]

This test was precisely about not having some bogus, artificial, calculated or computed result but about what actually and simply looks best at a given compression ratio (it is not about bitrate, really).

If there are enough participants in this and the upcoming tests to make it statistically valid, you'll have for the first time a sensible lead about which matrix to choose at which bitrate. Of course even then it still is up to your personal taste and differences in the look of your source, so you should always check whether your taste goes with the taste of the crowd.

[Sharktooth]

Choosing the right matrix depends on the source quality and type (DVD movie? DV capture? Anime? etc...), in how many bframes the codec will place at the end of the encode, how many keyframes the codec will place at the end of the encode, the bitrate you intend to use, the filters you will apply (and the amount of noise that remains after filtering), the amount of motion (low-mid-high) and some other minor factors.
If i forgot something plz feel free to add it to the list...

[Soulhunter]

Now we have 8.5 votes... :)


Bye

[Arcon]

Originally posted by Teegedeck
This test was precisely about not having some bogus, artificial, calculated or computed result but about what actually and simply looks best at a given compression ratio (it is not about bitrate, really).
maybe i said something wrong but i don't want to alter the test or the test-methods at all. i'd just like to see the bpp in the final result. so instead of 'ranking in the 900kbps class' i'd like to see something like 'ranking in the 900kbps class (0.284bpp)' so it's easier to estimate what scenario equals my current encoding project the most so that i can pick the right matrix. thats all.

it's just a convenience information so that i don't have to calculate the bpp manually from res/framerate/bitrate.

[Sharktooth]

bpp is not an universal and safe method to determine what matrix to use.
bpp is useful only when certain "variables" match certain values.
For example a high-motion source will need a higher bitrate (and higher bpp) to achieve a good quality encode. A low motion one can achieve a good quality at a much lower bitrate (and bpp).
Using bpp is not as good as using compressibility to estabilish what matrix and/or resolution is needed. Hence there are 2 tests (low motion and high motion) at different bitrates.
Once estabilished what are your personal preferences at those bitrates and motion you have a list of matrices that are good for that scenario.

[Alxemi]

One more vote here :D

[CruNcher]

Bump

[Teegedeck]

Soulhunter, I wouldn't mind if you tried to draw some more attention to any comparison currently running by including a capital "OPEN" in the subject title of the thread.

[Soulhunter]

Originally posted by Teegedeck

Soulhunter, I wouldn't mind if you tried to draw some more attention to any comparison currently running by including a capital "OPEN" in the subject title of the thread.

Or even a "small" link in my sign... :D


Bye

[Sharktooth]

bump!

[*.mp4 guy]

and another bump

[Audionut]

I really don't understand the point of this 900kbps test.

You say that you don't want to remove grain etc, etc.
Yet the quality of your encodes is crap.
And I think it's obvious that nobody would be happy if there results were the same as this test.

ie: they'd delete the file and start again.

I understand where you're coming from, but at the end of the day when the quality of the encode is shit house, and nobody would watch it, what's the point.

Here is my effort, with my new matrix.
I've removed grain, resized etc, etc.
But it's watchable.

http://pic.integrityp2p.com/Audionut/test.avi

[Teegedeck]

I don't think you really wanted to sound that rude. But I have to warn you anyway not to try to advertise your matrix at the expense of other forum members.

If you don't change that tone the resulting comments could take on a similar tone which would lead to the result of me issueing strikes and editing and closing the whole thread - something I don't want to do just because of your matrix; which BTW has nothing to do with this thread.

Thus, please add that link to the discussion-thread about your matrix and remove it from this thread, it would prevent things from going more off-topic here.

Edit: It is too late to submit your matrix for this round of the test, but maybe for the upcoming ones?

[CruNcher]

I agree with Teegedeck here you could have opened a new topic like "Let´s compare Penises" or something and not try to hijack this one ;) but be warned what you show their is semi impressive i also have some impressive things in my pants :D :rolleyes:

[Soulhunter]

@ Audionut

This is not a "which CQM produces DVD quality @ 900kbps" test...

Its more a "which CQM works best under such hard conditions" test !!!


And yes, I fully agree with Teegedeck and CruNcher @ this...

If you wanna "advertise" this CQM, put a link in your sign like I do it !!!


Bye

[Sharktooth]

@Audionut: The test encodes are full anamorphic encodes (cropped black bars) with AR signaling (704x432 AR 16:9) and only the mosquito noise was removed...
That was because this test was meant to stress the codec and the CQMs to produce a distinguishable difference between matrices.
Producing a watchable 704x288 encode with a blurred and filtered source is easy... but in that case an untrained eye won't spot any difference from matrix X and matrix Y, and i wish to remember you this comparison is open to everyone...

BTW here's a couple of SS...
This is clip1 CMQ11... (704x432)
http://ebola.gamersrevolt.it/temp/st.jpg

and this is your encode stretched to 704x432 for a comparison...
http://ebola.gamersrevolt.it/temp/an.jpg

The SS were taken from the same frame. note how your filtering and different resolution have distorted the image.
Now tell me what's best...

[Didée]

Originally posted by Sharktooth
with a blurred and filtered source ... an untrained eye won't spot any difference from matrix X and matrix Y
Same applies to the current situation. To an untrained eye, everything looks just the same.

BTW, remember that under such hard "high compression" conditions, you are not testing the performance of a matrix solely. What you are really testing is the interaction of matrix and codec's ME. (Already have had a look on the motion vectors?)

Furtheron, I think we agree that we would not do a serious encoding in the way it was done in this 900kbps test, since the results are obviously not good. We would either use a higher bitrate, or stronger filtering. Or both. But *then* the codec would make other decisions in ME again ... probably "more natural" ones. And it's a problem getting obvious especially with strong compression, when the "mathematically ideal" motion vectors don't align with the natural motion.

Also, in this test many matrices look rather similar because they actually *do* *become* very similar. Look at the used quantizer ranges (wohoo!) - at these quants the biggest part of available DCT values gets quantized away anyways :) - And if the matrices actually are only working with a short handful of leftover values from the upper-left low-frq range, they all become very similar indeed ...

[Sharktooth]

Didée... come on... look at the first clip and at the last clip in the zip. Only a blind cant distinguish the differences...

[Soulhunter]

@ Sharktooth

I think there went something wrong @ the second picture...

Have you maybe used nearest neighbor for the resizing ???


Bye

[Didée]

You won't believe how visually insensitive most people are! (though the relative quantity of "most" might be somewhat more graceful amongst this' forum members, compared to the rest of "real world" - hopefully.)

Of course there are differences. But I repeat that in this test, personally I tend to differentiate between two groups only, perhaps three with some good will.
... Except of course for the matrix no. *censored*, which I'd rate the highest of all, for obvious reasons ;)


edit:

I think there went something wrong @ the second picture...
Have you maybe used nearest neighbor for the resizing ???
I'd say no, you would see that on the hi-contrast edges. But that effect is only on the smoother gradient transition. Smells a little like DCTFilter overdone. Or it's a characteristica of that matrix he used, can't tell that.

[Soulhunter]

Originally posted by Didée

I'd say no, you would see that on the hi-contrast edges. But that effect is only on the smoother gradient transition. Smells a little like DCTFilter overdone. Or it's a characteristica of that matrix he used, can't tell that.

Audionut's source:
http://img37.exs.cx/img37/2770/Source.jpg

Bilinear resized:
http://img37.exs.cx/img37/7768/Bilinear.jpg

Nearest neighbor:
http://img37.exs.cx/img37/7968/NearestN.jpg

So, Sharktooth's sample was resized with nearest neighbor !!!

Btw, for some reason his pic shows a block @ the left side...

I guess there went something wrong while decoding !?!


Bye

[Sharktooth]

Uhm. Maybe there's something wrong... i'll check.
EDIT: Video card driver problem... now the image is correctly (bicubic) resized.

[Soulhunter]

Btw, I think its time to finish this round !?!

I gonna post the results @ Monday... ;)


Bye

[Audionut]

Com'on, there are over 90 words in the mentioned post, and you are all getting concerned over,,
Originally posted by Audionut
Here is my effort, with my new matrix.


My point is that the encode that I posted is watchable

Originally posted by Soulhunter
Its more a "which CQM works best under such hard conditions" test !!!

Which confirms the point that I am "trying" to make.
Who cares how well these matrices perform "at such conditions", when nobody is going to put these matrices "through such conditions" in "real life", because it simply provides "useless" results.
IE: crap quality encodes.

Please tell me you understand WTF i'm talking about!

[Teegedeck]

The problem is that you mixed two things in your post that you shouldn't have mixed.

Your criticism about the merits of this kind of test is valid and has been voiced before. So far so good. Then you set against this your own matrix and your own encoding. This has nothing to do with this test, as the encode hasn't been performed under the same circumstances. But by doing this you set the results your own matrix produced against the results of the other matrices used in the test.

Thus you should under not have let it sound as if the avi you linked to could prove that your matrix was better in any way than the matrices used in this test.

The two points you mentioned would have been OK if they stood just isolatedly for themsevles; one of them belonged into the thread about your matrix, as I've said before. Please try to avert giving wrong impressions in the future.

[Sharktooth]

Originally posted by Audionut
Who cares how well these matrices perform "at such conditions", when nobody is going to put these matrices "through such conditions" in "real life", because it simply provides "useless" results.
Well, if a matrix performs better than the other matrices in those conditions i think that the winning matrix could be "elected" as the best matrix for low bitrates and not just 900... (maybe even lower!!!).
Why? coz a 704x288 encode with all filterings and stuff @ 700-800kbit (or less) could be a goal for someone...

[Soulhunter]

Sharktooth is right...

Testing stuff under "harder than normal" conditions, exposes the weak points better !!!


In theory this means...

The CQM that scores best @ 900kbps, should be the best for low bitrates in general !!!


Bye

[Teegedeck]

Audionut, the place to criticize the test-setup would have been the pre-test discussion-thread, it's too late now. Don't PM me about it.

And BTW, I didn't like the set-up, either, but not liking something is no reason to get worked-up about it. Keep your comments clean and don't use rude language.

[Audionut]

Originally posted by Teegedeck
Audionut, the place to criticize the test-setup would have been the pre-test discussion-thread, it's too late now. Don't PM me about it.
Here I was thinking that I was taking this thread way off topic, and that the best option would be a PM.
And you simply quote rules and regs back at me.

Originally posted by Teegedeck
And BTW, I didn't like the set-up, either, but not liking something is no reason to get worked-up about it. Keep your comments clean and don't use rude language.

Soulhunter and Sharktooth provided helpful answers as to why they run the test the way that they have. To which I am Gratefull.
They also throughly critised my encode, pointing out all the problems with it.

You simply, singled out the fact that I had used my own matrix, and pounced on that fact.

[Teegedeck]

I am not attacking you in any way but only watching that no-one here gets abusive - please re-read your first post in this thread and you may understand why I reacted to it. Let it go now.

It is understandable that you feel singled-out but please try to be objective and you might see that this is nothing personal. Take your time to reply, perhaps after a good night's sleep. I'm gonna do just that. Good night. :)

[Soulhunter]

@ Audionut

The encodes for the 1600/2200kbps round are already done n' uploaded !!!

But if you want, I will use ya matrix for my CQM comparison V4 then... ;)


Bye

[Soulhunter]

Matrices and appropriate numbers


CQM_NO_ 001 = YACQM
CQM_NO_ 002 = Soulhunters v6
CQM_NO_ 003 = SixOfNine HVS
CQM_NO_ 004 = EQM V3 LR
CQM_NO_ 005 = HVS Best
CQM_NO_ 006 = HVS Good
CQM_NO_ 007 = Standard MPEG
CQM_NO_ 008 = Bach1
CQM_NO_ 009 = H.263
CQM_NO_ 010 = Jawors 1CD
CQM_NO_ 011 = EQM v3 ULR



Results from best to worst


CLIP-01:

4,60 = EQM v3 ULR
4,15 = H.263
3,90 = Jawors 1CD
3,55 = Bach1
3,50 = Standard MPEG
2,70 = HVS Best
2,40 = HVS Good
2,25 = EQM V3 LR
2,15 = Soulhunters v6
1,85 = SixOfNine HVS
1,35 = YACQM


CLIP-02:

4,25 = EQM v3 ULR
4,05 = H.263
4,05 = Jawors 1CD
3,75 = Bach1
3,20 = Standard MPEG
2,80 = HVS Best
2,65 = HVS Good
2,40 = EQM V3 LR
2,35 = Soulhunters v6
2.35 = SixOfNine HVS
1,85 = YACQM


Comments...


- EQM v3 ULR, H.263 and Jawors 1CD matrix are superior @ very low bitrates !!!

- The bad results for my CQM's proofs than I dont manipulate the votes... :D

- The standard MPEG matrix is much better than I thought !!!

- Thanks @ all 9 participants for taking the time... :)


Bye

[Sharktooth]

Wow, H.263 scored a great 2nd place. I thought Jawor's matrix could easily beat it...
It's time for round 2 :)

[len0x]

Originally posted by Sharktooth
Wow, H.263 scored a great 2nd place. I thought Jawor's matrix could easily beat it...

Well, having a look at best 3 samples now - I would say that its still obvious that Jawor's matrix is doing much better than H263 and even V3ULR. IMHO higher amount of details cannot compensate for larger blockiness... And during playback blocks are more visible than small details...

[Alxemi]

Very interesting results!!!
Thank you soulhunter for the work in the comparison and Sharktootk & Jawor for your matrices.
Next round please!:D

[Soulhunter]

Think I gonna start the 2nd round tomorrow... ;)


Bye

[Chainmax]

My eyesight isn't trained enough to be of use in these tests, so I want to extend my gratitude to Soulhunter for creating these tests and to all the people that submitted their votes :).

[pwh04]

Another big thanks to Soul for the testing and to all who have provided the matrices.. I use them and it's much appreciated!

[Teegedeck]

Another big thank-you, and I guess we all hope that the next round will be even more interesting!

The big surprise for me was that I had expected Javor's matrix to win it all, and also Bach's scored better than I expected. The good marks for the MPEG-quant didn't surprise me that much.

In the 1st clip, EQM v3 ULR clearly wins, while Javor's matrix and H.263 are so close that I'd say the error margin makes it a tie for 2nd place with Bach and MPEG-standard as a close runner-up in 3rd place. The rest of the field falls behind.

In the 2nd clip, EQM v3 ULR, Javor's matrix and H.263 are so close together that I'd say they tie, Bach following in 2nd.

[Chainmax]

May I suggest a future round with anime/cartoon sources?

[Soulhunter]

Originally posted by Chainmax

May I suggest a future round with anime/cartoon sources?


I already planed to include a chapter of Animatrix in custom matrix comparison V4... ;)


Bye

[Teegedeck]

BTW, was denoising used for the 2nd round, also (/hoping that the answer is 'no')?

[Soulhunter]

Originally posted by Teegedeck

BTW, was denoising used for the 2nd round, also (/hoping that the answer is 'no')?

Nope, no filtering for the 2nd and 3rd round... ;)


Bye

[Sharktooth]

Originally posted by Soulhunter
I already planed to include a chapter of Animatrix in custom matrix comparison V4... ;)
I've done TONS of animatrix encodes for testing... :D
However,if you're interested, i should have a 720p animatrix trailer somewhere in my HD...

[Soulhunter]

Originally posted by Sharktooth

However,if you're interested, i should have a 720p animatrix trailer somewhere in my HD...

Sure, would be nice... :)

But is it the same trailer that was used on DVDs ???

Coz, iirc its mixed with much cgi and mainly fast motion... :o


Bye

[Sharktooth]

Yep, same trailer.

[Soulhunter]

Well, then Im not sure if it fits the "which CQM is best for anime" purpose... :rolleyes:

But lets discuss this in the preparation thread for CQM comparison V4 then !!!


Bye

[Soulhunter]

Ok, round 2 (http://forum.doom9.org/showthread.php?s=&threadid=85455) started... :)


Bye