Custom matrix comparison - V3 - Round 1 - Finished

[Soulhunter]

Once again...


1. How is it supposed to work ???


We did a encode of this source...

Gladiator (PAL/R2) / Chapter No. 19 (3:57 min.)


With this AVS...

LoadPlugin("C:\SOFTS\AVSFILES\MPEG2DEC3.DLL")
LoadPlugin("C:\SOFTS\AVSFILES\TEMPORALCLEANER.DLL")

Mpeg2Source("gladiator.d2v", idct=2)

Crop(6,76,704,432)

TemporalCleaner(5,10)
UnDot()


And this settings...

XviD v.1.0.2 (Koepis) / Defaults / 2Pass @ 900kbps (VHQ4) / Trellis


With 11 different matrices !!!


We took 2 samples of each encode, and named them like this:

CQM_NO_006_CLIP-01.avi


The red text indicates the matrix that was used...

CQM_NO_006_CLIP-01.avi


The blue text indicates the scene that was used...

CQM_NO_006_CLIP-01.avi



2. But how to vote ???


Simple, watch the samples under your "usually" playback conditions...

When you got a good impression, write down a number between 1-5 !!!

1 = Crap
2 = Bad
3 = Tolerable
4 = Good
5 = Perfect


Then send me your votes like this...

CLIP-01:

CQM_NO_001 = 2.0

CQM_NO_002 = 4.5

CQM_NO_003 = 5.0

etc...


And after some time (maybe 4 weeks), I post the results !!!



3. How to get the samples ???


1st. Click here... (http://service.gmx.net/mc/wTqkmJdHCNtdUjFvrxssJnzFwaAzPB)

2nd. Press this button -> http://img60.exs.cx/img60/6854/bt_gmxmediacenterstarten.gif

3rd. Download the standard or broad-band zip file !!!


Hope this explains it more or less... :rolleyes:


EDIT: Comparison ended, you can find the results here (http://forum.doom9.org/showthread.php?s=&postid=570152#post570152) !!!


Tia n' Bye

[Teegedeck]

Awww, they all look like '2' to me... :(

I don't know if I really can rate this but I'm gonna try.

Thanks for giving us that opportunity, hard as it may be. ;)

[Soulhunter]

I would like to use some longer samples (10 seconds or so), but this would raise the download time... :(

The 2200kbps round is already 25MB huge, now imagine I would double the length of the samples !!!

Dont know how much people would download 50MB to participate @ this test... :rolleyes:

I could add a additional zip with longer samples for broad-band users !?!


Bye

[708145]

A special broadband version would be nice.

:sly: The upper limit is given by GMX I guess (1GB).

bis besser,
T0B1A5

[Soulhunter]

Ok, guess Ive done the cutting + uploading in two hours or so... ;)

Btw, thanks again for the GMX hint !!!


Bye

[ChronoCross]

the quality makes me kinda sad. I wouldn't give these more than a 1 out of 10. there is no real difference between the clips. Sorry.

[Soulhunter]

Ok, have the broad-band samples up now !!!

@ ChronoCross

Yeah the quality is indeed very low... :(

But what have you expected @ 900kbps !?!


Bye

[ChronoCross]

yes indeed I knew what the quality would be at 900kbps. I don't think that a CQM comparison can be done because the difference like I said is so neglegable.

[Sharktooth]

i can clearly see the differences, however look at the artifacts (ringing and blocking) and details (both on still and moving objects) in a frame-by-frame comparison.
Hint: There are 2 or 3 clips that are clearly better than the others.

[Soulhunter]

Originally posted by Sharktooth

Hint: There are 2 or 3 clips that are clearly better than the others.

Really ???

Deny myself to make a comment @ this... :D :rolleyes: ;)


Bye

[Sharktooth]

Oh well... i was generous saying "2 or 3"... :D

[*.mp4 guy]

Yes they are all very similar, but there are some that i think were noticibly better and a few that were a bit worse(some were less blocky on the sword and the arm in the begining of sample 1 in my opinion). There was practically no detail in any of them. well thats my 1/2 cent.

[Audionut]

Sorry I haven't been around lately, and haven't had a chance to reply to the other thread.

IMHO you should have used a chapter either side of chapter 19.
Chapter 19 requires a HIGH bitrate for decent quality. Chapters either side require substantially less bitrate.
Would have made xvid's job alot easier at encoding the samples at 900kbps. while maintaing good quality.


Removegrain does the same shit as undot, just better.
Get a sample clip, and undot it at a const. quant.
Now change undot for removegrain() and const. quant encode it.
Note the difference in filesize!

AFAI Concerned this sample at 900kbps needs to be resized.

My script recommendation, although to late is this.

LoadPlugin("C:\DGDecode.dll")
LoadPlugin("C:\Unfilter.dll")
Loadplugin("C:\removegrain.dll")
LoadPlugin("C:\FluxSmooth.dll")

Mpeg2Source("C:\gladiator.d2v")

removegrain()
fluxsmootht()
Crop(8,76,-8,-70)

Lanczos4resize(704,288)
removegrain()
unfilter(-5,-5)


Note:the reason for the largeish crop left & right is this.
1.Removegrain and fluxsmooth do not process the very edges of frames.
2.maintains mod16 width

[Teegedeck]

Originally posted by Sharktooth
i can clearly see the differences, however look at the artifacts (ringing and blocking) and details (both on still and moving objects) in a frame-by-frame comparison.
Frame-by-frame comparisons are VERBOTEN this once! :D

The idea to compare sample clips instead of frames was precisely about having a more realistic comparison. Usually, we don't watch a movie frame-by-frame. So if a viewer cannot tell the difference by viewing the samples, then there simply is no difference worth mentioning.

Also I doubt whether it makes sense trying to judge which artifact looks better than the other (blockfest beauty competition?). Or whether ringing looks better than flickering blocks or something similar. Both are annoying and should lead to a bad mark IMHO. :o

@Soulhunter: I'm having problems with the relative rating system. I cannot attain a 'worst', 'average' or 'best' mark if all I can relate to are the samples themselves because I cannot tell enough differences to spread the marks across the whole span of marks from 1 to 10. I take it you actually meant something in a manner like this: 1= blockfest, 2=unbearable, 3=hardly bearable 4=very annoying, 5=annoying, 6=mildly annoying, 7=not true to the original but not annoying, 8=close to the original, 9=hard to tell from the original, 10=transparent. Hm, I'd need the original to compare it to for that. Tough.
How about: 1= blockfest, 2=unbearable, 3=hardly bearable 4=very annoying, 5=annoying, 6=mildly annoying, 7=not pleasing but not annoying, 8=almost pleasing to watch, 9=almost entirely pleasing to watch, 10=pleasing to the eye.

:-/

[Sharktooth]

Originally posted by Audionut
Removegrain does the same shit as undot, just better.
Filtering was discussed long time ago but we wanted to remove noise and preserve film grain and details. The only alternative was "No filtering at all".
FluxSmoothT and RemoveGrain were tested and did not produced the expecet results.
Unfilter(-5,-5) is out of place because blurring = removing details.
Resizing was also dropped in favour of AR signaling.

Originally posted by Teegedeck
Frame-by-frame comparisons are VERBOTEN this once! :D

The idea to compare sample clips instead of frames was precisely about having a more realistic comparison. Usually, we don't watch a movie frame-by-frame. So if a viewer cannot tell the difference by viewing the samples, then there simply is no difference worth mentioning.
Maybe a Frame-by-frame comparison is not necessary but evaluating quality or spotting details/defects in a moving clip is quite difficult while it's easy to spot moving artifacts...

Originally posted by Teegedeck
Also I doubt whether it makes sense trying to judge which artifact looks better than the other (blockfest beauty competition?). Or whether ringing looks better than flickering blocks or something similar. Both are annoying and should lead to a bad mark IMHO. :o
I meant to look at the quantity of artifacts and how much they annoy your eyes. There are samples with more annoying artifacts than others...

[Teegedeck]

Results sent.

[Sharktooth]

I voted too...

[Teegedeck]

BTW, the second clip is a really high-motion one. I casted separate votes for the clip at normal speed and at 1/4 speed. And I oriented them by the rating system I suggested above. I'd say if we were strict about reproducing 'normal' viewing-conditions we'd have to question whether comparing such clips makes sense.

@Sharktooth: Of course I understand what you mean but the comparison gets more problematic with rising compression.
First we haven't got an original to compare the clips to - which is a technical problem and can't be helped - then we have the clips getting farther and farther away from that absent original's look as compression gets higher. And with this, verdicts about quality become more and more verdicts about which form of encoding errors one prefers over others. What is worse, more smoothing or some more blocks? A matter of common or individual taste. A form of taste which is 'artificial' or 'synthetic' IMHO in a sense that we normaly perceive a video-image as good if it looks life-like - i.e. close to the thing it depicts.
Such a quality of similarity is the thing we get very far detached from with such strong compresson. And 'similarity to an (absent) original' becomes a very vague notion by then. That is why I find it difficult to make big distinctions between the samples.

[Sharktooth]

Oh, well, i have it, but it's 180Mbytes...

[Teegedeck]

The more important aspect is the copies becoming too dissimilar from the original, anyway. Not that it stopped me from expressing _my_ taste, with all its vagueness.

[Soulhunter]

Wow, already 3 ppl voted... :)

Big thanks !!!

[708145]

Wow, already 3 ppl voted...

Big thanks !!!

Make that 4.

bis besser,
Tobias

[kurt]

kurt voted too :D

cya

[JarrettH]

I'm working on it....:p
Thanks so much for the effort. I remember encoding Alien 3 SE using a bitrate of 910kbps with AC3 audio (2CD) and the movie was 2hrs 25min. This should be an interesting comparison then (Gladiator is 10min shorter).;)

[Soulhunter]

Nice, 7 votes so far... :cool:


Bye

[((( atom )))]

i am currently downloading the files but as i read from the thread the bitrate used on the clips was rather low..

ok, there are some matrices designed for low bitrates and these will look rather fine i expect, but what about the high br ones?
i always encode at a compressibility above 80%, targeting for either 1400mb or 2gb, if it isn't enough. bytes _are_ cheap meanwhile..

i also have a lare collection of mostly old, meaning very noisy, movies.

so all in all - and i guess i am not alone here - seing the matrices perform at higher bitrates would be very interesting.

comments, please..

[((( atom )))]

hmm, i just wiewed the clips shortly and find it very hard to build an opinion..

i have the feeling the comparision actually is "which matrix produces the least artifacts" wich also is interesting for the mentioned lower bitrate targets.

"wich matrix preserves the most detail" for instance looks like a different story for me.

personally i would suggest to offer more test aiming at different questions: least artifacts and most detail.

the ideal matrix will always be a compromise between thes two, right? that is then where taste comes into play...

it was mentioned above that comparing single frames would be verboten, i am sorry but i disagree :(
when i watch a scene, i can not se everything at once and i will see different things every time again, especially when the scene is of high motion nature.
it is extremely difficult judging a running movie, that is why so many people are satisfied with low bitrates, i believe.
also i experienced many times, that a movie, wich i thought was of decent quality on a given playout set, suddenly looked worse lets say on a projector, due to the fact that i could follow small details more easily even through motion because they were a lot bigger and easier to keep up with.

if the frame is good, surely the running film will be.

[Soulhunter]

Originally posted by ((( atom )))


so all in all - and i guess i am not alone here - seing the matrices perform at higher bitrates would be very interesting.

comments, please...

As discussed in the preparation thread (http://forum.doom9.org/showthread.php?s=&threadid=82022), there will be a 1600kbps and 2200kbps round... ;)


Originally posted by ((( atom )))

hmm, i just wiewed the clips shortly and find it very hard to build an opinion...


My suggestion...

- First, try to separate between the "very good" and the "very bad" ones

- Now give the best sample 10 points and the worst sample 1 point

- Finally, try to judge the rest in contrast to the best/worst one


Bye

[708145]

- First, try to separate between the "very good" and the "very bad" ones
- Now give the best sample 10 points and the worst sample 1 point
- Finally, try to judge the rest in contrast to the best/worst one


Oops! :eek: Then you should scale all my results. I used a different scale.

bis besser,
T0B1A5

[Soulhunter]

Originally posted by 708145

Oops! :eek: Then you should scale all my results. I used a different scale.

bis besser,
T0B1A5

Yap, Ive scaled them... ;)

Simply used multiplication to get the min/max = 1/10 values !!!


EDIT1:

Think next time I should include this under "how to vote" !?!

Thought the "1=worst/10=best" would be enough... :rolleyes:


EDIT2:

Ok, Ive added the suggestion to my first post !!!


Bye

[((( atom )))]

Originally posted by Soulhunter
[B]As discussed in the preparation thread (http://forum.doom9.org/showthread.php?s=&threadid=82022), there will be a 1600kbps and 2200kbps round... ;)ok, didn't know that thread. i will look it up.

[Soulhunter]

Originally posted by ((( atom )))

ok, didn't know that thread. i will look it up.

Ehm, what I remember you have posted in this thread yesterday !!!

I know, sometimes the huge amount of threads is confusing... :D


Bye

[Teegedeck]

I'd like to voice some protest against this 'upscaling'. :o

That method artificially creates clear winners and loosers where actually the participants in this test found it very hard to find differences at all, let alone judge them.

While your method will even overemphasize a 'clarity' which wasn't there, subjective tests actually don't give clear results; on the contrary such tests leave a broad room for statistical error and uncertainty. Which is why the statistical analysis of a test is perhaps even more of a task than its preparation. If you look at the presentation of the results of Roberto's listening tests, as an example of good practice, only if two test samples' scores differ enough to exclude statistical error as a reason for that difference, one of the test samples can be said to have scored better than the other with an acceptable level of certainty (in Roberto's listening tests that means with 95% confidence). See here (http://www.rjamorim.com/test/32kbps/results.html).

Here's a little link on statistical analysis (http://davidmlane.com/hyperstat/index.html).

I don't ask you to perform an ANOVA analyis (there probably won't be enough results to justify performing one, anyway) or anything like that but I urge you to give up stretching those results over the 1-10 spectrum. If test participants mean to say 'those samples all look that bad, there hardly is a difference', then you shouldn't try to create the image that there were big differences.

[Soulhunter]

Originally posted by Teegedeck

I'd like to voice some protest against this 'upscaling'. :o

That method artificially creates clear winners and losers where actually the participants in this test found it very hard to find differences at all, let alone judge them.

While your method will even overemphasize a 'clarity' which wasn't there, subjective tests actually don't give clear results; on the contrary such tests leave a broad room for statistical error and uncertainty. Which is why the statistical analysis of a test is perhaps even more of a task than its preparation. If you look at the presentation of the results of Roberto's listening tests, as an example of good practice, only if two test samples' scores differ enough to exclude statistical error as a reason for that difference, one of the test samples can be said to have scored better than the other with an acceptable level of certainty (in Roberto's listening tests that means with 95% confidence). See here (http://www.rjamorim.com/test/32kbps/results.html).

Here's a little link on statistical analysis (http://davidmlane.com/hyperstat/index.html).

I don't ask you to perform an ANOVA analysis (there probably won't be enough results to justify performing one, anyway) or anything like that but I urge you to give up stretching those results over the 1-10 spectrum. If test participants mean to say 'those samples all look that bad, there hardly is a difference', then you shouldn't try to create the image that there were big differences.

Generally I agree with you... :)

But 708145 "authorized" me in the pm to scale his results !!!

Anyhow, I will study the info you gave me before resuming discussion... ;)


Thanks n' Bye

[Teegedeck]

Well, that's all I could ask; it's your test! :)

[Soulhunter]

@ Teegedeck

PM is on the way !!!

Btw, Ive discovered that this GMX hosting has a traffic limit of 2GB... :scared:


Bye

[Soulhunter]

Ok, we (actually Teegedeck) found a easy solution...

If you upscale for example my results to your relative scale, this would express my judgment of the matrices' quality correctly, though with large overemphasis. If you now downscale all results again to a range of 1-5, instead of 1-10, (still on a relative scale!) you at least deemphasize those differences again, especially if you express the margin of error in a similar manner as Roberto when you present the results.

[Teegedeck]

Just noticed that it doesn't make much sense the way I expressed it. Must of course read: scale up all results to 1-10, then divide by 2.

[Soulhunter]

Originally posted by Teegedeck

Just noticed that it doesn't make much sense the way I expressed it. Must of course read: scale up all results to 1-10, then divide by 2.

Hmm, but thats the way I understood it... :confused:

Scale the results -> 1-10 -> 1-5 !!!

Thats a division of 2, no...


Bye

[CruNcher]

Ok rated those 2 samples, but i have to say im not happy with the Content that was chosen the 2nd clip was ok but for me the 1st clip is nothing else it shows high or ok maybe middle motion but it´s definatly not low motion like a direct conversation shot, i´m a little upset about this :(

[Soulhunter]

Originally posted by CruNcher

Ok rated those 2 samples, but i have to say im not happy with the Content that was chosen the 2nd clip was ok but for me the 1st clip is nothing else it shows high or ok maybe middle motion but it´s definatly not low motion like a direct conversation shot, i´m a little upset about this :(

Imo its mainly mid/low motion !!!

Its hard to make everyone happy... :(


Bye

[Teegedeck]

Originally posted by Soulhunter
Hmm, but thats the way I understood it... :confused:

Scale the results -> 1-10 -> 1-5 !!!

Thats a division of 2, no...


Bye
Yeees, you understood what I meant. After all it seems I didn't express it so badly after all. :p

[Arcon]

it would be nice if you could also include the quality factor in bits per pixel for the final results. i think its much easier to find the right matrix based on the bpp instead of the bitrate/resolution/framerate given (assuming that we won't have one matrix that wins all bitrate contests ;)).

[len0x]

Originally posted by Arcon
i think its much easier to find the right matrix based on the bpp instead of the bitrate/resolution/framerate given

Do ppl still believe that? Both ways are not appropriate to decide on matrix...

[Arcon]

Originally posted by len0x
Do ppl still believe that? Both ways are not appropriate to decide on matrix...
but i have to choose one way :( i can't wait for an inspiration particle to solve this problem and a bpp seems to me more general than a bitrate. if there is an error in reasoning please tell me.

[Teegedeck]

This test was precisely about not having some bogus, artificial, calculated or computed result but about what actually and simply looks best at a given compression ratio (it is not about bitrate, really).

If there are enough participants in this and the upcoming tests to make it statistically valid, you'll have for the first time a sensible lead about which matrix to choose at which bitrate. Of course even then it still is up to your personal taste and differences in the look of your source, so you should always check whether your taste goes with the taste of the crowd.

[Sharktooth]

Choosing the right matrix depends on the source quality and type (DVD movie? DV capture? Anime? etc...), in how many bframes the codec will place at the end of the encode, how many keyframes the codec will place at the end of the encode, the bitrate you intend to use, the filters you will apply (and the amount of noise that remains after filtering), the amount of motion (low-mid-high) and some other minor factors.
If i forgot something plz feel free to add it to the list...

[Soulhunter]

Now we have 8.5 votes... :)


Bye

[Arcon]

Originally posted by Teegedeck
This test was precisely about not having some bogus, artificial, calculated or computed result but about what actually and simply looks best at a given compression ratio (it is not about bitrate, really).
maybe i said something wrong but i don't want to alter the test or the test-methods at all. i'd just like to see the bpp in the final result. so instead of 'ranking in the 900kbps class' i'd like to see something like 'ranking in the 900kbps class (0.284bpp)' so it's easier to estimate what scenario equals my current encoding project the most so that i can pick the right matrix. thats all.

it's just a convenience information so that i don't have to calculate the bpp manually from res/framerate/bitrate.

[Sharktooth]

bpp is not an universal and safe method to determine what matrix to use.
bpp is useful only when certain "variables" match certain values.
For example a high-motion source will need a higher bitrate (and higher bpp) to achieve a good quality encode. A low motion one can achieve a good quality at a much lower bitrate (and bpp).
Using bpp is not as good as using compressibility to estabilish what matrix and/or resolution is needed. Hence there are 2 tests (low motion and high motion) at different bitrates.
Once estabilished what are your personal preferences at those bitrates and motion you have a list of matrices that are good for that scenario.