Invitation to 5th H.264/AVC codecs comparison

[Sagekilla]

G_M_C: How do you tell if one video looks better than another, if you had nothing but those two videos on hand? You'd watch them, and say which one looks better to you. Same idea applies here, only with more restrictions to prevent skewing from having extra knowledge (which video was encoded with what settings, etc).

[G_M_C]

Originally Posted by Sagekilla

G_M_C: How do you tell if one video looks better than another, if you had nothing but those two videos on hand? You'd watch them, and say which one looks better to you. Same idea applies here, only with more restrictions to prevent skewing from having extra knowledge (which video was encoded with what settings, etc).

A "double blind" (*)test/screening you mean ? But you still need a fair number of people to make the test statistically sigificant.

(*)[funny]However the "Blind" part of it doesnt seem appropriate [/funny]

[Sagekilla]

No, doesn't necessarily have to be double blind. Yes, it is helpful to have the extra degree but you can get away with the testers knowing which video is which.

[Sharktooth]

blind test is the way to go for testing quality, since as i've said, metrics are juts... metrics. that means they measure differencies between samples weigthing them in some way. but the fact is, a better metric doesnt represent better visual quality.
lets make an example:
sample A: high quality picture
sample B: mid quality picture
sample C: low quality picture

all pictures represent the same image... just with different quality.
picture B is our source for comparation.

results for metrics>
picture A: low metric.
picture B: highest metric.
picture C: low metric.
conclusion: picture B has the highest metric. picture A even if it is the highest quality picture of the pack, is rated "low" coz it differs from picture B that is our source for comparation and has a low metric...

results for human eye>
picture A: highest quality
picture B: mid quality
picture C: worst quality
conclusion: if you look at the pictures with your eyes you will have no doubt that A is the best one... while metrics are telling you something else...

so, all in all, this is the demonstration that metrics do not represent quality and you must never trust them if you compare different encoders.
that's also the reason why elecard has higher psnr than x264 but x264 produces a much higher visual quality (and some "smart" people produce docs with graphs but without any kind of visual comparison...).

[Sagittaire]

Blind test don't work for video simply it's really hard to evaluate overall quality for long sequence (simple example is VBR vs CBR for the same codec). In practice blind test for video are generally less accurate than metric. It's like that. Speak about that with developper ... they don't trust generaly blind test for video. I have never see even here on doom9 really reliable blind test ...

[Sagittaire]

that's also the reason why elecard has higher psnr than x264 but x264 produces a much higher visual quality (and some "smart" people produce docs with graphs but without any kind of visual comparison...).

It's simply false ... for all affirmation ... lol

1) x264 without psy (AQ and SSD) produce better metric than Mainconcet SDK without psy (AQ and FGO) and with relative large margin. x264 is in fact the best in area for metric test ... ;-)
x264 use simply psy tools by default and not Mainconcept.

2) Some people even here on doom9 forum find that Mainconcept SDK produce better visual result than x264. The principal particulary for HVS is it's ... subjective ... and by definition you can't contradict that. In fact IMO x264 and Mainconcept SDK produce in most case comparable visual result and IMO you can notice real difference only for really particular sequences at really low quality encoding.

3) graph and metric are usefull for particular test like speed test simply because you must have really reliable quality reference. Make speed test with subjective comparison and without metric test is simply impossible. No way. It's like that. No possible discution here. Final point.

[Sharktooth]

Originally Posted by Sagittaire

Blind test don't work for video simply it's really hard to evaluate overall quality for long sequence (simple example is VBR vs CBR for the same codec). In practice blind test for video are generally less accurate than metric. It's like that. Speak about that with developper ... they don't trust generaly blind test for video. I have never see even here on doom9 really reliable blind test ...

in practice blind tests made lame the best mp3 encoder out there.
that is the proof blind tests work for encoders development.

Originally Posted by Sagittaire

It's simply false ... for all affirmation ... lol

1) x264 without psy (AQ and SSD) produce better metric than Mainconcet SDK without psy (AQ and FGO) and with relative large margin. x264 is in fact the best in area for metric test ... ;-)
x264 use simply psy tools by default and not Mainconcept.

2) Some people even here on doom9 forum find that Mainconcept SDK produce better visual result than x264. The principal particulary for HVS is it's ... subjective ... and by definition you can't contradict that. In fact IMO x264 and Mainconcept SDK produce in most case comparable visual result and IMO you can notice real difference only for really particular sequences at really low quality encoding.

3) graph and metric are usefull for particular test like speed test simply because you must have really reliable quality reference. Make speed test with subjective comparison and without metric test is simply impossible. No way. It's like that. No possible discution here. Final point.

1) there are no doubts x264 - if we talk about quality - is better than mainconcept h.264 encoder

2) psy opts are hardly subjective. if you have a source with grain, you expect the encoder to keep the grain... otherwise you filter it out before encoding. another point is artifacts. you dont want them if they're not on the source

3) i disagree since, as i said, metrics do not represent quality in any way.

[Shinigami-Sama]

metrics are good for sanity tests
for quality I'll stick to my eyes

[Sagekilla]

Metrics are useful for comparing how similar your image is to the source image. That's about it. Unfortunately it's metric optimal to do a number of visually poor things like prefer a soft image.

With that said, they can be useful for providing a rough speed vs quality tradeoff in an encoder, or among encoders, as long as psy optimizations are disabled. But, any serious quality comparison should be done with psy-rd and subjectively, not with metrics.

[Sagittaire]

in practice blind tests made lame the best mp3 encoder out there.
that is the proof blind tests work for encoders development.

Video don't work like Audio. Audio algo are massively based on psychoacoustic model and not video codec. Make psnr test with audio (direct comparison between source source and encoding) is non sens here but not for video. Anyway mp3 codec is simply mathematical algorithm and you can always evaluate efficacity with metric if you use metric based on psychoaccoustic model. Try to find good blind test for video and speak after about that.

1) there are no doubts x264 - if we talk about quality - is better than mainconcept h.264 encoder

No doubt only for your eyes. No doubt for psnr (delta is generaly at 0.3-0.5 dB in my test). IMO with good setting (aka psy tools) mainconcept provide generaly similar quality to x264.

2) psy opts are hardly subjective. if you have a source with grain, you expect the encoder to keep the grain... otherwise you filter it out before encoding. another point is artifacts. you dont want them if they're not on the source

Not always. x264 psy tools produce catastrophic result in some case (like for anime for example). For many people grain is artefact by itself and for these people codec with low grain preservetion produce good visual result and better visual experience than the source ...

3) i disagree since, as i said, metrics do not represent quality in any way.

It's false. You don't know simply how work metric and how psy tools change metric. If you don't trust metric you can't simply make speed comparison. It's like that. Final point.

[Gabriel_Bouvigne]

Originally Posted by Sagekilla

No, doesn't necessarily have to be double blind. Yes, it is helpful to have the extra degree but you can get away with the testers knowing which video is which.

Subjective testing MUST be double blind. You can not trust results of people who know which codec is used for a specific candidate.

Originally Posted by Sagittaire

Blind test don't work for video simply it's really hard to evaluate overall quality for long sequence (simple example is VBR vs CBR for the same codec). In practice blind test for video are generally less accurate than metric. It's like that. Speak about that with developper ... they don't trust generaly blind test for video. I have never see even here on doom9 really reliable blind test ...

DBT work for video. There is no more issue related to long content within video than within audio tests, and your example of CBR vs VBR for the same codec is not more an issue for video codecs than it is for audio codecs.
The main point is simply that objective grading is way cheaper and faster than subjective grading, thus as long as you can manage to reliably use objective grading it is a better choice to use it.

Originally Posted by Sagittaire

Video don't work like Audio. Audio algo are massively based on psychoacoustic model and not video codec. Make psnr test with audio (direct comparison between source source and encoding) is non sens here but not for video. Anyway mp3 codec is simply mathematical algorithm and you can always evaluate efficacity with metric if you use metric based on psychoaccoustic model. Try to find good blind test for video and speak after about that.

This is because until now, there were enough progress made on video encoder just by using brute force. Pure brute force (like extensive RD) and mathematical improvements are less risky that any psychoacoustic or psychovisual trick.
In no way this means that audio coding is more inherently dependent on psy models than video coding is.

Now, about the point that mp3 would be a simple mathematical algorithm, that is not more the case for mp3 than for h.264.
Objective grading of any psy-model based encoder (audio or video) requires the use of another psy-model. The big caveat is that the model used for evaluation must then be fully reliable. If it was fully reliable, that would mean that a fully reliable model would exist, and then you could simply put that model within your encoder and that would result into a fully reliable psy-model based encoder.
I am sorry, but until now such a beast doesn't exist.
State of the art objective grading of audio based on a psymodel only achieve about 60% of correlation with human results. While a 60% correlation is a significant achievement, it is still far from being able to totally replace DBT.

Something like a simple naive PSNR computation, which is throw away by a simple +1 to every sample or any ROI based optimisation, can not be considered as an overall good quality estimator.
It can only be used in some specific, controlled, and limited cases.

[Dark Shikari]

Originally Posted by Gabriel_Bouvigne

Subjective testing MUST be double blind. You can not trust results of people who know which codec is used for a specific candidate.

Even that may not be enough. An experienced and well-informed viewer can almost certainly tell an x264 from a Mainconcept stream, for example, especially at low bitrates.

You'd basically have to restrict yourself to viewers and testers who are unable to identify which stream comes from which encoder.

[Gabriel_Bouvigne]

Originally Posted by Dark Shikari

Even that may not be enough. An experienced and well-informed viewer can almost certainly tell an x264 from a Mainconcept stream, for example, especially at low bitrates.

You'd basically have to restrict yourself to viewers and testers who are unable to identify which stream comes from which encoder.

That's true, as if you keep those subjects your test is not double blind anymore (test subject being able to know which encoder is used)

Btw, for those who think that DBT are useless for video coding, ITU doesn't seem to agree on that:
ITU-R BT.500-11
ITU-R BT.700

[Chengbin]

Originally Posted by Dark Shikari

Even that may not be enough. An experienced and well-informed viewer can almost certainly tell an x264 from a Mainconcept stream, for example, especially at low bitrates.

You'd basically have to restrict yourself to viewers and testers who are unable to identify which stream comes from which encoder.

Wow.

I'm curious about your "especially at low bitrates" comment. What characteristics are there? Is one better than the other or something?

[Esurnir]

Originally Posted by Chengbin

Wow.

I'm curious about your "especially at low bitrates" comment. What characteristics are there? Is one better than the other or something?

--Touhou-RD 1:0

The difference is striking.

[Sagittaire]

Originally Posted by Esurnir

http://mirror05.x264.nl/Dark/website/compare.html

The difference is striking.

No ... it's more subtil ... and this comparison doen't mean anything because it's possible to obtain really better result for Mainconcept SDK, Nero or VP7.
The only good test is for touhou (x264 is able to reproduce very well static part with high complexity) but it's a really and too specific sample.

[Shinigami-Sama]

if that's not obvious across the board its time to invest in some glasses, and a new monitor

[Dark Shikari]

Originally Posted by Chengbin

Wow.

I'm curious about your "especially at low bitrates" comment. What characteristics are there? Is one better than the other or something?

AQ and especially psy-RD have a very distinct visual signature, especially when one can compare to an encode without them.

[Sharktooth]

well, gabriel already said everything that was needed to say.
however:

o doubt only for your eyes. No doubt for psnr (delta is generaly at 0.3-0.5 dB in my test). IMO with good setting (aka psy tools) mainconcept provide generaly similar quality to x264.

this is meaningless since i stated "if we speak about quality", and a metric measure is not about quality...
and your "final point" is pointless since there is no perfect mathematical model of the human eye and if you think about psy opts, they exists since metrics cant replace visual perception...

[Sagittaire]

Originally Posted by Sharktooth

this is meaningless since i stated "if we speak about quality", and a metric measure is not about quality...
and your "final point" is pointless since there is no perfect mathematical model of the human eye and if you think about psy opts, they exists since metrics cant replace visual perception...

psy tool for x264 are metric tools by itself ... lol

- AQ measure complexity block
- psy rdo use simply SSD metric for choose better RD decision.

I have impression when I read your thread that psy for x264 is like magical tools. Psy tools use simply other metric tools for better complexity/texture detection.