Invitation to 5th H.264/AVC codecs comparison

[DmitriyV2]

Dear all,

Sorry for delay, but we move forward with 5th H.264 codecs comparison.

Please forward this information to video codecs developers!

Moscow State University Graphics & Multimedia Laboratory starts next
5-th H.264 codecs comparison. There is some information about it
below. Let us know if any questions.

Main issues:

* COMPARISON WILL BE SOON! (Sorry for delay).

* We are planning to include new codecs that did not participate in
previous comparison by chosing presets for them ourself.
We are hope to include more codecs into this comparison.

===========================================================================
CALL FOR MPEG4-AVC/H.264 CODECS
Fifth H.264 video codec comparison
For practical researchers and developers in the field of high-end video compression
===========================================================================

Scope of Test
-------------

* Encoding time, speed/quality analysis
* Objective quality measurements (PSNR, SSIM, Average Advantage, etc.)
* Analysis of averaged objective results
* Leaders in different areas
* Special analysis of codec parts

Important Dates
---------------

February, 20 - Deadline for preliminary receipt of a H.264 codecs
February, 27 - Deadline for receipt of a H.264 codec with required presets
March, 16 - Deadline for settling technical problems with codec’s functioning
April, 7 - Draft version of report that will be sent to all participants
April, 14 - Deadline for reception of comments to the draft
April, 28 - Comparison report release


Enhancements in comparison to Previous H.264/AVC Comparison
-----------------------------------------------------------

* We are planning to include new codecs that did not participate in
previous comparison by chosing presets for them ourself. For that
task we will use option analysis.

Anyway we will be glad to have a direct contact with codec
developers. The main benefit of direct participation for developers
is receiving Pro version of comparison free of charge.

* Codecs options analysis (see example at Options Analysis of MPEG-4
AVC/H.264 Codec x264)
http://compression.ru/video/codec_comparison/x264_options_analysis_08_en.html

* New type of special analysis for codecs
* Using natural sequences' special modification
* Using synthetic sequences
* Separate analysis of codecs main subsystems

* New sequences


Developer Deliverables
----------------------

The following deliverables should be provided by each developer:
* Codec files (CLI executable file is preferable)
* Short description of codec parameters
* Codec's presets with mentioning what H.264/AVC profiles are used

The full text of Call for Codecs is available at
http://compression.ru/video/codec_comparison/call_for_codecs_09.html

Variants of Participation
-------------------------

There are two variants for companies to participate in our comparison:

1. Participation for free. All results of your codec will be published,
except special cases of measurements problems due to codec
instability.

2. Private participation. A special report will be prepared only for your
company. This report contains:
* Your codec results and all material from the free version
* Special additional analysis of your codec

If you are interested in the private participation, please contact us
for details.

Useful Links
------------

* Fourth Annual MSU MPEG-4 AVC/H.264 Video Codec Comparison
http://compression.ru/video/codec_comparison/mpeg-4_avc_h264_2007_en.html

* Options Analysis of MPEG-4 AVC/H.264 Codec x264
http://compression.ru/video/codec_comparison/x264_options_analysis_08_en.html

* Subjective Comparison of Modern Video Codecs
http://compression.ru/video/codec_comparison/subjective_codecs_comparison_en.html

Sincerely yours,
Dr. Vatolin

[Dark Shikari]

Let's just say that we have evil plans for our entry in this comparison ;)

Also, your mail server is down and I can't email any of you (message returned with "message too large" for even small messages).

[Cyber-Mav]

Let's just say that we have evil plans for our entry in this comparison ;)

hmmm, threaded lookahead ??

[Chengbin]

Let's just say that we have evil plans for our entry in this comparison ;)

A secret weapon under development that is almost done? Nice!

[IgorC]

I wonder if there is any H.264 encoder comparable to last revision of x264 at high quality settings.
And don't see the reason why x264 is compared on opsnr and ssim when its psy uses more advanced internal metrics.

[Raptus]

And don't see the reason why x264 is compared on opsnr and ssim when its psy uses more advanced internal metrics.
True, but what do you suggest? Doing a decent subjective test for video that shall produce statistically relevant results is quite difficult. A university should have the means to do it, though...
ITU-R has recommendations for that in the BT Series http://www.itu.int/rec/R-REC-BT/e

[Dark Shikari]

One of the problems I'm finding with this comparison so far is that the "fast" preset is more like a "slow" preset and the "slow" preset is more like an "insanely slow" preset. If I can use subme9/trellis2 on the "fast" preset, something is wrong.

[Audionut]

I'd like to do a subjective comparison. But I don't think I'd get enough participation!!

I could spam it at a few forums I guess to increase participation.

[Sagittaire]

One of the problems I'm finding with this comparison so far is that the "fast" preset is more like a "slow" preset and the "slow" preset is more like an "insanely slow" preset. If I can use subme9/trellis2 on the "fast" preset, something is wrong.

"slow" and "fast" are sujective if you don't have quality reference. For example ATI encoder is certainely really fast but produce insanely bad quality.


I wonder if there is any H.264 encoder comparable to last revision of x264 at high quality settings.
And don't see the reason why x264 is compared on opsnr and ssim when its psy uses more advanced internal metrics.

It's easy to desactivate psy option like AQ and SSD ...


I'd like to do a subjective comparison. But I don't think I'd get enough participation!!

I could spam it at a few forums I guess to increase participation.

Expenrience from dev show that subjective comparison produce not really good result.

[Audionut]

Expenrience from dev show that subjective comparison produce not really good result.

How can it produce not a good result?


Since x264's recent psy optimizations have proven quite definitively that one can achieve greatly improved visual quality while decreasing both PSNR and SSIM, will their be a subjective comparison as well as an objective one? In addition, what about the fact that encoders have different settings optimized for PSNR, SSIM, and visual quality,
respectively?

[Chengbin]

One of the problems I'm finding with this comparison so far is that the "fast" preset is more like a "slow" preset and the "slow" preset is more like an "insanely slow" preset. If I can use subme9/trellis2 on the "fast" preset, something is wrong.

LOL! I think x264 is fast enough on that machine to use any insanely slow settings and still qualify for the fps requirement. This comparison will be VERY interesting.

I was reading the 2006 video codec comparison, and x264, at r544, fared up very well. This comparison could be very funny, as x264 will completely own the other codecs.

It is too bad that they don't have an i7 rig to test these codecs. x264 will have even more of an advantage.

[Raptus]

How can it produce not a good result?
He is referring to the quality/accuracy of the conclusion of those tests, not how well x264 might do in it.

I was just reading through the old test papers and found quite a few interesting bits. For instance, i'd very much like to get a current version of this (http://compression.ru/video/codec_comparison/pdf/x264_options_analysis_08.pdf) paper. Doesn't cover all settings and isn't based on subjective metric but gives you a good overview of the speed/quality impact of different settings.

[Sagittaire]

This test (http://compression.ru/video/codec_comparison/pdf/x264_options_analysis_08.pdf) is really impressive ...

Here my little (and old) comparison with similar protocole ...

http://jfl1974.free.fr/Sample/Speed.PNG

Anyway for this old test aku say that i8x8,i4x4,p8x8 is the best for quality/speed

[IgorC]

No ssim no psnr are required. They are obsolete.
I thought it was already time to think different.

[Sagittaire]

No ssim no psnr are required. They are obsolete.
I thought it was already time to think different.

Unfortunaly there are not other way for make objective quality test or speed test.
Moreover SSIM and PSNR work very well if you work with high delta.
Moreover psy optimisation don't always work very well for all source or ... for all eyes.

[G_M_C]

Let's just say that we have evil plans for our entry in this comparison ;)

Also, your mail server is down and I can't email any of you (message returned with "message too large" for even small messages).

Since the "competition" should be well underway by now ....

... how did the EVIL PLAN work out Dark Shikari ?

And will the code of that evil plan be usable by us ?
/me is curious as to what the evil plan entailed

[Dark Shikari]

Since the "competition" should be well underway by now ....

... how did the EVIL PLAN work out Dark Shikari ?

And will the code of that evil plan be usable by us ?
/me is curious as to what the evil plan entailed The code of the evil plan was (mostly) already committed (http://git.videolan.org/?p=x264.git;a=commit;h=2dca5f5413051a26cbba4e20f3c77ff69b694ba3) :p

[Sharktooth]

i hope MSU labs changed the test methods since, despite what Sagittaire says, metrics do not represent quality at all.

[G_M_C]

The code of the evil plan was (mostly) already committed (http://git.videolan.org/?p=x264.git;a=commit;h=2dca5f5413051a26cbba4e20f3c77ff69b694ba3) :p

Ahh, i see; Quite a patch, that was. I'm still enjoying those improvements :)

i hope MSU labs changed the test methods since, despite what Sagittaire says, metrics do not represent quality at all.

True. But i wonder how you'd objectively measure differences in encoders without using "hard numbers". Or did you have other ways of measuring in mind ?

[Sharktooth]

blind tests.

[Sagekilla]

G_M_C: How do you tell if one video looks better than another, if you had nothing but those two videos on hand? You'd watch them, and say which one looks better to you. Same idea applies here, only with more restrictions to prevent skewing from having extra knowledge (which video was encoded with what settings, etc).

[G_M_C]

G_M_C: How do you tell if one video looks better than another, if you had nothing but those two videos on hand? You'd watch them, and say which one looks better to you. Same idea applies here, only with more restrictions to prevent skewing from having extra knowledge (which video was encoded with what settings, etc).

A "double blind" (*)test/screening you mean ? But you still need a fair number of people to make the test statistically sigificant.

(*)However the "Blind" part of it doesnt seem appropriate :D

[Sagekilla]

No, doesn't necessarily have to be double blind. Yes, it is helpful to have the extra degree but you can get away with the testers knowing which video is which.

[Sharktooth]

blind test is the way to go for testing quality, since as i've said, metrics are juts... metrics. that means they measure differencies between samples weigthing them in some way. but the fact is, a better metric doesnt represent better visual quality.
lets make an example:
sample A: high quality picture
sample B: mid quality picture
sample C: low quality picture

all pictures represent the same image... just with different quality.
picture B is our source for comparation.

results for metrics>
picture A: low metric.
picture B: highest metric.
picture C: low metric.
conclusion: picture B has the highest metric. picture A even if it is the highest quality picture of the pack, is rated "low" coz it differs from picture B that is our source for comparation and has a low metric...

results for human eye>
picture A: highest quality
picture B: mid quality
picture C: worst quality
conclusion: if you look at the pictures with your eyes you will have no doubt that A is the best one... while metrics are telling you something else...

so, all in all, this is the demonstration that metrics do not represent quality and you must never trust them if you compare different encoders.
that's also the reason why elecard has higher psnr than x264 but x264 produces a much higher visual quality (and some "smart" people produce docs with graphs but without any kind of visual comparison...).

[Sagittaire]

Blind test don't work for video simply it's really hard to evaluate overall quality for long sequence (simple example is VBR vs CBR for the same codec). In practice blind test for video are generally less accurate than metric. It's like that. Speak about that with developper ... they don't trust generaly blind test for video. I have never see even here on doom9 really reliable blind test ...

[Sagittaire]

that's also the reason why elecard has higher psnr than x264 but x264 produces a much higher visual quality (and some "smart" people produce docs with graphs but without any kind of visual comparison...).

It's simply false ... for all affirmation ... lol

1) x264 without psy (AQ and SSD) produce better metric than Mainconcet SDK without psy (AQ and FGO) and with relative large margin. x264 is in fact the best in area for metric test ... ;-)
x264 use simply psy tools by default and not Mainconcept.

2) Some people even here on doom9 forum find that Mainconcept SDK produce better visual result than x264. The principal particulary for HVS is it's ... subjective ... and by definition you can't contradict that. In fact IMO x264 and Mainconcept SDK produce in most case comparable visual result and IMO you can notice real difference only for really particular sequences at really low quality encoding.

3) graph and metric are usefull for particular test like speed test simply because you must have really reliable quality reference. Make speed test with subjective comparison and without metric test is simply impossible. No way. It's like that. No possible discution here. Final point.

[Sharktooth]

Blind test don't work for video simply it's really hard to evaluate overall quality for long sequence (simple example is VBR vs CBR for the same codec). In practice blind test for video are generally less accurate than metric. It's like that. Speak about that with developper ... they don't trust generaly blind test for video. I have never see even here on doom9 really reliable blind test ...
in practice blind tests made lame the best mp3 encoder out there.
that is the proof blind tests work for encoders development.

It's simply false ... for all affirmation ... lol

1) x264 without psy (AQ and SSD) produce better metric than Mainconcet SDK without psy (AQ and FGO) and with relative large margin. x264 is in fact the best in area for metric test ... ;-)
x264 use simply psy tools by default and not Mainconcept.

2) Some people even here on doom9 forum find that Mainconcept SDK produce better visual result than x264. The principal particulary for HVS is it's ... subjective ... and by definition you can't contradict that. In fact IMO x264 and Mainconcept SDK produce in most case comparable visual result and IMO you can notice real difference only for really particular sequences at really low quality encoding.

3) graph and metric are usefull for particular test like speed test simply because you must have really reliable quality reference. Make speed test with subjective comparison and without metric test is simply impossible. No way. It's like that. No possible discution here. Final point.
1) there are no doubts x264 - if we talk about quality - is better than mainconcept h.264 encoder

2) psy opts are hardly subjective. if you have a source with grain, you expect the encoder to keep the grain... otherwise you filter it out before encoding. another point is artifacts. you dont want them if they're not on the source

3) i disagree since, as i said, metrics do not represent quality in any way.

[Shinigami-Sama]

metrics are good for sanity tests
for quality I'll stick to my eyes

[Sagekilla]

Metrics are useful for comparing how similar your image is to the source image. That's about it. Unfortunately it's metric optimal to do a number of visually poor things like prefer a soft image.

With that said, they can be useful for providing a rough speed vs quality tradeoff in an encoder, or among encoders, as long as psy optimizations are disabled. But, any serious quality comparison should be done with psy-rd and subjectively, not with metrics.

[Sagittaire]

in practice blind tests made lame the best mp3 encoder out there.
that is the proof blind tests work for encoders development.

Video don't work like Audio. Audio algo are massively based on psychoacoustic model and not video codec. Make psnr test with audio (direct comparison between source source and encoding) is non sens here but not for video. Anyway mp3 codec is simply mathematical algorithm and you can always evaluate efficacity with metric if you use metric based on psychoaccoustic model. Try to find good blind test for video and speak after about that.


1) there are no doubts x264 - if we talk about quality - is better than mainconcept h.264 encoder

No doubt only for your eyes. No doubt for psnr (delta is generaly at 0.3-0.5 dB in my test). IMO with good setting (aka psy tools) mainconcept provide generaly similar quality to x264.


2) psy opts are hardly subjective. if you have a source with grain, you expect the encoder to keep the grain... otherwise you filter it out before encoding. another point is artifacts. you dont want them if they're not on the source

Not always. x264 psy tools produce catastrophic result in some case (like for anime for example). For many people grain is artefact by itself and for these people codec with low grain preservetion produce good visual result and better visual experience than the source ...


3) i disagree since, as i said, metrics do not represent quality in any way.

It's false. You don't know simply how work metric and how psy tools change metric. If you don't trust metric you can't simply make speed comparison. It's like that. Final point.

[Gabriel_Bouvigne]

No, doesn't necessarily have to be double blind. Yes, it is helpful to have the extra degree but you can get away with the testers knowing which video is which.
Subjective testing MUST be double blind. You can not trust results of people who know which codec is used for a specific candidate.

Blind test don't work for video simply it's really hard to evaluate overall quality for long sequence (simple example is VBR vs CBR for the same codec). In practice blind test for video are generally less accurate than metric. It's like that. Speak about that with developper ... they don't trust generaly blind test for video. I have never see even here on doom9 really reliable blind test ...
DBT work for video. There is no more issue related to long content within video than within audio tests, and your example of CBR vs VBR for the same codec is not more an issue for video codecs than it is for audio codecs.
The main point is simply that objective grading is way cheaper and faster than subjective grading, thus as long as you can manage to reliably use objective grading it is a better choice to use it.

Video don't work like Audio. Audio algo are massively based on psychoacoustic model and not video codec. Make psnr test with audio (direct comparison between source source and encoding) is non sens here but not for video. Anyway mp3 codec is simply mathematical algorithm and you can always evaluate efficacity with metric if you use metric based on psychoaccoustic model. Try to find good blind test for video and speak after about that.
This is because until now, there were enough progress made on video encoder just by using brute force. Pure brute force (like extensive RD) and mathematical improvements are less risky that any psychoacoustic or psychovisual trick.
In no way this means that audio coding is more inherently dependent on psy models than video coding is.

Now, about the point that mp3 would be a simple mathematical algorithm, that is not more the case for mp3 than for h.264.
Objective grading of any psy-model based encoder (audio or video) requires the use of another psy-model. The big caveat is that the model used for evaluation must then be fully reliable. If it was fully reliable, that would mean that a fully reliable model would exist, and then you could simply put that model within your encoder and that would result into a fully reliable psy-model based encoder.
I am sorry, but until now such a beast doesn't exist.
State of the art objective grading of audio based on a psymodel only achieve about 60% of correlation with human results. While a 60% correlation is a significant achievement, it is still far from being able to totally replace DBT.

Something like a simple naive PSNR computation, which is throw away by a simple +1 to every sample or any ROI based optimisation, can not be considered as an overall good quality estimator.
It can only be used in some specific, controlled, and limited cases.

[Dark Shikari]

Subjective testing MUST be double blind. You can not trust results of people who know which codec is used for a specific candidate.Even that may not be enough. An experienced and well-informed viewer can almost certainly tell an x264 from a Mainconcept stream, for example, especially at low bitrates.

You'd basically have to restrict yourself to viewers and testers who are unable to identify which stream comes from which encoder.

[Gabriel_Bouvigne]

Even that may not be enough. An experienced and well-informed viewer can almost certainly tell an x264 from a Mainconcept stream, for example, especially at low bitrates.

You'd basically have to restrict yourself to viewers and testers who are unable to identify which stream comes from which encoder.
That's true, as if you keep those subjects your test is not double blind anymore (test subject being able to know which encoder is used)

Btw, for those who think that DBT are useless for video coding, ITU doesn't seem to agree on that:
ITU-R BT.500-11
ITU-R BT.700

[Chengbin]

Even that may not be enough. An experienced and well-informed viewer can almost certainly tell an x264 from a Mainconcept stream, for example, especially at low bitrates.

You'd basically have to restrict yourself to viewers and testers who are unable to identify which stream comes from which encoder.

Wow.

I'm curious about your "especially at low bitrates" comment. What characteristics are there? Is one better than the other or something?

[Esurnir]

Wow.

I'm curious about your "especially at low bitrates" comment. What characteristics are there? Is one better than the other or something?

--Touhou-RD 1:0 (http://mirror05.x264.nl/Dark/website/compare.html)

The difference is striking.

[Sagittaire]

http://mirror05.x264.nl/Dark/website/compare.html

The difference is striking.

No ... it's more subtil ... and this comparison doen't mean anything because it's possible to obtain really better result for Mainconcept SDK, Nero or VP7.
The only good test is for touhou (x264 is able to reproduce very well static part with high complexity) but it's a really and too specific sample.

[Shinigami-Sama]

if that's not obvious across the board its time to invest in some glasses, and a new monitor

[Dark Shikari]

Wow.

I'm curious about your "especially at low bitrates" comment. What characteristics are there? Is one better than the other or something?AQ and especially psy-RD have a very distinct visual signature, especially when one can compare to an encode without them.

[Sharktooth]

well, gabriel already said everything that was needed to say.
however:
o doubt only for your eyes. No doubt for psnr (delta is generaly at 0.3-0.5 dB in my test). IMO with good setting (aka psy tools) mainconcept provide generaly similar quality to x264.
this is meaningless since i stated "if we speak about quality", and a metric measure is not about quality...
and your "final point" is pointless since there is no perfect mathematical model of the human eye and if you think about psy opts, they exists since metrics cant replace visual perception...

[Sagittaire]

this is meaningless since i stated "if we speak about quality", and a metric measure is not about quality...
and your "final point" is pointless since there is no perfect mathematical model of the human eye and if you think about psy opts, they exists since metrics cant replace visual perception...

psy tool for x264 are metric tools by itself ... lol

- AQ measure complexity block
- psy rdo use simply SSD metric for choose better RD decision.

I have impression when I read your thread that psy for x264 is like magical tools. Psy tools use simply other metric tools for better complexity/texture detection.

[dj_tjerk]

The goal is to be as close to the original as possible right? I don't know how well eyes can compare two videos; I just have a feeling that ears are a lot better at picking out small differences in audio than eyes are at picking out differences in video. Add that eyes only focus on a very small part of the screen (1cm^2 at about 50cm distance). You'd need to compare iamges if you want to compare the quality of the entire frame, but then you get high-motion/slow-motion issues/grain retention/etc..

Comparing images is easier, there have a couple of those here at doom9 before. It's easier to see changes 'happening' when you click an the "original" and it changes to "encode". (And you can do so multiple times).

Anyway, if this comparison judges codecs based on SSIM, then the encoders should just optimize for SSIM. It might give some indication of how much detail (and what kind of) detail it retains and how long an encoder takes to do that.

[Dark Shikari]

psy rdo use simply SSD metric for choose better RD decision.Perhaps if people keep thinking this it'll be longer before another encoder implements something comparable... :sly:

[Gabriel_Bouvigne]

The goal is to be as close to the original as possible right?
Only at mid/high bitrate (1.5Mbps is NOT low bitrate) and if the encoder is only using brute-force RD optims. Once you start adding things like psychovisual frame optimizations, ROI or anti-flickering, then simple frame by frame comparisons using psnr or ssim are starting to be less relevant.

(unless you are not encoding video for human viewers but for stuff like latter-stage segmentation, objects extractions,...)

[Sharktooth]

psy tool for x264 are metric tools by itself ... lol

- AQ measure complexity block
- psy rdo use simply SSD metric for choose better RD decision.

I have impression when I read your thread that psy for x264 is like magical tools. Psy tools use simply other metric tools for better complexity/texture detection.
so you're basically saying the default metrics (PSNR, SSIM) fails miserably and other metrics are needed to enhance the picture. maybe there are some other metrics that will enhance the picture for dogs or cats...
that's again a proof that metrics are not representing perceived quality and a proof there is no metric that can represent the human eye perception.
please, stop... it's been years you're telling always the same things... but you always forget that:
A) a number cant represent a picture.
B) metrics measure differencies, not quality
C) quality IS what please the EYE (details, vivid colors, no artifacts... etc.) not the PSNR or other metrics.

[dj_tjerk]

Heh, vivid colors; so if the encoder does tweak(sat=1.1) internally, it's a better encode? (Because the human eye, or rather brain, likes vivid colors.)

[Sagekilla]

Not quite. If one encoder somehow produces a duller looking image than the source, then a comparable encoder that can produce an image with identical saturation to the source would look better than the duller one.

[Esurnir]

so you're basically saying the default metrics (PSNR, SSIM) fails miserably and other metrics are needed to enhance the picture. maybe there are some other metrics that will enhance the picture for dogs or cats...
that's again a proof that metrics are not representing perceived quality and a proof there is no metric that can represent the human eye perception.
please, stop... it's been years you're telling always the same things... but you always forget that:
A) a number cant represent a picture.
B) metrics measure differencies, not quality
C) quality IS what please the EYE (details, vivid colors, no artifacts... etc.) not the PSNR or other metrics.

A number certainly can represent a picture, my picasa's album is entirely made of number and they are well satisfied of the quality of it,

B) Metric in the end is the -only- way you can make choice in an encoders save if you want to place every single motion vectors and do every single quantisation by hands you'll have to use a metric somewhere, and even the Sum of Average Difference is enough to reject the most obvious bad choices.

C) The aim of an encoder is to make a picture that is "high-fidelity" that is the nearest of the original pictures as possible, no matter how crappy the source was. If the source was a youtube .flv file blocky as hell with washed out colors then the best encoders will output a picture blocky as hell with washed out color. To make a picture that please is the job of the pre-processing which can adjust saturation, enhance contrast, remove noise and sharpen things. After that on the picture I showed, it is clear than psy-rdo preserve details better than the blurrier murkier pictures of other h.264 encoder, even if the psnr could be lower, even if the grain is not where it should be and some grain could very well be ringing, it's still grainy, and that's what my eyes will remember.

[Sagekilla]

Esurnir: That's not news that you need to use metrics to make decisions. This is well known. There is no known metric out there that can accurately provide a mathematical model of how the human eye perceives images. Therefore, we can't optimize for a metric that is visually pleasing.

If such a metric existed, we could simply try to encode for whatever provides the highest value for that hypothetical metric. Until then, we have only poor approximations that consider soft images to look better than sharper ones. (Read: no psy-rd vs psy-rd)

[Sharktooth]

A number certainly can represent a picture, my picasa's album is entirely made of number and they are well satisfied of the quality of it,

B) Metric in the end is the -only- way you can make choice in an encoders save if you want to place every single motion vectors and do every single quantisation by hands you'll have to use a metric somewhere, and even the Sum of Average Difference is enough to reject the most obvious bad choices.

C) The aim of an encoder is to make a picture that is "high-fidelity" that is the nearest of the original pictures as possible, no matter how crappy the source was. If the source was a youtube .flv file blocky as hell with washed out colors then the best encoders will output a picture blocky as hell with washed out color. To make a picture that please is the job of the pre-processing which can adjust saturation, enhance contrast, remove noise and sharpen things. After that on the picture I showed, it is clear than psy-rdo preserve details better than the blurrier murkier pictures of other h.264 encoder, even if the psnr could be lower, even if the grain is not where it should be and some grain could very well be ringing, it's still grainy, and that's what my eyes will remember.
A) i didnt say "numbers"... i said "a number" meaning a single number of (in our case) 2 digits and some decimals.

B) using metrics for decisions is "optimizing". that's usually done for encoding speed. instead of taking into account all the "candidates", some get discarded. that, for sure, hurts quality. how much quality is loss depends on the algos.

C) wrong. the aim of an encoder is to reach the best quality/compression ratio. so if an encoder can, for example, sistematically reconstruct details from an artifact and keep a reasonable compression, then it's a good encoder. that's why ppl use filters before encoding... a similar reason justifies the use of psy-trellis in x264 (it doesnt "keep" the grain exactly as in the source, it just fools the eyes... and in most cases, when it's needed, it does it damn good...).