When is PSNR applicable?

[berrinam]

It seems that many people don't approve of PSNR as a measure of quality, because it has no HVS modelling of any sort. However, it would clearly be useful to have a metric for comparing quality, partly just because of the time savings involved. My question is this: does it make sense to use PSNR as an indication of quality when tweaking settings in x264 which are not HVS-based, namely reference frames, mixed references, RDO, and ME method?

[Sharktooth]

b-frames were "invented" as a HVS method: lowering quality on certain frames coz the human eye cant spot the difference. so the answare is "no" unless you disable b-frames too. at least that is my POV.

[akupenguin]

Yes, PSNR is less applicable to comparing an encode with B-frames against an encode without B-frames. But that doesn't mean you can't use PSNR to compare variations in other options just because B-frames are present.

[Sharktooth]

well, that's for sure, but PSNR as a measure of quality (as stated in the first post) is only relative to encodes without HVS tuning, while it's pretty usefull to compare quality of 2 or more encodes for tweaking settings (supposing those encodes are made with comparable HVS tunings).

[foxyshadis]

PSNR has its place as a rough measure of quality; SSIM has largely displaced it, aligning better with HVS. Of course it's also dog-slow for a whole video. The only perfect metric is sitting down with several encodes with different options and a double-blind tester until your eyes bleed.

For one, adaptive quantization can produce noticably better video (and will likely improve further) in some cases, but hardly changes PSNR metrics.

[Sharktooth]

Well PSNR can be easily tricked (for example by smoothing). a clear proof is DivX 6...
it scores better than xvid on PSNR metrics, but xvid is visually better.

[dragongodz]

or to put it another way, metrics such as PSNR and SSIM are more useful for development purposes rather than for telling people what they are actually going to like.

[Manao]

a clear proof is DivX 6... it scores better than xvid on PSNR metrics, but xvid is visually better.Lets stop the myth : XviD tries to optimize the PSNR. The only thing in XviD's ME / mb type decision that doesn't try to optimize it is the too_small_int_limit. And "XviD is visually better" is a very subjective statement ( hint, x264 is visually better is objective :P )

[berrinam]

Thanks everyone for your responses. Basically, I am asking, because I have been doing some few tests with x264 keeping all the settings the same except for, as I said above, reference frames, mixed references, RDO, and ME method. I was comparing PSNR and fps achieved, to work out some settings which simply do not make sense to use, according to these results, like using 16 references with RDO 1, because an encode would have a higher fps AND higher PSNR by simply enabling mixed references and dropping the number of reference frames to 3.

Are these sorts of conclusions valid, because, having encoded the same file about 20 times, I don't particularly want to do all of the 190 ABX tests that would be required.

By the way, I am aware that settings like AQ are HVS-modelled (I didn't know about b-frames), which is why I am avoiding them in PSNR-based tests. I doubt my use of b-frames would affect the results, because I am using the same (turbo) first pass for all of the tests, so the frame types should all be the same.

[Manao]

BTW, another myth should be forgotten : bframes don't necessary work by decreasing quality. BFrames by themself are more efficient than PFrames, even (and especially) when their quantizers are the same. Just set the pbratio to 1, and do a constant quantizer encode with and without adaptive bframes, you'll see.

[stephanV]

BTW, another myth should be forgotten : bframes don't necessary work by decreasing quality. BFrames by themself are more efficient than PFrames, even (and especially) when their quantizers are the same. Just set the pbratio to 1, and do a constant quantizer encode with and without adaptive bframes, you'll see.
Only for h264 no? ASP-b-frames are worse than p-frame methinks... and IIRC visualy AVC -b frames where still a bit smoother at the same quant..

[Manao]

ASP B's are better than P's ( and it's true at least for mpeg2 too ) ( better --> in an RD sense )

[yaz]

BTW, another myth should be forgotten : bframes don't necessary work by decreasing quality ...highly agreed. it seems as if it shouldn't be repeated enough times.

anyway, what about considering these 'quality' metrics rather as 'fidelity' metrics. i mean a kinda 'similarity' here. the higher these metrics are the more similar is the encode to the original ... and no word about quality as it is always depends on personal preferences. the most common interpretation of quality : 'good is what i like bad is what i don't'. ;)

the bests
y

[Sagittaire]

Well PSNR can be easily tricked (for example by smoothing). a clear proof is DivX 6...
it scores better than xvid on PSNR metrics, but xvid is visually better.

1) For many user DivX6 is visually better than XviD
2) DivX6 is not very better than XviD for metric ... result are similar for APSNR, OPSNR and SSIM.


ASP B's are better than P's ( and it's true at least for mpeg2 too )

Bframe are not only HVS optimisation ... metric can detect bframe optimisation ...

1) with same quant for all frame encoding with bframe is little better than the encoding without bframe.
2) with ratio/offset for bframe encoding with bframe is very better than the encoding without bframe.


Ponctualy codec A can be clearly visually better than codec B and codec B can be better than codec A for APSNR or OPSNR or SSIM or VQM or JDN but nener codec A could be visually better than codec B if codec A is the best for APSNR and OPSNR and SSIM and VQM and JDN. If you have only one example show it ...

[foxyshadis]

anyway, what about considering these 'quality' metrics rather as 'fidelity' metrics. i mean a kinda 'similarity' here. the higher these metrics are the more similar is the encode to the original ... and no word about quality as it is always depends on personal preferences. the most common interpretation of quality : 'good is what i like bad is what i don't'. ;)
Yes, but there are well developed fields of study is psychovisual and psychoacoustic modeling, codecs as old as indeo 3 and sorensen included many scientificly supported tradeoffs of absolute quality for visual quality. Absolute quality is only useful for archival, editing, and ease of comparison purposes. Because the final output is intended for people, there's no point in aiming solely for metric improvement if metrics don't accurately model the actual experience for the vast majority of the viewers.

On the other hand using something like mpeg-4 as an intermediate format or for master archival, understanding what is closest to absolute zero difference, because future formats will likely use different tricks that won't play nicely with current ones. In the same way, knowing whether the PSNR of an mpeg-2 master you have today is low enough to be indistiguishable from original film is important in deciding how much to store and how long, or whether you should sacrifice for huffyuv or retransfer from film every time.

[dragongodz]

yaz - :goodpost:

first about B-frames.
to quote syskin(from way long ago here, http://edu.bnhof.de/pipermail/xvid-devel/2003-January/001950.html )
B-frames might have horrible PSNR but still look very good. The picture,
when compared pixel-by-pixel, might be different than the original, but
it's still sharp, because both references are sharp, and is not
blocky, because SAD wouldn't allow visible blocks (that's the way SAD
is). As a result, b-frames don't look bad even when someone would look
at a still picture, and at the same time original might look very
different - especially the noise is very different.
so B-frames will not always look the same as a P-frame would have. also when compared to close by P-frames they may indeed look different or even lower in quality but this is ignoring the purpose of B-frames. that is B-frames in general require less space/size for similar quality. this space saving can then be passed on to other frames(P-frames) to increase their quality for the same overall/average bitrate.

BFrames by themself are more efficient than PFrames, even (and especially) when their quantizers are the same. Just set the pbratio to 1, and do a constant quantizer encode with and without adaptive bframes, you'll see.
the only reason to have B-frames at the same quant as P-frames is if you are hitting bitrate saturation since a B-frame at the same quant can easily be larger than the equivilent P-frame. this is of course why B-frames are actually encoded with a higher quant ,except in rare cases, and so may look different.

now about PSNR.
first a link to show where PSNR has been fooled in real life since people seem to need to be shown it.
http://forum.doom9.org/showthread.php?p=419365&highlight=psnr#post419365

and i think i will leave all this(including posting anymore on this) by quoting a post from the Divx forum by stephenV, who is one of the mods there by the way.
I disagree, PSNR tests are not a qualitative test, as PSNR doesn't
measure quality but the difference between frame A source and frame A
encoding. PSNR tells us nothing if those difference are noticeable or not. For
example, a slight change in brightness would have a very negative effect on
PSNR but would probably imperceivable to the human eye. So PSNR is basically
a quantitative test,it measures how much an encoded frame differs from the
source, but not if therefor a frame looks good or not. So what if codec A is
better for metrics than codec B? About quality, you know just as much as you
did before.

[Sharktooth]

ok, maybe the divx vs xvid thing should not be taken into consideration (too much fans from both sides...).
However h.263 quantization gets highly PSNR metrics results than almost any custom matrix and the visual result is clearly in favour of the custom matrices...
Also, talking about myths and b-frames, most codec default options are set to rise the quantizers of b-frames. Ok, they're more efficient but not enaugh to compensate for the higher quantizer usually applied. The question is: why those codecs use higher quantizers for b-frames? The answer is: coz it's a measure to rise compression/efficiency/perceived quality by reducing quality/frame size on some frames without having the human eye spotting the difference: HVS tuning.

[Sagittaire]

now about PSNR.
first a link to show where PSNR has been fooled in real life since people seem to need to be shown it.
http://forum.doom9.org/showthread.php?p=419365&highlight=psnr#post419365


Average PSNR or Overall PSNR ... I want example for APSNR and OPSNR and SSIM ... ???



and i think i will leave all this(including posting anymore on this) by quoting a post from the Divx forum by stephenV, who is one of the mods there by the way.
I disagree, PSNR tests are not a qualitative test, as PSNR doesn't
measure quality but the difference between frame A source and frame A
encoding. PSNR tells us nothing if those difference are noticeable or not. For
example, a slight change in brightness would have a very negative effect on
PSNR but would probably imperceivable to the human eye. So PSNR is basically
a quantitative test,it measures how much an encoded frame differs from the
source, but not if therefor a frame looks good or not. So what if codec A is
better for metrics than codec B? About quality, you know just as much as you
did before.


1) You must compare equivalent image with PSNR ... source with preprocess will be "more pleasant for the eyes" than the same same without preprocess (denoising, sharpening ...) but compare these encoding with PSNR is totaly useless because source are not equivalent. Why not to compare King Kong 1933 version and King Kong 2005 version with PSNR ... ???

2) SSIM is able to compare source with different brightness or constrast for example
http://www.cns.nyu.edu/~zwang/files/research/quality_index/demo_lena.html

[Sharktooth]

sagittaire, we're talking about PSNR... no SSIM ... no APSNR ... no OPSNR... no JDB... etc...

[Sagittaire]

However h.263 quantization gets highly metrics than almost any custom matrix and the visual result is clearly in favour of the custom matrices...

Certainely not ... H263 is visually always in the top matrix (see Soulhunter blind test) in all situation (low, medium and high bitrate) for all source (natural, CGI, anime ...)

And for some developper
1) H263 is the default quantisation for XviD and not HVS CM
2) DigitAl56K say "H263 is the best and by far"

CM are good for blocking and not for ringing (generaly too agressive for high frequencies) but with PP4 H263 is IMO by far visually more efficient and by far better for metric ...

[Sharktooth]

Yes, as you said, BETTER FOR METRIC... can i add: NOT FOR QUALITY?
however "good for blocking but not for ringing" depends on how the custom matrix was made, it could be the other way as well...
Also JVT said (and proved) custom matrices (if made correctly) are better than default quantization... that's the reason why they were implemented in h.264 HIGH PROFILE (the profile supposed to bring HIGH/BETTER QUALITY).
Talking about post processing that's another (in)famous way to rise PSNR by smoothing out details. A good encode doesnt need to be post processed.

[Sagittaire]

Yes, as you said, BETTER FOR METRIC... can i add: NOT FOR QUALITY?.
Better for quality too for some dev and some user I think ...


however "good for blocking but not for ringing" depends on how the custom matrix was made, it could be the other way as well...
Also JVT said (and proved) custom matrices (if made correctly) are better than default quantization... that's the reason why they were implemented in h.264 HIGH PROFILE (the profile supposed to bring HIGH/BETTER QUALITY).

I don't know ... I don't test and use CM with H264 ... but MPEG4 AVC is not MPEG4 ASP. For me 8*8 DCT is more interessing than CM in MPEG4 AVC HP.


Talking about post processing that's another (in)famous way to rise PSNR by smoothing out details. A good encode doesnt need to be post processed.

H264 use deblocking in inloop process ... and is not infamous ... ;)
For High quant PP4 is necessary with MPEG4 ASP (quant 4 and more) and you can use threshold for deblocking in ffdshow for example.

[Sharktooth]

The soulhunter blind test shown h.263 was not the best...
However custom matrices for h.264 have been proven (as for ASP) to be better than the default quantization.
h.264 uses deblocking in the inloop filter coz h.264 has issues with blocks even at low quantizers due to the codec itself. however the inloop deblock in h.264 is "weighted" and completely different from post-processing deblocking.

[Sagittaire]

However custom matrices for h.264 have been proven (as for ASP) to be better than the default quantization.

How ... ???
Prove that implicate very complexe biological HVS study ???
It certainely very interessing for all video codec dev ... !!!

The soulhunter blind test shown h.263 was not the best...
Not for all round but for example with H263 PP0
http://forum.doom9.org/showthread.php?s=&postid=570152#post570152
http://forum.doom9.org/showthread.php?s=&postid=578992#post578992

[Sharktooth]

The proof for h.264 was the JVT document.
The first link shows h.263 is not the best for lowest bitrates, the second instead shows everyone liked it for the mid range. Lets wait for high bitrates... but an answer is ppl already use custom matrices for high bitrates and for everyday encodes coz h.263 is not enaugh detailed, limited, blocky... etc depending on the situation.
However i would like to remember the soulhunter blind test was done with a small sample of testers (not his fault though)...

Professional Extension Profiles (Fidelity Range Profiles) of AVC is aimed at coding of high-resolution pictures, including HD pictures. As such, high visual fidelity is appreciated greatly in HD picture presentation. When an AVC Professional Extension Profiles is used, it is naturally desirable is to allow the superior coding efficiency of AVC to directly benefit visual quality to the greatest extent. With the aim to improve HD subjective quality, we propose here a quantization weighting scheme that enables non-uniform quantization weightings to block transform coefficients. We consider non-uniform quantization tools to be critical for the following reasons:

1. Improving visual fidelity of decoded pictures
2. Non-uniform quantization makes it possible for quantization adjustment to be done in proportion to human vision sensitivity, which improves coding efficiency in picture fidelity.
3. Providing flexible choices in controlling end picture quality, strongly required by high quality content creation industry.

The quantization weighting scheme proposed here includes the following major features:

1. Specification of the quantization weighting scheme using completely integer based, division free, requiring only 16-bit memory operation at decoder, which introduces no complexity increase compared to uniform quantization scheme.
2. The non-uniform quantization scheme proposed is based on 8x8 transform on luma since we consider this is the transform that preserves better picture textures, one of the most important content elements that affect the subjective impression of high quality pictures.
3. The non-uniform quantization weightings are applied to residuals in both intra-prediction and inter-prediction

Our simulation and viewing in various HD display devices show subjective quality improvement in all video sequences and significant improvement in many of the test videos including some of the JVT sequences and film content we obtained from movie studios.
full document: http://www.webalice.it/f.corriga/temp/JVT-K029r1.7z

That's as true for AVC as for any other codec based on the same basic principles (ASP included...).

[IgorC]

Guys, topic about relationship PSNR and true visual quality was discussed many many times. It was clear that PSNR helps to develop codec but it´s not 100% indicator of quality. And I agree with Sharktooth Xvid+custom matrices is visually better than divX with H.263++ however not for PSNR.
and it´s not myth but reality.

[Manao]

IgorC : your comments on how XviD and DivX compare can't hardly be called objective.

Sharktooth : yet, h263 scored 2nd and 1st. You might personnaly disagree ( I do ), yet, it proves that at least some people ( among the ~10 that took part to the test ) liked it.

@all : we all agree that PSNR is far from being perfect. Yet, all codecs try to optimize it.

For XviD, three things aren't tweaked for PSNR : lumimasking ( some like it, others don't ), too_small_int_limit, and rate control ( a highly discussed matter between devs ). But everybody agrees on saying VHQ is great, yet the sole purpose of VHQ is to increase PSNR at all cost.

For x264, RDO does exactly the same. And everybody agrees that the more RDO, the better.

So that mainly means PSNR can safely be used to compare codecs within the same category ( XviD / DivX and x264 / nero ), when their psychovisual tools aren't in use ( rate control is psychovisual, so are custom matrices and adaptive quantization ).

Now, for the bframes : you're all forgetting a good reason to raise their quantizer : they won't be used as reference, so they can be trashed. x264 for example raises more the quantizer of bframes that aren't used as reference than those which will be. That's half the reason for raising the quantizer ( the other being that since they are more efficient, you can raise it without losing much PSNR, and so increase coding efficiency ( aka overall PSNR ) )

[Sharktooth]

here we fall in the case where PSNR is good for developing, since (presumably) a developer doesnt try to fool or trick the metrics to gain the best PSNR without improving the codec efficiency and final encoding quality.
But, obtaining a higher PSNR tweaking settings without visually checking the final results doesnt necessarily lead to a better visual quality.
So, IMHO, trusting PSNR as an absolute quality meter is not a good idea... adding other metric tests that take into account other factors (like SSIM, JDB, etc - as sagittaire said in previous posts) is necessary to limit the visual tests (that requires a lot of time and effort) to the minimum.

[Sagittaire]

@ Sharktooth

1) Where is the proof that the CM of MPEG4 ASP are better than the quantisation H263 ... ? I would be really very curious to see that ... !

2) This document speak about only for compare uniform matrix and non-uniform matrix for MPEG4 AVC HP. But default matrix in AVC is a CM itself with only number 16 for all matrix coefficient. H263 for MPEG4 ASP is not a matrix but a particular quantisation system. You can't compare uniform matrix vs non-uniform matrix for MPEG4 AVC and H263 vs CM for MPEG4 ASP.

3) This document say: "Our non-uniform matrix is better than the default uniform". Perhabs that it's true ... perhabs not ... but this JVT doc don't prove absolutely anything ...

Our simulation and viewing in various HD display devices show subjective quality improvement in all video sequences and significant improvement in many of the test videos including some of the JVT sequences and film content we obtained from movie studios.

[Sharktooth]

Sagittaire are you joking? or maybe you're blind...

which one looks best for you? (1st image is the original)
http://www.webalice.it/f.corriga/temp/orig.png
http://www.webalice.it/f.corriga/temp/1.png
http://www.webalice.it/f.corriga/temp/2.png
http://www.webalice.it/f.corriga/temp/3.png

[Sagittaire]

Sagittaire are you joking? or maybe you're blind...

yes ... you are certainely right finally ... JVT matrix is certainely the "best matrix" for H264. CM uniforme is certainely a very bad matrix and H263 quantisation too ... :stupid:

[Sharktooth]

have a look at the images in my previous post and choose the one which looks best for you (excluding the first which is the original).
if h263 is the best you should choose the right one...

[Manao]

Beware, the test migh be flawed : same quantizer --> size might differ. Edit: /me should read irc :D

[Sharktooth]

lol, filesize is almost the same (i've chosen the same target bitrate for all the encodings). There's a small difference though: 1.png 339KB, 2.png 340KB, 3.png 336KB. There's a 4th file that hit 356KB... but...

[IgorC]

IgorC : your comments on how XviD and DivX compare can't hardly be called objective.

The same way your comments about PSNR also can´t be objective. I´ve choosed Xvid not because it open source , not because I don´t like divx , not because of higher psnr or ssim nothing personal , but my own visual perception.
So please don´t tell me that my comments aren´t objective. It´s not only mine but aprox 60-70% of people opinion.

[Sagittaire]

The same way your comments about PSNR also can´t be objective. I´ve choosed Xvid not because it open source , not because I don´t like divx , not because of higher psnr or ssim nothing personal , but my own visual perception.
So please don´t tell me that my comments aren´t objective. It´s not only mine but aprox 60-70% of people opinion.

But certainely more than 60-70% of MPEG4 user encode with DivX ... ;)

[IgorC]

totally disagree

[Sagittaire]

have a look at the images in my previous post and choose the one which looks best for you (excluding the first which is the original).
if h263 is the best you should choose the right one...

My eyes say image 1, image 2 and finaly image 3 because:
- high ringing for image 2 and image 3
- high blocking for image 3 and perhabs little more blocking for image 1 than image 2

PSNR (with black zone for ffdshow OSD) say:
image 1: 43.4021 dB
image 2: 42.9003 dB
image 3: 41.9996 dB

Conclusion: PSNR is very good finaly ... :thanks:

[dragongodz]

Guys, topic about relationship PSNR and true visual quality was discussed many many times.
and no doubt will be many more times in the future. not much can be done about that while people believe metrics are some sort of absolute.

It was clear that PSNR helps to develop codec but it´s not 100% indicator of quality.
umm didnt i say that on the first page ? ;)

anyway i didnt WANT to have to post again but i guess i may aswell. :)

Average PSNR or Overall PSNR ... I want example for APSNR and OPSNR and SSIM ... ???
Makira did the tests ,not me, so you would have to ask him. the point is you have asked many times for proof of PSNR being fooled/wrong and i have told you before in which thread the proof was. now i have given you the actual link straight to it and you still dont want to believe it. i wont bother to keep banging my head against a wall because you dont want to accept proof when given to you.

about h263 in Xvid.
Better for quality too for some dev and some user I think
well here is 1 developers view.
http://66.102.7.104/search?q=cache:n-Q1VHPnt_IJ:www.xvid.org/modules.php%3Fop%3Dmodload%26name%3DphpBB2%26file%3Dviewtopic%26p%3D8611%26sid%3D9433e999819a9fe7606d0aa395adc802+xvid+psnr+site:www.xvid.org&hl=en
so bitrate is also a factor along with footage type etc etc etc. an "always best" to everyone is simply not possible. sharktooth did show you a test where h263 did not sweepingly win yet you still want to argue the point by saying it did win some and then even asking for proof again as if the blind test proved nothing ????? this is all starting to sound like argueing over how long a piece of string is. :sly:

you're all forgetting a good reason to raise their quantizer : they won't be used as reference, so they can be trashed
sorry but i see that more as a necessity of purpose and not a reason in itself, specifically for earlier encoding engines(such as mpeg 1/2 and 4part2). :)
That's half the reason for raising the quantizer ( the other being that since they are more efficient, you can raise it without losing much PSNR, and so increase coding efficiency ( aka overall PSNR ) )
no not half the reason at all. this is infact the reason period(coding efficiency that is). also coding efficiency does not equel OPSNR. as already said multiple time developers can often use metrics to try and increase the coding efficiency. that doesnt mean it always works or is always the best. see the link i previously gave showing where using PSNR to try and increase efficiency failed if you need proof.

finally.
It´s not only mine but aprox 60-70% of people opinion.
But certainely more than 60-70% of MPEG4 user encode with DivX
guys please dont start throwing out silly numbers like that unless you have something to back them up with. i dont know how many people overall would prefer Divx over Xvid for different encodes or vice versa. i also dont know how many people use eaither codec or have even tried both to make any opinion on which they may prefer. i somehow doubt you guys know either so saying those numbers makes no sense at all.

[Sharktooth]

Completely agree with dragondooz.
and however here's the magical h.263 screenshot (with pp4):
http://www.webalice.it/f.corriga/temp/4.png
(yes, none of the above were h.263...)
As you can see deblocked areas are clearly visible (without PP it was a blockfest), details are lacking (well they were lacking even without deblocking...) but probably the PSNR will be higher...

pics details:
1.png - flat matrix
2.png - jawor's 1CD
3.png - ULR rev4 (beta)
4.png - h.263 with postprocessing enabled (deblocking only)

[Manao]

Ok, so basically, the flat matrix ( ie, the closest to h263 ), wins the contest where h263 wasn't present ? And it happens to be the matrix that also give the highest PSNR ? Come on, I know you guys can find better counter examples than that ( because that surely won't convince Sagittaire, nor me ( though I don't need conversion ) ).

IgorC : please, tell me where my comments on PSNR weren't objective

[stephanV]

So instead of using PSNR we are now gonna use one sample, uhm, make that SCREENSHOT to determine what the best matrix is. Yes of course, thats a much better testing methodology...

I'ts funny how things can go off-topic BTW - discussing ASP matrices in the AVC forum...

[Sharktooth]

we're discussing if PSNR is useful to determine quality.
however the fact that "h.263 is the best of the best with the best PSNR and the best quality coz it's the best quantization and best PSNR means best everything in every case" came out at a certain point during the discussion.

@manao: if flat is the closest thing to h.263 (the "particular quantization system"...) then the JVT assertions are true even for ASP... dont you think?
however h.263 is not the "magical" instrument for the best encoding, just take 6of9 as an example of how a custom matrix can be much better than it. obviously matrices have a target compressibility, while h.263 is somewhat "adapting"...
also h.263 can be saturated at much lower bitrates than the majority of high bitrate matrices (including MPEG default!!!) and that means it wont reach higher quality even if it will reach higher PSNR...

[Manao]

then the JVT assertions are true even for ASP... dont you think?Too far fetched : hpel is different, transform is different, deblocking is added, intra works differently... there's simply too much difference between ASP and AVC.
that means it wont reach higher quality even if it will reach higher PSNR...But then, in a RD sense, both bitrate & distorsion must be taken into account, so that hardly matters ( and also, if they can reach higher bitrates / psnr, it means they can't reach lower bitrates, of which nobody cares but which still exist ).
if flat is the closest thing to h.263It's the closest in the sense that all frequencies are equally treated. There're however subtilities on how the quant / dequant must be done.

And I'll play the devil's advocate by pointing out that you were the first to critisize h263 in this thread.

Back on the topic ( well, almost ) :

dragongodz : i asked pengvado why he used a lower quantizer for bref than for bframe, and why it was still higher than for pframes : the answer is "because B-refs are referenced less than P-frames".

[Sharktooth]

It's the closest in the sense that all frequencies are equally treated. There're however subtilities on how the quant / dequant must be done.
well, then (out of any doubts) it's a uniform quantization.
The JVT assertion was:
2. Non-uniform quantization makes it possible for quantization adjustment to be done in proportion to human vision sensitivity, which improves coding efficiency in picture fidelity.
that's (out of any doubts) true for every codec based on certain principles... do we agree on that?
HVS tweaking means some details/frequencies/etc, invisible to human eyes are removed or higly quantized to gain bits to spend on more visible (for the human eye) areas. Such gain is not linear coz it depends on how the encoder decides to spend the bits.
Question: is PSNR a HVS proof metric test?
Answer: no. it's not based on any HVS principles.
Question: So can PSNR be inaccurate in respect of human vision system when HVS tunings are applied?
Answer: yes.
Question: So is PSNR directly related to perceived quality when applying HVS tunings?
Answer: no.
Question: How much HVS tunings are used in MPEG codecs?
Answer: a $h!tload. starting from b-frames, rate control, adaptive quantization, custom quantization matrices etc...
Final Question: So, in conclusion, is PSNR directly related to perceived quality?
Answer: NO!

[stephanV]

we're discussing if PSNR is useful to determine quality.
And here is the answer: "Yes it has its limited use, as long as you keep the restrictions it has in mind. It surely won't give a definitive answer." (but hey, nothing will)

however the fact that "h.263 is the best of the best with the best PSNR and the best quality coz it's the best quantization and best PSNR means best everything in every case" came out at a certain point during the discussion.
I haven't seen anyone (including Sagitaire) say that. He did say that (according) to him, h263 was always "in the top", meaning its always one of the better matrices. (Based on soulhunters tests BTW)

I think it was you who made this statement 1st:
However h.263 quantization gets highly PSNR metrics results than almost any custom matrix and the visual result is clearly in favour of the custom matrices...
Which is in contrast with soulhunters tests, since there h263 ended 1st and 2nd. So what is so "clearly" about this?

[Sharktooth]

@stephanV: the soulhunter blind tests were somewhat flawed unless they had more participants.
we cant determine in what conditions the clips where watched and that's another flaw...
however the fact that a linear quantization (h.263) is inferior to a non linear quantization (a well made custom matrix) is a fact and JVT tested and proved it.
however here: http://forum.doom9.org/showthread.php?p=729016#post729016 sagittaire said h.263 is better for both quality and psnr...

[stephanV]

the soulhunter blind tests were somewhat flawed unless they had more participants.
It's at least indicative that it's not so "clearly" as you put it.

however the fact that a linear quantization (h.263) is inferior to a non linear quantization (a custom matrix) is a fact and JVT tested and proved it.
I couldn't find any samples, nor results of any test (only the claim that test have shown...) in the doc you linked to and as while I have no idea how far we can stretch the AVC results to other standards, I'm going to rely on Manao's judgement on that for now.

however here: http://forum.doom9.org/showthread.p...9016#post729016 sagittaire said h.263 is better than...
What did he say???

Better for quality too for some dev and some user I think ...
That's a more refined statement than I've seen you make in this thread.

[Sagittaire]

Question: is PSNR a HVS proof metric test?
Answer: no. it's not based on any HVS principles.
not exactly, it's very complicated ... for example:
1) PSNR test is make in YV12 4:2:0 space color. YV12 4:2:0 is HVS optimisation for codec (HVS more sensitive for luma)
2) Overall PSNR is able to help for Rate Control variability. Good Rate Control is HVS optimisation (HVS more sensitive for low motion than for high motion)

Question: So can PSNR be inaccurate in respect of human vision system when HVS tunings are applied?
Answer: yes.
Yes with for example:
- adaptative quantisation (SSIM is able to detect optimisation)
- Custom Matrix (SSIM is not able detect optimisation but nobody is able to prove that Custom Matrix X is better than Custom Matrix Y -> Personnaly I choose metric sense or/and my eyes)

Question: So is PSNR directly related to perceived quality when applying HVS tunings?
Answer: no.
Rate Control tuning is HVS tunnig, RDO is HVS tunning, bframe is HVS tunnig ...


Question: How much HVS tunings are used in MPEG codecs?
Answer: a $h!tload. starting from b-frames, rate control, adaptive quantization, custom quantization matrices etc...
Final Question: So, in conclusion, is PSNR directly related to perceived quality?
Answer: NO!
not exactly, it's very complicated ... for example:
1) If Codec A is better than codec B with 0.1 dB OPSNR, it's very difficult to say "Codec A will be visually better than codec B". But if Codec A is better than codec B with 1.0 dB OPSNR Codec then "codec A will be with high propability very better visually than the codec B".
2) PSNR is relatively good for high quantizer but not for very high quality if codec A obtain 55 dB and codec B 50 dB ... visually encoding A and encoding B are in practice transparent encoding


But it's true: PSNR is not perfect:
for exemple IMO NDAVC is visually more stable than x264 (less block flicking for NDAVC) and IMO NDAVC MP is perhabs visually better than x264 HP. PSNR is not able to say that because they not test dynamique image sequencies but test only separate image ...

[Sharktooth]

not exactly, it's very complicated ... for example:
1) PSNR test is make in YV12 4:2:0 space color. YV12 4:2:0 is HVS optimisation for codec (HVS more sensitive for luma)
2) Overall PSNR is able to help for Rate Control variability. Good Rate Control is HVS optimisation (HVS more sensitive for low motion than for high motion)
1) Well, obviously PSNR was made for YV12, or it will be useless since that colorspace is used by the majority of video codecs.
2) Overall PSNR is different from PSNR.

Yes with for example:
- adaptative quantisation (SSIM is able to detect optimisation)
- Custom Matrix (SSIM is not able detect optimisation but nobody is able to prove that Custom Matrix X is better than Custom Matrix Y -> Personnaly I choose metric sense or/and my eyes)
SSIM is very limited in other aspects and can be tricked as well as PSNR. however as i said PSNR alone (and SSIM alone too) is not an index of quality.


Rate Control tuning is HVS tunnig, RDO is HVS tunning, bframe is HVS tunnig ...
yes, i stated it in a previous post.

not exactly, it's very complicated ... for example:
1) If Codec A is better than codec B with 0.1 dB OPSNR, it's very difficult to say "Codec A will be visually better than codec B". But if Codec A is better than codec B with 1.0 dB OPSNR Codec then "codec A will be with high propability very better visually than the codec B".
2) PSNR is relatively good for high quantizer but not for very high quality if codec A obtain 55 dB and codec B 50 dB ... visually encoding A and encoding B are in practice transparent encoding
we can assume that very big differences in PSNR is an index of a better quality, but still visual checks or other metric tests are needed.

But it's true: PSNR is not perfect:
for exemple IMO NDAVC is visually more stable than x264 (less block flicking for NDAVC) and IMO NDAVC MP is perhabs visually better than x264 HP. PSNR is not able to say that because they not test dynamique image sequencies but test only separate image ...
that's exactly the point.