Do you beleive to metric result? or Do you agree with them?

[IgorC]

Last time SSIM and OPSNR tests became even more popular between hobbists and devs of video compression.

Personally many times I was disagree with OPSNR metrics. When the difference was +/- 0.35-0.5 db but visually I prefered the video with lower opsnr. And the difference was noticeble.
My visual perception is more related with ssim metrics. I always was agree with the ssim results when the difference was noticeble . At least +/- 0.2 ssim .

[siddharthagandhi]

Yes, generally the metrics dont lie. Generally, I say.

[Mug Funky]

i agree with the wording that "PSNR and SSIM are reliable metrics", because that statement says nothing about perceptual quality...

i judge quality with my eyes, but numbers are a very good guide. for example, when i do MPEG-2 (which is all the time), i'll do a bitrateview scan and look for the bits where the quants visibly spike. you're unlikely to have quality problems with q2 or q1 footage. but q17 stuff requires visual inspection...

so i guess i'd have to say my quality metric is quantiser rather than PSNR or SSIM, as i hardly ever use either of them.

also, i read a lot of threads with metric comparisons, and PSNR and SSIM are almost always in agreement. they only differ greatly when the visual quality starts getting bad, or in special cases that i can't think of offhand.

[foxyshadis]

PSNR has a noise area, based on what I can tell it's something like 1-2 db at low values and more as you go higher... any two encodings within that area may be better, worse, or identical, only when the scores are very different are they reliable (at which point visual quality differences are very obvious anyway). SSIM is the same, but much smaller intervals, which is the only reason it's more useful. Still fine differences can throw it off, and it isn't really optimized for all the different kinds of artifacts on different kinds of video that encoders can cause.

[Doom9]

c't performs codec test using the most complex metric of 'em all: SND. Yet, in my codec comparisons, the codec they find to be great do not do so well.. hence I trust only my eyes. No matter how much time and maths you invest to model the HVS, each and everyone of us has a different visual trace and places focus on different areas. The occasional block may be distracting, but personally if the alternative means a lack of crispness, then when I watch both clips one after another, I'll still rate the crisper but blockier one higher than the washed out one. But I think Joe Average would rate them just the reverse.. there are enough people that are still happy with VHS quality after all and that just looks terrible. Even a lot of DVDs don't look that great and the fact that directors add arbitrary noise only makes the situation worse.

[GodofaGap]

I think for an intra-codec comparison (look for one codec what options are better) PSNR certainly is an useful value. Except for CQM's I think any option that is supposed to increase quality will also increase PSNR. Codecs seem to be mostly tuned on achieving the highest PSNR, anyway.

The problem with PSNR is that there is no useful scale that correlates with subjective judgments like good, ok, bad, etc. It is very easy to pump a PSNR score by adding letterboxing (which won't increase quality of course) and it is also very easy to decrease PSNR by changing the brightness just the slightest bit (which won't really decrease the quality of course). So if you have codec A with score X and codec B with score Y, it is very difficult to say which one is better, because we lack this scale. Except of course when one codec scores idiotically low.

[Sagittaire]

PSNR compare simply equivalent pixels : if there are high convergence between source and video then PSNR is high


It is very easy to pump a PSNR score by adding letterboxing (which won't increase quality of course)

PSNR is not absolute quality measurement : only relative test. You must compare codec A vs Codec B or functionnality A vs functionnality B with exacty the same source.


it is also very easy to decrease PSNR by changing the brightness just the slightest bit (which won't really decrease the quality of course).

You can flip (horizontaly or verticaly) video or desynchronyse frame and obtain too very bad PSNR. But in this case source are not equivalente. Denoising, sharpening, tweak brighness, saturation or contraste decrease PSNR but these transformation make not equivalent source and no codec in the world make these pre-process for the source simply because it's not objective of video codec. The only objective for video codec is to make the highest possible convergence for eyes between output and input and PSNR can check that very well.

So if you have codec A with score X and codec B with score Y, it is very difficult to say which one is better

Well PSNR can really easy say:
MPEG4 ASP is better than MPEG2
MPEG4 AVC is better than MPEG4 ASP

For make PSNR judgement you must use threshold: if codec A obtain 43.0 dB and codec B obtain 43.1 dB we can't say that "codec B is better than codec A". But if codec A obtain 43.0 dB and codec B obtain 44.0 dB we can say with very high probablity that "codec B is better than codec A".

the threshold of uncertainty for PSNR is higher than the threshold of uncertainty for SSIM simply because HVS evaluation for SSIM is better than HVS evaluation for PSNR.

you do not believe in the metrics : well metric test for codec done exactly the same result that the last doom9 codec test ... try explain that like you want. All the codec developper in the world use metric to improve their codec ... try explain that like you want.

[Doom9]

you do not believe in the metrics : well metric test for codec done exactly the same result that the last doom9 codec test ... try explain that like you want. All the codec developper in the world use metric to improve their codec ... try explain that like you want.But there are people who disagree with my results ;) I don't have their eyes so unless claims are really outrageous, I can't really say their opinion is wrong.

And I gess I got SSIM eyes then because SND doesn't agree with my eyes.

[Mug Funky]

Well PSNR can really easy say:
MPEG4 ASP is better than MPEG2
MPEG4 AVC is better than MPEG4 ASP

but it can't prove it except under a very limited set of conditions.

it's used by codec developers to improve their codecs because they are doing several small controlled tests - they'll vary one thing and check the metrics. usually it's because the differences are imperceptible that metrics are used - we can't see 1 pixel difference in luma around the edge of a man's glasses while said man is being shot out of a cannon at high shutter-speed, but an objective metric can.

then there's codec optimisation techniques that would make it impractical to use visual inspection: i remember reading here Pengvado's way of finding an optimal command line for (i think) mencoder - construct a genetic algorithm with bitrate and PSNR as selection criteria, then breed the settings together. doing that with visual inspection would yeild possibly stupid and superfluous settings, but using PSNR helped remove the crap. contrast this with LAME's old "r3mix" preset which was a mixture of good sense and stupidity. however the analogy ends there because objective metrics mean even less to audio, where blind testing and statistical analysis is the only way to trust one's results.

it would be difficult to compare AVC with ASP when the loop filter is turned on - perceptual quality increases and PSNR drops.

and as psychovisual modelling in codecs becomes more sophisticated, we will expect to see more of this kind of thing happening - PSNR going down while perceptual quality goes up.

[Sagittaire]

But there are people who disagree with my results I don't have their eyes so unless claims are really outrageous, I can't really say their opinion is wrong.

well perhaps in DivX vs XviD fight but certainely not in ASP vs ASP fight.


And I gess I got SSIM eyes then because SND doesn't agree with my eyes.

You must use average comparison for user too. If SDN show not the same judgement then SDN is simply bad metric.


it would be difficult to compare AVC with ASP when the loop filter is turned on - perceptual quality increases and PSNR drops.

why ... ???

and inloop increase always PSNR:
- best size for -2:-2 (constant quant encoding)
- best PSNR for 0:0 (constant quant or size encoding)
- best SSIM generaly for -1:-1 (constant quant or size encoding)
- best default value for developper is in [-2;0] interval. x264 use 0:0, Elecard use -1:-1 and Ateme -2:-2

[dragongodz]

All the codec developper in the world use metric to improve their codec ... try explain that like you want.
lets be clear about why devs can use metrics before trying to espouse them saying metrics = quality. lets go back in the past a little
http://forum.doom9.org/showthread.php?p=698634#post698634

i give my reply a few posts down about how you misunderstand the statement you use/quote. so yes devs can and do use metrics to try and give clues or ideas where there codec/encoder may need improvement. this does not mean they suddenly think it will tell you whats the best quality all the time or to everybody because that simply is not the case.

EDIT:
oh and PLEASE stop saying silly things like "All the codec developper in the world" since you have absolutly no way of knowing such a thing.

[Sagittaire]

oh and PLEASE stop saying silly things like "All the codec developper in the world" since you have absolutly no way of knowing such a thing

here exhaustive list:
- Ateme developper
- Elecard developper
- x264 developper
- RV10 developper
- XviD developper
- DivX developper
- VP7 developper
- VC1 developper

... etc etc etc

and most of them use metric to compare their codecs with the concurence.



PSNR at constant quantizer gives a good idea of the raw efficiency since this is what core encoder try to optimize most of the time.
MPSNR/OPSNR add some hints about rate-control and SSIM (JND metrics or whatever) add clues about adaptive quantization.

A codec which is inferior is those 3 metrics will almost surely be seen as subjectively inferior.

Besides, subjective testing is also very difficult to do and is not always reliable.

Actually I have no example to contradict that ... but if you find this example make demonstration ... :)

[dragongodz]

here exhaustive list:
... etc etc etc
oh come on, thats plain insulting. when you can list all the codecs and encoders and show where they have said they use metrics then you have a case. until then you do NOT know that they all have. saying "many devs" would be more realistic, not this "all" you have repeatedly used.

Originally Posted by Babayaga
i already commented on that quote when you used it before. yes metrics can be used to try and spot problem areas or to try and increase efficiency etc etc and yes the end goal is to have the best quality with best efficiency as possible. that does NOT mean metrics give you an absolute on quality.
take the line
A codec which is inferior is those 3 metrics will almost surely be seen as subjectively inferior.
notice a very important word ? ALMOST. thats right , not an absolute.
so please stop pushing that quote as some evidence that metrics will ALWAYS show you whats best quality to everybody because it simply is NOT true.

Actually I have no example to contradict that ... but if you find this example make demonstration ... :)
are you asking me to show where a metric has failed ? if so i already answered that. its in the giant Rejig thread where Makira made a change that increased PSNR but when people started testing it the quality was actually inferior. now you have come back against that saying it cant have been OPSNR etc, well you would have to ask Makira as he was the one who did it. the point is its a real example of a metric failing which you choose to constantly not believe.

i am rather tired of having to say the same things to you about this so i wont bother anymore.

[Sagittaire]

notice a very important word ? ALMOST. thats right , not an absolute.
so please stop pushing that quote as some evidence that metrics will ALWAYS show you whats best quality to everybody because it simply is NOT true.

well I didn't say always and everyone. I say:
"for very large majority with high probability if there are subtancial difference"

little example :

at this time impossible to say which is the best between XviD and DivX with metric simply metric test are really close. Many users find that XviD is visually the best and many users find that DivX is visually the best.

-> That's the reality


at this time really easy to say which is the best between ASP and AVP with metric simply metric test are very best for AVC. Large user's majority find that AVC is by far visually better than ASP.

-> That's the reality


Many developper use metric to compare their codec with concurence and for me too "subjective testing is also very difficult to do and is not always reliable" simply because overall sujective judgement is really really hard (very easy for me to make demonstration here)

-> That's the reality

[Mug Funky]

i wonder what would look better between an AVC encode with +2,+2 vs an ASP encode with a higher PSNR? obviously this is strongly dependent on the amount of quantization going on, but my point is that the ASP encode would have visible blocking where the AVC wouldn't. on a lot of sources the AVC would look much better in spite of being further away from the original, metric wise.

hopefully one can see that such a metric-based comparison breaks down when it's applied to drastically different technologies.

however (and i suspect this is what codec developers would be doing, though i can't make such a call and i can't be bothered sitting up all night digging up quotes), "quality" metrics are valid and very useful when tweaking settings within the same codec, on the same source. tests of this kind must be controlled or they are invalid and useless (like going xvid + fft3dfilter versus divx + conv3d in a PSNR battle would say nothing about either the codecs or the filters).

also, though i will look at the numbers when doing an encode, nothing goes on a DVD master that hasn't been watched by a human in a controlled environment. though quants/metrics can tell us where the problem areas might be, they wont say whether there's other problems going on. in the case of a total A/V dropout, PSNR can go to -inf, and quants will drop to 1, but that doesn't mean the quality is good because in fact it's a total disaster and will require several hours in photoshop cursing the name of certain equipment manufacturers.

[edit]

Many developper use metric to compare their codec with concurence and for me too "subjective testing is also very difficult to do and is not always reliable" simply because overall sujective judgement is really really hard (very easy for me to make demonstration here)
if the codecs are subjectively tied, it surely just goes to prove that the metrics have a long way to go in modelling the human visual system! i mean if the metrics say there's 3dB difference when you see none, what on earth does that say about the metrics? such a comparison is meaningless IMHO unless someone is interested in a "my codec is bigger than yours" zealot battle with numbers as ammunition.

[Sagittaire]

however (and i suspect this is what codec developers would be doing, though i can't make such a call and i can't be bothered sitting up all night digging up quotes), "quality" metrics are valid and very useful when tweaking settings within the same codec, on the same source. tests of this kind must be controlled or they are invalid and useless (like going xvid + fft3dfilter versus divx + conv3d in a PSNR battle would say nothing about either the codecs or the filters).

also, though i will look at the numbers when doing an encode, nothing goes on a DVD master that hasn't been watched by a human in a controlled environment. though quants/metrics can tell us where the problem areas might be, they wont say whether there's other problems going on. in the case of a total A/V dropout, PSNR can go to -inf, and quants will drop to 1, but that doesn't mean the quality is good because in fact it's a total disaster and will require several hours in photoshop cursing the name of certain equipment manufacturers.

Doen't mean anything for metric test: you must always compare exactly equivalent source. The only objective for video codec must be to obtain the most possible convergence for eyes between input and output. If input is crap objective to video codec must be simply to make exactly the same crap video in output. Improve quality of the source is not the objectif of a video codec ... :rolleyes: ... only pre-process for that.


i wonder what would look better between an AVC encode with +2,+2 vs an ASP encode with a higher PSNR? obviously this is strongly dependent on the amount of quantization going on, but my point is that the ASP encode would have visible blocking where the AVC wouldn't. on a lot of sources the AVC would look much better in spite of being further away from the original, metric wise.

1) PSNR for AVC is always better than ASP and by far at same size
2) "result better than original" doesn't mean anything for video codec

[foxyshadis]

Doen't mean anything for metric test: you must always compare exactly equivalent source. The only objective for video codec must be to obtain the most possible convergence for eyes between input and output. If input is crap objective to video codec must be simply to make exactly the same crap video in output. Improve quality of the source is not the objectif of a video codec ... :rolleyes: ... only pre-process for that.

1) PSNR for AVC is always better than ASP and by far at same size
2) "result better than original" doesn't mean anything for video codec
This is why metrics that compare only against the original are fundamentally flawed as arbiters of quality. Being told that a frame has a 20db difference from the original is useless when what you really want to know is if the output is blocking, noisy, ringy, blurry, blended, or has other common video artifacts, whether the original is or not, as I said in the other thread. There's so many ways that the encoder can change the quality, different types of reductions can be more or less irritating at different times, in different areas.

I hope someone comes up with a quality evaluator like that someday, it'd be so useful. (For one thing, guis could use it to place certain filters in their scripts, and tweak them in certain frame ranges as necessary, extending the trend of automated filtering.)

[Sagittaire]

This is why metrics that compare only against the original are fundamentally flawed as arbiters of quality. Being told that a frame has a 20db difference from the original is useless when what you really want to know is if the output is blocking, noisy, ringy, blurry, blended, or has other common video artifacts,

but SSIM make that ... :search:
http://www.cns.nyu.edu/~zwang/files/research/quality_index/demo_lena.html
http://www.cns.nyu.edu/~zwang/files/research/quality_index/demo_blur.html
http://www.cns.nyu.edu/~zwang/files/research/quality_index/demo_jpg.html
http://www.cns.nyu.edu/~zwang/files/research/quality_index/demo_couple.html

you speak about metric but you don't use it or you don't know their real capacity and limitation ... ;-)

[siddharthagandhi]

Both are reliable tests. SSIM may be better, but not that much. However, they are by no means perfect and they really need to be improved. But they generally should be good.

I am using PSNR for my upcoming codec comparison.

[Soulhunter]

I think it always depends on... what you wanna test with the metrics, how you to do the test, and how you interpret the results... In some cases metric test can be reliable, in others not, eh!? ^^


Bye

[dragongodz]

well I didn't say always and everyone. I say:
"for very large majority with high probability if there are subtancial difference"
hmm and who exactly is it that whenever some says metrics are not always reliable jumps in to defend them to the hilt again ? take the thread i already linked to. i said
then again its been proved before that metrics are not always reliable
to which you respond with
All video codec developper in the world think the opposite ...
so is it they are always reliable or they are not ? you cant have it both ways. if you are now saying that they are mostly reliable then that makes your previous statement in that other thread false.

Many developper use metric to compare their codec with concurence and for me too "subjective testing is also very difficult to do and is not always reliable" simply because overall sujective judgement is really really hard (very easy for me to make demonstration here)
by this i assume you mean that you get x amount of people liking sample A while x amount like sample B etc etc so which x are you mean to believe ? if anything thats just more proof that anything or anyone who tries to say what looks best to everybody is never going to be correct ALL the time. that doesnt mean metrics can not be used to aid in development but to think it is the final arbiter of quality is to ignore exactly what the target is for the end product, people that is.

-> That's the reality
hahaha what exactly was putting that after each statement supposed to prove ? it just looks silly. :D

[zambelli]

I personally think PSNR and SSIM are useful in giving one a good general idea of what might look better. The final test of quality should always be made visually. However, when you have 200 possible encoding options - PSNR and SSIM are very useful in narrowing down the candidates. Why waste time manually evaluating something that has a 3dB lower PSNR than another encode?

[siddharthagandhi]

3db= how much PSNR?

[foxyshadis]

but SSIM make that ... :search:
http://www.cns.nyu.edu/~zwang/files/research/quality_index/demo_lena.html
http://www.cns.nyu.edu/~zwang/files/research/quality_index/demo_blur.html
http://www.cns.nyu.edu/~zwang/files/research/quality_index/demo_jpg.html
http://www.cns.nyu.edu/~zwang/files/research/quality_index/demo_couple.html

you speak about metric but you don't use it or you don't know their real capacity and limitation ... ;-)
????

I don't see SSIM spitting out a blurring measure, a blocking measure, a ringing measure, a rainbowing measure, a film grain measure, a blending measure, a shifting textures measure. I'm asking for a measure that gives a qualitative assessment, and not just a stupid catch-all number. This would be useful to all the rest of the people in the world who aren't just testing codec options, who want to know what's being spit out (and what's being input so they don't blame it all on the codec ;)).

I understand how hard a problem this is, and I'm not saying SSIM is useless as a way to quickly test options and codecs - but as an indication of video quality? I can never accept that, except in the case of a source with absolutely perfect quality. (say, CGI.) Online trailers, and especially DVD sources are not perfect quality sources, even filtered, and from what I've seen aren't even close in many cases.

[Sagittaire]

????

I don't see SSIM spitting out a blurring measure, a blocking measure, a ringing measure, a rainbowing measure, a film grain measure, a blending measure, a shifting textures measure. I'm asking for a measure that gives a qualitative assessment, and not just a stupid catch-all number. This would be useful to all the rest of the people in the world who aren't just testing codec options, who want to know what's being spit out (and what's being input so they don't blame it all on the codec ;)).

I understand how hard a problem this is, and I'm not saying SSIM is useless as a way to quickly test options and codecs - but as an indication of video quality? I can never accept that, except in the case of a source with absolutely perfect quality. (say, CGI.) Online trailers, and especially DVD sources are not perfect quality sources, even filtered, and from what I've seen aren't even close in many cases.

Absolute video quality doesn't mean anything: it's pure subjective and artistic statement. Metric was, are and will never able to evaluate absolute quality source. Example : add noise, add blur, add grain, add 35 mn old film degradation ... can be purely artistic "degradation" for reproduce old film sequencies for example. In this case denoising or sharpening is nonsense.

1) The only objective for video codec must be to obtain the most possible convergence for eyes between input and output.

2) Modification for video (denoising, sharpening ...) must be always make in pre-process because all "video quality improvement" are purely subjective and "artistic".

3) Subjective or Objective comparison must be always make in comparison with original source. If source is crap then best codec is simply the codec who reproduce the best possible crap source. Perfect codec is by definition lossless codec.


hahaha what exactly was putting that after each statement supposed to prove ? it just looks silly

the opinion of babayaga about compression efficiency is more important for me ...

Metric done very good evaluation (not perfect anyway) of the video compression and that is a fact ...

[Wilbert]

3db= how much PSNR?
PSNR is measured in decibels. It's defined as

PSNR(I,K) = 20 * log10 ( 255/sqrt(MSE(I,K)) )

with

MSE(I,K) = 1/M * sum_{j,k} || I(j,k) - K(j,k) ||^2

and (j,k) runs over all pixels in a frame, and M is the number of pixels in a frame.

http://en.wikipedia.org/wiki/PSNR

[Manao]

When the PSNR ranges in 30 - 45, there's a thumb rule saying that 1% bitrate <-> 0.05 dB. So 3dB is huge in that regard ( 60% of bitrate, which represents more than the difference between quantizer 20 and quantizer 26 in h264, for example )

PSNR is a dumb metric, but it has several advantages for the developpers that makes it an important tool : it's fast & simple.
it supposes that each macroblocks are independant, so it's rather easy to optimize for it.
it's used by everybody, so any comparison may begin by a PSNR test to see quickly, painlessly, what it's like.

Of course, the second point makes it completely invalid as soon as we want to take psychovisual information into account, but if we use it knowing its drawbacks, it doesn't matter.

And, for the record, since I've started working on codecs, I don't like SSIM that much because : it's slower than PSNR
it's not as universal as PSNR
it makes the macroblock interdependant, so it's harder to optimize for it
it doesn't take the temporal aspect into account, while it is the most important part of the psychovisual perception. And that's the major drawback

So basically, SSIM is great for still pictures, but imho as shitty and slower than PSNR for videos, as soon as we want to use it to rank codecs, because the most important psychovisual effect isn't taken into account.

[CruNcher]

PSNR + Visuall Test the way to go for me and Visuall Tests allways get higher priority then PSNR results :D


it doesn't take the temporal aspect into account, while it is the most important part of the psychovisual perception. And that's the major drawback


exactly the reason why i don't trust it as much as i trust my eyes ;)

[Sagittaire]

PSNR + Visuall Test the way to go for me and Visuall Tests allways get higher priority then PSNR results :D

exactly the reason why i don't trust it as much as i trust my eyes ;)

Well I have recent example who invalid that ...

Recently I test new codec Y: developpers send me HPII trailer for my metric challenge:

first visual impression:
Codec Y is visually better than codec X the previous version

metric test
Codec X done better Overall PSNR than codec Y
Codec X done better SSIM than codec Y

for first time my eyes evaluation don't have the same result that metric test. I try to explain that and I make Graph PSNR to compare codec X and codec Y:

Codec X vs Codec Y
http://multimediacom.free.fr/Video/VCMvsVC1-450.PNG

Graph PSNR show clearly that codec X is better than Codec Y for these green sequencies. Codec X is better than codec Y for overall PSNR and Codec Y is probaly really better than Codec X for Average PSNR. Bad frames are really important for Overall PSNR and codec Y produce very bad frame certainely in high motion part.

I make sample in high motion part (550 frames) and I make new visual comparison : this time Codec X is visually better than codec Y for these sample in high motion part.

Now I compare size for this high motion part:
Codec Y done 1503 Ko
Codec X done 1707 Ko
Ateme done 1808 Ko
Elecard done 1657 Ko
x264 done 1735 Ko

Result is better for codec X in high motion part simply because bitrate for this part is higher ( 620 Kbps vs 546 Kbps)

Conclusion: IMO codec Y is optimized for Average PSNR and not for Overall PSNR. codec Y produce very bad frame in High Motion part because Rate Control is really agressive for these part and quality is visually really not constant. IMO codec Y efficiency seem really better than previous codec X and IMO can potentially produce better Overall PSNR with better Rate Control and thus certainely very better overall subjective quality.
But why my brain say codec Y is visually better than codec X for first visual impression? Simply because first frame are subjectly really important and first frame are low motion and are better for new codec in this trailer. Simply because it's really hard for brain to make real overall judgement for more than 3000 frames like for this trailer and certainely impossible for more than 180 000 frames ...

[CruNcher]

@Sagittaire
ehh you didn't test my EDP build here did you ? ;)
actually that's the same HVS RC approach i use in my build (looks like they stollen it from me grml ;) )


But why my brain say codec Y is visually better than codec X for first visual impression? Simply because first frame are subjectly really important and first frame are low motion and are better for new codec in this trailer. Simply because it's really hard for brain to make real overall judgement for more than 3000 frames like for this trailer and certainely impossible for more than 180 000 frames ...


very simple answer you're doing to much Metric tests and get very confused here and your brain and eyes almost never realize the quality problems in those short high motion scenes the problem is the runtime of those, for example if you have a car drive by the camera very fast and then cuts into another scene a second after that you wont really realize the few bits that are taken from that scene and distributed to enhance a low motion one that lasts over a couple of minutes (where your eyes constantly focused on the scene) :) But when this High Motion scene is for example a Fight Scene like in the Matrix (that lasts over a longer time period) where you can follow the action with your eyes it gets problematic to use this approach and that's what im currently trying to detect and compensate (but also doing some other balance stuff also for H.264 HD and some special Profile for CoreAVC and Realtime Pocketpc Playback) in my build, it's nice to see others trying the same good work whoever changed to that aproach now :)

Look here and you realize the same as in the graph above hehe http://forum.doom9.org/showthread.php?p=586938#post586938

btw we allready discussed about that http://forum.doom9.org/showthread.php?p=589109#post589109

[Sagittaire]

very simple answer you're doing to much Metric tests and get very confused here and your brain and eyes almost never realize the quality problems in those short high motion scenes the problem is the runtime of those, for example if you have a car drive by the camera very fast and then cuts into another scene a second after that you wont really realize the few bits that are taken from that scene and distributed to enhance a low motion one that lasts over a couple of minutes (where your eyes constantly focused on the scene) But when this High Motion scene is for example a Fight Scene like in the Matrix (that lasts over a longer time period) where you can follow the action with your eyes it gets problematic to use this approach

Well but if you change sequences order the visual impression will be different in this example simply because quality for first frame are more important for visual impression. IMO if you want make real blind test you must cut you video in multiple sequencies with no more than 250 Frame.



ehh you didn't test my EDP build here did you ?
actually that's the same HVS RC approach i use in my build (looks like they stollen it from me grml )

well I don't know : previous codec (codec X and codec Y) use inloop and post-process. Inloop is very usefull in high motion part if you use curve compression for Rate Control simply because quality degradation is visually lower with inloop. AVC codec can make that but I don't know if it's good idea for ASP codec. Don't forget that:
- ASP use bframe in low motion part generaly with ratio/offset
- Inter quantisation can be more or less agressive than Intra quantisation with CQM (for example with HVS matrix)

Libavcodec ASP can change curve quantisation compression too with vqcomp (with high default value curve compression vqcomp = 0.5) and prestigious user like doom9 ( see last doom9 comparison and libavcodec disqualification) ) find that quality in high motion part are very bad. Many user like me prefer constant quantizer encoding for ASP (in fact not real constant quantisation with ratio/offset for bframe and with CQM).

Anyway I will test your build simply because say "it's bad" without testing is not my philosophy