How do you quantify acceptable performance/quality tradeoffs?

[zambelli]

As we all know, better image quality in encoding most commonly means more complex encoding work and consequently slower encoding times

That poses a question: how much better does the quality need to be (and how does one quantify it) in order to justify the slower encoding performance?

To be clear: I'm talking about the quality/performance tradeoff of the same codec. Comparing various codecs against each other in this regard is another interesting, but completely different topic.

For example, if you can improve the average PSNR of a particular encoding mode by 0.5dB, what would you consider a justified slowdown in encoding time? 5%? 15%? 50%?

I'm interested in hearing your theories, and if anyone has examples and data to back them up - even better. :)

[Revgen]

I will do anything to insure that a video I encode looks as close to the original as possible while being at least half the size.

I'm usualy willing to encode it as long as possible (days not weeks) in order to do this.

[Mug Funky]

the tradeoffs are wholly dependent on scenario.

for example, 1-pass vs 2-pass seems like an easy choice, but even then there's a huge number of things to consider.

at work if i'm using a software encoder i'll go 1-pass because it's much much faster, and the bitrates are high enough to absorb any (rare) difficult scenes. even if it fails on a scene, it's quicker to drop in a re-encoded segment than do the whole thing at 2 pass.

on the other hand, if i'm using one of the hardware encoders (all 3 are c-cube based MPX3000 cards), i'll do 2-pass for anything under 8 megabits simply because the CBR is good, the 2-pass is good, but the 1-pass is woeful and almost totally useless.

[zambelli]

Hmm, I'm not sure you understood the question - or perhaps I didn't phrase it well enough.

It's not a top-level "which encoding method do you use" kind of question. It's more about "how much speed are you willing to give up for better quality" and I was curious to see if anybody would attempt to quantify it with real metrics and numbers.

Imagine you already made up your mind about the encoding method and the encoding parameters you want to use, and have already encoded the content once. Now, the codec authors come and say, "We've optimized the codec a little bit with regards to image quality. Give it a try." So you try the new codec build and find that indeed, the image quality for the same encode has gone up by, say, 0.5dB in Overall PSNR... but you also find out that the encoder is now 25% slower than before.

My question is: where do you draw the line? What would you define as an acceptable tradeoff between image quality and performance? At which point do you proclaim the increased encoding time to be unworthy of the quality gain?

[GodofaGap]

The problem with your question is that we lack a reliable and objective method for measuring video quality.

[zambelli]

The problem with your question is that we lack a reliable and objective method for measuring video quality.
The guys at MSU would probably disagree. :) There are plenty of methods available: PSNR, Delta, MSAD, MSE, SSIM, VQM, etc. A part of the question is also about how you quantify quality and which metrics you trust.

[GodofaGap]

I think you missed the word 'reliable' there. :p

MSU guys have made this test (http://www.compression.ru/video/codec_comparison/subjective_codecs_comparison_en.html) remember. If you look at the correlation between metrics and subjective scores, you can see that for 2 out of 3 metrics there is none. For SSIM it's not too bad, but also not mind-blowing.

[unmei]

Even if we stick with PSNR for once, it makes a difference whether you are at 35dB or at 50dB already (numbers for example only)..

If i am at 35, for example because i am under extreme bitrate constraints, i will most likely want enable every single feature that gives me even a 0.2dB boost. On the other hand, basing on "50dB setup" it is most likely not not worth to add encoder features that gain 0.1dB at a 50% longer encode time.

Normally, i think, my encodes are around 44-49dB "global" psnr (from what x264 tells me) and enable about every feature in x264 except exhaustive search and i limit the number of reference frames to 3..5

[dragongodz]

Hmm, I'm not sure you understood the question - or perhaps I didn't phrase it well enough.
hmm and i would question first if there are really that many people who would or do already do a metric test ,even on just a sample, before doing an encode.
more likely they would encode a small sample and just use their eyes to decide if it was worth turning on extra quality features. even then a lot would probably just go by bitrate etc. that is if they were encoding to a low bitrate they would turn on the options and if to high bitrate they may (or not) turn some off.

[zambelli]

hmm and i would question first if there are really that many people who would or do already do a metric test ,even on just a sample, before doing an encode.
I must be one of the few then. :) Checking everything manually can be really time-consuming if you're dealing with a lot of parameters. If I have a setting that I've never used before and I want to know how it affects quality, there's nothing easier than writing a short batch file to kick of multiple encodes with varying setting values, and then running PSNR and SSIM metrics on them. Rather than inspecting every encoded video frame by frame, it's much easier to look at the numbers, isolate the top 2-3 contenders and *then* inspect them manually.

[zambelli]

If i am at 35, for example because i am under extreme bitrate constraints, i will most likely want enable every single feature that gives me even a 0.2dB boost. On the other hand, basing on "50dB setup" it is most likely not not worth to add encoder features that gain 0.1dB at a 50% longer encode time.
A fine point (and exactly the kind of observation I'm looking for here). It sounds like you already have an opinion as to which quality increases are worth waiting for, and which aren't. Would it be fair to say then that the "worthy quality increases" grow with the overall quality of the encode?

[Mug Funky]

i understood the question. i was pointing out that it's not black and white, and that it depends very strongly on many constraints and variables. so much so, that to give a simple linear relationship like "time/quality" would be senseless except for 1 situation and probably only 1. perhaps a plot of the acceptable time/quality tradeoff versus bitrate would be useful, but it'd only be relevant for 1 encoder with 1 set of parameters, probably on only 1 source.

also, for .5 dB i wouldn't even flinch :) i'd choose the fastest method unless the difference was more significant (in the order of 3dB).

hypothetically, if a codec developer slowed their baby down by 25% for a .5dB gain, and didn't at least allow the old behaviour to be used, i simply wouldn't update, (actually, i'd request the developer to allow us to use the old method via a checkbox or similar).

but to give an idea where i'm coming from, i encode almost exclusively above 4 mbps, MPEG-2. usually it's 1-pass because the current line of encoders out there have good 1-pass ratecontrol, and at 4+mbps, quite often the quantizers are pegged at 1 for the length of the movie, except in action scenes, or in interlaced video. if the quant is pegged at 1, then 2-pass is obviously not going to give an advantage.

[siddharthagandhi]

Personally, I don't care how long it takes if the quality is really good. For example, if I could get 10 mbps bitrate quality at say 15 mb/minute I would let my computer run 24 hours + in order to get that. It (the quality) has to be much better if the time is longer than an hour or so.