OPV & Multipass Experiments

[Sir Didymus]

Let me post again some links about the first test performed:

xls sheet & original images:
http://wpop14.libero.it/cgi-bin/vlink.cgi?Link=http%3A//webmail.libero.it/r.php%3Fd%3Dlibero.it%26wr%3D3zqcxucf%26ws%3D3zqcxucf%26e%3Dlibero.it%26c%3Dn9LxHgvnSfiQE0J593lQOkFLcLcYwiY16165

original m2v:
http://wpop14.libero.it/cgi-bin/vlink.cgi?Link=http%3A//webmail.libero.it/r.php%3Fd%3Dlibero.it%26wr%3D3zqcxucf%26ws%3D3zqcxucf%26e%3Dlibero.it%26c%3D6XF6oayyjkR3El935ac0ULDST0yfBbvy6287

test 1+1 mv2:
http://wpop14.libero.it/cgi-bin/vlink.cgi?Link=http%3A//webmail.libero.it/r.php%3Fd%3Dlibero.it%26wr%3D3zqcxucf%26ws%3D3zqcxucf%26e%3Dlibero.it%26c%3DLcmtwHau1jksPjtj1bgR3pRNoov4XyEJ6221

All links are working. Just tested. They are supposed to be active until five more days. [Deadline 04.12.2004 03:30:48 PM CET]

Currently performing some encodings at lower bitrates than 4250, and with different quanti matrices. I will post the results (that are personal evaluations, [if it will be allowed to me]) in the next, few, days.

Indeed it seems to me very important to add the following:

1. No evident mistakes seems to me are present in what I proposed last week on the subject. I have already demonstrated to be a correct person and to be friendly and open minded towards everybody: if someone demonstrates my tests are wrong I am ready to say sorry and to change my opinions; if I discover some of my tests are wrong it is my interest to inform asap everybody. This is the only way I know (that is to apply the scientific method) to ensure the reliability and the replicability of experiments.

2. Troubles on uploading links are due to the service I am using (that is clearly not very stable), and the five days timeout is a limit of the service, that isn't under my control. Since I don't have an ftp site available to host my files, I am using this way to share big chunks of information.

3. I do not accept ultimatum deadlines on my testings from anybody. I am doing the experiments on my spare time and (I hope) I am free to use the time of my weekends with my family, and without being pressed or available on line, if I decide so.

4. Everybody, even the most stupid person on the earth, starting from the single source m2v, and following the detailed procedure I described, is supposed to be able to replicate the test, without any need of the other files. Using the posted ecl, it is possible and easy to generate all the needed m2v files (from and including 1+1, to 1+9) and to visually compare them.

5. Since I was the target of some very heavy and full of poison personal comments, I want to thank very much Jdobbs for closing the thread on the multipass evaluation. Well done and really appreciated.

[mod edit]

SD

[Rockas]

@Sir Didymus
Keep up the good work... sorry... the excellent work :)
... by the way... did you take a look to the help files?
Your opinion will be welcome

keep it UP

[Sir Didymus]

Thanks, Rockas :)

Even if it is a little bit OT, your post is very appreciated. Really.
While we are OT: of course I did..., And it seems to me very, very good...
Asap I will give to you some more structured comments, in the Help File Thread...

[TheSeeker]

@Sir Didymus

Thank you very much for you hard work on these tests. I know I personally am very anxious to see what you have found. ANd what you will find in the future of your testing. [mod edit] Anyways keep up the good work. Now to check out thoes links.....

[Sir Didymus]

Hi TheSeeker!

Thanks a lot for your comments. Really appreciated!!!

Just let me say:
1. I didn't find or discover anything relevant. That's sure. I mean, this thread is about questionable opinions, that may be supported by SSIM tests and analysis, but aren't facts, you know quality is subjective...
2. So, no reason to be anxious... Even because tests with std matrices are now complete, and in the next few (one or two) days I will start posting something...
3. If you can not wait 2 days and you want to exercise yourself a little bit with Italian, you may want to start getting a look here:
http://forum.doom9.it/viewtopic.php?p=58662#58662

Cheers
SD

[TheSeeker]

Hey Didy,

I noticed from the results in that link that OPV seems to be providing a higher SSIM in most cases. What would explain this and does this mean that OPV may produce better quality than multipass. Other than that though it seems like 1+3 passes seems to be the magic number, at least for SSIM accuracy.

[Sir Didymus]

Sssss!!!

Do not say such things at such loud voice !!! :cool:

Edit : your Italian teacher did a damn quick job...
Edit2: next time I will post before into my favourite Japanise forum...

[TheSeeker]

Sorry about that... I can edit out my post if you want me to do so.

[Sir Didymus]

No, no, just kidding...
There are no secrets to keep veiled... (I hope) :scared:
Just need some little more time to elaborate...

Cheers,
SD

[pg55555]

SD

Very good job.
I was peeking into your italian posting and found the result very interesting.

I have some comments but I will wait to the posting of your results in this forum.

Pablo

[ShadowKnight]

Can't wait to hear more :) sounds like a great idea, and definately something that is subjective, but very important to discuss. I'm very interested to see the results and hear the opinions :)

[Rockas]

@Sir Didymus
Thank you :)

keep it UP

[gizzin]

Haha, OPV will never produce better quality than multipass. But still good work Sir Didymus. I really appreciate the compliancy work. Keep it up.

[TheSeeker]

It may not produce better quality but that isnt what SSIM measures. It measures the videos similarity to the original stream. With OPV the encoded stream may be closer in similarity to the original but be of lower quality. Same with many transcoders. They might have a higher similarity to the original stream, but for high compression we all know that encoders do a much better job, as far as quality.

[Boulder]

Thanks SD, I'll just have to wait for my broadband connection to get fixed:mad:

@gizzin: one point of the comparison is surely to check whether it's true or not. Never say never;)

EDIT: results look quite interesting, very good job!

[Sir Didymus]

Originally posted by gizzin
Haha, OPV will never produce better quality than multipass. But still good work Sir Didymus. I really appreciate the compliancy work. Keep it up.

Hi gizzin. :)

Your appreciation ...is appreciated... By the way, while we are OT, I may add that at least in the compliancy job, even when I did some very strong statements I never received personal insults from anybody... So maybe better to switch back to that when these my never ending encoding sessions will hopefully conclude... :scared:

Ok. let me give you some answer (and I am trying to do this in the most educate way I can. If you feel I am not, let's meet to discuss the issue with the arms of your choice, as gentlemens, or in a coffee shop in front of a couple of beers [preferred...] ;) ...).

In short terms: when your say: "Haha" I will tell you "Hihi"...

How can you just state that "OPV will never produce better quality than multipass" ? Tout court ?

Get a look to this graph:

http://img86.exs.cx/img86/8075/graph1-4250.gif

It is extracted from the encoding at 4250 Kbps, where actually the VBR seems having in average a better behaviour (just to be clear: I am speaking about the SSIM behaviour). You may easily understand that even if in some few points the OPV encoding is clearily worse, in much more points (frames) OPV shows a little bit better figure. You should not be a matematician to understand that it may happen that in average the OPV behaviour may be better right ?

Ok I have also some hypotheses that may explain why it may happen, in some (not rare) cases that OPV is better...

Do you have any information or references or experiences or hypotheses to explain why OPV will never produce better quality than multipass ?

Cheers,
SD

[Sir Didymus]

Ok. Even though most subjective evaluations are still pending, let me anticipate some test results with SSIM and CCE encoding (quanti mat standard).

A very weak point on the whole test is that it was just encoded a single cell -V1C3- of the Kill Bill Vol. 1 movie. This is very limitative. Nevertheless the cell contains both action and static scenes, so let's stay with it.

Following the procedure described in another post, it has been generated a 4250 kbit/s ecl control file using the standard settings of DVD-RB.

This single ecl file has been manually tweeked for producing VBR encoding sessions from 1 to 9 passes, with bitrates of 4250, 3750, 3250, 2750, 2250 Kbit/s and two OPV encoding sessions finding EMPIRICALLY the Q factors producing m2v files nearest as much as possible to the target bitrate. This is to simulate the DVD-RB behaviour in case of a good estimate of the Q factor.

I have to take (or better to save from a thread that for other reasons is better to forget) some comments, that I totally share, on the matter of robot1, Jdobbs and Dragongodz. The synthesis is:


--> to be very careful before drawing conclusions on anything, since
SSIM index IS NOT SUITABLE to compare encodings generated by
different encoders/transcoders (this is fundamental, since the
GOP structure is mandatory it is the same, otherwise it may be
very frequent to compare I frames with B frames, and such thing
is meaningless).
--> do not blindly believe in the numbers and in statistics!
--> the only really reliable and sure tools for evaluating the
quality of your encodings are your eyes.


by robot1:
While I think that SSIM results are valid in test with the same encoder (as SD did with CCE), I doubt about the results when you compare different encoders (or transcoders). I did some tests (posted on the doom9 italian forum, if someone is interested I could find the link) with CCE, Shrink2.1 (not the last one!) and Procoder. Quality of Procoder and CCE was very good, while Shrink had a lots of blocks (not true for the last version, but I did the test in july). Anyway SSIM of the Shrinked clip was better than the one of the CCE clip (which was perfect, but the colors were just a bit washed). Looking at the backup, the CCE one was very pleasant, and you couldn't notice any differences from the original in normal play. Procoder quality was at the same level (or even better, for my taste).

In short: I think one should use SSIM to test only clips encoded by the same encoder.


by Jdobbs:
As was mentioned earlier -- be careful if you intend to use SSIM to compare different packages. I have seen comparisons that to the eye were intuitively obvious to the casual observer, but in SSIM values said the opposite. Numbers can lie.

My favorite example is the statistics of the hospital. In the U.S. ten times more people die in hospitals than other places. So the numbers say you have a 10:1 better chance of survival if you don't go to the hospital... hmmm...

SSIM is great for comparing apples to apples...


by Dragongodz:
if i may offer a couple of small comments.

while looking at charts done for SSIM or whatever quality evaluation tool is interesting it will never replace the 1 tool you will always use. that is your own eyes. they will give you more real feedback than anything else and can never be fooled as to what you will think looks better.

also certain things can be taken as true, such as throw enough bitrate at something and no new artifact or blocks etc will be created and it will look like the original(or pretty damn close). however comparing the quality of different encodes will always be individual and opinion(which should not need to be said) as everyone is an individual with individual tastes. person A will like encode A better while person B will like encode B better, thats life.

so please remember that when asking for proof of quality or such.


Here is the synthesis of the SSIM values aggregated for each test, reporting the average SSIM, the standard deviation and the filesize of the produced m2v files. The single subjective test I did on the matter (and I am very confident about this) is that for any given bitrate encoding family (I mean, for ALL of the encoding performed, for example at 3750 Kbit/s) it was absolutely impossible to myself to VISUALLY distinguish any difference... Even using the still frame comparison based on the avs script I posted previously. Further comments will follow, but for the moment I just did some VISUAL comparison between 4250 and 2250, and in this case the difference IS actually very noticeable: the noise and the little artifacts presents for the lower bitrate encoding, especially compared at still frames, is evident...

http://img118.exs.cx/img118/9862/stdmatravgssim.gif

http://img118.exs.cx/img118/1858/stdmatrstddeviation.gif

http://img118.exs.cx/img118/4465/stdmatrfilesize.gif

[Boulder]

The single subjective test I did on the matter (and I am very confident about this) is that for any given bitrate encoding family (I mean, for ALL of the encoding performed, for example at 3750 Kbit/s) it was absolutely impossible to myself to VISUALLY distinguish any difference... Even using the still frame comparison based on the avs script I posted previously.


This is one point why I prefer OPV. No need for those extra passes, saving precious time.

One point for using multipass VBR is getting strictly the desired bitrate. But even then, 1+1 passes should be enough.

However, with a good prediction method, the Q value can be predicted quite accurately. Tylo has achieved this with his D2SRoBa plugin for DVD2SVCD, and it also includes the possibility to do a sizing pass (or use a transcoder) if the filesize error is over the boundaries that the user sets. Based on my couple of tests, RB-Opt gets close to the target as well.

[Sir Didymus]

Just forgot to mention, that the full set of xls files can be downloaded from:

http://webmail.libero.it/r.php?d=libero.it&wr=3zqcxucf&ws=3zqcxucf&e=libero.it&c=pRY537m6OHGbLviOhR9IOTJVOq4vUQDl8021

Starting from these files, it is very easy (with just a minimum of Excel practice) to perform specific SSIM frame by frame comparisons and generation of graphs similar to the one posted before...

Again sorry, but the deadline for downloading is until 6-dic-2004 18.39. Clearly, after that date, you may ask me via PM to get them.

Cheers,
SD

[robot1]

Great works, SD.
I think I will never do more than 1+1 passes!
OPV is great for saving time, the only drawback I can see is the size prediction, which could be not very accurate.

Hint for DVD-RB users: if you set an OPV project, and the final size isn't right (but differs less than 10% from the target size), you can change the project in VBR using RB-Opt, and DVD-RB will perform the second pass, keeping the .vaf

[Sir Didymus]

Hi :(

I have one bad new, and one good new on the topic...

I am sorry to inform that I discovered one very little mistake in the data I posted.

1. Using VirtualDub 1.6.1, it happens that the dubbing ends to the frame before the last in almost every play I did for computing the SSIM values. This is introducing a very little systematic error in the LAST TWO FRAMES of the SSIM xls files. The recovery is very easy (and I am almost sure that nothing will change in the posted information) and consists in using the first 6795 of the 6797 SSIM values, but this implies the two tables of SSIM average and standard deviation should be edited. I will try to do the job asap.

2. Full encodings using a low bitrate matrix for doing the same experiment are complete. Hopefully this evening I will be able to post similar tables for the low bitrate quanti mat...

Cheers,
SD

[Boulder]

To get the thread back on track: have you considered any PSNR measuring? The folks at XviD department seem to use it with their quant matrix tests and when testing new unstable builds.

[Sir Didymus]

Hi...

Please find in this post the corrected tables reporting Average and Standard Deviation for the SSIM values produced in the test...

I have to say and to admit (as you may see) that the errors present in the original tables were two and very nasty...

So I decided not to remove the previous (buggy) tables, and to post the new tables here, adding some comments for explaining these errors...

Error 1:
In the calculation of the average SSIM values, the last two frames of the cell were included in the test. These frames were sometimes totally wrong, due to some VirtuaDub troubles... Overall effect wasn't to introduce a systematic error (as I hoped), but a seriuos mistake capable of changing some conclusions regarding the OPV behaviour at 4250 Kbit/s.

The shortcoming has been simply not to use these two last frames in any of the calculation of the SSIM average values.

Error 2:
Big mistake (and this was really my fault): I didn't properly understood that the overall average value (generated by the SSIM plugin and reported in my data) is not the simple aritmethical average of the frame-by-frame SSIM comparison, but it is a weighted average (where the weights are included in the second column of the csv files).
This is very interesting (and useful) since it allows to give for example no relevance to black frames, and to give more weight to frames judged relevant by the SSIM evaluation algorithm... Anyway, in the previous posted tables, the Std. Deviation was calculated arithmetically, without considering the weights...
So the Average tables were just a little bit wrong, and with a little bit of Excel work it was not difficult to recalculate the right values, while the Std Deviation table (which is a fundamental information to consider in the test) was definitely wrong... And its recalculation took to me some more time than I expected...

Hope the attached information, and the whole work it took to produce it, is worthwile...

Cheers,
SD

http://img98.exs.cx/img98/6944/ssimaveaggrcorrected8va.gif

http://img32.exs.cx/img32/685/ssimsdaggrcorrected5xc.gif

http://img98.exs.cx/img98/3687/filesizeaggrcorrected8hz.gif

Edit 12-12-04: Included filesize (third) table. It is the same than on page 1; just to improve "random access" reading...

[Sir Didymus]

Originally posted by Boulder
To get the thread back on track: have you considered any PSNR measuring? The folks at XviD department seem to use it with their quant matrix tests and when testing new unstable builds.

Very good idea, and useful comment, Boulder.
I initially considered it as a possible alternative to SSIM, but now I think it could be very useful as an additional test, if this thread will survive..., since all encodings are available, so it should be not so complex to do...

My biggest concern, on the other hand, is the limitation of the test to a single cell of a single movie...

P.S. Sorry for the delay in posting some results about the test based on the Very Low Bitrate Quanti Mat of CCE. The test is complete, but I should perform the data analysis.

[Boulder]

My biggest concern, on the other hand, is the limitation of the test to a single cell of a single movie...


That shouldn't be a big problem as the test is quite extensive nevertheless and as you said, there's fast and low motion scenes involved. Besides, a test can never cover everything, there's only 24 hours in a day;)

[dragongodz]

That shouldn't be a big problem as the test is quite extensive nevertheless and as you said, there's fast and low motion scenes involved. Besides, a test can never cover everything, there's only 24 hours in a day
obviously but if you want more reliable results you do need to do larger/longer encodes(maybe not a whole movie but a reasonbably large section) and to do multiple varied sources. that would show clearer under what conditions OPV may give better results and where it wouldnt. yes a huge task that would take a very long time. :)

[wmansir]

I just went thru this thread and removed a lot of personal attacks and garbage. I have PM'ed those involved and hopefully the thread can remain productive.

If you feel a post is in violation of the rules (including personal attacks) please use the "report this post" link and refrain from responding in public. That includes asking for a mod to do something, which may just provoke the situation and have less of an effect because we may not see it.

Let's try and keep this on topic.

[Boulder]

OK, time to make a clean thread and keep it that way.


that would show clearer under what conditions OPV may give better results and where it wouldnt.


If someone is willing to do such a test, I suggest using Saving Private Ryan as the source material. It has ultra shaky camera movement, and many feel that multipass VBR excels when encoding the movie. It's also quite long so the average bitrate for the whole movie would become quite low compared to many other movies.

Probably would provide some clues regarding the pros and cons of VBR and OPV.

[dragongodz]

also a lot of anime can be difficult so picking a suitable title may show further interesting results aswell, probably different again from what SPR would show. :)

[Boulder]

I agree, anime and cartoons are a tough case and have their own distinctive ways. Just need to make sure the source isn't a crappy conversion:D

[Sir Didymus]

Originally posted by dragongodz
obviously but if you want more reliable results you do need to do larger/longer encodes(maybe not a whole movie but a reasonbably large section) and to do multiple varied sources. that would show clearer under what conditions OPV may give better results and where it wouldnt. yes a huge task that would take a very long time. :)
Totally agreed, and happy to see you underline this aspect: never forget the scientific approach. A single cell of a single movie is not sufficient for taking general conclusions.

For the moment let me just complete the first test (now that the whole thread has been "edited" I am a little bit more confident this is feasible). By the way, taking the opportunity to say where this work is assumed to go, and what conclusions are expected from my point of view:

1. The whole test is based on the usage of the SSIM tool. This is a limitation.

2. The test is based on a single cell (6795 frames) of a single movie. This is a big limitation

3. First step (completed) was to generate some tables with encodings produced "starting from" an eclcce file generated by DVD-RB. For the five bitrates of 4250, 3750, 3250, 2750, 2250 Kbit/s it has been executed 9 VBR encodings (from 1+1 to 1+9 passes) and two OPV encodings, finding empirically the Q values better matching the targer bitrates. THIS STEP HAS BEEN COMPLETED USING THE DEFAULT STANDARD QUANTI MAT.

4. Step 3 has been repeated for the bitrates of 3750, 3250, 2750, 2250, 1750, using the Very Low Bitrate Quanti Mat of CCE. (encodings completed, still pending the data analysis.

5. The same, for bitrates of 3250, 2750, 2250, 1750, 1250 should be carried out using both the Ultra Low Bitrate Quanti Mat of CCE and the Notch Matrix (here again just completed the encodings for the CCE Matrix, nothing done with the Notch). Here just want to point out that I think I will use the Notch Matrix with DC precision set to 10 bits, while the recommended is 8 bit, for making omogeneous comparisons with all other encodings performed with DC precision set to 10 bit. I hope this makes the test with Notch still useful. If someone is frequently using the Notch matrix (me not...), please give me some advice...

6. At the end, when all encoding and data analysis will be done based on SSIM, I am very interested in making the same comparisons based on PSTN, as suggestd by Boulder; it shouldn't take too much more time...


EXPECTED CONCLUSIONS (Sorry most are obvious...).

1. Change your feelings and your way of encoding only if you are very confident on what you do; and especially follow what your eyes are telling: if you actually are capable of seeing differences between encodings performed using OPV and 1+3 passes, or 1+9 passes follow what is better for yourself.

2. I think that the discriminatory power of SSIM regarding the bitrate can be accepted as a fact: the different classes of encodings at different bitrates produce different SSIM values.

3. The SSIM tool is actually not discriminating between VBR encodings: for a given bitrate it actually seems all VBR encodings produce similar results.

4. Further conclusions will follow as soon steps 4. 5. and 6. above will conclude...

SD

[TheSeeker]

This is a little OT but I was wondering about your comment on DC precision as it relates to the Notch matrice. First what exactly is the DC precision and when if ever should I think about changing it? I have always used 10 bits and I frequently use the notch matrice in low bitrate situations. You mentioned Sir Didy that it is recommended to use 8 bit dc precision with the notch matrice. Why is that?

[Boulder]

Actually there should be no relation between the matrix and DC precision, it's the avg bitrate that counts. I use 8 bits for low bitrates (in my mind ~2500kbps and below), 9 bits for mid bitrates (2500-4000kbps) and 10 bits beyond that. If the video is extra colorful and detailed, I use a higher value than I normally would. Higher DC precision value produces slightly higher filesizes (but only very slightly).

[Sir Didymus]

I was exactely asking some hints from people using Notch, since I really prefere just changing the quanti mat matrix used in the encoding, without changing the DC precision parameter too. I just quickly read into kvcd.net this 8 bits recommendation...

I suppose that DC precision at 8 bits may be helpful for improving compressibility of streams... But what about the quality ? And you know, better asking before making "improper settings"...

Cheers,
SD

Edit: by the way, here there are some tables about the test using the Very Low Bitrate Quanti Mat of CCE...

Nice w/e to everybody...

http://img100.exs.cx/img100/8182/04-AvgSSIMVeryLowBR.gif

http://img100.exs.cx/img100/7071/d7-SDSSIMVeryLowBR.gif

http://img100.exs.cx/img100/8385/2e-FilesizeSSIMVeryL.gif

[Boulder]

I don't think there is an easily perceivable difference between the different DC precisions. The 8-bit recommendation comes from the fact that the Notch matrix was developed for lowish-bitrate XVCDs, in which the max bitrate was 2500kbps.

Maybe it would be worth it doing a small test, with maybe 1+1 passes with all three different precisions and see how the values change?

For low bitrates, I'd go for either the Notch or Bach's matrix (a slight compression gain over Notch).

[jdobbs]

2. The test is based on a single cell (6795 frames) of a single movie. This is a big limitation I think you are going to find this isn't as large a limitation as you think. In my early testing with DVD-RB I was concerned about encoding cells individually because it wouldn't take full advantage of VBR due to the limited sample across which to balance the bandwidth... As it turned out that was a wrong assumption. The "magic threshold" in sample size was much smaller than a typical cell.

There's only one way to prove it of course -- but I think you'll find that even if you encode 20 complete movies the results will deviate only slightly from what you have already found.

Excellent work.

[jdobbs]

Also, I am not surprised at all at the lack of distinction between multiple passes. As I have said in the past -- once you have gone through the file once you have collected all the information necessary to decide the best allocation of bits across all frames... and you can apply that in the second pass.

[pg55555]

The problem I see with SIMM is its poor discrimination:
From SD tables you got that for 4250 kbps the SIMM is 0.981480 and fr 2250 kbps it is 0.965700, a 1.6% reduction for a compression ratio of 52%. More important, SD reported the quality degradation with the lower bitrate was noticeable

Looking at the original paper of Dr. Zhou Wang (one of the developers of SIMM) at http://www.cns.nyu.edu/~zwang/files/papers/ssim.html (link porvided by Sir Didymus in other thread)one can see this is a characteristic of SIMM: in a graphic where it is plotted the subjective quality evaluation of a series of JPEG images Mean Opinion Score or MOS vs the SIMM index, a big perceived quality degradation (MOS from 83 to 75)resulted in a SIMM index still above 0.97.

On favor of SIMM, as SD itself indicated, lower bitrates correlates with lower SIMM.

I would add that, if not completely convinced, I have already moved from my standard setting in CCE from 1+4 passes to 1+3. And probably will reduce it further to 1+2. I'm not sure if I would dare to 1+1, but if the PSNR tests show similar results, I should.

By the way, from the same paper, PSNR discrimination properties are just the opposite, small differences of subjective quality (MOS) produced big changes in the PSNR measure.

SD, thanks for a great work. I truly belive your work is enriching us all.

Pablo

[dragongodz]

once you have gone through the file once you have collected all the information necessary to decide the best allocation of bits across all frames... and you can apply that in the second pass.
so the vaf pass is (roughly)equivilent to a normal first pass as done by other encoders ? i once asked about this but from memory nobody was sure. :)
if so then yes 1+1 should give good results a lot of the time but 1+2 should never be worse and should sometimes be better. the source can play a major part in that which is why i also said multiple types of source etc should show up differences in the different encoding types. anything over 1+3 is overkill and i wouldnt worry about, would save quite a bit of test encoding time too. :)

[dragongodz]

no artificial quality test is accurate. PSNR is even worse than SSIM because you can actually get a higher PSNR with a worse video. :)

[jdobbs]

Originally posted by dragongodz
so the vaf pass is (roughly)equivilent to a normal first pass as done by other encoders ? i once asked about this but from memory nobody was sure. :)
if so then yes 1+1 should give good results a lot of the time but 1+2 should never be worse and should sometimes be better. the source can play a major part in that which is why i also said multiple types of source etc should show up differences in the different encoding types. anything over 1+3 is overkill and i wouldnt worry about, would save quite a bit of test encoding time too. :) Yes the VAF pass is nothing more than a first pass. In you want, you can even replace it with a OPV or CBR pass for creating the VAF.

[Rockas]

Yes the VAF pass is nothing more than a first pass. In you want, you can even replace it with a OPV or CBR pass for creating the VAF.

I think that if people stop for a little and think could get that conclusion easily :)
If it weren't that way what good could the basic version on CCE be?
That's why I alway make my encodes 1+1 and rarely 1+2 :)

keep it UP

[jdobbs]

Other than testing I've never done more than 1+2.

[Sir Didymus]

Originally posted by jdobbs
I think you are going to find this isn't as large a limitation as you think. In my early testing with DVD-RB I was concerned about encoding cells individually because it wouldn't take full advantage of VBR due to the limited sample across which to balance the bandwidth... As it turned out that was a wrong assumption. The "magic threshold" in sample size was much smaller than a typical cell.

There's only one way to prove it of course -- but I think you'll find that even if you encode 20 complete movies the results will deviate only slightly from what you have already found.

Excellent work.

Thanks a lot for the compliment, boss, it's really appreciated.

Your comment sound really interesting (and very relevant IMHO)...

Of course it's totally out of my very limited time to try following a systematic approach testing 20 movies or even 20 single cells (also it's out of the possibilities of my poor storage space: up to now the whole fileset for the test of just one single cell is taking more than 18GB... and I didn't even start Notch Quanti Mat Encodings...).

Not to talk about some "little allergic reaction" I am actually suffering cause of this SSIM indigestion of these last weeks... :scared:

Cheers,
SD

[Sir Didymus]

Ok. Sorry this test is taking more time than I expected. Here are some results based on the CCE Ultra Low Bitrate Quantimat...

http://img53.exs.cx/img53/812/avgssimultralowbr4bg.gif

http://img54.exs.cx/img54/4222/sdssimultralowbr1hi.gif

http://img36.exs.cx/img36/812/filesizessimultralowbr0nf.gif

I have to explain that for encodings at 1250 Kbit/s it was necessary to perform three OPV encodings, instead of the usual two. As you may see (in case encoding mistakes are absent, as I hope, since attached values have been triple checked...), at such low bitrates, the quality factor vs bitrate curve is almost flat, and the VBR bitrate hitting precision is not so high.

So the first Q factor has been choosen in order to produce an encoding sized more than the maximum of the nine VBRs, the third Q factor in order to produce an encoding sized less than the minimum, the second Q factor in order to produce an encoding sized as much as possible near to the average...

Next "episode" will be based on Notch Quantimat...

Cheers,
SD

[dragongodz]

hmm interesting...but i will refrain from comments/observations until more test are done. :)

i would suggest though that 1+6 and over is really just wasting your time and you would get much more done by just dropping them from the testing. up to you of course.

can you please include the actual sizes produced aswell ? that way we can see hopw close the OPV's are to the desired end size.

[Sir Didymus]

Hi dragongodz :)

Thanks a lot for your full understanding, both when you say you'd prefere waiting for some more tests done and for your suggestion of dropping unrelevant (1+6 and more...) VBR encodings.

Especially appreciated since I did very little subjective evaluations, apart some very general ones, already posted.

I am actually near to complete the encodings based on the Notch Quanti mat, so I think that in more or less one more week the encoding sessions (including maybe also the DDogg Bach1 matrix) will be done... At this time, just preferring spend some more time in encoding with the same nr of passes and applying the same testing procedure for all of the matrices...

What do you mean when you ask to include the actual sizes produced ? The third table I posted (in all of the tests) is actualy reporting the filesize, in Kilobytes, for each one of the encoded segments...

I included this table exactely for showing what you are asking: the Q values for the OPV encodings have been choosen in order to match as much as possible the sizes produced by the VBR encodings. For the tests using the STD matrix and the Very Low Bitrate Matrix it was possible to identify exactely two Q values matching very well (by excess and by defeat) the encoding sizes generated by the VBR encodings. With the Ultra Low Bitrate Quantimat, just at 1250 Kbyte/s, it was necessary to use three Q values in the OPV encodings (and please note these Q values are absurdely high...). The reason was explained in my previous post...

Cheers,
SD

[dragongodz]

The third table I posted (in all of the tests) is actualy reporting the filesize, in Kilobytes, for each one of the encoded segments..
woops missed that. look at the first lot of results on page 2, no size table. so i was just looking at the first table and not scrolling down. :)

once you are finished testing different matrices are you considering testing on different types of source ,with just 1 matrix, to compare what effect that has aswell ?

[Sir Didymus]

Originally posted by dragongodz
look at the first lot of results on page 2, no size table...

once you are finished testing different matrices are you considering testing on different types of source ,with just 1 matrix, to compare what effect that has aswell ?

Yeah. The filesize table for std quantimat results (page 2) is posted on page 1. It was necessary to post two times the SSIM values, for the standard CCE matrix, due to some calculation errors, and I didn't include the filesize table on page 2 since it is the same table than the one on page 1. But (as you have experienced...) this makes not linear the reading of the thread, so I will edit the post on page 2 in order to improve the readability, including the filesize table...

Well, my plans, (for the future)... are to complete the test for the Kill Bill Vol 1 and making some subjective evaluations on some (few) specific encodings, with my personal comments and comparisons...

Then I have the idea of evaluating the PSNR for the whole set of the produced encodings. As a further step I would repeat the whole test (but limiting to 4 or 5 the VBR encodings) to at least another cell of another movie. Maybe an anime (Balto ?)... Maybe even a third cell could be tested, but not more...

I know three cells it is very limited data set, but I can not spend the whole spare time of my life into this test... :D

By the way, my experience in encoding anime movie is very very limited. I am almost sure it is better to change at least quality_prec from 16 to 24 or to 32, from the default of 16. Have you further suggestions on the matter ?

Cheers,
SD

[Rockas]

@Sir Didymus
I know that words are taken by the wind and we forget them quickly and they also have another "malfunction", they aren't enough to say how much your work should be appreciated by everyone.
Anyway...
Thank you

and keep it (really) UP

[dragongodz]

(but limiting to 4 or 5 the VBR encodings)
i imagine you will be able to get the test done in half the time or twice as many in the same time doing that. :)

I would repeat the whole test
so you want to have factors such as, dif matrix + bitrate + footage type + passes = result ? which will show not only the difference pass count makes to different types of footage but also how different matrices have an effect aswell. just a little bit of work. :)

Maybe an anime (Balto ?)
Balto ? umm i think we should be able to come up with something better than that when the time comes. :D

my experience in encoding anime movie is very very limited. I am almost sure it is better to change at least quality_prec from 16 to 24 or to 32, from the default of 16. Have you further suggestions on the matter ?
well i dont own CCE so not really, i have only used the trial version in regard to working on calling its plugin(by DVDx). that may raise the question then why am i even interested i know. well short answer is because i just find it interesting, besides which i may decide to buy CCE Basic in the future(i could never afford SP). i dont own a top of the line sports car either but still find it interesting to read about their capabilities and how they perform. :)

[akupenguin]

Originally posted by dragongodz
i imagine you will be able to get the test done in half the time or twice as many in the same time doing that. :)
Why would skipping 5 out of 20 passes save half the time?

@Didymus: You are just running a total of 10 vbr passes and 10 opv passes and reporting the results after each one, right?

[dragongodz]

Why would skipping 5 out of 20 passes save half the time?
i didnt mean literally. its just a saying meaning a great deal of time would be saved. geez.

[Sir Didymus]

Originally posted by akupenguin
Why would skipping 5 out of 20 passes save half the time?

@Didymus: You are just running a total of 10 vbr passes and 10 opv passes and reporting the results after each one, right?

Hi, akupenguin, :)

Well, no...

Actually the test, for each of the given Quanti Matrices, and for each of the given bitrates, is performing OPV and VBR encodings with passes from 1+1 to 1+9. So the total number of encoding passes is 56 for each "column" of the SSIM comparison tables... [except in the very specific situation of the Ultra Low Quanti Mat at 1250 Kbytes/s, where three OPV encodings instead of two were done...].

So limiting the test up to 1+5 VBR passes will save 34 encoding passes for each column, and the time saving would be 34/56 which is much more than 50% of the time...

Hope it is clear...

Cheers,
SD

Edit. If you need some further description/information about the procedure adopted for the test, you may find some ones in this thread, (it was closed due it went "a little bit" off track..., but it still contains some useful background info...):
http://forum.doom9.org/showthread.php?s=&threadid=85331&perpage=20&pagenumber=2

[Sir Didymus]

Originally posted by dragongodz
so you want to have factors such as, dif matrix + bitrate + footage type + passes = result ? which will show not only the difference pass count makes to different types of footage but also how different matrices have an effect aswell. just a little bit of work. :)

Exactely...
I really agree it may be not simple to draw some conclusions (if any is even possible, due to the known limitations of the "objectivation" tool, which is SSIM) from such big amount of data, but I really prefere being systematic and complete as much as possible...

After all, while carrying out these tests, I have the feeling (the hope ?) I am improving my experience with the CCE encoder much more than just doing some backups of my DVDs... So the time I am spending on the subject should be hopefully a good investment, from my very personal point of view...


Balto ? umm i think we should be able to come up with something better than that when the time comes. :D

Hey dragon, what do you have against the most lovely wolf in the world ? :p Please consider Christmas is approaching... And nothing can make your heart wormer than a movie of Balto... :cool: [glasses just to mask the eyes...]

It is very popular in this timeframe here in Italy...

Anyway I am open to discuss about VALID alternatives... :)

[dragongodz]

what do you have against the most lovely wolf in the world ?
nothing. i bought my son the first one years ago and he enjoys watching it. the fact is though that it is not anime but american animation(amblin).

I am open to discuss about VALID alternatives.
yes maybe we could even do a poll as to what anime would be a typical representation. if all else fails then Balto would be better than nothing. ;)

[akupenguin]

Originally posted by Sir Didymus
Actually the test, for each of the given Quanti Matrices, and for each of the given bitrates, is performing OPV and VBR encodings with passes from 1+1 to 1+9. So the total number of encoding passes is 56 for each "column" of the SSIM comparison tables...

So limiting the test up to 1+5 VBR passes will save 34 encoding passes for each column, and the time saving would be 34/56 which is much more than 50% of the time...

Disclaimer: I have never used CCE, and if I ever decide to encode something to MPEG2, I will be using ffmpeg. But my high-level discussion of ratecontrol should be independent of the codec.

Here's my proposed encoding process, assuming I don't grossly misunderstand the operation of CCE:
1st pass just performs compexity analysis, and doesn't output any video.
2nd pass (1+1) takes that analysis, performs bit/quantizer allocation based on it, and encodes. It also performs another round of complexity analysis. You keep this output video, and run your SSIM or whatever on it.
3rd pass (1+2) takes the analysis from 2nd pass to refine ratecontrol, encodes to another output video, and runs another round of analysis. You keep this output video, and run your SSIM or whatever on it.
...
10th pass (1+9) does the same.

Total: 10 passes, 9 video files. (plus the OPV encodes)


I'm assuming you don't have to tell CCE in advance how many passes you're going to give it. I could imagine a very smart codec might decide what options to evaluate based on how many passes it's going to run, and use all previous stats rather than just the last pass. But if CCE did that, then you wouldn't be seeing an oscillation in quality.

If I'm right, this will reduce your total time from 56 passes per setting to 21, while producing outputs what are bitwise identical to what you're getting now.

[Sir Didymus]

Originally posted by akupenguin
Disclaimer: I have never used CCE, and if I ever decide to encode something to MPEG2, I will be using ffmpeg. But my high-level discussion of ratecontrol should be independent of the codec.

...

Total: 10 passes, 9 video files. (plus the OPV encodes)

...

I'm assuming you don't have to tell CCE in advance how many passes you're going to give it.

...



I now understand what you mean...
I have to tell you assumption is wrong: actually you have to specify to CCE how many VBR encoding passes you want to perform in advance.

If you want to produce (and compare) two CCE encodings, for example based on 1+2 and 1+3 VBR passes, you should start two different CCE sessions, performing in total 7 encoding passes. And the second session should restart from the beginning. It would be very nice and useful to follow your idea, but it is simply not feasible...

Cheers,
SD

[akupenguin]

OK, now I'm wondering whether it actually makes use of the knowledge, or whether the restriction is just a misfeature. I mean, I could easily understand that that they might do it that way in the name of user-friendliness, without any technical justification.

And people wonder why I consider open source to be one of the most important features of a codec...

[robot1]

Originally posted by Sir Didymus
If you want to produce (and compare) two CCE encodings, for example based on 1+2 and 1+3 VBR passes, you should start two different CCE sessions, performing in total 7 encoding passes. And the second session should restart from the beginning. It would be very nice and useful to follow your idea, but it is simply not feasible...

Cheers,
SD Probably you could act as suggested, doing always one pass of VBR pass, using the last .vaf created.
If you've done 1+1, the result will be just like the output of 2+1.
(You could do a test to check with a binary compare).
With every added pass, you get a result for your test.

[Sir Didymus]

Hi robot1, :)

Could you please point me out where, in the CCE User Manual, the behaviour you suggest guarantees to obtain the result you mention ?

I heard the same other times (never personally verified, I have to say...), but in order to be totally sure the data reported in the test are actually the same as the ones generated by DVD-RB, I can not do anything else than exactely replicate what is doing DVD-RB in its encoding sessions.

I can not assume undocumented features (even though commonly accepted as valid) as facts. Even if the binary compare shows what you say for one single (or even for many) encoding(s), nothing guarantees this holds for all encoding passes...

Of course, ready to change my position in light of further news/info on the matter...

All the best,
SD

[akupenguin]

While I can't find any official info reguarding the workings of multipass, the user's guide does guarantee that you can use the same 1st pass VAF for all of the encodes. See http://www.cinemacraft.com/files/doc/ccs_270e.pdf page 35.

[Trahald]

doing a 1+3 (vaf+3) gives output that is is bitwise equivelant as doing 1+1 three times (using the same vaf file the 2nd and 3rd time.. it works for any number of passes. (did some quick testing to be sure)

[jdobbs]

Robot1 and Trahold are correct. I can assure you that running a single VBR pass four times is the same thing as running a four pass VBR encode. I wouldn't call it three (1+1) encodes, though, as each time you run it it will only do a single pass (because the VAF exists and is used from the previous pass. CCE will not do a .VAF pass if the .VAF already exists.

[Trahald]

yeah.. wasnt the best explanation. its more a 1+1 +1 +1

[Sir Didymus]

Hmmm, well, good to know...
Especially in light of the big time saving it implies...

I have to say that the User Manual is of no help: it clearly describes what to do for re using the VAF file, but it never states what you get...

I will do a simple further check (mostly for my own confidence), binary comparing the results, already available, of the Notch encoding...

If everything goes right, gonna changing the encoding procedure in order to perform the test with the Bach1 Quanti Mat (and the following...) using the described approach.

MANY THANKS for the fundamental contribution to akupenguin, robot1, Trahald and Jdobbs...

If everything is ok, in one of my next post I will report the number of encoding passes up to now spent unnecessarily... :scared:

Cheers,
SD

[Sir Didymus]

Well, as indicated, (many thanks guys!) the encoding following the scheme:

1+1 +1 +1 ...

works fine, and the results of the stepwise encodings are totally equivalent to perform the multipass as a whole. Tested on a couple of "columns" of the Notch Quanti Mat test... I have to admit my ignorance about this feature (let me just underlne again the CCE User Manual is not providing any enlightening on the subject...)...

Anyway, this will reduce the whole encoding time for the test at someting like the 20% of the previous...

Of course the way I am looking at this nice feature is not the thinking at the (huge :scared: ) amount of time wasted up to now... but considering that I am now working like having a PC running at 15 GHz !!! WOW!!!

Many thanks again to akupenguin and robot1 for the great suggestion...

Taking the opportunity for posting the SSIM results based on the KVCD Notch Quantimat Encodings:

http://img94.exs.cx/img94/9703/avgssimkvcdnotchmatr4oi.gif

http://img109.exs.cx/img109/7022/sdssimkvcdnotchmatr9ca.gif

http://img110.exs.cx/img110/1037/filesizekvcdnotchmatr5qk.gif

Cheers,
SD

[tylo]

Originally posted by jdobbs
Yes the VAF pass is nothing more than a first pass. In you want, you can even replace it with a OPV or CBR pass for creating the VAF.
Chapter 3.3.4 of the CCE manual:

Unless above parameters are changed, you don’t need to recreate
video information file. If the setting of the bitrate is a major change,
however, it is better to recreate the video information file because a
better encoding result can be obtained with less number of passes.
When an average bitrate is set to twice or more or half or less than
that previously specified, recreating the video information file is recommended.

This indicates that the .vaf file is somewhat different depending on avg. bitrate or Q (if OPV) chosen when creating it...

[jdobbs]

Absolutely.

If you do an OPV or CBR pass the .VAF is going to be based upon the Q or bitrate you selected. If you set it more than twice or half the target bitrate you've really over/underestimated...

BTW, haven't heard from you in a while, Tylo (the OPV master), how's it going?

[tylo]

I'm doing great, thank you! A little quite here yes, but I did make a small update on my little precious (d2sroba) a few days ago! :D

Btw: Interesting thread. Nice to see you people are keeping up the good work.

[Sir Didymus]

Just learning from the masters...
:)
By the way, your previous post and the one of Jdobbs helped me to understand better what was Boulder referring to, when he spoke, few pages above, about the possibility of performing one OPV pass for producing a VAF file, followed by a VBR pass, in order to have a very precise matching with a desired target avg bitrate.

Do you think (while I am doing 100 encodings, one more costs just quite little) it is worthwile to include this "mixed mode" encoding in the SSIM comparison tables ?

Cheers,
SD

[Boulder]

There is a slight difference when the vaf file is created by a OPV encode or a simple first pass of a multipass encode. If the Q value is predicted well enough, which it usually is, the average bitrate of the OPV encode is closer to the final target than the one that comes out with the first pass.

To me it looks like that the standard vaf file creation pass does almost CBR, and it often ends going much lower than the target. OPV is more like constant quantizer encoding. If there's anything to refer to, I'd like to point out that XviD does the first pass with a constant quantizer and DivX5 uses constant bitrate. Many people feel that DivX5 needs the third pass to come out with matching quality. Of course, these assumptions need to be taken with a grain of salt, but it's some food for thought anyway.

In short, it might be worth checking. However, it's your own time you're using so the decision is of course yours:)

[TheSeeker]

I guess im kind of lost as to what we are exploring here. Are you guys saying that using OPV to generate the vaf file might give a more accurate roadmap to the video stream than the first vaf pass of a multipass encode?

[jdobbs]

I really don't think so -- but I wouldn't be my life on it. The second pass is going to readjust upward (on high demand) and downward (on low demand) to approximately the same level either way. So performing a CBR first pass at the target average bitrate would pretty much do the same thing.

[J-Wo]

hey guys, I just stumbled upon this thread which was so timely because I was jsut doing my own informal comparisions b/t OPV vs multipass (1+1), and notch matrix vs standard. For years now I've been using the notch matrix and OPV with great success, but it is only now with Rb that I've tried others. Oh, and for those that say the disadvantage of OPV is poor file prediction, you should give Rb-Opt a try! It performs the prediction for you flawlessly and in a very short time too.

Based on the movies I've done, I find that notch and OPV makes natural film grain look more blocky, and very dark scenes lack detail and manifests itself as dancing dark grey/blue/purple squares where shadows should be. So for now I'm sticking with std matrix and 1+1 VBR. I'd love to go over a bit more of your results, sir didymus.

One question tho: for your notch encodes, did you also change DC precision to 8, Gop length to 15, and Quantizer characteristics to 25? These settings were recommended for DVD-Rb use in a thread over at kvcd.net and can be set using Rb-Opt.

[Sir Didymus]

Originally posted by J-Wo
One question tho: for your notch encodes, did you also change DC precision to 8, Gop length to 15, and Quantizer characteristics to 25? These settings were recommended for DVD-Rb use in a thread over at kvcd.net and can be set using Rb-Opt.

In short: no, no, no...

I mean, I know the recommended setting at kvcd.net is not just limited to the usage of the Notch Matrix in itself, but in the performed test, for obtaining results comparable with the other encodings (where just the usage of different matrices is a parameter) it has been necessary not to change any other parameter that may affect the compressibility.

In other words, thanks to the excellent application of robot1, it is very easy (and maybe convenient) to change a lot of CCE parameters. For example, it is very clear to me that extending the GOP length to 15 frames lead to an improvement of the compressibility.

But this equally holds for all of the matrices..., so since it is not a standard setting of DVD-RB, just preferred not to modify more than a single parameter (the matrix) for each test...

I think the focus of the test is just to perform a relative comparison among OPV and VBR multipass encodings, trying to understend if it is really useful to perform many VBR passes, how many, and why.

Even with such a "little" objective, possible answers will be necessarily subjective and limited. If you want to find the best possible CCE settings I think you will get no answers at all in this thread... ;)

Cheers,
SD

[Sir Didymus]

Originally posted by TheSeeker
I guess im kind of lost as to what we are exploring here. Are you guys saying that using OPV to generate the vaf file might give a more accurate roadmap to the video stream than the first vaf pass of a multipass encode?

Hi, friend, :)

That was my question !!!

Just to clarify: Due to my deep ignorance on the subject, (and to the dammit writers of the CCE User Manual :devil: ), I have been a little bit surprised from the understanding that the nth step of a VBR multipass encoding is just obtainable with a single VBR step, adopting the VAF file produced at the nth-1 step.

This means that the VAF file in itself is extremely important in the multipass encoding, since it basically contains all information about the previous n-1 steps.

As a consequence the way of producing this VAF file is fundamental...
...And it may be the case to add just another row in the tables...

My feeling is totally matching with what stated lastly by Jdobbs, and I am quite convinced that an extraordinary change in the SSIM values will be hardly obtainable, but you know, I am curious by nature, and this "discovery" :D about the VAF relevance is a nice stimulus to do some more encodings...

SD

[Sir Didymus]

Originally posted by Boulder
In short, it might be worth checking. However, it's your own time you're using so the decision is of course yours:)

Hey Boulder...

LOL...
That is exactely the deeply motivating and encouraging type of answer I was looking for... :D

[Boulder]

Could you feel the weight of the sudden additional responsibility?

:D

[pg55555]

Sir Didymus
That is exactely the deeply motivating and encouraging type of answer I was looking for... How about this:

Thanks to your testing I have reduced my standard encoding setting from 1+4 to 1+2, roughly reducing my encoding times from 5 hours to 3 hours, a 2 hours saving. This actually was not so important for myself, as I ussually do the encoding overnight (and I try to sleep more than 5 hours).

But it means two less hours for encoding in which my Pentium 4 will not be running at almost 100%, consuming 90W.

So, you have helped me (and possible many others) to "Save the Earth"!!! (and some pesos from my power bill)

Keep the good work

Pablo

[tylo]

I am not convinced that CCE manages to adjust the bitrate to be perfectly distributed in the second pass. In theory it should, but I then wonder what is going on in the subsequent passes...

However, I think it is possible to test whether an OPV created vaf is "better" than a CBR/VBR one by:
1. Create an OPV vaf with some chosen Q.
2. Add an extra VBR pass to it with exactly the bitrate that the OPV pass produced.
3. Create a new VBR + 1pass file with bitrate as above
4. Compare the two videos produced

[Sir Didymus]

Hi tylo,

Just to undestand what you mean:

a. Which way do you calculate the exact bitrate of your point 1. ?
b. due to some unavoidable rounding errors it seems to me it will be very difficult -- or better almost impossible -- to feed CCE with exactely this value... so is your argument, let me say, theoric or are you really suggesting an experiment to carry out ?
c. When you say "compare", in your point 4., do you mean binary compare, evaluate SSIM differences, or through subjective comparison ?

Cheers,
SD

[tylo]

Hi SD, I meant that you can carry out the test if you want.

a)
Select a realistic Q about 25 ~ 30, encode.
bitrate (kbps) = 8 * mpv_filesize_bytes / number_of_frames * framerate / 1000.

b) You can compute exactly the bitrate. Num of frames (shown in CCE), filesize in bytes (Properties menu in explorer), and framerate is 25 (PAL) or 29.976 (NTSC)...

c) I meant both evaluate SSIM differences and subjective comparison, although the latter will be difficult, I can assure you...

Cheers.

/Add: you may try a higher Q - it will likely give bigger differences (if any).

/Add 2: If you do the tests (I for one will be grateful), can you supply the SSIM values after the OPV pass too. Some have claimed that a following VBR pass, which not resize, may actually degrade the quality, or at least not improve it, because it is optimal already...

[Sir Didymus]

Ok. with the Bach1 matrix (see the enclosed results), the encodings and SSIM calculations for this cell of Kill Bill Vol.1 are complete...

At the usual results, I added the SSIM evaluations for including a single VBR step using the VAF file after a OPV encoding. Since the tables becomes even more extended, in a minimum attempt to reduce their visual size and to improve the general readability, the results are posted as thumbnail. I will edit the previously posted tables in the same way in the next days...

As you may see this matrix shows some interesting characteristics from the point of view of improving the encoding compression. I will do some subjective evaluations in the next days (of the next year, most probably... :D ) in order to check what are the drawbacks, if any...

Cheers,
SD

http://img65.exs.cx/img65/3913/avgssimddoggbach1matr6ag.th.gif (http://img65.exs.cx/my.php?loc=img65&image=avgssimddoggbach1matr6ag.gif)

http://img65.exs.cx/img65/4214/sdssimddoggbach1matr7vs.th.gif (http://img65.exs.cx/my.php?loc=img65&image=sdssimddoggbach1matr7vs.gif)

http://img76.exs.cx/img76/7172/filesizeddoggbach1matr6zs.th.gif (http://img76.exs.cx/my.php?loc=img76&image=filesizeddoggbach1matr6zs.gif)

[Sir Didymus]

Well, I carried out some test...
Hope it is useful in some way on this fascinating comparison subject...

First of all (just want to state it again, especially at the benefit of all occasional readers), I will never say enough that this SSIM index is a very nice and appropriate tool to numerically evaluate ïmage similarities, having many strong and good properties, but it shouldn't be taken as a metric for the quality assessment of your encodings. Quality is subjective, and, as already stated many times by people much more experienced ans skilled than myself, your eyes are the only valid instrument to use for measuring the quality...

Said this, please find attached another table... Well, that's right, it looks like recently I was not able to post anything else than tables... so I am a now quite seriously worried my way of communicating is becoming, a would say, a little bit too schematic :) ...

That's why, in order to balance the numbers, I'd like to give some explainations on how to read it (not to the masters, of course, but again, think occasional readers may find useful to have some hints on the subject).

The starting stream is composed by one cell (6797 frames) extracted from the movie Kill Bill Vol1 - R2 - PAL - Italian Edition; it is the famous scene where Vivica Fox is attacked in her house by Uma Thurman... The cell is composed by high dynamics parts (very high I would say)... together with some other static parts...

The stream has been encoded adopting tkeaked versions of an ecl file generated with DVD-RB...

In the table there are three blocks, since the test suggested by Tylo has been repeated three times, in order to check different Q values (Q28, Q34, Q42).

Initially the stream has been encoded using a single OPV pass, and the relative SSIM index has been calculated. The completion of this step allows to fill the first four columns of the table, i.e. filesize of the encoded m2v stream, bitrate, using the formula pointed out by Tylo himself, and SSIM values (average and std. deviation, calculated on the first 6795 frames of the stream).

As you may see the obtained bitrate, which have a precise value, but is not an integer, generates two bitrate values (ceil and floor rounding of the bitrate), so two 1 pass VBR encodings have been produced after this OPV step. For example, in the first row of the table the bitrate obtained after the OPV step is 4286.643 Kbit/s, and a VBR step was performed with the average bitrate values of 4286 and 4287 fed to CCE.

The completion of this step allows to fill the last columns of the table, including the filesize of the m2v stream at the end of this second step, its bitrate, and the relative SSIM values. As it is possible to observe, the obtained final bitrate is in some ways not optimal, so two more bitrate values (in the example 4291 and 4292 kbit/s) have been empirically identified in order to produce, if applied as avg bitrate of a single VBR pass after the OPV, a bitrate better matching the target. In the example, the given bitrates, fed to CCE, generate the two bitrate values of 4282.477 and 4292.003 for hitting the target of 4286.643.

The other rows in each test have been produced, using a normal 1+1 pass VBR encoding, for tabulate all of the meaningful bitrates involved in the experiment. In the example, all of the bitrate values between 4286 and 4292 have been tested.

The table contains three Q values, just for trying to replicate the test on some different bitrate "domains".

Conclusions:

- I am really embarassed this very simple test proposed by Tylo has assumed this huge dimension: the whole fileset for this "little experiment" alone is 3.53 GB... I was surprised on my own, while doing the test of its growing... And we'd better considering it is still improper to draw any general conclusion: just a single cell was used, of just a single movie, with just three Q values... Almost nothing from the statistical point of view...

- didn't even try to perform some subjective analysis: when average SSIM differences are (like in this experiment) in the order of few ppm, it's absolutely impossible to observe any relevant difference. This is my experience. So please do not to ask to my poor eyes to perform a meaningless effort...

- by looking at the SSIM values, I would say that there is some argument in favour of people claiming that a OPV pass, followed by VBR one, which is not resizing, may actually degrade the quality, or at least not improve it, because it is already optimal...

All the best,
SD

P.S. maybe I am a bit early, but let me take the opportunity for wishing all the friends and nice people of this beautiful forum a Merry Christmas and an happy new year...

http://img67.exs.cx/img67/2595/constqtest1kp.th.gif (http://img67.exs.cx/my.php?loc=img67&image=constqtest1kp.gif)

[bobwillis]

Re: VBR and OPV.

Similar discussions have occured in the past; here are my previous thoughts (bottom of page).

http://forum.doom9.org/showthread.php?s=&threadid=73959&highlight=vbr+bias

For me, it comes down to whether or not you prefer a constant quantization encode. The visual differences become more apparent as the Q level increases.

Regards,
Bob

[Sir Didymus]

@bobwillis.

Hey, thanks for the reference!!!

Just ended the reading of the thread, and...

Well, it seems incredible to me, I am reading almost daily the posts in this forum, since the beginning, but I have to admit I completely missed that very intersting and helpful discussion... :rolleyes:

All the best,
SD

Edit: Anyway, I know the topic has been widely discussed and investigated in a much more strong and complete manner elsewere. I just started to carry our some tests on my own, since I had the perception that too much discussions and opinions were proposed, while not so many documented results were posted to balance them...

[jdobbs]

Hey, Sir Didymus...

Go shopping, get some rest, drink some eggnog... Merry Christmas.

[Sir Didymus]

LOL, :D

Quoted, quoted...
Really need to do all you suggest...

SD