OPV & Multipass Experiments

[Sir Didymus]

Hi

I have one bad new, and one good new on the topic...

I am sorry to inform that I discovered one very little mistake in the data I posted.

1. Using VirtualDub 1.6.1, it happens that the dubbing ends to the frame before the last in almost every play I did for computing the SSIM values. This is introducing a very little systematic error in the LAST TWO FRAMES of the SSIM xls files. The recovery is very easy (and I am almost sure that nothing will change in the posted information) and consists in using the first 6795 of the 6797 SSIM values, but this implies the two tables of SSIM average and standard deviation should be edited. I will try to do the job asap.

2. Full encodings using a low bitrate matrix for doing the same experiment are complete. Hopefully this evening I will be able to post similar tables for the low bitrate quanti mat...

Cheers,
SD

[Boulder]

To get the thread back on track: have you considered any PSNR measuring? The folks at XviD department seem to use it with their quant matrix tests and when testing new unstable builds.

[Sir Didymus]

Hi...

Please find in this post the corrected tables reporting Average and Standard Deviation for the SSIM values produced in the test...

I have to say and to admit (as you may see) that the errors present in the original tables were two and very nasty...

So I decided not to remove the previous (buggy) tables, and to post the new tables here, adding some comments for explaining these errors...

Error 1:
In the calculation of the average SSIM values, the last two frames of the cell were included in the test. These frames were sometimes totally wrong, due to some VirtuaDub troubles... Overall effect wasn't to introduce a systematic error (as I hoped), but a seriuos mistake capable of changing some conclusions regarding the OPV behaviour at 4250 Kbit/s.

The shortcoming has been simply not to use these two last frames in any of the calculation of the SSIM average values.

Error 2:
Big mistake (and this was really my fault): I didn't properly understood that the overall average value (generated by the SSIM plugin and reported in my data) is not the simple aritmethical average of the frame-by-frame SSIM comparison, but it is a weighted average (where the weights are included in the second column of the csv files).
This is very interesting (and useful) since it allows to give for example no relevance to black frames, and to give more weight to frames judged relevant by the SSIM evaluation algorithm... Anyway, in the previous posted tables, the Std. Deviation was calculated arithmetically, without considering the weights...
So the Average tables were just a little bit wrong, and with a little bit of Excel work it was not difficult to recalculate the right values, while the Std Deviation table (which is a fundamental information to consider in the test) was definitely wrong... And its recalculation took to me some more time than I expected...

Hope the attached information, and the whole work it took to produce it, is worthwile...

Cheers,
SD







Edit 12-12-04: Included filesize (third) table. It is the same than on page 1; just to improve "random access" reading...

[Sir Didymus]

Originally posted by Boulder
To get the thread back on track: have you considered any PSNR measuring? The folks at XviD department seem to use it with their quant matrix tests and when testing new unstable builds.

Very good idea, and useful comment, Boulder.
I initially considered it as a possible alternative to SSIM, but now I think it could be very useful as an additional test, if this thread will survive..., since all encodings are available, so it should be not so complex to do...

My biggest concern, on the other hand, is the limitation of the test to a single cell of a single movie...

P.S. Sorry for the delay in posting some results about the test based on the Very Low Bitrate Quanti Mat of CCE. The test is complete, but I should perform the data analysis.

[Boulder]

My biggest concern, on the other hand, is the limitation of the test to a single cell of a single movie...

That shouldn't be a big problem as the test is quite extensive nevertheless and as you said, there's fast and low motion scenes involved. Besides, a test can never cover everything, there's only 24 hours in a day

[dragongodz]

That shouldn't be a big problem as the test is quite extensive nevertheless and as you said, there's fast and low motion scenes involved. Besides, a test can never cover everything, there's only 24 hours in a day

obviously but if you want more reliable results you do need to do larger/longer encodes(maybe not a whole movie but a reasonbably large section) and to do multiple varied sources. that would show clearer under what conditions OPV may give better results and where it wouldnt. yes a huge task that would take a very long time.

[wmansir]

I just went thru this thread and removed a lot of personal attacks and garbage. I have PM'ed those involved and hopefully the thread can remain productive.

If you feel a post is in violation of the rules (including personal attacks) please use the "report this post" link and refrain from responding in public. That includes asking for a mod to do something, which may just provoke the situation and have less of an effect because we may not see it.

Let's try and keep this on topic.

[Boulder]

OK, time to make a clean thread and keep it that way.

that would show clearer under what conditions OPV may give better results and where it wouldnt.

If someone is willing to do such a test, I suggest using Saving Private Ryan as the source material. It has ultra shaky camera movement, and many feel that multipass VBR excels when encoding the movie. It's also quite long so the average bitrate for the whole movie would become quite low compared to many other movies.

Probably would provide some clues regarding the pros and cons of VBR and OPV.

[dragongodz]

also a lot of anime can be difficult so picking a suitable title may show further interesting results aswell, probably different again from what SPR would show.

[Boulder]

I agree, anime and cartoons are a tough case and have their own distinctive ways. Just need to make sure the source isn't a crappy conversion

[Sir Didymus]

Originally posted by dragongodz
obviously but if you want more reliable results you do need to do larger/longer encodes(maybe not a whole movie but a reasonbably large section) and to do multiple varied sources. that would show clearer under what conditions OPV may give better results and where it wouldnt. yes a huge task that would take a very long time.

Totally agreed, and happy to see you underline this aspect: never forget the scientific approach. A single cell of a single movie is not sufficient for taking general conclusions.

For the moment let me just complete the first test (now that the whole thread has been "edited" I am a little bit more confident this is feasible). By the way, taking the opportunity to say where this work is assumed to go, and what conclusions are expected from my point of view:

1. The whole test is based on the usage of the SSIM tool. This is a limitation.

2. The test is based on a single cell (6795 frames) of a single movie. This is a big limitation

3. First step (completed) was to generate some tables with encodings produced "starting from" an eclcce file generated by DVD-RB. For the five bitrates of 4250, 3750, 3250, 2750, 2250 Kbit/s it has been executed 9 VBR encodings (from 1+1 to 1+9 passes) and two OPV encodings, finding empirically the Q values better matching the targer bitrates. THIS STEP HAS BEEN COMPLETED USING THE DEFAULT STANDARD QUANTI MAT.

4. Step 3 has been repeated for the bitrates of 3750, 3250, 2750, 2250, 1750, using the Very Low Bitrate Quanti Mat of CCE. (encodings completed, still pending the data analysis.

5. The same, for bitrates of 3250, 2750, 2250, 1750, 1250 should be carried out using both the Ultra Low Bitrate Quanti Mat of CCE and the Notch Matrix (here again just completed the encodings for the CCE Matrix, nothing done with the Notch). Here just want to point out that I think I will use the Notch Matrix with DC precision set to 10 bits, while the recommended is 8 bit, for making omogeneous comparisons with all other encodings performed with DC precision set to 10 bit. I hope this makes the test with Notch still useful. If someone is frequently using the Notch matrix (me not...), please give me some advice...

6. At the end, when all encoding and data analysis will be done based on SSIM, I am very interested in making the same comparisons based on PSTN, as suggestd by Boulder; it shouldn't take too much more time...


EXPECTED CONCLUSIONS (Sorry most are obvious...).

1. Change your feelings and your way of encoding only if you are very confident on what you do; and especially follow what your eyes are telling: if you actually are capable of seeing differences between encodings performed using OPV and 1+3 passes, or 1+9 passes follow what is better for yourself.

2. I think that the discriminatory power of SSIM regarding the bitrate can be accepted as a fact: the different classes of encodings at different bitrates produce different SSIM values.

3. The SSIM tool is actually not discriminating between VBR encodings: for a given bitrate it actually seems all VBR encodings produce similar results.

4. Further conclusions will follow as soon steps 4. 5. and 6. above will conclude...

SD

[TheSeeker]

This is a little OT but I was wondering about your comment on DC precision as it relates to the Notch matrice. First what exactly is the DC precision and when if ever should I think about changing it? I have always used 10 bits and I frequently use the notch matrice in low bitrate situations. You mentioned Sir Didy that it is recommended to use 8 bit dc precision with the notch matrice. Why is that?

[Boulder]

Actually there should be no relation between the matrix and DC precision, it's the avg bitrate that counts. I use 8 bits for low bitrates (in my mind ~2500kbps and below), 9 bits for mid bitrates (2500-4000kbps) and 10 bits beyond that. If the video is extra colorful and detailed, I use a higher value than I normally would. Higher DC precision value produces slightly higher filesizes (but only very slightly).

[Sir Didymus]

I was exactely asking some hints from people using Notch, since I really prefere just changing the quanti mat matrix used in the encoding, without changing the DC precision parameter too. I just quickly read into kvcd.net this 8 bits recommendation...

I suppose that DC precision at 8 bits may be helpful for improving compressibility of streams... But what about the quality ? And you know, better asking before making "improper settings"...

Cheers,
SD

Edit: by the way, here there are some tables about the test using the Very Low Bitrate Quanti Mat of CCE...

Nice w/e to everybody...

[Boulder]

I don't think there is an easily perceivable difference between the different DC precisions. The 8-bit recommendation comes from the fact that the Notch matrix was developed for lowish-bitrate XVCDs, in which the max bitrate was 2500kbps.

Maybe it would be worth it doing a small test, with maybe 1+1 passes with all three different precisions and see how the values change?

For low bitrates, I'd go for either the Notch or Bach's matrix (a slight compression gain over Notch).

[jdobbs]

2. The test is based on a single cell (6795 frames) of a single movie. This is a big limitation

I think you are going to find this isn't as large a limitation as you think. In my early testing with DVD-RB I was concerned about encoding cells individually because it wouldn't take full advantage of VBR due to the limited sample across which to balance the bandwidth... As it turned out that was a wrong assumption. The "magic threshold" in sample size was much smaller than a typical cell.

There's only one way to prove it of course -- but I think you'll find that even if you encode 20 complete movies the results will deviate only slightly from what you have already found.

Excellent work.

[jdobbs]

Also, I am not surprised at all at the lack of distinction between multiple passes. As I have said in the past -- once you have gone through the file once you have collected all the information necessary to decide the best allocation of bits across all frames... and you can apply that in the second pass.

[pg55555]

The problem I see with SIMM is its poor discrimination:
From SD tables you got that for 4250 kbps the SIMM is 0.981480 and fr 2250 kbps it is 0.965700, a 1.6% reduction for a compression ratio of 52%. More important, SD reported the quality degradation with the lower bitrate was noticeable

Looking at the original paper of Dr. Zhou Wang (one of the developers of SIMM) at http://www.cns.nyu.edu/~zwang/files/papers/ssim.html (link porvided by Sir Didymus in other thread)one can see this is a characteristic of SIMM: in a graphic where it is plotted the subjective quality evaluation of a series of JPEG images Mean Opinion Score or MOS vs the SIMM index, a big perceived quality degradation (MOS from 83 to 75)resulted in a SIMM index still above 0.97.

On favor of SIMM, as SD itself indicated, lower bitrates correlates with lower SIMM.

I would add that, if not completely convinced, I have already moved from my standard setting in CCE from 1+4 passes to 1+3. And probably will reduce it further to 1+2. I'm not sure if I would dare to 1+1, but if the PSNR tests show similar results, I should.

By the way, from the same paper, PSNR discrimination properties are just the opposite, small differences of subjective quality (MOS) produced big changes in the PSNR measure.

SD, thanks for a great work. I truly belive your work is enriching us all.

Pablo

[dragongodz]

once you have gone through the file once you have collected all the information necessary to decide the best allocation of bits across all frames... and you can apply that in the second pass.

so the vaf pass is (roughly)equivilent to a normal first pass as done by other encoders ? i once asked about this but from memory nobody was sure.
if so then yes 1+1 should give good results a lot of the time but 1+2 should never be worse and should sometimes be better. the source can play a major part in that which is why i also said multiple types of source etc should show up differences in the different encoding types. anything over 1+3 is overkill and i wouldnt worry about, would save quite a bit of test encoding time too.

[dragongodz]

no artificial quality test is accurate. PSNR is even worse than SSIM because you can actually get a higher PSNR with a worse video.