OPV & Multipass Experiments

[Sir Didymus]

Let me post again some links about the first test performed:

xls sheet & original images:
http://wpop14.libero.it/cgi-bin/vlink.cgi?Link=http%3A//webmail.libero.it/r.php%3Fd%3Dlibero.it%26wr%3D3zqcxucf%26ws%3D3zqcxucf%26e%3Dlibero.it%26c%3Dn9LxHgvnSfiQE0J593lQOkFLcLcYwiY16165

original m2v:
http://wpop14.libero.it/cgi-bin/vlink.cgi?Link=http%3A//webmail.libero.it/r.php%3Fd%3Dlibero.it%26wr%3D3zqcxucf%26ws%3D3zqcxucf%26e%3Dlibero.it%26c%3D6XF6oayyjkR3El935ac0ULDST0yfBbvy6287

test 1+1 mv2:
http://wpop14.libero.it/cgi-bin/vlink.cgi?Link=http%3A//webmail.libero.it/r.php%3Fd%3Dlibero.it%26wr%3D3zqcxucf%26ws%3D3zqcxucf%26e%3Dlibero.it%26c%3DLcmtwHau1jksPjtj1bgR3pRNoov4XyEJ6221

All links are working. Just tested. They are supposed to be active until five more days. [Deadline 04.12.2004 03:30:48 PM CET]

Currently performing some encodings at lower bitrates than 4250, and with different quanti matrices. I will post the results (that are personal evaluations, [if it will be allowed to me]) in the next, few, days.

Indeed it seems to me very important to add the following:

1. No evident mistakes seems to me are present in what I proposed last week on the subject. I have already demonstrated to be a correct person and to be friendly and open minded towards everybody: if someone demonstrates my tests are wrong I am ready to say sorry and to change my opinions; if I discover some of my tests are wrong it is my interest to inform asap everybody. This is the only way I know (that is to apply the scientific method) to ensure the reliability and the replicability of experiments.

2. Troubles on uploading links are due to the service I am using (that is clearly not very stable), and the five days timeout is a limit of the service, that isn't under my control. Since I don't have an ftp site available to host my files, I am using this way to share big chunks of information.

3. I do not accept ultimatum deadlines on my testings from anybody. I am doing the experiments on my spare time and (I hope) I am free to use the time of my weekends with my family, and without being pressed or available on line, if I decide so.

4. Everybody, even the most stupid person on the earth, starting from the single source m2v, and following the detailed procedure I described, is supposed to be able to replicate the test, without any need of the other files. Using the posted ecl, it is possible and easy to generate all the needed m2v files (from and including 1+1, to 1+9) and to visually compare them.

5. Since I was the target of some very heavy and full of poison personal comments, I want to thank very much Jdobbs for closing the thread on the multipass evaluation. Well done and really appreciated.

[mod edit]

SD

[Rockas]

@Sir Didymus
Keep up the good work... sorry... the excellent work :)
... by the way... did you take a look to the help files?
Your opinion will be welcome

keep it UP

[Sir Didymus]

Thanks, Rockas :)

Even if it is a little bit OT, your post is very appreciated. Really.
While we are OT: of course I did..., And it seems to me very, very good...
Asap I will give to you some more structured comments, in the Help File Thread...

[TheSeeker]

@Sir Didymus

Thank you very much for you hard work on these tests. I know I personally am very anxious to see what you have found. ANd what you will find in the future of your testing. [mod edit] Anyways keep up the good work. Now to check out thoes links.....

[Sir Didymus]

Hi TheSeeker!

Thanks a lot for your comments. Really appreciated!!!

Just let me say:
1. I didn't find or discover anything relevant. That's sure. I mean, this thread is about questionable opinions, that may be supported by SSIM tests and analysis, but aren't facts, you know quality is subjective...
2. So, no reason to be anxious... Even because tests with std matrices are now complete, and in the next few (one or two) days I will start posting something...
3. If you can not wait 2 days and you want to exercise yourself a little bit with Italian, you may want to start getting a look here:
http://forum.doom9.it/viewtopic.php?p=58662#58662

Cheers
SD

[TheSeeker]

Hey Didy,

I noticed from the results in that link that OPV seems to be providing a higher SSIM in most cases. What would explain this and does this mean that OPV may produce better quality than multipass. Other than that though it seems like 1+3 passes seems to be the magic number, at least for SSIM accuracy.

[Sir Didymus]

Sssss!!!

Do not say such things at such loud voice !!! :cool:

Edit : your Italian teacher did a damn quick job...
Edit2: next time I will post before into my favourite Japanise forum...

[TheSeeker]

Sorry about that... I can edit out my post if you want me to do so.

[Sir Didymus]

No, no, just kidding...
There are no secrets to keep veiled... (I hope) :scared:
Just need some little more time to elaborate...

Cheers,
SD

[pg55555]

SD

Very good job.
I was peeking into your italian posting and found the result very interesting.

I have some comments but I will wait to the posting of your results in this forum.

Pablo

[ShadowKnight]

Can't wait to hear more :) sounds like a great idea, and definately something that is subjective, but very important to discuss. I'm very interested to see the results and hear the opinions :)

[Rockas]

@Sir Didymus
Thank you :)

keep it UP

[gizzin]

Haha, OPV will never produce better quality than multipass. But still good work Sir Didymus. I really appreciate the compliancy work. Keep it up.

[TheSeeker]

It may not produce better quality but that isnt what SSIM measures. It measures the videos similarity to the original stream. With OPV the encoded stream may be closer in similarity to the original but be of lower quality. Same with many transcoders. They might have a higher similarity to the original stream, but for high compression we all know that encoders do a much better job, as far as quality.

[Boulder]

Thanks SD, I'll just have to wait for my broadband connection to get fixed:mad:

@gizzin: one point of the comparison is surely to check whether it's true or not. Never say never;)

EDIT: results look quite interesting, very good job!

[Sir Didymus]

Originally posted by gizzin
Haha, OPV will never produce better quality than multipass. But still good work Sir Didymus. I really appreciate the compliancy work. Keep it up.

Hi gizzin. :)

Your appreciation ...is appreciated... By the way, while we are OT, I may add that at least in the compliancy job, even when I did some very strong statements I never received personal insults from anybody... So maybe better to switch back to that when these my never ending encoding sessions will hopefully conclude... :scared:

Ok. let me give you some answer (and I am trying to do this in the most educate way I can. If you feel I am not, let's meet to discuss the issue with the arms of your choice, as gentlemens, or in a coffee shop in front of a couple of beers [preferred...] ;) ...).

In short terms: when your say: "Haha" I will tell you "Hihi"...

How can you just state that "OPV will never produce better quality than multipass" ? Tout court ?

Get a look to this graph:

http://img86.exs.cx/img86/8075/graph1-4250.gif

It is extracted from the encoding at 4250 Kbps, where actually the VBR seems having in average a better behaviour (just to be clear: I am speaking about the SSIM behaviour). You may easily understand that even if in some few points the OPV encoding is clearily worse, in much more points (frames) OPV shows a little bit better figure. You should not be a matematician to understand that it may happen that in average the OPV behaviour may be better right ?

Ok I have also some hypotheses that may explain why it may happen, in some (not rare) cases that OPV is better...

Do you have any information or references or experiences or hypotheses to explain why OPV will never produce better quality than multipass ?

Cheers,
SD

[Sir Didymus]

Ok. Even though most subjective evaluations are still pending, let me anticipate some test results with SSIM and CCE encoding (quanti mat standard).

A very weak point on the whole test is that it was just encoded a single cell -V1C3- of the Kill Bill Vol. 1 movie. This is very limitative. Nevertheless the cell contains both action and static scenes, so let's stay with it.

Following the procedure described in another post, it has been generated a 4250 kbit/s ecl control file using the standard settings of DVD-RB.

This single ecl file has been manually tweeked for producing VBR encoding sessions from 1 to 9 passes, with bitrates of 4250, 3750, 3250, 2750, 2250 Kbit/s and two OPV encoding sessions finding EMPIRICALLY the Q factors producing m2v files nearest as much as possible to the target bitrate. This is to simulate the DVD-RB behaviour in case of a good estimate of the Q factor.

I have to take (or better to save from a thread that for other reasons is better to forget) some comments, that I totally share, on the matter of robot1, Jdobbs and Dragongodz. The synthesis is:


--> to be very careful before drawing conclusions on anything, since
SSIM index IS NOT SUITABLE to compare encodings generated by
different encoders/transcoders (this is fundamental, since the
GOP structure is mandatory it is the same, otherwise it may be
very frequent to compare I frames with B frames, and such thing
is meaningless).
--> do not blindly believe in the numbers and in statistics!
--> the only really reliable and sure tools for evaluating the
quality of your encodings are your eyes.


by robot1:
While I think that SSIM results are valid in test with the same encoder (as SD did with CCE), I doubt about the results when you compare different encoders (or transcoders). I did some tests (posted on the doom9 italian forum, if someone is interested I could find the link) with CCE, Shrink2.1 (not the last one!) and Procoder. Quality of Procoder and CCE was very good, while Shrink had a lots of blocks (not true for the last version, but I did the test in july). Anyway SSIM of the Shrinked clip was better than the one of the CCE clip (which was perfect, but the colors were just a bit washed). Looking at the backup, the CCE one was very pleasant, and you couldn't notice any differences from the original in normal play. Procoder quality was at the same level (or even better, for my taste).

In short: I think one should use SSIM to test only clips encoded by the same encoder.


by Jdobbs:
As was mentioned earlier -- be careful if you intend to use SSIM to compare different packages. I have seen comparisons that to the eye were intuitively obvious to the casual observer, but in SSIM values said the opposite. Numbers can lie.

My favorite example is the statistics of the hospital. In the U.S. ten times more people die in hospitals than other places. So the numbers say you have a 10:1 better chance of survival if you don't go to the hospital... hmmm...

SSIM is great for comparing apples to apples...


by Dragongodz:
if i may offer a couple of small comments.

while looking at charts done for SSIM or whatever quality evaluation tool is interesting it will never replace the 1 tool you will always use. that is your own eyes. they will give you more real feedback than anything else and can never be fooled as to what you will think looks better.

also certain things can be taken as true, such as throw enough bitrate at something and no new artifact or blocks etc will be created and it will look like the original(or pretty damn close). however comparing the quality of different encodes will always be individual and opinion(which should not need to be said) as everyone is an individual with individual tastes. person A will like encode A better while person B will like encode B better, thats life.

so please remember that when asking for proof of quality or such.


Here is the synthesis of the SSIM values aggregated for each test, reporting the average SSIM, the standard deviation and the filesize of the produced m2v files. The single subjective test I did on the matter (and I am very confident about this) is that for any given bitrate encoding family (I mean, for ALL of the encoding performed, for example at 3750 Kbit/s) it was absolutely impossible to myself to VISUALLY distinguish any difference... Even using the still frame comparison based on the avs script I posted previously. Further comments will follow, but for the moment I just did some VISUAL comparison between 4250 and 2250, and in this case the difference IS actually very noticeable: the noise and the little artifacts presents for the lower bitrate encoding, especially compared at still frames, is evident...

http://img118.exs.cx/img118/9862/stdmatravgssim.gif

http://img118.exs.cx/img118/1858/stdmatrstddeviation.gif

http://img118.exs.cx/img118/4465/stdmatrfilesize.gif

[Boulder]

The single subjective test I did on the matter (and I am very confident about this) is that for any given bitrate encoding family (I mean, for ALL of the encoding performed, for example at 3750 Kbit/s) it was absolutely impossible to myself to VISUALLY distinguish any difference... Even using the still frame comparison based on the avs script I posted previously.


This is one point why I prefer OPV. No need for those extra passes, saving precious time.

One point for using multipass VBR is getting strictly the desired bitrate. But even then, 1+1 passes should be enough.

However, with a good prediction method, the Q value can be predicted quite accurately. Tylo has achieved this with his D2SRoBa plugin for DVD2SVCD, and it also includes the possibility to do a sizing pass (or use a transcoder) if the filesize error is over the boundaries that the user sets. Based on my couple of tests, RB-Opt gets close to the target as well.

[Sir Didymus]

Just forgot to mention, that the full set of xls files can be downloaded from:

http://webmail.libero.it/r.php?d=libero.it&wr=3zqcxucf&ws=3zqcxucf&e=libero.it&c=pRY537m6OHGbLviOhR9IOTJVOq4vUQDl8021

Starting from these files, it is very easy (with just a minimum of Excel practice) to perform specific SSIM frame by frame comparisons and generation of graphs similar to the one posted before...

Again sorry, but the deadline for downloading is until 6-dic-2004 18.39. Clearly, after that date, you may ask me via PM to get them.

Cheers,
SD

[robot1]

Great works, SD.
I think I will never do more than 1+1 passes!
OPV is great for saving time, the only drawback I can see is the size prediction, which could be not very accurate.

Hint for DVD-RB users: if you set an OPV project, and the final size isn't right (but differs less than 10% from the target size), you can change the project in VBR using RB-Opt, and DVD-RB will perform the second pass, keeping the .vaf

[Sir Didymus]

Hi :(

I have one bad new, and one good new on the topic...

I am sorry to inform that I discovered one very little mistake in the data I posted.

1. Using VirtualDub 1.6.1, it happens that the dubbing ends to the frame before the last in almost every play I did for computing the SSIM values. This is introducing a very little systematic error in the LAST TWO FRAMES of the SSIM xls files. The recovery is very easy (and I am almost sure that nothing will change in the posted information) and consists in using the first 6795 of the 6797 SSIM values, but this implies the two tables of SSIM average and standard deviation should be edited. I will try to do the job asap.

2. Full encodings using a low bitrate matrix for doing the same experiment are complete. Hopefully this evening I will be able to post similar tables for the low bitrate quanti mat...

Cheers,
SD

[Boulder]

To get the thread back on track: have you considered any PSNR measuring? The folks at XviD department seem to use it with their quant matrix tests and when testing new unstable builds.

[Sir Didymus]

Hi...

Please find in this post the corrected tables reporting Average and Standard Deviation for the SSIM values produced in the test...

I have to say and to admit (as you may see) that the errors present in the original tables were two and very nasty...

So I decided not to remove the previous (buggy) tables, and to post the new tables here, adding some comments for explaining these errors...

Error 1:
In the calculation of the average SSIM values, the last two frames of the cell were included in the test. These frames were sometimes totally wrong, due to some VirtuaDub troubles... Overall effect wasn't to introduce a systematic error (as I hoped), but a seriuos mistake capable of changing some conclusions regarding the OPV behaviour at 4250 Kbit/s.

The shortcoming has been simply not to use these two last frames in any of the calculation of the SSIM average values.

Error 2:
Big mistake (and this was really my fault): I didn't properly understood that the overall average value (generated by the SSIM plugin and reported in my data) is not the simple aritmethical average of the frame-by-frame SSIM comparison, but it is a weighted average (where the weights are included in the second column of the csv files).
This is very interesting (and useful) since it allows to give for example no relevance to black frames, and to give more weight to frames judged relevant by the SSIM evaluation algorithm... Anyway, in the previous posted tables, the Std. Deviation was calculated arithmetically, without considering the weights...
So the Average tables were just a little bit wrong, and with a little bit of Excel work it was not difficult to recalculate the right values, while the Std Deviation table (which is a fundamental information to consider in the test) was definitely wrong... And its recalculation took to me some more time than I expected...

Hope the attached information, and the whole work it took to produce it, is worthwile...

Cheers,
SD

http://img98.exs.cx/img98/6944/ssimaveaggrcorrected8va.gif

http://img32.exs.cx/img32/685/ssimsdaggrcorrected5xc.gif

http://img98.exs.cx/img98/3687/filesizeaggrcorrected8hz.gif

Edit 12-12-04: Included filesize (third) table. It is the same than on page 1; just to improve "random access" reading...

[Sir Didymus]

Originally posted by Boulder
To get the thread back on track: have you considered any PSNR measuring? The folks at XviD department seem to use it with their quant matrix tests and when testing new unstable builds.

Very good idea, and useful comment, Boulder.
I initially considered it as a possible alternative to SSIM, but now I think it could be very useful as an additional test, if this thread will survive..., since all encodings are available, so it should be not so complex to do...

My biggest concern, on the other hand, is the limitation of the test to a single cell of a single movie...

P.S. Sorry for the delay in posting some results about the test based on the Very Low Bitrate Quanti Mat of CCE. The test is complete, but I should perform the data analysis.

[Boulder]

My biggest concern, on the other hand, is the limitation of the test to a single cell of a single movie...


That shouldn't be a big problem as the test is quite extensive nevertheless and as you said, there's fast and low motion scenes involved. Besides, a test can never cover everything, there's only 24 hours in a day;)

[dragongodz]

That shouldn't be a big problem as the test is quite extensive nevertheless and as you said, there's fast and low motion scenes involved. Besides, a test can never cover everything, there's only 24 hours in a day
obviously but if you want more reliable results you do need to do larger/longer encodes(maybe not a whole movie but a reasonbably large section) and to do multiple varied sources. that would show clearer under what conditions OPV may give better results and where it wouldnt. yes a huge task that would take a very long time. :)

[wmansir]

I just went thru this thread and removed a lot of personal attacks and garbage. I have PM'ed those involved and hopefully the thread can remain productive.

If you feel a post is in violation of the rules (including personal attacks) please use the "report this post" link and refrain from responding in public. That includes asking for a mod to do something, which may just provoke the situation and have less of an effect because we may not see it.

Let's try and keep this on topic.

[Boulder]

OK, time to make a clean thread and keep it that way.


that would show clearer under what conditions OPV may give better results and where it wouldnt.


If someone is willing to do such a test, I suggest using Saving Private Ryan as the source material. It has ultra shaky camera movement, and many feel that multipass VBR excels when encoding the movie. It's also quite long so the average bitrate for the whole movie would become quite low compared to many other movies.

Probably would provide some clues regarding the pros and cons of VBR and OPV.

[dragongodz]

also a lot of anime can be difficult so picking a suitable title may show further interesting results aswell, probably different again from what SPR would show. :)

[Boulder]

I agree, anime and cartoons are a tough case and have their own distinctive ways. Just need to make sure the source isn't a crappy conversion:D

[Sir Didymus]

Originally posted by dragongodz
obviously but if you want more reliable results you do need to do larger/longer encodes(maybe not a whole movie but a reasonbably large section) and to do multiple varied sources. that would show clearer under what conditions OPV may give better results and where it wouldnt. yes a huge task that would take a very long time. :)
Totally agreed, and happy to see you underline this aspect: never forget the scientific approach. A single cell of a single movie is not sufficient for taking general conclusions.

For the moment let me just complete the first test (now that the whole thread has been "edited" I am a little bit more confident this is feasible). By the way, taking the opportunity to say where this work is assumed to go, and what conclusions are expected from my point of view:

1. The whole test is based on the usage of the SSIM tool. This is a limitation.

2. The test is based on a single cell (6795 frames) of a single movie. This is a big limitation

3. First step (completed) was to generate some tables with encodings produced "starting from" an eclcce file generated by DVD-RB. For the five bitrates of 4250, 3750, 3250, 2750, 2250 Kbit/s it has been executed 9 VBR encodings (from 1+1 to 1+9 passes) and two OPV encodings, finding empirically the Q values better matching the targer bitrates. THIS STEP HAS BEEN COMPLETED USING THE DEFAULT STANDARD QUANTI MAT.

4. Step 3 has been repeated for the bitrates of 3750, 3250, 2750, 2250, 1750, using the Very Low Bitrate Quanti Mat of CCE. (encodings completed, still pending the data analysis.

5. The same, for bitrates of 3250, 2750, 2250, 1750, 1250 should be carried out using both the Ultra Low Bitrate Quanti Mat of CCE and the Notch Matrix (here again just completed the encodings for the CCE Matrix, nothing done with the Notch). Here just want to point out that I think I will use the Notch Matrix with DC precision set to 10 bits, while the recommended is 8 bit, for making omogeneous comparisons with all other encodings performed with DC precision set to 10 bit. I hope this makes the test with Notch still useful. If someone is frequently using the Notch matrix (me not...), please give me some advice...

6. At the end, when all encoding and data analysis will be done based on SSIM, I am very interested in making the same comparisons based on PSTN, as suggestd by Boulder; it shouldn't take too much more time...


EXPECTED CONCLUSIONS (Sorry most are obvious...).

1. Change your feelings and your way of encoding only if you are very confident on what you do; and especially follow what your eyes are telling: if you actually are capable of seeing differences between encodings performed using OPV and 1+3 passes, or 1+9 passes follow what is better for yourself.

2. I think that the discriminatory power of SSIM regarding the bitrate can be accepted as a fact: the different classes of encodings at different bitrates produce different SSIM values.

3. The SSIM tool is actually not discriminating between VBR encodings: for a given bitrate it actually seems all VBR encodings produce similar results.

4. Further conclusions will follow as soon steps 4. 5. and 6. above will conclude...

SD

[TheSeeker]

This is a little OT but I was wondering about your comment on DC precision as it relates to the Notch matrice. First what exactly is the DC precision and when if ever should I think about changing it? I have always used 10 bits and I frequently use the notch matrice in low bitrate situations. You mentioned Sir Didy that it is recommended to use 8 bit dc precision with the notch matrice. Why is that?

[Boulder]

Actually there should be no relation between the matrix and DC precision, it's the avg bitrate that counts. I use 8 bits for low bitrates (in my mind ~2500kbps and below), 9 bits for mid bitrates (2500-4000kbps) and 10 bits beyond that. If the video is extra colorful and detailed, I use a higher value than I normally would. Higher DC precision value produces slightly higher filesizes (but only very slightly).

[Sir Didymus]

I was exactely asking some hints from people using Notch, since I really prefere just changing the quanti mat matrix used in the encoding, without changing the DC precision parameter too. I just quickly read into kvcd.net this 8 bits recommendation...

I suppose that DC precision at 8 bits may be helpful for improving compressibility of streams... But what about the quality ? And you know, better asking before making "improper settings"...

Cheers,
SD

Edit: by the way, here there are some tables about the test using the Very Low Bitrate Quanti Mat of CCE...

Nice w/e to everybody...

http://img100.exs.cx/img100/8182/04-AvgSSIMVeryLowBR.gif

http://img100.exs.cx/img100/7071/d7-SDSSIMVeryLowBR.gif

http://img100.exs.cx/img100/8385/2e-FilesizeSSIMVeryL.gif

[Boulder]

I don't think there is an easily perceivable difference between the different DC precisions. The 8-bit recommendation comes from the fact that the Notch matrix was developed for lowish-bitrate XVCDs, in which the max bitrate was 2500kbps.

Maybe it would be worth it doing a small test, with maybe 1+1 passes with all three different precisions and see how the values change?

For low bitrates, I'd go for either the Notch or Bach's matrix (a slight compression gain over Notch).

[jdobbs]

2. The test is based on a single cell (6795 frames) of a single movie. This is a big limitation I think you are going to find this isn't as large a limitation as you think. In my early testing with DVD-RB I was concerned about encoding cells individually because it wouldn't take full advantage of VBR due to the limited sample across which to balance the bandwidth... As it turned out that was a wrong assumption. The "magic threshold" in sample size was much smaller than a typical cell.

There's only one way to prove it of course -- but I think you'll find that even if you encode 20 complete movies the results will deviate only slightly from what you have already found.

Excellent work.

[jdobbs]

Also, I am not surprised at all at the lack of distinction between multiple passes. As I have said in the past -- once you have gone through the file once you have collected all the information necessary to decide the best allocation of bits across all frames... and you can apply that in the second pass.

[pg55555]

The problem I see with SIMM is its poor discrimination:
From SD tables you got that for 4250 kbps the SIMM is 0.981480 and fr 2250 kbps it is 0.965700, a 1.6% reduction for a compression ratio of 52%. More important, SD reported the quality degradation with the lower bitrate was noticeable

Looking at the original paper of Dr. Zhou Wang (one of the developers of SIMM) at http://www.cns.nyu.edu/~zwang/files/papers/ssim.html (link porvided by Sir Didymus in other thread)one can see this is a characteristic of SIMM: in a graphic where it is plotted the subjective quality evaluation of a series of JPEG images Mean Opinion Score or MOS vs the SIMM index, a big perceived quality degradation (MOS from 83 to 75)resulted in a SIMM index still above 0.97.

On favor of SIMM, as SD itself indicated, lower bitrates correlates with lower SIMM.

I would add that, if not completely convinced, I have already moved from my standard setting in CCE from 1+4 passes to 1+3. And probably will reduce it further to 1+2. I'm not sure if I would dare to 1+1, but if the PSNR tests show similar results, I should.

By the way, from the same paper, PSNR discrimination properties are just the opposite, small differences of subjective quality (MOS) produced big changes in the PSNR measure.

SD, thanks for a great work. I truly belive your work is enriching us all.

Pablo

[dragongodz]

once you have gone through the file once you have collected all the information necessary to decide the best allocation of bits across all frames... and you can apply that in the second pass.
so the vaf pass is (roughly)equivilent to a normal first pass as done by other encoders ? i once asked about this but from memory nobody was sure. :)
if so then yes 1+1 should give good results a lot of the time but 1+2 should never be worse and should sometimes be better. the source can play a major part in that which is why i also said multiple types of source etc should show up differences in the different encoding types. anything over 1+3 is overkill and i wouldnt worry about, would save quite a bit of test encoding time too. :)

[dragongodz]

no artificial quality test is accurate. PSNR is even worse than SSIM because you can actually get a higher PSNR with a worse video. :)

[jdobbs]

Originally posted by dragongodz
so the vaf pass is (roughly)equivilent to a normal first pass as done by other encoders ? i once asked about this but from memory nobody was sure. :)
if so then yes 1+1 should give good results a lot of the time but 1+2 should never be worse and should sometimes be better. the source can play a major part in that which is why i also said multiple types of source etc should show up differences in the different encoding types. anything over 1+3 is overkill and i wouldnt worry about, would save quite a bit of test encoding time too. :) Yes the VAF pass is nothing more than a first pass. In you want, you can even replace it with a OPV or CBR pass for creating the VAF.

[Rockas]

Yes the VAF pass is nothing more than a first pass. In you want, you can even replace it with a OPV or CBR pass for creating the VAF.

I think that if people stop for a little and think could get that conclusion easily :)
If it weren't that way what good could the basic version on CCE be?
That's why I alway make my encodes 1+1 and rarely 1+2 :)

keep it UP

[jdobbs]

Other than testing I've never done more than 1+2.

[Sir Didymus]

Originally posted by jdobbs
I think you are going to find this isn't as large a limitation as you think. In my early testing with DVD-RB I was concerned about encoding cells individually because it wouldn't take full advantage of VBR due to the limited sample across which to balance the bandwidth... As it turned out that was a wrong assumption. The "magic threshold" in sample size was much smaller than a typical cell.

There's only one way to prove it of course -- but I think you'll find that even if you encode 20 complete movies the results will deviate only slightly from what you have already found.

Excellent work.

Thanks a lot for the compliment, boss, it's really appreciated.

Your comment sound really interesting (and very relevant IMHO)...

Of course it's totally out of my very limited time to try following a systematic approach testing 20 movies or even 20 single cells (also it's out of the possibilities of my poor storage space: up to now the whole fileset for the test of just one single cell is taking more than 18GB... and I didn't even start Notch Quanti Mat Encodings...).

Not to talk about some "little allergic reaction" I am actually suffering cause of this SSIM indigestion of these last weeks... :scared:

Cheers,
SD

[Sir Didymus]

Ok. Sorry this test is taking more time than I expected. Here are some results based on the CCE Ultra Low Bitrate Quantimat...

http://img53.exs.cx/img53/812/avgssimultralowbr4bg.gif

http://img54.exs.cx/img54/4222/sdssimultralowbr1hi.gif

http://img36.exs.cx/img36/812/filesizessimultralowbr0nf.gif

I have to explain that for encodings at 1250 Kbit/s it was necessary to perform three OPV encodings, instead of the usual two. As you may see (in case encoding mistakes are absent, as I hope, since attached values have been triple checked...), at such low bitrates, the quality factor vs bitrate curve is almost flat, and the VBR bitrate hitting precision is not so high.

So the first Q factor has been choosen in order to produce an encoding sized more than the maximum of the nine VBRs, the third Q factor in order to produce an encoding sized less than the minimum, the second Q factor in order to produce an encoding sized as much as possible near to the average...

Next "episode" will be based on Notch Quantimat...

Cheers,
SD

[dragongodz]

hmm interesting...but i will refrain from comments/observations until more test are done. :)

i would suggest though that 1+6 and over is really just wasting your time and you would get much more done by just dropping them from the testing. up to you of course.

can you please include the actual sizes produced aswell ? that way we can see hopw close the OPV's are to the desired end size.

[Sir Didymus]

Hi dragongodz :)

Thanks a lot for your full understanding, both when you say you'd prefere waiting for some more tests done and for your suggestion of dropping unrelevant (1+6 and more...) VBR encodings.

Especially appreciated since I did very little subjective evaluations, apart some very general ones, already posted.

I am actually near to complete the encodings based on the Notch Quanti mat, so I think that in more or less one more week the encoding sessions (including maybe also the DDogg Bach1 matrix) will be done... At this time, just preferring spend some more time in encoding with the same nr of passes and applying the same testing procedure for all of the matrices...

What do you mean when you ask to include the actual sizes produced ? The third table I posted (in all of the tests) is actualy reporting the filesize, in Kilobytes, for each one of the encoded segments...

I included this table exactely for showing what you are asking: the Q values for the OPV encodings have been choosen in order to match as much as possible the sizes produced by the VBR encodings. For the tests using the STD matrix and the Very Low Bitrate Matrix it was possible to identify exactely two Q values matching very well (by excess and by defeat) the encoding sizes generated by the VBR encodings. With the Ultra Low Bitrate Quantimat, just at 1250 Kbyte/s, it was necessary to use three Q values in the OPV encodings (and please note these Q values are absurdely high...). The reason was explained in my previous post...

Cheers,
SD

[dragongodz]

The third table I posted (in all of the tests) is actualy reporting the filesize, in Kilobytes, for each one of the encoded segments..
woops missed that. look at the first lot of results on page 2, no size table. so i was just looking at the first table and not scrolling down. :)

once you are finished testing different matrices are you considering testing on different types of source ,with just 1 matrix, to compare what effect that has aswell ?

[Sir Didymus]

Originally posted by dragongodz
look at the first lot of results on page 2, no size table...

once you are finished testing different matrices are you considering testing on different types of source ,with just 1 matrix, to compare what effect that has aswell ?

Yeah. The filesize table for std quantimat results (page 2) is posted on page 1. It was necessary to post two times the SSIM values, for the standard CCE matrix, due to some calculation errors, and I didn't include the filesize table on page 2 since it is the same table than the one on page 1. But (as you have experienced...) this makes not linear the reading of the thread, so I will edit the post on page 2 in order to improve the readability, including the filesize table...

Well, my plans, (for the future)... are to complete the test for the Kill Bill Vol 1 and making some subjective evaluations on some (few) specific encodings, with my personal comments and comparisons...

Then I have the idea of evaluating the PSNR for the whole set of the produced encodings. As a further step I would repeat the whole test (but limiting to 4 or 5 the VBR encodings) to at least another cell of another movie. Maybe an anime (Balto ?)... Maybe even a third cell could be tested, but not more...

I know three cells it is very limited data set, but I can not spend the whole spare time of my life into this test... :D

By the way, my experience in encoding anime movie is very very limited. I am almost sure it is better to change at least quality_prec from 16 to 24 or to 32, from the default of 16. Have you further suggestions on the matter ?

Cheers,
SD

[Rockas]

@Sir Didymus
I know that words are taken by the wind and we forget them quickly and they also have another "malfunction", they aren't enough to say how much your work should be appreciated by everyone.
Anyway...
Thank you

and keep it (really) UP