[Test Codec] H264, VP6, RV10, WMV9, DivX3, XviD, DivX5 .....

[Sagittaire]

Matrix II 720*304

Source DVD PAL 720*576 MPEG2 MP@ML, chapter 20 to 26, ~35 000 frames

Test in french (http://jfl1974.free.fr/HTM/32_Test_Codec_Video.htm) ... but Capture, Sample & Graphs are universal language

Codecs
- H264 Ateme
- XviD 1.0.1
- RV10 Elysean
- VP6 6.2
- DivX 5.2.1
- WMV9 VCM
- DivX 3 ffvfw


Tests
- 500-2000 Kbps and 25-100% compressibility (/quant2 XviD H263)
- SSIM, Average PSNR, Overall PSNR and Frames PSNR test
- Blind test
- Samples
- Diaporama

[bilu]

What's the criteria difference between the two images?

One is called PSNR and the other PSNR Overall... :confused:

EDIT: does that trailer have high-motion parts?

Bilu

[Sirber]

Nice comparison :D

[DAvenger]

Ah, could you please explain the results to us uneducated plebs? :p

Thanks

[TheXung]

Could you use more differing colors for xvid and vp6 or at least make the lines thicker than a pixel.

[Assault]

@ Sagittaire

Did you use XviD with GMC and Trellis? And for your b-frame settings I would recommend 1/150/100. I think this should deliver a little better results but altogether it's a very nice comparison. Thanks :)

@ all

When you look at the two graphs it's very interesting to see that VP6 is a good codec but has a bad 2-pass algorithm. Its results are much better for average psnr than for overall psnr. I think that's because average psnr doesn't take into account how constant the quality is. So I suspect that VP6 has both frames with VERY good quality and frames with VERY bad quality. That should be the reason why the results for overall psnr arent really good. If On2 improves the 2-pass mode VP6 will be a very good codec in my opinion.

Assault

[CruNcher]

@Sagittaire

please post more infos filesize and time needed to encode

[SeeMoreDigital]

Originally posted by Assault
When you look at the two graphs it's very interesting to see that VP6 is a good codec but has a bad 2-pass algorithm. Its results are much better for average psnr than for overall psnr. I think that's because average psnr doesn't take into account how constant the quality is. So I suspect that VP6 has both frames with VERY good quality and frames with VERY bad quality. That should be the reason why the results for overall psnr arent really good. If On2 improves the 2-pass mode VP6 will be a very good codec in my opinion.

Assault It's interesting you should mention that!

I've been trying to generate 2pass VBR encodes with VP6 (using MPEGmediator) but on every occasion the encode looks either the same or only very slightly better than a 1pass CBR VP6 encode.

Personally, I've been thinking that I may be doing something wrong when generating 2pass encodes. But I don't know for sure!

I really would like to get to the bottom of this. Because if VP6 is able to look as good as it does in CBR mode, it should look astonishing in VBR mode. I just wish I could get it right!

Cheers

EDIT: Anyway, what's with the DivX 5.1.2 ident. I thought the latest is DivX 5.1.1 beta2.... Or am I wrong?

[Assault]

@ SeeMoreDigital

I don't think that you're doing something wrong when creating 2pass encodes. I wasn't content with the results I got in my tests either. I also remember C0mPr355 saying that VP6 is their first codec with a pass mode. So perhaps that's the reason for the bad results. :(
Concerning DivX I think you're right. It's probably a typo.

Assault

[SeeMoreDigital]

Thanks for that Assault.

I thought I was begining to flip out there for a second - twice!

Yesterday I did some more low bitrate CBR tests with VP6 using 720x576 image pixel frame sizes. It really is very good!

However, I personally would like to be able to enter kb's instead of KB's per second. And maybe the ability to store the stream in an MP4 container instead of (or as well as) on AVI.

I realize that this codec was originally concieved for use with cellular phone networks. So with that in mind I would have thought being able to shove the video stream in an MP4 container would have been more practical!

Or am I really flipping out now???

Cheers

[Assault]

@ SeeMoreDigital

AFAIK VP6 is no mpeg4 codec. I don't think that it's possible to put non mpeg4 compliant codecs in an MP4 container. And even if it were possible it wouldn't be a good idea in my opinion because the resulting MP4 wouldn't be spec compliant.

Assault

[CruNcher]

@ SeeMoreDigital
i think Christian is allready lurking arround and allready talks with the other Matroska guys howto support it in MKV aswell hehe im sure about that why ??? because i know him :D

[Sagittaire]

Update: VP6 setting and file size

[vinouz]

So please could someone explain (Sagittaire ?) what the difference between PSNR and PSNROverall is ? Do that mean that on PSNR you take the 1324th frame's PSNR ? :p
As I'm the third to ask and no one seemend to answer, I think it would be glad all the others (who surely understood, if they discussed the results without asking what they mean) or Sagittaire make a little step down the stairs to educate at least three ordinary folks.
If the results weren't that different between first and second graph I wouldn't have been much worried about my lack of comprehension...

Thanks in advance,
Vinouz.

[C0mPr355]

@ Sagi

I believe the best results for psnr you can get on vp6 should have the post processing level set to "Best PSNR." This can be done in Start>settings>control panel>vp6 decoder settings.

also could you retry your test with an auto keyframe of 360 rather than 120? I have stated that 360 is a much better number to use because of the way that vp6 generates keyframes.

Please only use Spatial and/or Temporal resampling when you are unable to hit the datarate on much lower bitrate clips. A datarate of 500-1000 really should not have a need to drop frames or dynamically downsize the encoding resolution to hit the datarate.

[Assault]

@ vinouz

I'll try to explain the difference between Overall PSNR and Average PSNR as far a I understand it. The average PSNR (first graph) is calculated by the average PSNR values (as the name already says ;))of the single frames. This means for the final result it doesn't matter how constant the quality of the video is. As an example you can take three frames. Codec A delivers a result of 10dB for the first frame 40dB for the second and 70dB for the third. Codec B delivers 39dB, 40dB and 41dB. The average PSNR for those two codecs will be exactly the same. Nevertheless in "real life" you would certainly prefer the quality of codec B because nobody wants a frame with 10dB in his encode. ;)
Overall PSNR on the other hand takes into account how constant the quality of the video is. But please don't ask for the exact formula of Overall PSNR. It's quite complicated (You can search in the Avisynth forum when you are interested).
So IMHO the Overall PSNR is preferable because it's the aim of the 2pass mode to deliver as constant quality as possible.
I hope that helps.

Assault

[Sagittaire]

Post-Process decrease PSNR and "Best PSNR" too ... for my trailer in 500~1000 Kbps interval

keyframe maxi 360 or 120 frames obtain the same PSNR because the encoder himself choose the keyframe position and frequency for bframe is generaly inferior at 360 frames ...

The VP6 is really bad with the PSNR Overall and really good with¨PSNR because it encode very well the not very complex frames which are in a majority and very badly the complex frames which are minority. Many blocking and ringing in the fast scenes which are cleaned with a very good Post-Process. The VP6 is thus not a good codec to make true VBR ...

[vinouz]

@Assault : thanks for the explanation.

[MfA]

"~~ PSNR, SSIM, VQM Test ~~"

?

[C0mPr355]

@ Sagi

can you please post the filters you used for Avisynth to resize the image as well as where the source is from (the dvd ?).

ty

[superdump]

Vinouz: The PSNR formula involves a logarithm. So if you average the PSNR values across the frames it gives a different and biased result as opposed to averaging all the MSE values and then plugging that into the PSNR formula at the end which gives the overall PSNR. Search for PSNR formula on google or something. You'll see what I'm talking about. :)

[Blueseb]

Well, I watched XviD and VP6 samples and, as far as I can trust my eyes, it seems that is VP6 that have a more constant quality (& generally less detail) against XviD that have a more variable quality (& generally more detail). All this IMHO, correct me if I'm wrong :p

[vinouz]

@superdump : thanks. All I wanted to know is what "Overall" was relating to. For mathematics I'm OK (E[ln(dx)] != ln(E[dx]) of course). What I didn't know is that "PSNROverall" meant PSNR over the average SAD of all the frames, instead of average of PSNRs of all the frames for "PSNR". Is that what you meant ?

[SeeMoreDigital]

PSNR test's are all well and good. But they can be easily fiddled by altering the brightness, contrast or colour of the source prior to encoding!

They are no replacement to a good set of eyes. Even leaning your head to one side can make an image look worse. And if you don't believe me, try watching say, the TV with your head at 90 degrees to the image. The closer you are to the screen the worse the lines look!

Amazing, the tricks your eyes can play on you, in it!

Cheers

[Sharktooth]

Had a look at those avis and there are some things i've noticed.

xvid-600.avi has good details but shows extreme "blockiness" in high motion scenes, some texture motion visible in still backgrounds.

divx-600.avi has good details and no blockiness even in high motion scenes but background texture movement is clearly visible in still backgrounds.

vp6-600.avi has very good details in low motion scenes but blurs out in high motion. No blockiness but shows weird colors when playing very high contrast scenes.

vmw9-600.avi shows variable details from nearly good to very good. No blockiness seen even in high motion scenes.

My opinion is vmw9 and divx are the best encodes.

EDIT: I didnt watch the RV9 clip.

[Sirber]

My opinion is vmw9 and divx are the best encodes.Have you seen that RV9 beats every codec at 600kbps looking at the PNSR scores? I bet you haven't tryed the RV9 clip. How can you give your opinion whitout watching the whole thing?

[ookzDVD]

I think we should add the new 3ivx into the test. :)

[Sharktooth]

Originally posted by Sirber
Have you seen that RV9 beats every codec at 600kbps looking at the PNSR scores? I bet you haven't tryed the RV9 clip. How can you give your opinion whitout watching the whole thing?
If you carefully read my post you see i didnt watch the RV9 encode. The reason is WE ALL KNOW ITS THE BEST.
I just watched the other encodes and draw some conclusion.

[Sirber]

Is it forbidden to talk about the best codec? to review it with others? If someone read your post and don't know much about video compression, what will he think?

[superdump]

We don't all know it's the best. We all know it gets the best PSNRs. I still find it slightly blurry though the high frequency detail retention thing Karl added had made it better. I still prefer the details (and yes Sirber, I can live with some blocking occasionally :P ;)) of XviD myself but, each to their own.

[Sharktooth]

I haven't the time to discuss about silly arguments so I edited my post...
Hope its all clear now.

[iwod]

I asumer when PSNR is being compared Xvid and DIvx don't have pp on........

But i would think RV9 would have it as by default.

So how would PSNR have anything to do with quality?

[deXtoRious]

That's the real problem with these tests. They only test the codecs technically, they test how the video is encoded. They don't really have much to do with real quality because the final movie that we see is always (unless of course you are some kind of a masochist not using pp ;)) irrevokably altered by post-processing in order to increase the relative quality that we perceive. It is rather pointless to try to rationalize this as it does not really give us feasable information that could be used to determine the best codec available (yet some people try to interpret these results as absolute ratings of quality). However, we do it anyway :D

[Sagittaire]

Update ... ;-)

1) SSIM Test
2) VQM Test
3) Script used for encoding and tests
4) 3ivX Demo
5) Vob source: 75 Mo ...

[SeeMoreDigital]

Originally posted by Sagittaire
Update ... ;-)

1) SSIM Test
2) VQM Test
3) Script used for encoding and tests
4) 3ivX Demo
5) Vob source: 75 Mo ... Thanks for posting the 3vix encode.

It doen't look half bad. Better than I've been able to manage so far (mine have been 720x576 anamorphic).

Can you confirm the settings you used please?

Cheers

[Sagittaire]

Defaut setting and FouCC XviD for ffdshow decoding ... or desactive Post-process with 3ivX decodeur ...

PP = :devil:

[Sharktooth]

Sagittaire (if you have some spare time) can you plz add the final divx5.1.1 to your comparison?

[bilu]

Why do the VQM results seem opposed to the PSNR and SSIM ones? :confused:

Bilu

[superdump]

Because a lower VQM value is better. :)

[On2Tech]

Thanks for performing this comparison. It's helped On2 find some bugs in VP6. We'll have a new version out in the next couple of weeks.

Here's a preview of where we stand now: I'll post a description of the bug fixes in my next post.

Clip Size ---- PSNR -----
Overall Avg
======= ======= =======
HP.rmvb 10,387KB 41.4208 43.9264
HPVP6.avi 10,424KB 41.5209 44.3266

Description
===================
HP.rmvb Real clip produced with RMFactory & your settings
HPVP6.avi VP6 clip built with the prealpha new version.

NOTE: When the codec is finally released the results may differ ( we hope they'll actually be better).

[On2Tech]

1) Missed Key Frames

The vp6 encoding missed marking 20 some-odd frames as keyframes. Since VP6 generally encodes the keyframes at a higher quality than surrounding interframes this hurt the overall quality.

2) Keyframes that were way too big.

We realized that the VP6 keyframes were being compressed at datarates that are much higher than they should have been. This meant we effectively wasted a large number of bits making the first frame of a cut scene pristine at the expense of the rest of the frames of the section.

3) Two Pass Quality Control

We were very conservative in our two pass datarate control. More aggressive quantizers seem to help the Overall PSNR number by quite a bit.

4) Exposed Encoder Complexity Number.

We have some control over how much time we spend performing motion searches, we had it disabled because we wanted to make sure it ran relatively quickly by default.

[On2Tech]

One point about these clips. I agree with those people who claim that both PSNR and Overall PSNR are extremely poor measures of a clip's quality. Especially in the case of a clip such as this Harry Potter trailer in which the entire clip is a conglomeration of a large number of different small sections.

In this clip there are a large number of black frames. Getting these black frames exactly right means you get a PSNR score of 100 in the avg PSNR metric. Some of the encodings we did of VP6 produced an exact match of these black frames others produced psnrs near 55. Problem is they all look black. Visually there is no major difference between 55 and 100 db on a black frame but there is a huge difference when it comes to average PSNR.

Likewise Overall PSNR is heavily biased against the odd bad frame. In our clip of 3176 frames a single really awful frame ( let's say with a PSNR of 10) would drag the entire clip's overall PSNR number down by 1.6 db even though its just 1 frame.

I suggest that PSNR is far more relevant on small single scene clips like those in the MPEG-2 test suite.

[CruNcher]

Originally posted by On2Tech
Getting these black frames exactly right means you get a PSNR score of 100 in the avg PSNR metric.

ehm isn't it actually infinite then ?

[Sagittaire]

PSNR isn't a very good video quality mesurement but Overall PSNR, SSIM and VQM are convergent results and not in favour to VP6 ...

[Sirber]

J'té cassé! (Brice de Nice)

(eng: I broke you!)

[On2Tech]

When we made some simple fixes we beat Real 9 's Overall PSNR score on the Harry Potter Clip by .1 db !!! I'll check the other quality metrics tomorrow.

B.t.w. We used rmfactory downloaded from the site yesterday with the exact audience you listed in your first test.

The fact is 2 or 3 bugs in the two pass compressor hurt vp6 badly on that clip. Don't take my word for it. Wait a couple of weeks download the codec and try again ( or pm me and/or Compr355 and we might get you an early copy).

You will have your shot at it soon...

Note: The psnr isn't the only thing that improved. Visually the quality is much better than the vp6 file you posted. I'll test using the other quality metrics as well, but I have no doubt that you will see massive improvements.

P.p.s. With the VP6 post processor set at deblock only ( ie no deringing, no noise filtration or black clamping etc) helped the clip we built by .5 db on overall psnr ( like .3db on avg). I am not sure why your results differed.

[On2Tech]

Originally posted by CruNcher
ehm isn't it actually infinite then ?

You are correct, you do get infinite psnr for perfect reconstruction. Unfortunately I was referring to the results you get when you use the Compare function within avisynth. For whatever the person who wrote that code settled on a perfect reconstruction being 100.

I think 60 might have been a better choice but that's a post for a different forum.

[Stux]

Originally posted by On2Tech
I think 60 might have been a better choice but that's a post for a different forum.

Probably clamping to 60 then, because you can get near perfect reconstruction and actually end up with 70, 80 or 90dB

[Sharktooth]

Glad to see VP6 is getting better.
Keep up the good work guys.

[Sagittaire]

VP6 600 Kbps


Minimum Average Maximum
Mean Absolute Deviation: 0.0007 1.2970 3.7648
Mean Deviation: -0.1555 +0.0432 +0.2624
PSNR: 32.7076 43.6361 79.3822
Overall PSNR: 41.2033


RV9 600 Kbps


Minimum Average Maximum
Mean Absolute Deviation: 0.0006 1.2024 2.6072
Mean Deviation: -0.2449 +0.1311 +0.7298
PSNR: 36.4788 43.7111 80.0075
Overall PSNR: 42.6571


Very good average PSNR but Overall PSNR is very bad (same result with SSIM and VQM). Encoding is very good for slow motion (frames generaly in majority) and very bad for high motion (frames generaly in minority) thus good average PSNR but very bad overall PSNR (More influance for complexe frames in PSNR overall)

SSIM and VQM (Psycho video quality mesurement) are very bad with VP6 I don't understand why ... ??

Best Profil for PSNR and RV9


<?xml version="1.0" encoding="UTF-8"?>
<audience xmlns="http://ns.real.com/tools/audience.2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://ns.real.com/tools/audience.2.0 http://ns.real.com/tools/audience.2.0.xsd">
<avgBitrate type="uint">580000</avgBitrate>
<maxBitrate type="uint">5800000</maxBitrate>
<streams>
<stream xsi:type="videoStream">
<pluginName type="string">rn-videocodec-realvideo</pluginName>
<codecName type="string">rv9</codecName>
<codecProperties type="bag">
<encoderComplexity type="uint">100</encoderComplexity>
<customPacketSize type="uint">16000</customPacketSize>
<noisyEdgeFilter type="bool">false</noisyEdgeFilter>
<patternAdaptivity type="uint">1</patternAdaptivity>
<scalingFactor type="uint">80</scalingFactor>
<calcPSNR type="bool">true</calcPSNR>
</codecProperties>
<encodingType type="string">vbrBitrate</encodingType>
<maxStartupLatency type="double">60</maxStartupLatency>
<maxFrameRate type="double">25</maxFrameRate>
<maxKeyFrameInterval type="double">10</maxKeyFrameInterval>
<enableLossProtection type="bool">false</enableLossProtection>
</stream>
</streams>
</audience>