My recent tests w/ Nic's latest build

[Bulletproof]

These are some tests I did with Nic's latest build. Keep in mind that PSNR is not a definitive measure of quality, these numbers are best used as just a factor in your decisions and not a single deciding factor. I have also included two matrix files that I have done, one of them is my "Heavy compression" matrix, this will make a smaller file than using the H.263 matrix, however you may notice some artifacting around objects, I only advise you to use this matrix on live action movies with lots of movement. The other matrix I have included is the "High quality" matrix, this will make a bigger filesize, but not extremely large and is worth the quality it can give in my opinion. The file I worked with is 5608 frames long, and is of a 3D CG movie, the PSNR tests were done with 300 vital frames of the movie which contain many varying forms of motion, color, and detail. All filesizes that are displayed show the entire 5608 frames encoded and not 300 frames. These were all done using constant quantizer 2.


Original File Size: 565,639,168

H263:

Average PSNR (Y U V): 44.57, 47.45, 47.08
Total Average PSNR: 46.37
Filesize: 123,009,024

H263 + Luminance Masking:

Average PSNR (Y U V): 43.19, 46.63, 46.16
Total Average PSNR: 45.33
Filesize: 113,905,664

H263 + Qpel:

Average PSNR (Y U V): 43.90, 47.32, 46.96
Total Average PSNR: 46.06
Filesize: 128,686,080

H263 + GMC:

Average PSNR (Y U V): 44.57, 47.46, 47.07
Total Average PSNR: 46.37
Filesize: 123,211,776

H263 + Chroma:

Average PSNR (Y U V): 44.69, 47.67, 47.26
Total Average PSNR: 46.54
Filesize: 120,586,240

H263 + Chroma Optimizer:

Average PSNR (Y U V): 44.51, 47.32, 46.97
Total Average PSNR: 46.27
Filesize: 123,131,904

H263 + VHQ1:

Average PSNR (Y U V): 45.05, 48.13, 47.73
Total Average PSNR: 46.97
Filesize: 110,811,136

H263 + VHQ2:

Average PSNR (Y U V): 44.85, 48.05, 47.66
Total Average PSNR: 46.85
Filesize: 108,177,408

H263 + VHQ3:

Average PSNR (Y U V): 44.77, 48.03, 47.62
Total Average PSNR: 46.81
Filesize: 107,204,608

H263 + VHQ4:

Average PSNR (Y U V): 44.72, 47.99, 47.58
Total Average PSNR: 46.76
Filesize: 106,463,232

H263 + B-Frames 1 150 100:

Average PSNR (Y U V): 23.67, 38.88, 37.25
Total Average PSNR: 33.27
Filesize: 121,378,816

H263 + B-Frames 1 200 100:

Average PSNR (Y U V): 23.67, 38.88, 37.25
Total Average PSNR: 33.27
Filesize: 121,102,336

H263 + B-Frames 1 150 150:

Average PSNR (Y U V): 23.67, 38.88, 37.25
Total Average PSNR: 33.27
Filesize: 121,378,816

H263 + B-Frames 1 300 200:

Average PSNR (Y U V): 23.68, 38.89, 37.26
Total Average PSNR: 33.28
Filesize: 120,647,680

H263 + B-Frames 2 150 100:

Average PSNR (Y U V): 23.67, 38.88, 37.25
Total Average PSNR: 33.27
Fielsize: 121,405,440

H263 + B-Frames 6 150 100:

Average PSNR (Y U V): 23.67, 38.88, 37.25
Total Average PSNR: 33.27
Filesize: 121,415,680

H263 + B-Frames 1 150 100 + Luma + GMC + Chroma + Chroma Optimizer + Qpel:

Average PSNR (Y U V): 23.65, 38.71, 37.12
Total Average PSNR: 33.16
Filesize: 115,097,600

H263 + B-Frames 1 150 100 + Luma + VHQ4 + Chroma + Chroma Optimizer + Qpel:

Average PSNR (Y U V): 23.65, 38.68, 37.11
Total Average PSNR: 33.14
Filesize: 106,608,640

BHQ Matrix (No extra settings):

Average PSNR (Y U V): 44.31, 47.20, 46.83
Total Average PSNR: 46.11
Filesize: 119,066,624

BHC Matrix (No extra settings):

Average PSNR (Y U V): 40.09, 46.27, 45.59
Total Average PSNR: 43.98
Filesize: 93,372,416

MPEG Matrix (No extra settings):

Average PSNR (Y U V): 17.32, 30.26, 26.37
Total Average PSNR: 24.65
Filesize: 124,174,336


Facts about the research for this particular video:

1. Q-Pel increased the filesize by 5 megs and has lower PSNR by .30. :(

2. Luminance Masking took off 10 megs and only has a difference of .4 PSNR. :) :) :)

3. GMC increased the filesize by 200k but there is no difference on the Y plane (luma) PSNR, but the U plane has a better PSNR of .1 and the V plane has a lower PSNR of .1. :(

4. Chroma Motion made the PSNR better by .20 and took off 3 megs. :) :)

5. Chroma Optimizer increased the filesize by 100k but the PSNR is lower by .10. :(

6. VHQ1 made the PSNR better by .60 and took off 13 megs. :) :) :)

7. VHQ is a great addition, in all 4 modes the PSNR was always higher than the original H263 encode and has a lower filesize. :) :) :)

8. The default B-Frames setting took off only 2 megs but has a large PSNR drop of 13.10. However upon visual inspection, the change in quality did not seem that dramatic.

9. B-Frames with 200 100 was smaller by 200k yet has the exact same PSNR as the default B-frames setting. I did not visually inspect this.

10. Default settings with an offset of 150 just made an identical file that would've been made with 150 100 (default B-Frames setting).

11. High B-Frames settings at 300 200 is smaller by a meg versus the regular setting and has a higher psnr of .1. Upon visual inspection the video did actually seem to look better and it had a smaller filesize. :)

12. 2 B-frames with regular settings increased the size by 300k but no PSNR change. :(

13. 6 B-frames with regular setting increased the size by 300k but no PSNR change.

14. When using VHQ4 instead of GMC with all extra options enabled, it takes off 9 megs with only a PSNR difference of .2 versus using GMC. :) :) :)

15. My high quality matrix gets a lower PSNR rating of .26 versus H263, however upon visual inspection, the high quality matrix looks better, and the matrix made the filesize 4 megs smaller which surprised me. This matrix works good for all resolutions.

16. My heavy compression matrix lowered the filesize by 30 megs, the PSNR difference is 2.00 lower than H263 but upon visual inspection it does not look that bad and seemed very acceptable quality, but as I said, use this only for live action stuff with good motion. Also, I have only used this with constant quantizer of 2, it may not be wise to use this with high quantizers, and this matrix MUST be used on high resolution video, this means 640x480 and above, otherwise it will start to look really bad. I have not yet tested it on 16:9 resized videos.

17. The MPEG matrix increased the filesize by a meg and greatly reduced the PSNR, however upon visual inspection it doesnt look bad at all.

18. After the test I have realized that some of the B-frames tests are flawed. I am using a constant quantizer, therefore the B-frames cannot variate and be used to their advantage.


Here are the matrixes I made:

http://www.boomspeed.com/boya/BPMTX.rar

[ookzDVD]

Wow...

That's a huge test :) I hope developers can take advantage of your test results.

[BoNz1]

Thank you for your extensive testing Bulletproof, this will help me in deciding which settings to use. I am also quite eager to test your matrices. :)

[Gazza]

Bulletproof,

Just awesome. In tests I did, I noticed that Nic's build with qpel, mpeg, vhq=1, bframes 3/150/100, chrome motion produced noticeable 'pixellation' or very mobile artifacts (if you magnified the images). Did you see this in your tests?

[Blight]

Bulletproof:
Nice review... but... us drones ( :D ) don't like to think to much, so if you could end your review with: "for best quality and filesize ratio use the following settings" and "for good quality and minimal filesize use the following settings".

[CruNcher]

@ Bulletproof

Nice tests good work it's mirroring mostly the results i got :)
but i think you should also add that VHQ1 is faster around (3-4 fps) then Chroma Me. Qpel not only decreases PSNR it's also visible in the encode even with SimpleIdct used as in Nics, Koepis and many other Builds.
People @ the moment should be carefull with custom matrices @ least if they mainly useing ffdshow for playback, cause it looks like that with too high values ffdshow is produceing dequantization errors
can be seen with SofTMatrix by Trbarry, but this is no XviD problem its a libavc problem as it seems. Also with Chroma Optimizer the stepping effect gets reduced and Text in the Video Stream is also better rendered :)

[sysKin]

Thanks for great test, Bulletproof! I have a question though: how did you measure this psnr? I'm asking because of bframes results: you can't measure by decoding avi, either by vfw nor by directshow! Bframes will cause delay, and you will not compare encoded frames to original frames, but rather 'some frames' to 'some frames' and this will not work.

I'm also asking because I don't know any sensible way of computing psnr with bframes and I'd really like to do that.

Radek

[jpl]

To check PSNR with B-frames couldn't you just convert the XVID file with B-frames to a lossless HUFFYUV file and then do the PSNR calcs? Or would the change in color space change the results?

Thanks
JPL

[sysKin]

Originally posted by jpl
To check PSNR with B-frames couldn't you just convert the XVID file with B-frames to a lossless HUFFYUV file and then do the PSNR calcs? Or would the change in color space change the results?
Frames in huffyuv would still be delayed, so the clips still wouldn't be in sync...

[Blight]

How about converting to huffyuv using GraphEdit? Wouldn't the decoder filter compensate for the sync?

[Bulletproof]

Perhaps that is why the PSNR drop was so large when activating B-frames, and upon visual inspection the video did not seem bad at all. The program I used for PSNR inspection was by Vanguard Software, they are also the makers of their own H.264 codec, so there is a chance that somehow they did take B-frames into account, but I can't be certain, and from the current test it seems the PSNR dropped by 13.10, so I think it probably isn't doing proper comparison.

I suppose the Chroma Optimizer's PSNR is lower because of what Cruncher says, because its actually changing the picture so the PSNR result should be lower. To be honest, I don't really see any difference when putting the Chroma Optimizer on, but it did obviously do something to the picture if the PSNR is lower by .10.

[crusty]

Bulletproof could you tell us a bit more about the source ?
Is it high motion or low motion material ?
Is it a dark scene or a light scene ?
Is it easy-compressable or hard to compress stuff like dust and smoke?
Is it a clean source or is noisy?
Is it real-world video or is it animated ?

I mean it's hard to conclude anything about your tests if we don't know anything about the source material.

Different types of sources compress very differently, and the effect of a certain option can be an increase of filesize in one instance and a decrease of filesize on the other. Just think about the effect of GMC in clips with lots of panning and zooming.

[trbarry]

Bulletproof -

Great series of tests.

I guess we have to discount the b-frame results. I'm not sure you can trust any contstant quant tests to be fair if you vary either the quants or the matrices. For awhile I had myself convinced that the mpeg matrix was much worse than the H263 because of fixed quant tests. But that was partly an illusion since fixed quants are not fixed once you change the matrices.

For a pure numeric test accross matrices I guess now I only trust constant file size (bit rate) vs. PSNR.

But all this still seems like a big win for VHQ, which I know nothing about.

Does anybody happen to know if there is any decoding CPU speed penalty for VHQ?

- Tom

[Bulletproof]

Yes I think the B-frame tests should be ignored in this. About the source material, it is CG rendered material, I specifically chose a scene where there is High, low and medium motion. There is also landscapes (high detail) and also some flat shaded scenes. There is no dust in this video as it is really "digital" uncompressed video. If I remember correctly, GMC right now ignores panning and zooming because it is not complete yet. I believe VHQ is specifically in the encoding process and does not change the decoding in any way, however I did notice there is a large speed penalty in using VHQ during encoding.

[Gazza]

It looks as if VHQ provides some real advantage if you want to preserve as much quality as possible if you do 1 cd encodes or if the video is long (> 2hours?). With the average video length and a 2 cd encode do you really need to use VHQ?

trbarry - I can't say that VHQ in itself has impacted on cpu load, however I have noticed that when playing back tests produced from the latest versions of xvid (nic's and koepi's) cpu load is about 10 to 15% higher. I noticed the problem when recent tests are more likely to halt or pause because the processor (P3, 1GHz, 512Meg memory) is maxed out during replay. I normally test with bframes (3/150/100), qpel, vhq, chroma estimation, no filters except for Convolution3d and LanczosResize. Maybe these features are less buggy and performing closer to the spec but this is having a detrimental effect on playback?

[trbarry]

I previously already believed that qpel, GMC, and b frames had a CPU cost during playback. Since I encode mostly HDTV at resolutions like 1280x720 I've avoided these. My own practices are probably more limited by the playback than file size since I can fit most anything on a single DVD-R.

But I'd be willing to pay a slowdown penalty during encoding if it gave me better quality at the same bit rate, at least as long as it didn't make them harder to play. So it seems this VHQ might be worth exploring.

- Tom

[crusty]

@Bulletproof

Thanks for the clarification on the source. I guess I was thinking about Divx 5 when I was talking about gmc.

I would really like to see some test results on sources that are consisting of only one type of content, because I find it very hard to see the effect of one particular setting on different types of material. The forum also isn't very enlightening on this subject.

Take for instance a low-motion, very dark scene from Dark City and a high motion, very dark scene from the matrix (the intro fight would be a nice example I think) and then see what the effect is of individual settings.

Or compare a very clean source with a very noisy source (like an old B/W movie) and see the effects.

I find the results of different settings on combined source doesn't really say much because ppl out here want to use the best settings for a particular movie, which could be completely different from the a test clip containg a mixture of different motion-sources.
In general one can say that in many movies one type of motion will be more present than the others. And this offcourse differs completely from movie to movie.
Your test, although helpfull, really don't tell us a lot about that.

[PerCIVaL]

@Bulletproof

really nice tests, thank you

[Alex_e_Basta]

Hi all
I'm used to calculating psnr values using the Avisynth 2.0x Compare filter.
I noticed that Xvid files coded with B frames seem to be shifted 1 frame backward so,
if you compare in VDub the original VOB to the coded avi, the Xvid frame N is really the VOB's
frame N-1.
I don't know if this delay arises either from Xvid or Avisynth/VDub, likewise I don't know if this is a "real" delay or not
but for a correct psnr' evaluation, you need to take count of it.

I think this "delay" is constant but I may be wrong.

Here my avs script to compare a Divx with an Xvid encoding.

LoadPlugin("c:\Programmi\mpeg2dec.dll")
orig=mpeg2source("C:\Avisynth\PH.d2v").trim(0,800).crop(0,75,720,426)
encoded1=avisource("c:\Avisynth\PH 720x296 Divx be1.avi").BicubicResize(720,426,0,0.75)
encoded2=avisource("c:\Avisynth\PH 720x296 xvid be2.avi").trim(1,801).BicubicResize(720,426,0,0.75)
video1=compare(encoded1,orig,"","",False)
video2=compare(encoded2,orig,"","",False)
stackvertical(video1.BicubicResize(720,296,0,0.75),video2.BicubicResize(720,296,0,0.75))

As you can notice, I trim Xvid file 1,801 and the VOB/Divx 0,800; this is not needed if Xvid is coded without B frames.
Both encoding are Quality based 100%, Xvid Bframes = 2, no other stuff applied:
final average Divx 5.03 psnr(YUV) is 35.02, Xvid psnr is 35.03. ;)

Ciao
Alex

[Sigmatador]

Originally posted by Alex_e_Basta
I noticed that Xvid files coded with B frames seem to be shifted 1 frame
exactly, my first test give me the same bad result with b-frames than bulletproof. After shifting this frame, i got a better PSNR by .5db with b-frames ^^

Motion search 6 + H.263 + QPel + Chroma motion + Lumimasking + VHQ1 + Bframes 3/150/100
--> the best PSNR/File Size ratio i think...

BUT!
@BulletProof
your BHC matrix seems to be very interresting for a full 2 pass encode (with a big first pass size).

@trbarry&Gazza
i read the VHQ Manual and i don't understand why it can produce cpu penalty for decoding... only encoding i think... a "bruteforce" of frame coding, but coding process stay mpeg-4 compliant... (i am wrong ? lol did i understand something in the vhq manual :D :D )

[mf]

You do know that several features work on the human eye, while PSNR doesn't take that into account :). Chroma optimizer introduces invisible (well in 4:4:4 material invisble) errors which cause PSNR to drop, but increase percieved quality, because chroma subsampling (4:2:2) artifacts get reduced. However this is much more obvious at encodes which are 512x or lower. If you have ever noticed grey areas around black or white lines, chroma optimizer will fix it.

[Sigmatador]

the problem: sometimes chroma optmizer give me some strange purple bluring