[TEST] Rv9, Divx 51, H.264 R-D Plots

[FastMike]

Ok guys,

I did some tests using the VQEG 'Mobile & Calendar' sequence. I deinterlaced it using Tom's MoComp, converted to YV12, and cropped it to 720x480. I encoded using all the slowest settings with all pre/post processing disabled. Each video followed the GOP structure IBPBPBP...

Target Bitrates (k): 256, 512, 768, 1500, 2000, 3000, 4000,6000

mobile_lg.png (http://web.ics.purdue.edu/~migarta/mobile_lg.png)

Divx 5.1 2-Pass , slowest, no PyschoVisual

RealVideo 9 - 2-Pass, CBR, encoderComplexity=100, noisyEdgeFilter=false, MSL=60, patternAdaptivity=1, scalingFactor=20

H.264 - JM 6.1e, 5 Reference frames, CABAC, R-D on, Hadamard on. Since no rate control exists, Iframe QP = Pframe = Bframe QP - 2 (about 20% larger step size)

I would love to include WM9, but I don't know of an easy to way to perform command line encoding to ease batch encoding.

I am trying to keep this a fair comparison as possible, so please feel free to posts any questions, comments, and suggestions you might have. I will be posting more data as soon as I have it.

[temporance]

I think this is an excellent testing methodology and for once I can't offer any constructive criticism. All I can suggest is to try the same test but with some different sequences.

Also I'd like to see WMV9 in the comparison, but I don't know any way to run it from command line other than to launch vdub with a pre-prepared job file.

[bilu]

2-pass CBR on RV9 ???

What's 2-pass CBR? You probably meant VBR :)

And can you add XVID to it? I don't see the need to way for dev-api-4 to come out... :)

Bilu

[Sirber]

I aggree with Bilu, why CBR??? Also, why doesn't all codecs begin where H264 begins? Also #2, use EHQ=85, it will be faster and have the same Q.

[FastMike]

Originally posted by bilu
2-pass CBR on RV9 ???

What's 2-pass CBR? You probably meant VBR :)

And can you add XVID to it? I don't see the need to way for dev-api-4 to come out... :)

Bilu

Well its 'cbr' in the RV9 jobs file. There's an 'Analysis' pass so I called it 2-pass (which may be incorrect). I'll do a run with RV9 VBR, but I don't think there will be much of a difference.

I'll include Xvid well, using the Koepei's 24062003 build.

I will post more results as I have them. I would also like to include some HD sequences as well, but I have not found any good sources yet.

[bilu]

Helix Community link about encoding modes (https://producerapps.helixcommunity.org/cmdproducer/docs/AudienceFile.htm#encodingType-rn-realvideo)


Video Stream Encoding Type

Selects between one of the following encoding types:

* cbr - Constant Bit Rate (CBR) based on avgBitrate only (maxBitrate and quality are ignored)
* vbrBitrate - Vaiable Bit Rate (VBR) based on avgBitrate and maxBitate (quality is ignored)
* vbrQuality - Vaiable Bit Rate (VBR) based on maxBitate and quality (avgBitrate is ignored)
* vbrUnconstrainedQuality - Vaiable Bit Rate (VBR) based on quality only (avgBitrate and maxBitate are ignored)
* vbrUnconstrainedBitrate - Vaiable Bit Rate (VBR) based on maxBitate only (avgBitrate and quality are ignored)

VBR encoding types provide higher quality but are not as well suited to low bit rate streaming situations and can cause network congestion due to spikes in bit rate when streaming. Use CBR for low bandwidth dial-up streaming, VBR constrained types for streaming at higher bit rates where network capacity is available, VBR unconstrained types for download. The quality versus bitrate types allow the output to be targeted towards a specified level of quality or a specified file size.



Default:
cbr

Values:
cbr, vbrBitrate, vbrQuality, vbrUnconstrainedQuality, or vbrUnconstrainedBitrate

Type:
string

Example:
<encodingType type="string">cbr</encodingType>


EDIT: Also 2-pass CBR is pointless in any codec. 2-pass process is used to generate statistics from one pass to another so you can spend more bits where needed. You don't have the option to choose how much bits to spend on CBR encodes :o


Bilu

[FastMike]

Originally posted by bilu
EDIT: Also 2-pass CBR is pointless in any codec. 2-pass process is used to generate statistics from one pass to another so you can spend more bits where needed. You don't have the option to choose how much bits to spend on CBR encodes :o
Bilu [/B]

2-pass CBR is not pointless. The encoder can "spend bits" in a variety of ways, such as choosing an I,P,B frames, quantization level, intra-MB or inter-MB, etc. How much you spend on one frame will affect how many bits you need for a predictive frame that references that frame.

I believe VBR encoding is more of an advantage in longer clips with scene changes. All the VQEG clips I am using are mostly < 10 seconds long (~260-300 frames) without a scene change. Informal testing show's the VBR encoding does not change the quantization much from frame to frame (for a several codecs).

Actually the *main* reason I am using 2-pass encoding is that many codecs cannot meet the desired bitrate in CBR, especially with such a short video clip. Divx 5.0.5 often produced files *twice* the desired size, albeit Divx 5.1 is much more accurate in meeting the bitrate.

[bilu]

I guess you're right, although VBR is more related to reality.

But I think tests should use clips with 5 minutes at least, not 10 seconds. After all, VBR decisions should be a part of the benchmark as well.


Bilu

[karl_lillevold]

@FastMike: interesting data, but I am afraid something appears to go quite wrong with your RV9 measurements. MobCal is one of the sequences I test with, incl measurements for H264 and DivX.

Looking at your parameters, please do not use scalingFactor or patternAdaptivity for RV9. Leave these at default, i.e. adaptive as recommended. scalingFactor = 20 is very suboptimal for MobCal, with B frame QP almost the same as P frame QP. RV9 step sizes are not the same as in MPEG-4.

Also, for a PSNR plot, I would recommend to use fixed Quality (QP) for all codecs, since rate control for a 10 second clip can be quite hazardous, and 1-pass, since 2-pass makes no sense for fixed Q, or for such a short clip, and then just plot the points on the curve as the bitrate turns out.

One last datapoint, how did you obtain RV9 PSNR numbers? If with Avisynth compare, did you make sure to take into account the one missing frame?

[karl_lillevold]

Sorry, I forgot one item:

Please specify maxPacketSize 16000 as well for RV9. This is required for good performance for such high bitrates that are needed for MobCal. If you use the latest Producer Milestone, it automatically sets large packetsizes, but I think only for VBR high bitrates. If a slightly older Producer is used, maxPacketSize always needs to be specified, so it is best to make sure 16000 is used by explicitly specifying it.

[FastMike]

Karl, how did I figure you would have something to say about RV9? :)

* I was trying to keep the GOP structure the same for all codecs. I did test RV9 with scalingFactor and patternAdaptivity set to default values. I was shooting for a 20% higher step sizes for the B-frames, which I thought scalingFactor = 20 was achieving that?

* I obtained PSNR numbers from the rv9log.txt file. I haven't yet verified myself the numbers to be accurate. :)

I will look into using constant quality and setting the larger packet size.

Oh, and all the graphs don't start in the same place because Divx & RV9 don't seem to follow the intended bitrate at those low of levels. Hopefully using constant quality will help fix that.

All of the standard test sequences used in video coding research are all short. I am trying to utilize a method similiar to the MPEG Call for Proposals method (CfP) which does not allow rate control. I am trying to remove as many variables as possible, and rate control is a huge variable.

[Prettz]

Why not add tests using one of the visual quality-based metrics in addition to PSNR. I would guess that PSNR is even more useless than normal when encoding at very low bitrates.

[Animaniac]

Wow, I would have never expected the curves to be so smooth...

[Nibor]

Question:
Why are the PSNR values so low?
At 3'000 kB/s just 34?
Is it not average or overall PSNR?
Or am I missing something completely? :confused:

.:: Nibor ::.

[FastMike]

As for why the PSNR values are so low, consider two things:

- The source material is 720x480, pretty much using the entire area.

- This sequence is known to be one of the harder sequences to compress

The PSNR is the average of all the PSNR's for each frame.

As for more metrics, I am not fully convinced they're worth anything yet. :) I will look into some, e.g. SSIM.

[MfA]

According to the original inventor SSIM correlated better with subjective quality testing than PSNR on the VQEG test set, with fewer outliers too so we can have a little faith in that being true in general, if SSIM is worth nothing PSNR is worth just as much :)

[Sagittaire]

For Video
I make Test with RV9 EHQ, WMV9, DivX 5.10 and XviD Devapi with PSNR, SSIM and VQM. XviD and DivX are iso-compliant but not RV9 and WMV9. The best codecs are in order:

For animates
RV9 EHQ crushes everyone
WMV9
DivX 5.10
XviD Devapi4

For movies:
RV9 EHQ is the best
WMV9
DivX 5.10 Slowest 3 pass
XviD Devapi4

For Audio
The advanced audio codec are very similar for quality exepted MP3. but I make test with audio and for me the best codec are in order:

He-AAC 64~96
Vorbis 64~96
MP3Pro 64~96
RA8 Cook 64~96
WMV9 64~96
MP3 VBR 80~128 (with musicmatch jukebox encoder)

For Container
The best solution for container are:

1) RV9 EHQ / RA8 Cook / MKV

2) XviD Devapi4 / AAC / MP4
Menus integrated with Language and Sub selection but necessary envivio Plugin

3) WMV9 / WMA9 / MKV

4) DivX 5.10 / MP3 VBR / AVI
best hardware and software compatibility

[Gaia]

I did my own test with my bran new 32" TV.

I encoded Matrix Reloaded with RV9 EHQ, latest DivX Beta and XviD api3. Used bitrate was 1200. I did short of blind test. RV9 was very easy to spot, it looked too blurry and kind of unnatural. Also RV9 EHQ is too slow for my old cpu. DivX and XviD looked both really good. Hard to say wich is better but because latest DivX beta is still too slow(if i use "best" settings") for my Duron so winner is XviD. Quality is very good and detailed. Speed is decent too.

This test proves me lot more than any mathematical crap.

I just wanted see what is best for me and my hardware. I am not saying XviD is the best codec for everybody.

Sagittaire your results are very strange. Especially audio and containers. You need to have lots of codecs installed if you want to watch RV9 mkv files. How you can say that's the best solution?
For normal PC users(i am not talking about geeks) avi+mp3 is still best solution. All you need to do is install ffdshow.

You and Sirber might be only people who says RA8 Cook 64~96 is a winner.

Everybody should forget these damn tests and use what suits best for their needs.

[Sirber]

Originally posted by Gaia
You and Sirber might be only people who says RA8 Cook 64~96 is a winner.Was your test about Video or Audio codecs? :p

[Lefungus]

@Gaia
The mathematical crap as you said it, is not meant to replace subjective tests, but is an addendum to it. There are cases when you just can't watch everything and judge. That's where mathematical video metrics comes into play.
You made the right decision to test different codecs on your TV, and pick the best for your eyes, but stop the bashing on video metrics.

@Sagittaire
We're all glad you told us what is best. Unfortunately, you displayed it as if it was a general truth. Unfortunately, it isn't. RV9 EHQ doesn't crush everyone or whatever. I feel offended when you think so. Some may like the look of it, i personally find it "flat". Keep this in mind when you'll affirm something is "best" in your next post.

[Sagittaire]

@ Lefungus

RV9 EHQ are best in PSNR, SSIM and VQM test ... it's mathematical test and better is high. RV9 obtain the best result for this tests ...

I find too visualy the RV9 EHQ is better than iso-MPEG4 but it's my opignon specially for animate ...

@ Gaia

I also made the encoding of Matrix Reloaded, Hulk and XMen II in french & english with sub on XCD of 80 min and RV9 EHQ is by far that which gives the best results for these movies ...

RA8 Cook isn't best audio codec but RV9 EHQ / RA8 Cook Surround / SRT / MKV are the best solution for this movies ...

In fact the very best and powerfull solution is RV9 EHQ / He-AAC / SSA / MKV but it's a very complexe solution ...

With 1200 Kbps all codec give very good results (divx3 SBC with ffvfw too) and make a comparaison with this bitrate is useless

[Gaia]

@ Lefungus

I was not bashing any testing methods. Like i already said, i wanted to try what is best for me in real life use. What i mean there are many mathematical tests in this board already and they are all different. Compare CruNcher's test and Sagittaire's tests. Who is right who is wrong?

@ Sagittaire

Like Lefungus said, those are just your personal opinions. Not ultimate truth. Ofcourse RV9 EHQ is a winner in low bitrate encodes but so? Everybody knows it...I don't do anymore low bitrate encodes and i always keep original ac3 audio. I can't understand why somebody want's to encode original 5.1 to lower bitrate 5.1 audio? It's just stupid. I sounds bad if you have good equipment.

[karl_lillevold]

at 1200 kbps RV9 would look its best at full anamorphic resolution, the native DVD resolution. If you used 640x272, this corresponds to 0.29 bits/pixel. Anamorphic 720x360 (or so) encoded rez, stretched to 860x360 on playback, would have corresponded to 0.19 bits/pixel, vhere RV9-EHQ would have done really well at preserving the original resolution.

On a 32" TV, encoded videos tend to look very different than on high resolution computer displays or HDTVs, and there is probably not even a benefit to anamorphic encoding, since the TV is more or less limited to 640 pixels wide. MPEG-4 ringing lends a certain sharpness to the experience, so for very high bitrates, targeted for smallish TV playback, that's probably your best solution then. As always, there is no answer to what is best for every scenario.

That said, we have three small improvements coming out shortly, a VBR 2-pass improvement to give more bits to high action scenes, an encoder improvement to preserve high frequencies better, and a decoder post-"filter" that does something surprising.

@Gaia: AC3 is not a very efficient compression method, and it is possible to preserve the original quality very well, at a lower bitrate. I have not done the listening tests required to find exactly which bitrate this is though. Again it all depends on the target playback mechanism and one's sensitivity to any degradation in the soundtrack quality.

[HomiE FR]

@ Sagittaire

I really agree with other people here: your use of the words "best", "better" and the likes is getting on my nerves. Everytime I read those posts about tests where "RV9 EHQ is best..." and it's meaningless : even though tests are here to have some "mathematical" facts about video compression, it doesn't mean you can use them as absolute truth to say that something IS best.

I only read "in my opinion" in some cases, when people told you to stop those "best"/"better"/... posts. So please stop these posts, I could as well ignore these threads but I believe that such statements can be confusing for "newbies".

Thanks in advance.

[Sagittaire]

Sorry but my english is limited ... ;-)

For video codec test
But for tests PSNR, SSIM and VQM it is very simple ... it is enough to make the tests and to compare the values. I don't know if these tests are representative but in any case they are used by all the developers of codecs and a thing is sure with these tests in fact the RV9 EHQ is most powerful, then the WMV9, DivX 5.10 and finally XviD Devapi4 ... and in more they correspond so that I observe visually ... it's all ...

For audio test it's here
http://audio.ciara.us/test/64test/results.html

For Container tests
I am making demo for my site

RV9 / RA8 / MKV
WMV9 / WMA9 / MKV
XviD / AAC / MP4

I prefer personally XviD/AAC/MP4 with integrated menu but that who am most powerful is well the RV9/RA8/MKV with which have will be able to also make integrated menus if the spec are respected and to put sub directly at the format vobsub if gabest moves this bottom ... ;-)

[bilu]

Originally posted by karl_lillevold
That said, we have three small improvements coming out shortly, a VBR 2-pass improvement to give more bits to high action scenes, an encoder improvement to preserve high frequencies better, and a decoder post-"filter" that does something surprising.

Do you mean audio or video when talking about high frequencies?
I'm looking forward to see some action ;)
And if the post-filter adds grain the XVID crowd will love it :D


Bilu

[CruNcher]

Originally posted by karl_lillevold
That said, we have three small improvements coming out shortly, a VBR 2-pass improvement to give more bits to high action scenes, an encoder improvement to preserve high frequencies better, and a decoder post-"filter" that does something surprising.

:eek: that sounds really good that could make me indeed, happy :D
especialy the "improvement to preserve high frequencies better" im looking forward to it. Bilu nah no ugly grain tricks thats odd :P

[Sirber]

What is high frequency for video? :confused:

[FastMike]

Originally posted by Sirber
What is high frequency for video? :confused:

High frequency = high spatial frequency = fast changing parts of the image

Examples: object edges, areas of high detail, etc.

Low frequency stuff would be like a slowly fading sky, etc.

I'm really curious to see these new RV9 features.

Ok, while I'm here:

* Human visual system (HVS) is more sensitive to lower frequency parts of the image than higher frequency parts

* The DCT (Discrete Cosine Transform) seperates the image into its different frequency parts

* "MPEG" type quantization uses heavier quantization (heavy rounding or less precision) for the higher frequency information

* "H.263" quantization uses the same quantization (same precision/weighting) for low and high frequency information.

[MfA]

Originally posted by FastMike
* Human visual system (HVS) is more sensitive to lower frequency parts of the image than higher frequency parts

I think that is debatable (of course I would given that Im a SSIM fan). I would personally phrase that differently, our visual system is usually less sensitive to quantization error in the high frequency components than in the low frequency components. That is more because of how those types of errors usually relate to errors in the spatial domain than because the HVS really internally uses a localized frequency domain.

It is very tempting to try to capture a HVS model in nice seperable channels ... because it is so elegant and simple. I think it is a trap though.

[bilu]

Originally posted by CruNcher
Bilu nah no ugly grain tricks thats odd :P

I know, but when you hear some MPEG-4 veterans opinion about RV9 you get the impression that grain is what they're missing. :)

Bilu

[Sirber]

Maybe they are missing all the squares and rigning.. ;)

[Lefungus]

Originally posted by Sirber
Maybe they are missing all the squares and rigning.. ;)

Next time, just don't post, as you aren't tolerant when someone jokes the other way, about rv9...

@Karl
I'm glad some improvement are coming for rv9, as it's still a very good codec.
The "grain" of the video is very important in my opinion. Not an artificial one, but the grain created by the 35mm process. Some directors spend a lot of time to ensure the DVD compression won't destroy it, so a codec should "try" to keep it too.

[bilu]

@Lefungus

I've seen some new AVS filters around that claim to have a very good grain effect. If that's true and can be used in realtime post-processing then we can have both compression and grain effect, I think :)


Bilu

[Lefungus]

@Bilu I'm currently testing that :)

[Sirber]

What would be the best is to separate the grain from the main video and encode it in another stream that will play on top of the main video on playback.

[bilu]

??? :confused:
No need to encode grain on another layer, it can be easily generated on realtime.


Bilu

[Sirber]

Generated with infos from the source saved in the bitstream? Like SBR... :D

[MfA]

You can do more than just synthesizing grain, you can go as far as synthesizing fine texture (http://www.google.com/url?sa=U&start=1&q=http://dewww.epfl.ch/~nadenau/Research/Pub_12.htm&e=747).

The problem with that is that grain is motion independent, and that sort of stuff isnt.

[hubereevez]

hi,

In fact the very best and powerfull solution is RV9 EHQ / He-AAC / SSA / MKV but it's a very complexe solution ...

No it's not......And this produce a real good quality mkv. Excepet the ssa subs, mpc doesn't permit to resize them well (e.g. increase font)....it's not mkv's fault....

ciya

[Sirber]

In fact the very best and powerfull solution is RV9 EHQ / He-AAC / SSA / MKV but it's a very complexe solution ...Surreal UI will be able to produce that automaticly from DVD or AVI. I just need to finish it :rolleyes: