Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 20th October 2003, 09:09   #1  |  Link
FastMike
Registered User
 
Join Date: May 2003
Posts: 20
[TEST] Rv9, Divx 51, H.264 R-D Plots

Ok guys,

I did some tests using the VQEG 'Mobile & Calendar' sequence. I deinterlaced it using Tom's MoComp, converted to YV12, and cropped it to 720x480. I encoded using all the slowest settings with all pre/post processing disabled. Each video followed the GOP structure IBPBPBP...

Target Bitrates (k): 256, 512, 768, 1500, 2000, 3000, 4000,6000

mobile_lg.png

Divx 5.1 2-Pass , slowest, no PyschoVisual

RealVideo 9 - 2-Pass, CBR, encoderComplexity=100, noisyEdgeFilter=false, MSL=60, patternAdaptivity=1, scalingFactor=20

H.264 - JM 6.1e, 5 Reference frames, CABAC, R-D on, Hadamard on. Since no rate control exists, Iframe QP = Pframe = Bframe QP - 2 (about 20% larger step size)

I would love to include WM9, but I don't know of an easy to way to perform command line encoding to ease batch encoding.

I am trying to keep this a fair comparison as possible, so please feel free to posts any questions, comments, and suggestions you might have. I will be posting more data as soon as I have it.
FastMike is offline   Reply With Quote
Old 20th October 2003, 10:51   #2  |  Link
temporance
Registered User
 
Join Date: Mar 2002
Posts: 486
I think this is an excellent testing methodology and for once I can't offer any constructive criticism. All I can suggest is to try the same test but with some different sequences.

Also I'd like to see WMV9 in the comparison, but I don't know any way to run it from command line other than to launch vdub with a pre-prepared job file.
temporance is offline   Reply With Quote
Old 20th October 2003, 12:38   #3  |  Link
bilu
Registered User
 
bilu's Avatar
 
Join Date: Oct 2002
Location: Portugal
Posts: 1,182
2-pass CBR on RV9 ???

What's 2-pass CBR? You probably meant VBR

And can you add XVID to it? I don't see the need to way for dev-api-4 to come out...

Bilu
bilu is offline   Reply With Quote
Old 20th October 2003, 13:14   #4  |  Link
Sirber
retired developer
 
Sirber's Avatar
 
Join Date: Oct 2002
Location: Canada
Posts: 8,978
I aggree with Bilu, why CBR??? Also, why doesn't all codecs begin where H264 begins? Also #2, use EHQ=85, it will be faster and have the same Q.
__________________
Detritus Software
Sirber is offline   Reply With Quote
Old 20th October 2003, 14:49   #5  |  Link
FastMike
Registered User
 
Join Date: May 2003
Posts: 20
Quote:
Originally posted by bilu
2-pass CBR on RV9 ???

What's 2-pass CBR? You probably meant VBR

And can you add XVID to it? I don't see the need to way for dev-api-4 to come out...

Bilu
Well its 'cbr' in the RV9 jobs file. There's an 'Analysis' pass so I called it 2-pass (which may be incorrect). I'll do a run with RV9 VBR, but I don't think there will be much of a difference.

I'll include Xvid well, using the Koepei's 24062003 build.

I will post more results as I have them. I would also like to include some HD sequences as well, but I have not found any good sources yet.
FastMike is offline   Reply With Quote
Old 20th October 2003, 15:05   #6  |  Link
bilu
Registered User
 
bilu's Avatar
 
Join Date: Oct 2002
Location: Portugal
Posts: 1,182
Helix Community link about encoding modes

Quote:
Video Stream Encoding Type

Selects between one of the following encoding types:

* cbr - Constant Bit Rate (CBR) based on avgBitrate only (maxBitrate and quality are ignored)
* vbrBitrate - Vaiable Bit Rate (VBR) based on avgBitrate and maxBitate (quality is ignored)
* vbrQuality - Vaiable Bit Rate (VBR) based on maxBitate and quality (avgBitrate is ignored)
* vbrUnconstrainedQuality - Vaiable Bit Rate (VBR) based on quality only (avgBitrate and maxBitate are ignored)
* vbrUnconstrainedBitrate - Vaiable Bit Rate (VBR) based on maxBitate only (avgBitrate and quality are ignored)

VBR encoding types provide higher quality but are not as well suited to low bit rate streaming situations and can cause network congestion due to spikes in bit rate when streaming. Use CBR for low bandwidth dial-up streaming, VBR constrained types for streaming at higher bit rates where network capacity is available, VBR unconstrained types for download. The quality versus bitrate types allow the output to be targeted towards a specified level of quality or a specified file size.



Default:
cbr

Values:
cbr, vbrBitrate, vbrQuality, vbrUnconstrainedQuality, or vbrUnconstrainedBitrate

Type:
string

Example:
<encodingType type="string">cbr</encodingType>
EDIT: Also 2-pass CBR is pointless in any codec. 2-pass process is used to generate statistics from one pass to another so you can spend more bits where needed. You don't have the option to choose how much bits to spend on CBR encodes


Bilu

Last edited by bilu; 20th October 2003 at 15:13.
bilu is offline   Reply With Quote
Old 20th October 2003, 16:47   #7  |  Link
FastMike
Registered User
 
Join Date: May 2003
Posts: 20
Quote:
Originally posted by bilu
EDIT: Also 2-pass CBR is pointless in any codec. 2-pass process is used to generate statistics from one pass to another so you can spend more bits where needed. You don't have the option to choose how much bits to spend on CBR encodes
Bilu [/B]
2-pass CBR is not pointless. The encoder can "spend bits" in a variety of ways, such as choosing an I,P,B frames, quantization level, intra-MB or inter-MB, etc. How much you spend on one frame will affect how many bits you need for a predictive frame that references that frame.

I believe VBR encoding is more of an advantage in longer clips with scene changes. All the VQEG clips I am using are mostly < 10 seconds long (~260-300 frames) without a scene change. Informal testing show's the VBR encoding does not change the quantization much from frame to frame (for a several codecs).

Actually the *main* reason I am using 2-pass encoding is that many codecs cannot meet the desired bitrate in CBR, especially with such a short video clip. Divx 5.0.5 often produced files *twice* the desired size, albeit Divx 5.1 is much more accurate in meeting the bitrate.
FastMike is offline   Reply With Quote
Old 20th October 2003, 17:04   #8  |  Link
bilu
Registered User
 
bilu's Avatar
 
Join Date: Oct 2002
Location: Portugal
Posts: 1,182
I guess you're right, although VBR is more related to reality.

But I think tests should use clips with 5 minutes at least, not 10 seconds. After all, VBR decisions should be a part of the benchmark as well.


Bilu
bilu is offline   Reply With Quote
Old 20th October 2003, 18:15   #9  |  Link
karl_lillevold
Moderator
 
karl_lillevold's Avatar
 
Join Date: Oct 2002
Location: Seattle
Posts: 1,584
@FastMike: interesting data, but I am afraid something appears to go quite wrong with your RV9 measurements. MobCal is one of the sequences I test with, incl measurements for H264 and DivX.

Looking at your parameters, please do not use scalingFactor or patternAdaptivity for RV9. Leave these at default, i.e. adaptive as recommended. scalingFactor = 20 is very suboptimal for MobCal, with B frame QP almost the same as P frame QP. RV9 step sizes are not the same as in MPEG-4.

Also, for a PSNR plot, I would recommend to use fixed Quality (QP) for all codecs, since rate control for a 10 second clip can be quite hazardous, and 1-pass, since 2-pass makes no sense for fixed Q, or for such a short clip, and then just plot the points on the curve as the bitrate turns out.

One last datapoint, how did you obtain RV9 PSNR numbers? If with Avisynth compare, did you make sure to take into account the one missing frame?
__________________
This information is provided "AS IS" with no warranties, grants no rights, and reflects my personal opinion.
karl_lillevold is offline   Reply With Quote
Old 20th October 2003, 19:40   #10  |  Link
karl_lillevold
Moderator
 
karl_lillevold's Avatar
 
Join Date: Oct 2002
Location: Seattle
Posts: 1,584
Sorry, I forgot one item:

Please specify maxPacketSize 16000 as well for RV9. This is required for good performance for such high bitrates that are needed for MobCal. If you use the latest Producer Milestone, it automatically sets large packetsizes, but I think only for VBR high bitrates. If a slightly older Producer is used, maxPacketSize always needs to be specified, so it is best to make sure 16000 is used by explicitly specifying it.
__________________
This information is provided "AS IS" with no warranties, grants no rights, and reflects my personal opinion.
karl_lillevold is offline   Reply With Quote
Old 20th October 2003, 20:20   #11  |  Link
FastMike
Registered User
 
Join Date: May 2003
Posts: 20
Karl, how did I figure you would have something to say about RV9?

* I was trying to keep the GOP structure the same for all codecs. I did test RV9 with scalingFactor and patternAdaptivity set to default values. I was shooting for a 20% higher step sizes for the B-frames, which I thought scalingFactor = 20 was achieving that?

* I obtained PSNR numbers from the rv9log.txt file. I haven't yet verified myself the numbers to be accurate.

I will look into using constant quality and setting the larger packet size.

Oh, and all the graphs don't start in the same place because Divx & RV9 don't seem to follow the intended bitrate at those low of levels. Hopefully using constant quality will help fix that.

All of the standard test sequences used in video coding research are all short. I am trying to utilize a method similiar to the MPEG Call for Proposals method (CfP) which does not allow rate control. I am trying to remove as many variables as possible, and rate control is a huge variable.
FastMike is offline   Reply With Quote
Old 22nd October 2003, 03:02   #12  |  Link
Prettz
easily bamboozled user
 
Prettz's Avatar
 
Join Date: Sep 2002
Location: Atlanta
Posts: 373
Why not add tests using one of the visual quality-based metrics in addition to PSNR. I would guess that PSNR is even more useless than normal when encoding at very low bitrates.
Prettz is offline   Reply With Quote
Old 22nd October 2003, 08:32   #13  |  Link
Animaniac
Registered User
 
Animaniac's Avatar
 
Join Date: Mar 2003
Location: Cambridge, MA
Posts: 286
Wow, I would have never expected the curves to be so smooth...
Animaniac is offline   Reply With Quote
Old 22nd October 2003, 18:00   #14  |  Link
Nibor
Registered User
 
Nibor's Avatar
 
Join Date: Apr 2003
Location: Switzerland
Posts: 140
Question:
Why are the PSNR values so low?
At 3'000 kB/s just 34?
Is it not average or overall PSNR?
Or am I missing something completely?

.:: Nibor ::.
__________________
Nobody but you... and me.
Nibor is offline   Reply With Quote
Old 22nd October 2003, 21:36   #15  |  Link
FastMike
Registered User
 
Join Date: May 2003
Posts: 20
As for why the PSNR values are so low, consider two things:

- The source material is 720x480, pretty much using the entire area.

- This sequence is known to be one of the harder sequences to compress

The PSNR is the average of all the PSNR's for each frame.

As for more metrics, I am not fully convinced they're worth anything yet. I will look into some, e.g. SSIM.
FastMike is offline   Reply With Quote
Old 23rd October 2003, 03:53   #16  |  Link
MfA
Registered User
 
Join Date: Mar 2002
Posts: 1,075
According to the original inventor SSIM correlated better with subjective quality testing than PSNR on the VQEG test set, with fewer outliers too so we can have a little faith in that being true in general, if SSIM is worth nothing PSNR is worth just as much
MfA is offline   Reply With Quote
Old 25th October 2003, 11:46   #17  |  Link
Sagittaire
Testeur de codecs
 
Sagittaire's Avatar
 
Join Date: May 2003
Location: France
Posts: 2,484
For Video
I make Test with RV9 EHQ, WMV9, DivX 5.10 and XviD Devapi with PSNR, SSIM and VQM. XviD and DivX are iso-compliant but not RV9 and WMV9. The best codecs are in order:

For animates
RV9 EHQ crushes everyone
WMV9
DivX 5.10
XviD Devapi4

For movies:
RV9 EHQ is the best
WMV9
DivX 5.10 Slowest 3 pass
XviD Devapi4

For Audio
The advanced audio codec are very similar for quality exepted MP3. but I make test with audio and for me the best codec are in order:

He-AAC 64~96
Vorbis 64~96
MP3Pro 64~96
RA8 Cook 64~96
WMV9 64~96
MP3 VBR 80~128 (with musicmatch jukebox encoder)

For Container
The best solution for container are:

1) RV9 EHQ / RA8 Cook / MKV

2) XviD Devapi4 / AAC / MP4
Menus integrated with Language and Sub selection but necessary envivio Plugin

3) WMV9 / WMA9 / MKV

4) DivX 5.10 / MP3 VBR / AVI
best hardware and software compatibility
__________________
Le Sagittaire ... ;-)

1- Ateme AVC or x264
2- VP7 or RV10 only for anime
3- XviD, DivX or WMV9

Last edited by Sagittaire; 25th October 2003 at 11:49.
Sagittaire is offline   Reply With Quote
Old 25th October 2003, 14:59   #18  |  Link
Gaia
Registered User
 
Join Date: Feb 2002
Posts: 267
I did my own test with my bran new 32" TV.

I encoded Matrix Reloaded with RV9 EHQ, latest DivX Beta and XviD api3. Used bitrate was 1200. I did short of blind test. RV9 was very easy to spot, it looked too blurry and kind of unnatural. Also RV9 EHQ is too slow for my old cpu. DivX and XviD looked both really good. Hard to say wich is better but because latest DivX beta is still too slow(if i use "best" settings") for my Duron so winner is XviD. Quality is very good and detailed. Speed is decent too.

This test proves me lot more than any mathematical crap.

I just wanted see what is best for me and my hardware. I am not saying XviD is the best codec for everybody.

Sagittaire your results are very strange. Especially audio and containers. You need to have lots of codecs installed if you want to watch RV9 mkv files. How you can say that's the best solution?
For normal PC users(i am not talking about geeks) avi+mp3 is still best solution. All you need to do is install ffdshow.

You and Sirber might be only people who says RA8 Cook 64~96 is a winner.

Everybody should forget these damn tests and use what suits best for their needs.

Last edited by Gaia; 25th October 2003 at 15:11.
Gaia is offline   Reply With Quote
Old 25th October 2003, 15:37   #19  |  Link
Sirber
retired developer
 
Sirber's Avatar
 
Join Date: Oct 2002
Location: Canada
Posts: 8,978
Quote:
Originally posted by Gaia
You and Sirber might be only people who says RA8 Cook 64~96 is a winner.
Was your test about Video or Audio codecs?
__________________
Detritus Software
Sirber is offline   Reply With Quote
Old 25th October 2003, 15:42   #20  |  Link
Lefungus
Registered User
 
Lefungus's Avatar
 
Join Date: Apr 2002
Location: Bretagne
Posts: 192
@Gaia
The mathematical crap as you said it, is not meant to replace subjective tests, but is an addendum to it. There are cases when you just can't watch everything and judge. That's where mathematical video metrics comes into play.
You made the right decision to test different codecs on your TV, and pick the best for your eyes, but stop the bashing on video metrics.

@Sagittaire
We're all glad you told us what is best. Unfortunately, you displayed it as if it was a general truth. Unfortunately, it isn't. RV9 EHQ doesn't crush everyone or whatever. I feel offended when you think so. Some may like the look of it, i personally find it "flat". Keep this in mind when you'll affirm something is "best" in your next post.
__________________

Some old avisynth filters
Lefungus is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 21:41.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.