Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
20th October 2003, 09:09 | #1 | Link |
Registered User
Join Date: May 2003
Posts: 20
|
[TEST] Rv9, Divx 51, H.264 R-D Plots
Ok guys,
I did some tests using the VQEG 'Mobile & Calendar' sequence. I deinterlaced it using Tom's MoComp, converted to YV12, and cropped it to 720x480. I encoded using all the slowest settings with all pre/post processing disabled. Each video followed the GOP structure IBPBPBP... Target Bitrates (k): 256, 512, 768, 1500, 2000, 3000, 4000,6000 mobile_lg.png Divx 5.1 2-Pass , slowest, no PyschoVisual RealVideo 9 - 2-Pass, CBR, encoderComplexity=100, noisyEdgeFilter=false, MSL=60, patternAdaptivity=1, scalingFactor=20 H.264 - JM 6.1e, 5 Reference frames, CABAC, R-D on, Hadamard on. Since no rate control exists, Iframe QP = Pframe = Bframe QP - 2 (about 20% larger step size) I would love to include WM9, but I don't know of an easy to way to perform command line encoding to ease batch encoding. I am trying to keep this a fair comparison as possible, so please feel free to posts any questions, comments, and suggestions you might have. I will be posting more data as soon as I have it. |
20th October 2003, 10:51 | #2 | Link |
Registered User
Join Date: Mar 2002
Posts: 486
|
I think this is an excellent testing methodology and for once I can't offer any constructive criticism. All I can suggest is to try the same test but with some different sequences.
Also I'd like to see WMV9 in the comparison, but I don't know any way to run it from command line other than to launch vdub with a pre-prepared job file. |
20th October 2003, 13:14 | #4 | Link |
retired developer
Join Date: Oct 2002
Location: Canada
Posts: 8,978
|
I aggree with Bilu, why CBR??? Also, why doesn't all codecs begin where H264 begins? Also #2, use EHQ=85, it will be faster and have the same Q.
__________________
Detritus Software |
20th October 2003, 14:49 | #5 | Link | |
Registered User
Join Date: May 2003
Posts: 20
|
Quote:
I'll include Xvid well, using the Koepei's 24062003 build. I will post more results as I have them. I would also like to include some HD sequences as well, but I have not found any good sources yet. |
|
20th October 2003, 15:05 | #6 | Link | |
Registered User
Join Date: Oct 2002
Location: Portugal
Posts: 1,182
|
Helix Community link about encoding modes
Quote:
Bilu Last edited by bilu; 20th October 2003 at 15:13. |
|
20th October 2003, 16:47 | #7 | Link | |
Registered User
Join Date: May 2003
Posts: 20
|
Quote:
I believe VBR encoding is more of an advantage in longer clips with scene changes. All the VQEG clips I am using are mostly < 10 seconds long (~260-300 frames) without a scene change. Informal testing show's the VBR encoding does not change the quantization much from frame to frame (for a several codecs). Actually the *main* reason I am using 2-pass encoding is that many codecs cannot meet the desired bitrate in CBR, especially with such a short video clip. Divx 5.0.5 often produced files *twice* the desired size, albeit Divx 5.1 is much more accurate in meeting the bitrate. |
|
20th October 2003, 17:04 | #8 | Link |
Registered User
Join Date: Oct 2002
Location: Portugal
Posts: 1,182
|
I guess you're right, although VBR is more related to reality.
But I think tests should use clips with 5 minutes at least, not 10 seconds. After all, VBR decisions should be a part of the benchmark as well. Bilu |
20th October 2003, 18:15 | #9 | Link |
Moderator
Join Date: Oct 2002
Location: Seattle
Posts: 1,584
|
@FastMike: interesting data, but I am afraid something appears to go quite wrong with your RV9 measurements. MobCal is one of the sequences I test with, incl measurements for H264 and DivX.
Looking at your parameters, please do not use scalingFactor or patternAdaptivity for RV9. Leave these at default, i.e. adaptive as recommended. scalingFactor = 20 is very suboptimal for MobCal, with B frame QP almost the same as P frame QP. RV9 step sizes are not the same as in MPEG-4. Also, for a PSNR plot, I would recommend to use fixed Quality (QP) for all codecs, since rate control for a 10 second clip can be quite hazardous, and 1-pass, since 2-pass makes no sense for fixed Q, or for such a short clip, and then just plot the points on the curve as the bitrate turns out. One last datapoint, how did you obtain RV9 PSNR numbers? If with Avisynth compare, did you make sure to take into account the one missing frame?
__________________
This information is provided "AS IS" with no warranties, grants no rights, and reflects my personal opinion. |
20th October 2003, 19:40 | #10 | Link |
Moderator
Join Date: Oct 2002
Location: Seattle
Posts: 1,584
|
Sorry, I forgot one item:
Please specify maxPacketSize 16000 as well for RV9. This is required for good performance for such high bitrates that are needed for MobCal. If you use the latest Producer Milestone, it automatically sets large packetsizes, but I think only for VBR high bitrates. If a slightly older Producer is used, maxPacketSize always needs to be specified, so it is best to make sure 16000 is used by explicitly specifying it.
__________________
This information is provided "AS IS" with no warranties, grants no rights, and reflects my personal opinion. |
20th October 2003, 20:20 | #11 | Link |
Registered User
Join Date: May 2003
Posts: 20
|
Karl, how did I figure you would have something to say about RV9?
* I was trying to keep the GOP structure the same for all codecs. I did test RV9 with scalingFactor and patternAdaptivity set to default values. I was shooting for a 20% higher step sizes for the B-frames, which I thought scalingFactor = 20 was achieving that? * I obtained PSNR numbers from the rv9log.txt file. I haven't yet verified myself the numbers to be accurate. I will look into using constant quality and setting the larger packet size. Oh, and all the graphs don't start in the same place because Divx & RV9 don't seem to follow the intended bitrate at those low of levels. Hopefully using constant quality will help fix that. All of the standard test sequences used in video coding research are all short. I am trying to utilize a method similiar to the MPEG Call for Proposals method (CfP) which does not allow rate control. I am trying to remove as many variables as possible, and rate control is a huge variable. |
22nd October 2003, 18:00 | #14 | Link |
Registered User
Join Date: Apr 2003
Location: Switzerland
Posts: 140
|
Question:
Why are the PSNR values so low? At 3'000 kB/s just 34? Is it not average or overall PSNR? Or am I missing something completely? .:: Nibor ::.
__________________
Nobody but you... and me. |
22nd October 2003, 21:36 | #15 | Link |
Registered User
Join Date: May 2003
Posts: 20
|
As for why the PSNR values are so low, consider two things:
- The source material is 720x480, pretty much using the entire area. - This sequence is known to be one of the harder sequences to compress The PSNR is the average of all the PSNR's for each frame. As for more metrics, I am not fully convinced they're worth anything yet. I will look into some, e.g. SSIM. |
23rd October 2003, 03:53 | #16 | Link |
Registered User
Join Date: Mar 2002
Posts: 1,075
|
According to the original inventor SSIM correlated better with subjective quality testing than PSNR on the VQEG test set, with fewer outliers too so we can have a little faith in that being true in general, if SSIM is worth nothing PSNR is worth just as much
|
25th October 2003, 11:46 | #17 | Link |
Testeur de codecs
Join Date: May 2003
Location: France
Posts: 2,484
|
For Video
I make Test with RV9 EHQ, WMV9, DivX 5.10 and XviD Devapi with PSNR, SSIM and VQM. XviD and DivX are iso-compliant but not RV9 and WMV9. The best codecs are in order: For animates RV9 EHQ crushes everyone WMV9 DivX 5.10 XviD Devapi4 For movies: RV9 EHQ is the best WMV9 DivX 5.10 Slowest 3 pass XviD Devapi4 For Audio The advanced audio codec are very similar for quality exepted MP3. but I make test with audio and for me the best codec are in order: He-AAC 64~96 Vorbis 64~96 MP3Pro 64~96 RA8 Cook 64~96 WMV9 64~96 MP3 VBR 80~128 (with musicmatch jukebox encoder) For Container The best solution for container are: 1) RV9 EHQ / RA8 Cook / MKV 2) XviD Devapi4 / AAC / MP4 Menus integrated with Language and Sub selection but necessary envivio Plugin 3) WMV9 / WMA9 / MKV 4) DivX 5.10 / MP3 VBR / AVI best hardware and software compatibility
__________________
Le Sagittaire ... ;-) 1- Ateme AVC or x264 2- VP7 or RV10 only for anime 3- XviD, DivX or WMV9 Last edited by Sagittaire; 25th October 2003 at 11:49. |
25th October 2003, 14:59 | #18 | Link |
Registered User
Join Date: Feb 2002
Posts: 267
|
I did my own test with my bran new 32" TV.
I encoded Matrix Reloaded with RV9 EHQ, latest DivX Beta and XviD api3. Used bitrate was 1200. I did short of blind test. RV9 was very easy to spot, it looked too blurry and kind of unnatural. Also RV9 EHQ is too slow for my old cpu. DivX and XviD looked both really good. Hard to say wich is better but because latest DivX beta is still too slow(if i use "best" settings") for my Duron so winner is XviD. Quality is very good and detailed. Speed is decent too. This test proves me lot more than any mathematical crap. I just wanted see what is best for me and my hardware. I am not saying XviD is the best codec for everybody. Sagittaire your results are very strange. Especially audio and containers. You need to have lots of codecs installed if you want to watch RV9 mkv files. How you can say that's the best solution? For normal PC users(i am not talking about geeks) avi+mp3 is still best solution. All you need to do is install ffdshow. You and Sirber might be only people who says RA8 Cook 64~96 is a winner. Everybody should forget these damn tests and use what suits best for their needs. Last edited by Gaia; 25th October 2003 at 15:11. |
25th October 2003, 15:37 | #19 | Link | |
retired developer
Join Date: Oct 2002
Location: Canada
Posts: 8,978
|
Quote:
__________________
Detritus Software |
|
25th October 2003, 15:42 | #20 | Link |
Registered User
Join Date: Apr 2002
Location: Bretagne
Posts: 192
|
@Gaia
The mathematical crap as you said it, is not meant to replace subjective tests, but is an addendum to it. There are cases when you just can't watch everything and judge. That's where mathematical video metrics comes into play. You made the right decision to test different codecs on your TV, and pick the best for your eyes, but stop the bashing on video metrics. @Sagittaire We're all glad you told us what is best. Unfortunately, you displayed it as if it was a general truth. Unfortunately, it isn't. RV9 EHQ doesn't crush everyone or whatever. I feel offended when you think so. Some may like the look of it, i personally find it "flat". Keep this in mind when you'll affirm something is "best" in your next post. |
|
|