[20-06-2005] Metric Benchmark Challenge ...

[CruNcher]

im useing mplayer to get the correct bitrate

My .d2v is an old project (DVD2AVI 1.76) but isn't a big problem : one or two frames droped ...


that's a joke right ? every of us should use the same or this here is useless

[Sagittaire]

Originally posted by CruNcher
im useing mplayer to get the correct bitrate


that's a joke right ? every of us should use the same or this here is useless

no an error ... I use the last DGDecode.dll with last dgindex project version ... lol

In my previous test (matrix reloaded) I use DVD2AVI 1.76 project ... tired when i write this line ... lol

And now i'm tired by this size/overhead problem ...

I work Thursday and Friday and I resolve this problem during the weekend ...

[sparky]

My DivX6 results:

For 450 kbps target:
-bvn1 457000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2
-bvnn 457000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2 -complexity_modulation=0.1
7104512 bytes = 6938 kb
SSIM 65.13
OPSNR 40.6269
APSNR 41.4548

For 900 kbps target:
-bvn1 919000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2
-bvnn 919000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2 -complexity_modulation=0.1
14206976 bytes = 13874 kb
SSIM 78.34
OPSNR 43.5476
APSNR 44.3324

Feel free to retest

[Sagittaire]

Originally posted by sparky
My DivX6 results:

For 450 kbps target:
-bvn1 457000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2
-bvnn 457000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2 -complexity_modulation=0.1
7104512 bytes = 6938 kb
SSIM 65.13
OPSNR 40.6269
APSNR 41.4548

For 900 kbps target:
-bvn1 919000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2
-bvnn 919000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2 -complexity_modulation=0.1
14206976 bytes = 13874 kb
SSIM 78.34
OPSNR 43.5476
APSNR 44.3324

Feel free to retest

:cool:

uptade this weekend

[iapir]

I agree the overhead issue is probably very short compared to the differences in size allocation by each codec. But it's always better to compare what can be compared.

If a codec uses only I frames and a codec like AVC uses complex references, maybe the data size will be the same. But the storage space needed will be different on the container side. That's why I think it should be taken in account too. Since in the end you won't use raw data files but files in a container...

[PatchWorKs]

What about SNOW ? And Theora ? Any1 ?

[stephanV]

I agree the overhead issue is probably very short compared to the differences in size allocation by each codec. But it's always better to compare what can be compared.

The problem is, I can mux things in AVI, Matroska and MP4 with different apps and get different values for overhead, and then i didnt even mention changing the settings.

Should AVC be used in MP4 for this test? What if a lot of people would want to use it with AC3 or Vorbis? Would the test become invalid for them?

Just compare raw data.

[Manao]

StephanV : i doubt anyone can compute psnr from a raw h264 stream. Moreover, it's hard to get raw data size for some codecs. I still think that choosing the container & tha application that minimizes the filesize for a particular codec is still the way to go.

And if each codecs are tested with their 'optimal' container, somehow, i guess the overhead won't be that different.

PatchWorKs: i too would be interested in Snow's results. However, the fact that snow in ffdshow has been broken for a long time now doesn't help testing it. As for theora, it really would hurt it, i think, if you were to make a comparison with the codecs used here.

[stephanV]

Originally posted by Manao
StephanV : i doubt anyone can compute psnr from a raw h264 stream.
But that isnt necessary, you only need the raw data to determine the size of the video stream. I think the container would not influence the PSNR right?

Moreover, it's hard to get raw data size for some codecs. I still think that choosing the container & tha application that minimizes the filesize for a particular codec is still the way to go.

And if each codecs are tested with their 'optimal' container, somehow, i guess the overhead won't be that different.
I would advice everyone to use AVIMux GUI. (matroska, set video lacing to 5 frames. :) )

[Sagittaire]

The problem is: container is part of "codec" ?

It's true : the efficacity of ovehead container is not user problem ... it's developper problem. Developper must use the best possible container ...

It's wrong : codec can or must use different container for X or Y reason. AVC use mp4, avi or mkv ; VP6 use avi and .vp6 ... for example.

Like say Manao use the best possible for each codec is perhabs good solution ... but for information I will indicate the real bitrate of video ES. And perhabs size overhead is not very different (I hope).

[Tommy Carrot]

Originally posted by PatchWorKs
What about SNOW ? I tried, and it looks like it's finally not borked in the latest ffdshow, but due to the lack of ratecontrol, i could only get the bitrate reasonably close to the target, but it's impossible to force the filesize exactly to the target range, so those encodes are not really relevant. But based on them i would say that Snow has slightly higher PSNR than the ASP codecs, but lower than the h.264 codecs.

About the container issue: simply use the container which is most widely used with the given codec (in other words leave them as they are now), that would give the most meaningful results IMO (as far as metric measures can be meaningful anyway).

[stephanV]

does anyone have a 3ivx Beta he/she is allowed to try? Their official codec from the site doesnt even outperform NanDub...

[IgorC]

AutoSSIM from italian Doom9 http://forum.doom9.it/download.php?id=853&sid=422a2f090dc3ae25098cf232eaab8e54. it very comfortable

P.S. It supports only MPEG1(2)

[Sagittaire]

Originally posted by Manao
Btw, you praise psnr overall over psnr average when it comes to rate control comparison, however ssim is computed as average psnr is, so ssim should only be used to compare frames together, not to compare rate control algorithms.



|---------------|---------|---------|
| Variability | OPSNR | SSIM |
|---------------|---------|---------|
| 000% | 39.3871 | 64.05 |
| 010% | 39.6695 | 65.23 |
| 020% | 39.9377 | 66.31 |
| 030% | 40.1266 | 67.08 |
| 040% | 40.2793 | 67.70 |
| 050% | 40.3925 | 68.19 |
| 060% | 40.4644 | 68.50 |
| 070% | 40.5003 | 68.66 |
| 080% | 40.4851 | 68.63 |
| 090% | 40.4357 | 68.51 |
| 100% | 40.3381 | 68.13 |
|---------------|---------|---------|


SSIM seem very good too for RC tweak ... perhabs not classic average calculation for SSIM ... :confused:

[IgorC]

The same result here. I obtained best aver. SSIM at 75-76% VBR, 50% Key Boost, 35-40% B reduction , deblocking -2 or -3. But I´m not sure if subjective quality is better, may be even worse.

[SpaceV]

what I would like to know, are the On2 guys responsive
to your test results, do they agree, do you get updated versions
of VP7 ?
do you see results and improvements, is there progress?

please let us know at least that much.

I am very interested in VP7.

Thanks!

[Sharktooth]

SpaceV you posted the same questions in the other thread.
Dont spam over the forum, it is an unwanted behaviour and goes against the forum rules.

[Sagittaire]

test in progress ... lol

I try to make better result with VP7 (only RC tweak) than Nero H264 (with best setting in recode)

[SpaceV]

Sorry, I thought not every VP7 tester is reading both
message threads and to increase my chance of getting an answer back..... which I still have not, at least not to my questions.

[Sagittaire]

Update:
New complete test
New samples with better setting
Metric faq
Archive in download

You think that your codec is the best : prove that ... !!?

[Manao]

The best codecs (H264 or VP7) are 75 % better than MPEG2 MP@ML: MPEG2 MP@ML with good encoder is always a good codec.That doesn' mean anything at all.

[IgorC]

Sagitarrie, are you going to do 2-CD Rip (maybe 1/4 , 1/3 , 1/2 of DVD RIP, HD-DVD 5mbit (raw video source) ) test later? It will be very interesting to know about performance of each codec (AVC, ASP, MPEG2 , etc)

[Manao]

Frankly, looking at the quality of the 450 kbit samples, knowing that a trailer is notoriously more complicated to encode than a movie, a 2CDs encode is imho an overkill now, even for such a long movie as Harry Potter. Anyway, in this particular case, 450 kbit is 1CD, and 900 kbit is 2CDs.

[Sagittaire]

Originally posted by Manao
That doesn' mean anything at all.

I use the same logic than One2, Microsoft or DXN ...

MPEG2 must use 1560 Kbps for the same "metric quality" than H264 900 Kbps : 1560 is 75% more than 900 ...

example: VP(x+1) is 15% better than VPx, DivX is 30% better than WMV9, WMV9 is 50% better than MPEG2 ... ect ect ect

All these company use metric to make these equivalence!


Originally posted by Manao
Frankly, looking at the quality of the 450 kbit samples, knowing that a trailer is notoriously more complicated to encode than a movie, a 2CDs encode is imho an overkill now, even for such a long movie as Harry Potter. Anyway, in this particular case, 450 kbit is 1CD, and 900 kbit is 2CDs.


Source and bitrate are not important because I use quant reference for my test: With this source I use q4 XviD for "1CDR Quality" bitrate reference and q8 XviD for "Streaming Quality" bitrate reference. Q4 in 720*304 use praticaly (perhabs 10% less) the same bitrate than Q3 in 640*272. For example my last encoding (The incredibles) use average ~q4 (for I,P frame) in 720*304 like the majority of my 1CDR encoding. Encoding with 50% for compressibility (Q2 H263 for reference) is an average ~Q4 encoding ...

[SpaceV]

guys, on the yahoo message board,
some people complain about the bad quality
of the new VP7 clips. There seem to be more that dislike
the new quality than people that praise it.

Some say that the first VP7 clips looked much better.

I think you can see it the best at the end of the Troy
trailer. The clouds with the text in the forgeround look really bad
and blocky. Even the ones at @700!

Did anyone watch those clips there, do you guys think they
represent the best ablilities of VP7.

[Sagittaire]

Originally posted by SpaceV
guys, on the yahoo message board,
some people complain about the bad quality
of the new VP7 clips. There seem to be more that dislike
the new quality than people that praise it.

Some say that the first VP7 clips looked much better.

I think you can see it the best at the end of the Troy
trailer. The clouds with the text in the forgeround look really bad
and blocky. Even the ones at @700!

Did anyone watch those clips there, do you guys think they
represent the best ablilities of VP7.

and ... :confused:

Troy clip use perhabs high quant because it's hard trailer ... bitrate and quality are 2 very differents notions. With XviD, q1 at 150 Kbps is possible and q10 at 1500 Kbps is possible and with the same resolution ... lol

If you want compare use another codec with the same source ... :devil:

[aeternitas]

You have a couple problems that shake ones faith in this being an accurite comparison.

1. +/- X = Unfair comparison, where X is anything other than 0.
3. Each compressed sample should contain exactly the same source with no dropped frames unless a rule states it as required for a better overall finished product.
2. Ones faith in your methods are not exactly secured when one realizes you didn't even use the right compression method to compress your graphs. Try PNG8.

[sparky]

Something is still wrong with your numbers.

Firstly, the script you suggest for measuring OPSNR is faulty.

LoadPlugin("dgdecode.dll")
LoadPlugin("CompareYV12.dll")
source=Mpeg2Source("mpeg2\HPII.d2v")
source=Trim(source,70,3145)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)

#video=DirectShowSource("DivX6.avi",fps=25)
video=DirectShowSource("XvID-450.mkv",fps=25)

# --> PSNR analysis <--
compareYV12(video,source,"YUV","PSNR-DivX6-450.log")


This will not work because the default output format of DirectShowSource is YUY2. At least it is so in avisynth 2.5.5.0 ( which is the latest available on doom9 ), with decoders from xvid 1.0.3 and divx fusion.

Secondly, even after you add an explicit ConvertToYV12(), CompareYV12.dll you gave in the first post does not work. I tried it with avisynth 2.5.5.0, VirtualDub 1.5.0 and VirtualDubMod 1.5.10.1. I presume the destructor never gets called ( no numbers are output to the log ).

Thirdly, the PSNR numbers in the first post are different from what I receive from standard avisynth Compare(). SSIM's are exactly the same, but my OPSNRs are about 0.7 db higher than yours.

You have these numbers:
DivX 450 kbps - 39.5729 db
XviD 450 kbps - 39.6601 db
DivX 900 kbps - 42.6220 db

With your latest clips I see:
DivX 450 kbps - 40.3340 db
XviD 450 kbps - 40.2208 db
DivX 900 kbps - 43.3076 db

This is not a big deal because the algorithms used by Compare() and CompareYV12() could be different.

Finally, the settings you actually used to create DivX clips are somehow different from both my suggested settings and your claimed settings. Using standard avisynth Compare() and SSIM, I got these results:

your clip ( 6,918,301 bytes ) - OPSNR 40.3340 db; SSIM 63.49
my own encoding using your claimed settings and bitrate 444 kbps ( 6,907,904 bytes ) - OPSNR 40.4812 db; SSIM 64.27
my encoding with the same settings minus GMC ( 6,907,904 bytes ) - OPSNR 40.5055 db; SSIM 64.42

For reference, my DivX CLI:
-bvn1 444000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2

[Sagittaire]

Originally posted by aeternitas
You have a couple problems that shake ones faith in this being an accurite comparison.

1. +/- X = Unfair comparison, where X is anything other than 0.
2. Each compressed sample should contain exactly the same source with no dropped frames unless a rule states it as required for a better overall finished product.
3. Ones faith in your methods are not exactly secured when one realizes you didn't even use the right compression method to compress your graphs. Try PNG8.

1) +/- 1 kbps for Elementary Video Stream (VES) represent +/- 15Ko for 450 Kbps sample. My tolerance is good bacause it's very diffucult to make better and don't change the result (Overhead Problem, RC target, container ... ect ect). In my test the maximum diff for VES is 22 Ko for "6750 Ko sample" and 28 Ko for "13500 Ko sample". Try to make better if you want ...

2) it's generaly a splitter problem but 1 frame dropped (for example last frame like for RV10) and don't change the result. It's only 1 frame for 3075 and the global test don't change:


3074 0.4859 +0.0578 18 -15 48.1109 39.4580
0 0.0214 -0.0159 3 -3 64.4317 39.4594


the "0" frame is dropped by the RealSplitter and Overall PSNR is increase to 0.0014 dB with artificial black frame ...

3) it's very important ... :confused:
it's a test for video codec and not for picture codec ... lol

[Sagittaire]

Originally posted by sparky
Something is still wrong with your numbers.

Firstly, the script you suggest for measuring OPSNR is faulty.

LoadPlugin("dgdecode.dll")
LoadPlugin("CompareYV12.dll")
source=Mpeg2Source("mpeg2\HPII.d2v")
source=Trim(source,70,3145)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)

#video=DirectShowSource("DivX6.avi",fps=25)
video=DirectShowSource("XvID-450.mkv",fps=25)

# --> PSNR analysis <--
compareYV12(video,source,"YUV","PSNR-DivX6-450.log")


This will not work because the default output format of DirectShowSource is YUY2. At least it is so in avisynth 2.5.5.0 ( which is the latest available on doom9 ), with decoders from xvid 1.0.3 and divx fusion.

Secondly, even after you add an explicit ConvertToYV12(), CompareYV12.dll you gave in the first post does not work. I tried it with avisynth 2.5.5.0, VirtualDub 1.5.0 and VirtualDubMod 1.5.10.1. I presume the destructor never gets called ( no numbers are output to the log ).

Thirdly, the PSNR numbers in the first post are different from what I receive from standard avisynth Compare(). SSIM's are exactly the same, but my OPSNRs are about 0.7 db higher than yours.

You have these numbers:
DivX 450 kbps - 39.5729 db
XviD 450 kbps - 39.6601 db
DivX 900 kbps - 42.6220 db

With your latest clips I see:
DivX 450 kbps - 40.3340 db
XviD 450 kbps - 40.2208 db
DivX 900 kbps - 43.3076 db

This is not a big deal because the algorithms used by Compare() and CompareYV12() could be different.

Finally, the settings you actually used to create DivX clips are somehow different from both my suggested settings and your claimed settings. Using standard avisynth Compare() and SSIM, I got these results:

your clip ( 6,918,301 bytes ) - OPSNR 40.3340 db; SSIM 63.49
my own encoding using your claimed settings and bitrate 444 kbps ( 6,907,904 bytes ) - OPSNR 40.4812 db; SSIM 64.27
my encoding with the same settings minus GMC ( 6,907,904 bytes ) - OPSNR 40.5055 db; SSIM 64.42

For reference, my DivX CLI:
-bvn1 444000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2

1):readrule: ... lol


You must mux or remux x264, VP6, VP7, WMV9, DivX6 and XviD in avi container (with VDM for example) to solve problem with color space output, synchronisation or acquisition for metric test. You must use AviSource in avisynth for VP6, VP7 and WMV9. You must use DirectShowSource in avisynth for MPEG4 ASP, MPEG4 AVC and RV10. You must change trim for frame synchronization or for clip length if you have some probleme with SSIM plugin. You must use these AviSynth type script for metric test:


I use matroska container for optimized Overhead size and compare [codec+container] with the same bitrate for Video Elementary Stream


2) You must use AviSource for VP6 (don't work with DS), VP7 (YUY2 output only with DS) and WMV9 (desynchro with DS). You must use DirectShowSource with MPEG4 ASP (you must force YV12 output in XviD dec and DivX dec)

CompareYV12 has little bug: you must open a first time and make preview in VD and open a second time

PSNR in YV12 and in YUY2 are not equivalent: it's better to use complete YV12 preocess because all codec use YV12. In YUY2 4:2:2 the chroma is more important and better codec for chroma could have better result ...

3) I will update with your setting: IMO it's very strange that GMC decrease metric : GMC decision is perhabs not good for this trailer ... I will update too for XviD ... lol

[sparky]

You must use AviSource for VP6 (don't work with DS), VP7 (YUY2 output only with DS) and WMV9 (desynchro with DS). You must use DirectShowSource with MPEG4 ASP (you must force YV12 output in XviD dec and DivX dec)


DivX dec can't be forced into YV12 output, it does not have such feature. Only XviD dec can do that. For DivX, DirectShowSource() gives me YUY2 regardless of container format. Yes, I do have "Output YUV 4:2:0 when supported" enabled.


CompareYV12 has little bug: you must open a first time and make preview in VD and open a second time


I tried opening, previewing, opening again, opening other clip, explicitly closing .avs with Ctrl-W, etc. Right now my log file looks like this:


Comparing channel(s) YUV

Mean Max Max
Absolute Mean Pos. Neg.
Frame Dev. Dev. Dev. Dev. PSNR (dB) Avg. PSNR
---------------------------------------------------------------

Comparing channel(s) YUV

Mean Max Max
Absolute Mean Pos. Neg.
Frame Dev. Dev. Dev. Dev. PSNR (dB) Avg. PSNR
---------------------------------------------------------------

Comparing channel(s) YUV

Mean Max Max
Absolute Mean Pos. Neg.
Frame Dev. Dev. Dev. Dev. PSNR (dB) Avg. PSNR
---------------------------------------------------------------

Comparing channel(s) YUV

Mean Max Max
Absolute Mean Pos. Neg.
Frame Dev. Dev. Dev. Dev. PSNR (dB) Avg. PSNR
---------------------------------------------------------------

Comparing channel(s) YUV

Mean Max Max
Absolute Mean Pos. Neg.
Frame Dev. Dev. Dev. Dev. PSNR (dB) Avg. PSNR
---------------------------------------------------------------

Comparing channel(s) YUV

Mean Max Max
Absolute Mean Pos. Neg.
Frame Dev. Dev. Dev. Dev. PSNR (dB) Avg. PSNR
---------------------------------------------------------------

Comparing channel(s) YUV

Mean Max Max
Absolute Mean Pos. Neg.
Frame Dev. Dev. Dev. Dev. PSNR (dB) Avg. PSNR
---------------------------------------------------------------


(you've guessed the rest...)

IMO it's very strange that GMC decrease metric : GMC decision is perhabs not good for this trailer

GMC does not make a lot of difference here. Something else must be affecting your encodings ( 0.75 difference of SSIM scores between your & my clips ).

[aeternitas]

Originally posted by Sagittaire
1) +/- 1 kbps for Elementary Video Stream (VES) represent +/- 15Ko for 450 Kbps sample. My tolerance is good bacause it's very diffucult to make better and don't change the result (Overhead Problem, RC target, container ... ect ect). In my test the maximum diff for VES is 22 Ko for "6750 Ko sample" and 28 Ko for "13500 Ko sample". Try to make better if you want ...

2) it's generaly a splitter problem but 1 frame dropped (for example last frame like for RV10) and don't change the result. It's only 1 frame for 3075 and the global test don't change:



the "0" frame is dropped by the RealSplitter and Overall PSNR is increase to 0.0014 dB with artificial black frame ...

3) it's very important ... :confused:
it's a test for video codec and not for picture codec ... lol


1. 1 frame does change the result. It results in an imperfect comparison based on guessing instead of facts. 1 frame is infintly more than 0 frame loss. It's not as important as other factors, but its not something you throw to the wind if you want a serious comparison.

2. Yu should be useing the same container types and _everything else besides the video codec should be identical_ to the point where the only changes would be from compatibility issues cuased by the video codec.

3. Yes, your lack of knowlage in image compression tells more than you care to admit about how youre not very well rounded in the general compression scene.

This comparison is flawed. Flaws that can be fixed. If your comparison were an application it would be in beta.

You need to upload images of your compared scenes. PSNR and whatnot are lame representations of quality. They are not more representitive of quality than more bps is. More does not always mean better.

[huang_ch]

I'm really confused about that why RV10 get the WORST score even in 450Kbps. My original knowledge is that RV10 is really good at low bps, while XVID/DIVX/WMV codecs are more focused at higher bps and don't play well in low bps compare to RV9/10, but in this test I saw XVID/DIVX/WMA are better than RV10, can anyone tell me why?

[Shinobu]

metrics and eyes are different, that's all ^^.
for exemple if you sharp only a little a video, it may look better for you, but for metrics it'll look like hell.

metrics are cool for "pure math test" , the only way to corectly compare codecs power is to do blind test and use your eyes ^^.
on the same encoding, i can find rv10 better and you can find vp7 better, it's a matter of taste, someone perfers a little moscito noise but mutch details, some other no mosquito but less details.... and that can't be calculated by metrics.

for my eyes rv10 looks also better than any others at low bitrates (may be not nero avc in non-anime case), but eyes are not metric ...

++

[Sagittaire]

Originally posted by aeternitas
1. 1 frame does change the result. It results in an imperfect comparison based on guessing instead of facts. 1 frame is infintly more than 0 frame loss. It's not as important as other factors, but its not something you throw to the wind if you want a serious comparison.

2. Yu should be useing the same container types and _everything else besides the video codec should be identical_ to the point where the only changes would be from compatibility issues cuased by the video codec.

3. Yes, your lack of knowlage in image compression tells more than you care to admit about how youre not very well rounded in the general compression scene.

This comparison is flawed. Flaws that can be fixed. If your comparison were an application it would be in beta.

You need to upload images of your compared scenes. PSNR and whatnot are lame representations of quality. They are not more representitive of quality than more bps is. More does not always mean better.



You need to upload images of your compared scenes.

very stupid comparison method : your lack of knowlage in video compression tells more than you care to admit about how youre not very well rounded in the general compression scene. Images comparison (PNG, JPG or uncompressed 4:4:4 RGB 1024 bit ... lol) are very unable to make video codec comparison. Local difference like I,P,S,B,b or RC difference are not the same for all codec. Conclusion for N frame is not the same that conclusion for N+1 frame. If you want make visual test download sample ...



:readrule:

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...


Read pdf SSIM ... if you contest SSIM test speak with dev and not with me ... it's perhabs Lame like you say ... :devil:



1) not correct and I prove that when you want. One frame for 3075 don't change the result: with my source (Perhabs maxi 0.005 dB less or more for OPSNR and the first and last frame for this source is black frame for solve this problem). Source is the same for all codec ... dropped frame is little parser or decoder problem

2) not correct and I prove that when you want. I can use different container if I want. The most important is the VES bitrate. In my test bitrate for VES and files are the same ... mp4 container or matroska container use in practice the same overhead for this test.

3) I use JPG if I want for my graph ... it's not picture compression test ... you are really strange or particulary ... "Lame" like you say ... :confused:

[Sagittaire]

for exemple if you sharp only a little a video, it may look better for you, but for metrics it'll look like hell.

That does not mean anything and you did not understand what must be a codec. Sharp it's an post-process modification possible with all codec ... but you must compare equivalent frame and Sharp frame isn't the same frame than source. Source with denoising look better than original source but the codec objective is not denoising: codec must restitute the noise if source is noisy. Metric compare convergence between input and output codec ... that's all.



@ huang_ch & Shinobu

:readrule:

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...


Read pdf SSIM ... if you contest SSIM test speak with SSIM dev and not with me ...

If you want speak about visual quality vs metric test open another thread

For end you must use the best Post-Process and decoder for all codec:
- PP1 for WMV9 (http://multimediacom.free.fr/Video/WMVPostpross.exe)
- PP4 with DivX and DivX dec
- PP4 with XviD and XviD dec
- Don't use HFE 2.1 for RV10 with sharp but HFE 1.

you must see too all frame of the trailer: for exemple x264 isn't good for the first scene (warner presentation) visually and with metric but after the codec is very good ...

[aeternitas]

1) not correct and I prove that when you want. One frame for 3075 don't change the result: with my source (Perhabs maxi 0.005 dB less or more for OPSNR and the first and last frame for this source is black frame for solve this problem). Source is the same for all codec ... dropped frame is little parser or decoder problem

2) not correct and I prove that when you want. I can use different container if I want. The most important is the VES bitrate. In my test bitrate for VES and files are the same ... mp4 container or matroska container use in practice the same overhead for this test.

3) I use JPG if I want for my graph ... it's not picture compression test ... you are really strange or particulary ... "Lame" like you say ... :confused: [/B]

0."Conclusion for N frame is not the same that conclusion for N+1 frame." Exactly why ONE FRAME MATTERS in these sorts of tests. The whole would-be sequance has a good chance of being changed for several frames down the line.

Also, you can definitly find and pull a P or B frame that is placed in a similer set string of the sequance thoughout each encode.

1. You can make excuses to how negligible that is all day, but a difference is a differance in the source.

2. You can use any container you want, I will give you that, as long as the source for all encodes are identical and the bps given to each second of video per codec is the same.

3. You can use BMP if you want. Thats not the point. The point is its on the same level. Its a compression. I just found it ironic that you used the worst possible compression method for those images while you're trying to convince people how wonderfully fair and thought out your video compression tests are.

[Manao]

sparky : compareyv12 works a lot better if instead of giving as a filename 'foo.txt', you give the full path.

sagittaire : it's not because firms use their own twisted logic to say that their codec is 75 % better than another that you should do the same. What can be said is that, coming from the same source, and aiming at the same quality ( in that case, PSNR ), an h264 encode's size will be between 50 to 70 % of the size of the mpeg2 encode's one.

aeternitas : i found some of your critics miss the point. Why are you attacking Sagittaire on the compression used for his graph's screenshots ? Indeed, png was the best choice, but perhaps jpg was the most convenient for him ( it depends on the program he used to save his graph, after all ). That is not correlated to his encoding skills at all and that's imho a gratuitous attack.

Then, you're criticizing some approximations made by Sagittaire. Indeed, there are some : bitrates aren't all equal, but can't be except if you're ready to keep encoding over and over the same clip with the same codec in order to bypass ratecontrol imprecision and containers' discrepancies. Indeed, the looseness on the missing frames & co is unwelcomed, but circumvent them would have taken some time and the imprecision they bring is by far smaller than anything you could notice with your eyes.

I still wonder, btw, why you're asking Sagittaire to post screenshots, when he gave access to the whole clips ( source + results )

So if you want to have a row with him, make it private please.

huang_ch : download the resulting clips and make your own visual impressions. PSNR and SSIM and only slightly correlated to visual impression.

[Sharktooth]

Here's another "i know it all, you all suck (http://forum.doom9.org/showthread.php?s=&threadid=89093&perpage=20&pagenumber=2#post622608)" newbie...
aeternitas please keep your comments private, they're not relevant.

[Sagittaire]

Originally posted by Manao
sagittaire : it's not because firms use their own twisted logic to say that their codec is 75 % better than another that you should do the same. What can be said is that, coming from the same source, and aiming at the same quality ( in that case, PSNR ), an h264 encode's size will be between 50 to 70 % of the size of the mpeg2 encode's one.

yes it's true ... perhabs not very pertinent comparison:

Equivalent metric for MPEG2 at 1560 Kbps and MPEG4 AVC at 900 Kbps

With H264 reference bitrate "H264 is 73% better than MPEG2"
With MPEG2 reference bitrate "H264 is 42% better than MPEG2"

but it's easy to understand for newbie than "h264 done same result than MPEG2 with X% less for bitrate" only with metric ...

Update in progress ...

[sparky]

Originally posted by Manao
[B]sparky : compareyv12 works a lot better if instead of giving as a filename 'foo.txt', you give the full path.


!!!

Everything works now, thanks

[hdonly]

http://on2.com/duckutils/duck_license.php3?class=player

Fixes CPU complexity problem. Clips look better too!

[eqbal]

how can i determine PSNR value for a movie?
is there any software?

[Manao]

Search the forum for Video Quality Studio ( and more generally, search the forum for such question )

An alternative way could be avisynth ( the compare or compareYV12 function ), but it is more complicated if you don't know how to use avisynth.

[trbarry]

It might be interesting to have a test like this where everybody instead had to match or beat the same SSIM, say 80 or better.

The winner would be the one with the codec/parms/container with the smallest total file size. Then you could truly say one codec was xx% better than another, at least for some given clip and agreed on SSIM target.

The only true test is probably to view the moving videos but I still tend to trust the metrics for comparisons.

- Tom

[Sagittaire]

@ trbarry

perhabs next test with HDTV resolution ...

[trbarry]

perhabs next test with HDTV resolution ...

That would be near & dear to my own heart. ;)

But there is a problem with testing real HD sources because there aren't many to test. Most HD captures seem to need only maybe 544p or at most 720p to duplicate since they really don't have any more effective detail than that anyway, and a little is always lost in re-encoding. So maybe a nice test in that range would be appropriate if enough folks here were interested in HD yet.

- Tom

[Sagittaire]

Update

- New VP6 Trailers
Better Rate Control setting

- New VP7 Trailer
Encoding with PIV class CPU for "450" trailer only. VP7 seem little buggy with MMX/SSE class CPU like PIII.

- New DivX Trailers
For the first time DivX is better than XviD for OPSNR and SSIM. Adaptative Quantisation (psy mode) is not optimized with DivX Fusion : with good A.Q DivX will be very better for SSIM. It's perhabs possible to make better encoding with RC tweak for XviD/DivX (variablity and bframe ratio/offset) ... if MPEG4 ASP's fan want to try ... ???

- New RV10 trailers
Very better Rate Control setting


Technical Information

Originally posted by huang_ch
I'm really confused about that why RV10 get the WORST score even in 450Kbps. My original knowledge is that RV10 is really good at low bps, while XVID/DIVX/WMV codecs are more focused at higher bps and don't play well in low bps compare to RV9/10, but in this test I saw XVID/DIVX/WMA are better than RV10, can anyone tell me why?

1) My eyes seem say that too. And for the first time I don't understand why. Perhabs that it's true : metric are unable to test video quality ... !!?


|---------------|-----------|------------|----------|---------|---------|
| Codec | P-Process | ES Bitrate | Size | OPSNR | SSIM |
|---------------|-----------|------------|----------|---------|---------|
| DivX | PP4 | 447 kbps | 6757 Ko | 39.5729 | 63.49 |
| XviD | PP4 | 447 kbps | 6760 Ko | 39.6601 | 64.36 |
| RV10 old | HF1 | 447 kbps | 6767 Ko | 39.4594 | 62.83 |
|---------------|-----------|------------|----------|---------|---------|
| DivX | PP4 | 896 kbps | 13490 Ko | 42.6220 | 77.47 |
| XviD | PP4 | 896 kbps | 13498 Ko | 42.6654 | 77.95 |
| RV10 old | HF1 | 896 kbps | 13501 Ko | 42.5815 | 77.21 |
|---------------|-----------|------------|----------|---------|---------|


2) I make Graph PSNR test to compare with other codec and I see that with default RC setting for RV10:

RV10 vs XviD
http://multimediacom.free.fr/Video/RV10-Problem.PNG

With this trailer and defaut RC setting the last frames of the trailer are dramaticaly worst than all other codec. It's typicaly a Rate Control Problem. In fact RV10 RC use to high size for the begin and too low size for the end. Visually and for metric these last frames MPEG4 ASP and all the other codec are very better than RV10.


3) I remade "450" encoding with new and really optimized metric RC tweak
- Size quant prediction: rcPFrameRefQuant = 17 for better prediction
- Bframe ratio : rcBFrameRefQuant = 23 for better metric result
- Rate Contro : rcLowBitrateBoost = rcHighBitrateReduce = 20 for better metric result

I remade too "450" with old RC and it's too very better than Elysian RC with defaut RC setting for this trailer. The new encoding are very better for the complete trailer and very better for the last frames.


|---------------|-----------|------------|----------|---------|---------|
| Codec | P-Process | ES Bitrate | Size | OPSNR | SSIM |
|---------------|-----------|------------|----------|---------|---------|
| DivX | PP4 | 446 kbps | 6755 Ko | 39.7552 | 64.44 |
| XviD | PP4 | 446 kbps | 6760 Ko | 39.6601 | 64.36 |
| RV10 new | HF1 | 446 kbps | 6743 Ko | 40.0457 | 65.52 |
|---------------|-----------|------------|----------|---------|---------|
| DivX | PP4 | 896 kbps | 13508 Ko | 42.7455 | 77.93 |
| XviD | PP4 | 896 kbps | 13498 Ko | 42.6654 | 77.95 |
| RV10 new | HF1 | 896 kbps | 13493 Ko | 42.7431 | 77.73 |
|---------------|-----------|------------|----------|---------|---------|


RV10 update vs RV10 previous
http://multimediacom.free.fr/Video/RV10-Solve.PNG

4)
Originally posted by Shinobu
for my eyes rv10 looks also better than any others at low bitrates (may be not nero avc in non-anime case), but eyes are not metric ...


Yes, your (and my) eyes are not metric : your (and my) eyes were unable to detect this Rate Control artefact and very visible problem for the 250 last frames perhabs simply because the clip was not entirely views ... perhabs because for eyes the first scene is more important than the last ... perhabs because it's very difficult for eyes to compare 3075 frames and make objective overall observation ...

:readrule: :readrule: :readrule: :readrule: :readrule: :readrule: :readrule: :readrule:

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...


Conclusion: Make yourself the conclusion ... !!!

[IgorC]

x264 rev157? Why sample of x264 wasn´t updated? The last revision has better SSIM result.

[Sagittaire]

perhabs 6 or 7 days between rev 157 and rev 173 ... lol

bref frame doen't work wery well at this time (dec problem for me) ... rev 173 is perhabs little better than rev 157 but not revolution ...