Log in

View Full Version : [20-06-2005] Metric Benchmark Challenge ...


Pages : 1 2 [3] 4 5 6

IgorC
15th March 2005, 16:56
What significate MinQ 8/16? Is it MinQ=8 for 900kbps and MinQ=16 for 450kbps, right?

EDIT
660 kbit/s test
rev 157 - SSIM 77.64
rev 174 - SSIM 77.76

Sagittaire
15th March 2005, 17:03
Originally posted by IgorC
What significate MinQ 8/16? Is it MinQ=8 for 900kbps and MinQ=16 for 450kbps, right?

yes ... One2tech RC optimisations


Originally posted by Tommy Carrot
There were other changes in x264 other than bref, chroma ME should improve the quality by quite a bit, for example. But if you want to wait for the bigger improvements, wait a few days, as RDO support seems to be very near to completion.

I will wait ... :)

IgorC
15th March 2005, 21:03
Why Ateme encavc beta4a used as NERO AVC codec in this test? Who has it? Only beta-testers had this opportunity. ateme beta4a lets to put 16r ref while Recode2 only 8 ref. So is it fair to compare on this mode?

Sagittaire
15th March 2005, 21:31
ateme beta4a and current ND AVC for recode are very similar (I make test for that in ateme beta thread). The only difference is maxbframe number and maxpref number. 8 or 16 for maxpref is not very important ... with more 5 pref the metric gain is very little with MPEG4 AVC ...

I use ateme CLI because batch mode is very good for test ...

IgorC
15th March 2005, 22:31
Sagittaire, what version of Xvid 1.1 beta 1 do you use in your last test?
http://www.aziendeassociate.it/./cd///XviD.cvs.head.exe (14 march)
in 660kbit/s test it gives more +0.1 SSIM than Xvid 1.1 beta 1 official 16 jan. since in this test all kind of version ( release, revision, beta etc.). The difeernce between Divx-Xvid is too small, so it will be enough to say that Xvid better (iqual/better on 450kbit/s) than Divx6

Sagittaire
15th March 2005, 22:53
In fact I use special RC metric tweak for XviD Koepi 1.1.0 beta 1 like for RV10:

- Size quant prediction: first pass in Q4/Q8 for 900/450 trailer
- Variability: HighBitrateReduce and LowBitrateBoost optimisation

perhabs possible to make better with bframe ratio/offset ... if XviD fan want make test ...

huang_ch
16th March 2005, 03:25
Great, thanks for the explanation for RV10 issues. And I know metric is only metric, I've made several visual test these days, I think to my eyes RV10 and Nero AVC is almost at the same level in low bitrate for Animes (300-500kbps), and x264 is a bit worse than them. I don't try the VP6 & VP7 codec. Since I'm not experts in doing this kind of test, so indeed I can't distinguish the quality between RV10 & AVC even by frame and frame in VirtualDub. But for the encoding time, I think AVC is 1.x-2.x slower than RV10 and x264 is at least 2X faster than RV10. :cool:

IgorC
16th March 2005, 04:16
Originally posted by Sagittaire
In fact I use special RC metric tweak ..

Saggitarie. Can you answer to some my questions relationed to XVID and x264?
Does it mean 1st past at Q8 (for 450kbit/s) and 2 pass as usual with decribed settings ?
What setings fot HighBitrateReduce and LowBitrateBoost did you use?
What setings of x264 (KEyBoost and Breduction) did you use?. I obtain the same SSIM for your trailer x264-450 (68.24), but my trailer encoded with the same rev157 and same settings gives me another result.
Where can I find HFE1? There is only HFE2


x264(your trailer) rev157 SSIM=68.24 , OPSNR = 40.4423
VP7 (your trailer) SSIM=68.57 , OPSNR = 40.6802
x264 rev174 SSIM=68.77 , OPSNR = 40.4737

settings for x264 rev174 - Cabac, Deblock -1, Bframes 2 (Temp), Weight, ref15, keyboost 50% , Breduc 40%, VBR 75%, 5 max Quality , macroblock partitions ALL on. I donītīknow if these settings are best for this source but parametlicly (and visually too) gived a better result.
As a conclusion , x264 > VP7 for SSIM test.
x264 < VP7 for OPSNR test.
But maybe there is oportunity to find better balance between OPSNR and SSIM changing values for RC.

redfordxx
21st March 2005, 17:08
Originally posted by Sagittaire
A - Introduction
The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...
Hi,
with metrics can be measured other parameters. I cannot download samples ATM, but I guess there could be some blocks:). I suggest following script to measure amount of blocks. Maybe someone can say metrics are not relevant compared to visual tests. I doubt it can be said about this blocktest.LoadVirtualdubPlugin("...\deblock.vdf", "MSUDeBlock")
....

video=DirectShowSource("G:\...\RV10-450.mkv",fps=25)
deblocked=video.ConvertToRGB32(matrix="pc.601").MSUDeBlock(1,1,1).ConvertBackToYUY2(matrix="pc.601").ConvertToYV12()

# --> PSNR analysis <--
return SSIM(deblocked,video,"results.csv","SSIM-RV10-450.txt",lumimask=true) Maybe you find it useful to add one column to your table for this (+value for source itself).
I think some other artifact tests could be invented...just to find the correct filter which aims on the specific problem of codec, e.g. mosquito noise (well I read of it here often, but can someone describe exactly, what is it?).

temporance
21st March 2005, 17:13
Neat idea, but this only measures blocks that can be deblocked. Other blocks (such as those translated by motion compensation) cannot be deblocked and will not be measured by your script.

Bear in mind that there is a whole industry devoted to perceptual video quality measurement - VQEG uses all sorts of signal processing techniques in its various metrics.

Sagittaire
21st March 2005, 18:01
Originally posted by redfordxx
with metrics can be measured other parameters. I cannot download samples ATM, but I guess there could be some blocks:). I suggest following script to measure amount of blocks. Maybe someone can say metrics are not relevant compared to visual tests. I doubt it can be said about this blocktest.

I use PP4 (deblocking YUV in post-process) for best result with MPEG4 ASP. With PP4 metric are very better than PP0 (perhabs ~1 dB better)

Conclusion: metric can detect blocking ...

redfordxx
21st March 2005, 18:20
Originally posted by Sagittaire
I use PP4 (deblocking YUV in post-process) for best result with MPEG4 ASP. With PP4 metric are very better than PP0 (perhabs ~1 dB better)
I see, so you are not interested in real encode quality but in quality after PP... is that relevant? I read here many misunderstandings who measured what value, but it could be with different postprocessing...
Conclusion: metric can detect blocking ...
How do I decide (and check) which codec is used for decoding in avisynth script. I dont know, it could be XviD or FFDShow. I know in VDM is used XviD, althoug I'd like FFDS...

Sagittaire
21st March 2005, 19:06
Originally posted by redfordxx
I see, so you are not interested in real encode quality but in quality after PP... is that relevant? I read here many misunderstandings who measured what value, but it could be with different postprocessing...

How do I decide (and check) which codec is used for decoding in avisynth script. I dont know, it could be XviD or FFDShow. I know in VDM is used XviD, althoug I'd like FFDS...

Objective is best possible metric ... for best metric with MPEG4 ASP you must use PP4 (not PP2 or PP6 but PP4). PP4 is in fact real quality encoding : XviD dec or DivX dec use PP4 for default setting. Chip can use PP4 in hardware decoding. This test is co-dec test: I use DivX dec for DivX, XviD dec for XviD but ffdshow dec is not better than XviD dec ...

DirectShowSource use the default decoder, if XviD dec is active for XviD playback in MPC then DirectShowSource will use XviD dec in Avisynth script ...

redfordxx
21st March 2005, 19:17
Originally posted by Sagittaire
ffdshow dec is not better than XviD dec ...
But, there has been the issue with green color in XviD dec. Sure, I didnt check the recent releases, since I'm using FFDShow.

WorBry
28th March 2005, 10:02
Sagittaire "Testeur de Codecs"

I would be appreciate some advice on PSNR testing. I have been in the process of comparing different codecs for compression of captured Type II Pal DV videos and of course this has lead me into the realm of metric measurement comparison. Until recently, I have been using Everwicked's Video Quality Studio v3.2 to compare DivX, XviD and VP6.2 encodes. This version of VQS gives the raw PSNR values for Y, U and V, as well as VQM. For the reference file I've been using the same source AVS script that I now use for all DV conversions:

avisource("Test Pal DV Type II clip.avi")
ConvertToYUY2(interlaced=true)
ColorYUV(off_y=-16,gain_y=16)
KernelDeint(order=0,sharp=true)
LanczosResize(720,544)
Convolution3D(1,4,5,3,4,2.8,0)
RemoveGrain(mode=2)

(Note: the ColorYUV adjustment is to correct for the luma shift that occurs when using the Sony DV Software codec for decompression)

My simplistic approach to evaluating the data was to derive average PSNR values for each of the channels and...compare them. On this basis all three codecs were giving PSNR's in the typical range with VP6.2 as the clear leader. I wont go into details of the codec settings, but these were all 2-pass encodes at 2000kbps. From reading your posts I learn that is not the best way to do it and that I should be looking at Overall PSNR. In addition I've now extended my test repertoire to other formats including RV10 and NeroAVC and of course found that I could not load the encodes in VQS, being DirectShow and not vfw formats. So, I've tried to use the AviSynth Compare and CompareYV12 filters instead. Again I've been using the same source AVS script as the reference which outputs as YUY2. Consequently, I've been mostly using the Compare filter and then adding ConvertToYUY2() to the test file line (Avisource or DirectShowSource, as appropriate). Now I'm acheiving OPSNR results in the expected range and my conclusions, so far, are XviD > NeroAVC > DivX Fusion > DivX (I'll maybe post the results later, after I've played around with NeroAVC some more). I finally got RV10 to load as a DirectShowSource after installing RealMediaSplitter. VP6.2 appeared to load OK as an AviSource. However, both RV10 and VP6.2 give ridiculously low OPSNR values, down in the 15-20 range, which, for is contrary to the PSNR's I was acheiving for VP6.2 with VQS and those for RV10 recorded in the AutoRV10 PSNR log files - which are all up in the expected 40 - 43 range. From the results you posted at the start of this thread, you clearly dont have this problem. Any idea what I am obviously doing wrong and how to correct it?

Many thanks.

On2Tech
28th March 2005, 11:07
When you do CompareYV12(x,y,"","result.txt") the first of the two is what is displayed on the screen. I usually choose to set it up so that the decoded video is displayed rather than the source as this can sometimes give advanced warning if something is wrong.

My guess in your case is that the RV10 and VP6 clips may be decoding such that they are flipped vertically (upside down comapred to source) when played in this way through avisynth.

If this is the case you would of course get silly psnr numbers. This can be solved by adding a video=FlipVertical(video) before the compare.

For example

video=AviSource("vp6.avi")
video=FlipVertical(video)
CompareYV12(video,source,"","result.txt")

I hope this helps,

On2Tech

Originally posted by WorBry
Sagittaire "Testeur de Codecs"

I would be appreciate some advice on PSNR testing. I have been in the process of comparing different codecs for compression of captured Type II Pal DV videos and of course this has lead me into the realm of metric measurement comparison. Until recently, I have been using Everwicked's Video Quality Studio v3.2 to compare DivX, XviD and VP6.2 encodes. This version of VQS gives the raw PSNR values for Y, U and V, as well as VQM. For the reference file I've been using the same source AVS script that I now use for all DV conversions:

avisource("Test Pal DV Type II clip.avi")
ConvertToYUY2(interlaced=true)
ColorYUV(off_y=-16,gain_y=16)
KernelDeint(order=0,sharp=true)
LanczosResize(720,544)
Convolution3D(1,4,5,3,4,2.8,0)
RemoveGrain(mode=2)

(Note: the ColorYUV adjustment is to correct for the luma shift that occurs when using the Sony DV Software codec for decompression)

My simplistic approach to evaluating the data was to derive average PSNR values for each of the channels and...compare them. On this basis all three codecs were giving PSNR's in the typical range with VP6.2 as the clear leader. I wont go into details of the codec settings, but these were all 2-pass encodes at 2000kbps. From reading your posts I learn that is not the best way to do it and that I should be looking at Overall PSNR. In addition I've now extended my test repertoire to other formats including RV10 and NeroAVC and of course found that I could not load the encodes in VQS, being DirectShow and not vfw formats. So, I've tried to use the AviSynth Compare and CompareYV12 filters instead. Again I've been using the same source AVS script as the reference which outputs as YUY2. Consequently, I've been mostly using the Compare filter and then adding ConvertToYUY2() to the test file line (Avisource or DirectShowSource, as appropriate). Now I'm acheiving OPSNR results in the expected range and my conclusions, so far, are XviD > NeroAVC > DivX Fusion > DivX (I'll maybe post the results later, after I've played around with NeroAVC some more). I finally got RV10 to load as a DirectShowSource after installing RealMediaSplitter. VP6.2 appeared to load OK as an AviSource. However, both RV10 and VP6.2 give ridiculously low OPSNR values, down in the 15-20 range, which, for is contrary to the PSNR's I was acheiving for VP6.2 with VQS and those for RV10 recorded in the AutoRV10 PSNR log files - which are all up in the expected 40 - 43 range. From the results you posted at the start of this thread, you clearly dont have this problem. Any idea what I am obviously doing wrong and how to correct it?

Many thanks.

WorBry
28th March 2005, 19:46
Thanks On2Tech. You were right, the VP6.2 encode had become inverted in the Compare script and FlipVertical corrected for this. However, the RV10 encode was not inverted and I'm still trying to figure out why I'm getting these very low PSNR's with both CompareYV12 and Compare(YUY2).

Richard Berg
31st March 2005, 15:14
I'm getting weird numbers just trying to do the test. 24.5 PSNR, 11.1 SSIM. When I playback the original + encode with StackVertical, they line up fine all the way to the last frame (which I trim off in the comparison).

Sagittaire
3rd April 2005, 19:19
Update 03/04/05

- x264 rev 184
- VP7 7.0.0.4

IgorC
3rd April 2005, 21:06
0 is not optimal deblcoking for VP7 and x264 for this source. However it makes a scence to compare with deblocking filter by default = 0.

Sagittaire
3rd April 2005, 22:17
Originally posted by IgorC
0 is not optimal deblcoking for VP7 and x264 for this source. However it makes a scence to compare with deblocking filter by default = 0.

yes ... it's always true for VP7 ... ???

I use inloop = 0 for x264 (best for x264's dev and other H264's dev) and post process level 2 (deblocking YUV I think and best for One2Tech). Perhabs possible that inloop +/- 1 could be ponctually the best for some source but difference for metric will be very ligth I think ...

could be good idea to make test with different deblock level:
deblock = -6 -> OPSNR = X6
deblock = -5 -> OPSNR = X5

IgorC
4th April 2005, 01:00
Yes, it was supposed before that settings of x264 would be discussed with devs of x264 and H.264
But for x264 3 b-frames isnīt optimal(at least for this source). The best is 2 bframes. I checked with the same source for SSIM test. And deblocking -1 is better than 0 (in most of cases, i didnīt tested anime). I have the more or less (a little desviation from your results +/- 0.02-0.03 )same SSIM, OPSNR for VP7, but with settnigs 2b + deblock -1 x264 has a higher SSIM (although visually VP7 is better).

Sagittaire
27th April 2005, 21:51
in metrics at least (well, it looks quite good visually too).

Like say Koepi "don't trust in magic number" ... llllooollll

update in progress with last codec

test in progress with HDTV resolution ... :)

IgorC
27th April 2005, 23:49
About magic numbers. Of course SSIM, PSNR arenīt ideal test, but looking at numeric results of both, you have idea whatīs going on.
Iīm very interesting on HD test and update Vp7.06, Xvid beta2, snow and LAVC. LAVC seems to be not bad for ASP quality. itīs still buggy and has somewhere artefacts but image is very sharp.

CyberGuy
30th April 2005, 21:15
JM 9.5 Reference Encoder

Size: 7,003,927 Bytes (.264)
SSIM: 70.90

I am enoding using some optimizing and with more optimization I doubt that anybody will be able to beat it, that's why I love the AVC/H.264 Reference Encoder. Now I only need to get the distributed encoding working so I can encode in a reasonable time.

babayaga
13th May 2005, 15:10
Originally posted by CyberGuy
JM 9.5 Reference Encoder

Size: 7,003,927 Bytes (.264)
SSIM: 70.90

I am enoding using some optimizing and with more optimization I doubt that anybody will be able to beat it, that's why I love the AVC/H.264 Reference Encoder. Now I only need to get the distributed encoding working so I can encode in a reasonable time.

You shouldn't doubt anymore, using a special experimental version of an encoder, one can reach slightly higher SSIM : 7x.93, size = 7 004 847 Bytes (.264). I let you guess which value is x ;)

Lefungus
13th May 2005, 17:34
Where are the csv files ?
A mean value is so uninteresting :)

temporance
14th May 2005, 13:48
x=0

babayaga
14th May 2005, 15:25
Originally posted by temporance
x=0

No, x > 0 :p

temporance
16th May 2005, 20:29
x=1

My final offer :)

Sharktooth
16th May 2005, 20:38
x = Int((9*Rnd)+1)

babayaga
17th May 2005, 13:53
Originally posted by Sharktooth
x = Int((9*Rnd)+1)

You might be right, it evolves almost every day :)

Just a glimpse of our next beta:
450kbps, .MP4 size = 6747kB : SSIM = 71.69 (nearest contender VP7 = 68.84)
900kbps, .MP4 size = 13491kB : SSIM = 81.57 (nearest contender VP7 = 79.33)

455kbps, .264 size = 7 003 809 bytes : SSIM = 72.08. Who said that the JM could not be defeated ? :p

JohnV
17th May 2005, 14:25
Originally posted by babayaga
You might be right, it evolves almost every day :)

Just a glimpse of our next beta:
450kbps, .MP4 size = 6747kB : SSIM = 71.69 (nearest contender VP7 = 68.84)
900kbps, .MP4 size = 13491kB : SSIM = 81.57 (nearest contender VP7 = 79.33)

455kbps, .264 size = 7 003 809 bytes : SSIM = 72.08. Who said that the JM could not be defeated ? :p Looks good. :) Can't wait to have this integrated into Nero Recode. :) Looks like VP7 has some catching up to do. ;)

Sagittaire
17th May 2005, 14:26
Saperlipopette ... lol


High profil or Main profil ... ???

Sharktooth
17th May 2005, 14:28
Originally posted by JohnV
Looks good. :) Can't wait to have this integrated into Nero Recode. :)
:p

Sagittaire
17th May 2005, 14:33
if it's true for SSIM, it's perhabs more than 42 dB for OPSNR. In bitrate equivalence it's ~800 Kbps for MPEG4 ASP (with very best codec and very best setting) and more than 75% bitrate for the same quality result ... :eek:

-> it's by far (and really far this time) the most powerfull codec in the world ...

babayaga
17th May 2005, 15:55
Originally posted by Sagittaire
if it's true for SSIM, it's perhabs more than 42 dB for OPSNR.

No it's not the case ;) OPSNR and SSIM are sometimes not so closely related. That's why SSIM exists btw.

Regarding perceptual quality, you're right. AVC 450kbps is quite close to the best ASP at ~750kbps
(We've just made a subjective comparison with your xvid 900kbps, some prefer AVC, some other prefer Xvid)


-> it's by far (and really far this time) the most powerfull codec in the world ...
There was a doubt ? ;)

Sagittaire
17th May 2005, 20:02
Originally posted by babayaga
No it's not the case ;) OPSNR and SSIM are sometimes not so closely related. That's why SSIM exists btw.


Perhabs you use psy optimisation ... ???


There was a doubt ? ;)


IMO VP7 (high quality) and x264 (high speed) are very serious challengers for NDAVC Main Profil ... lol

I think that I will wait your next beta (if I'am "trusted tester" ... lol) for my HDTV test.

Ghim
17th May 2005, 22:27
These results seems extremely promising for the evolution of H.264 based codecs in the future !!

The only problem is that these numbers (OPSNR, SSIM, ...) can't be taken as a proof that a codec is subjectively better than an other one. Even if it's not the point here, It would be really interesting to see how the codecs perform in a proper, standardised, subjective test ^^ in order to really know which ones perform better (I know I'm dreaming because doing subjective tests is too expensive for just comparing codecs ^^).

I also don't deny that OPSNR and SSIM are a good start, but it's hard to measure how much a codec is better compared to another one by using these (OPSNR or SSIM) figures. By example, if you say that the SSIM is 71.86, how does it subjectively compare to a score of 70.50 ? A normalised MOS or DMOS scale could be more appropriate to quantify how much a codec is better compared to another one.

superdump
17th May 2005, 23:21
I wouldn't say subjective comparison is expensive if you can find a community willing and able to carry out such tests. I think a number of us would be willing and are able if people were to organise such a test.

LordRPI
17th May 2005, 23:31
Didn't everwicked have some subjective video quality measurement applications that could make something like this a lot easier?

babayaga
17th May 2005, 23:51
Originally posted by Ghim
These results seems extremely promising for the evolution of H.264 based codecs in the future !!

The only problem is that these numbers (OPSNR, SSIM, ...) can't be taken as a proof that a codec is subjectively better than an other one. Even if it's not the point here, It would be really interesting to see how the codecs perform in a proper, standardised, subjective test ^^ in order to really know which ones perform better (I know I'm dreaming because doing subjective tests is too expensive for just comparing codecs ^^).


PSNR at constant quantizer gives a good idea of the raw efficiency since this is what core encoder try to optimize most of the time.
MPSNR/OPSNR add some hints about rate-control and SSIM (JND metrics or whatever) add clues about adaptive quantization.

A codec which is inferior is those 3 metrics will almost surely be seen as subjectively inferior.

Besides, subjective testing is also very difficult to do and is not always reliable.

Sagittaire
18th May 2005, 00:19
Yes, yes and yes ... :D

Originally posted by babayaga
Besides, subjective testing is also very difficult to do and is not always reliable.

for little group of frame it's possible but for high nomber of frame it's impossible in this case to make a global subjectif testing ...

Ghim
18th May 2005, 07:59
Originally posted by babayaga
Besides, subjective testing is also very difficult to do and is not always reliable. [/B]

When I meant subjective test, I meant a proper test done in a controlled environement (Subjective Test Facility) and using a standardised method (ACR,DSCQS). The tools created by everwicked follow a standardised method, but reproducing a proper controlled environement is quite impossible to do by yourself.
This kind of test would give very reliable results but as Sagittaire mentioned, the big drawback in the current subjective tests methods is that they're designed for assessing the quality of 8-10 seconds segment. I know some people are working on long sample subjective tests (length in minutes), but there's nothing standard yet and it's in the speech domain.

In order to get good results, you would need to encode videos 2 minutes by example, and once encoded you would cut them in 8-10 seconds chuncks that you would use in the tests. And during the analyse of the results you would need to regroup the scores of the different parts of the video. In order to test the codecs properly, you would need to organise severall subjective tests (one would not be enough).
But as I mentioned earlier, this would be VERY expensive (in terms of money and time) and would need to be redone for each new version ^^.

Sagittaire
18th May 2005, 08:39
In order to get good results, you would need to encode videos 2 minutes by example, and once encoded you would cut them in 8-10 seconds chuncks that you would use in the tests.


yes but make this supposition: source 100 sec with 90 sec of low motion and 10 sec of high motion (for last frames). Cut this sample in 10 part for blind test.

I compare the source in CBR for each segment (or VBR buffer 10 sec) and in real unconstrained VBR:
- 9/10 CBR is better than VBR
- 1/10 VBR is very very better than CBR
-> conclusion: CBR is better than VBR ... ???

you must compare codec in quality mode (constant quality / constant quant) and constant size condition for good conclusion but RC tweak is a very important part for codec quality and in practice RC optimisation must be very different between different codec ...

Ghim
18th May 2005, 09:01
That's the reason I said you would need more subjective tests...
You wouldn't use just 1 sequence of 2 minutes, it wouldn't be enough.

But you wouldn't just see which codec one is the best on each 8 seconds segment. You would use at least an average of the scores over the segments or an other way to have a good MOS (or DMOS) estimate over the overall sequence.

So, If I follow your example:
- 9/10 CBR is better than VBR (but let's say the difference is very small)
- 1/10 VBR is very very better than CBR (higher difference)

You would then by example make the average of the scores, regardless the number of 8-seconds segments where one performs better than the other. If on average CBR is still higher, I would say that ON THIS SEQUENCE, CBR performs better. That's why you would also need enough sequences covering a larger range of motion.

Let's say you use 8 2-minutes sequences => 96 10-seconds segments with 8 different codecs and 2 settings per codec => 16 conditions per segment
=> It would represent 1536 files. If for one subjective test session, you consider the maximum length of 200 files (with ACR method, that's the maximum you can get), you would need to organise
8 subjective tests. This is, as I mentioned before, a HUGE number for simply comparing codecs.

An other way would be to use more complex subjective models that should give reliable enough results. The major advantage of using this kind of model compared to PSNR and SSIM metrics would be to have results corresponding better to subjective opinion. It would as well give results on a scale where it's easier to estimate the quality difference between codecs.

Sagittaire
18th May 2005, 14:42
hmmm IMO very heavy blind test is useless for video test (perhabs for audio codec but not for video codec). Like say Babayaka: "A codec which is inferior is those 3 metrics (APSNR,OPSNR and SSIM) will almost surely be seen as subjectively inferior."

Latexxx
18th May 2005, 16:04
Originally posted by Sagittaire
hmmm IMO very heavy blind test is useless for video test (perhabs for audio codec but not for video codec). Like say Babayaka: "A codec which is inferior is those 3 metrics (APSNR,OPSNR and SSIM) will almost surely be seen as subjectively inferior."
But yet guys at EBU do proper blind tests.

IgorC
19th May 2005, 21:14
Just small observation

New version of VP7.06 with sames options of tested VP7.04:

SSIM
VP7.06(srtCpuFRee=0) - 69.66
VP7.06 (wihout srtCPUfree on 2GHZ) - 69.30
http://rapidshare.de/files/1861889/2vp443.mkv.html

During playback of VP7 CPU usage is 100%. Trailer Harry Poter 720x304 450 kbit/s

I know 2 ghz CPU isn´t enough powerfull , but it´s critical to play some video of higher resolution + sound track + subtitles even on top CPU.
Even same trailer of beta Ateme High Profile has on my PC less CPU usage (between 50%-65%) and SSIM of High profile is 71.68


x264 rev233(cabac, 3pass, deblocking -1, VRB 75, refer. 16, 2 b-frames others by default) - SSIM 68.55 http://rapidshare.de/files/1862844/3x264.avi.html

babayaga
19th May 2005, 23:00
Originally posted by IgorC
Just small observation

New version of VP7.06 with sames options of tested VP7.04:

SSIM
VP7.06(srtCpuFRee=0) - 69.66
VP7.06 (wihout srtCPUfree on 2GHZ) - 69.30
http://rapidshare.de/files/1861889/2vp443.mkv.html

During playback of VP7 CPU usage is 100%. Trailer Harry Poter 720x304 450 kbit/s

I know 2 ghz CPU isn´t enough powerfull , but it´s critical to play some video of higher resolution + sound track + subtitles even on top CPU.
Even same trailer of beta Ateme High Profile has on my PC less CPU usage (between 50%-65%) and SSIM of High profile is 71.68


x264 rev233(cabac, 3pass, deblocking -1, VRB 75, refer. 16, 2 b-frames others by default) - SSIM 68.55 http://rapidshare.de/files/1862844/3x264.avi.html

VP7 also slower on decoding on my Athlon 2800+ : 36% mean load compared to 20% for H.264 files (Main or High profile).
This is for the files you provided and Ateme HP2 450k AVC HP.