[20-06-2005] Metric Benchmark Challenge ...

[akupenguin]

0.1 dB isn't negligible if it's more than the difference between XviD and DivX in your table.

[CruNcher]

ok how did you get 462 kbps for XviD now ?

| DivX          |    PP4    | 456 kbps |  6842 Ko | 41.3023 | 40.3475 |  63.70  |
| XviD          |    PP4    | 462 kbps |  6944 Ko | 41.3018 | 40.3777 |  63.90  |

XviD = 457.0 kbps
DivX = 450.0 kbps

at least the clips that are downloadable

[LordRPI]

Sagittaire,

You also switched the DivX and the XviD downloads on your first post.

[Sagittaire]

it's simple ... i want 6850 Ko (taget size in XviD) but RC for small trailer isn't very good with defaut setting and high quant for second pass encoding (perhabs too small overflow) ...

but i uptade if you want ...

[bobololo]

Originally posted by Sagittaire
6811 Ko for Nero AVC and 6852 Ko for VP7 ... it's less than 1% ... difficult to make better ...

It can easily make better You can consider this as nitpicking, but if you want the best settings providing the highest results respecting your rules (max allowable size = 6960 KiB which is equivalent to 463396 b/s), then for ND/AVC you can safely set the target bitrate to 460500 b/s. This will provide enough room for the container and the potential RC errors so that the final size won't exceed 6960 KiB).

This with the addition of the full quality mode, you'll reach tremendous values j/k of course.

edit: I agree with akupenguin, 0.1 dB for PSNR is significant.

[CruNcher]

i meant why are the bitrate of the samples don't match with the table results ?

XviD EDP H263 VHQ4 noaq_noqpel_gmc  = AVG: 41.9555 OV: 40.3158 SSIM: 63.56 BR: 450.3 kbps

FFdshow settings




XviD-450

[Sagittaire]

i have 0.06 dB for 100 Ko difference with XviD (6944 vs 6844 and 1.5%) : after update with XviD all the files will have less 0.5% difference for size ...

[sparky]

A few notes.
1. Bitrate should be specified more precisely. Variation of bitrate by 2% is roughly equal to 0.1 db difference in quality. 0.1 db is significant in our case.
2. It should be allowed to target each of the 3 quality metrics individually. Maxima for SSIM and Overall PSNR could be achieved by different codec configurations. For example, DivX deringing increases PSNR but lowers SSIM.

[Manao]

For XviD i get these results:

| XviD          |    PP4    |        |  6888Ko | 41.6998 | 40.4912 |  64.59  |

( the SSIM value changed because i computed it without lumimask the first time )

Settings were defaults + bvhq + vhq4 ( i dropped the GMC, it brings almost nothing ). Playback with ffdshow + nic's PP 4 custom, as CruNcher

CompareYV12 gives the overall, but not the average ( though you can compute it with excel, since you have all the frame's psnr )

For the different length, CruNcher thinks it's due to an outdated version of dgdecode ( for Sagittaire ).

[Sagittaire]

Originally posted by Manao
For XviD i get these results:

| XviD          |    PP4    |        |  6888Ko | 41.6998 | 40.4912 |  64.59  |

( the SSIM value changed because i computed it without lumimask the first time )

Settings were defaults + bvhq + vhq4 ( i dropped the GMC, it brings almost nothing ). Playback with ffdshow + nic's PP 4 custom, as CruNcher

CompareYV12 gives the overall, but not the average ( though you can compute it with excel, since you have all the frame's psnr )

For the different length, CruNcher thinks it's due to an outdated version of dgdecode ( for Sagittaire ).

SSIM = 64.59 ... you are sure for that ...

My .d2v is an old project (DVD2AVI 1.76) but isn't a big problem : one or two frames droped ...

[Manao]

Yes, i'm sure, and you can notice that springl, with his settings, even got an higher ssim than me.

[springl]

@Manao
..even though PSNR values are very similar to yours.
I don't understand if you get SSIM=64.59 with lumimask=True and SSIM=63.72 with lumimask=True or viceversa or something else.
@Sagittaire
I will unveil my settings later on.
If you have a webspace I will be happy to upload my files

Greetings
Springl

[Manao]

springl : when i coputed the first ssim, it was with a tweaked ( or old , i don't remember ) version of SSIM.dll which outputs SSIM & lumimasking values for each frames, but which doesn't give the average SSIM of the clip.

When i computed the first SSIM, i did only a simple average of SSIM scores, without taking lumimasking into account ( and even though lumimasking was enabled ). Moreover, some SSIM value where null ( where they should have been 1.0 ). All that gave a lower SSIM value.

So i computed SSIM again this morning with the proper SSIM.dll

[Sagittaire]

@ springl
Perhabs solutions later ...

@ Manao
I use XviD dec PP4 and DivX dec PP4 for decoding ... it's a co-dec test like say Digital56K ...

@ Cruncher
Possible to make your best for XviD sample and post them (6850 Ko +/-15 Ko and 13850 Ko +/-30 Ko) ... thx

I will update test with:
- new interval for size (perhabs bitrate +/- 0.25%)
- new XviD sample (perhabs directly with Cruncher sample)
- new Nero H264 sample (similar to recode with more accurate target bitrate)
- new Nero H264 sample with full quality (not excessible with Nero and thus not in the test but just for fun)
- new x264 sample with complete RC variability test (RC for x264 is IMO the most powerfull, simple to understand and to use ... very good job for your RC akupenguin and your codec too ... )

[Irwin]

"TMPGEnc : MPEG2 MP@ML
find size for metric equivalence with best codec
Coder:
decoder:
files:"

I'm waiting for this above

And good work for test. Vp7 is good but h264 it's better for me. (quality & speed) On my PIII 500Mhz - decode speed VP7 is totally terrible. H264 is much better.

[Manao]

Sagittaire : Did you see springl post stating that xvid & ffdshow postprocessing yield the same results ? And the emphasis and CO-DEC is meaningless to me, it's still the same norm. And that doesn't prevent you from testing x264 which can't decode.

[springl]

@Manao
When you computed SSIM again this morning with the proper SSIM.dll,did you set lumimask true or false in your avs script :
ssim(clip1,clip2,"results.csv","averageSSIM.txt",lumimask= ? ) ?
I'm trying to understand how you got your results 'cause I want to better understand the reliability of mine.

Greetings
Springl

[Manao]

I set it to true.

[springl]

thank you for your quick reply
Springl

[Sagittaire]

One2tech speack about overhead problem in PM ... it's a problem (.rmvb, .avi, .mp4 and .wmv). Perhabs Matroska container for all codec could be better for regular test ... ???