[20-06-2005] Metric Benchmark Challenge ...

[Sagittaire]

~ Movie Metric Benchmark Challenge ~

You think that your codec is the best : prove that ... !!?


A - Introduction

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...



B - Rules

Rule 1 : Source

You must use this source for the test : Harry Potter II trailer, 720*576 MPEG2 MP@ML 4.5 Mbps
HPII Part1 winrar (http://multimediacom.free.fr/Download/HPII.part1.rar), HPII Part2 winrar (http://multimediacom.free.fr/Download/HPII.part2.rar), HPII Part3 winrar (http://multimediacom.free.fr/Download/HPII.part3.rar), HPII Part4 winrar (http://multimediacom.free.fr/Download/HPII.part4.rar) & HPII Part5 winrar (http://multimediacom.free.fr/Download/HPII.part5.rar)


Rule 2 : Pre-process and encoding

You must use DGindex with 32-bit SSE/MMX iDCT algorithm for .d2v project and DGDecode.dll for MPEG2 decoding in avisynth scipt. You must use VirtualDub or VirtualDubMod in "fast compress" mod for VFW codec. You must use YV12 space color for complete process with all codec. You must use this avisynth script for make encoding:

source=Mpeg2Source("C:\...\HPII.d2v",idct=2)
source=Trim(source,70,3145)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)
Return(source)



Rule 3 : Bitrate

You must use 2 bitrates for encoding. The first use XviD in quant8 for reference, it's the "Streaming Quality". The second use XviD in quant4 for reference, it's the "1CDR Quality". You can use container of your choice for all Video Elementary Stream.

Streaming: 450 Kbps for Video Elementary Stream and container or 6750 Ko with +/- 0.30 % for tolerance. The size must be in [6730 Ko - 6770 Ko] interval.

1CDR: 900 Kbps for Video Elementary Stream and container or 13500 Ko with +/- 0.30 % for tolerance. The size must be in [13460 Ko - 13540 Ko] interval.


Rule 4 : Metric Tests

SSIM (YV12): IMO the best metric. it uses some HVS properties but not all ...
PDF description for VSSIM (http://www.cns.nyu.edu/~zwang/files/papers/vssim.pdf)

Overall PSNR (YV12): Very good metric test, particullary for Rate Control Quality test ...
MSE = Sum t = 1 -> NFrame Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (NFrame*width*height)
Overall PSNR = 10 * Log ( 255² / MSE)

Average PSNR (YV12):The worse metric test but Frame PSNR graph is very good to compare RC strategy ...
MSE = Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (width*height)
Average PSNR = Sum t = 1 -> NFrame 10 * Log ( 255² / MSE) / Nframe

Here (http://multimediacom.free.fr/Download/Test.rar) the test archives if you want all original Avisynth script for metric test.


SSIM vs PSNR

"For a given reference video and a given compressed video, it is meant to compute a quality metric, based on perceived visual distortion. Unlike the well-known PSNR measure, it's not purely mathematical, and should correlate much better with human vision."

"A higher MSE (and so lower PSNR) should mean that the compressed clip is a worse image but MSE and PSNR are flawed in this respect as numerous tests have shown. However with SSIM, according to tests carried out on the VQEG dataset, a higher Q (SSIM value) has a much better relation to the visual quality of the compressed clip. Despite this, bear in mind the SSIM metric still isn't perfect."

Here (http://multimediacom.free.fr/Download/SSIM-0.24.rar) last AviSynth plugin for VSSIM in YV12


Average PSNR vs Overall PSNR

It's easy to increase artificially Average PSNR with Rate Control tweak. In certain case CBR (0% for variability) will be better than real VBR (100% for variability or constant quant). It's generaly not possible with Overall PSNR. Overall PSNR is very good to test Rate Control quality. If difference between Average PSNR and Overall PSNR is very high, "local quality difference" (difference between Pframe or Bframe for exemple) or "scene quality difference" (difference between low motion and high motion for exemple) will be too high and Rate Control for this codec not very good.

Here (http://multimediacom.free.fr/Download/CompareYV12.zip) last AviSynth plugin for Overall PSNR in YV12


You must mux or remux x264, VP6, VP7, WMV9, DivX6 and XviD in avi container (with VDM for example) to solve problem with color space output, synchronisation or acquisition for metric test. You must use AviSource in avisynth for VP6, VP7 and WMV9. You must use DirectShowSource in avisynth for MPEG4 ASP, MPEG4 AVC and RV10. You must change trim for frame synchronization or for clip length if you have some probleme with SSIM plugin. You must use these AviSynth type script for metric test:

# --> Source Opening <--
source=Mpeg2Source("G:\...\HPII.d2v",idct=2)
source=Trim(source,70,3145)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)

# --> Video Opening <--
video=DirectShowSource("G:\...\DivX6-450.avi",fps=25)

# --> PSNR analysis <--
compareYV12(video,source,"YUV","PSNR-DivX6-450.log")



# --> Source Opening <--
source=Mpeg2Source("G:\...\HPII.d2v",idct=2)
source=Trim(source,70,3144)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)

# --> Video Opening <--
video=DirectShowSource("G:\...\RV10-450.mkv",fps=25)

# --> SSIM analysis <--
return SSIM(source,video,"results.csv","SSIM-RV10-450.txt",lumimask=2)



C - Results

At this time here my result ... but done better if you can ... :devil:

|--------------|---------|---------|---------|---------|---------|---------|
| Codec | PProc | Bitrate | Size | OPSNR | SSIM 0 | SSIM 2 |
|--------------|---------|---------|---------|---------|---------|---------|
| DivX6 ASP | PP4 | 446 | 6757 | 39.85 | 70.17 | 65.02 |
| XviD ASP | PP4 | 446 | 6756 | 39.66 | 69.55 | 64.33 |
| LAVC ASP | PP4 | 447 | 6775 | 39.71 | 69.51 | 63.82 |
| Nero AVC HP | PP0 | 446 | 6747 | 41.10 | 76.30 | 72.10 |
| x264 AVC HP | PP0 | 446 | 6747 | 40.85 | 74.82 | 70.40 |
| Nero AVC MP | PP0 | 446 | 6747 | 40.72 | 74.65 | 70.25 |
| x264 AVC MP | PP0 | 446 | 6750 | 40.64 | 73.83 | 69.23 |
| VP7 | PP2 | 446 | 6760 | 40.84 | 74.42 | 70.05 |
| VP6 | PP4 | 446 | 6769 | 40.28 | 71.86 | 66.89 |
| WMV9 | PP1 | 446 | 6746 | 39.51 | 69.58 | 64.07 |
| RV10 | HF2 | 446 | 6743 | 39.84 | 69.86 | 64.60 |
| DivX3 | PP4 | 447 | 6771 | 38.70 | 65.11 | 58.56 |
|--------------|---------|---------|---------|---------|---------|---------|
| DivX6 ASP | PP4 | 896 | 13493 | 42.78 | 81.14 | 77.99 |
| XviD ASP | PP4 | 896 | 13489 | 42.66 | 80.99 | 77.96 |
| LAVC ASP | PP4 | 897 | 13528 | 42.77 | 80.95 | 77.77 |
| Nero AVC HP | PP0 | 896 | 13490 | 43.70 | 84.39 | 81.54 |
| x264 AVC HP | PP0 | 896 | 13491 | 43.46 | 83.27 | 80.50 |
| Nero AVC MP | PP0 | 896 | 13491 | 43.43 | 83.45 | 80.63 |
| x264 AVC MP | PP0 | 896 | 13498 | 43.26 | 82.66 | 79.75 |
| VP7 | PP2 | 897 | 13534 | 43.34 | 82.75 | 80.02 |
| VP6 | PP4 | 896 | 13481 | 42.99 | 81.39 | 78.41 |
| WMV9 | PP1 | 896 | 13510 | 42.52 | 79.88 | 76.73 |
| RV10 | HF2 | 896 | 13493 | 42.77 | 80.82 | 77.72 |
| DivX3 | PP4 | 896 | 13508 | 41.84 | 78.27 | 74.40 |
|--------------|---------|---------|---------|---------|---------|---------|
SSIM 0: Lumimask Off
SSIM 1: Lumimask On (Original Lumimask)
SSIM 2: Lumimask On (One2Tech Patch)



MPEG4 ASP : DivX6
Coder: DivX6 Codec
Setting: 2 pass, Unconstrained, Insame Quality, Multi-Bframe, H263 optmized, BM = 0.1
decoder: DivX decoder with PP4
files: DivX 450 Kbps (http://multimediacom.free.fr/Video/DivX6-450.mkv) & DivX 900 Kbps (http://multimediacom.free.fr/Video/DivX6-900.mkv)


MPEG4 ASP : XviD
Coder: Koepi XviD 1.1.0 beta2
Setting: 2 pass, Unconstrained, Motion Ultra-High, VHQ4, BVHQ, Chroma Motion, Trellis, Adaptative Quantisation, GMC, BFrame 2/1.50/1.00
decoder: XviD decoder with PP4
files: XviD 450 Kbps (http://multimediacom.free.fr/Video/XviD-450.mkv) & XviD 900 Kbps (http://multimediacom.free.fr/Video/XviD-900.mkv)


MPEG4 ASP : Libavcodec
Coder: MEncoder dev-CVS-050426-18:49-3.4.2
Setting: 3 pass, adaptative BFrame 2/1.50/1.00, Chroma Motion, Trellis, RDO, SAD, 6 diamond size
decoder: ffdshow decoder with PP4
files: Libavcodec ASP 450 Kbps (http://multimediacom.free.fr/Video/LAVC-450.mkv) & Libavcodec ASP 900 Kbps (http://multimediacom.free.fr/Video/LAVC-900.mkv)


DivX 3.11 : Libavcodec
Coder: MEncoder dev-CVS-050426-18:49-3.4.2
Setting: 3 pass, Chroma Motion, RDO, SAD, 6 diamond size
decoder: ffdshow decoder with PP4
files: DivX3 450 Kbps (http://multimediacom.free.fr/Video/DivX3-450.mkv) & DivX3 900 Kbps (http://multimediacom.free.fr/Video/DivX3-900.mkv)


MPEG4 AVC : Ateme AVC Main Profil
Coder: Ateme encavc 1.2.0.17
Setting: 2 pass, Quality Full, 8 Ref, 8 BRef, 2 Bframes, CABAC, BPred, MPart, WPred, Inloop strength 0, Chroma, psy 3
decoder: Ateme AVC decoder
files: Nero AVC Main Profil 450 Kbps (http://multimediacom.free.fr/Video/NDAVCMP-450.mp4) & Nero AVC Main Profil 900 Kbps (http://multimediacom.free.fr/Video/NDAVCMP-900.mp4)


MPEG4 AVC : Ateme AVC High Profil
Coder: Ateme encavc 1.2.0.14
Setting: 2 pass, Quality Full, 16 Ref, 16 BRef, 2 Bframes, CABAC, BPred, MPart, WPred, Inloop strength 0, Chroma, psy 3, transform 8x8
decoder: Ateme AVC decoder
files: Nero AVC High Profil 450 Kbps (http://multimediacom.free.fr/Video/NDAVCHP-450.mp4) & Nero AVC High Profil 900 Kbps (http://multimediacom.free.fr/Video/NDAVCHP-900.mp4)


MPEG4 AVC : x264 Main Profil
Coder: x264 VFW rev 264
Setting: 3 pass, variabilty 75%, Quality Max, 16 Ref frames, 2 Bframes, CABAC, WPred, Inloop deblock strength -1, RDO
decoder: ffdshow H264 decoder
files: x264 Main Profil 450 Kbps (http://multimediacom.free.fr/Video/x264MP-450.mp4) & x264 Main Profil 900 Kbps (http://multimediacom.free.fr/Video/x264MP-900.mp4)


MPEG4 AVC : x264 High Profil
Coder: x264 VFW rev 264
Setting: 3 pass, variabilty 75%, Quality Max, 16 Ref frames, 2 Bframes, CABAC, WPred, Inloop deblock strength -1 , 8x8dct, RDO
decoder: ffdshow H264 decoder
files: x264 High Profil 450 Kbps (http://multimediacom.free.fr/Video/x264HP-450.mp4) & x264 High Profil 900 Kbps (http://multimediacom.free.fr/Video/x264HP-900.mp4)


One2Tech : VP7
Coder: VP7 VFW Codec 7.0.0.7
Setting: 2 pass, best quality, VBR, Noisered 0, Sharpeness 0, Default Rate Control setting
decoder: VP7 decoder PP2
files: VP7 450 Kbps (http://multimediacom.free.fr/Video/VP7-450.mkv) & VP7 900 Kbps (http://multimediacom.free.fr/Video/VP7-900.mkv)


One2Tech : VP6
Coder: VP6 VFW Codec 6.4.1.0
Setting: 2 pass, best quality, VBR, Noisered=0, Sharpeness=0, MinQ 6/12, MaxQ 56, Vbr bias 70, Vbr min 30, Vbr max 200, Undershoot 95%
decoder: VP6 decoder PP4
files: VP6 450 Kbps (http://multimediacom.free.fr/Video/VP6-450.mkv) & VP6 900 Kbps (http://multimediacom.free.fr/Video/VP6-900.mkv)


Microsoft : WMV9
Coder: WMV9 VCM
Setting: 2 pass VBR, best quality, decoder complex
decoder: WMV3 DMO decoder PP1
files: WMV9 450 Kbps (http://multimediacom.free.fr/Video/WMV9-450.mkv) & WMV9 900 Kbps (http://multimediacom.free.fr/Video/WMV9-900.mkv)


RealNetwork : RV10
Coder: erv4 10.0.0.2
Setting: 2 pass, EHQ 100, old Rate Control, chroma
decoder: RV10 decoder HFE2
files: RV10 450 Kbps (http://multimediacom.free.fr/Video/RV10-450.mkv) & RV10 900 Kbps (http://multimediacom.free.fr/Video/RV10-900.mkv)


MPEG2 MP@ML : MEncoder dev-CVS-050426-18:49-3.4.2
find size for metric equivalence with best codec for quality "1CDR"
"Quality" must be at ~ 43 dB for OPSNR or at ~ 80 for SSIM
files: MPEG2 1250 Kbps (http://multimediacom.free.fr/Video/MPEG2-1250.m2v)



D - Conclusion

At this time for this trailer

MPEG4 AVC codecs: NDAVC vs x264
Ateme H264 and x264 are comparable codec for OPSNR but not for SSIM (visually NDAVC is IMO better than x264 too). x264 is the best Open Source Codec and XviD can't fight with the last version.

MPEG4 ASP codecs: XviD vs DivX vs LAVC
DivX6 is the best MPEG4 ASP codec for OPSNR and SSIM but XviD, DivX6 and Libavcodec ASP are very close codec.

Proprietary "free" codecs: RV10 vs VP6 vs WMV9
VP6 is always better than RV10 or WMV9 for OPSNR or SSIM. VP6 are better than all MPEG4 ASP codec for OPSNR and SSIM. RV10 is an excellent codec for high quant encoding (low quality or "low bitrate").

Higest performance codecs: VP7 vs NDAVC vs x264
VP7, Ateme AVC or x264 are really better than the others codecs. VP7 seem better than Ateme H264 Main Profil for very high quant (low quality or "low bitrate") and Ateme H264 Main Profil seem better than VP7 for lower quant (high quality or "high bitrate"). x264 High Profil is the best codec actually available. Ateme H264 is by far the best codec in this test ...

MPEG2 vs all
The best codecs (H264 or VP7) are 50% better than MPEG2 MP@ML but MPEG2 MP@ML with good encoder is always a good codec. IMO Mencoder is by far the best MPEG2 encoder for low bitrate ...



Annexe - Update

14.07.05
- Ateme codec Full quality update

19.06.05
- Ateme codec Full quality
- VP7 7.0.0.7 better RC

17.06.05
- x264 rev 263 Main Profil
- x264 rev 263 High Profil
- DivX6
- VP7 7.0.0.7
- MPEG2 MP@ML with Mencoder

08.06.05
- x264 rev 253 Main Profil
- x264 rev 253 High Profil

24.05.05
- SSIM plugin
- Nero Digital High Profil for demo only

22.05.05
- x264 rev 232
- VP7 7.0.0.6
- Libavcodec MPEG4 ASP (MEncoder dev-CVS-050426-18:49-3.4.2)
- Libavcodec DivX 3.11 (MEncoder dev-CVS-050426-18:49-3.4.2)
- XviD 1.1.0 Beta2

03.04.05
- x264 rev 184
- VP7 7.0.0.4

[Manao]

You encoded XviD at fixed quantizer. Encoding it two-passes will give you a higher psnr. Furthermore, you stated 450 kbps / 6882 KB, which isn't coherent ( 1024 / 1000 issue ). You're better off targetting 458kbps ( the difference been greater than the error margin, those 8 kbps are important ).

I made the test with XviD @ '450' kbps ( actually, in order to hit the wanted bitrate, i had to type 480 in the bitrate box ). I obtained a psnr of 40.49 dB overall ( 41.69 average ).

Edit : btw, the loss implied by doing a constant quantizer encoding is 0.8 dB, which is _huge_. I hope nobody still believe that constant quantizer yields better quality that a two passes encoding.

[Sagittaire]

Originally posted by Manao
You encoded XviD at fixed quantizer. Encoding it two-passes will give you a higher psnr.

For me priority are in order:
- SSIM
- Overall PSNR
- Average PSNR


Originally posted by Manao
Furthermore, you stated 450 kbps / 6882 KB, which isn't coherent ( 1024 / 1000 issue ). You're better off targetting 458kbps ( the difference been greater than the error margin, those 8 kbps are important ).

Yes ... it's true ... 458 Kbps +/-1% and 919 Kbps +/-1% exactly (I edit)


Originally posted by Manao
I made the test with XviD @ '450' kbps ( actually, in order to hit the wanted bitrate, i had to type 480 in the bitrate box ). I obtained a psnr of 40.49 dB overall ( 41.69 average ).
Yes I use one pass q4 (q4 for I and P, q6 for B) ... but your encoding seem very good (perhabs not for SSIM) ... you can post this file or your setting ...

[Manao]

SSIM : 63.72
Settings : XviD defaults + bvhq + vhq4 + GMC

Btw, you praise psnr overall over psnr average when it comes to rate control comparison, however ssim is computed as average psnr is, so ssim should only be used to compare frames together, not to compare rate control algorithms.

Edit : forgot : playback with ffdshow Nic's PP4

[springl]

My results with Xvid:

Streaming.avi
size: 6894 KB
SSIM_PP4: 64.86

1CD.avi
size: 13806 KB
SSIM_PP4: 78.27

Settings :
XviD defaults + bvhq + vhq4 + GMC + MIN/MAX I/P/B frame 2/31 + some tuning in
overflow treatment and curve compression.

playback with ( Xvid decoder + PP4 ) or ( ffdshow + Nic's PP4 ) will produce same results.

Greetings :)
Springl

[Manao]

Sagittaire :

CruNcher and I doublechecked your psnr results for your XviD @ 450 kbps. We obtain an overall of 40.40 and a SSIM of 63.94. How did you manage to get that wrong ? What can we assume for the other figures for the other codecs ?

[bobololo]

@Sagittaire: Just a stupid question, why don't you use compareYV12() instead of compare() ? Don't all the codecs here work natively in YUV 4:2:0 colorspace ?

[LordRPI]

Sagittaire,

Would it be possible to list the codec settings you used for all comparisons?

[Sagittaire]

@Bobololo
compare (http://www.avisynth.org/Compare) is only YUY2 or RGB compatible in avisynth ... i don't understand why ... perhabs function begin in avisynth YUY2 2.0x version but avisynth 2.5x is YV12 now ...

You can confirm my result with your AVC codec and with my test protocol ... perhabs you can make better encoding ... ???


@ Manao
Problem seem to be ffdshow for MPEG4 ASP class codec ... I will use XviD decoder and DivX decoder now ... update in progress


@LordRPI
update in progress ...


@springl
please if possible post your encoding or your setting with complete reg file for exemple:


Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\GNU\XviD]
"mode"=dword:00000002
"bitrate"=dword:000001ca
"desired_size"=dword:00001ac2
"use_2pass_bitrate"=dword:00000000
"desired_quant"=dword:00000190
"quant_type"=dword:00000000
"lum_masking"=dword:00000000
"interlacing"=dword:00000000
"tff"=dword:00000000
"qpel"=dword:00000000
"gmc"=dword:00000000
"use_bvop"=dword:00000001
"max_bframes"=dword:00000002
"bquant_ratio"=dword:00000096
"bquant_offset"=dword:00000064
"packed"=dword:00000001
"ar_mode"=dword:00000000
"aspect_ratio"=dword:00000000
"par_x"=dword:00000001
"par_y"=dword:00000001
"ar_x"=dword:00000004
"ar_y"=dword:00000003
"num_zones"=dword:00000001
"rc_reaction_delay_factor"=dword:00000010
"rc_averaging_period"=dword:00000064
"rc_buffer"=dword:00000064
"discard1pass"=dword:00000001
"full1pass"=dword:00000000
"keyframe_boost"=dword:0000000a
"kfreduction"=dword:00000014
"kfthreshold"=dword:00000001
"curve_compression_high"=dword:00000000
"curve_compression_low"=dword:00000000
"overflow_control_strength"=dword:00000005
"twopass_max_overflow_improvement"=dword:00000005
"twopass_max_overflow_degradation"=dword:00000005
"container_type"=dword:00000001
"target_size"=dword:000a2800
"subtitle_size"=dword:00000000
"hours"=dword:00000001
"minutes"=dword:0000001e
"seconds"=dword:00000000
"fps"=dword:00000002
"audio_mode"=dword:00000000
"audio_type"=dword:00000000
"audio_rate"=dword:00000080
"audio_size"=dword:00000000
"motion_search"=dword:00000006
"vhq_mode"=dword:00000004
"vhq_bframe"=dword:00000001
"chromame"=dword:00000001
"turbo"=dword:00000000
"max_key_interval"=dword:0000012c
"frame_drop_ratio"=dword:00000000
"min_iquant"=dword:00000001
"max_iquant"=dword:0000001f
"min_pquant"=dword:00000001
"max_pquant"=dword:0000001f
"min_bquant"=dword:00000001
"max_bquant"=dword:0000001f
"trellis_quant"=dword:00000001
"fourcc_used"=dword:00000000
"debug"=dword:00000000
"vop_debug"=dword:00000000
"display_status"=dword:00000001
"Brightness"=dword:00000000
"Deblock_Y"=dword:00000001
"Deblock_UV"=dword:00000001
"Dering_Y"=dword:00000000
"Dering_UV"=dword:00000000
"FilmEffect"=dword:00000000
"profile"="(unrestricted)"
"stats"="\\video.pass"
"qmatrix_intra"=hex:08,11,12,13,15,17,19,1b,11,12,13,15,17,19,1b,1c,14,15,16,\
17,18,1a,1c,1e,15,16,17,18,1a,1c,1e,20,16,17,18,1a,1c,1e,20,23,17,18,1a,1c,\
1e,20,23,26,19,1a,1c,1e,20,23,26,29,1b,1c,1e,20,23,26,29,2d
"qmatrix_inter"=hex:10,11,12,13,14,15,16,17,11,12,13,14,15,16,17,18,12,13,14,\
15,16,17,18,19,13,14,15,16,17,18,1a,1b,14,15,16,17,19,1a,1b,1c,15,16,17,18,\
1a,1b,1c,1e,16,17,18,1a,1b,1c,1e,1f,17,18,19,1b,1c,1e,1f,21
"zone0_frame"=dword:00000000
"zone0_mode"=dword:00000000
"zone0_weight"=dword:00000064
"zone0_quant"=dword:00000190
"zone0_type"=dword:00000000
"zone0_greyscale"=dword:00000000
"zone0_chroma_opt"=dword:00000000
"zone0_bvop_threshold"=dword:00000000
"zone0_cartoon_mode"=dword:00000000
"ForceColorspace"=dword:00000001
"FlipVideo"=dword:00000000
"Supported_4CC"=dword:00000000
"Videoinfo_Compat"=dword:00000000

[DigitAl56K]

Which settings were used for DivX? I might be missing something obvious here, but I don't see them. Did you constrain all codecs via VBV, or was this test a mixed bag? Was there any emphasis on comparible testing (e.g. time to encode, CE compatibility, etc.) or was every codec simply allowed to run wild with the features it used?

What were the settings used for WM9? I'm surprised at this result.

Also, when you use a decoder you really ought to use the decoder for the codec you are testing, i.e. use the DivX decoder for DivX, the XVID decoder for XVID, etc. Each codec may implement different technologies in their decoder and these are no less a part of the format than any encode-side features. For DivX we recommend full deblock, no dering, no frame delay, no film effect. We notice you also used DirectShowSource. When using this input filter please ensure that the DivX Decoder is configured so that it does not overlay the DivX Video logo during decoding.

[sparky]

Streaming: 458 Kbps or 6885 Ko

Just to clarify. Do you mean 458 kbps = 458000 bps = (458000/8)*(3076/25) bytes = 7044040 bytes = 6879 Kb, and 919 kbps = 919000 bps = 14134220 bytes = 13803 Kb?

Also, you have Trim(70,3144) in SSIM script, and Trim(70,3145) in all other scripts. It causes an error ( SSIM won't work with different-length clips ). You want to change 3144->3145.

[Sagittaire]

@ DigitAl56K
problem with ffdshow for MPEG4 ASP class codec (i use DivX dec and XviD dec now ...) and I update bench for MPEG4 ASP codec ...

For WMV9 I use best quality with complex dec in 2 two pass VBR. I use PP1 DMO (http://multimediacom.free.fr/Video/WMVPostpross.exe) for Post-process ...

For DivX I use unconstrained, insame quality, H263, multibframe, GMC, Bitrate Modultion = 0.25. You can propose your encoding with setting used if you want ...


Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\DivXNetworks\DivX4Windows]
"Auto PP Level"=dword:00000000
"Auto PP"=dword:00000000
"Disable PP"=dword:00000000
"Dering PP Level"=dword:00000000
"Deblock PP Level"=dword:0000000f
"Warmth Strength"=dword:00000002
"Warmth Filter"=dword:00000000
"Postprocessing"=dword:00000028
"Brightness"=dword:00000000
"Contrast"=dword:00000000
"Saturation"=dword:00000000
"AspectRatio_X"=dword:00000000
"AspectRatio_Y"=dword:00000000
"Smooth Playback"=dword:00000000
"Extended Yuv Mode"=dword:00000001
"Double Buffering"=dword:00000000
"Extended Overlay Mode"=dword:00000000
"Disable Logo"=dword:00000001
"ATI Disable"=dword:00000000
"Support Generic Mpeg-4"=dword:00000001
"Bitrate"=dword:000be6e0
"VBV Channel Bitrate"=dword:006a1120
"VBV Buffer Size"=dword:00300000
"VBV Occupancy"=dword:00240000
"Complexity Modulation"="0.000000"
"Max Keyframe Interval"=dword:0000012c
"Quality"=dword:00000005
"VBR Mode"=dword:00000000
"VBR Quality v5"=dword:00000028
"Frame Dropping"=dword:00000000
"Deinterlace"=dword:00000000
"CPU Limit"=dword:00000064
"Use CPU Limit"=dword:00000000
"Data Partitioning"=dword:00000000
"Bidirectional Encoding"=dword:00000001
"Scene change threshold"=dword:00000032
"do_add"=dword:00000001
"do_mc"=dword:00000001
"do_mc_b"=dword:00000001
"show_gmc"=dword:00000000
"no_vbr_rc"=dword:00000000
"use_mvfile"=dword:00000000
"do_writeyuv"=dword:00000000
"do_writeyuvmul"=dword:00000000
"Psychovisual Enhancement"=dword:00000000
"Psychovisual Strength - Frame Level"="0.200000"
"Psychovisual Strength - Macroblock level"="0.200000"
"Write MP4 file"=dword:00000000
"Use GMC"=dword:00000000
"Quarter-pel"=dword:00000000
"Crop"=dword:00000000
"Resize"=dword:00000000
"Crop left"=dword:00000000
"Crop right"=dword:00000000
"Crop top"=dword:00000000
"Crop bottom"=dword:00000000
"Resize width"=dword:00000280
"Resize height"=dword:000001e0
"Resize mode"=dword:00000000
"Bicubic spline B"="0.000000"
"Bicubic spline C"="0.500000"
"Temporal enable"=dword:00000000
"Spatial passes"=dword:00000000
"Temporal level"="0.000000"
"Spatial level"="0.000000"
"Interlace mode"=dword:00000000
"Use dialogs"=dword:00000000
"Disable feedback"=dword:00000001
"Write log file"=dword:00000000
"5.0 Psychovisual Mode"=dword:00000000
"5.0 Preprocessing Mode"=dword:ffffffff
"5.0 Resize Mode"=dword:00000001
"Write nth pass logfile"=dword:00000000
"Disable Profile Page"=dword:00000000
"Active Profile"=dword:00000002
"Nth Pass debug file name"="c:\\newrc.txt"
"Log File Name"="c:\\divx.log"
"MP4 File Name"="c:\\test.divx"
"YUV Dir Name"="c:\\yuv"
"MV File Name"="c:\\mvinfo.bin"
"Top field first"=dword:00000001
"MPEG Quant"=dword:00000000
"Rotate Artwork"=dword:00000001
"Check for Updates"=dword:00000001
"Disable Notifications"=dword:00000000
"Intra quant matrix"=hex:00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00
"Inter quant matrix"=hex:00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00
"Feedback refresh"=dword:00000001
"Force Color Mode"=dword:00000000


command line:
-bvnn 915000 -vbv 27336400,100663296,75497472 -pq 8610 -g -complexity_modulation=0.25

[Sagittaire]

Originally posted by sparky
Just to clarify. Do you mean 458 kbps = 458000 bps = (458000/8)*(3076/25) bytes = 7044040 bytes = 6879 Kb, and 919 kbps = 919000 bps = 14134220 bytes = 13803 Kb?

Also, you have Trim(70,3144) in SSIM script, and Trim(70,3145) in all other scripts. It causes an error ( SSIM won't work with different-length clips ). You want to change 3144->3145.

yes for bitrate ...

not explain for variable length clip ... clip for SSIM function must have the same length and not for compare function. Length problem is perhabs bframe problem for MPEG4 ASP, for RV10 i don't know ... but it isn't big problem with trim modification ...

[bobololo]

Originally posted by Sagittaire
@Bobololo
compare (http://www.avisynth.org/Compare) is only YUY2 or RGB compatible in avisynth ... i don't understand why ... perhabs function begin in avisynth YUY2 2.0x version but avisynth 2.5x is YV12 now ...

You can confirm my result with your AVC codec and with my test protocol ... perhabs you can make better encoding ... ???

For native YV12 PSNR computation, try compareYV12 (http://www.avisynth.org/warpenterprises/files/compareyv12_25_dll_20030724.zip).

Regarding your result with our codec, let me know your settings first. Also, there is an undocumented quality level "full" which is insanely slow but that can surely achieve slightly higher figures ;)

[Sagittaire]

Originally posted by bobololo
For native YV12 PSNR computation, try compareYV12 (http://www.avisynth.org/warpenterprises/files/compareyv12_25_dll_20030724.zip).

Regarding your result with our codec, let me know your settings first. Also, there is an undocumented quality level "full" which is insanely slow but that can surely achieve slightly higher figures ;)

yes, I know but just Average PSNR with this plugin ... :(
YUY2 test order is not different to YV12 ... there are good scaling with result between YUY2 and YV12 test ... perhabs Average PSNR in YV12 and Overall PSNR in YUY2 will be good solution ...


for your AVC codec I use "my best setting" ... lol
encavc.exe -i azerty.avs -o clip.mp4 -qual extra -rcmode 1st -log 1pass.log -br 450000 -deblock 0 -ref 5 -setef wpred -cartoon -psy 1 -priority idle
encavc.exe -i azerty.avs -o clip.mp4 -qual extra -rcmode 2nd -log 1pass.log -br 450000 -deblock 0 -ref 5 -setef wpred -cartoon -psy 1 -priority idle


Also, there is an undocumented quality level "full" which is insanely slow but that can surely achieve slightly higher figures ;)

[méthode couet=On]
i want this command line ... i want this command line ... i want this command line ... lol
[méthode couet=Off]

[Sagittaire]

XviD update:
new sample, new bench, new graph

@Bobololo

-qual full ... ah yes ... 0.16 fps ... lol

[French=On]
je comprend vite mais faut quand même m'expliquer longtemps ... lol
[French=Off]

[bobololo]

Originally posted by Sagittaire
yes, I know but just Average PSNR with this plugin ... :(
YUY2 test order is not different to YV12 ... there are good scaling with result between YUY2 and YV12 test ... perhabs Average PSNR in YV12 and Overall PSNR in YUY2 will be good solution ...


Ah you're right, I didn't notice that compareYV12 doesn't report overall PSNR. That's really weird it isn't included. It should be easy to add it. I'll try to make tomorrow.

for your AVC codec I use "my best setting" ... lol
encavc.exe -i azerty.avs -o clip.mp4 -qual extra -rcmode 1st -log 1pass.log -br 450000 -deblock 0 -ref 5 -setef wpred -cartoon -psy 1 -priority idle
encavc.exe -i azerty.avs -o clip.mp4 -qual extra -rcmode 2nd -log 1pass.log -br 450000 -deblock 0 -ref 5 -setef wpred -cartoon -psy 1 -priority idle


I believe the target bitrate was 458000 bits/s, am I wrong ?

[méthode couet=On]
i want this command line ... i want this command line ... i want this command line ... lol
[méthode couet=Off]

I explicitly gave it ;)

[Sagittaire]

[i]Originally posted by bobololo

I believe the target bitrate was 458000 bits/s, am I wrong ?

I explicitly gave it ;)

6811 Ko for Nero AVC and 6852 Ko for VP7 ... it's less than 1% ... difficult to make better ... ;)

[CruNcher]

Sagittaire please keep the values correct the XviD-450.avi is not 450 kbps as stated in the table it's 457.0 kbps

[Sagittaire]

All my files are upload and hyperlink in post ...

And 450 must be 450 Kbit/s too (like for VP6 or WMV9) ... in fact the most important for size/bitrate is to respect [6810 Ko - 6960 Ko] interval and [13660 Ko - 13940 Ko] interval. With same codec/setting difference between 2% for size is very small for metric (perhabs 0.1 dB for PSNR and 0.1 for SSIM)

[akupenguin]

0.1 dB isn't negligible if it's more than the difference between XviD and DivX in your table.

[CruNcher]

ok how did you get 462 kbps for XviD now ?


| DivX | PP4 | 456 kbps | 6842 Ko | 41.3023 | 40.3475 | 63.70 |
| XviD | PP4 | 462 kbps | 6944 Ko | 41.3018 | 40.3777 | 63.90 |


XviD = 457.0 kbps
DivX = 450.0 kbps

at least the clips that are downloadable

[LordRPI]

Sagittaire,

You also switched the DivX and the XviD downloads on your first post.

[Sagittaire]

it's simple ... i want 6850 Ko (taget size in XviD) but RC for small trailer isn't very good with defaut setting and high quant for second pass encoding (perhabs too small overflow) ...

but i uptade if you want ... :D

[bobololo]

Originally posted by Sagittaire
6811 Ko for Nero AVC and 6852 Ko for VP7 ... it's less than 1% ... difficult to make better ... ;)

It can easily make better ;) You can consider this as nitpicking, but if you want the best settings providing the highest results respecting your rules (max allowable size = 6960 KiB which is equivalent to 463396 b/s), then for ND/AVC you can safely set the target bitrate to 460500 b/s. This will provide enough room for the container and the potential RC errors so that the final size won't exceed 6960 KiB).

This with the addition of the full quality mode, you'll reach tremendous values ;) j/k of course.

edit: I agree with akupenguin, 0.1 dB for PSNR is significant.

[CruNcher]

i meant why are the bitrate of the samples don't match with the table results ?


XviD EDP H263 VHQ4 noaq_noqpel_gmc = AVG: 41.9555 OV: 40.3158 SSIM: 63.56 BR: 450.3 kbps

FFdshow settings
http://cruncher.mufflastig.com/XviD/samples/pp4.png

http://cruncher.mufflastig.com/XviD/samples/psnr.png

XviD-450 (http://cruncher.mufflastig.com/XviD/samples/XviD-450.3-edp-h263.avi)

[Sagittaire]

i have 0.06 dB for 100 Ko difference with XviD (6944 vs 6844 and 1.5%) : after update with XviD all the files will have less 0.5% difference for size ... :devil:

[sparky]

A few notes.
1. Bitrate should be specified more precisely. Variation of bitrate by 2% is roughly equal to 0.1 db difference in quality. 0.1 db is significant in our case.
2. It should be allowed to target each of the 3 quality metrics individually. Maxima for SSIM and Overall PSNR could be achieved by different codec configurations. For example, DivX deringing increases PSNR but lowers SSIM.

[Manao]

For XviD i get these results:

| XviD | PP4 | | 6888Ko | 41.6998 | 40.4912 | 64.59 |
( the SSIM value changed because i computed it without lumimask the first time )

Settings were defaults + bvhq + vhq4 ( i dropped the GMC, it brings almost nothing ). Playback with ffdshow + nic's PP 4 custom, as CruNcher

CompareYV12 gives the overall, but not the average ( though you can compute it with excel, since you have all the frame's psnr )

For the different length, CruNcher thinks it's due to an outdated version of dgdecode ( for Sagittaire ).

[Sagittaire]

Originally posted by Manao
For XviD i get these results:

| XviD | PP4 | | 6888Ko | 41.6998 | 40.4912 | 64.59 |
( the SSIM value changed because i computed it without lumimask the first time )

Settings were defaults + bvhq + vhq4 ( i dropped the GMC, it brings almost nothing ). Playback with ffdshow + nic's PP 4 custom, as CruNcher

CompareYV12 gives the overall, but not the average ( though you can compute it with excel, since you have all the frame's psnr )

For the different length, CruNcher thinks it's due to an outdated version of dgdecode ( for Sagittaire ).

SSIM = 64.59 ... you are sure for that ... :confused:

My .d2v is an old project (DVD2AVI 1.76) but isn't a big problem : one or two frames droped ...

[Manao]

Yes, i'm sure, and you can notice that springl, with his settings, even got an higher ssim than me.

[springl]

@Manao
..even though PSNR values are very similar to yours.
I don't understand if you get SSIM=64.59 with lumimask=True and SSIM=63.72 with lumimask=True or viceversa or something else.
@Sagittaire
I will unveil my settings later on.
If you have a webspace I will be happy to upload my files

Greetings:)
Springl

[Manao]

springl : when i coputed the first ssim, it was with a tweaked ( or old , i don't remember ) version of SSIM.dll which outputs SSIM & lumimasking values for each frames, but which doesn't give the average SSIM of the clip.

When i computed the first SSIM, i did only a simple average of SSIM scores, without taking lumimasking into account ( and even though lumimasking was enabled ). Moreover, some SSIM value where null ( where they should have been 1.0 ). All that gave a lower SSIM value.

So i computed SSIM again this morning with the proper SSIM.dll

[Sagittaire]

@ springl
Perhabs solutions later ...

@ Manao
I use XviD dec PP4 and DivX dec PP4 for decoding ... it's a co-dec test like say Digital56K ...

@ Cruncher
Possible to make your best for XviD sample and post them (6850 Ko +/-15 Ko and 13850 Ko +/-30 Ko) ... thx

I will update test with:
- new interval for size (perhabs bitrate +/- 0.25%)
- new XviD sample (perhabs directly with Cruncher sample)
- new Nero H264 sample (similar to recode with more accurate target bitrate)
- new Nero H264 sample with full quality (not excessible with Nero and thus not in the test but just for fun)
- new x264 sample with complete RC variability test (RC for x264 is IMO the most powerfull, simple to understand and to use ... very good job for your RC akupenguin and your codec too ... :D )

[Irwin]

"TMPGEnc : MPEG2 MP@ML
find size for metric equivalence with best codec
Coder:
decoder:
files:"

I'm waiting for this above

And good work for test. Vp7 is good but h264 it's better for me. (quality & speed) On my PIII 500Mhz - decode speed VP7 is totally terrible. H264 is much better.

[Manao]

Sagittaire : Did you see springl post stating that xvid & ffdshow postprocessing yield the same results ? And the emphasis and CO-DEC is meaningless to me, it's still the same norm. And that doesn't prevent you from testing x264 which can't decode.

[springl]

@Manao
When you computed SSIM again this morning with the proper SSIM.dll,did you set lumimask true or false in your avs script :
ssim(clip1,clip2,"results.csv","averageSSIM.txt",lumimask= ? ) ?
I'm trying to understand how you got your results 'cause I want to better understand the reliability of mine.

Greetings :)
Springl

[Manao]

I set it to true.

[springl]

thank you for your quick reply :)
Springl

[Sagittaire]

One2tech speack about overhead problem in PM ... it's a problem (.rmvb, .avi, .mp4 and .wmv). Perhabs Matroska container for all codec could be better for regular test ... ???

[Manao]

Instead of using the same container, i'd rather use the 'best' container for each codec. That way, it won't cripple artificially codecs that can be put efficiently into some container.

[iapir]

AFAIK only WMV is not natively supported by Matroska. And you can still use Gabest's tool to much WMV files in Matroska. But yes, then it's not optimal.

The same number of frames of the same duration should give a good idea of the actual file size for each file. Including possible complex (or not) references like in AVC+MP4.

[Sagittaire]

What is the Graal for other codec? At this time it's Nero H264 with full search: H264 full search (http://multimediacom.free.fr/Video/H264-450-full.mp4). It's just for fun and I don't use this file for benchmark. Setting not accessible in recode ... 0.11 fps and 8H00 for 3075 frames with my celeron 1.5 Ghz and very simple script ... lol



Minimum Average Maximum
Mean Absolute Deviation: 0.0061 1.3457 3.0761
Mean Deviation: -0.4678 +0.1607 +0.7250
PSNR: 34.5903 42.7184 69.2643
Overall PSNR: 41.5768


SSIM = 70.15

[IgorC]

Originally posted by Sagittaire
Nero H264 with full search: SSIM = 70.15 [/B]

what version did you use for this video? is it a future version of nero h.264?

[Sagittaire]

No it's an old CLI beta version of recode ... but recode done the same result than CLI with usual setting. Recode is perhabs little better for Rate Control. CLI is good for me because CLI can make 2 separed pass with stat files and can make easy multi-job in batch file ...

[stephanV]

Originally posted by Manao
Instead of using the same container, i'd rather use the 'best' container for each codec. That way, it won't cripple artificially codecs that can be put efficiently into some container.

Looking at container overhead is... weird... Are there no reliable tools for mp4 to determine the bit rate of a video stream?

It is a metrical quality evalution of codecs, dont let it get cluttered up by overhead. If necessary just dump the raw video data and look at the size of that...

[G-Slide]

So, who is the winner ???:confused:

:devil: :D

[Sagittaire]

Note: In fact real size target is very important for "450" sample : metric function are in high variability zone particulary SSIM. dSSIM/dbitrate are more and more high with less bitrate -> SSIM is like a*(1-e^(-bx)) form function ...

for exemple:
ND AVC, 6811 Ko, SSIM = 69.49
ND AVC, 6886 Ko, SSIM = 69.82

It's true in order with "SSIM-450" bench , "APSNR-450" bench and "OPSNR-450" bench

In these condition influence for Overhead container could be considerable ... and shit

It's not true with "900" bench -> dSSIM/dbitrate is not high and in relatively low variability zone



Originally posted by G-Slide
So, who is the winner ???:confused:

:devil: :D

it's a challenge ... not conclusion for the moment ... see XviD for exemple ... spectaculary evolution with setting optimisation and it's only for XviD ay this time ...

[bobololo]

Regarding the bitrate and container overhead issues, I would just suggest 2 things :

1) use as rule a max allowable bitrate instead of a target with a tolerance margin. Therefore the aim is to approach this bitrate without exceeding it (whatever the value you give to the encoder to obtain the result).

2) Consider the elementary stream bitrate instead of the overall file size including the container overhead. vdub gives the ES bitrate in the info page for AVI files. encavc also reports the ES bitrate when completed. Now I don't know how to retrieve this info from rvmb, but I guess it should exist some tool for that.

What do you think ?

[Sagittaire]

Originally posted by bobololo
Regarding the bitrate and container overhead issues, I would just suggest 2 things :

1) use as rule a max allowable bitrate instead of a target with a tolerance margin. Therefore the aim is to approach this bitrate without exceeding it (whatever the value you give to the encoder to obtain the result).

2) Consider the elementary stream bitrate instead of the overall file size including the container overhead. vdub gives the ES bitrate in the info page for AVI files. encavc also reports the ES bitrate when completed. Now I don't know how to retrieve this info from rvmb, but I guess it should exist some tool for that.

What do you think ?

1) I reduce in 6840 Ko - 6890 Ko ... but the problem is each codec aren't very good RC for target bitrate with high precision

2) Use matroska is possible if overhead is identical for each class codec ... it's an indirect method but very simple ???

elementary stream + constant overhead in mkv (if overhead is constant for each class codec) could be a good methode for compare real size of elementary video stream ... ?

It's simple and could to use for each codec of this benchmark ... even RV10 with .rmvb container or WMV9 with .wmv container ...