[20-06-2005] Metric Benchmark Challenge ...

[Sagittaire]

~ Movie Metric Benchmark Challenge ~

You think that your codec is the best : prove that ... !!?


A - Introduction

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...



B - Rules

Rule 1 : Source

You must use this source for the test : Harry Potter II trailer, 720*576 MPEG2 MP@ML 4.5 Mbps
HPII Part1 winrar (http://multimediacom.free.fr/Download/HPII.part1.rar), HPII Part2 winrar (http://multimediacom.free.fr/Download/HPII.part2.rar), HPII Part3 winrar (http://multimediacom.free.fr/Download/HPII.part3.rar), HPII Part4 winrar (http://multimediacom.free.fr/Download/HPII.part4.rar) & HPII Part5 winrar (http://multimediacom.free.fr/Download/HPII.part5.rar)


Rule 2 : Pre-process and encoding

You must use DGindex with 32-bit SSE/MMX iDCT algorithm for .d2v project and DGDecode.dll for MPEG2 decoding in avisynth scipt. You must use VirtualDub or VirtualDubMod in "fast compress" mod for VFW codec. You must use YV12 space color for complete process with all codec. You must use this avisynth script for make encoding:

source=Mpeg2Source("C:\...\HPII.d2v",idct=2)
source=Trim(source,70,3145)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)
Return(source)



Rule 3 : Bitrate

You must use 2 bitrates for encoding. The first use XviD in quant8 for reference, it's the "Streaming Quality". The second use XviD in quant4 for reference, it's the "1CDR Quality". You can use container of your choice for all Video Elementary Stream.

Streaming: 450 Kbps for Video Elementary Stream and container or 6750 Ko with +/- 0.30 % for tolerance. The size must be in [6730 Ko - 6770 Ko] interval.

1CDR: 900 Kbps for Video Elementary Stream and container or 13500 Ko with +/- 0.30 % for tolerance. The size must be in [13460 Ko - 13540 Ko] interval.


Rule 4 : Metric Tests

SSIM (YV12): IMO the best metric. it uses some HVS properties but not all ...
PDF description for VSSIM (http://www.cns.nyu.edu/~zwang/files/papers/vssim.pdf)

Overall PSNR (YV12): Very good metric test, particullary for Rate Control Quality test ...
MSE = Sum t = 1 -> NFrame Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (NFrame*width*height)
Overall PSNR = 10 * Log ( 255² / MSE)

Average PSNR (YV12):The worse metric test but Frame PSNR graph is very good to compare RC strategy ...
MSE = Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (width*height)
Average PSNR = Sum t = 1 -> NFrame 10 * Log ( 255² / MSE) / Nframe

Here (http://multimediacom.free.fr/Download/Test.rar) the test archives if you want all original Avisynth script for metric test.


SSIM vs PSNR

"For a given reference video and a given compressed video, it is meant to compute a quality metric, based on perceived visual distortion. Unlike the well-known PSNR measure, it's not purely mathematical, and should correlate much better with human vision."

"A higher MSE (and so lower PSNR) should mean that the compressed clip is a worse image but MSE and PSNR are flawed in this respect as numerous tests have shown. However with SSIM, according to tests carried out on the VQEG dataset, a higher Q (SSIM value) has a much better relation to the visual quality of the compressed clip. Despite this, bear in mind the SSIM metric still isn't perfect."

Here (http://multimediacom.free.fr/Download/SSIM-0.24.rar) last AviSynth plugin for VSSIM in YV12


Average PSNR vs Overall PSNR

It's easy to increase artificially Average PSNR with Rate Control tweak. In certain case CBR (0% for variability) will be better than real VBR (100% for variability or constant quant). It's generaly not possible with Overall PSNR. Overall PSNR is very good to test Rate Control quality. If difference between Average PSNR and Overall PSNR is very high, "local quality difference" (difference between Pframe or Bframe for exemple) or "scene quality difference" (difference between low motion and high motion for exemple) will be too high and Rate Control for this codec not very good.

Here (http://multimediacom.free.fr/Download/CompareYV12.zip) last AviSynth plugin for Overall PSNR in YV12


You must mux or remux x264, VP6, VP7, WMV9, DivX6 and XviD in avi container (with VDM for example) to solve problem with color space output, synchronisation or acquisition for metric test. You must use AviSource in avisynth for VP6, VP7 and WMV9. You must use DirectShowSource in avisynth for MPEG4 ASP, MPEG4 AVC and RV10. You must change trim for frame synchronization or for clip length if you have some probleme with SSIM plugin. You must use these AviSynth type script for metric test:

# --> Source Opening <--
source=Mpeg2Source("G:\...\HPII.d2v",idct=2)
source=Trim(source,70,3145)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)

# --> Video Opening <--
video=DirectShowSource("G:\...\DivX6-450.avi",fps=25)

# --> PSNR analysis <--
compareYV12(video,source,"YUV","PSNR-DivX6-450.log")



# --> Source Opening <--
source=Mpeg2Source("G:\...\HPII.d2v",idct=2)
source=Trim(source,70,3144)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)

# --> Video Opening <--
video=DirectShowSource("G:\...\RV10-450.mkv",fps=25)

# --> SSIM analysis <--
return SSIM(source,video,"results.csv","SSIM-RV10-450.txt",lumimask=2)



C - Results

At this time here my result ... but done better if you can ... :devil:

|--------------|---------|---------|---------|---------|---------|---------|
| Codec | PProc | Bitrate | Size | OPSNR | SSIM 0 | SSIM 2 |
|--------------|---------|---------|---------|---------|---------|---------|
| DivX6 ASP | PP4 | 446 | 6757 | 39.85 | 70.17 | 65.02 |
| XviD ASP | PP4 | 446 | 6756 | 39.66 | 69.55 | 64.33 |
| LAVC ASP | PP4 | 447 | 6775 | 39.71 | 69.51 | 63.82 |
| Nero AVC HP | PP0 | 446 | 6747 | 41.10 | 76.30 | 72.10 |
| x264 AVC HP | PP0 | 446 | 6747 | 40.85 | 74.82 | 70.40 |
| Nero AVC MP | PP0 | 446 | 6747 | 40.72 | 74.65 | 70.25 |
| x264 AVC MP | PP0 | 446 | 6750 | 40.64 | 73.83 | 69.23 |
| VP7 | PP2 | 446 | 6760 | 40.84 | 74.42 | 70.05 |
| VP6 | PP4 | 446 | 6769 | 40.28 | 71.86 | 66.89 |
| WMV9 | PP1 | 446 | 6746 | 39.51 | 69.58 | 64.07 |
| RV10 | HF2 | 446 | 6743 | 39.84 | 69.86 | 64.60 |
| DivX3 | PP4 | 447 | 6771 | 38.70 | 65.11 | 58.56 |
|--------------|---------|---------|---------|---------|---------|---------|
| DivX6 ASP | PP4 | 896 | 13493 | 42.78 | 81.14 | 77.99 |
| XviD ASP | PP4 | 896 | 13489 | 42.66 | 80.99 | 77.96 |
| LAVC ASP | PP4 | 897 | 13528 | 42.77 | 80.95 | 77.77 |
| Nero AVC HP | PP0 | 896 | 13490 | 43.70 | 84.39 | 81.54 |
| x264 AVC HP | PP0 | 896 | 13491 | 43.46 | 83.27 | 80.50 |
| Nero AVC MP | PP0 | 896 | 13491 | 43.43 | 83.45 | 80.63 |
| x264 AVC MP | PP0 | 896 | 13498 | 43.26 | 82.66 | 79.75 |
| VP7 | PP2 | 897 | 13534 | 43.34 | 82.75 | 80.02 |
| VP6 | PP4 | 896 | 13481 | 42.99 | 81.39 | 78.41 |
| WMV9 | PP1 | 896 | 13510 | 42.52 | 79.88 | 76.73 |
| RV10 | HF2 | 896 | 13493 | 42.77 | 80.82 | 77.72 |
| DivX3 | PP4 | 896 | 13508 | 41.84 | 78.27 | 74.40 |
|--------------|---------|---------|---------|---------|---------|---------|
SSIM 0: Lumimask Off
SSIM 1: Lumimask On (Original Lumimask)
SSIM 2: Lumimask On (One2Tech Patch)



MPEG4 ASP : DivX6
Coder: DivX6 Codec
Setting: 2 pass, Unconstrained, Insame Quality, Multi-Bframe, H263 optmized, BM = 0.1
decoder: DivX decoder with PP4
files: DivX 450 Kbps (http://multimediacom.free.fr/Video/DivX6-450.mkv) & DivX 900 Kbps (http://multimediacom.free.fr/Video/DivX6-900.mkv)


MPEG4 ASP : XviD
Coder: Koepi XviD 1.1.0 beta2
Setting: 2 pass, Unconstrained, Motion Ultra-High, VHQ4, BVHQ, Chroma Motion, Trellis, Adaptative Quantisation, GMC, BFrame 2/1.50/1.00
decoder: XviD decoder with PP4
files: XviD 450 Kbps (http://multimediacom.free.fr/Video/XviD-450.mkv) & XviD 900 Kbps (http://multimediacom.free.fr/Video/XviD-900.mkv)


MPEG4 ASP : Libavcodec
Coder: MEncoder dev-CVS-050426-18:49-3.4.2
Setting: 3 pass, adaptative BFrame 2/1.50/1.00, Chroma Motion, Trellis, RDO, SAD, 6 diamond size
decoder: ffdshow decoder with PP4
files: Libavcodec ASP 450 Kbps (http://multimediacom.free.fr/Video/LAVC-450.mkv) & Libavcodec ASP 900 Kbps (http://multimediacom.free.fr/Video/LAVC-900.mkv)


DivX 3.11 : Libavcodec
Coder: MEncoder dev-CVS-050426-18:49-3.4.2
Setting: 3 pass, Chroma Motion, RDO, SAD, 6 diamond size
decoder: ffdshow decoder with PP4
files: DivX3 450 Kbps (http://multimediacom.free.fr/Video/DivX3-450.mkv) & DivX3 900 Kbps (http://multimediacom.free.fr/Video/DivX3-900.mkv)


MPEG4 AVC : Ateme AVC Main Profil
Coder: Ateme encavc 1.2.0.17
Setting: 2 pass, Quality Full, 8 Ref, 8 BRef, 2 Bframes, CABAC, BPred, MPart, WPred, Inloop strength 0, Chroma, psy 3
decoder: Ateme AVC decoder
files: Nero AVC Main Profil 450 Kbps (http://multimediacom.free.fr/Video/NDAVCMP-450.mp4) & Nero AVC Main Profil 900 Kbps (http://multimediacom.free.fr/Video/NDAVCMP-900.mp4)


MPEG4 AVC : Ateme AVC High Profil
Coder: Ateme encavc 1.2.0.14
Setting: 2 pass, Quality Full, 16 Ref, 16 BRef, 2 Bframes, CABAC, BPred, MPart, WPred, Inloop strength 0, Chroma, psy 3, transform 8x8
decoder: Ateme AVC decoder
files: Nero AVC High Profil 450 Kbps (http://multimediacom.free.fr/Video/NDAVCHP-450.mp4) & Nero AVC High Profil 900 Kbps (http://multimediacom.free.fr/Video/NDAVCHP-900.mp4)


MPEG4 AVC : x264 Main Profil
Coder: x264 VFW rev 264
Setting: 3 pass, variabilty 75%, Quality Max, 16 Ref frames, 2 Bframes, CABAC, WPred, Inloop deblock strength -1, RDO
decoder: ffdshow H264 decoder
files: x264 Main Profil 450 Kbps (http://multimediacom.free.fr/Video/x264MP-450.mp4) & x264 Main Profil 900 Kbps (http://multimediacom.free.fr/Video/x264MP-900.mp4)


MPEG4 AVC : x264 High Profil
Coder: x264 VFW rev 264
Setting: 3 pass, variabilty 75%, Quality Max, 16 Ref frames, 2 Bframes, CABAC, WPred, Inloop deblock strength -1 , 8x8dct, RDO
decoder: ffdshow H264 decoder
files: x264 High Profil 450 Kbps (http://multimediacom.free.fr/Video/x264HP-450.mp4) & x264 High Profil 900 Kbps (http://multimediacom.free.fr/Video/x264HP-900.mp4)


One2Tech : VP7
Coder: VP7 VFW Codec 7.0.0.7
Setting: 2 pass, best quality, VBR, Noisered 0, Sharpeness 0, Default Rate Control setting
decoder: VP7 decoder PP2
files: VP7 450 Kbps (http://multimediacom.free.fr/Video/VP7-450.mkv) & VP7 900 Kbps (http://multimediacom.free.fr/Video/VP7-900.mkv)


One2Tech : VP6
Coder: VP6 VFW Codec 6.4.1.0
Setting: 2 pass, best quality, VBR, Noisered=0, Sharpeness=0, MinQ 6/12, MaxQ 56, Vbr bias 70, Vbr min 30, Vbr max 200, Undershoot 95%
decoder: VP6 decoder PP4
files: VP6 450 Kbps (http://multimediacom.free.fr/Video/VP6-450.mkv) & VP6 900 Kbps (http://multimediacom.free.fr/Video/VP6-900.mkv)


Microsoft : WMV9
Coder: WMV9 VCM
Setting: 2 pass VBR, best quality, decoder complex
decoder: WMV3 DMO decoder PP1
files: WMV9 450 Kbps (http://multimediacom.free.fr/Video/WMV9-450.mkv) & WMV9 900 Kbps (http://multimediacom.free.fr/Video/WMV9-900.mkv)


RealNetwork : RV10
Coder: erv4 10.0.0.2
Setting: 2 pass, EHQ 100, old Rate Control, chroma
decoder: RV10 decoder HFE2
files: RV10 450 Kbps (http://multimediacom.free.fr/Video/RV10-450.mkv) & RV10 900 Kbps (http://multimediacom.free.fr/Video/RV10-900.mkv)


MPEG2 MP@ML : MEncoder dev-CVS-050426-18:49-3.4.2
find size for metric equivalence with best codec for quality "1CDR"
"Quality" must be at ~ 43 dB for OPSNR or at ~ 80 for SSIM
files: MPEG2 1250 Kbps (http://multimediacom.free.fr/Video/MPEG2-1250.m2v)



D - Conclusion

At this time for this trailer

MPEG4 AVC codecs: NDAVC vs x264
Ateme H264 and x264 are comparable codec for OPSNR but not for SSIM (visually NDAVC is IMO better than x264 too). x264 is the best Open Source Codec and XviD can't fight with the last version.

MPEG4 ASP codecs: XviD vs DivX vs LAVC
DivX6 is the best MPEG4 ASP codec for OPSNR and SSIM but XviD, DivX6 and Libavcodec ASP are very close codec.

Proprietary "free" codecs: RV10 vs VP6 vs WMV9
VP6 is always better than RV10 or WMV9 for OPSNR or SSIM. VP6 are better than all MPEG4 ASP codec for OPSNR and SSIM. RV10 is an excellent codec for high quant encoding (low quality or "low bitrate").

Higest performance codecs: VP7 vs NDAVC vs x264
VP7, Ateme AVC or x264 are really better than the others codecs. VP7 seem better than Ateme H264 Main Profil for very high quant (low quality or "low bitrate") and Ateme H264 Main Profil seem better than VP7 for lower quant (high quality or "high bitrate"). x264 High Profil is the best codec actually available. Ateme H264 is by far the best codec in this test ...

MPEG2 vs all
The best codecs (H264 or VP7) are 50% better than MPEG2 MP@ML but MPEG2 MP@ML with good encoder is always a good codec. IMO Mencoder is by far the best MPEG2 encoder for low bitrate ...



Annexe - Update

14.07.05
- Ateme codec Full quality update

19.06.05
- Ateme codec Full quality
- VP7 7.0.0.7 better RC

17.06.05
- x264 rev 263 Main Profil
- x264 rev 263 High Profil
- DivX6
- VP7 7.0.0.7
- MPEG2 MP@ML with Mencoder

08.06.05
- x264 rev 253 Main Profil
- x264 rev 253 High Profil

24.05.05
- SSIM plugin
- Nero Digital High Profil for demo only

22.05.05
- x264 rev 232
- VP7 7.0.0.6
- Libavcodec MPEG4 ASP (MEncoder dev-CVS-050426-18:49-3.4.2)
- Libavcodec DivX 3.11 (MEncoder dev-CVS-050426-18:49-3.4.2)
- XviD 1.1.0 Beta2

03.04.05
- x264 rev 184
- VP7 7.0.0.4

[Manao]

You encoded XviD at fixed quantizer. Encoding it two-passes will give you a higher psnr. Furthermore, you stated 450 kbps / 6882 KB, which isn't coherent ( 1024 / 1000 issue ). You're better off targetting 458kbps ( the difference been greater than the error margin, those 8 kbps are important ).

I made the test with XviD @ '450' kbps ( actually, in order to hit the wanted bitrate, i had to type 480 in the bitrate box ). I obtained a psnr of 40.49 dB overall ( 41.69 average ).

Edit : btw, the loss implied by doing a constant quantizer encoding is 0.8 dB, which is _huge_. I hope nobody still believe that constant quantizer yields better quality that a two passes encoding.

[Sagittaire]

Originally posted by Manao
You encoded XviD at fixed quantizer. Encoding it two-passes will give you a higher psnr.

For me priority are in order:
- SSIM
- Overall PSNR
- Average PSNR


Originally posted by Manao
Furthermore, you stated 450 kbps / 6882 KB, which isn't coherent ( 1024 / 1000 issue ). You're better off targetting 458kbps ( the difference been greater than the error margin, those 8 kbps are important ).

Yes ... it's true ... 458 Kbps +/-1% and 919 Kbps +/-1% exactly (I edit)


Originally posted by Manao
I made the test with XviD @ '450' kbps ( actually, in order to hit the wanted bitrate, i had to type 480 in the bitrate box ). I obtained a psnr of 40.49 dB overall ( 41.69 average ).
Yes I use one pass q4 (q4 for I and P, q6 for B) ... but your encoding seem very good (perhabs not for SSIM) ... you can post this file or your setting ...

[Manao]

SSIM : 63.72
Settings : XviD defaults + bvhq + vhq4 + GMC

Btw, you praise psnr overall over psnr average when it comes to rate control comparison, however ssim is computed as average psnr is, so ssim should only be used to compare frames together, not to compare rate control algorithms.

Edit : forgot : playback with ffdshow Nic's PP4

[springl]

My results with Xvid:

Streaming.avi
size: 6894 KB
SSIM_PP4: 64.86

1CD.avi
size: 13806 KB
SSIM_PP4: 78.27

Settings :
XviD defaults + bvhq + vhq4 + GMC + MIN/MAX I/P/B frame 2/31 + some tuning in
overflow treatment and curve compression.

playback with ( Xvid decoder + PP4 ) or ( ffdshow + Nic's PP4 ) will produce same results.

Greetings :)
Springl

[Manao]

Sagittaire :

CruNcher and I doublechecked your psnr results for your XviD @ 450 kbps. We obtain an overall of 40.40 and a SSIM of 63.94. How did you manage to get that wrong ? What can we assume for the other figures for the other codecs ?

[bobololo]

@Sagittaire: Just a stupid question, why don't you use compareYV12() instead of compare() ? Don't all the codecs here work natively in YUV 4:2:0 colorspace ?

[LordRPI]

Sagittaire,

Would it be possible to list the codec settings you used for all comparisons?

[Sagittaire]

@Bobololo
compare (http://www.avisynth.org/Compare) is only YUY2 or RGB compatible in avisynth ... i don't understand why ... perhabs function begin in avisynth YUY2 2.0x version but avisynth 2.5x is YV12 now ...

You can confirm my result with your AVC codec and with my test protocol ... perhabs you can make better encoding ... ???


@ Manao
Problem seem to be ffdshow for MPEG4 ASP class codec ... I will use XviD decoder and DivX decoder now ... update in progress


@LordRPI
update in progress ...


@springl
please if possible post your encoding or your setting with complete reg file for exemple:


Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\GNU\XviD]
"mode"=dword:00000002
"bitrate"=dword:000001ca
"desired_size"=dword:00001ac2
"use_2pass_bitrate"=dword:00000000
"desired_quant"=dword:00000190
"quant_type"=dword:00000000
"lum_masking"=dword:00000000
"interlacing"=dword:00000000
"tff"=dword:00000000
"qpel"=dword:00000000
"gmc"=dword:00000000
"use_bvop"=dword:00000001
"max_bframes"=dword:00000002
"bquant_ratio"=dword:00000096
"bquant_offset"=dword:00000064
"packed"=dword:00000001
"ar_mode"=dword:00000000
"aspect_ratio"=dword:00000000
"par_x"=dword:00000001
"par_y"=dword:00000001
"ar_x"=dword:00000004
"ar_y"=dword:00000003
"num_zones"=dword:00000001
"rc_reaction_delay_factor"=dword:00000010
"rc_averaging_period"=dword:00000064
"rc_buffer"=dword:00000064
"discard1pass"=dword:00000001
"full1pass"=dword:00000000
"keyframe_boost"=dword:0000000a
"kfreduction"=dword:00000014
"kfthreshold"=dword:00000001
"curve_compression_high"=dword:00000000
"curve_compression_low"=dword:00000000
"overflow_control_strength"=dword:00000005
"twopass_max_overflow_improvement"=dword:00000005
"twopass_max_overflow_degradation"=dword:00000005
"container_type"=dword:00000001
"target_size"=dword:000a2800
"subtitle_size"=dword:00000000
"hours"=dword:00000001
"minutes"=dword:0000001e
"seconds"=dword:00000000
"fps"=dword:00000002
"audio_mode"=dword:00000000
"audio_type"=dword:00000000
"audio_rate"=dword:00000080
"audio_size"=dword:00000000
"motion_search"=dword:00000006
"vhq_mode"=dword:00000004
"vhq_bframe"=dword:00000001
"chromame"=dword:00000001
"turbo"=dword:00000000
"max_key_interval"=dword:0000012c
"frame_drop_ratio"=dword:00000000
"min_iquant"=dword:00000001
"max_iquant"=dword:0000001f
"min_pquant"=dword:00000001
"max_pquant"=dword:0000001f
"min_bquant"=dword:00000001
"max_bquant"=dword:0000001f
"trellis_quant"=dword:00000001
"fourcc_used"=dword:00000000
"debug"=dword:00000000
"vop_debug"=dword:00000000
"display_status"=dword:00000001
"Brightness"=dword:00000000
"Deblock_Y"=dword:00000001
"Deblock_UV"=dword:00000001
"Dering_Y"=dword:00000000
"Dering_UV"=dword:00000000
"FilmEffect"=dword:00000000
"profile"="(unrestricted)"
"stats"="\\video.pass"
"qmatrix_intra"=hex:08,11,12,13,15,17,19,1b,11,12,13,15,17,19,1b,1c,14,15,16,\
17,18,1a,1c,1e,15,16,17,18,1a,1c,1e,20,16,17,18,1a,1c,1e,20,23,17,18,1a,1c,\
1e,20,23,26,19,1a,1c,1e,20,23,26,29,1b,1c,1e,20,23,26,29,2d
"qmatrix_inter"=hex:10,11,12,13,14,15,16,17,11,12,13,14,15,16,17,18,12,13,14,\
15,16,17,18,19,13,14,15,16,17,18,1a,1b,14,15,16,17,19,1a,1b,1c,15,16,17,18,\
1a,1b,1c,1e,16,17,18,1a,1b,1c,1e,1f,17,18,19,1b,1c,1e,1f,21
"zone0_frame"=dword:00000000
"zone0_mode"=dword:00000000
"zone0_weight"=dword:00000064
"zone0_quant"=dword:00000190
"zone0_type"=dword:00000000
"zone0_greyscale"=dword:00000000
"zone0_chroma_opt"=dword:00000000
"zone0_bvop_threshold"=dword:00000000
"zone0_cartoon_mode"=dword:00000000
"ForceColorspace"=dword:00000001
"FlipVideo"=dword:00000000
"Supported_4CC"=dword:00000000
"Videoinfo_Compat"=dword:00000000

[DigitAl56K]

Which settings were used for DivX? I might be missing something obvious here, but I don't see them. Did you constrain all codecs via VBV, or was this test a mixed bag? Was there any emphasis on comparible testing (e.g. time to encode, CE compatibility, etc.) or was every codec simply allowed to run wild with the features it used?

What were the settings used for WM9? I'm surprised at this result.

Also, when you use a decoder you really ought to use the decoder for the codec you are testing, i.e. use the DivX decoder for DivX, the XVID decoder for XVID, etc. Each codec may implement different technologies in their decoder and these are no less a part of the format than any encode-side features. For DivX we recommend full deblock, no dering, no frame delay, no film effect. We notice you also used DirectShowSource. When using this input filter please ensure that the DivX Decoder is configured so that it does not overlay the DivX Video logo during decoding.

[sparky]

Streaming: 458 Kbps or 6885 Ko

Just to clarify. Do you mean 458 kbps = 458000 bps = (458000/8)*(3076/25) bytes = 7044040 bytes = 6879 Kb, and 919 kbps = 919000 bps = 14134220 bytes = 13803 Kb?

Also, you have Trim(70,3144) in SSIM script, and Trim(70,3145) in all other scripts. It causes an error ( SSIM won't work with different-length clips ). You want to change 3144->3145.

[Sagittaire]

@ DigitAl56K
problem with ffdshow for MPEG4 ASP class codec (i use DivX dec and XviD dec now ...) and I update bench for MPEG4 ASP codec ...

For WMV9 I use best quality with complex dec in 2 two pass VBR. I use PP1 DMO (http://multimediacom.free.fr/Video/WMVPostpross.exe) for Post-process ...

For DivX I use unconstrained, insame quality, H263, multibframe, GMC, Bitrate Modultion = 0.25. You can propose your encoding with setting used if you want ...


Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\DivXNetworks\DivX4Windows]
"Auto PP Level"=dword:00000000
"Auto PP"=dword:00000000
"Disable PP"=dword:00000000
"Dering PP Level"=dword:00000000
"Deblock PP Level"=dword:0000000f
"Warmth Strength"=dword:00000002
"Warmth Filter"=dword:00000000
"Postprocessing"=dword:00000028
"Brightness"=dword:00000000
"Contrast"=dword:00000000
"Saturation"=dword:00000000
"AspectRatio_X"=dword:00000000
"AspectRatio_Y"=dword:00000000
"Smooth Playback"=dword:00000000
"Extended Yuv Mode"=dword:00000001
"Double Buffering"=dword:00000000
"Extended Overlay Mode"=dword:00000000
"Disable Logo"=dword:00000001
"ATI Disable"=dword:00000000
"Support Generic Mpeg-4"=dword:00000001
"Bitrate"=dword:000be6e0
"VBV Channel Bitrate"=dword:006a1120
"VBV Buffer Size"=dword:00300000
"VBV Occupancy"=dword:00240000
"Complexity Modulation"="0.000000"
"Max Keyframe Interval"=dword:0000012c
"Quality"=dword:00000005
"VBR Mode"=dword:00000000
"VBR Quality v5"=dword:00000028
"Frame Dropping"=dword:00000000
"Deinterlace"=dword:00000000
"CPU Limit"=dword:00000064
"Use CPU Limit"=dword:00000000
"Data Partitioning"=dword:00000000
"Bidirectional Encoding"=dword:00000001
"Scene change threshold"=dword:00000032
"do_add"=dword:00000001
"do_mc"=dword:00000001
"do_mc_b"=dword:00000001
"show_gmc"=dword:00000000
"no_vbr_rc"=dword:00000000
"use_mvfile"=dword:00000000
"do_writeyuv"=dword:00000000
"do_writeyuvmul"=dword:00000000
"Psychovisual Enhancement"=dword:00000000
"Psychovisual Strength - Frame Level"="0.200000"
"Psychovisual Strength - Macroblock level"="0.200000"
"Write MP4 file"=dword:00000000
"Use GMC"=dword:00000000
"Quarter-pel"=dword:00000000
"Crop"=dword:00000000
"Resize"=dword:00000000
"Crop left"=dword:00000000
"Crop right"=dword:00000000
"Crop top"=dword:00000000
"Crop bottom"=dword:00000000
"Resize width"=dword:00000280
"Resize height"=dword:000001e0
"Resize mode"=dword:00000000
"Bicubic spline B"="0.000000"
"Bicubic spline C"="0.500000"
"Temporal enable"=dword:00000000
"Spatial passes"=dword:00000000
"Temporal level"="0.000000"
"Spatial level"="0.000000"
"Interlace mode"=dword:00000000
"Use dialogs"=dword:00000000
"Disable feedback"=dword:00000001
"Write log file"=dword:00000000
"5.0 Psychovisual Mode"=dword:00000000
"5.0 Preprocessing Mode"=dword:ffffffff
"5.0 Resize Mode"=dword:00000001
"Write nth pass logfile"=dword:00000000
"Disable Profile Page"=dword:00000000
"Active Profile"=dword:00000002
"Nth Pass debug file name"="c:\\newrc.txt"
"Log File Name"="c:\\divx.log"
"MP4 File Name"="c:\\test.divx"
"YUV Dir Name"="c:\\yuv"
"MV File Name"="c:\\mvinfo.bin"
"Top field first"=dword:00000001
"MPEG Quant"=dword:00000000
"Rotate Artwork"=dword:00000001
"Check for Updates"=dword:00000001
"Disable Notifications"=dword:00000000
"Intra quant matrix"=hex:00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00
"Inter quant matrix"=hex:00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00
"Feedback refresh"=dword:00000001
"Force Color Mode"=dword:00000000


command line:
-bvnn 915000 -vbv 27336400,100663296,75497472 -pq 8610 -g -complexity_modulation=0.25

[Sagittaire]

Originally posted by sparky
Just to clarify. Do you mean 458 kbps = 458000 bps = (458000/8)*(3076/25) bytes = 7044040 bytes = 6879 Kb, and 919 kbps = 919000 bps = 14134220 bytes = 13803 Kb?

Also, you have Trim(70,3144) in SSIM script, and Trim(70,3145) in all other scripts. It causes an error ( SSIM won't work with different-length clips ). You want to change 3144->3145.

yes for bitrate ...

not explain for variable length clip ... clip for SSIM function must have the same length and not for compare function. Length problem is perhabs bframe problem for MPEG4 ASP, for RV10 i don't know ... but it isn't big problem with trim modification ...

[bobololo]

Originally posted by Sagittaire
@Bobololo
compare (http://www.avisynth.org/Compare) is only YUY2 or RGB compatible in avisynth ... i don't understand why ... perhabs function begin in avisynth YUY2 2.0x version but avisynth 2.5x is YV12 now ...

You can confirm my result with your AVC codec and with my test protocol ... perhabs you can make better encoding ... ???

For native YV12 PSNR computation, try compareYV12 (http://www.avisynth.org/warpenterprises/files/compareyv12_25_dll_20030724.zip).

Regarding your result with our codec, let me know your settings first. Also, there is an undocumented quality level "full" which is insanely slow but that can surely achieve slightly higher figures ;)

[Sagittaire]

Originally posted by bobololo
For native YV12 PSNR computation, try compareYV12 (http://www.avisynth.org/warpenterprises/files/compareyv12_25_dll_20030724.zip).

Regarding your result with our codec, let me know your settings first. Also, there is an undocumented quality level "full" which is insanely slow but that can surely achieve slightly higher figures ;)

yes, I know but just Average PSNR with this plugin ... :(
YUY2 test order is not different to YV12 ... there are good scaling with result between YUY2 and YV12 test ... perhabs Average PSNR in YV12 and Overall PSNR in YUY2 will be good solution ...


for your AVC codec I use "my best setting" ... lol
encavc.exe -i azerty.avs -o clip.mp4 -qual extra -rcmode 1st -log 1pass.log -br 450000 -deblock 0 -ref 5 -setef wpred -cartoon -psy 1 -priority idle
encavc.exe -i azerty.avs -o clip.mp4 -qual extra -rcmode 2nd -log 1pass.log -br 450000 -deblock 0 -ref 5 -setef wpred -cartoon -psy 1 -priority idle


Also, there is an undocumented quality level "full" which is insanely slow but that can surely achieve slightly higher figures ;)

[méthode couet=On]
i want this command line ... i want this command line ... i want this command line ... lol
[méthode couet=Off]

[Sagittaire]

XviD update:
new sample, new bench, new graph

@Bobololo

-qual full ... ah yes ... 0.16 fps ... lol

[French=On]
je comprend vite mais faut quand même m'expliquer longtemps ... lol
[French=Off]

[bobololo]

Originally posted by Sagittaire
yes, I know but just Average PSNR with this plugin ... :(
YUY2 test order is not different to YV12 ... there are good scaling with result between YUY2 and YV12 test ... perhabs Average PSNR in YV12 and Overall PSNR in YUY2 will be good solution ...


Ah you're right, I didn't notice that compareYV12 doesn't report overall PSNR. That's really weird it isn't included. It should be easy to add it. I'll try to make tomorrow.

for your AVC codec I use "my best setting" ... lol
encavc.exe -i azerty.avs -o clip.mp4 -qual extra -rcmode 1st -log 1pass.log -br 450000 -deblock 0 -ref 5 -setef wpred -cartoon -psy 1 -priority idle
encavc.exe -i azerty.avs -o clip.mp4 -qual extra -rcmode 2nd -log 1pass.log -br 450000 -deblock 0 -ref 5 -setef wpred -cartoon -psy 1 -priority idle


I believe the target bitrate was 458000 bits/s, am I wrong ?

[méthode couet=On]
i want this command line ... i want this command line ... i want this command line ... lol
[méthode couet=Off]

I explicitly gave it ;)

[Sagittaire]

[i]Originally posted by bobololo

I believe the target bitrate was 458000 bits/s, am I wrong ?

I explicitly gave it ;)

6811 Ko for Nero AVC and 6852 Ko for VP7 ... it's less than 1% ... difficult to make better ... ;)

[CruNcher]

Sagittaire please keep the values correct the XviD-450.avi is not 450 kbps as stated in the table it's 457.0 kbps

[Sagittaire]

All my files are upload and hyperlink in post ...

And 450 must be 450 Kbit/s too (like for VP6 or WMV9) ... in fact the most important for size/bitrate is to respect [6810 Ko - 6960 Ko] interval and [13660 Ko - 13940 Ko] interval. With same codec/setting difference between 2% for size is very small for metric (perhabs 0.1 dB for PSNR and 0.1 for SSIM)

[akupenguin]

0.1 dB isn't negligible if it's more than the difference between XviD and DivX in your table.

[CruNcher]

ok how did you get 462 kbps for XviD now ?


| DivX | PP4 | 456 kbps | 6842 Ko | 41.3023 | 40.3475 | 63.70 |
| XviD | PP4 | 462 kbps | 6944 Ko | 41.3018 | 40.3777 | 63.90 |


XviD = 457.0 kbps
DivX = 450.0 kbps

at least the clips that are downloadable

[LordRPI]

Sagittaire,

You also switched the DivX and the XviD downloads on your first post.

[Sagittaire]

it's simple ... i want 6850 Ko (taget size in XviD) but RC for small trailer isn't very good with defaut setting and high quant for second pass encoding (perhabs too small overflow) ...

but i uptade if you want ... :D

[bobololo]

Originally posted by Sagittaire
6811 Ko for Nero AVC and 6852 Ko for VP7 ... it's less than 1% ... difficult to make better ... ;)

It can easily make better ;) You can consider this as nitpicking, but if you want the best settings providing the highest results respecting your rules (max allowable size = 6960 KiB which is equivalent to 463396 b/s), then for ND/AVC you can safely set the target bitrate to 460500 b/s. This will provide enough room for the container and the potential RC errors so that the final size won't exceed 6960 KiB).

This with the addition of the full quality mode, you'll reach tremendous values ;) j/k of course.

edit: I agree with akupenguin, 0.1 dB for PSNR is significant.

[CruNcher]

i meant why are the bitrate of the samples don't match with the table results ?


XviD EDP H263 VHQ4 noaq_noqpel_gmc = AVG: 41.9555 OV: 40.3158 SSIM: 63.56 BR: 450.3 kbps

FFdshow settings
http://cruncher.mufflastig.com/XviD/samples/pp4.png

http://cruncher.mufflastig.com/XviD/samples/psnr.png

XviD-450 (http://cruncher.mufflastig.com/XviD/samples/XviD-450.3-edp-h263.avi)

[Sagittaire]

i have 0.06 dB for 100 Ko difference with XviD (6944 vs 6844 and 1.5%) : after update with XviD all the files will have less 0.5% difference for size ... :devil:

[sparky]

A few notes.
1. Bitrate should be specified more precisely. Variation of bitrate by 2% is roughly equal to 0.1 db difference in quality. 0.1 db is significant in our case.
2. It should be allowed to target each of the 3 quality metrics individually. Maxima for SSIM and Overall PSNR could be achieved by different codec configurations. For example, DivX deringing increases PSNR but lowers SSIM.

[Manao]

For XviD i get these results:

| XviD | PP4 | | 6888Ko | 41.6998 | 40.4912 | 64.59 |
( the SSIM value changed because i computed it without lumimask the first time )

Settings were defaults + bvhq + vhq4 ( i dropped the GMC, it brings almost nothing ). Playback with ffdshow + nic's PP 4 custom, as CruNcher

CompareYV12 gives the overall, but not the average ( though you can compute it with excel, since you have all the frame's psnr )

For the different length, CruNcher thinks it's due to an outdated version of dgdecode ( for Sagittaire ).

[Sagittaire]

Originally posted by Manao
For XviD i get these results:

| XviD | PP4 | | 6888Ko | 41.6998 | 40.4912 | 64.59 |
( the SSIM value changed because i computed it without lumimask the first time )

Settings were defaults + bvhq + vhq4 ( i dropped the GMC, it brings almost nothing ). Playback with ffdshow + nic's PP 4 custom, as CruNcher

CompareYV12 gives the overall, but not the average ( though you can compute it with excel, since you have all the frame's psnr )

For the different length, CruNcher thinks it's due to an outdated version of dgdecode ( for Sagittaire ).

SSIM = 64.59 ... you are sure for that ... :confused:

My .d2v is an old project (DVD2AVI 1.76) but isn't a big problem : one or two frames droped ...

[Manao]

Yes, i'm sure, and you can notice that springl, with his settings, even got an higher ssim than me.

[springl]

@Manao
..even though PSNR values are very similar to yours.
I don't understand if you get SSIM=64.59 with lumimask=True and SSIM=63.72 with lumimask=True or viceversa or something else.
@Sagittaire
I will unveil my settings later on.
If you have a webspace I will be happy to upload my files

Greetings:)
Springl

[Manao]

springl : when i coputed the first ssim, it was with a tweaked ( or old , i don't remember ) version of SSIM.dll which outputs SSIM & lumimasking values for each frames, but which doesn't give the average SSIM of the clip.

When i computed the first SSIM, i did only a simple average of SSIM scores, without taking lumimasking into account ( and even though lumimasking was enabled ). Moreover, some SSIM value where null ( where they should have been 1.0 ). All that gave a lower SSIM value.

So i computed SSIM again this morning with the proper SSIM.dll

[Sagittaire]

@ springl
Perhabs solutions later ...

@ Manao
I use XviD dec PP4 and DivX dec PP4 for decoding ... it's a co-dec test like say Digital56K ...

@ Cruncher
Possible to make your best for XviD sample and post them (6850 Ko +/-15 Ko and 13850 Ko +/-30 Ko) ... thx

I will update test with:
- new interval for size (perhabs bitrate +/- 0.25%)
- new XviD sample (perhabs directly with Cruncher sample)
- new Nero H264 sample (similar to recode with more accurate target bitrate)
- new Nero H264 sample with full quality (not excessible with Nero and thus not in the test but just for fun)
- new x264 sample with complete RC variability test (RC for x264 is IMO the most powerfull, simple to understand and to use ... very good job for your RC akupenguin and your codec too ... :D )

[Irwin]

"TMPGEnc : MPEG2 MP@ML
find size for metric equivalence with best codec
Coder:
decoder:
files:"

I'm waiting for this above

And good work for test. Vp7 is good but h264 it's better for me. (quality & speed) On my PIII 500Mhz - decode speed VP7 is totally terrible. H264 is much better.

[Manao]

Sagittaire : Did you see springl post stating that xvid & ffdshow postprocessing yield the same results ? And the emphasis and CO-DEC is meaningless to me, it's still the same norm. And that doesn't prevent you from testing x264 which can't decode.

[springl]

@Manao
When you computed SSIM again this morning with the proper SSIM.dll,did you set lumimask true or false in your avs script :
ssim(clip1,clip2,"results.csv","averageSSIM.txt",lumimask= ? ) ?
I'm trying to understand how you got your results 'cause I want to better understand the reliability of mine.

Greetings :)
Springl

[Manao]

I set it to true.

[springl]

thank you for your quick reply :)
Springl

[Sagittaire]

One2tech speack about overhead problem in PM ... it's a problem (.rmvb, .avi, .mp4 and .wmv). Perhabs Matroska container for all codec could be better for regular test ... ???

[Manao]

Instead of using the same container, i'd rather use the 'best' container for each codec. That way, it won't cripple artificially codecs that can be put efficiently into some container.

[iapir]

AFAIK only WMV is not natively supported by Matroska. And you can still use Gabest's tool to much WMV files in Matroska. But yes, then it's not optimal.

The same number of frames of the same duration should give a good idea of the actual file size for each file. Including possible complex (or not) references like in AVC+MP4.

[Sagittaire]

What is the Graal for other codec? At this time it's Nero H264 with full search: H264 full search (http://multimediacom.free.fr/Video/H264-450-full.mp4). It's just for fun and I don't use this file for benchmark. Setting not accessible in recode ... 0.11 fps and 8H00 for 3075 frames with my celeron 1.5 Ghz and very simple script ... lol



Minimum Average Maximum
Mean Absolute Deviation: 0.0061 1.3457 3.0761
Mean Deviation: -0.4678 +0.1607 +0.7250
PSNR: 34.5903 42.7184 69.2643
Overall PSNR: 41.5768


SSIM = 70.15

[IgorC]

Originally posted by Sagittaire
Nero H264 with full search: SSIM = 70.15 [/B]

what version did you use for this video? is it a future version of nero h.264?

[Sagittaire]

No it's an old CLI beta version of recode ... but recode done the same result than CLI with usual setting. Recode is perhabs little better for Rate Control. CLI is good for me because CLI can make 2 separed pass with stat files and can make easy multi-job in batch file ...

[stephanV]

Originally posted by Manao
Instead of using the same container, i'd rather use the 'best' container for each codec. That way, it won't cripple artificially codecs that can be put efficiently into some container.

Looking at container overhead is... weird... Are there no reliable tools for mp4 to determine the bit rate of a video stream?

It is a metrical quality evalution of codecs, dont let it get cluttered up by overhead. If necessary just dump the raw video data and look at the size of that...

[G-Slide]

So, who is the winner ???:confused:

:devil: :D

[Sagittaire]

Note: In fact real size target is very important for "450" sample : metric function are in high variability zone particulary SSIM. dSSIM/dbitrate are more and more high with less bitrate -> SSIM is like a*(1-e^(-bx)) form function ...

for exemple:
ND AVC, 6811 Ko, SSIM = 69.49
ND AVC, 6886 Ko, SSIM = 69.82

It's true in order with "SSIM-450" bench , "APSNR-450" bench and "OPSNR-450" bench

In these condition influence for Overhead container could be considerable ... and shit

It's not true with "900" bench -> dSSIM/dbitrate is not high and in relatively low variability zone



Originally posted by G-Slide
So, who is the winner ???:confused:

:devil: :D

it's a challenge ... not conclusion for the moment ... see XviD for exemple ... spectaculary evolution with setting optimisation and it's only for XviD ay this time ...

[bobololo]

Regarding the bitrate and container overhead issues, I would just suggest 2 things :

1) use as rule a max allowable bitrate instead of a target with a tolerance margin. Therefore the aim is to approach this bitrate without exceeding it (whatever the value you give to the encoder to obtain the result).

2) Consider the elementary stream bitrate instead of the overall file size including the container overhead. vdub gives the ES bitrate in the info page for AVI files. encavc also reports the ES bitrate when completed. Now I don't know how to retrieve this info from rvmb, but I guess it should exist some tool for that.

What do you think ?

[Sagittaire]

Originally posted by bobololo
Regarding the bitrate and container overhead issues, I would just suggest 2 things :

1) use as rule a max allowable bitrate instead of a target with a tolerance margin. Therefore the aim is to approach this bitrate without exceeding it (whatever the value you give to the encoder to obtain the result).

2) Consider the elementary stream bitrate instead of the overall file size including the container overhead. vdub gives the ES bitrate in the info page for AVI files. encavc also reports the ES bitrate when completed. Now I don't know how to retrieve this info from rvmb, but I guess it should exist some tool for that.

What do you think ?

1) I reduce in 6840 Ko - 6890 Ko ... but the problem is each codec aren't very good RC for target bitrate with high precision

2) Use matroska is possible if overhead is identical for each class codec ... it's an indirect method but very simple ???

elementary stream + constant overhead in mkv (if overhead is constant for each class codec) could be a good methode for compare real size of elementary video stream ... ?

It's simple and could to use for each codec of this benchmark ... even RV10 with .rmvb container or WMV9 with .wmv container ...

[CruNcher]

im useing mplayer to get the correct bitrate

My .d2v is an old project (DVD2AVI 1.76) but isn't a big problem : one or two frames droped ...


that's a joke right ? every of us should use the same or this here is useless

[Sagittaire]

Originally posted by CruNcher
im useing mplayer to get the correct bitrate


that's a joke right ? every of us should use the same or this here is useless

no an error ... I use the last DGDecode.dll with last dgindex project version ... lol

In my previous test (matrix reloaded) I use DVD2AVI 1.76 project ... tired when i write this line ... lol

And now i'm tired by this size/overhead problem ...

I work Thursday and Friday and I resolve this problem during the weekend ...

[sparky]

My DivX6 results:

For 450 kbps target:
-bvn1 457000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2
-bvnn 457000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2 -complexity_modulation=0.1
7104512 bytes = 6938 kb
SSIM 65.13
OPSNR 40.6269
APSNR 41.4548

For 900 kbps target:
-bvn1 919000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2
-bvnn 919000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2 -complexity_modulation=0.1
14206976 bytes = 13874 kb
SSIM 78.34
OPSNR 43.5476
APSNR 44.3324

Feel free to retest

[Sagittaire]

Originally posted by sparky
My DivX6 results:

For 450 kbps target:
-bvn1 457000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2
-bvnn 457000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2 -complexity_modulation=0.1
7104512 bytes = 6938 kb
SSIM 65.13
OPSNR 40.6269
APSNR 41.4548

For 900 kbps target:
-bvn1 919000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2
-bvnn 919000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2 -complexity_modulation=0.1
14206976 bytes = 13874 kb
SSIM 78.34
OPSNR 43.5476
APSNR 44.3324

Feel free to retest

:cool:

uptade this weekend

[iapir]

I agree the overhead issue is probably very short compared to the differences in size allocation by each codec. But it's always better to compare what can be compared.

If a codec uses only I frames and a codec like AVC uses complex references, maybe the data size will be the same. But the storage space needed will be different on the container side. That's why I think it should be taken in account too. Since in the end you won't use raw data files but files in a container...

[PatchWorKs]

What about SNOW ? And Theora ? Any1 ?

[stephanV]

I agree the overhead issue is probably very short compared to the differences in size allocation by each codec. But it's always better to compare what can be compared.

The problem is, I can mux things in AVI, Matroska and MP4 with different apps and get different values for overhead, and then i didnt even mention changing the settings.

Should AVC be used in MP4 for this test? What if a lot of people would want to use it with AC3 or Vorbis? Would the test become invalid for them?

Just compare raw data.

[Manao]

StephanV : i doubt anyone can compute psnr from a raw h264 stream. Moreover, it's hard to get raw data size for some codecs. I still think that choosing the container & tha application that minimizes the filesize for a particular codec is still the way to go.

And if each codecs are tested with their 'optimal' container, somehow, i guess the overhead won't be that different.

PatchWorKs: i too would be interested in Snow's results. However, the fact that snow in ffdshow has been broken for a long time now doesn't help testing it. As for theora, it really would hurt it, i think, if you were to make a comparison with the codecs used here.

[stephanV]

Originally posted by Manao
StephanV : i doubt anyone can compute psnr from a raw h264 stream.
But that isnt necessary, you only need the raw data to determine the size of the video stream. I think the container would not influence the PSNR right?

Moreover, it's hard to get raw data size for some codecs. I still think that choosing the container & tha application that minimizes the filesize for a particular codec is still the way to go.

And if each codecs are tested with their 'optimal' container, somehow, i guess the overhead won't be that different.
I would advice everyone to use AVIMux GUI. (matroska, set video lacing to 5 frames. :) )

[Sagittaire]

The problem is: container is part of "codec" ?

It's true : the efficacity of ovehead container is not user problem ... it's developper problem. Developper must use the best possible container ...

It's wrong : codec can or must use different container for X or Y reason. AVC use mp4, avi or mkv ; VP6 use avi and .vp6 ... for example.

Like say Manao use the best possible for each codec is perhabs good solution ... but for information I will indicate the real bitrate of video ES. And perhabs size overhead is not very different (I hope).

[Tommy Carrot]

Originally posted by PatchWorKs
What about SNOW ? I tried, and it looks like it's finally not borked in the latest ffdshow, but due to the lack of ratecontrol, i could only get the bitrate reasonably close to the target, but it's impossible to force the filesize exactly to the target range, so those encodes are not really relevant. But based on them i would say that Snow has slightly higher PSNR than the ASP codecs, but lower than the h.264 codecs.

About the container issue: simply use the container which is most widely used with the given codec (in other words leave them as they are now), that would give the most meaningful results IMO (as far as metric measures can be meaningful anyway).

[stephanV]

does anyone have a 3ivx Beta he/she is allowed to try? Their official codec from the site doesnt even outperform NanDub...

[IgorC]

AutoSSIM from italian Doom9 http://forum.doom9.it/download.php?id=853&sid=422a2f090dc3ae25098cf232eaab8e54. it very comfortable

P.S. It supports only MPEG1(2)

[Sagittaire]

Originally posted by Manao
Btw, you praise psnr overall over psnr average when it comes to rate control comparison, however ssim is computed as average psnr is, so ssim should only be used to compare frames together, not to compare rate control algorithms.



|---------------|---------|---------|
| Variability | OPSNR | SSIM |
|---------------|---------|---------|
| 000% | 39.3871 | 64.05 |
| 010% | 39.6695 | 65.23 |
| 020% | 39.9377 | 66.31 |
| 030% | 40.1266 | 67.08 |
| 040% | 40.2793 | 67.70 |
| 050% | 40.3925 | 68.19 |
| 060% | 40.4644 | 68.50 |
| 070% | 40.5003 | 68.66 |
| 080% | 40.4851 | 68.63 |
| 090% | 40.4357 | 68.51 |
| 100% | 40.3381 | 68.13 |
|---------------|---------|---------|


SSIM seem very good too for RC tweak ... perhabs not classic average calculation for SSIM ... :confused:

[IgorC]

The same result here. I obtained best aver. SSIM at 75-76% VBR, 50% Key Boost, 35-40% B reduction , deblocking -2 or -3. But I´m not sure if subjective quality is better, may be even worse.

[SpaceV]

what I would like to know, are the On2 guys responsive
to your test results, do they agree, do you get updated versions
of VP7 ?
do you see results and improvements, is there progress?

please let us know at least that much.

I am very interested in VP7.

Thanks!

[Sharktooth]

SpaceV you posted the same questions in the other thread.
Dont spam over the forum, it is an unwanted behaviour and goes against the forum rules.

[Sagittaire]

test in progress ... lol

I try to make better result with VP7 (only RC tweak) than Nero H264 (with best setting in recode)

[SpaceV]

Sorry, I thought not every VP7 tester is reading both
message threads and to increase my chance of getting an answer back..... which I still have not, at least not to my questions.

[Sagittaire]

Update:
New complete test
New samples with better setting
Metric faq
Archive in download

You think that your codec is the best : prove that ... !!?

[Manao]

The best codecs (H264 or VP7) are 75 % better than MPEG2 MP@ML: MPEG2 MP@ML with good encoder is always a good codec.That doesn' mean anything at all.

[IgorC]

Sagitarrie, are you going to do 2-CD Rip (maybe 1/4 , 1/3 , 1/2 of DVD RIP, HD-DVD 5mbit (raw video source) ) test later? It will be very interesting to know about performance of each codec (AVC, ASP, MPEG2 , etc)

[Manao]

Frankly, looking at the quality of the 450 kbit samples, knowing that a trailer is notoriously more complicated to encode than a movie, a 2CDs encode is imho an overkill now, even for such a long movie as Harry Potter. Anyway, in this particular case, 450 kbit is 1CD, and 900 kbit is 2CDs.

[Sagittaire]

Originally posted by Manao
That doesn' mean anything at all.

I use the same logic than One2, Microsoft or DXN ...

MPEG2 must use 1560 Kbps for the same "metric quality" than H264 900 Kbps : 1560 is 75% more than 900 ...

example: VP(x+1) is 15% better than VPx, DivX is 30% better than WMV9, WMV9 is 50% better than MPEG2 ... ect ect ect

All these company use metric to make these equivalence!


Originally posted by Manao
Frankly, looking at the quality of the 450 kbit samples, knowing that a trailer is notoriously more complicated to encode than a movie, a 2CDs encode is imho an overkill now, even for such a long movie as Harry Potter. Anyway, in this particular case, 450 kbit is 1CD, and 900 kbit is 2CDs.


Source and bitrate are not important because I use quant reference for my test: With this source I use q4 XviD for "1CDR Quality" bitrate reference and q8 XviD for "Streaming Quality" bitrate reference. Q4 in 720*304 use praticaly (perhabs 10% less) the same bitrate than Q3 in 640*272. For example my last encoding (The incredibles) use average ~q4 (for I,P frame) in 720*304 like the majority of my 1CDR encoding. Encoding with 50% for compressibility (Q2 H263 for reference) is an average ~Q4 encoding ...

[SpaceV]

guys, on the yahoo message board,
some people complain about the bad quality
of the new VP7 clips. There seem to be more that dislike
the new quality than people that praise it.

Some say that the first VP7 clips looked much better.

I think you can see it the best at the end of the Troy
trailer. The clouds with the text in the forgeround look really bad
and blocky. Even the ones at @700!

Did anyone watch those clips there, do you guys think they
represent the best ablilities of VP7.

[Sagittaire]

Originally posted by SpaceV
guys, on the yahoo message board,
some people complain about the bad quality
of the new VP7 clips. There seem to be more that dislike
the new quality than people that praise it.

Some say that the first VP7 clips looked much better.

I think you can see it the best at the end of the Troy
trailer. The clouds with the text in the forgeround look really bad
and blocky. Even the ones at @700!

Did anyone watch those clips there, do you guys think they
represent the best ablilities of VP7.

and ... :confused:

Troy clip use perhabs high quant because it's hard trailer ... bitrate and quality are 2 very differents notions. With XviD, q1 at 150 Kbps is possible and q10 at 1500 Kbps is possible and with the same resolution ... lol

If you want compare use another codec with the same source ... :devil:

[aeternitas]

You have a couple problems that shake ones faith in this being an accurite comparison.

1. +/- X = Unfair comparison, where X is anything other than 0.
3. Each compressed sample should contain exactly the same source with no dropped frames unless a rule states it as required for a better overall finished product.
2. Ones faith in your methods are not exactly secured when one realizes you didn't even use the right compression method to compress your graphs. Try PNG8.

[sparky]

Something is still wrong with your numbers.

Firstly, the script you suggest for measuring OPSNR is faulty.

LoadPlugin("dgdecode.dll")
LoadPlugin("CompareYV12.dll")
source=Mpeg2Source("mpeg2\HPII.d2v")
source=Trim(source,70,3145)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)

#video=DirectShowSource("DivX6.avi",fps=25)
video=DirectShowSource("XvID-450.mkv",fps=25)

# --> PSNR analysis <--
compareYV12(video,source,"YUV","PSNR-DivX6-450.log")


This will not work because the default output format of DirectShowSource is YUY2. At least it is so in avisynth 2.5.5.0 ( which is the latest available on doom9 ), with decoders from xvid 1.0.3 and divx fusion.

Secondly, even after you add an explicit ConvertToYV12(), CompareYV12.dll you gave in the first post does not work. I tried it with avisynth 2.5.5.0, VirtualDub 1.5.0 and VirtualDubMod 1.5.10.1. I presume the destructor never gets called ( no numbers are output to the log ).

Thirdly, the PSNR numbers in the first post are different from what I receive from standard avisynth Compare(). SSIM's are exactly the same, but my OPSNRs are about 0.7 db higher than yours.

You have these numbers:
DivX 450 kbps - 39.5729 db
XviD 450 kbps - 39.6601 db
DivX 900 kbps - 42.6220 db

With your latest clips I see:
DivX 450 kbps - 40.3340 db
XviD 450 kbps - 40.2208 db
DivX 900 kbps - 43.3076 db

This is not a big deal because the algorithms used by Compare() and CompareYV12() could be different.

Finally, the settings you actually used to create DivX clips are somehow different from both my suggested settings and your claimed settings. Using standard avisynth Compare() and SSIM, I got these results:

your clip ( 6,918,301 bytes ) - OPSNR 40.3340 db; SSIM 63.49
my own encoding using your claimed settings and bitrate 444 kbps ( 6,907,904 bytes ) - OPSNR 40.4812 db; SSIM 64.27
my encoding with the same settings minus GMC ( 6,907,904 bytes ) - OPSNR 40.5055 db; SSIM 64.42

For reference, my DivX CLI:
-bvn1 444000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2

[Sagittaire]

Originally posted by aeternitas
You have a couple problems that shake ones faith in this being an accurite comparison.

1. +/- X = Unfair comparison, where X is anything other than 0.
2. Each compressed sample should contain exactly the same source with no dropped frames unless a rule states it as required for a better overall finished product.
3. Ones faith in your methods are not exactly secured when one realizes you didn't even use the right compression method to compress your graphs. Try PNG8.

1) +/- 1 kbps for Elementary Video Stream (VES) represent +/- 15Ko for 450 Kbps sample. My tolerance is good bacause it's very diffucult to make better and don't change the result (Overhead Problem, RC target, container ... ect ect). In my test the maximum diff for VES is 22 Ko for "6750 Ko sample" and 28 Ko for "13500 Ko sample". Try to make better if you want ...

2) it's generaly a splitter problem but 1 frame dropped (for example last frame like for RV10) and don't change the result. It's only 1 frame for 3075 and the global test don't change:


3074 0.4859 +0.0578 18 -15 48.1109 39.4580
0 0.0214 -0.0159 3 -3 64.4317 39.4594


the "0" frame is dropped by the RealSplitter and Overall PSNR is increase to 0.0014 dB with artificial black frame ...

3) it's very important ... :confused:
it's a test for video codec and not for picture codec ... lol

[Sagittaire]

Originally posted by sparky
Something is still wrong with your numbers.

Firstly, the script you suggest for measuring OPSNR is faulty.

LoadPlugin("dgdecode.dll")
LoadPlugin("CompareYV12.dll")
source=Mpeg2Source("mpeg2\HPII.d2v")
source=Trim(source,70,3145)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)

#video=DirectShowSource("DivX6.avi",fps=25)
video=DirectShowSource("XvID-450.mkv",fps=25)

# --> PSNR analysis <--
compareYV12(video,source,"YUV","PSNR-DivX6-450.log")


This will not work because the default output format of DirectShowSource is YUY2. At least it is so in avisynth 2.5.5.0 ( which is the latest available on doom9 ), with decoders from xvid 1.0.3 and divx fusion.

Secondly, even after you add an explicit ConvertToYV12(), CompareYV12.dll you gave in the first post does not work. I tried it with avisynth 2.5.5.0, VirtualDub 1.5.0 and VirtualDubMod 1.5.10.1. I presume the destructor never gets called ( no numbers are output to the log ).

Thirdly, the PSNR numbers in the first post are different from what I receive from standard avisynth Compare(). SSIM's are exactly the same, but my OPSNRs are about 0.7 db higher than yours.

You have these numbers:
DivX 450 kbps - 39.5729 db
XviD 450 kbps - 39.6601 db
DivX 900 kbps - 42.6220 db

With your latest clips I see:
DivX 450 kbps - 40.3340 db
XviD 450 kbps - 40.2208 db
DivX 900 kbps - 43.3076 db

This is not a big deal because the algorithms used by Compare() and CompareYV12() could be different.

Finally, the settings you actually used to create DivX clips are somehow different from both my suggested settings and your claimed settings. Using standard avisynth Compare() and SSIM, I got these results:

your clip ( 6,918,301 bytes ) - OPSNR 40.3340 db; SSIM 63.49
my own encoding using your claimed settings and bitrate 444 kbps ( 6,907,904 bytes ) - OPSNR 40.4812 db; SSIM 64.27
my encoding with the same settings minus GMC ( 6,907,904 bytes ) - OPSNR 40.5055 db; SSIM 64.42

For reference, my DivX CLI:
-bvn1 444000 -vbv 7800000,1835008,1376256 -pq 8610 -b 3 -nf -quantization=2

1):readrule: ... lol


You must mux or remux x264, VP6, VP7, WMV9, DivX6 and XviD in avi container (with VDM for example) to solve problem with color space output, synchronisation or acquisition for metric test. You must use AviSource in avisynth for VP6, VP7 and WMV9. You must use DirectShowSource in avisynth for MPEG4 ASP, MPEG4 AVC and RV10. You must change trim for frame synchronization or for clip length if you have some probleme with SSIM plugin. You must use these AviSynth type script for metric test:


I use matroska container for optimized Overhead size and compare [codec+container] with the same bitrate for Video Elementary Stream


2) You must use AviSource for VP6 (don't work with DS), VP7 (YUY2 output only with DS) and WMV9 (desynchro with DS). You must use DirectShowSource with MPEG4 ASP (you must force YV12 output in XviD dec and DivX dec)

CompareYV12 has little bug: you must open a first time and make preview in VD and open a second time

PSNR in YV12 and in YUY2 are not equivalent: it's better to use complete YV12 preocess because all codec use YV12. In YUY2 4:2:2 the chroma is more important and better codec for chroma could have better result ...

3) I will update with your setting: IMO it's very strange that GMC decrease metric : GMC decision is perhabs not good for this trailer ... I will update too for XviD ... lol

[sparky]

You must use AviSource for VP6 (don't work with DS), VP7 (YUY2 output only with DS) and WMV9 (desynchro with DS). You must use DirectShowSource with MPEG4 ASP (you must force YV12 output in XviD dec and DivX dec)


DivX dec can't be forced into YV12 output, it does not have such feature. Only XviD dec can do that. For DivX, DirectShowSource() gives me YUY2 regardless of container format. Yes, I do have "Output YUV 4:2:0 when supported" enabled.


CompareYV12 has little bug: you must open a first time and make preview in VD and open a second time


I tried opening, previewing, opening again, opening other clip, explicitly closing .avs with Ctrl-W, etc. Right now my log file looks like this:


Comparing channel(s) YUV

Mean Max Max
Absolute Mean Pos. Neg.
Frame Dev. Dev. Dev. Dev. PSNR (dB) Avg. PSNR
---------------------------------------------------------------

Comparing channel(s) YUV

Mean Max Max
Absolute Mean Pos. Neg.
Frame Dev. Dev. Dev. Dev. PSNR (dB) Avg. PSNR
---------------------------------------------------------------

Comparing channel(s) YUV

Mean Max Max
Absolute Mean Pos. Neg.
Frame Dev. Dev. Dev. Dev. PSNR (dB) Avg. PSNR
---------------------------------------------------------------

Comparing channel(s) YUV

Mean Max Max
Absolute Mean Pos. Neg.
Frame Dev. Dev. Dev. Dev. PSNR (dB) Avg. PSNR
---------------------------------------------------------------

Comparing channel(s) YUV

Mean Max Max
Absolute Mean Pos. Neg.
Frame Dev. Dev. Dev. Dev. PSNR (dB) Avg. PSNR
---------------------------------------------------------------

Comparing channel(s) YUV

Mean Max Max
Absolute Mean Pos. Neg.
Frame Dev. Dev. Dev. Dev. PSNR (dB) Avg. PSNR
---------------------------------------------------------------

Comparing channel(s) YUV

Mean Max Max
Absolute Mean Pos. Neg.
Frame Dev. Dev. Dev. Dev. PSNR (dB) Avg. PSNR
---------------------------------------------------------------


(you've guessed the rest...)

IMO it's very strange that GMC decrease metric : GMC decision is perhabs not good for this trailer

GMC does not make a lot of difference here. Something else must be affecting your encodings ( 0.75 difference of SSIM scores between your & my clips ).

[aeternitas]

Originally posted by Sagittaire
1) +/- 1 kbps for Elementary Video Stream (VES) represent +/- 15Ko for 450 Kbps sample. My tolerance is good bacause it's very diffucult to make better and don't change the result (Overhead Problem, RC target, container ... ect ect). In my test the maximum diff for VES is 22 Ko for "6750 Ko sample" and 28 Ko for "13500 Ko sample". Try to make better if you want ...

2) it's generaly a splitter problem but 1 frame dropped (for example last frame like for RV10) and don't change the result. It's only 1 frame for 3075 and the global test don't change:



the "0" frame is dropped by the RealSplitter and Overall PSNR is increase to 0.0014 dB with artificial black frame ...

3) it's very important ... :confused:
it's a test for video codec and not for picture codec ... lol


1. 1 frame does change the result. It results in an imperfect comparison based on guessing instead of facts. 1 frame is infintly more than 0 frame loss. It's not as important as other factors, but its not something you throw to the wind if you want a serious comparison.

2. Yu should be useing the same container types and _everything else besides the video codec should be identical_ to the point where the only changes would be from compatibility issues cuased by the video codec.

3. Yes, your lack of knowlage in image compression tells more than you care to admit about how youre not very well rounded in the general compression scene.

This comparison is flawed. Flaws that can be fixed. If your comparison were an application it would be in beta.

You need to upload images of your compared scenes. PSNR and whatnot are lame representations of quality. They are not more representitive of quality than more bps is. More does not always mean better.

[huang_ch]

I'm really confused about that why RV10 get the WORST score even in 450Kbps. My original knowledge is that RV10 is really good at low bps, while XVID/DIVX/WMV codecs are more focused at higher bps and don't play well in low bps compare to RV9/10, but in this test I saw XVID/DIVX/WMA are better than RV10, can anyone tell me why?

[Shinobu]

metrics and eyes are different, that's all ^^.
for exemple if you sharp only a little a video, it may look better for you, but for metrics it'll look like hell.

metrics are cool for "pure math test" , the only way to corectly compare codecs power is to do blind test and use your eyes ^^.
on the same encoding, i can find rv10 better and you can find vp7 better, it's a matter of taste, someone perfers a little moscito noise but mutch details, some other no mosquito but less details.... and that can't be calculated by metrics.

for my eyes rv10 looks also better than any others at low bitrates (may be not nero avc in non-anime case), but eyes are not metric ...

++

[Sagittaire]

Originally posted by aeternitas
1. 1 frame does change the result. It results in an imperfect comparison based on guessing instead of facts. 1 frame is infintly more than 0 frame loss. It's not as important as other factors, but its not something you throw to the wind if you want a serious comparison.

2. Yu should be useing the same container types and _everything else besides the video codec should be identical_ to the point where the only changes would be from compatibility issues cuased by the video codec.

3. Yes, your lack of knowlage in image compression tells more than you care to admit about how youre not very well rounded in the general compression scene.

This comparison is flawed. Flaws that can be fixed. If your comparison were an application it would be in beta.

You need to upload images of your compared scenes. PSNR and whatnot are lame representations of quality. They are not more representitive of quality than more bps is. More does not always mean better.



You need to upload images of your compared scenes.

very stupid comparison method : your lack of knowlage in video compression tells more than you care to admit about how youre not very well rounded in the general compression scene. Images comparison (PNG, JPG or uncompressed 4:4:4 RGB 1024 bit ... lol) are very unable to make video codec comparison. Local difference like I,P,S,B,b or RC difference are not the same for all codec. Conclusion for N frame is not the same that conclusion for N+1 frame. If you want make visual test download sample ...



:readrule:

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...


Read pdf SSIM ... if you contest SSIM test speak with dev and not with me ... it's perhabs Lame like you say ... :devil:



1) not correct and I prove that when you want. One frame for 3075 don't change the result: with my source (Perhabs maxi 0.005 dB less or more for OPSNR and the first and last frame for this source is black frame for solve this problem). Source is the same for all codec ... dropped frame is little parser or decoder problem

2) not correct and I prove that when you want. I can use different container if I want. The most important is the VES bitrate. In my test bitrate for VES and files are the same ... mp4 container or matroska container use in practice the same overhead for this test.

3) I use JPG if I want for my graph ... it's not picture compression test ... you are really strange or particulary ... "Lame" like you say ... :confused:

[Sagittaire]

for exemple if you sharp only a little a video, it may look better for you, but for metrics it'll look like hell.

That does not mean anything and you did not understand what must be a codec. Sharp it's an post-process modification possible with all codec ... but you must compare equivalent frame and Sharp frame isn't the same frame than source. Source with denoising look better than original source but the codec objective is not denoising: codec must restitute the noise if source is noisy. Metric compare convergence between input and output codec ... that's all.



@ huang_ch & Shinobu

:readrule:

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...


Read pdf SSIM ... if you contest SSIM test speak with SSIM dev and not with me ...

If you want speak about visual quality vs metric test open another thread

For end you must use the best Post-Process and decoder for all codec:
- PP1 for WMV9 (http://multimediacom.free.fr/Video/WMVPostpross.exe)
- PP4 with DivX and DivX dec
- PP4 with XviD and XviD dec
- Don't use HFE 2.1 for RV10 with sharp but HFE 1.

you must see too all frame of the trailer: for exemple x264 isn't good for the first scene (warner presentation) visually and with metric but after the codec is very good ...

[aeternitas]

1) not correct and I prove that when you want. One frame for 3075 don't change the result: with my source (Perhabs maxi 0.005 dB less or more for OPSNR and the first and last frame for this source is black frame for solve this problem). Source is the same for all codec ... dropped frame is little parser or decoder problem

2) not correct and I prove that when you want. I can use different container if I want. The most important is the VES bitrate. In my test bitrate for VES and files are the same ... mp4 container or matroska container use in practice the same overhead for this test.

3) I use JPG if I want for my graph ... it's not picture compression test ... you are really strange or particulary ... "Lame" like you say ... :confused: [/B]

0."Conclusion for N frame is not the same that conclusion for N+1 frame." Exactly why ONE FRAME MATTERS in these sorts of tests. The whole would-be sequance has a good chance of being changed for several frames down the line.

Also, you can definitly find and pull a P or B frame that is placed in a similer set string of the sequance thoughout each encode.

1. You can make excuses to how negligible that is all day, but a difference is a differance in the source.

2. You can use any container you want, I will give you that, as long as the source for all encodes are identical and the bps given to each second of video per codec is the same.

3. You can use BMP if you want. Thats not the point. The point is its on the same level. Its a compression. I just found it ironic that you used the worst possible compression method for those images while you're trying to convince people how wonderfully fair and thought out your video compression tests are.

[Manao]

sparky : compareyv12 works a lot better if instead of giving as a filename 'foo.txt', you give the full path.

sagittaire : it's not because firms use their own twisted logic to say that their codec is 75 % better than another that you should do the same. What can be said is that, coming from the same source, and aiming at the same quality ( in that case, PSNR ), an h264 encode's size will be between 50 to 70 % of the size of the mpeg2 encode's one.

aeternitas : i found some of your critics miss the point. Why are you attacking Sagittaire on the compression used for his graph's screenshots ? Indeed, png was the best choice, but perhaps jpg was the most convenient for him ( it depends on the program he used to save his graph, after all ). That is not correlated to his encoding skills at all and that's imho a gratuitous attack.

Then, you're criticizing some approximations made by Sagittaire. Indeed, there are some : bitrates aren't all equal, but can't be except if you're ready to keep encoding over and over the same clip with the same codec in order to bypass ratecontrol imprecision and containers' discrepancies. Indeed, the looseness on the missing frames & co is unwelcomed, but circumvent them would have taken some time and the imprecision they bring is by far smaller than anything you could notice with your eyes.

I still wonder, btw, why you're asking Sagittaire to post screenshots, when he gave access to the whole clips ( source + results )

So if you want to have a row with him, make it private please.

huang_ch : download the resulting clips and make your own visual impressions. PSNR and SSIM and only slightly correlated to visual impression.

[Sharktooth]

Here's another "i know it all, you all suck (http://forum.doom9.org/showthread.php?s=&threadid=89093&perpage=20&pagenumber=2#post622608)" newbie...
aeternitas please keep your comments private, they're not relevant.

[Sagittaire]

Originally posted by Manao
sagittaire : it's not because firms use their own twisted logic to say that their codec is 75 % better than another that you should do the same. What can be said is that, coming from the same source, and aiming at the same quality ( in that case, PSNR ), an h264 encode's size will be between 50 to 70 % of the size of the mpeg2 encode's one.

yes it's true ... perhabs not very pertinent comparison:

Equivalent metric for MPEG2 at 1560 Kbps and MPEG4 AVC at 900 Kbps

With H264 reference bitrate "H264 is 73% better than MPEG2"
With MPEG2 reference bitrate "H264 is 42% better than MPEG2"

but it's easy to understand for newbie than "h264 done same result than MPEG2 with X% less for bitrate" only with metric ...

Update in progress ...

[sparky]

Originally posted by Manao
[B]sparky : compareyv12 works a lot better if instead of giving as a filename 'foo.txt', you give the full path.


!!!

Everything works now, thanks

[hdonly]

http://on2.com/duckutils/duck_license.php3?class=player

Fixes CPU complexity problem. Clips look better too!

[eqbal]

how can i determine PSNR value for a movie?
is there any software?

[Manao]

Search the forum for Video Quality Studio ( and more generally, search the forum for such question )

An alternative way could be avisynth ( the compare or compareYV12 function ), but it is more complicated if you don't know how to use avisynth.

[trbarry]

It might be interesting to have a test like this where everybody instead had to match or beat the same SSIM, say 80 or better.

The winner would be the one with the codec/parms/container with the smallest total file size. Then you could truly say one codec was xx% better than another, at least for some given clip and agreed on SSIM target.

The only true test is probably to view the moving videos but I still tend to trust the metrics for comparisons.

- Tom

[Sagittaire]

@ trbarry

perhabs next test with HDTV resolution ...

[trbarry]

perhabs next test with HDTV resolution ...

That would be near & dear to my own heart. ;)

But there is a problem with testing real HD sources because there aren't many to test. Most HD captures seem to need only maybe 544p or at most 720p to duplicate since they really don't have any more effective detail than that anyway, and a little is always lost in re-encoding. So maybe a nice test in that range would be appropriate if enough folks here were interested in HD yet.

- Tom

[Sagittaire]

Update

- New VP6 Trailers
Better Rate Control setting

- New VP7 Trailer
Encoding with PIV class CPU for "450" trailer only. VP7 seem little buggy with MMX/SSE class CPU like PIII.

- New DivX Trailers
For the first time DivX is better than XviD for OPSNR and SSIM. Adaptative Quantisation (psy mode) is not optimized with DivX Fusion : with good A.Q DivX will be very better for SSIM. It's perhabs possible to make better encoding with RC tweak for XviD/DivX (variablity and bframe ratio/offset) ... if MPEG4 ASP's fan want to try ... ???

- New RV10 trailers
Very better Rate Control setting


Technical Information

Originally posted by huang_ch
I'm really confused about that why RV10 get the WORST score even in 450Kbps. My original knowledge is that RV10 is really good at low bps, while XVID/DIVX/WMV codecs are more focused at higher bps and don't play well in low bps compare to RV9/10, but in this test I saw XVID/DIVX/WMA are better than RV10, can anyone tell me why?

1) My eyes seem say that too. And for the first time I don't understand why. Perhabs that it's true : metric are unable to test video quality ... !!?


|---------------|-----------|------------|----------|---------|---------|
| Codec | P-Process | ES Bitrate | Size | OPSNR | SSIM |
|---------------|-----------|------------|----------|---------|---------|
| DivX | PP4 | 447 kbps | 6757 Ko | 39.5729 | 63.49 |
| XviD | PP4 | 447 kbps | 6760 Ko | 39.6601 | 64.36 |
| RV10 old | HF1 | 447 kbps | 6767 Ko | 39.4594 | 62.83 |
|---------------|-----------|------------|----------|---------|---------|
| DivX | PP4 | 896 kbps | 13490 Ko | 42.6220 | 77.47 |
| XviD | PP4 | 896 kbps | 13498 Ko | 42.6654 | 77.95 |
| RV10 old | HF1 | 896 kbps | 13501 Ko | 42.5815 | 77.21 |
|---------------|-----------|------------|----------|---------|---------|


2) I make Graph PSNR test to compare with other codec and I see that with default RC setting for RV10:

RV10 vs XviD
http://multimediacom.free.fr/Video/RV10-Problem.PNG

With this trailer and defaut RC setting the last frames of the trailer are dramaticaly worst than all other codec. It's typicaly a Rate Control Problem. In fact RV10 RC use to high size for the begin and too low size for the end. Visually and for metric these last frames MPEG4 ASP and all the other codec are very better than RV10.


3) I remade "450" encoding with new and really optimized metric RC tweak
- Size quant prediction: rcPFrameRefQuant = 17 for better prediction
- Bframe ratio : rcBFrameRefQuant = 23 for better metric result
- Rate Contro : rcLowBitrateBoost = rcHighBitrateReduce = 20 for better metric result

I remade too "450" with old RC and it's too very better than Elysian RC with defaut RC setting for this trailer. The new encoding are very better for the complete trailer and very better for the last frames.


|---------------|-----------|------------|----------|---------|---------|
| Codec | P-Process | ES Bitrate | Size | OPSNR | SSIM |
|---------------|-----------|------------|----------|---------|---------|
| DivX | PP4 | 446 kbps | 6755 Ko | 39.7552 | 64.44 |
| XviD | PP4 | 446 kbps | 6760 Ko | 39.6601 | 64.36 |
| RV10 new | HF1 | 446 kbps | 6743 Ko | 40.0457 | 65.52 |
|---------------|-----------|------------|----------|---------|---------|
| DivX | PP4 | 896 kbps | 13508 Ko | 42.7455 | 77.93 |
| XviD | PP4 | 896 kbps | 13498 Ko | 42.6654 | 77.95 |
| RV10 new | HF1 | 896 kbps | 13493 Ko | 42.7431 | 77.73 |
|---------------|-----------|------------|----------|---------|---------|


RV10 update vs RV10 previous
http://multimediacom.free.fr/Video/RV10-Solve.PNG

4)
Originally posted by Shinobu
for my eyes rv10 looks also better than any others at low bitrates (may be not nero avc in non-anime case), but eyes are not metric ...


Yes, your (and my) eyes are not metric : your (and my) eyes were unable to detect this Rate Control artefact and very visible problem for the 250 last frames perhabs simply because the clip was not entirely views ... perhabs because for eyes the first scene is more important than the last ... perhabs because it's very difficult for eyes to compare 3075 frames and make objective overall observation ...

:readrule: :readrule: :readrule: :readrule: :readrule: :readrule: :readrule: :readrule:

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...


Conclusion: Make yourself the conclusion ... !!!

[IgorC]

x264 rev157? Why sample of x264 wasn´t updated? The last revision has better SSIM result.

[Sagittaire]

perhabs 6 or 7 days between rev 157 and rev 173 ... lol

bref frame doen't work wery well at this time (dec problem for me) ... rev 173 is perhabs little better than rev 157 but not revolution ...

[IgorC]

What significate MinQ 8/16? Is it MinQ=8 for 900kbps and MinQ=16 for 450kbps, right?

EDIT
660 kbit/s test
rev 157 - SSIM 77.64
rev 174 - SSIM 77.76

[Sagittaire]

Originally posted by IgorC
What significate MinQ 8/16? Is it MinQ=8 for 900kbps and MinQ=16 for 450kbps, right?

yes ... One2tech RC optimisations


Originally posted by Tommy Carrot
There were other changes in x264 other than bref, chroma ME should improve the quality by quite a bit, for example. But if you want to wait for the bigger improvements, wait a few days, as RDO support seems to be very near to completion.

I will wait ... :)

[IgorC]

Why Ateme encavc beta4a used as NERO AVC codec in this test? Who has it? Only beta-testers had this opportunity. ateme beta4a lets to put 16r ref while Recode2 only 8 ref. So is it fair to compare on this mode?

[Sagittaire]

ateme beta4a and current ND AVC for recode are very similar (I make test for that in ateme beta thread). The only difference is maxbframe number and maxpref number. 8 or 16 for maxpref is not very important ... with more 5 pref the metric gain is very little with MPEG4 AVC ...

I use ateme CLI because batch mode is very good for test ...

[IgorC]

Sagittaire, what version of Xvid 1.1 beta 1 do you use in your last test?
http://www.aziendeassociate.it/./cd///XviD.cvs.head.exe (14 march)
in 660kbit/s test it gives more +0.1 SSIM than Xvid 1.1 beta 1 official 16 jan. since in this test all kind of version ( release, revision, beta etc.). The difeernce between Divx-Xvid is too small, so it will be enough to say that Xvid better (iqual/better on 450kbit/s) than Divx6

[Sagittaire]

In fact I use special RC metric tweak for XviD Koepi 1.1.0 beta 1 like for RV10:

- Size quant prediction: first pass in Q4/Q8 for 900/450 trailer
- Variability: HighBitrateReduce and LowBitrateBoost optimisation

perhabs possible to make better with bframe ratio/offset ... if XviD fan want make test ...

[huang_ch]

Great, thanks for the explanation for RV10 issues. And I know metric is only metric, I've made several visual test these days, I think to my eyes RV10 and Nero AVC is almost at the same level in low bitrate for Animes (300-500kbps), and x264 is a bit worse than them. I don't try the VP6 & VP7 codec. Since I'm not experts in doing this kind of test, so indeed I can't distinguish the quality between RV10 & AVC even by frame and frame in VirtualDub. But for the encoding time, I think AVC is 1.x-2.x slower than RV10 and x264 is at least 2X faster than RV10. :cool:

[IgorC]

Originally posted by Sagittaire
In fact I use special RC metric tweak ..

Saggitarie. Can you answer to some my questions relationed to XVID and x264?
Does it mean 1st past at Q8 (for 450kbit/s) and 2 pass as usual with decribed settings ?
What setings fot HighBitrateReduce and LowBitrateBoost did you use?
What setings of x264 (KEyBoost and Breduction) did you use?. I obtain the same SSIM for your trailer x264-450 (68.24), but my trailer encoded with the same rev157 and same settings gives me another result.
Where can I find HFE1? There is only HFE2


x264(your trailer) rev157 SSIM=68.24 , OPSNR = 40.4423
VP7 (your trailer) SSIM=68.57 , OPSNR = 40.6802
x264 rev174 SSIM=68.77 , OPSNR = 40.4737

settings for x264 rev174 - Cabac, Deblock -1, Bframes 2 (Temp), Weight, ref15, keyboost 50% , Breduc 40%, VBR 75%, 5 max Quality , macroblock partitions ALL on. I don´t´know if these settings are best for this source but parametlicly (and visually too) gived a better result.
As a conclusion , x264 > VP7 for SSIM test.
x264 < VP7 for OPSNR test.
But maybe there is oportunity to find better balance between OPSNR and SSIM changing values for RC.

[redfordxx]

Originally posted by Sagittaire
A - Introduction
The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...
Hi,
with metrics can be measured other parameters. I cannot download samples ATM, but I guess there could be some blocks:). I suggest following script to measure amount of blocks. Maybe someone can say metrics are not relevant compared to visual tests. I doubt it can be said about this blocktest.LoadVirtualdubPlugin("...\deblock.vdf", "MSUDeBlock")
....

video=DirectShowSource("G:\...\RV10-450.mkv",fps=25)
deblocked=video.ConvertToRGB32(matrix="pc.601").MSUDeBlock(1,1,1).ConvertBackToYUY2(matrix="pc.601").ConvertToYV12()

# --> PSNR analysis <--
return SSIM(deblocked,video,"results.csv","SSIM-RV10-450.txt",lumimask=true) Maybe you find it useful to add one column to your table for this (+value for source itself).
I think some other artifact tests could be invented...just to find the correct filter which aims on the specific problem of codec, e.g. mosquito noise (well I read of it here often, but can someone describe exactly, what is it?).

[temporance]

Neat idea, but this only measures blocks that can be deblocked. Other blocks (such as those translated by motion compensation) cannot be deblocked and will not be measured by your script.

Bear in mind that there is a whole industry devoted to perceptual video quality measurement - VQEG uses all sorts of signal processing techniques in its various metrics.

[Sagittaire]

Originally posted by redfordxx
with metrics can be measured other parameters. I cannot download samples ATM, but I guess there could be some blocks:). I suggest following script to measure amount of blocks. Maybe someone can say metrics are not relevant compared to visual tests. I doubt it can be said about this blocktest.

I use PP4 (deblocking YUV in post-process) for best result with MPEG4 ASP. With PP4 metric are very better than PP0 (perhabs ~1 dB better)

Conclusion: metric can detect blocking ...

[redfordxx]

Originally posted by Sagittaire
I use PP4 (deblocking YUV in post-process) for best result with MPEG4 ASP. With PP4 metric are very better than PP0 (perhabs ~1 dB better)
I see, so you are not interested in real encode quality but in quality after PP... is that relevant? I read here many misunderstandings who measured what value, but it could be with different postprocessing...
Conclusion: metric can detect blocking ...
How do I decide (and check) which codec is used for decoding in avisynth script. I dont know, it could be XviD or FFDShow. I know in VDM is used XviD, althoug I'd like FFDS...

[Sagittaire]

Originally posted by redfordxx
I see, so you are not interested in real encode quality but in quality after PP... is that relevant? I read here many misunderstandings who measured what value, but it could be with different postprocessing...

How do I decide (and check) which codec is used for decoding in avisynth script. I dont know, it could be XviD or FFDShow. I know in VDM is used XviD, althoug I'd like FFDS...

Objective is best possible metric ... for best metric with MPEG4 ASP you must use PP4 (not PP2 or PP6 but PP4). PP4 is in fact real quality encoding : XviD dec or DivX dec use PP4 for default setting. Chip can use PP4 in hardware decoding. This test is co-dec test: I use DivX dec for DivX, XviD dec for XviD but ffdshow dec is not better than XviD dec ...

DirectShowSource use the default decoder, if XviD dec is active for XviD playback in MPC then DirectShowSource will use XviD dec in Avisynth script ...

[redfordxx]

Originally posted by Sagittaire
ffdshow dec is not better than XviD dec ...
But, there has been the issue with green color in XviD dec. Sure, I didnt check the recent releases, since I'm using FFDShow.

[WorBry]

Sagittaire "Testeur de Codecs"

I would be appreciate some advice on PSNR testing. I have been in the process of comparing different codecs for compression of captured Type II Pal DV videos and of course this has lead me into the realm of metric measurement comparison. Until recently, I have been using Everwicked's Video Quality Studio v3.2 to compare DivX, XviD and VP6.2 encodes. This version of VQS gives the raw PSNR values for Y, U and V, as well as VQM. For the reference file I've been using the same source AVS script that I now use for all DV conversions:

avisource("Test Pal DV Type II clip.avi")
ConvertToYUY2(interlaced=true)
ColorYUV(off_y=-16,gain_y=16)
KernelDeint(order=0,sharp=true)
LanczosResize(720,544)
Convolution3D(1,4,5,3,4,2.8,0)
RemoveGrain(mode=2)

(Note: the ColorYUV adjustment is to correct for the luma shift that occurs when using the Sony DV Software codec for decompression)

My simplistic approach to evaluating the data was to derive average PSNR values for each of the channels and...compare them. On this basis all three codecs were giving PSNR's in the typical range with VP6.2 as the clear leader. I wont go into details of the codec settings, but these were all 2-pass encodes at 2000kbps. From reading your posts I learn that is not the best way to do it and that I should be looking at Overall PSNR. In addition I've now extended my test repertoire to other formats including RV10 and NeroAVC and of course found that I could not load the encodes in VQS, being DirectShow and not vfw formats. So, I've tried to use the AviSynth Compare and CompareYV12 filters instead. Again I've been using the same source AVS script as the reference which outputs as YUY2. Consequently, I've been mostly using the Compare filter and then adding ConvertToYUY2() to the test file line (Avisource or DirectShowSource, as appropriate). Now I'm acheiving OPSNR results in the expected range and my conclusions, so far, are XviD > NeroAVC > DivX Fusion > DivX (I'll maybe post the results later, after I've played around with NeroAVC some more). I finally got RV10 to load as a DirectShowSource after installing RealMediaSplitter. VP6.2 appeared to load OK as an AviSource. However, both RV10 and VP6.2 give ridiculously low OPSNR values, down in the 15-20 range, which, for is contrary to the PSNR's I was acheiving for VP6.2 with VQS and those for RV10 recorded in the AutoRV10 PSNR log files - which are all up in the expected 40 - 43 range. From the results you posted at the start of this thread, you clearly dont have this problem. Any idea what I am obviously doing wrong and how to correct it?

Many thanks.

[On2Tech]

When you do CompareYV12(x,y,"","result.txt") the first of the two is what is displayed on the screen. I usually choose to set it up so that the decoded video is displayed rather than the source as this can sometimes give advanced warning if something is wrong.

My guess in your case is that the RV10 and VP6 clips may be decoding such that they are flipped vertically (upside down comapred to source) when played in this way through avisynth.

If this is the case you would of course get silly psnr numbers. This can be solved by adding a video=FlipVertical(video) before the compare.

For example

video=AviSource("vp6.avi")
video=FlipVertical(video)
CompareYV12(video,source,"","result.txt")

I hope this helps,

On2Tech

Originally posted by WorBry
Sagittaire "Testeur de Codecs"

I would be appreciate some advice on PSNR testing. I have been in the process of comparing different codecs for compression of captured Type II Pal DV videos and of course this has lead me into the realm of metric measurement comparison. Until recently, I have been using Everwicked's Video Quality Studio v3.2 to compare DivX, XviD and VP6.2 encodes. This version of VQS gives the raw PSNR values for Y, U and V, as well as VQM. For the reference file I've been using the same source AVS script that I now use for all DV conversions:

avisource("Test Pal DV Type II clip.avi")
ConvertToYUY2(interlaced=true)
ColorYUV(off_y=-16,gain_y=16)
KernelDeint(order=0,sharp=true)
LanczosResize(720,544)
Convolution3D(1,4,5,3,4,2.8,0)
RemoveGrain(mode=2)

(Note: the ColorYUV adjustment is to correct for the luma shift that occurs when using the Sony DV Software codec for decompression)

My simplistic approach to evaluating the data was to derive average PSNR values for each of the channels and...compare them. On this basis all three codecs were giving PSNR's in the typical range with VP6.2 as the clear leader. I wont go into details of the codec settings, but these were all 2-pass encodes at 2000kbps. From reading your posts I learn that is not the best way to do it and that I should be looking at Overall PSNR. In addition I've now extended my test repertoire to other formats including RV10 and NeroAVC and of course found that I could not load the encodes in VQS, being DirectShow and not vfw formats. So, I've tried to use the AviSynth Compare and CompareYV12 filters instead. Again I've been using the same source AVS script as the reference which outputs as YUY2. Consequently, I've been mostly using the Compare filter and then adding ConvertToYUY2() to the test file line (Avisource or DirectShowSource, as appropriate). Now I'm acheiving OPSNR results in the expected range and my conclusions, so far, are XviD > NeroAVC > DivX Fusion > DivX (I'll maybe post the results later, after I've played around with NeroAVC some more). I finally got RV10 to load as a DirectShowSource after installing RealMediaSplitter. VP6.2 appeared to load OK as an AviSource. However, both RV10 and VP6.2 give ridiculously low OPSNR values, down in the 15-20 range, which, for is contrary to the PSNR's I was acheiving for VP6.2 with VQS and those for RV10 recorded in the AutoRV10 PSNR log files - which are all up in the expected 40 - 43 range. From the results you posted at the start of this thread, you clearly dont have this problem. Any idea what I am obviously doing wrong and how to correct it?

Many thanks.

[WorBry]

Thanks On2Tech. You were right, the VP6.2 encode had become inverted in the Compare script and FlipVertical corrected for this. However, the RV10 encode was not inverted and I'm still trying to figure out why I'm getting these very low PSNR's with both CompareYV12 and Compare(YUY2).

[Richard Berg]

I'm getting weird numbers just trying to do the test. 24.5 PSNR, 11.1 SSIM. When I playback the original + encode with StackVertical, they line up fine all the way to the last frame (which I trim off in the comparison).

[Sagittaire]

Update 03/04/05

- x264 rev 184
- VP7 7.0.0.4

[IgorC]

0 is not optimal deblcoking for VP7 and x264 for this source. However it makes a scence to compare with deblocking filter by default = 0.

[Sagittaire]

Originally posted by IgorC
0 is not optimal deblcoking for VP7 and x264 for this source. However it makes a scence to compare with deblocking filter by default = 0.

yes ... it's always true for VP7 ... ???

I use inloop = 0 for x264 (best for x264's dev and other H264's dev) and post process level 2 (deblocking YUV I think and best for One2Tech). Perhabs possible that inloop +/- 1 could be ponctually the best for some source but difference for metric will be very ligth I think ...

could be good idea to make test with different deblock level:
deblock = -6 -> OPSNR = X6
deblock = -5 -> OPSNR = X5

[IgorC]

Yes, it was supposed before that settings of x264 would be discussed with devs of x264 and H.264
But for x264 3 b-frames isn´t optimal(at least for this source). The best is 2 bframes. I checked with the same source for SSIM test. And deblocking -1 is better than 0 (in most of cases, i didn´t tested anime). I have the more or less (a little desviation from your results +/- 0.02-0.03 )same SSIM, OPSNR for VP7, but with settnigs 2b + deblock -1 x264 has a higher SSIM (although visually VP7 is better).

[Sagittaire]

in metrics at least (well, it looks quite good visually too).

Like say Koepi "don't trust in magic number" ... llllooollll

update in progress with last codec

test in progress with HDTV resolution ... :)

[IgorC]

About magic numbers. Of course SSIM, PSNR aren´t ideal test, but looking at numeric results of both, you have idea what´s going on.
I´m very interesting on HD test and update Vp7.06, Xvid beta2, snow and LAVC. LAVC seems to be not bad for ASP quality. it´s still buggy and has somewhere artefacts but image is very sharp.

[CyberGuy]

JM 9.5 Reference Encoder

Size: 7,003,927 Bytes (.264)
SSIM: 70.90

I am enoding using some optimizing and with more optimization I doubt that anybody will be able to beat it, that's why I love the AVC/H.264 Reference Encoder. Now I only need to get the distributed encoding working so I can encode in a reasonable time.

[babayaga]

Originally posted by CyberGuy
JM 9.5 Reference Encoder

Size: 7,003,927 Bytes (.264)
SSIM: 70.90

I am enoding using some optimizing and with more optimization I doubt that anybody will be able to beat it, that's why I love the AVC/H.264 Reference Encoder. Now I only need to get the distributed encoding working so I can encode in a reasonable time.

You shouldn't doubt anymore, using a special experimental version of an encoder, one can reach slightly higher SSIM : 7x.93, size = 7 004 847 Bytes (.264). I let you guess which value is x ;)

[Lefungus]

Where are the csv files ?
A mean value is so uninteresting :)

[temporance]

x=0

[babayaga]

Originally posted by temporance
x=0

No, x > 0 :p

[temporance]

x=1

My final offer :)

[Sharktooth]

x = Int((9*Rnd)+1)

[babayaga]

Originally posted by Sharktooth
x = Int((9*Rnd)+1)

You might be right, it evolves almost every day :)

Just a glimpse of our next beta:
450kbps, .MP4 size = 6747kB : SSIM = 71.69 (nearest contender VP7 = 68.84)
900kbps, .MP4 size = 13491kB : SSIM = 81.57 (nearest contender VP7 = 79.33)

455kbps, .264 size = 7 003 809 bytes : SSIM = 72.08. Who said that the JM could not be defeated ? :p

[JohnV]

Originally posted by babayaga
You might be right, it evolves almost every day :)

Just a glimpse of our next beta:
450kbps, .MP4 size = 6747kB : SSIM = 71.69 (nearest contender VP7 = 68.84)
900kbps, .MP4 size = 13491kB : SSIM = 81.57 (nearest contender VP7 = 79.33)

455kbps, .264 size = 7 003 809 bytes : SSIM = 72.08. Who said that the JM could not be defeated ? :p Looks good. :) Can't wait to have this integrated into Nero Recode. :) Looks like VP7 has some catching up to do. ;)

[Sagittaire]

Saperlipopette ... lol


High profil or Main profil ... ???

[Sharktooth]

Originally posted by JohnV
Looks good. :) Can't wait to have this integrated into Nero Recode. :)
:p

[Sagittaire]

if it's true for SSIM, it's perhabs more than 42 dB for OPSNR. In bitrate equivalence it's ~800 Kbps for MPEG4 ASP (with very best codec and very best setting) and more than 75% bitrate for the same quality result ... :eek:

-> it's by far (and really far this time) the most powerfull codec in the world ...

[babayaga]

Originally posted by Sagittaire
if it's true for SSIM, it's perhabs more than 42 dB for OPSNR.

No it's not the case ;) OPSNR and SSIM are sometimes not so closely related. That's why SSIM exists btw.

Regarding perceptual quality, you're right. AVC 450kbps is quite close to the best ASP at ~750kbps
(We've just made a subjective comparison with your xvid 900kbps, some prefer AVC, some other prefer Xvid)


-> it's by far (and really far this time) the most powerfull codec in the world ...
There was a doubt ? ;)

[Sagittaire]

Originally posted by babayaga
No it's not the case ;) OPSNR and SSIM are sometimes not so closely related. That's why SSIM exists btw.


Perhabs you use psy optimisation ... ???


There was a doubt ? ;)


IMO VP7 (high quality) and x264 (high speed) are very serious challengers for NDAVC Main Profil ... lol

I think that I will wait your next beta (if I'am "trusted tester" ... lol) for my HDTV test.

[Ghim]

These results seems extremely promising for the evolution of H.264 based codecs in the future !!

The only problem is that these numbers (OPSNR, SSIM, ...) can't be taken as a proof that a codec is subjectively better than an other one. Even if it's not the point here, It would be really interesting to see how the codecs perform in a proper, standardised, subjective test ^^ in order to really know which ones perform better (I know I'm dreaming because doing subjective tests is too expensive for just comparing codecs ^^).

I also don't deny that OPSNR and SSIM are a good start, but it's hard to measure how much a codec is better compared to another one by using these (OPSNR or SSIM) figures. By example, if you say that the SSIM is 71.86, how does it subjectively compare to a score of 70.50 ? A normalised MOS or DMOS scale could be more appropriate to quantify how much a codec is better compared to another one.

[superdump]

I wouldn't say subjective comparison is expensive if you can find a community willing and able to carry out such tests. I think a number of us would be willing and are able if people were to organise such a test.

[LordRPI]

Didn't everwicked have some subjective video quality measurement applications that could make something like this a lot easier?

[babayaga]

Originally posted by Ghim
These results seems extremely promising for the evolution of H.264 based codecs in the future !!

The only problem is that these numbers (OPSNR, SSIM, ...) can't be taken as a proof that a codec is subjectively better than an other one. Even if it's not the point here, It would be really interesting to see how the codecs perform in a proper, standardised, subjective test ^^ in order to really know which ones perform better (I know I'm dreaming because doing subjective tests is too expensive for just comparing codecs ^^).


PSNR at constant quantizer gives a good idea of the raw efficiency since this is what core encoder try to optimize most of the time.
MPSNR/OPSNR add some hints about rate-control and SSIM (JND metrics or whatever) add clues about adaptive quantization.

A codec which is inferior is those 3 metrics will almost surely be seen as subjectively inferior.

Besides, subjective testing is also very difficult to do and is not always reliable.

[Sagittaire]

Yes, yes and yes ... :D

Originally posted by babayaga
Besides, subjective testing is also very difficult to do and is not always reliable.

for little group of frame it's possible but for high nomber of frame it's impossible in this case to make a global subjectif testing ...

[Ghim]

Originally posted by babayaga
Besides, subjective testing is also very difficult to do and is not always reliable. [/B]

When I meant subjective test, I meant a proper test done in a controlled environement (Subjective Test Facility) and using a standardised method (ACR,DSCQS). The tools created by everwicked follow a standardised method, but reproducing a proper controlled environement is quite impossible to do by yourself.
This kind of test would give very reliable results but as Sagittaire mentioned, the big drawback in the current subjective tests methods is that they're designed for assessing the quality of 8-10 seconds segment. I know some people are working on long sample subjective tests (length in minutes), but there's nothing standard yet and it's in the speech domain.

In order to get good results, you would need to encode videos 2 minutes by example, and once encoded you would cut them in 8-10 seconds chuncks that you would use in the tests. And during the analyse of the results you would need to regroup the scores of the different parts of the video. In order to test the codecs properly, you would need to organise severall subjective tests (one would not be enough).
But as I mentioned earlier, this would be VERY expensive (in terms of money and time) and would need to be redone for each new version ^^.

[Sagittaire]

In order to get good results, you would need to encode videos 2 minutes by example, and once encoded you would cut them in 8-10 seconds chuncks that you would use in the tests.


yes but make this supposition: source 100 sec with 90 sec of low motion and 10 sec of high motion (for last frames). Cut this sample in 10 part for blind test.

I compare the source in CBR for each segment (or VBR buffer 10 sec) and in real unconstrained VBR:
- 9/10 CBR is better than VBR
- 1/10 VBR is very very better than CBR
-> conclusion: CBR is better than VBR ... ???

you must compare codec in quality mode (constant quality / constant quant) and constant size condition for good conclusion but RC tweak is a very important part for codec quality and in practice RC optimisation must be very different between different codec ...

[Ghim]

That's the reason I said you would need more subjective tests...
You wouldn't use just 1 sequence of 2 minutes, it wouldn't be enough.

But you wouldn't just see which codec one is the best on each 8 seconds segment. You would use at least an average of the scores over the segments or an other way to have a good MOS (or DMOS) estimate over the overall sequence.

So, If I follow your example:
- 9/10 CBR is better than VBR (but let's say the difference is very small)
- 1/10 VBR is very very better than CBR (higher difference)

You would then by example make the average of the scores, regardless the number of 8-seconds segments where one performs better than the other. If on average CBR is still higher, I would say that ON THIS SEQUENCE, CBR performs better. That's why you would also need enough sequences covering a larger range of motion.

Let's say you use 8 2-minutes sequences => 96 10-seconds segments with 8 different codecs and 2 settings per codec => 16 conditions per segment
=> It would represent 1536 files. If for one subjective test session, you consider the maximum length of 200 files (with ACR method, that's the maximum you can get), you would need to organise
8 subjective tests. This is, as I mentioned before, a HUGE number for simply comparing codecs.

An other way would be to use more complex subjective models that should give reliable enough results. The major advantage of using this kind of model compared to PSNR and SSIM metrics would be to have results corresponding better to subjective opinion. It would as well give results on a scale where it's easier to estimate the quality difference between codecs.

[Sagittaire]

hmmm IMO very heavy blind test is useless for video test (perhabs for audio codec but not for video codec). Like say Babayaka: "A codec which is inferior is those 3 metrics (APSNR,OPSNR and SSIM) will almost surely be seen as subjectively inferior."

[Latexxx]

Originally posted by Sagittaire
hmmm IMO very heavy blind test is useless for video test (perhabs for audio codec but not for video codec). Like say Babayaka: "A codec which is inferior is those 3 metrics (APSNR,OPSNR and SSIM) will almost surely be seen as subjectively inferior."
But yet guys at EBU do proper blind tests.

[IgorC]

Just small observation

New version of VP7.06 with sames options of tested VP7.04:

SSIM
VP7.06(srtCpuFRee=0) - 69.66
VP7.06 (wihout srtCPUfree on 2GHZ) - 69.30
http://rapidshare.de/files/1861889/2vp443.mkv.html

During playback of VP7 CPU usage is 100%. Trailer Harry Poter 720x304 450 kbit/s

I know 2 ghz CPU isn´t enough powerfull , but it´s critical to play some video of higher resolution + sound track + subtitles even on top CPU.
Even same trailer of beta Ateme High Profile has on my PC less CPU usage (between 50%-65%) and SSIM of High profile is 71.68


x264 rev233(cabac, 3pass, deblocking -1, VRB 75, refer. 16, 2 b-frames others by default) - SSIM 68.55 http://rapidshare.de/files/1862844/3x264.avi.html

[babayaga]

Originally posted by IgorC
Just small observation

New version of VP7.06 with sames options of tested VP7.04:

SSIM
VP7.06(srtCpuFRee=0) - 69.66
VP7.06 (wihout srtCPUfree on 2GHZ) - 69.30
http://rapidshare.de/files/1861889/2vp443.mkv.html

During playback of VP7 CPU usage is 100%. Trailer Harry Poter 720x304 450 kbit/s

I know 2 ghz CPU isn´t enough powerfull , but it´s critical to play some video of higher resolution + sound track + subtitles even on top CPU.
Even same trailer of beta Ateme High Profile has on my PC less CPU usage (between 50%-65%) and SSIM of High profile is 71.68


x264 rev233(cabac, 3pass, deblocking -1, VRB 75, refer. 16, 2 b-frames others by default) - SSIM 68.55 http://rapidshare.de/files/1862844/3x264.avi.html

VP7 also slower on decoding on my Athlon 2800+ : 36% mean load compared to 20% for H.264 files (Main or High profile).
This is for the files you provided and Ateme HP2 450k AVC HP.

[yaz]

may i ask how the quality indices mentioned above were derived ? was it the 'usual avisynthing' way or sg different ? i really wanna know how to perform reliable quality tests on sg like h264
thx
y

[On2Tech]

Originally posted by IgorC
New version of VP7.06 with sames options of tested VP7.04:

SSIM
VP7.06(srtCpuFRee=0) - 69.66
VP7.06 (wihout srtCPUfree on 2GHZ) - 69.30
http://rapidshare.de/files/1861889/2vp443.mkv.html


What were your PSNR numbers on this encode? Obviously your numbers are quite a bit better than the last numbers Sagittaire posted.

Note, I am still a bit wary of pixel weighting funnies in ssim. Perhaps we can get all codec developers to agree to a couple of sensible changes as proposed in this thread:

http://forum.doom9.org/showthread.php?s=&threadid=92532&highlight=ssim


During playback of VP7 CPU usage is 100%. Trailer Harry Poter 720x304 450 kbit/s

I know 2 ghz CPU isn´t enough powerfull , but it´s critical to play some video of higher resolution + sound track + subtitles even on top CPU.
Even same trailer of beta Ateme High Profile has on my PC less CPU usage (between 50%-65%) and SSIM of High profile is 71.68


We've got some good news to report on that front:

Using yet unreleased version 7.07:

On my 1.7 ghz computer, decode of the Harry Potter trailer at 450kb runs video decode only at 170fps. Since trailer is 25 fps and blit to yv12 surface should be relatively free ( given overlay hardware), this puts cpu usage at roughly 16%.

We've also solved a number of issues including fixing a few issues in which non Intel Machines were running a whole lot slower than they should.

Also the next version includes some huge improvements to encode quality at faster speeds. Now at Speed 5 on a 3ghz machine we encode ( + of course decode of vob file etc) at 35fps and get results :

446.129 Kbps Avg PSNR 41.133 Overall PSNR 40.210 SSIM 67.555

Note: this is an improvement of .3 db and 1.9 ssim over the last version.

Also best quality mode has been improved and so have all of the others.

[IgorC]

OPSNR for VP7.06 450 kbit/s

with strCPUfree=0 : 40.7832
without str... : 40.7405

[Sharktooth]

Quite impressive results. It will be interesting to see Ateme AVC-HP vs on2 VP7.0.7.0... :)

[Leo 69]

On my 1.7 ghz computer, decode of the Harry Potter trailer at 450kb runs video decode only at 170fps. Since trailer is 25 fps and blit to yv12 surface should be relatively free ( given overlay hardware), this puts cpu usage at roughly 16%.

@ On2Tech

So, it means I can play 720x304 HQ VP7 video on my girlfriend's 500mhz Celeron??? If it's really true, then I love you. When is the next 7.07 release?

[Sagittaire]

Update

22.05.05
- x264 rev 232
- VP7 7.0.0.6
- LAVC ASP (MEncoder dev-CVS-050426-18:49-3.4.2)
- DivX 3.11 (MEncoder dev-CVS-050426-18:49-3.4.2)
- XviD 1.1.0 Beta2

IMO the most impressive update is DivX3 codec ... DivX 5.2.1 in standart mode vs DivX 3.11 libavcodec could be a good test ... lol

[IgorC]

SSIM 2: Lumimask On (One2Tech Patch)
What patch it was?

[CyberGuy]

JM 9.6 Reference Encoder

Size: 7,003,927 Bytes (.264)
SSIM: 74.76

[IgorC]

Originally posted by CyberGuy
JM 9.6 Reference Encoder

Size: 7,003,927 Bytes (.264)
SSIM: 74.76

video file?

[CyberGuy]

Originally posted by IgorC
video file?
Where is a free place I can post it?

[IgorC]

www.rapidshare.de

Sagitarrie, why not changes the source. Using the same source each time it'll get ironian results.

[IgorC]

Originally posted by CyberGuy
JM 9.6 Reference Encoder

Size: 7,003,927 Bytes (.264)
SSIM: 74.76

File size is a little bigger than other (6,9xx,xxx)

[CyberGuy]

Originally posted by IgorC
video file? http://rapidshare.de/files/1923268/HPII.264.html

Use the Nero codec because this is a High Profile encode using 8x8 transform and also because the Nero codec still has an incorrect value in it’s CABAC initialization table, so I had to put the incorrect value into the JM encoder in order for it to playback using the Nero decoder. I’ve been informed that the new Ateme codec beta has this value fixed.

[Sagittaire]

Originally posted by IgorC
www.rapidshare.de

Sagitarrie, why not changes the source. Using the same source each time it'll get ironian results.

because I must reencode all trailer for each codec (WMV9, RV10, DivX6, NDAVC) and remake all the test.

but I will use another source for the next round in HDTV resolution ... perhabs batman begin 1920*816@1280*544 + light artificial noise ... :D

[Sagittaire]

@ CyberGuy

thank for the trailer but ...


Rule 3 : Bitrate

You must use 2 bitrates for encoding. The first use XviD in quant8 for reference, it's the "Streaming Quality". The second use XviD in quant4 for reference, it's the "1CDR Quality". You can use container of your choice for all Video Elementary Stream.

Streaming: 450 Kbps for Video Elementary Stream and container or 6750 Ko with +/- 0.30 % for tolerance. The size must be in [6730 Ko - 6770 Ko] interval.

1CDR: 900 Kbps for Video Elementary Stream and container or 13500 Ko with +/- 0.30 % for tolerance. The size must be in [13460 Ko - 13540 Ko] interval.


... in fact real size target is very important for "450" sample : metric function are in high variability zone particulary SSIM. dSSIM/dbitrate are more and more high with quant -> SSIM is like a*(1-e^(-bx)) form function ...

for exemple with same setting:
ND AVC, 6811 Ko, SSIM = 69.49
ND AVC, 6886 Ko, SSIM = 69.82

[IgorC]

Originally posted by CyberGuy
http://rapidshare.de/files/1923268/HPII.264.html

Use the Nero codec because this is a High Profile encode using 8x8 transform and also because the Nero codec still has an incorrect value in it’s CABAC initialization table, so I had to put the incorrect value into the JM encoder in order for it to playback using the Nero decoder. I’ve been informed that the new Ateme codec beta has this value fixed.

I coudln´t get SSIM . It has extension .264. Nero decoder plays it but
not decodes it vie directsource for SSIM test. :confused:
Visually is very impressive.

[CyberGuy]

Originally posted by Sagittaire
@ CyberGuy

thank for the trailer but ...



... in fact real size target is very important for "450" sample : metric function are in high variability zone particulary SSIM. dSSIM/dbitrate are more and more high with quant -> SSIM is like a*(1-e^(-bx)) form function ...

for exemple with same setting:
ND AVC, 6811 Ko, SSIM = 69.49
ND AVC, 6886 Ko, SSIM = 69.82
I did this back when you had stated on page one that the rate was 458, ie. 458 kbps = 458000 bps = (458000/8)*(3076/25) bytes = 7044040 bytes = 6879 Kb, and since it takes a lot of tweaking to get the size just right, I've stuck with that. I know this value has changed several times. The new Nero/Ateme beta mentioned a few posts back is over 7,000,000 also and I'm more trying to compete with that in a frendly way anyway since I think it's the only one really close, or better. :)

[CyberGuy]

Originally posted by IgorC
I coudln´t get SSIM . It has extension .264. Nero decoder plays it but
not decodes it vie directsource for SSIM test. :confused:
Visually is very impressive.
mp4creator.exe -create=HPII.264 -rate=25 HPII.MP4

[IgorC]

ok, thanks
Can you encode with the right size to figure out the true result?
And what options did you use? Time of encoding?

[CyberGuy]

Originally posted by IgorC
ok, thanks
Can you encode with the right size to figure out the true result?
And what options did you use? Time of encoding? Yes, but it would take about a week of trying different values and right now I've got too much going on to bother with it. This encode took 12 hours on a 3.2GHz Pentium 4. I'm doing a new encode with some more tweaking, but the encoding will take three weeks on a dual 3GHz Xeon! I was two weeks into it when a Windows Automatic Update rebooted the computer, but I started it again on Friday. Maybe the new values will make the size smaller, you will just have to wait and see.

[IgorC]

SSIM for FREXT.mp4 6823KB 70.90

I used the same .AVS for Nero MP, HP, FREXT

LoadPlugin("C:\ARCHIV~1\GORDIA~1\DGMPGDec\DGDecode.dll")
LoadPlugin("C:\Archivos de programa\GordianKnot\DGMPGDec\ssim.dll")

# --> Video Opening <--
# Trim old 70,3144

source=Mpeg2Source("C:\D\HP\hpjj.d2v",idct=2)
source=Trim(source,70,3145)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)

video=DirectShowSource("C:\D\HP\test\frext.mp4", fps=25)

# --> PSNR analysis <--
return SSIM(source,video,"results2.csv","SSIM-FREXT-450.txt",lumimask=true)

[CyberGuy]

Originally posted by IgorC
SSIM for FREXT.mp4 6823KB 70.90

I used the same .AVS for Nero MP, HP, FREXT

LoadPlugin("C:\ARCHIV~1\GORDIA~1\DGMPGDec\DGDecode.dll")
LoadPlugin("C:\Archivos de programa\GordianKnot\DGMPGDec\ssim.dll")

# --> Video Opening <--
# Trim old 70,3144

source=Mpeg2Source("C:\D\HP\hpjj.d2v",idct=2)
source=Trim(source,70,3145)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)

video=DirectShowSource("C:\D\HP\test\frext.mp4", fps=25)

# --> PSNR analysis <--
return SSIM(source,video,"results2.csv","SSIM-FREXT-450.txt",lumimask=true) That's correct for SIMM 1, SIMM 0 = 74.76.

[CyberGuy]

Here is an earlier one that I had saved, but with lower quality encoder settings. This should be within the target size, 6768.

http://rapidshare.de/files/1924176/HPII.MP4.html

Set: source=Trim(source,70,3144)

Size: 6,930,379 Bytes (.MP4)
SSIM 0: 74.45

[IgorC]

For SSIM0 (lumimask = false) for VP7.06 (MKV container) = 74.01 size 6.932.257 bytes
I´d steak to SSIM1 or move to source without black frames ;)

[CyberGuy]

Originally posted by IgorC
For SSIM0 (lumimask = false) for VP7.06 (MKV container) = 74.01 size 6.932.257 bytes
I´d steak to SSIM1 or move to source without black frames ;)
Last time I checked, 6,930,379 was smaller than 6,932,257 and 74.45 was still better then 74.01.

[babayaga]

Originally posted by babayaga

Just a glimpse of our next beta:

455kbps, .264 size = 7 003 809 bytes : SSIM = 72.08

On this special stream, SSIM0 is 76.53 :p

Is there a way to have the dll for SSIM2 ?

[On2Tech]

Originally posted by babayaga
On this special stream, SSIM0 is 76.53 :p

Is there a way to have the dll for SSIM2 ?
Sure:

anonymous ftp to ftp.on2.com

Grab the dll ( ssim.dll)
The source code is SSIMSrc-0.24.zip.

Basically exactly the same things were done as were discussed in the thread below:

http://forum.doom9.org/showthread.php?s=&threadid=92532&highlight=ssim

We did not try to adjust the U and V plane oddities discussed in that thread.

These still exist, also note that the lumimask option partially corrects for the issue described with magnitudes in u and v.

In our opinion the brightness component of this metric should be corrected so that an error in black should not count > 33 times as much as an equivalent error in white ( or pink vs green etc in uv).

Thoughts?

[bobololo]

Originally posted by babayaga
On this special stream, SSIM0 is 76.53 :p

The different clips produced by our work-in-progress encoder are now available here:

hp2-ateme-450k-hp.mp4 (ftp://mood.ateme.net/beta/hp2-ateme-450k-hp.mp4)
hp2-ateme-455k-hp.mp4 (ftp://mood.ateme.net/beta/hp2-ateme-455k-hp.mp4) (to match with CyberGuy results)
hp2-ateme-900k-hp.mp4 (ftp://mood.ateme.net/beta/hp2-ateme-900k-hp.mp4)

You can now measure any metrics by yourself :)

[IgorC]

SSIM1 for Ateme High Profile 900kbit/s 81.57
Many results but there is no still codec. Like song :"Look at us but do not touch" :)
CyberGuy, can you say if there is date of realse of JM 9.6? Untill now we know it's bloody slow but what result will be obtain after optimisation (speed/quality)

[CyberGuy]

Originally posted by bobololo
The different clips produced by our work-in-progress encoder are now available here:

hp2-ateme-450k-hp.mp4 (ftp://mood.ateme.net/beta/hp2-ateme-450k-hp.mp4)
hp2-ateme-455k-hp.mp4 (ftp://mood.ateme.net/beta/hp2-ateme-455k-hp.mp4) (to match with CyberGuy results)
hp2-ateme-900k-hp.mp4 (ftp://mood.ateme.net/beta/hp2-ateme-900k-hp.mp4)

You can now measure any metrics by yourself :)
I received a SSIM 0 value of 76.34779441 and a SSIM 1 value of 71.84 for hp2-ateme-455k-hp.mp4. These do not match the numbers that you provided, not that it makes much difference. :scared: I was just wandering if you are using a beta DirectShow codec and that's why the numbers are different. I'm using the latest codec's from http://www.nero.com. It may be futile, but I'm still trying to beat those numbers. :)

[CyberGuy]

Originally posted by IgorC
SSIM1 for Ateme High Profile 900kbit/s 81.57
Many results but there is no still codec. Like song :"Look at us but do not touch" :)
CyberGuy, can you say if there is date of realse of JM 9.6? Untill now we know it's bloody slow but what result will be obtain after optimisation (speed/quality)
It started on 05/19/2005 at 10:47 p.m. and is on frame 855, so I'll let you do the math. I'm not sure if it will have a higher SSIM value or not, it's just some settings that may help some.

[babayaga]

Originally posted by CyberGuy
I received a SSIM 0 value of 76.34779441 and a SSIM 1 value of 71.84 for hp2-ateme-455k-hp.mp4. These do not match the numbers that you provided, not that it makes much difference. :scared: I was just wandering if you are using a beta DirectShow codec and that's why the numbers are different. I'm using the latest codec's from http://www.nero.com. It may be futile, but I'm still trying to beat those numbers. :)
Such a difference might be cause by a colorspace conversion for instance (DirectShowSource is very difficult to control :mad:
It's odd you found a different result since Sagittaire measured the same Overall PSNR :confused:
But you're right, we use a whole set of internal tools for the tests, including a decoder.

[CyberGuy]

Originally posted by babayaga
Such a difference might be cause by a colorspace conversion for instance (DirectShowSource is very difficult to control :mad:
It's odd you found a different result since Sagittaire measured the same Overall PSNR :confused:
But you're right, we use a whole set of internal tools for the tests, including a decoder.
Sorry, I found the problem. I was using a SSIM AVS script that I had been testing that had Taps=8 in it. Now I'm on the same page and the values match. BTW, this changed the JM encode SSIM 0 value to 74.93.

[yaz]

Originally posted by On2Tech
anonymous ftp to ftp.on2.com
Grab the dll ( ssim.dll)
The source code is SSIMSrc-0.24.zip.
erhm ... would someone make it available for the mass. i can't get it this way :(
thx
y

[Sharktooth]

well... it's an ANONYMOUS FTP so it's available to the "mass".
I just got it.

[yaz]

Originally posted by Sharktooth
well... it's an ANONYMOUS FTP so it's available to the "mass".
I just got it. ahh .. it's good to know really (i'll sleep calm tonight ;) ) ... so let's reduce my msg :
i can't get it this way. would someone make it available for me, pls.
thx
y (the 'mass')

[Manao]

http://manao4.free.fr/SSIM.dll
http://manao4.free.fr/SSIMSrc-0.24.zip

[yaz]

@manao
thx a lot !
y

[Sagittaire]

New update

24.05.05
- SSIM plugin
- Nero Digital High Profil for demo only

[Lefungus]

If the SSIM plugin has been modified, why is it still named 0.24 ? Version should be properly incremented or we'll have multiple "0.24" versions floating around. We'll only get more confused people, riots and eventually the end of the world as we know it.

[IgorC]

Something is wrong with Xvid-450 beta2 sample. It has date 9th of march. Xvid Beta2 was realsed on 4th of april.

Test results :
Xvid OPSNR : 39.66 SSIM : 64.36 Size : 6756K

I get here slightly better result http://rapidshare.de/files/1980087/2Xvid.mkv.html
Xvid(beta2) OPSNR : 39:67 SSIM : 64.45 Size : 6754K
All conditions of test(iDCT 32 bits MMX, fast recopress, etc.) were observed.

[Sagittaire]

Originally posted by IgorC
Something is wrong with Xvid-450 beta2 sample. It has date 9th of march. Xvid Beta2 was realsed on 4th of april.

Test results :
Xvid OPSNR : 39.66 SSIM : 64.36 Size : 6756K

I get here slightly better result http://rapidshare.de/files/1980087/2Xvid.mkv.html
Xvid(beta2) OPSNR : 39:67 SSIM : 64.45 Size : 6754K
All conditions of test(iDCT 32 bits MMX, fast recopress, etc.) were observed.

Update in progress ... but IMO XviD and DivX are equivalent for metric.

It's possible to make slightly better for OPSNR or SSIM with post-process refinement in ffdshow (threshold and strenght)

[IgorC]

But in this case decoder was XVID

[subliminal]

hummm...

[Sagittaire]

Update

08.06.05
- x264 rev 253 Main Profil
- x264 rev 253 High Profil

[Sharktooth]

253 is b0rked. use 254.

[Manao]

The psnr / bitrate figures won't be changed between 253 / 254, so it's no big deal.

Edit : thinking about it, it's true that r253 won't be playable by anything else than a b0rked version of ffdshow / ffmpeg. My bad.

[dinolib2]

May be I've missed something... but yesterday night Nero AVC MP SSIM0 was 73.99. right?

[Sagittaire]

yes it's true ... update with 73.99 ... ;)

i will try to find setting refinement for better result with x264 HP than NDAVC MP ... :devil:

[IgorC]

Early I´ve already said that deblock -1 , 2 bframes are better for this source. It´s not first time I'm saying it. And answer is as always 'Test in progress' I got here with x264 HP better result than Nero H.264 MP.

[Sagittaire]

Early I´ve already said that deblock -1 , 2 bframes are better for this source. It´s not first time I'm saying it. And answer is as always 'Test in progress' I got here with x264 HP better result than Nero H.264 MP.

perhabs with old build but I think not with new build ... :D

send your result ... :thanks:

[dinolib2]

I've made my try :p
Only one difference: I've use ffdshow decoder.

Cmd line:
x264.exe --bframe 2 --merange 16 --ref 15 --bitrate 447 --pass 1 --8x8dct --filter -1:-1 --qcomp 0.75 --ipratio 1.10 --pbratio 1.40 --analyse "all" --weightb --me "umh" --subme 5

3 pass

Result:
SSIM: Structural Similarity Index Metric 0.23
Average SSIM= 74.01
Size 6750 KB

Just a little better :)

I haven't enought time to test different filters and curve compression.
Tried different bframes: 4 is worse, 8 is the same (sincerely I don't know why)

Same settings for 896 kbps give me 82.72 and 13523 KB

[zombi55]

@Sagittaire

Can we Please have your exactly command line for the x264 HP ?

And you do how many fps on average without going into the details ?

Ty

[On2Tech]

A new version of VP7 is up.

More details and a link are given in the VP7 thread but the basic best quality metrics with default settings are as follows.

Opsnr SSIM0 SSIM1 SSIM2

446K 40.837 74.33 70.22 70.195
896K 43.344 82.69 80.108 80.104

Note that the defaults have changed a little for this release to reflect other internal changes to the RC.

Most of the effort in this release has gone into the higher speed settings (for example good quality speed 5 SSIM has improved by almost 2 on the HP clip at 446k) and improving the decode speed. The latter will not directly effect metric results but will improve "real world" quality for many people with lower spec machines.

On2Tech

[JnZ]

The purpose of this challenge is to determine which is the best codec for the metrics
And what about of encoding time? Just for real imagination.

[On2Tech]

And what about of encoding time? Just for real imagination.


Encoding speed is similar to 7.0.6. Details of encode speeds for the different speed settings are given at the top of the VP7 thread.

[Sharktooth]

well... the improvment is neat. congratulations for the new release :)

[JnZ]

Encoding speed is similar to 7.0.6. Details of encode speeds for the different speed settings are given at the top of the VP7 thread.
I know about slow speed of VP7, tested. :D. (peace)

This question was pointed to Sagittaire. I mean globally speed of all codecs for imagination.

I'm meaning: In history was many compression algorithms,that was excelent compression ratios,but very slow. And where they are now? Disappeared from minds very quickly...

So no only quality is important, but good compromise of quality vs speed.

[Sharktooth]

Why complaining about speed? Use Good Quality mode that is a LOT faster and results are very close to Best Quality.
Think about Best Quality like an "Insane" mode to use ONLY if encoding time is not a big problem.

[JnZ]

Why complaining about speed? Use Good Quality mode that is a LOT faster and results are very close to Best Quality.
Think about Best Quality like an "Insane" mode to use ONLY if encoding time is not a big problem.
I know. But, anyway, I think, that encoding speed could be introducted in Sagittaire tab.
Eg: AVC codecs are very close,but we can't say that some is best, because it was best SSIM. We must take in consideration encoding speed.

[Sharktooth]

Agreed.

[On2Tech]

I know. But, anyway, I think, that encoding speed could be introducted in Sagittaire tab.
Eg: AVC codecs are very close,but we can't say that some is best, because it was best SSIM. We must take in consideration encoding speed.


I think your point that speed is also an important real world consideration for many users is well made (sorry if I misunderstood your previous post). It is just that speed was not one of the criteria for this test.

In fact, the "best" quality mode in VP7 does not represent the absolute best that could be acheived assuming speed did not matter at all. For example, if we examined all options and all possible motion vectors for every block, we would no doubt get significantly better results... eventually :D

I think you are wrong, however, in your assumption that some of the SUPER slow methods of the past died purely because of encode speed. For example, I worked for a time some years back on a project using "fractals". The codec did well to manage a frame a minute at 320x240, but even at that speed it did not come close to the quality of current generation codecs.

On2Tech

[Sagittaire]

Update 17.06.05
- x264 rev 263 Main Profil
- x264 rev 263 High Profil
- DivX6
- VP7 7.0.0.7
- MPEG2 MP@ML with Mencoder

Last Update for this trailer, Next round in HDTV resolution ... :)

[berrinam]

According to Google cache (http://64.233.179.104/search?q=cache:xPI3MC7sWXgJ:forum.doom9.org/showpost.php%3Fp%3D618225%26postcount%3D1+sagittaire+movie+metric+benchmark+challenge&hl=en), all of the metric values have gone down since the last update, except for x264. What's up?

[Sagittaire]

According to Google cache (http://64.233.179.104/search?q=cache:xPI3MC7sWXgJ:forum.doom9.org/showpost.php%3Fp%3D618225%26postcount%3D1+sagittaire+movie+metric+benchmark+challenge&hl=en), all of the metric values have gone down since the last update, except for x264. What's up?

better metric for x264 HP & MP, DivX6 and VP7
better SSIM for ateme MP and lower OPSNR with new psy mode

I not use the highest quality mode with Ateme HP & MP because it's very too slow (5 fps for extra vs 0.5 fps for full quality mode)
SSIM2 is delete because SSIM2 ~ SSIM3

[On2Tech]

better metric for x264 HP & MP, DivX6 and VP7
better SSIM for ateme MP and lower OPSNR with new psy mode

I not use the highest quality mode with Ateme HP & MP because it's very too slow (5 fps for extra vs 0.5 fps for full quality mode)
SSIM2 is delete because SSIM2 ~ SSIM3

Your numbers are way down on the numbers we get using all defaults for the new VP7. Note: some of our defaults have changed. We think using the new defaults will give you better results.

[Sagittaire]

19.06.05
- Ateme codec Full quality
- VP7 7.0.0.7 better RC

At this time for this trailer

MPEG4 AVC codecs: NDAVC vs x264
Ateme H264 and x264 are comparable codec for OPSNR but not for SSIM (visually NDAVC is IMO better than x264 too). x264 is the best Open Source Codec and XviD can't fight with the last version.

MPEG4 ASP codecs: XviD vs DivX vs LAVC
DivX6 is the best MPEG4 ASP codec for OPSNR and SSIM but XviD, DivX6 and Libavcodec ASP are very close codec.

Proprietary "free" codecs: RV10 vs VP6 vs WMV9
VP6 is always better than RV10 or WMV9 for OPSNR or SSIM. VP6 are better than all MPEG4 ASP codec for OPSNR and SSIM. RV10 is an excellent codec for high quant encoding (low quality or "low bitrate").

Higest performance codecs: VP7 vs NDAVC vs x264
VP7, Ateme AVC or x264 are really better than the others codecs. VP7 seem better than Ateme H264 Main Profil for very high quant (low quality or "low bitrate") and Ateme H264 Main Profil seem better than VP7 for lower quant (high quality or "high bitrate"). x264 High Profil is the best codec actually available. Ateme H264 is by far the best codec in this test ...

MPEG2 vs all
The best codecs (H264 or VP7) are 50% better than MPEG2 MP@ML but MPEG2 MP@ML with good encoder is always a good codec. IMO Mencoder is by far the best MPEG2 encoder for low bitrate ...

[IgorC]

MPEG4 AVC codecs: NDAVC vs x264
Ateme H264 and x264 are comparable codec for OPSNR but not for SSIM (visually NDAVC is IMO better than x264 too).


The same I can say about Divx6 and Xvid 1.1 beta 2 . Divx6 and Xvid have the same SSIM (Divx6 13493 KB - SSIM 77.99. Xvid 13489 KB - SSIM 77.96) but visually Imo Xvid is far better.
What about to use VQM test for HDTV test? It seems to be realistic test as SSIM.

[Rash]

Just a question: Atheme H264 = NeroDigital ?

[bond]

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics
to be more accurate and not claiming to have explained the whole codec world via one clip i would write it that way:

"The purpose of this challenge is to determine which is the best codec for the Harry Potter II trailer for the metrics and only for the metrics"

after all thats the truth, no?


MPEG4 ASP : XviD
Coder: Koepi XviD 1.1.0 beta2
Setting: 2 pass, Unconstrained, Motion Ultra-High, VHQ4, BVHQ, Chroma Motion, Trellis, Adaptative Quantisation, GMC, BFrame 2/1.50/1.00
decoder: XviD decoder with PP4
files: XviD 450 Kbps (http://multimediacom.free.fr/Video/XviD-450.mkv) & XviD 900 Kbps (http://multimediacom.free.fr/Video/XviD-900.mkv)definitely use qpel too, it helps preserving details a lot normally
what quant matrix did you use?

MPEG4 AVC : x264 Main Profil
Coder: x264 VFW rev 264
Setting: 3 pass, variabilty 75%, Quality Max, 16 Ref frames, 2 Bframes, CABAC, WPred, Inloop deblock strength -1, RDO
decoder: ffdshow H264 decoder
files: x264 Main Profil 450 Kbps (http://multimediacom.free.fr/Video/x264MP-450.mp4) & x264 Main Profil 900 Kbps (http://multimediacom.free.fr/Video/x264MP-900.mp4)

MPEG4 AVC : x264 High Profil
Coder: x264 VFW rev 264
Setting: 3 pass, variabilty 75%, Quality Max, 16 Ref frames, 2 Bframes, CABAC, WPred, Inloop deblock strength -1 , 8x8dct, RDO
decoder: ffdshow H264 decoder
files: x264 High Profil 450 Kbps (http://multimediacom.free.fr/Video/x264HP-450.mp4) & x264 High Profil 900 Kbps (http://multimediacom.free.fr/Video/x264HP-900.mp4)what is "quality max"?
why not use b-pyramid and 3 b-frames? x264 should be smart enough to use the correct amount of b-frames anyways
which partitions did you use? all?
i dont think its necessary to use 16 reference frames

[Manao]

bond : qpel and custom quant matrices lowers the PSNR in comparison to no qpel and h263.

[bond]

bond : qpel and custom quant matrices lowers the PSNR in comparison to no qpel and h263.
what about ssim?

[stephanV]

definitely use qpel too, it helps preserving details a lot normally

If you can give us settings that do better on PSNR and SSIM (after all thats whats being measured here), then give them. Just suggesting to use QPEL is a bit pointless as I'm pretty sure some people would have thought of (and tried) that by now.


why not use b-pyramid and 3 b-frames? x264 should be smart enough to use the correct amount of b-frames anyways
which partitions did you use? all?
i dont think its necessary to use 16 reference frames
Again the answer is: better metrics. If your settings do better with the suggested metrics, post them! Source material is available after all.

The whole point about this test was to prevent everyone posting his favorite custom settings along with a screaming contest and to provide a means to evaluate the quality (don't say quality is subjective, that is not entirely true, it are the terms we use to describe quality that are subjective) of the encode that is not discussable.

Of course, if you think a metric comparison doesn't have any value to you, you shouldn't pay too much attention to this thread.

[bond]

If you can give us settings that do better on PSNR and SSIM (after all thats whats being measured here), then give them.stephanV, you seem to think quality comparisons in a wrong way

a quality comparison is always about visual quality. it cant make sense to use settings aiming at maximizing the metrices outcome even if the visual quality gets hurt (if thats what this thread is about)...

now if the metrices are not able to reflect visual quality correctly you
- either have to use better metrices or
- your metrices values are useless as not accurately reflecting the reality


i indeed start paying attention to this thread when the outcome values of this comparison get posted in a divx6 thread without any further explanation, except "this codec is best", or link to this thread and all the newbies take this as the ultimate measure for codec's quality

[stephanV]

stephanV, you seem to think quality comparisons in a wrong way.

a quality comparison is always about visual quality. it cant make sense to use settings aiming at maximizing the metrices outcome even if the visual quality gets hurt (if thats what this thread is about)...
The whole point is that no one has ever bothered to make a good definition of "visual quality", or at least I have missed it. Then throw in some loosely defined terms as "crisp", "blurry" and "details" and now you know exactly why all those "subjective visual quality" comparisons based on screenshots of all things have no meaning to anyone other than to the person who did the comparison. If you want to do that, download the clips:

A - Introduction

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...

Please don't try to tell me I don't know what a comparison should be all about. This is actually one of the few comparisons I've seen that defines closely what its goal is and therefor it's probably more usefull than all random screenshot comparisons put together.

i indeed start paying attention to this thread when the outcome values of this comparison get posted in a divx6 thread without any further explanation, except "this codec is best",

That's a bit of a misquote: http://forum.doom9.org/showthread.php?t=95862&page=7&pp=20
For metric DivX6 is the best ... MPEG4 ASP codec. Good work DXN ...
And yes, maybe it would have been better if a link to this thread was posted there, but then you just should have said that instead of making suggestions for better metrics.

Again, if you don't like metrics just ignore this thread, or invalidate the test by using the same metrics and a different clip. Shooting things down from the side-line and throwing everything on "subjectiveness" is a bit too easy, no?

[bond]

ok if the only execrise discussed here is to maximize psnr and ssim on the harry potter clip no matter what effect this has on quality, than i have to say you are right i shouldnt have thrown in qpel and b-pyramid as suggestion without providing psnr/ssim values for that too

still noone discussed qpel and b-pyramid till now in this thread so i think it was a valid point to point at

The whole point is that no one has ever bothered to make a good definition of "visual quality", or at least I have missed it. Then throw in some loosely defined terms as "crisp", "blurry" and "details" and now you know exactly why all those "subjective visual quality" comparisons based on screenshots of all things have no meaning to anyone other than to the person who did the comparison. If you want to do that, download the clips:

Please don't try to tell me I don't know what a comparison should be all about. This is actually one of the few comparisons I've seen that defines closely what its goal is and therefor it's probably more usefull than all random screenshot comparisons put together.of course from a scientific point of view its easier to define and calculate some values than to develop and work with subjective measures
still this doesnt mean automatically that the mathematical method is any better, just because its more clearly defined, especially if the method is known already to be weak at accurately reflecting the reality (which is its only point after all)

i hoped ssim is better than psnr in that regard, but i am still waiting for manao's reply ;)

That's a bit of a misquote: http://forum.doom9.org/showthread.php?t=95862&page=7&pp=20
well he claimed to have found the best codec for metrices without mentioning any limitations.
obviously his values are only valid for the harry potter trailer, dont you agree?

so his claim is simply false
and especially misleading as he didnt link to that thread nor explained the method he used to come to that values

[berrinam]

@stephanV: You have got your link the wrong way around -- the URL is displayed and is linked to http://here.

[stephanV]

@berrinam: fixed. thank you.

of course from a scientific point of view its easier to define and calculate some values than to develop and work with subjective measures
still this doesn't mean automatically that the mathematical method is any better, just because its more clearly defined
It is exactly for that reason it is automatically better. Until you define things, discussion is pointless (Socrates already acknowledged that!). Now things are defined we can discuss what is better according to metric. And since the metric is closely defined too we can even discuss the merits of the metric itself and therefor the accuracy of this test. The participation in this thread/test from people of On2Tech, Ateme and DivX, is worthwhile mentioning I suppose.

On the other hand, discussing video quality only guided by "IMO" is pretty much worthless. Quite frankly, if you admit quality is subjective you should also admit that your opinion on it does not matter in the larger scheme of things (everybody else's opinion is just as valid, even if its the opposite of yours) and therefor that publishing subjective comparisons is pointless (unless you are a narcist maybe).

well he claimed to have found the best codec for metrices without mentioning any limitations.
obviously his values are only valid for the harry potter trailer, dont you agree?

so his claim is simply false
I wouldn't go that far. It is not unusual in science to allow for a certain extra-polation of results, and seeing that PSNR metrics play a large role in developing codecs it is not completely unrealistic to suggest that on another clip similar results would be obtained. I cannot judge if the differences in PSNR and SSIM between DivX6, XviD and LAVC are large enough to conclusively say that it's very likely DivX will do better metric-wise in most cases; I have no idea how big a difference of 0.1 dB really is.

and especially misleading as he didnt link to that thread nor explained the method he used to come to that values
Again, then you should have said something about that right away. When you started off it looked like you were angry that XviD didn't win the test and that DivX (of all codecs!) was called a better one. Just realize that this test does not try to invalidate your opinion.

[Sagittaire]

@ Bond

All the reponses are in the preceding posts ... there are very interessing post for exemple:

1) Trailer are not particullar source: all the frame in trailers are the same frame than the original movie. IMO make test with trailer is more interessing than only one sample because type scenes are more various. Trailers are not very compressible but if you make test with quant reference for bitrate (q4 for "1CDR" and q8 for "Streaming" here) it's not a probleme.

2) I use the best possible setting for all codec for this trailer (qpel, GMC or CM decrease always metric for MPEG4 ASP, bpyramid don't help here for x264 ...) with a reasonable speed (I not use "esa" for me with x264) with this trailer ...

3) IMO blind test or visual test are unable to make overall compararison. Metric are very better for that:

PSNR at constant quantizer gives a good idea of the raw efficiency since this is what core encoder try to optimize most of the time.
MPSNR/OPSNR add some hints about rate-control and SSIM (JND metrics or whatever) add clues about adaptive quantization.

A codec which is inferior is those 3 metrics will almost surely be seen as subjectively inferior.

Besides, subjective testing is also very difficult to do and is not always reliable.



Technical Information

Originally posted by huang_ch
I'm really confused about that why RV10 get the WORST score even in 450Kbps. My original knowledge is that RV10 is really good at low bps, while XVID/DIVX/WMV codecs are more focused at higher bps and don't play well in low bps compare to RV9/10, but in this test I saw XVID/DIVX/WMA are better than RV10, can anyone tell me why?

1) My eyes seem say that too. And for the first time I don't understand why. Perhabs that it's true : metric are unable to test video quality ... !!?


|---------------|-----------|------------|----------|---------|---------|
| Codec | P-Process | ES Bitrate | Size | OPSNR | SSIM |
|---------------|-----------|------------|----------|---------|---------|
| DivX | PP4 | 447 kbps | 6757 Ko | 39.5729 | 63.49 |
| XviD | PP4 | 447 kbps | 6760 Ko | 39.6601 | 64.36 |
| RV10 old | HF1 | 447 kbps | 6767 Ko | 39.4594 | 62.83 |
|---------------|-----------|------------|----------|---------|---------|
| DivX | PP4 | 896 kbps | 13490 Ko | 42.6220 | 77.47 |
| XviD | PP4 | 896 kbps | 13498 Ko | 42.6654 | 77.95 |
| RV10 old | HF1 | 896 kbps | 13501 Ko | 42.5815 | 77.21 |
|---------------|-----------|------------|----------|---------|---------|


2) I make Graph PSNR test to compare with other codec and I see that with default RC setting for RV10:

With this trailer and defaut RC setting the last frames of the trailer are dramaticaly worst than all other codec. It's typicaly a Rate Control Problem. In fact RV10 RC use to high size for the begin and too low size for the end. Visually and for metric these last frames MPEG4 ASP and all the other codec are very better than RV10.


3) I remade "450" encoding with new and really optimized metric RC tweak
- Size quant prediction: rcPFrameRefQuant = 17 for better prediction
- Bframe ratio : rcBFrameRefQuant = 23 for better metric result
- Rate Contro : rcLowBitrateBoost = rcHighBitrateReduce = 20 for better metric result

I remade too "450" with old RC and it's too very better than Elysian RC with defaut RC setting for this trailer. The new encoding are very better for the complete trailer and very better for the last frames.


|---------------|-----------|------------|----------|---------|---------|
| Codec | P-Process | ES Bitrate | Size | OPSNR | SSIM |
|---------------|-----------|------------|----------|---------|---------|
| DivX | PP4 | 446 kbps | 6755 Ko | 39.7552 | 64.44 |
| XviD | PP4 | 446 kbps | 6760 Ko | 39.6601 | 64.36 |
| RV10 new | HF1 | 446 kbps | 6743 Ko | 40.0457 | 65.52 |
|---------------|-----------|------------|----------|---------|---------|
| DivX | PP4 | 896 kbps | 13508 Ko | 42.7455 | 77.93 |
| XviD | PP4 | 896 kbps | 13498 Ko | 42.6654 | 77.95 |
| RV10 new | HF1 | 896 kbps | 13493 Ko | 42.7431 | 77.73 |
|---------------|-----------|------------|----------|---------|---------|


4)
Originally posted by Shinobu
for my eyes rv10 looks also better than any others at low bitrates (may be not nero avc in non-anime case), but eyes are not metric ...


Yes, your (and my) eyes are not metric : your (and my) eyes were unable to detect this Rate Control artefact and very visible problem for the 250 last frames perhabs simply because the clip was not entirely views ... perhabs because for eyes the first scene is more important than the last ... perhabs because it's very difficult for eyes to compare 3075 frames and make objective overall observation ...

Conclusion: Make yourself the conclusion ... !!!

[Sagittaire]

I'am not MPEG2 fan (with CCE or other encoder) thus if someone is able to make better result for MPEG2 MP@ML then he will be welcome ...

[bond]

ok before we continue this surely endlessly discussable discussion, which only shows that different people have different subjective opinions, valuing different things more than others, lemme make only one thing clear:

dont claim to have found the best codec (for metrices, one or all videos existing or whatever)!

there are so many things influencing a codec's performance (bitrates, settings, sources, moon phases...) that i dont think its possible to answer the question "what is the best codec for..."


therefore let me point out the open issues of my initial post (cause after all i posted because i wanted to say and know these things):

1)
The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics to be more accurate and not claiming to have explained the whole codec world via one clip i would write it that way:

"The purpose of this challenge is to determine which is the best codec for the Harry Potter II trailer for the metrics and only for the metrics"

after all thats the truth, no?

2)
MPEG4 ASP : XviD
Coder: Koepi XviD 1.1.0 beta2
Setting: 2 pass, Unconstrained, Motion Ultra-High, VHQ4, BVHQ, Chroma Motion, Trellis, Adaptative Quantisation, GMC, BFrame 2/1.50/1.00
decoder: XviD decoder with PP4
files: XviD 450 Kbps (http://multimediacom.free.fr/Video/XviD-450.mkv) & XviD 900 Kbps (http://multimediacom.free.fr/Video/XviD-900.mkv)what quant matrix did you use?

MPEG4 AVC : x264 Main Profil
Coder: x264 VFW rev 264
Setting: 3 pass, variabilty 75%, Quality Max, 16 Ref frames, 2 Bframes, CABAC, WPred, Inloop deblock strength -1, RDO
decoder: ffdshow H264 decoder
files: x264 Main Profil 450 Kbps (http://multimediacom.free.fr/Video/x264MP-450.mp4) & x264 Main Profil 900 Kbps (http://multimediacom.free.fr/Video/x264MP-900.mp4)

MPEG4 AVC : x264 High Profil
Coder: x264 VFW rev 264
Setting: 3 pass, variabilty 75%, Quality Max, 16 Ref frames, 2 Bframes, CABAC, WPred, Inloop deblock strength -1 , 8x8dct, RDO
decoder: ffdshow H264 decoder
files: x264 High Profil 450 Kbps (http://multimediacom.free.fr/Video/x264HP-450.mp4) & x264 High Profil 900 Kbps (http://multimediacom.free.fr/Video/x264HP-900.mp4)what is "quality max"?
which partitions did you use? all?


@ socrates
he acknowledged that people are seeing the world with their own eyes and also interpret the world they live in subjectively. the next step when realising this is to realise that everyone creates the world on his/her own. therefore the next step is to realise that nothing you think is true also has to be objectively true and therefore you have to admit that "you know nothing" (i hope i wrote it in an understandable way, cause technical people, who think that there is the ultimate truth out there, often dont understand that at all)

therefore i think you can take socrates as one who would talk out for subjective measures for judging video quality ("use your own eyes") instead of having some mathematicians develop indicators for measuring quality which claim to be able to objectively reflecting the truth, which is not the case

a wrong indicator is not better than the no indicator or even more important, the only indicator that counts for you, your eyes
but well its your subjective decision to see it the other way round ;)

[Sagittaire]

@ Bond

1) yes read the first post ...


D - Conclusion

At this time for this trailer ...


and perhabs too:
- for this bitrate
for example VP7 is very very powerfull for very high quant
for exemple it seem that MPEG2 is better than MPEG4 for very low quant
- for this resolution
conclusion for 720*304 is parhabs a good conclusion for DVD like resolution (720*576 or 720*480) but perhabs not for VCD or HDTV resolution ...
- for this pre-process
conclusion in YUV perhabs not the same with only Y. And with denoiser? And with noisy source ... ect ect ect

2) Metric graph is very good too: if you use similar RC setting and if for metric each frame of codec A is better than codec B then conclusion must be can be generalized . For exemple HPII trailer has more 3000 frame: if x264 HP is better than MPEG4 ASP for each frame then there will be strong chances that that is also the case for other similar source (same noise, same resolution, same average quant).

3) for all MPEG4 ASP h263 quantification + PP4 is the best setting by far for metric. IMO CM is good choice only with PP0 ...

I use this CLI for x264 test:
x264.exe --bframe 2 --ref 16 --filter -1:-1 --bitrate 447 --pass 1 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.40 --analyse "all" --weightb --me "umh" --subme 6 --progress -o NUL Encodage.avs
x264.exe --bframe 2 --ref 16 --filter -1:-1 --bitrate 447 --pass 3 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.40 --analyse "all" --weightb --me "umh" --subme 6 --progress -o NUL Encodage.avs
x264.exe --bframe 2 --ref 16 --filter -1:-1 --bitrate 447 --pass 2 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.40 --analyse "all" --weightb --me "umh" --subme 6 --progress -o x264MP-447.mp4 Encodage.avs

x264.exe --bframe 2 --ref 16 --filter -1:-1 --bitrate 896 --pass 1 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.40 --analyse "all" --weightb --me "umh" --subme 6 --progress -o NUL Encodage.avs
x264.exe --bframe 2 --ref 16 --filter -1:-1 --bitrate 896 --pass 3 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.40 --analyse "all" --weightb --me "umh" --subme 6 --progress -o NUL Encodage.avs
x264.exe --bframe 2 --ref 16 --filter -1:-1 --bitrate 896 --pass 2 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.40 --analyse "all" --weightb --me "umh" --subme 6 --progress -o x264MP-896.mp4 Encodage.avs

x264.exe --bframe 2 --8x8dct --ref 16 --filter -1:-1 --bitrate 447 --pass 1 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.40 --analyse "all" --weightb --me "umh" --subme 6 --progress -o NUL Encodage.avs
x264.exe --bframe 2 --8x8dct --ref 16 --filter -1:-1 --bitrate 447 --pass 3 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.40 --analyse "all" --weightb --me "umh" --subme 6 --progress -o NUL Encodage.avs
x264.exe --bframe 2 --8x8dct --ref 16 --filter -1:-1 --bitrate 447 --pass 2 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.40 --analyse "all" --weightb --me "umh" --subme 6 --progress -o x264HP-447.mp4 Encodage.avs

x264.exe --bframe 2 --8x8dct --ref 16 --filter -1:-1 --bitrate 896 --pass 1 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.40 --analyse "all" --weightb --me "umh" --subme 6 --progress -o NUL Encodage.avs
x264.exe --bframe 2 --8x8dct --ref 16 --filter -1:-1 --bitrate 896 --pass 3 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.40 --analyse "all" --weightb --me "umh" --subme 6 --progress -o NUL Encodage.avs
x264.exe --bframe 2 --8x8dct --ref 16 --filter -1:-1 --bitrate 896 --pass 2 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.40 --analyse "all" --weightb --me "umh" --subme 6 --progress -o x264HP-896.mp4 Encodage.avs

[babayaga]

19.06.05
- Ateme codec Full quality

I don't know if this is a "no-limit" contest, but the full quality mode is mostly a debug mode that will not be part of an official release; at least the way it is done today. The reason is simply its crawling speed that could discourage even the most addicted user :)

More seriously, you did the measurements in psy 3 which is experimental and was designed for very odd sequences. This is not the case of HP2 or custom trailers or movies (which is fortunate). Besides some have complained that psy 3 blurs too much.
So I reworked this psy level and it will be available in the next verison of our ongoing beta. The results are so far:

----------------------------------------------------------------------
| version | bitrate | size | OPSNR | SSIM0 | SSIM2 | speed |
----------------------------------------------------------------------
| HP 450kbps extra | 447kbps | 6746 | 41.00 | 75.91 | 71.73 | 6.0fps |
----------------------------------------------------------------------
| MP 450kbps extra | 447kbps | 6747 | 40.61 | 74.34 | 69.92 | 7.6fps |
----------------------------------------------------------------------
| HP 900kbps extra | 896kbps | 13491 | 43.61 | 83.96 | 81.25 | 4.8fps |
----------------------------------------------------------------------
| MP 900kbps extra | 896kbps | 13492 | 43.32 | 83.16 | 80.31 | 6.0fps |
----------------------------------------------------------------------
| HP 450kbps full | 447kbps | 6747 | 41.10 | 76.28 | 72.10 | 1.0fps |
----------------------------------------------------------------------
| MP 450kbps full | 447kbps | 6746 | 40.72 | 74.70 | 70.33 | 1.4fps |
----------------------------------------------------------------------
| HP 900kbps full | 896kbps | 13492 | 43.70 | 84.25 | 81.53 | 0.9fps |
----------------------------------------------------------------------
| MP 900kbps full | 896kbps | 13491 | 43.44 | 83.16 | 80.63 | 1.3fps |
----------------------------------------------------------------------

Encoding parameters are: 2 pass, 8 Ref, 3 BRef, max 2 Bframes, CABAC, BPred, MPart, WPred, Inloop strength -1, Chroma, next psy 3

The speed was measured on an Athlon 3500+ while I was working.

[Ice =A=]

I just wanted to say (just after the recent posts) that this comparison is in my opinion very well made with a lot of love for detail, proper explanations and a good concept. Thank you for that!

One more thing: You maybe really can't objectively compare video quality or even define that concept, since one is more sensitive to (not) smooth colour gradients and the other to details preservation. But then there is hardly anything in the world you really can compare! You can try to find some good compromise, something rather representative, and I think that's what Sagittaire is trying to do here and what he has spend quite some time for!
And (about you socrates argument :)): Some of the most important concepts in science are not so well defined, actualls! Just think of physics and tell me: What exactly is time?! You can really well compute with the concept "time", but you can't say much more about it, then that it's measured with a watch...

[Sulik]

I'm very impressed with the performance of MEncoder @ 1250Kbps.

Could you provide more information about what build of MEncoder and MPEG-2 codec did you use and more detailed parameters for the MPEG-2 encoding ?
(I don't see MEncoder available in the download section)

[Sagittaire]

yes, IMO it's by far the best MPEG2 encoder for very low bitrate ...

mencoder.exe HDTV2.avs -o MPEG2-CM.m2v -ovc lavc -lavcopts vcodec=mpeg2video:vpass=1:turbo:vqmin=1:vqscale=2:vrc_maxrate=9800:vrc_buf_size=1835:keyint=15:vqcomp=1.00:vmax_b_frames=2:vb_qfactor=1.50:vb_qoffset=0.00:preme=2:dia=6:predia=6:mbd=2:mbcmp=262:ildctcmp=262:precmp=262:cmp=256:subcmp=262:qprd:mv0:trell:cbp:psnr:aspect=16/9:intra_matrix=8,16,19,22,26,27,29,34,16,16,22,24,27,29,34,37,19,22,26,27,29,34,34,39,22,22,26,27,29,34,38,42,22,26,27,29,32,36,40,50,26,27,29,32,36,40,50,61,26,27,29,35,40,50,59,75,27,29,35,40,50,59,75,89:inter_matrix=16,17,18,19,20,21,22,23,17,18,19,20,21,22,23,25,18,19,20,21,22,23,24,26,19,20,21,22,23,24,26,28,20,21,22,23,25,26,28,29,21,22,23,24,26,28,29,31,22,23,24,26,28,29,31,34,23,24,25,28,29,31,34,38 -of rawvideo -ffourcc MPG2

mencoder.exe HDTV2.avs -o MPEG2-CM.m2v -ovc lavc -lavcopts vcodec=mpeg2video:vpass=2:vqmin=1:vbitrate=4500:vrc_maxrate=9800:vrc_buf_size=1835:keyint=15:vqcomp=1.00:vmax_b_frames=2:vb_qfactor=1.50:vb_qoffset=0.00:preme=2:dia=6:predia=6:mbd=2:mbcmp=262:ildctcmp=262:precmp=262:cmp=256:subcmp=262:qprd:mv0:trell:cbp:psnr:aspect=16/9:intra_matrix=8,16,19,22,26,27,29,34,16,16,22,24,27,29,34,37,19,22,26,27,29,34,34,39,22,22,26,27,29,34,38,42,22,26,27,29,32,36,40,50,26,27,29,32,36,40,50,61,26,27,29,35,40,50,59,75,27,29,35,40,50,59,75,89:inter_matrix=16,17,18,19,20,21,22,23,17,18,19,20,21,22,23,25,18,19,20,21,22,23,24,26,19,20,21,22,23,24,26,28,20,21,22,23,25,26,28,29,21,22,23,24,26,28,29,31,22,23,24,26,28,29,31,34,23,24,25,28,29,31,34,38 -of rawvideo -ffourcc MPG2

[Sagittaire]

update with new ateme beta ... more and more powerfull

Difference between NDAVC HP and DivX6 is more important than difference between DivX6 and DivX 3 (with libavcodec) ... :eek:

[IgorC]

Improvements for Ateme H.264 was only for 450 kbps. For 900 the results didn't change.

900 kbps
PSNR SSIM2
Beta 1 43.63 81.60
Beta 2 43.70 81.54

For Ateme H.264 was used -full quality mode -> speed 0.2 fps
For x264 was used optimal settings -> 1-2 fps

Make conclusions yourself.

[Sagittaire]

...... ???

Ateme encavc 1.2.0.17 is better for all bitrate. Result change with new psy 3 ...

Ateme is very better for SSIM (visually too) and faster than x264 ... for example in my test NDAVC MP result are the same than x264 HP ...

see bobololo test for speed with 1.2.0.17 build:


----------------------------------------------------------------------
| version | bitrate | size | OPSNR | SSIM0 | SSIM2 | speed |
----------------------------------------------------------------------
| HP 450kbps extra | 447kbps | 6746 | 41.00 | 75.91 | 71.73 | 6.0fps |
----------------------------------------------------------------------
| MP 450kbps extra | 447kbps | 6747 | 40.61 | 74.34 | 69.92 | 7.6fps |
----------------------------------------------------------------------
| HP 900kbps extra | 896kbps | 13491 | 43.61 | 83.96 | 81.25 | 4.8fps |
----------------------------------------------------------------------
| MP 900kbps extra | 896kbps | 13492 | 43.32 | 83.16 | 80.31 | 6.0fps |
----------------------------------------------------------------------
| HP 450kbps full | 447kbps | 6747 | 41.10 | 76.28 | 72.10 | 1.0fps |
----------------------------------------------------------------------
| MP 450kbps full | 447kbps | 6746 | 40.72 | 74.70 | 70.33 | 1.4fps |
----------------------------------------------------------------------
| HP 900kbps full | 896kbps | 13492 | 43.70 | 84.25 | 81.53 | 0.9fps |
----------------------------------------------------------------------
| MP 900kbps full | 896kbps | 13491 | 43.44 | 83.16 | 80.63 | 1.3fps |
----------------------------------------------------------------------

[IgorC]

I'm NOT blind and there is NO difereence
900 kbps
PSNR SSIM2
Beta 1 43.63 81.60
Beta 2 43.70 81.54

PSNR +0.07 and SSIM -0.06 and that's call Difference? is there any problem with math?

[Sagittaire]

in fact ME is always better but psy 3 is new: psy 3 in encavc 14 is not the same in encavc 17. You must compare with comparable setting for psy mode.

Compare beta 1 and beta 2 with psy 3 it's like if you compare beta 1 and beta 2 with different matrix ... :o

[IgorC]

I know that there were impovement of psy 3 in beta 2. But you said that is better for ALL bitrates. So what is 'better' for you? 900 kbps +0.07 for PSNR and -0.06 for SSIM. Is that better for you?
It´s exactly same quality for 900 kbps. New psy mode just like a new matrice.
Look, I understand that you like Ateme codec . So do I. But new psy mode is better only for low bitrate 450 kbit/s and you're saying that is better for all bitrates. And that's not true.

[IgorC]

Difference between NDAVC HP and DivX6 is more important than difference between DivX6 and DivX 3 (with libavcodec) ... :eek:

SSIM(Divx6) - SSIM (divx3) = 65.02 -58.56 = 6.46
SSIM (Nero HP) - SSIM (Divx6) = 7.08

But the difference of 6.46 for number 65.02(Divx6 ) is bigger than 7.08 for 72.10(Ateme HP)
6.46/65.02 = 0.09935...
7.08/72.10 = 0.0981969.....

I'm not sure if it's correct your or my calculation because SSIM has not a lineal function.

[Sagittaire]

In fact you choose quality level for all codec ( OPSNR = x or SSIM = y) and you make encoding for each codec with this quality level.

Codec A : u Kbps
Codec B : v Kbps
Codec C : w Kbps

After you compare (v-u) and (w-v) ...

[easyfab]

Sagittaire have you planed to do some benchmark on HD clips?
I'm very interested to know if avc codecs are also the best in a 1920*1080 resolution for a bitrate between 3500 and 4500 kps or if XVID rules for this usage.

[lrms]

MPEG4 ASP : Libavcodec
Coder: MEncoder dev-CVS-050426-18:49-3.4.2
Setting: 3 pass, adaptative BFrame 2/1.50/1.00, Chroma Motion, Trellis, RDO, SAD, 6 diamond size
decoder: ffdshow decoder with PP4
files: Libavcodec ASP 450 Kbps & Libavcodec ASP 900 Kbps

Sagittaire: did you use "v4mv" (4 montion vectors per macroblock) for "MPEG4 ASP : Libavcodec"?
In my tests it seemed to improve average PNSR a bit (~0.1 for the test source @900kbps). By the way, could post the complete mencoder command line you used for "MPEG4 ASP : Libavcodec"? (I found it hard to guess the whole thing just by the text)

[Sagittaire]

Sagittaire: did you use "v4mv" (4 montion vectors per macroblock) for "MPEG4 ASP : Libavcodec"?
In my tests it seemed to improve average PNSR a bit (~0.1 for the test source @900kbps). By the way, could post the complete mencoder command line you used for "MPEG4 ASP : Libavcodec"? (I found it hard to guess the whole thing just by the text)

CLI in batch file. 3 pass with best setting for this source. I think that good masking can improuve SSIM ... ;)

mencoder.exe encodage.avs -o LAVC-900.avi -ovc lavc -lavcopts vstrict=1:vpass=1:vbitrate=896:vb_qfactor=1.5:vb_qoffset=1.0:vqcomp=0.75:vmax_b_frames=2:vb_strategy=1:preme=2:dia=6:predia=6:mbd=2:mbcmp=262:ildctcmp=262:precmp=262:cmp=256:subcmp=262:qprd:v4mv:mv0:trell:cbp:psnr -of avi -ffourcc MP4V
mencoder.exe encodage.avs -o LAVC-900.avi -ovc lavc -lavcopts vstrict=1:vpass=3:vbitrate=896:vb_qfactor=1.5:vb_qoffset=1.0:vqcomp=0.75:vmax_b_frames=2:vb_strategy=1:preme=2:dia=6:predia=6:mbd=2:mbcmp=262:ildctcmp=262:precmp=262:cmp=256:subcmp=262:qprd:v4mv:mv0:trell:cbp:psnr -of avi -ffourcc MP4V
mencoder.exe encodage.avs -o LAVC-900.avi -ovc lavc -lavcopts vstrict=1:vpass=3:vbitrate=896:vb_qfactor=1.5:vb_qoffset=1.0:vqcomp=0.75:vmax_b_frames=2:vb_strategy=1:preme=2:dia=6:predia=6:mbd=2:mbcmp=262:ildctcmp=262:precmp=262:cmp=256:subcmp=262:qprd:v4mv:mv0:trell:cbp:psnr -of avi -ffourcc MP4V

[akupenguin]

mbd=2:mbcmp=262:ildctcmp=262:precmp=262:cmp=256:subcmp=262

mbcmp is ignored because you're using mbd=2 (RD mb decision).
ildctcmp is ignored because your video isn't interlaced.
and you have precmp=RD while main cmp=SAD, which is backwards.
Try
mbd=2:precmp=258:cmp=262:subcmp=262
and a possible addition of qns=2 If you haven't already.

[lrms]

Well, just to be sure I did try the settings used by Sagittarie and then the changes suggested by akupenguin. Sagittarie settings still seem to be the best I could find for this test.
At least by the builtin encoder PSNR numbers:

Original Sagittarie settings:
Average PSNR: Y:42.00, Cb:45.11, Cr:45.93, All:42.89

With comp. functions change as suggested by akupenguin:
Average PSNR: Y:42.00, Cb:45.10, Cr:45.91, All:42.88

Plus qns=1:
Average PSNR: Y:41.94, Cb:45.14, Cr:45.95, All:42.84

Plus qns=2:
Average PSNR: Y:41.94, Cb:45.11, Cr:45.93, All:42.84

All were tree pass encodings done using mencoder 050626 (Sascha Sommer mingw build - http://ftp.mplayerhq.hu/MPlayer/releases/win32-beta/). Notice that these results are not to be compared with other PSNR measurments in this thread because the procedure was all done with mencoder, but I expect the numbers using the suggested procedure at least follow the same trend as mencoder numbers.

[zombi55]

No update anymore :confused:

[redfordxx]

I made H.264 test encodes, I write approx info, coz I am not home and I will redo tests more exact and relevant.
Source 1440x816 anamorph WMV9Pro cca 6Mbps, blocky, noisy,2076frames
Preprocessing SmoothD deblock & denoise THEN -->1280x544
Nero 7 Premium Recode --> SSIM=cca 85 (visually more blocky)
x264 Sharktooth's v368 --> SSIM=cca 88
SSIM with luma, no deblock inloop (slow machine=2600+;-), 3bframes, cca4200kbps
quite in contrast with results on 1st page
Do you think I should make tests w/o bframes too to make it more relevant?

[EDIT] video 1:26, CPU time for playback BSP Overlay mixer: Nero 0:43, x264 0:51

[Ice =A=]

Good work!
But why should no bframes make it more relevant?

[redfordxx]

From some discussion somewhere I realized that b-frames are HVS feature and they may look good and sharp but differ from original frame siginificantly at the same time. So maybe good to know whether the b frames are the origin of the Neros lower performance

[stephanV]

B-frames mostly improve PSNR/SSIM and I'm not sure if it is (purely) an HVS feature.

[redfordxx]

B-frames mostly improve PSNR/SSIM and I'm not sure if it is (purely) an HVS feature.
Search for it, I am sure there is a discussion like "whether metrics are relevant and for which features". And Sharktooth mentioned it as a HVS like feature and I agree because reasons I mentioned post above.

[stephanV]

Aah, that famous thread. :)

I think Manao mentioned there that b-frames are usually better RD wise too and thus should improve PSNR and i guess normally SSIM too.

[redfordxx]

RD wise What is RD wise?should improve PSNR and i guess normally SSIM too.hmm I agree coz I remember some of my older experiences too (when properly set the amount)

Anyway, few days ago I thought I will choose Nero because the playback is faster (20%), according to this challenge is metrics better and read some visual opinions that it is better.
But now, 85 SSIM vs. 88 is significant difference and I should reconsider. Maybe the chart in this thread is old and x264 improved and Manao wrote Nero is not Highprofile anymore ...

[stephanV]

What is RD wise?
RD = rate distortion. Very simpy put it is bitrate/PSNR. RD optimisations are aiming to optimise that number.


But now, 85 SSIM vs. 88 is significant difference and I should reconsider. Maybe the chart in this thread is old and x264 improved and Manao wrote Nero is not Highprofile anymore ...
This chart shows Ateme's beta encoder, not the encoder in Nero Recode.

[redfordxx]

This chart shows Ateme's beta encoder, not the encoder in Nero Recode.Well I was suspicious I didn't understand the whole truth with the fact, that Nero is using Ateme now.
So, to end up my confusion, please where can I get the Ateme's encoder?

[Manao]

http://forum.doom9.org/showthread.php?p=731493#post731493

[redfordxx]

Hope this is suitable thread for this post...

I have seen often recommendations when making 3pass encoding to make (now speaking about ME) low quality 1st pass and hi qual 2nd and 3rd pass. Then I thought that maybe it is better to make 1st and 3rd hi qual and 2nd lo quality. I am running tests on Harry Potter now and I will prove the hypothesis ASAP.

Reason: (correct me if anything wrong)
IIUC the quality of encoding depends on following:
- optimally assign frame types
- optimally distribute the bitrate (curve)
- effectively use references instead of DCT

I believe
- frame types are assigned in 1st pass - nowhere else.
- quality of bitrate curve of any pass is dependent of the quality of bitrate curve in input stats file
- qual of refs etc. depends on the last pass itself

Conclusion:
I think the best quality/time ratio (provided we decided for 3pass) we can reach with following strategy:
Focus 1st pass setting on correct I,P,B frames distribution, nothing else
Focus 2nd pass setting on bitrate curve approx (forget frame types, the are already solved)
3rd pass with full quality (or maybe some advanced trade/off in settings related to frametypes distribution if there are some)
The proposed settings tweaking is for more experienced users than me who understand the x264 --- I appreciate help with your opinions.

I have read somewhere that the 3pass style "fast slow slow" makes 2.9 pass quality iin 2.1pass time. I think it is 2.8pass quality for the 2.9pass quality is reached with the "slow fast slow" sheme... just joking ;) Nevertheless, I believe (expressing with this logic) it is possible (with advanced settings) to reach the 2.8pass or so quality in less than 2pass time. It is maybe worth thinking...

[Manao]

- frame types are assigned in 1st pass - nowhere else.Wrong ( but it's a correct assertion for XviD ). Anyway, encoding quality should hardly influence frame type decision.

And also, a third pass is mainly useless as soon as the video is long enough ( long --> > 1000 frames ), so I wouldn't bother with it. The proper way is : first pass as fast as you can, second pass as slow as you can afford.

[redfordxx]

I have finished my tests:
To see the difference I made it really extreme ME dia vs. esa. I had two kinds of settings e (stands for esa) and d (stands for dia) as shown below--keyint 250 --ref 16 --mixed-refs --bframes 3 --b-bias 0 --pbratio 1.3 --b-pyramid --nf --subme 7 --b-rdo --weightb --trellis 2 --analyse all --8x8dct --me esa --merange 32
--keyint 250 --ref 16 --mixed-refs --bframes 3 --b-bias 0 --pbratio 1.3 --b-pyramid --nf --subme 7 --b-rdo --weightb --trellis 2 --analyse all --8x8dct --me dia --merange 16In following I will code the filenames with these letters like:
eee stands for 3pass file with all passes made with ME esa settings
dee stands for 3pass file with 1st pass made with ME dia settings
edx stands for 2pass file with 2nd pass made with ME dia settings

So, it was HPII without some mainly black scenes, 1000kbps,2714frames.

Assuming eee is the best we can reach, I compared all other encodes to eee with SSIM.


Here are the results:
ddx 81,45
edx 81,76
dex 83,22
eex 83,97
dde 85,79
dee 86,38
ede 88,04

So here, the middle fast makes better than first fast. Of course, this is only one example, but still...

[EDIT] Damn, now I got it: the B frames distribution is different... so this is not very relevant

[redfordxx]

frame types are assigned in 1st pass - nowhere else.
Wrong ( but it's a correct assertion for XviD ).Then on what condition it is changed? In all of my test encodings I have never seen that frametypecount changed between passes.
It changed only depending on 1st pass quant, bitrate, Bs, bias, me...

And also, a third pass is mainly useless as soon as the video is longAgree

[Manao]

Wrong ( but it's a correct assertion for XviD ).Mmm, indeed I'm wrong ( I checked the source code ). Strange, because I definitely seem to remember that frame types could be changed. So indeed, frame type depends on quality ( since ME during frame type decision is made with the ME settings chosen, on half resolution frames ). From what I read however, ref shouldn't play a part, but subme & me, merange will )

[redfordxx]

Sorry, too much information encoded in too short text...(for me)
So indeed, frame type depends on qualityQuality of what? 1st pass?
(since ME during frame type decision is made with the ME settings chosen, on half resolution frames ).Do I understand correctly, that when deciding the frame type, ReducedByTwo resolution frame is used? That implies especially important subme?
ref shouldn't play a part,
So ref can be 1 in 1st pass. In fact, it seems logical because I frame decision is important on scene change, which is two neighbour frame comparision. Bframe decision is I believe whether the previous and next frame are similar enough to create Bf. Is Bframe made only by references or DCT data also?

[DeathTheSheep]

-DivX6.1
-XviD1.1 Release1.1 build
-VP7.0.9
-AtemeHP b3

Do ya think these guys will make it into the testing arena? ;)

[IgorC]

When xvid.org will say final 1.1 it will be while Xvid 1.1 is not still realesed.
Xvid 1.1 new SSIM here result 78.02 while old Xvid beta1/2 was 77.96 and Divx6 old 77.99

Last x264 should be also checked. It's very close to Ateme beta now but still slow. There were a hundred improvements since then.


Indeed you can do yorself. Videotrailer and ssim/opsnr.dll are here for download.

[Sagittaire]

Actually I prepare another test:

video codec
- Ateme AVC MP (Nero) and HP
- x264 MP and HP
- XviD 1.1 b2
- DivX 6.1
- WMV9 VCM
- VP7 7.0.9

resolution
- portable resolution ~ 320*240
- DVD resolution ~ 720*400
- HD resolution ~ 1280*720

source
- Ice Age Trailer
- HPIV trailer
- King Kong Trailer

Quality Test
- metrix test : APSNR, OPSNR and SSIM
- screenshoot
- little blind test

Speed test
and for the first time a real speed test ...
- graph fps = fct (PSNR)
- graph fps = fct (SSIM)
IMO with very surprising result ... ;)

[redfordxx]

Actually I prepare another test:
source
- HPIV trailer
Isn't it better to test HPII for "backward compatibility"? I mean to see the improvement over time...

[temporance]

M. Sagittaire,

I like to try to replicate (parts of) codec shootouts so would like to know where you found your sources. Presumably they're downloaded HD trailers. Can you upload them somewhere like you did with the HP sources from your last comparison?

Look forward to seeing some of your results :)

[redfordxx]

Presumably they're downloaded HD trailers.
I don't know exactly the sources S. mentioned, but if interested in general, look here (http://www.highdefforum.com/showthread.php?t=6537).

[Sagittaire]

IMO the best HD source are here (http://www.apple.com/trailers/)

-> Very better quality than HD-WMV sources
-> Very better than *.TS HDTV 720p/1080i sources
-> Real Full HD progressive source

You can make that with these sources:
http://forum.surdvd.com/viewtopic.php?t=43651

[MuTeK]

Sagittaire
Please, clear PM.

[redfordxx]

Speed test
and for the first time a real speed test ...
- graph fps = fct (PSNR)
- graph fps = fct (SSIM)
IMO with very surprising result ... ;)
What is the best method to measure encoding speed (of x264) to eliminate avisynth script slow down?

Related question: Presuming, each input frame is decoded only once during the encoding process and then kept in memory until not used anymore?

[temporance]

I don't know exactly the sources S. mentioned, but if interested in general, look here (http://www.highdefforum.com/showthread.php?t=6537).
What codec was used for these source? I'm not familiar with .mov's.

[redfordxx]

What codec was used for these source? I'm not familiar with .mov's.various, they are not all the same..
usually inside the *.exe are wmv

[temporance]

Sorry, I meant to ask Sagittaire about the trailers at http://www.apple.com/trailers/ - what codec did Apple use?

[Sagittaire]

H264 for HD Quick Time trailer