Sagittaire
1st August 2007, 13:06
~ HD Benchmark Challenge ~
You think that your standard is the best : prove that ... !!?
A - Introduction
The purpose of this challenge is to determine which is the best codec format encoding for objective test.
HD-DVD & BD can use MPEG2, H264 and VC-1 for video codec.
B - Rules
Rule 1 : Source
Uncompressed source is available Here (http://orange.blender.org/blog/original-lossless-source-available/) for reproduce the test.
Open source Elephant Dream movie, video 1920*1080 PNG lossless, audio 5.1 Flac lossless, 15 691 frames
Rule 2 : Pre-process and encoding
You must use this avisynth script for make encoding:
Source=ImageSource("C:\LossLess\images\%05d.png", start=1, end=15691, fps=23.976)
Source=ConvertToYV12(source, matrix="Rec709")
return source
Other pre-process are not autorized. It's possible to propose example with high quality pre-process (like sharp ... ect) but just for demonstration and not for this challenge.
Video Stream must be like HD-DVD/BD compliant stream. You can use all the compliant audio codec and all the compliant bitrate for audio. This test is just video challenge and not audio challenge. All the video stream must be compliant with these setting:
MPEG2 Encoding
Profil & Level: MP@HL except specific restrictions
Max GOP lenght: 14 frames
Maximum bitrate: 20.0 Mbps, 24.0 Mbps and 28.0 Mbps
Buffer size: 9781 Kbits for principal HD video stream
Horizontal Vector Range: +/- 1024 pixels
Vertical Vector Range: +/- 128 pixels
Other Restrictions Setting: max adaptative GOP at 14, max adaptative bframe at 7
VC-1 Encoding
Profil & Level: AP@L3 except specific restrictions
Max GOP lenght: 14 frames
Maximum bitrate: 20.0 Mbps, 24.0 Mbps and 28.0 Mbps
Buffer size: 14745 Kbits for principal HD video stream
Horizontal Vector Range: unlimited
Vertical Vector Range: unlimited
Other Restrictions Setting: max adaptative GOP at 14, max adaptative bframe at 7
H264 Encoding
Profil & Level: HP@L4.1 except specific restrictions
Max GOP lenght: 14 frames
Maximum bitrate: 20.0 Mbps, 24.0 Mbps and 28.0 Mbps
Buffer size: 14745 Kbits for principal HD video stream
Horizontal Vector Range: +/- 1024 pixels
Vertical Vector Range: +/- 512 pixels
Other Restrictions Setting: max adaptative GOP at 14, max adaptative bframe at 7,
Max reference at 4, Max breference at 3, no film grain modeling
Rule 3 : Bitrate and Max Bitrate
You must use these bitrate/size for encoding:
HD-DVD with "super bitrate" video stream and simple HDDVD authoring:
18 Mbps (Max at 28.0 Mbps) for video stream with +/- 0.5 % for bitrate tolerance
HD-DVD with "medium bitrate" video stream and standard HDDVD authoring:
12 Mbps (Max at 24.0 Mbps) for video stream with +/- 0.5 % for bitrate tolerance
HD-DVD with "low bitrate" video stream and standard HDDVD authoring:
6 Mbps (Max at 20.0 Mbps) for video stream with +/- 0.5 % for bitrate tolerance
NB: we can use HD-DVD structure too on simple DVD DL 12 cm at 8.5 GB
Rule 4 : Visual Tests
Here you can find encoding elementary stream for subjective test ... try to find yourself the best visual quality:
HD-DVD & BD MPEG2 at 18 Mbps: not available
HD-DVD & BD MPEG2 at 12 Mbps: not available
HD-DVD & BD MPEG2 at 6 Mbps (http://jfl1974.free.fr/HDDVD/MPEG2/6Mbps/MPEG2_6Mbps.m2v)
HD-DVD & BD VC-1 at 18 Mbps: not available
HD-DVD & BD VC-1 at 12 Mbps: not available
HD-DVD & BD VC-1 at 6 Mbps (http://jfl1974.free.fr/HDDVD/VC1/6Mbps/VC1_6Mbps.vc1)
HD-DVD & BD H264 at 18 Mbps (http://jfl1974.free.fr/HDDVD/H264/18Mbps/)
HD-DVD & BD H264 at 12 Mbps (http://jfl1974.free.fr/HDDVD/H264/12Mbps/)
HD-DVD & BD H264 at 6 Mbps (http://jfl1974.free.fr/HDDVD/H264/6Mbps/)
Rule 5 : Metrics Tests
SSIM (YV12): IMO the best metric. it uses some HVS properties but not all ...
PDF description for VSSIM (http://www.cns.nyu.edu/~zwang/files/papers/vssim.pdf)
Overall PSNR (YV12): Very good metric test, particullary for Rate Control Quality test ...
MSE = Sum t = 1 -> NFrame Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (NFrame*width*height)
Overall PSNR = 10 * Log ( 255² / MSE)
Average PSNR (YV12):The worse metric test but Frame PSNR graph is very good to compare RC strategy ...
MSE = Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (width*height)
Average PSNR = Sum t = 1 -> NFrame 10 * Log ( 255² / MSE) / Nframe
SSIM vs PSNR
"For a given reference video and a given compressed video, it is meant to compute a quality metric, based on perceived visual distortion. Unlike the well-known PSNR measure, it's not purely mathematical, and should correlate much better with human vision."
"A higher MSE (and so lower PSNR) should mean that the compressed clip is a worse image but MSE and PSNR are flawed in this respect as numerous tests have shown. However with SSIM, according to tests carried out on the VQEG dataset, a higher Q (SSIM value) has a much better relation to the visual quality of the compressed clip. Despite this, bear in mind the SSIM metric still isn't perfect."
Here (http://multimediacom.free.fr/Download/SSIM.rar) last AviSynth plugin for VSSIM in YV12
Average PSNR vs Overall PSNR
It's easy to increase artificially Average PSNR with Rate Control tweak. In certain case CBR (0% for variability) will be better than real VBR (100% for variability or constant quant). It's generaly not possible with Overall PSNR. Overall PSNR is very good to test Rate Control quality. If difference between Average PSNR and Overall PSNR is very high, "local quality difference" (difference between Pframe or Bframe for exemple) or "scene quality difference" (difference between low motion and high motion for exemple) will be too high and Rate Control for this codec not very good.
Here (http://multimediacom.free.fr/Download/CompareYV12.zip) last AviSynth plugin for Overall PSNR in YV12
Open your avs metric script with VirtualDub. Select file > preview input for run the test. Select file > close video file for obtain final result in log metric files. You must use these AviSynth type script for metric test:
# --> Source Opening <--
Source=ImageSource("C:\LossLess\images\%05d.png", start=1, end=15691, fps=24)
Source=ConvertToYV12(source, matrix="Rec709")
# --> Video Opening <--
video=Mpeg2Source("D:\...\MPEG2_24Mbps.d2v",idct=2)
# --> PSNR analysis <--
compareYV12(video,source,"YUV","OPSNR_MPEG2_24Mbps.log")
# --> Source Opening <--
Source=ImageSource("C:\LossLess\images\%05d.png", start=1, end=15691, fps=24)
Source=ConvertToYV12(source, matrix="Rec709")
# --> Video Opening <--
video=Mpeg2Source("D:\...\MPEG2_24Mbps.d2v",idct=2)
# --> SSIM analysis <--
return SSIM(source,video,"results.csv","SSIM_MPEG2_24Mbps.txt",lumimask=2)
C - Results
At this time here the best result (http://jfl1974.free.fr/HDDVD/Test_Metric.rar) ... but make better if you can ... :devil:
|--------------|---------|---------|----------|---------|---------|
| Codec | PProc | Bitrate | Size | OPSNR | SSIM 2 |
|--------------|---------|---------|----------|---------|---------|
| MPEG2 | No | 5999 | 479233 | 43.08 | 82.55 |
| VC-1 | No | 5994 | 479405 | 44.15 | 85.85 |
| H264 | No | 6007 | 479766 | 46.35 | 89.54 |
|--------------|---------|---------|----------|---------|---------|
| MPEG2 | No | 12001 | 958666 | 46.38 | 91.27 |
| VC-1 | No | 11987 | 957575 | 47.30 | 92.25 |
| H264 | No | 12008 | 959329 | 49.11 | 94.18 |
|--------------|---------|---------|----------|---------|---------|
| MPEG2 | No | 18002 | 1438146 | 47.17 | 93.26 |
| VC-1 | No | 18005 | 1438379 | 49.00 | 94.60 |
| H264 | No | 18009 | 1438862 | 50.53 | 95.74 |
|--------------|---------|---------|----------|---------|---------|
SSIM 0: Lumimask Off
SSIM 1: Lumimask On (Original Lumimask)
SSIM 2: Lumimask On (One2Tech Patch)
H264 vs VC1 vs MPEG2 at 6Mbps
http://jfl1974.free.fr/HDDVD/6Mbps_small.PNG (http://jfl1974.free.fr/HDDVD/6Mbps.PNG)
H264 vs VC1 vs MPEG2 at 12Mbps
http://jfl1974.free.fr/HDDVD/12Mbps_small.PNG (http://jfl1974.free.fr/HDDVD/12Mbps.PNG)
H264 vs VC1 vs MPEG2 at 18Mbps
http://jfl1974.free.fr/HDDVD/18Mbps_small.PNG (http://jfl1974.free.fr/HDDVD/18Mbps.PNG)
H264 at 6Mbps vs MPEG2 at 12 Mbps
http://jfl1974.free.fr/HDDVD/H264_6M_MPEG2_12M_small.PNG (http://jfl1974.free.fr/HDDVD/H264_6M_MPEG2_12M.PNG)
H264 at 12Mbps vs VC1 at 18 Mbps
http://jfl1974.free.fr/HDDVD/H264_12M_VC1_18M_small.PNG (http://jfl1974.free.fr/HDDVD/H264_12M_VC1_18M.PNG)
VC1 encoding by "Anonyme"
Encoding with vc1_enc.exe from VC1 PEP software
See Annexe for complete profil
Video stream are compliant with HDDVD "NTSC" profil
H264 encoding by "Sagittaire"
Encoding with x264.exe
See Annexe for complete profil
Video stream are compliant with HDDVD "NTSC" profil
MPEG2 encoding by "Sagittaire"
Encoding with mencoder.exe from libavcodec MPEG2
See Annexe for complete profil
Video stream are compliant with HDDVD "NTSC" profil
D - Conclusion
At this time for this movie
1) I use certainely the best MPEG2 encoder available in the area for metric: Libavcodec. The result is simply impressive for the old MPEG2 codec. With PP4 the result is even better. Libavcodec produce by far much better metric result here than all the other implementations (HCEnc, TMPGEnc, Mainconcept, Procoder ...).
2) VC1 is a very good surprise for me. For "low/medium bitrate" (high quantisation level) VC1 in unable to fight with H264 but for "High bitrate" the SSIM for VC1 is close to H264. MS say "VC1 is particulary optimized for high bitrate with high resolution" and it's true. Anyway MS claim that VC1 is able to produce the same metric that H264 but it's clearly false here.
3) H264 is simply the best for metric in all situation and particulary for "low bitrate" situation. Anyway differences between VC1 and H264 are not so high for low quantisation encoding.
4) VC1 and H264 at 6/20 Mbps (average bitrate/max bitrate) are unable to produce better quality than MPEG2 at 12/24 Mbps. H264 at 12/24 Mbps produce better quality than MPEG2 at 18/28 Mbps but not VC1. H264 at 12/24 Mbps is unable to produce better quality than VC1 at 18/28 Mbps. MicroSoft annonce (http://www.microsoft.com/windows/windowsmedia/howto/articles/vc1techoverview.aspx#VC1ComparedtoOtherCodecs) ratio between 1:2 and 1:3 for MPEG2 vs VC1 but it's really not the case here. The x264 particularly optimized in its development to obtain very good PSNR is not able to obtain a ratio of 2:1 with MPEG2 in this challenge.
5) Use constrained vbv rate control produce very inconstant quality and use overall or average metric results in this case is very difficult because complex part (motion and/or texture) can produce very bad results for metric. For better results it will be better to use metric graph for part by part comparison. If a codec want produce good overall/average metric here then the codec must have particulary good vbv Rate Control optimization.
IF YOU DON'T AGREE WITH THAT, TRY WITH YOUR ENCODER ... !!!
You think that your standard is the best : prove that ... !!?
A - Introduction
The purpose of this challenge is to determine which is the best codec format encoding for objective test.
HD-DVD & BD can use MPEG2, H264 and VC-1 for video codec.
B - Rules
Rule 1 : Source
Uncompressed source is available Here (http://orange.blender.org/blog/original-lossless-source-available/) for reproduce the test.
Open source Elephant Dream movie, video 1920*1080 PNG lossless, audio 5.1 Flac lossless, 15 691 frames
Rule 2 : Pre-process and encoding
You must use this avisynth script for make encoding:
Source=ImageSource("C:\LossLess\images\%05d.png", start=1, end=15691, fps=23.976)
Source=ConvertToYV12(source, matrix="Rec709")
return source
Other pre-process are not autorized. It's possible to propose example with high quality pre-process (like sharp ... ect) but just for demonstration and not for this challenge.
Video Stream must be like HD-DVD/BD compliant stream. You can use all the compliant audio codec and all the compliant bitrate for audio. This test is just video challenge and not audio challenge. All the video stream must be compliant with these setting:
MPEG2 Encoding
Profil & Level: MP@HL except specific restrictions
Max GOP lenght: 14 frames
Maximum bitrate: 20.0 Mbps, 24.0 Mbps and 28.0 Mbps
Buffer size: 9781 Kbits for principal HD video stream
Horizontal Vector Range: +/- 1024 pixels
Vertical Vector Range: +/- 128 pixels
Other Restrictions Setting: max adaptative GOP at 14, max adaptative bframe at 7
VC-1 Encoding
Profil & Level: AP@L3 except specific restrictions
Max GOP lenght: 14 frames
Maximum bitrate: 20.0 Mbps, 24.0 Mbps and 28.0 Mbps
Buffer size: 14745 Kbits for principal HD video stream
Horizontal Vector Range: unlimited
Vertical Vector Range: unlimited
Other Restrictions Setting: max adaptative GOP at 14, max adaptative bframe at 7
H264 Encoding
Profil & Level: HP@L4.1 except specific restrictions
Max GOP lenght: 14 frames
Maximum bitrate: 20.0 Mbps, 24.0 Mbps and 28.0 Mbps
Buffer size: 14745 Kbits for principal HD video stream
Horizontal Vector Range: +/- 1024 pixels
Vertical Vector Range: +/- 512 pixels
Other Restrictions Setting: max adaptative GOP at 14, max adaptative bframe at 7,
Max reference at 4, Max breference at 3, no film grain modeling
Rule 3 : Bitrate and Max Bitrate
You must use these bitrate/size for encoding:
HD-DVD with "super bitrate" video stream and simple HDDVD authoring:
18 Mbps (Max at 28.0 Mbps) for video stream with +/- 0.5 % for bitrate tolerance
HD-DVD with "medium bitrate" video stream and standard HDDVD authoring:
12 Mbps (Max at 24.0 Mbps) for video stream with +/- 0.5 % for bitrate tolerance
HD-DVD with "low bitrate" video stream and standard HDDVD authoring:
6 Mbps (Max at 20.0 Mbps) for video stream with +/- 0.5 % for bitrate tolerance
NB: we can use HD-DVD structure too on simple DVD DL 12 cm at 8.5 GB
Rule 4 : Visual Tests
Here you can find encoding elementary stream for subjective test ... try to find yourself the best visual quality:
HD-DVD & BD MPEG2 at 18 Mbps: not available
HD-DVD & BD MPEG2 at 12 Mbps: not available
HD-DVD & BD MPEG2 at 6 Mbps (http://jfl1974.free.fr/HDDVD/MPEG2/6Mbps/MPEG2_6Mbps.m2v)
HD-DVD & BD VC-1 at 18 Mbps: not available
HD-DVD & BD VC-1 at 12 Mbps: not available
HD-DVD & BD VC-1 at 6 Mbps (http://jfl1974.free.fr/HDDVD/VC1/6Mbps/VC1_6Mbps.vc1)
HD-DVD & BD H264 at 18 Mbps (http://jfl1974.free.fr/HDDVD/H264/18Mbps/)
HD-DVD & BD H264 at 12 Mbps (http://jfl1974.free.fr/HDDVD/H264/12Mbps/)
HD-DVD & BD H264 at 6 Mbps (http://jfl1974.free.fr/HDDVD/H264/6Mbps/)
Rule 5 : Metrics Tests
SSIM (YV12): IMO the best metric. it uses some HVS properties but not all ...
PDF description for VSSIM (http://www.cns.nyu.edu/~zwang/files/papers/vssim.pdf)
Overall PSNR (YV12): Very good metric test, particullary for Rate Control Quality test ...
MSE = Sum t = 1 -> NFrame Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (NFrame*width*height)
Overall PSNR = 10 * Log ( 255² / MSE)
Average PSNR (YV12):The worse metric test but Frame PSNR graph is very good to compare RC strategy ...
MSE = Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (width*height)
Average PSNR = Sum t = 1 -> NFrame 10 * Log ( 255² / MSE) / Nframe
SSIM vs PSNR
"For a given reference video and a given compressed video, it is meant to compute a quality metric, based on perceived visual distortion. Unlike the well-known PSNR measure, it's not purely mathematical, and should correlate much better with human vision."
"A higher MSE (and so lower PSNR) should mean that the compressed clip is a worse image but MSE and PSNR are flawed in this respect as numerous tests have shown. However with SSIM, according to tests carried out on the VQEG dataset, a higher Q (SSIM value) has a much better relation to the visual quality of the compressed clip. Despite this, bear in mind the SSIM metric still isn't perfect."
Here (http://multimediacom.free.fr/Download/SSIM.rar) last AviSynth plugin for VSSIM in YV12
Average PSNR vs Overall PSNR
It's easy to increase artificially Average PSNR with Rate Control tweak. In certain case CBR (0% for variability) will be better than real VBR (100% for variability or constant quant). It's generaly not possible with Overall PSNR. Overall PSNR is very good to test Rate Control quality. If difference between Average PSNR and Overall PSNR is very high, "local quality difference" (difference between Pframe or Bframe for exemple) or "scene quality difference" (difference between low motion and high motion for exemple) will be too high and Rate Control for this codec not very good.
Here (http://multimediacom.free.fr/Download/CompareYV12.zip) last AviSynth plugin for Overall PSNR in YV12
Open your avs metric script with VirtualDub. Select file > preview input for run the test. Select file > close video file for obtain final result in log metric files. You must use these AviSynth type script for metric test:
# --> Source Opening <--
Source=ImageSource("C:\LossLess\images\%05d.png", start=1, end=15691, fps=24)
Source=ConvertToYV12(source, matrix="Rec709")
# --> Video Opening <--
video=Mpeg2Source("D:\...\MPEG2_24Mbps.d2v",idct=2)
# --> PSNR analysis <--
compareYV12(video,source,"YUV","OPSNR_MPEG2_24Mbps.log")
# --> Source Opening <--
Source=ImageSource("C:\LossLess\images\%05d.png", start=1, end=15691, fps=24)
Source=ConvertToYV12(source, matrix="Rec709")
# --> Video Opening <--
video=Mpeg2Source("D:\...\MPEG2_24Mbps.d2v",idct=2)
# --> SSIM analysis <--
return SSIM(source,video,"results.csv","SSIM_MPEG2_24Mbps.txt",lumimask=2)
C - Results
At this time here the best result (http://jfl1974.free.fr/HDDVD/Test_Metric.rar) ... but make better if you can ... :devil:
|--------------|---------|---------|----------|---------|---------|
| Codec | PProc | Bitrate | Size | OPSNR | SSIM 2 |
|--------------|---------|---------|----------|---------|---------|
| MPEG2 | No | 5999 | 479233 | 43.08 | 82.55 |
| VC-1 | No | 5994 | 479405 | 44.15 | 85.85 |
| H264 | No | 6007 | 479766 | 46.35 | 89.54 |
|--------------|---------|---------|----------|---------|---------|
| MPEG2 | No | 12001 | 958666 | 46.38 | 91.27 |
| VC-1 | No | 11987 | 957575 | 47.30 | 92.25 |
| H264 | No | 12008 | 959329 | 49.11 | 94.18 |
|--------------|---------|---------|----------|---------|---------|
| MPEG2 | No | 18002 | 1438146 | 47.17 | 93.26 |
| VC-1 | No | 18005 | 1438379 | 49.00 | 94.60 |
| H264 | No | 18009 | 1438862 | 50.53 | 95.74 |
|--------------|---------|---------|----------|---------|---------|
SSIM 0: Lumimask Off
SSIM 1: Lumimask On (Original Lumimask)
SSIM 2: Lumimask On (One2Tech Patch)
H264 vs VC1 vs MPEG2 at 6Mbps
http://jfl1974.free.fr/HDDVD/6Mbps_small.PNG (http://jfl1974.free.fr/HDDVD/6Mbps.PNG)
H264 vs VC1 vs MPEG2 at 12Mbps
http://jfl1974.free.fr/HDDVD/12Mbps_small.PNG (http://jfl1974.free.fr/HDDVD/12Mbps.PNG)
H264 vs VC1 vs MPEG2 at 18Mbps
http://jfl1974.free.fr/HDDVD/18Mbps_small.PNG (http://jfl1974.free.fr/HDDVD/18Mbps.PNG)
H264 at 6Mbps vs MPEG2 at 12 Mbps
http://jfl1974.free.fr/HDDVD/H264_6M_MPEG2_12M_small.PNG (http://jfl1974.free.fr/HDDVD/H264_6M_MPEG2_12M.PNG)
H264 at 12Mbps vs VC1 at 18 Mbps
http://jfl1974.free.fr/HDDVD/H264_12M_VC1_18M_small.PNG (http://jfl1974.free.fr/HDDVD/H264_12M_VC1_18M.PNG)
VC1 encoding by "Anonyme"
Encoding with vc1_enc.exe from VC1 PEP software
See Annexe for complete profil
Video stream are compliant with HDDVD "NTSC" profil
H264 encoding by "Sagittaire"
Encoding with x264.exe
See Annexe for complete profil
Video stream are compliant with HDDVD "NTSC" profil
MPEG2 encoding by "Sagittaire"
Encoding with mencoder.exe from libavcodec MPEG2
See Annexe for complete profil
Video stream are compliant with HDDVD "NTSC" profil
D - Conclusion
At this time for this movie
1) I use certainely the best MPEG2 encoder available in the area for metric: Libavcodec. The result is simply impressive for the old MPEG2 codec. With PP4 the result is even better. Libavcodec produce by far much better metric result here than all the other implementations (HCEnc, TMPGEnc, Mainconcept, Procoder ...).
2) VC1 is a very good surprise for me. For "low/medium bitrate" (high quantisation level) VC1 in unable to fight with H264 but for "High bitrate" the SSIM for VC1 is close to H264. MS say "VC1 is particulary optimized for high bitrate with high resolution" and it's true. Anyway MS claim that VC1 is able to produce the same metric that H264 but it's clearly false here.
3) H264 is simply the best for metric in all situation and particulary for "low bitrate" situation. Anyway differences between VC1 and H264 are not so high for low quantisation encoding.
4) VC1 and H264 at 6/20 Mbps (average bitrate/max bitrate) are unable to produce better quality than MPEG2 at 12/24 Mbps. H264 at 12/24 Mbps produce better quality than MPEG2 at 18/28 Mbps but not VC1. H264 at 12/24 Mbps is unable to produce better quality than VC1 at 18/28 Mbps. MicroSoft annonce (http://www.microsoft.com/windows/windowsmedia/howto/articles/vc1techoverview.aspx#VC1ComparedtoOtherCodecs) ratio between 1:2 and 1:3 for MPEG2 vs VC1 but it's really not the case here. The x264 particularly optimized in its development to obtain very good PSNR is not able to obtain a ratio of 2:1 with MPEG2 in this challenge.
5) Use constrained vbv rate control produce very inconstant quality and use overall or average metric results in this case is very difficult because complex part (motion and/or texture) can produce very bad results for metric. For better results it will be better to use metric graph for part by part comparison. If a codec want produce good overall/average metric here then the codec must have particulary good vbv Rate Control optimization.
IF YOU DON'T AGREE WITH THAT, TRY WITH YOUR ENCODER ... !!!