Sagittaire
19th August 2006, 13:44
~ HD-DVD Benchmark Challenge ~
You think that your standard is the best : prove that ... !!?
A - Introduction
The purpose of this challenge is to determine which is the best codec format encoding.
HD-DVD & BD can use MPEG2, H264 and VC-1 for video codec.
B - Rules
Rule 1 : Source
Uncompressed source is available Here (http://orange.blender.org/blog/original-lossless-source-available/) for reproduce the test.
Open source Elephant Dream movie, video 1920*1080 PNG lossless, audio 5.1 Flac lossless, 15 691 frames
Rule 2 : Pre-process and encoding
You must use this avisynth script for make encoding:
Source=ImageSource("C:\LossLess\images\%05d.png", start=1, end=15691, fps=23.976)
Source=ConvertToYV12(source, matrix="Rec709")
return source
Other pre-process is not autorized. It's possible to propose example with high quality pre-process (objective like high color conversion or subjective like sharp process ... ect) but just for demonstration and not for this challenge.
Video Stream must be like HD-DVD compliant stream. You can use all the compliant audio codec and all the compliant bitrate for audio. This test is just video challenge and not audio challenge. All the video stream must be compliant with these setting:
MPEG2 Encoding
Profil & Level: MP@HL except specific restrictions
Max GOP lenght: 14 frames
Maximum bitrate: 20.0 Mbps, 24.0 Mbps and 28.0 Mbps
Buffer size: 9781 Kbits for principal HD video stream
Horizontal Vector Range: +/- 1024 pixels
Vertical Vector Range: +/- 128 pixels
Other Restrictions Setting: max adaptative GOP at 14, max adaptative bframe at 2
VC-1 Encoding
Profil & Level: AP@L3 except specific restrictions
Max GOP lenght: 14 frames
Maximum bitrate: 20.0 Mbps, 24.0 Mbps and 28.0 Mbps
Buffer size: 14745 Kbits for principal HD video stream
Horizontal Vector Range: +/- 1024 pixels
Vertical Vector Range: +/- 256 pixels
Other Restrictions Setting: max adaptative GOP at 14, max adaptative bframe at 2
H264 Encoding
Profil & Level: HP@L4.1 except specific restrictions
Max GOP lenght: 14 frames
Maximum bitrate: 20.0 Mbps, 24.0 Mbps and 28.0 Mbps
Buffer size: 14745 Kbits for principal HD video stream
Horizontal Vector Range: +/- 1024 pixels
Vertical Vector Range: +/- 512 pixels
Other Restrictions Setting: max adaptative GOP at 14, max adaptative bframe at 2,
Max reference at 4, Max breference at 3, no film grain modeling
Rule 3 : Bitrate and Max Bitrate
You must use these bitrate/size for encoding:
HD-DVD with "super bitrate" video stream and simple HDDVD authoring:
18 Mbps (Max at 28.0 Mbps) for video stream with +/- 0.5 % for bitrate tolerance
HD-DVD with "medium bitrate" video stream and standard HDDVD authoring:
12 Mbps (Max at 24.0 Mbps) for video stream with +/- 0.5 % for bitrate tolerance
HD-DVD with "low bitrate" video stream and standard HDDVD authoring:
6 Mbps (Max at 20.0 Mbps) for video stream with +/- 0.5 % for bitrate tolerance
NB: we can use HD-DVD structure too on simple DVD DL 12 cm at 8.5 GB
Rule 4 : Visual Tests
Here you can find encoding in ifo & evo structure for subjective test ... try to find yourself the best quality:
HD-DVD & BD MPEG2 at 18 Mbps :
File: not available
HD-DVD & BD MPEG2 at 12 Mbps :
File: not available
HD-DVD & BD MPEG2 at 6 Mbps :
File: not available
HD-DVD & BD VC-1 at 18 Mbps :
File: not available
HD-DVD & BD VC-1 at 12 Mbps :
File: not available
HD-DVD & BD VC-1 at 6 Mbps :
File: not available
HD-DVD & BD H264 at 18 Mbps :
File: not available
HD-DVD & BD H264 at 12 Mbps :
File: not available
HD-DVD & BD H264 at 6 Mbps :
File: not available
Rule 5 : Metrics Tests
SSIM (YV12): IMO the best metric. it uses some HVS properties but not all ...
PDF description for VSSIM (http://www.cns.nyu.edu/~zwang/files/papers/vssim.pdf)
Overall PSNR (YV12): Very good metric test, particullary for Rate Control Quality test ...
MSE = Sum t = 1 -> NFrame Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (NFrame*width*height)
Overall PSNR = 10 * Log ( 255² / MSE)
Average PSNR (YV12):The worse metric test but Frame PSNR graph is very good to compare RC strategy ...
MSE = Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (width*height)
Average PSNR = Sum t = 1 -> NFrame 10 * Log ( 255² / MSE) / Nframe
SSIM vs PSNR
"For a given reference video and a given compressed video, it is meant to compute a quality metric, based on perceived visual distortion. Unlike the well-known PSNR measure, it's not purely mathematical, and should correlate much better with human vision."
"A higher MSE (and so lower PSNR) should mean that the compressed clip is a worse image but MSE and PSNR are flawed in this respect as numerous tests have shown. However with SSIM, according to tests carried out on the VQEG dataset, a higher Q (SSIM value) has a much better relation to the visual quality of the compressed clip. Despite this, bear in mind the SSIM metric still isn't perfect."
Here (http://multimediacom.free.fr/Download/SSIM.rar) last AviSynth plugin for VSSIM in YV12
Average PSNR vs Overall PSNR
It's easy to increase artificially Average PSNR with Rate Control tweak. In certain case CBR (0% for variability) will be better than real VBR (100% for variability or constant quant). It's generaly not possible with Overall PSNR. Overall PSNR is very good to test Rate Control quality. If difference between Average PSNR and Overall PSNR is very high, "local quality difference" (difference between Pframe or Bframe for exemple) or "scene quality difference" (difference between low motion and high motion for exemple) will be too high and Rate Control for this codec not very good.
Here (http://multimediacom.free.fr/Download/CompareYV12.zip) last AviSynth plugin for Overall PSNR in YV12
Open your avs metric script with VirtualDub. Select file > preview input for run the test. Select file > close video file for obtain final result in log metric files. You must use these AviSynth type script for metric test:
# --> Source Opening <--
Source=ImageSource("C:\LossLess\images\%05d.png", start=1, end=15691, fps=24)
Source=ConvertToYV12(source, matrix="Rec709")
# --> Video Opening <--
video=Mpeg2Source("D:\...\MPEG2_24Mbps.d2v",idct=2)
# --> PSNR analysis <--
compareYV12(video,source,"YUV","OPSNR_MPEG2_24Mbps.log")
# --> Source Opening <--
Source=ImageSource("C:\LossLess\images\%05d.png", start=1, end=15691, fps=24)
Source=ConvertToYV12(source, matrix="Rec709")
# --> Video Opening <--
video=Mpeg2Source("D:\...\MPEG2_24Mbps.d2v",idct=2)
# --> SSIM analysis <--
return SSIM(source,video,"results.csv","SSIM_MPEG2_24Mbps.txt",lumimask=2)
C - Results
At this time here the best result ... but done better if you can ... :devil:
|--------------|---------|---------|---------|---------|---------|
| Codec | PProc | Bitrate | Size | OPSNR | SSIM 2 |
|--------------|---------|---------|---------|---------|---------|
| MPEG2 | PP4 | 0000 | 0000 | 00.00 | 00.00 |
| VC-1 | PP1 | 0000 | 0000 | 00.00 | 00.00 |
| H264 | PP0 | 0000 | 0000 | 00.00 | 00.00 |
|--------------|---------|---------|---------|---------|---------|
| MPEG2 | PP4 | 00000 | 0000 | 00.00 | 00.00 |
| VC-1 | PP1 | 00000 | 0000 | 00.00 | 00.00 |
| H264 | PP0 | 00000 | 0000 | 00.00 | 00.00 |
|--------------|---------|---------|---------|---------|---------|
| MPEG2 | PP4 | 00000 | 0000 | 00.00 | 00.00 |
| VC-1 | PP1 | 00000 | 0000 | 00.00 | 00.00 |
| H264 | PP0 | 00000 | 0000 | 00.00 | 00.00 |
|--------------|---------|---------|---------|---------|---------|
SSIM 0: Lumimask Off
SSIM 1: Lumimask On (Original Lumimask)
SSIM 2: Lumimask On (One2Tech Patch)
D - Conclusion
At this time for this movie
THE BEST IS ??????????
IF YOU DON'T AGREE WITH THAT, TRY WITH YOUR ENCODER ... !!!
Annexe - Update
01.07.2007 - NTSC setting for encoding replace PAL setting
26.10.2006 - New rules for more real HDDVD encoding (new bitrate, new vbv)
03.10.2006 - buffer for H264 is confidential but use 14400 Kbits is certainely compliant with HDDVD
02.10.2006 - official buffer for VC1 is 14400 Kbits or 480 ms for better HD-DVD compliancy
26.09.2006 - MPEG2 and H264 files with audio are available
19.09.2006 - MPEG2 metric result
18.09.2006 - H264 metric result
will coming ... if you want ... !!!
All the developpers are wellcome ... !!?
You think that your standard is the best : prove that ... !!?
A - Introduction
The purpose of this challenge is to determine which is the best codec format encoding.
HD-DVD & BD can use MPEG2, H264 and VC-1 for video codec.
B - Rules
Rule 1 : Source
Uncompressed source is available Here (http://orange.blender.org/blog/original-lossless-source-available/) for reproduce the test.
Open source Elephant Dream movie, video 1920*1080 PNG lossless, audio 5.1 Flac lossless, 15 691 frames
Rule 2 : Pre-process and encoding
You must use this avisynth script for make encoding:
Source=ImageSource("C:\LossLess\images\%05d.png", start=1, end=15691, fps=23.976)
Source=ConvertToYV12(source, matrix="Rec709")
return source
Other pre-process is not autorized. It's possible to propose example with high quality pre-process (objective like high color conversion or subjective like sharp process ... ect) but just for demonstration and not for this challenge.
Video Stream must be like HD-DVD compliant stream. You can use all the compliant audio codec and all the compliant bitrate for audio. This test is just video challenge and not audio challenge. All the video stream must be compliant with these setting:
MPEG2 Encoding
Profil & Level: MP@HL except specific restrictions
Max GOP lenght: 14 frames
Maximum bitrate: 20.0 Mbps, 24.0 Mbps and 28.0 Mbps
Buffer size: 9781 Kbits for principal HD video stream
Horizontal Vector Range: +/- 1024 pixels
Vertical Vector Range: +/- 128 pixels
Other Restrictions Setting: max adaptative GOP at 14, max adaptative bframe at 2
VC-1 Encoding
Profil & Level: AP@L3 except specific restrictions
Max GOP lenght: 14 frames
Maximum bitrate: 20.0 Mbps, 24.0 Mbps and 28.0 Mbps
Buffer size: 14745 Kbits for principal HD video stream
Horizontal Vector Range: +/- 1024 pixels
Vertical Vector Range: +/- 256 pixels
Other Restrictions Setting: max adaptative GOP at 14, max adaptative bframe at 2
H264 Encoding
Profil & Level: HP@L4.1 except specific restrictions
Max GOP lenght: 14 frames
Maximum bitrate: 20.0 Mbps, 24.0 Mbps and 28.0 Mbps
Buffer size: 14745 Kbits for principal HD video stream
Horizontal Vector Range: +/- 1024 pixels
Vertical Vector Range: +/- 512 pixels
Other Restrictions Setting: max adaptative GOP at 14, max adaptative bframe at 2,
Max reference at 4, Max breference at 3, no film grain modeling
Rule 3 : Bitrate and Max Bitrate
You must use these bitrate/size for encoding:
HD-DVD with "super bitrate" video stream and simple HDDVD authoring:
18 Mbps (Max at 28.0 Mbps) for video stream with +/- 0.5 % for bitrate tolerance
HD-DVD with "medium bitrate" video stream and standard HDDVD authoring:
12 Mbps (Max at 24.0 Mbps) for video stream with +/- 0.5 % for bitrate tolerance
HD-DVD with "low bitrate" video stream and standard HDDVD authoring:
6 Mbps (Max at 20.0 Mbps) for video stream with +/- 0.5 % for bitrate tolerance
NB: we can use HD-DVD structure too on simple DVD DL 12 cm at 8.5 GB
Rule 4 : Visual Tests
Here you can find encoding in ifo & evo structure for subjective test ... try to find yourself the best quality:
HD-DVD & BD MPEG2 at 18 Mbps :
File: not available
HD-DVD & BD MPEG2 at 12 Mbps :
File: not available
HD-DVD & BD MPEG2 at 6 Mbps :
File: not available
HD-DVD & BD VC-1 at 18 Mbps :
File: not available
HD-DVD & BD VC-1 at 12 Mbps :
File: not available
HD-DVD & BD VC-1 at 6 Mbps :
File: not available
HD-DVD & BD H264 at 18 Mbps :
File: not available
HD-DVD & BD H264 at 12 Mbps :
File: not available
HD-DVD & BD H264 at 6 Mbps :
File: not available
Rule 5 : Metrics Tests
SSIM (YV12): IMO the best metric. it uses some HVS properties but not all ...
PDF description for VSSIM (http://www.cns.nyu.edu/~zwang/files/papers/vssim.pdf)
Overall PSNR (YV12): Very good metric test, particullary for Rate Control Quality test ...
MSE = Sum t = 1 -> NFrame Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (NFrame*width*height)
Overall PSNR = 10 * Log ( 255² / MSE)
Average PSNR (YV12):The worse metric test but Frame PSNR graph is very good to compare RC strategy ...
MSE = Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (width*height)
Average PSNR = Sum t = 1 -> NFrame 10 * Log ( 255² / MSE) / Nframe
SSIM vs PSNR
"For a given reference video and a given compressed video, it is meant to compute a quality metric, based on perceived visual distortion. Unlike the well-known PSNR measure, it's not purely mathematical, and should correlate much better with human vision."
"A higher MSE (and so lower PSNR) should mean that the compressed clip is a worse image but MSE and PSNR are flawed in this respect as numerous tests have shown. However with SSIM, according to tests carried out on the VQEG dataset, a higher Q (SSIM value) has a much better relation to the visual quality of the compressed clip. Despite this, bear in mind the SSIM metric still isn't perfect."
Here (http://multimediacom.free.fr/Download/SSIM.rar) last AviSynth plugin for VSSIM in YV12
Average PSNR vs Overall PSNR
It's easy to increase artificially Average PSNR with Rate Control tweak. In certain case CBR (0% for variability) will be better than real VBR (100% for variability or constant quant). It's generaly not possible with Overall PSNR. Overall PSNR is very good to test Rate Control quality. If difference between Average PSNR and Overall PSNR is very high, "local quality difference" (difference between Pframe or Bframe for exemple) or "scene quality difference" (difference between low motion and high motion for exemple) will be too high and Rate Control for this codec not very good.
Here (http://multimediacom.free.fr/Download/CompareYV12.zip) last AviSynth plugin for Overall PSNR in YV12
Open your avs metric script with VirtualDub. Select file > preview input for run the test. Select file > close video file for obtain final result in log metric files. You must use these AviSynth type script for metric test:
# --> Source Opening <--
Source=ImageSource("C:\LossLess\images\%05d.png", start=1, end=15691, fps=24)
Source=ConvertToYV12(source, matrix="Rec709")
# --> Video Opening <--
video=Mpeg2Source("D:\...\MPEG2_24Mbps.d2v",idct=2)
# --> PSNR analysis <--
compareYV12(video,source,"YUV","OPSNR_MPEG2_24Mbps.log")
# --> Source Opening <--
Source=ImageSource("C:\LossLess\images\%05d.png", start=1, end=15691, fps=24)
Source=ConvertToYV12(source, matrix="Rec709")
# --> Video Opening <--
video=Mpeg2Source("D:\...\MPEG2_24Mbps.d2v",idct=2)
# --> SSIM analysis <--
return SSIM(source,video,"results.csv","SSIM_MPEG2_24Mbps.txt",lumimask=2)
C - Results
At this time here the best result ... but done better if you can ... :devil:
|--------------|---------|---------|---------|---------|---------|
| Codec | PProc | Bitrate | Size | OPSNR | SSIM 2 |
|--------------|---------|---------|---------|---------|---------|
| MPEG2 | PP4 | 0000 | 0000 | 00.00 | 00.00 |
| VC-1 | PP1 | 0000 | 0000 | 00.00 | 00.00 |
| H264 | PP0 | 0000 | 0000 | 00.00 | 00.00 |
|--------------|---------|---------|---------|---------|---------|
| MPEG2 | PP4 | 00000 | 0000 | 00.00 | 00.00 |
| VC-1 | PP1 | 00000 | 0000 | 00.00 | 00.00 |
| H264 | PP0 | 00000 | 0000 | 00.00 | 00.00 |
|--------------|---------|---------|---------|---------|---------|
| MPEG2 | PP4 | 00000 | 0000 | 00.00 | 00.00 |
| VC-1 | PP1 | 00000 | 0000 | 00.00 | 00.00 |
| H264 | PP0 | 00000 | 0000 | 00.00 | 00.00 |
|--------------|---------|---------|---------|---------|---------|
SSIM 0: Lumimask Off
SSIM 1: Lumimask On (Original Lumimask)
SSIM 2: Lumimask On (One2Tech Patch)
D - Conclusion
At this time for this movie
THE BEST IS ??????????
IF YOU DON'T AGREE WITH THAT, TRY WITH YOUR ENCODER ... !!!
Annexe - Update
01.07.2007 - NTSC setting for encoding replace PAL setting
26.10.2006 - New rules for more real HDDVD encoding (new bitrate, new vbv)
03.10.2006 - buffer for H264 is confidential but use 14400 Kbits is certainely compliant with HDDVD
02.10.2006 - official buffer for VC1 is 14400 Kbits or 480 ms for better HD-DVD compliancy
26.09.2006 - MPEG2 and H264 files with audio are available
19.09.2006 - MPEG2 metric result
18.09.2006 - H264 metric result
will coming ... if you want ... !!!
All the developpers are wellcome ... !!?