[Challenge Medium Bitrate] More efficient MPEG2 Encoder for objectives test

[Sagittaire]

~ Movie Metric Benchmark Challenge ~


A - Introduction

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...



B - Rules

Rule 1 : Source

Source is available Here for reproduce the test : memoirsofageisha_1080p.mov, 1920*900 MPEG4 AVC MP@L4 10 Mbps


Rule 2 : Pre-process and encoding

You must use ffdshow without post-process for MPEG4 AVC decoding in avisynth script. You must use MPEG2 for encoding with DVD standard MPEG2 MP@ML Pal DAR 16/9 PAR 16:11 profil. You must use this avisynth script for make encoding:

clip = DirectShowSource("C:\...\memoirsofageisha_1080p.mov", fps=23.976)
clip = AssumeFPS(clip, 25)
clip = Trim(clip, 100, 0) 
clip = Crop(clip, 4, 4, -4, -4)
clip = lanczos4resize(clip, 720, 496)
clip = addborders(clip, 0, 40, 0, 40)
clip = Limiter(clip) 
Return clip

Rule 3 : Bitrate

You must use these bitrate/size for encoding:

The second round use medium bitrate for MPEG2 : maximum bitrate is 2930 Kbps for DVDR5 profil encoding.
DVD5 Profil: Maximun 2930 Kbps for Video Elementary Stream or 50 850 Ko for m2v file.


Rule 4 : Metric Tests

SSIM (YV12): IMO the best metric. it uses some HVS properties but not all ...
PDF description for VSSIM

Overall PSNR (YV12): Very good metric test, particullary for Rate Control Quality test ...
MSE = Sum t = 1 -> NFrame Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (NFrame*width*height)
Overall PSNR = 10 * Log ( 255² / MSE)

Average PSNR (YV12):The worse metric test but Frame PSNR graph is very good to compare RC strategy ...
MSE = Sum x = 1 -> width Sum y = 1 -> height (F'(x,y,t)-F(x,y,t))² / (width*height)
Average PSNR = Sum t = 1 -> NFrame 10 * Log ( 255² / MSE) / Nframe


SSIM vs PSNR

"For a given reference video and a given compressed video, it is meant to compute a quality metric, based on perceived visual distortion. Unlike the well-known PSNR measure, it's not purely mathematical, and should correlate much better with human vision."

"A higher MSE (and so lower PSNR) should mean that the compressed clip is a worse image but MSE and PSNR are flawed in this respect as numerous tests have shown. However with SSIM, according to tests carried out on the VQEG dataset, a higher Q (SSIM value) has a much better relation to the visual quality of the compressed clip. Despite this, bear in mind the SSIM metric still isn't perfect."

Here last AviSynth plugin for VSSIM in YV12


Average PSNR vs Overall PSNR

It's easy to increase artificially Average PSNR with Rate Control tweak. In certain case CBR (0% for variability) will be better than real VBR (100% for variability or constant quant). It's generaly not possible with Overall PSNR. Overall PSNR is very good to test Rate Control quality. If difference between Average PSNR and Overall PSNR is very high, "local quality difference" (difference between Pframe or Bframe for exemple) or "scene quality difference" (difference between low motion and high motion for exemple) will be too high and Rate Control for this codec not very good.

Here last AviSynth plugin for Overall PSNR in YV12


Open your avs metric script with VirtualDub. Select file > preview input for run the test. Select file > close video file for obtain final result in log metric files. You must use these AviSynth type script for metric test:

# --> Source Opening <--
source=Mpeg2Source("G:\...\HPII.d2v",idct=2)
source=Trim(source,70,3145)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)

# --> Video Opening <--
video=DirectShowSource("G:\...\DivX6-450.avi",fps=25)

# --> PSNR analysis <--
compareYV12(video,source,"YUV","PSNR-DivX6-450.log")

# --> Source Opening <--
source=Mpeg2Source("G:\...\HPII.d2v",idct=2)
source=Trim(source,70,3144)
source=Crop(source,0,76,-0,-76)
source=LanczosResize(source,720,304)

# --> Video Opening <--
video=DirectShowSource("G:\...\RV10-450.mkv",fps=25)

# --> SSIM analysis <--
return SSIM(source,video,"results.csv","SSIM-RV10-450.txt",lumimask=2)

C - Results

At this time here the best result ... but done better if you can ...

|--------------|---------|---------|---------|---------|
| Codec        | Bitrate |   Size  |  OPSNR  |  SSIM 2 |
|--------------|---------|---------|---------|---------|
| TMPGEnc      |  2930   |  50847  |  44.66  |  83.95  |
| CCE          |  2930   |  50839  |  44.07  |  81.01  |
| Mencoder     |  2930   |  50805  |  45.40  |  85.36  |
| HCEnc        |  2930   |  50823  |  44.72  |  82.99  |
|--------------|---------|---------|---------|---------|
SSIM 0: Lumimask Off
SSIM 2: Lumimask On (One2Tech Patch)

D - Subjective "Blind Tests"

Here you can find encoding in ifo & vob structure for very little blind test:

Encoding A :
File: MPEG2 MP@ML 2930 Kbps

Encoding B :
File: MPEG2 MP@ML 2930 Kbps

Encoding C :
File: MPEG2 MP@ML 2930 Kbps

Encoding D :
File: MPEG2 MP@ML 2930 Kbps

You can vote for your prefered encoding with 1 points for the first, 2 points for the second, 3 points for the third and 4 points for the last. The encodind with the lower final score will be the winner of this very little "blind test".


E - Conclusion

At this time for this trailer

THE BEST FOR METRICS IS MENCODER

IF YOU DON'T AGREE WITH THAT, TRY WITH YOUR ENCODER ... !!!


Annexe - Update

will coming ... if you want ... !!!

[Guest]

>Best MPEG2 Encoder for DVDR5 encoding
>You think that your MPEG2 coder is the best : prove that ... !!?
>The purpose of this challenge is to determine which is the best codec...

Forum rule 12: Do not ask what's best.

You can say "Preferred" but not "Best".

[Sagittaire]

Originally Posted by neuron2

>Best MPEG2 Encoder for DVDR5 encoding
>You think that your MPEG2 coder is the best : prove that ... !!?
>The purpose of this challenge is to determine which is the best codec...

Forum rule 12: Do not ask what's best.

You can say "Preferred" but not "Best".

No ... rules 12 is perfectly respected... !!!

... because it's only objectif metric challenge ... there ar not subjective judgement in this test. read completely:

The purpose of this challenge is to determine which is the best codec for the metrics and only for the metrics : this test will not speak about subjective visual quality. If you want subjective visual comparison download sample and compare yourself ...

Metric test is very simply: best codec (for metric and only for metric) obtain higher result for metric. It's like for example lossless codec comparison : the more efficient codec obtain simply the lower final size/bitrate ... !!!

Mencoder is in this case (with this trailer, with this resolution, with this bitrate, with this avs script ... etc etc etc) the best for metric and not my prefered MPEG2 codec ... !!!

[Guest]

In that case, replace the generic use of "best" with something like "highest PSNR" or whatever. Your title, for example, refers only to "Best MPEG2 Encoder". It violates the rules. Please fix it.

[Sagittaire]

Originally Posted by neuron2

In that case, replace the generic use of "best" with something like "highest PSNR" or whatever. Your title, for example, refers only to "Best MPEG2 Encoder". It violates the rules. Please fix it.

Yes ... Perhabs ...

[Challenge Medium Bitrate] More efficient MPEG2 Encoder for objectives tests

[Guest]

That's perfect. Thank you very much. Now for some testing...!

[LauriM]

Sagittaire, I like your MPEG2 codec tests.

After your previous one I took a good look at MEncoder, which (as a Linux user) I really like. There's just one problem that I'd like to point out - perhaps someone here can figure out what's going on and even help writing a good bugreport.

MEncoder (with libavcodec) cannot handle high quality encodes from material with film grain. Noisy gradients in the source material seem to make the result always blocky. I tried some encodes with MEncoder and xvid to make sure it's not a problem with MEncoder itself, but with the MPEG2 codec (and perhaps MPEG4) from libavcodec.

There are a few samples at http://koti.welho.com/lmyllari/dvdtest/. The sample bitstreams are really short, but at low fps (try 3 for example) the differences are clear. The original (-ovc copy) obviously looks good, MPEG2/MPEG4 with libavcodec is quite blocky in gradients and loses the noise, Xvid with mpeg quant is quite good with occasional blocks.

I also tried with mpeg2enc and it can handle the source - although at high bitrate. It lacks 2pass, so I don't think it can really be used for dvd backups.

I only looked at very high quality settings here, but noticed the problem originally with a 2pass encode at "normal" bitrate. I thought that I didn't allow for high enough quant/bitrate.

Any insight would be most welcome. And sorry for going a bit offtopic here, but I thought this might be of interest to you and your readers.

[Audionut]

Sagittaire, SSIM not available from your link.

Will have results soon.

[Mug Funky]

@ LauriM:

try a higher bitrate matrix. the global quants will be up a bit, but it seems to come out better for me. though i've never seen anything that encodes grain like an MPX3000. too bad they have sucky ratecontrol and aren't made anymore.

[Sagittaire]

Originally Posted by Audionut

Sagittaire, SSIM not available from your link.

Will have results soon.

fixed ...

[IgorC]

Sagitarrie. But if I'll donwload videos and cheks PSNR and SSIM. I will know what encoder is.
I won't do that. And I will post my blind results.
what about Poll? Or you think it's idea with opsnr >0.01

[FlashBlade]

I was wondering if we could know what encoders are behind the blind tests A, B, C and D, now.
Especially A, actually.

[Sagittaire]

Encoding A : CCE
Encoding B : HCEnc
Encoding C : Mencoder
Encoding D : TMPGEnc

[mezzanine]

Sagitaire could you please post the cli options you used for mencoder to get these results?
Thanks