Codec Comparison for HD-Content

[Stavr0s]

Hi all,

Currently I'm preparing (work-related) a 'video coding standard' shoot-out for High Definition content. I've got a pretty clear picture of the basic setup, but I need to be sure that all the test encodings are 100% spec-compliant, and for some things I could really need some suggestions.

Sequences : all sequences are 1920x1080/24p ( SMTPE-sequences, so movie-material (no anime) ). Originally in DPX-format, but converted to 8-bit RGB raw AVI. For each AVI there is an avisynth script that serves as input for each codec.
AviSource("filename.avi")
AssumeFPS(24)
ConvertToYV12(matrix="Rec709")

Standards : VC-1, H.264, MJPEG-2000 and (as an old reference) MPEG-2 (once in Intra- and once in Inter-mode)

Bitrates : vary from low to extremely high (part of the results are used for an archival purposes), 2/3/5/8/12/16/32/64/100 Mbps.

Tests : PSNR (has to be included), OPSNR, and I'm also trying to squeeze in SSIM. A few months ago I took a look at the MSU tool and it seemed decent so I think I'm going to use that one (in conjunction with a DirectShowSource()-avisynth script, all decoding is handled through ffdshow). Later on, subjective visual test are going to be held.

H.264 : Used codec x264. 2-pass mode (1st pass TURBO)
High profile, Unrestricted Level
Deblocking enabled (0:0)
ME : UMH (range 16)
Subpixel refinement : 7 (RDO level 2)
CABAC
8 reference frames
All MB options + 8x8dct (FRext is allowed)
3 Bframes, adaptive, hierarchical, RDO
Keyframe interval : 48 (min-max)
This last one I cannot change, for VC-1 a key-frame interval of 2 secs has to be used and the basic GOP-structure for both codecs has to be alike. So it's even possible I have to disable adaptive B-frames. Output-container is MP4.
What I want to know : can Trellis encoding and Mixed Reference Frames be used, or are they not spec-compliant. Also, the number of reference frames is kind of a shot in the dark.

VC-1 : Used codec WMV. With the current release of WMP11/Beta this might change to WMV-AP (this is still part of the VC-1 spec I hope?). If I plan it right the VC-1 encodings can be delayed a couple of weeks and I can check out WM Encoder Studio. Otherwise I think I'll make use of Nic's WM Encoder. Also, I've recently discovered that the WMV-codec is actually 'customizable' (num. of Bframes, Lookahead, ...), but I'm not sure what their values should be to achieve maximum quality (old WMV doesn't use B-frames I think, is there a reason for this?). Output-container is WMV.

MPEG-2 : This one is giving me a headache. The only affordable MP@HL (needed for HD sequences) MPEG2-encoder I found was TMPGEnc plus 2.5. Unfortunatly it crashes after a while (can't remember the exact error, something about low-memory, even when I was using 'cache first pass on disk' or something like that). If anyone knows a MPEG-2 coder which can do the job i'd appreciate it a lot.

I'm aware that this is a big post. Thanks to the people who are still reading this. Of course when results are floating in I'll keep you updated.

Steven

[Golgot13]

Hi,

nice, for your test, you can use a matrice to optimize x264 encoded.

Because when you use VC1 from other software or hardware of microsoft, the quality is not same.
I think (to my mind) the VC1 from microsoft software is optimize with some filter to be best.

I sure that x264 will win at same bitrate. You can contact me to do best comparaison.



Golgot13

[GodofaGap]

Originally in DPX-format, but converted to 8-bit RGB raw AVI.
I hope you mean 24 bit RGB (8 bit per channel) :)


This last one I cannot change, for VC-1 a key-frame interval of 2 secs has to be used and the basic GOP-structure for both codecs has to be alike.
This is not true, with Nic WMenc you can set intervals for VC-1 anyway you like.

What I want to know : can Trellis encoding and Mixed Reference Frames be used, or are they not spec-compliant.
Well trellis is just an optimisation technique that doesn't need to be in the standard (like RDO). Mixed refs is in the standard. So you can safely use both.

Also, the number of reference frames is kind of a shot in the dark.
If you aim for quality only, use 16. If you aim for a tradeoff between speed and quality 8 is probably already too high.

[Stavr0s]

@Golgot13
I'm not going to use custom matrices. The strategy is to make a general comparison between basic 'standards' only, not possible features integrated in implementations of those standards. I do realize that I will NOT achieve maximum quality at each bitrate if I keep all settings between different bitrates identical. I'm just an 'advisor' in this comparison-project, I will not carry out the encodings myself, nor can I decide everything in this project myself.
Encoding will happen on the original AVI's without filtering (hence the use of AviSource i.s.o. DirectShowSource). Yes, this will also hurt quality at lower bitrates. Again, I'm not going to compare different bitrates within one standard, but rather different standards for one bitrate.
I sure that x264 will win at same bitrate.
I have the same feeling about this (except maybe MJPEG-2000 at high bitrates).

@GodofaGap
Never heard of 8-bit HD video? Great resolution, crappy colors :). You're right of course, they're 24-bit.
This is not true, with Nic WMenc you can set intervals for VC-1 anyway you like.
Yes that's true, but what I meant was that those '2 secs' is something I did not decide. The comparison is also made for several national Broadcasters, and they tend to panic if not everything is Intra-coded, so 2 seconds is already a lot.
Well trellis is just an optimisation technique that doesn't need to be in the standard (like RDO). Mixed refs is in the standard. So you can safely use both.
I don't really know the details of Trellis Quantization. It's some kind of vector quantization, but I haven't found a clear explanation yet. It reminds me of Viterbi, but then again I'm not sure if it's the same thing. Thanks for the info.
If you aim for quality only, use 16. If you aim for a tradeoff between speed and quality 8 is probably already too high.
The research is indeed mostly aimed at quality. I do however plan to log the encoding times and mention them as a small 'side-remark'.

Thanks again.
Steven

[Golgot13]

I have the same feeling about this (except maybe MJPEG-2000 at high bitrates).


I know it is use in new format DCI for cinema 2K and 4K.
The difference is if there is not enough bitrate for encoding video, the picture
will be blur (in MPEG we will see the bloc of DCT).

But I think H264 with same bitrate than DCI in JPEG2000 will be same quality in 2K resolution (H264 support it :) ).
There is not hardware chipset to play H264 at this resolution now...


Golgot13

[foxyshadis]

x264: Ref 4-6 is great for normal encoding, 8 for slightly higher quality, 10 for squeezing blood out of a turnip, anything higher than that is generally a waste of a lot of time. 6 or 8 is a great tradeoff, especially if it'll be used for broadcast.

VC-1: Now has a special registry preset "Compression Optimization Type" that if you set to 1, enables all quality options. That includes inloop, which isn't present in versions before this current beta.

MPEG-2: Maybe you can make this the incentive for Hank to release an HD version of HC? =p

[GodofaGap]

A note for VC1: Compression Optimization Type can be PSNR wise a very bad option. If PSNR plays a significant role in the comparision, use it with caution.

[Manao]

Custom matrices will reduce the PSNR, and most likely the SSIM, so if you compare only using those, don't use custom matrices. MJpeg2k won't beat h264. H264 intra only is already better than mjpeg2k... Coding efficiency isn't the only reason for choosing a codec, and cinemas have others constraints ( one of them being : being able to be decoded ).

Stavr0s : why the fixed gop structure ? I can understand the 2 seconds gop size, but why want to fix the bframes place ?

Mencoder is a fairly good mpeg2 encoder, and should support HD.

Finally, as soon as you mention speed - even as a sidenote - you should use "sane" settings. And, imho, 8 references is far from been sane. I'd 4. At most.

[zambelli]

I'd recommend trying both Main Profile and Advanced Profile for WMV9/VC-1. It'll make for an interesting comparison.

Enabling "Compression Optimization Type" and "Force LoopFilter" will probably help on lower bitrate encodes, but it will likely hurt the PSNR on higher bitrate encodes.

For best quality, try enabling these:
"Motion Search Range" = -1
"Motion Match Method" = 1 or -1
"Motion Search Level" = 2 or -1

Set encoder complexity to 3 or 4 (60 or 80 in -v_performance terms). 5 (100) is probably not worth the performance hit.

B frame distance and I frame distance should match whatever the other codecs are using.

[Stavr0s]

@Zambelli
Thanks for your advice, but is there that much of a difference between VC-1 MP and AP? I thought the most important addition was the support of interlaced encoding (which I don't need since all sequences are 24p), and higher bitrates allowed.

@Manao
You're probably right about the GOP-structure, as long as I keep the size more or less the same. It's just that I didn't want to end up comparing a 5Mbps WMV with no B-frames at all to a 5Mbps x264 with 3 consecutive B-frames.

After all 'numerical' comparisons (PSNR, etc..) some visual tests should follow. Maybe it's interesting to do something like MSU's Video Codecs Comparison and calculate correlations between the subjective opinion scores and the scientific measures.

[zambelli]

@Zambelli
Thanks for your advice, but is there that much of a difference between VC-1 MP and AP? I thought the most important addition was the support of interlaced encoding (which I don't need since all sequences are 24p), and higher bitrates allowed.

I haven't read the whole VC-1 spec, but I'm guessing there might be other differences. There's one sure way to find out though! :)

[benwaggoner]

I haven't read the whole VC-1 spec, but I'm guessing there might be other differences. There's one sure way to find out though! :)
Beyond stuff like metadata and a elementary stream format, Interlaced support is the big difference.

[Stavr0s]

A bump to this thread to inform you that the results are (almost) in. From what I've seen (which is only a small part) H.264 is in the lead, followed by VC-1.
What is remarkable, is that MJPEG-2000 (purely intra-based) seems to perform better than MPEG-2 (IBBPBBP...) at sequences with low bitrates (provided there is not too much movement). If these sequences are scaled down to regular SD-resolution MPEG-2 performs better than MJPEG-2000. Anyone got any comments about this?

[Manao]

Yep, what mpeg2 encoder did you use ?

[pandy]

Ok, i use a tmepgenc for HDTV encoding. Solution for memory problem is:
- add more memory (support of tmepgenc claims that more memory is better - but they dont know that tmpegenc utilise more than 2GB of memory - which seems to be common problem for all x86 32 bit apps)

- shorten GOP size - ie standard which mean 15 frames in GOP or shorter, from my knowledge for 1920x1080 tmpegenc utilise more than 1.4GB memory

- lower motion search method ie disable highest quality motion search which is no good for quality of final video...

btw
switches for MSWINT kernel such as: /3GB and/or /PAE seems to not working with the tmpegenc (ok maybe it works with the system which have more than 2GB phsysical RAM - not working withe virtual memory ie swap - i have only 2GB RAM in my computer and tmpegenc have the same "out of memory" banner )

[Manao]

You can also use a proper mpeg2 encoder, such as mencoder. Quality will be better ( hopefully better than mjpeg2000 ), and memory won't be a problem anymore

[akupenguin]

A bump to this thread to inform you that the results are (almost) in. From what I've seen (which is only a small part) H.264 is in the lead, followed by VC-1.
What is remarkable, is that MJPEG-2000 (purely intra-based) seems to perform better than MPEG-2 (IBBPBBP...) at sequences with low bitrates (provided there is not too much movement).
That's just backwards. Intra-only codecs don't care how much motion there is, whereas mpeg-* compresses better with less motion.
If these sequences are scaled down to regular SD-resolution MPEG-2 performs better than MJPEG-2000. Anyone got any comments about this?
If that is still true when you use a good mpeg-2 encoder, then here's my explanation:
mjpeg-2000 uses wavelets. Wavelets are scale-invariant. At low bitrates, you just quantize away all the high frequency subbands, and the resulting compressed image is essentially identical to a low resolution image plus an instruction to upscale on playback. (Though it can also have extra detail in places the encoder deems visually important.)
mpeg-2 uses 8x8 dct. If you quantize away all the high frequency coefficients, you're effectively left with a low resolution image coded with a smaller dct. But 8x8 was chosen because it's better than smaller sizes. (>8x8 could have been chosen for the standard, but it's slower.) Also, there's a minimum bit-cost overhead per macroblock, which can take a significant fraction of the total bitrate at high res+low bitrate (an optimal inter-codec would increase the size of each macroblock, but mpeg-2 can't). So at low bitrates you have to downscale to get the most out of mpeg codecs.

- lower motion search method ie disable highest quality motion search which is no good for quality of final video...
Does anyone know whytf memory usage would depend on motion search method?

[Stavr0s]

Thanks for your thoughts about the MJPEG-2000 vs MPEG-2 subject. I was thinking something along the same lines but definately not as detailed as this explanation :). Anyway, this 'problem' has nothing to do with real-world encoding, as no sane person would try to encode 1920x1080 video with MPEG-2 at 3Mbps. Just scientific curiosity.

PS: I should've mentioned that we used Mainconcept's MPEG-2 encoder

[benwaggoner]

A bump to this thread to inform you that the results are (almost) in. From what I've seen (which is only a small part) H.264 is in the lead, followed by VC-1.
What is remarkable, is that MJPEG-2000 (purely intra-based) seems to perform better than MPEG-2 (IBBPBBP...) at sequences with low bitrates (provided there is not too much movement). If these sequences are scaled down to regular SD-resolution MPEG-2 performs better than MJPEG-2000. Anyone got any comments about this?
What VC-1 options did you wind up using?

Can you share any of the output files?