Matrix and Quantizer Choice In CQ Mode

[Makaveli84]

Hello,
I use Avidemux for mpeg 2 encoding. There are two encoder options (mpeg2video and mpeg2enc). Both have a pre-bundled list of matrices to chose from for encoding (default, tmpgenc, anime and kvcd).
I need advice on which to go with for:

1) Live action TV Shows or Movies.

2) Adult cartoons (i.e. Family Guy, Simpsons, etc..)

Specifically, I would like to know if the Anime matrices in general (not merely for mpeg 2) are optimized for cartoons in general, or for the Anime genre specifically (the Japanese style cartoons).

Also, I tend to encode in CQ mode. Now Avidemux help states that mpeg2video is based on lavc which should only be used in 2-pass mode because it's not very good in CQ mode. They go on to say that mpeg2enc is better in CQ mode, but is slower and results in larger files. I tried them both, and mpeg2enc is slower and results in much larger files in CQ mode, but I couldn't really tell the quality tradeoff between these 2 encoders. Plus I am having some trouble with progressive vs interlaced and fps issues with the supposedly better quality encoder (mpeg2enc).

So here's how you can help:

a) Would encoding with Q=2 negate the bad quality of mpeg2video in CQ mode (although in testing, even with Q=2, mpeg2enc results in much larger files)?

b) If I were to use the supposedly superior encoder (mpeg2enc), what is the highest Q advisable for very good quality encoding? The Avidemux default setting is Q=4, but I'm not sure.. Should I go with Q=4? 3? Or 2?
Q=2 results in approx. 1 GB files for 22 mins of cartoon, which is higher than most commercial DVDs out there, so maybe it's a bit of an overreach? Or not??

Please, I need some advice people. I tried posting in the Avidemux forum, but that place doesn't look to vibrant and I didn't get anywhere there... Thx!!

[manolito]

Using AviDemux for MPEG2 encoding certainly is convenient, but both of the built-in encoders are second rate by todays standards. You will get higher quality if you use a standalone encoder like CCE or HC (HC is free).

Also the built-in custom quant matrices are quite outdated. They were created at the time when everybody still did SVCD conversions. The default MPEG2 matrix is probably your best choice.

MPEG2 custom quant matrices have long been the subject of some controversy. Just have a look at this thread:
http://forum.doom9.org/showthread.php?p=1465642&highlight=matrix#post1465642
Both CCE and HC use an exponential quantizer scale, so the choice of a suitable matrix is much less critical than for libav based encoders.

What is the final target format for your encodes? If you want to create DVDs you have to know that CQ based encodes will most likely result in noncompliant DVDs because the max bitrate is not observed.

Generally a quantizer of 2 is overkill, for a good quality encoding start with a quantizer of 4 and go lower if the quality is not good enough. But again, for DVD compliant results you should use 2-pass VBR for your encodes.



Cheers
manolito

[Makaveli84]

Using AviDemux for MPEG2 encoding certainly is convenient, but both of the built-in encoders are second rate by todays standards. You will get higher quality if you use a standalone encoder like CCE or HC (HC is free).

Also the built-in custom quant matrices are quite outdated. They were created at the time when everybody still did SVCD conversions. The default MPEG2 matrix is probably your best choice.

MPEG2 custom quant matrices have long been the subject of some controversy. Just have a look at this thread:
http://forum.doom9.org/showthread.php?p=1465642&highlight=matrix#post1465642
Both CCE and HC use an exponential quantizer scale, so the choice of a suitable matrix is much less critical than for libav based encoders.

What is the final target format for your encodes? If you want to create DVDs you have to know that CQ based encodes will most likely result in noncompliant DVDs because the max bitrate is not observed.

Generally a quantizer of 2 is overkill, for a good quality encoding start with a quantizer of 4 and go lower if the quality is not good enough. But again, for DVD compliant results you should use 2-pass VBR for your encodes.



Cheers
manolito

Thanks for this reply manolito, it was very helpful and informative. I will see about the standalone encoders, but as long as use Avidemux, I will go with the default matrix. However, I am still intrigued by the "Anime" matrix, and if it is "safe" to use with cartoons.

Now I am encoding for DVD, but I am setting a max bitrate of 8000 to go along with CQ. Is this being ignored? I was under the impression that I was using a CQ + max bitrate encoding mode to achieve best possible compliance...

[manolito]

Now I am encoding for DVD, but I am setting a max bitrate of 8000 to go along with CQ. Is this being ignored? I was under the impression that I was using a CQ + max bitrate encoding mode to achieve best possible compliance...
I am not too familiar with the latest versions of libavcodec, but AFAIK encoding in CQ mode does ignore any max bitrate settings. But maybe someone more familiar with libav and ffmpeg will prove me wrong...

What I do know for sure is that HCenc has a CQ_Maxbitrate mode. But it also has a "real" 1-pass VBR mode with a quick prediction pass (determines a base quantizer and establishes a compression curve). Kind of a poor man's 2-pass VBR, much faster than a real 2-pass encode, and the quality is amazing.



Cheers
manolito