HCenc 2nd pass, should I do anything about "*** INFO, adjusting average bitrate..." ?

[jclampy]

Hi, I have a 352x576 analog capture from TV of some concert footage so it is mostly dark scenes and also using multiple portable cameras so the footage is moving around alot. It is a metal concert and so the moving with the cameras and panning and zooming and quick changes between camera's is suppose to increase the intensity I guess.

So darks scenes and alot of movement in the footage.

Luckily the bitrate for the video encoding I can go up to maximum of 8000kb/s give or take about 200kb/s which I will keep as a maximum allowed overshoot safety buffer.

Anyhow, these are the settings I was going to go with based on the limited information I have been able to find:
I am using HCenc to encode and here are my settings:

interlacing options = progressive
chroma downsampling = progressive
progressive sequence = ticked
colorimetry = BT.470-2BG
MPEG profile@level = MP@ML
--------------------
| encoder settings |
--------------------
profile: BEST
framerate: 25.000
aspect ratio: 4:3
chroma format: 4:2:0
bitrate Kb/s: 7000
max. bitrate Kb/s: 8000
pulldown: no
closed gops: no
VBV check: yes
VBV size (bits): 1835008
scene change det: yes
interlaced: no, BFF
goplen,B-pic: AUTO 15
dc_precision: 10
intraVLC: auto
scan method: zigzag
bias: 100
chapter frames: 0
time code: 0 0 0 0
CPU: SSE2
priority: idle
SMP active: yes
matrix: MANONO1, adaptive
luminance gain: yes, level 4
adaptive quantization: yes, strength 4
deadzone quantization: auto
width*height: 352x576
fps: 25.000
*BITRATE 7000
*MAXBITRATE 8000
*FRAMES 4500 5999
*PROFILE best
*ASPECT 4:3
*AUTOGOP 15
*AQ 4
*DC_PREC 10
*PROGRESSIVE
*PROG_SEQ
*BFF
*FRAMELOG psnr
*BIAS 100
*MPEGLEVEL MP@ML
*INTRAVLC 2
*CUSTOMMATRIX
8 8 9 9 10 10 11 11
8 9 9 10 10 11 11 12
9 9 10 10 11 11 12 12
9 10 10 11 11 12 13 13
10 10 11 11 12 13 13 14
10 11 11 12 13 13 14 15
11 11 12 13 13 14 15 15
11 12 12 13 14 15 15 16

8 8 9 9 10 10 11 11
8 9 9 10 10 11 11 12
9 9 10 10 11 11 12 12
9 10 10 11 11 12 13 13
10 10 11 11 12 13 13 14
10 11 11 12 13 13 14 15
11 11 12 13 13 14 15 15
11 12 12 13 14 15 15 16
*LUMGAIN 4
*COLOUR 5

Are there any improvements I could make for the type of video I am encoding? Or have I got everything about right?

I originally was using avg and max bitrate both set at 8000kb/s but recently I have been trying more 'detail keeping' matrix's and have noticed messages like this; " *** INFO, adjusting average bitrate: -2.78 %" or similar coming up. So I thought maybe keep the max at 8000kb/s and drop the avg down to 7000kb/s might help? No idea what the error message means though.

Also, a couple of things I am not sure about:
1) When encoding 2nd pass I see Q: I=xxx P=xxx B=xxx can someone give me an idea what number ranges I should be trying to aim for there? I have heard something around 2-3, will different matrix's for a given bitrate make these numbers go up or down?

2) Does 'average Quantizer' in my log file mean PSNR value and I guess the same reasoning for reading this will be like 1) above? My last 1 minute test has a 'average Quantizer' of 4.310

3) I am using AQ4 because otherwise there is not enough 'padding' for my high 7000 to 8000 bitrates.

4) I am using Lumigain4 because I hope it will give me the most quality, but can it also be detrimental going that high?

PS: I read about "Rebuilder Matrix Editor" but I can't find it, is there a similar tool that people use now-a-days?

Edit:
Just noticed with avg 7000 and max 8000 the "*** INFO, adjusting average bitrate" message still comes up sometimes. Do I need to take notice of this at all or what is the problem and how would one fix it?

Thanks.

[hank315]

The message is pure informational, the encoder can't hold the bitrate you requested.

Some possible explanations:
- the difference between the average and maximum bitrate is small
- the source is very dark
- your source is only 352x576 pixels?, almost half the normal resolution

[jclampy]

Hi Hank,

Thanks for responding. Here are some test run results:
VBR Bias 100:
avg 8000 max 8000 = average bitrate adjusted 5 times + average encoded bitrate 7292kb/s + avg quant 4.249
avg 7500 max 8000 = average bitrate adjusted 4 times + average encoded bitrate 7307kb/s + avg quant 4.262
avg 7000 max 8000 = average bitrate adjusted 2 times + average encoded bitrate 6907kb/s + avg quant 4.569

VBR Bias 0:
avg 8000 max 8000 = average bitrate adjusted 5 times + average encoded bitrate 7290kb/s + avg quant 4.249

A couple of things I am not sure about:
1) When encoding 2nd pass I see Q: I=xxx P=xxx B=xxx do you have an idea what number ranges I should be trying to aim for there? If I am not getting numbers I should be what would adjust this?

2) Does 'average Quantizer' in my log file mean PSNR value and I guess the same reasoning for reading this will be like 1) above?

3) I am using AQ4 because I am using a high 7000 to 8000 bitrate. The should be nothing wrong with that right?

4) I am using Lumigain4 because I hope it will give me the most quality, but can it also be detrimental going that high? The documentation says something about artifacts may appear, but what type of artifacts as I got no idea what to look for?

Maybe the settings I am using are fine, or do you have any advise that might improve things even further?

Thanks.

[Mounir]

according to my tests:

aq:2 lum gain: 1
dc: 10 bits / profile best / autogop, scene change, smp
intra vlc: auto
dead zone: auto
matrix: avamat6
2 pass vbr 9000-9500 (high bitrate)

This is best to keep fine grain details, ideal for videos with lot of dark scenes
- tested with vhs sources mainly

[jclampy]

Hi Mounir, thankyou for the reply. I will run a test with your suggestions tomorrow and post the results.

May I ask what resolution you would be using and do you know what your average bitrate and average quantizer would be hitting?

Thanks.

[hank315]

1) When encoding 2nd pass I see Q: I=xxx P=xxx B=xxx do you have an idea what number ranges I should be trying to aim for there? If I am not getting numbers I should be what would adjust this?
That's difficult to say, the final quantization is always the product of the quantizer * quantmatrix.
So if you're using a matrix like AVAMAT6 you will get lower values for the quantizer than if you're using a matrix like FOX1.
As a rule of thumb, an average quantizer value of 4.0 will do in most cases.

2) Does 'average Quantizer' in my log file mean PSNR value and I guess the same reasoning for reading this will be like 1) above?
Don't fully understand the question but the average quantizer is simply what it says, the sum of all frame quantizers divided by the nr. of frames.
For PSNR, HCenc outputs only the luminance (Y component) PSNR.

3) I am using AQ4 because I am using a high 7000 to 8000 bitrate. The should be nothing wrong with that right?
It could introduce artifacts around sharp edges (ringing). These sharp edges require a lot of encoding bits, using a high AQ value means some bits are taken from these Macro Blocks and used for flat areas to prevent blocking artifacts in these flat areas.
In practice this means the quantizer may vary between the Macro Blocks of the frame, the decision is based on the variance of the MB.

4) I am using Lumigain4 because I hope it will give me the most quality, but can it also be detrimental going that high? The documentation says something about artifacts may appear, but what type of artifacts as I got no idea what to look for?
This command adapts the quant matrices, if the average luminance value in a GOP < 64, the matrix values are lowered for that GOP, more progressive for higher values of LUMGAIN.
This will not introduce artifacts but a few HW players don't like it and won't play it without errors (large artifacts). It certainly is DVD compliant, CCE also uses this technique of manupilating the quant matrices.


In your case I would use the FOX1 matrix, bias 0, bitrate 7000, max bitrate 8000, lumgain 4, AQ 2.

[jclampy]

Thankyou very much Hank, I will give it a shot and post results. I also ran some more tests as well but will post all results at same time. Thanks heaps for answering those questions I appreciate it.

Edit:
Here are some results, most tests were done before your response, I want to do some more tests tomorrow so, this is just to put some results up. I will add some comments later.

Frames 5210, VBR Bias = 100:
~~~~~~~~~~~~~~~~~~~~~~~~~
Test 2E: HCenc avg 8000, max 8000, AQ=4, LG=4, matrix = New Fox
Results HCenc log = min 3348, avg 7442, max 8000, avg bitrate adjusted 3 times (final -6.60%), avg quantizer 4.561
Results DGIndex preview = avg 7455, max 8274

Test 2D: HCenc avg 7000, max 8000, AQ=3, lG=3, matrix = New Fox
Results HCenc log = min 2377, avg 6997, max 7999, avg bitrate adjusted 0 times, avg quantizer 4.641
Results DGIndex preview = avg 7008, max 8192
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Frames 8808 VBR Bias = 100:
~~~~~~~~~~~~~~~~~~~~~~~~
Test 3B: HCenc avg 8000, max 8000, AQ=4, LG=4, matrix = New Fox
Results HCenc log = min 3629, avg 7292, max 7999, avg bitrate adjusted 5 times (final -9.10%), avg quantizer 4.249
Results DGIndex preview = avg 7295, max 8342

Test 3A1: HCenc avg 8000, max 8000, AQ=4, LG=4, matrix = New Fox
Results HCenc log = min 2527, avg 7374, max 7998, avg bitrate adjusted 5 times (final -7.87%), avg quantizer 5.759
Results DGIndex preview = avg 7377, max 8315

Test 3A2: HCenc avg 7500, max 8000, AQ=4, LG=4, matrix = New Fox
Results HCenc log = min 2150, avg 7307, max 7999, avg bitrate adjusted 4 times (final -2.54%), avg quantizer 4.262
Results DGIndex preview = avg 7310, max 8342

Test 2C: HCenc avg 7000, max 8000, AQ=4, LG=4, matrix = New Fox
Results HCenc log = min 1671, avg 6907, max 7999, avg bitrate adjusted 2 times (final -1.31%), avg quantizer 4.569
Results DGIndex preview = avg 6909, max 8124

Test 2A: HCenc avg 8000, max 8000, AQ=4, LG=4, matrix = AVAMAT6
Results HCenc log = min 4140, avg 7191, max 7999, avg bitrate adjusted 5 times (final -10.42%), avg quantizer 1.775
Results DGIndex preview = avg 7196, max 8260
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Frames 8808 VBR Bias = 0:
~~~~~~~~~~~~~~~~~~~~~~
Test 3K: HCenc avg 7500, max 8000, AQ=2, LG=3, matrix = New Fox
Results HCenc log = min 4804, avg 7401, max 7999, avg bitrate adjusted 4 times (final -1.28%), avg quantizer 3.693
Results DGIndex preview = avg 7406, max 8329

Test 3J: HCenc avg 7000, max 8000, AQ=2, LG=2, matrix = New Fox
Results HCenc log = min 3089, avg 7000, max 7996, avg bitrate adjusted 0 times, avg quantizer 3.870
Results DGIndex preview = avg 7003, max 8124

Test 3I: HCenc avg 7000, max 8000, AQ=2, LG=3, matrix = New Fox
Results HCenc log = min 3154, avg 6999, max 7998, avg bitrate adjusted 0 times, avg quantizer 3.961
Results DGIndex preview = avg 7004, max 8110

Test 3H: HCenc avg 7000, max 8000, AQ=2, LG=4, matrix = Fox1
Results HCenc log = min 3066, avg 6996, max 7998, avg bitrate adjusted 0 times, avg quantizer 4.286
Results DGIndex preview = avg 6999, max 8274

Test 3G: HCenc avg 7000, max 8000, AQ=2, LG=4, matrix = New Fox
Results HCenc log = min 3079, avg 6997, max 7999, avg bitrate adjusted 0 times, avg quantizer 4.037
Results DGIndex preview = avg 7000, max 8178

Test 3F: HCenc avg 7500, max 8000, AQ=3, LG=3, matrix = New Fox
Results HCenc log = min 1948, avg 7241, max 7998, avg bitrate adjusted 4 times (final -3.28%), avg quantizer 3.936
Results DGIndex preview = avg 7245, max 8206

Test 3E: HCenc avg 7500, max 8000, AQ=2, LG=2, matrix = New Fox
Results HCenc log = min 2915, avg 7384, max 7999, avg bitrate adjusted 4 times (final -1.52%), avg quantizer 3.608
Results DGIndex preview = avg 7388, max 8342

Test 3D: HCenc avg 7500, max 8000, AQ=2, LG=1, matrix = New Fox
Results HCenc log = min 4848, avg 7337, max 7999, avg bitrate adjusted 4 times (final -2.15%), avg quantizer 3.437
Results DGIndex preview = avg 7343, max 8451

Test 3C: HCenc avg 8000, max 8000, AQ=4, LG=4, matrix = New Fox
Results HCenc log = min 3629, avg 7290, max 7999, avg bitrate adjusted 5 times (final -9.12%), avg quantizer 4.249
Results DGIndex preview = avg 7293, max 8342
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Full Video Encode VBR Bias = 0:
~~~~~~~~~~~~~~~~~~~~~
Test 4: HCenc avg 7000, max 8000, AQ=2, LG=4, matrix = New Fox
Results HCenc log = min 2027, avg 6993, max 7999, avg bitrate adjusted 3 times (final -0.09%), avg quantizer 4.261
Results DGIndex preview = avg 6997, max 8383

Test 5: HCenc avg 7000, max 8000, AQ=2, LG=2, matrix = New Fox
Results HCenc log = min 1989, avg 6999, max 7999, avg bitrate adjusted 1 time (final -0.02%), avg quantizer 4.135
Results DGIndex preview = avg 7002, max 8397
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I decided to try a couple of full video encodes (Tests 4&5) using settings from Test3G & Test3J and unfortunately the maximum bitrate peaks went higher than 8264. That will put my overall total final bitrate beyond DVD specifications.

My goal at the moment is for video maximum bitrate to stay under 8264. Looks like I need to bite the bullet and drop my maximum bitrate in HCenc a little lower. Seems to be a scene around 14 minutes into the video that overshoots it to nearly 8400.

Here is where I read about New Fox Matrix, although I see is from 2005; http://forum.doom9.org/showthread.php?p=759321#post759321

In the meantime if anyone wants to make any comments on the results above then please do.

[jclampy]

Ok, this is weird. I found where abouts in the full video where the bitrate is overshooting by almost 400kb/s but when I encode just a small section (about a minute) around that time I cannot get HCenc to overshoot.

I am at a bit of a loss of how to progress along this path as HCenc does not appear to be consistant in this regard.

Maybe lowering the maximum bitrate further may solve this issue but unless running a full video encode testing is blind.

Edit:
Ok, I have done some more tests with a lower maximum bitrate and think I have reached the ultimate conclusion;
Frames 8808 VBR Bias = 0:
~~~~~~~~~~~~~~~~~~~~~~
Test 6B: HCenc avg 6656, max 7680, AQ=2, LG=4, matrix = New Fox
Results HCenc log = min 2756, avg 6653, max 7678, avg bitrate adjusted 0 times, avg quantizer 4.209
Results DGIndex preview = avg 6656, max 7796

Test 6C: HCenc avg 6656, max 7680, AQ=2, LG=2, matrix = New Fox
Results HCenc log = min 3030, avg 6655, max 7676, avg bitrate adjusted 0 times, avg quantizer 4.068
Results DGIndex preview = avg 6658, max 7810

Test 6D: HCenc avg 7168, max 7680, AQ=2, LG=3, matrix = New Fox
Results HCenc log = min 3031, avg 7071, max 7679, avg bitrate adjusted 4 times (final -1.27%), avg quantizer 3.867
Results DGIndex preview = avg 7075, max 7946

Test 6E: HCenc avg 7168, max 7680, AQ=2, LG=2, matrix = New Fox
Results HCenc log = min 4261, avg 7119, max 7679, avg bitrate adjusted 3 times (final -0.56%), avg quantizer 3.750
Results DGIndex preview = avg 7123, max 8028
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Full Video Encode VBR Bias = 0:
~~~~~~~~~~~~~~~~~~~~~
Test7: HCenc avg 7168, max 7680, AQ=2, LG=2, matrix = New Fox (based on Test 6E)
Results HCenc log = min 2835, avg 7098, max 7680, avg bitrate adjusted 30 times (final -0.98%), avg quantizer 4.039
Results DGIndex preview = avg 7102, max 8110
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

As you can see, I have reached my three goals:
1) max bitrate kept under 8264kb/s
2) avg quantizer closest to 4.0
3) avg bitrate adjusted to a minimum; just less than 1%

I have learned alot over this experience and I hope others reading this may also see how certain settings work and how you can manipulate different results by experimenting with different combinations of settings.

Thankyou Hank for HCenc!