To MB1 and Kwag - matrices and min. bitrate

[ninja.rogue]

Hi folks,
just for the sake of completeness, I have read almost all forum messages here and in vcdhelp.com about matrices.
There is one thing that I have to ask Mb1 and Kwag, about using or not using MB1 matrix with very high bitrates.
Mb1 has explained that his matrix is suitable for DV interlaced source and high bitrate encoding.
Kwag has now got back to default matrix in his templates.
As I am encoding with PAL DVD settings with very high bitrates (about 9000 avg) onto DVD-R, is it coherent to use MB1 matrix or is this matrix only for low bitrate encoding?
Or, to be even simplier, is MB1 matrix suitable for which min bitrate?
My general encoding settings are ok, as I am using highest possible quality, CQ100 or 2pass-VBR with max bitrate = avg bitrate.
I personally presume that Kwag's template are suitable for lower bitrate than MB1 has underlied his matrix - given the CQ value used by Kwag.
Your reply is welcome!!
Have a nice day.
Paolo

[mb1]

Matrices depend extremely on your source material (which you did not tell).
If your source is interlaced then use "mb1 interlaced DV" (no matter if it´s DV or analogue or dvd source) for your high bitrates.
The matrix should be used with following min bitrates for full D1 (720x576):

Encoding to
- 2.000 kbps for 2.35:1 letterboxed movies
- 3.000 kbps for 1.66/1.78/1.85:1 letterboxed movies and 2.35:1 anamorph
- 4.000 kbps for full 4:3 or 1.66/1.78/1.85:1 anamorph

Max bitrate from 7.600 - 9.500 (what your authoring prog allows).
Avg bitrate from 5.000 - 7.500 (or what you calculated to fill dvd-r)

For lower resolutions you can lower the bitrates, too.


I can´t speak for Kwags settings (as I do not know them in detail).
Especially I didn´t have a look at the KDVD template ...

But remember that interlaced sources need another scan order (alternate scan) and so the matrix should be optimized to that scan order (mine is :) ). Default ones aren´t ...

Another point:
...2pass-VBR with max bitrate = avg bitrate...
That´s pretty useless. If max bitrate is set equal to avg bitrate simply encode with CBR.
You don´t get the advantages of variable bitrates (lower space usage or better quality). I would even say this setting would be contraproductive as it forces the encoder to calculate with no variability on the upper borders in 'variability mode'. Result should be not as good as CBR which lets the encoder calculate in an appropriate way.

[rubic42]

How do you optimize a matrix for the alternate scan order? I like the mb1 interlaced DV matrix, but I actually get more compression out of the CCE Very Low and CCE Ultra Low bitrate matrices. How would I rearrange those matrices to work for interlaced DV material? Or is there a way to make your interlaced DV matrix even more efficient?

[ninja.rogue]

First of all, thanks for your prompt reply.
I am encoding DV interlaced videos from DV camcorder in 16:9 (anamorphic) format.
I use Tmpgenc for encoding, and would like to know which scan order it is using. Indeed, I have noticed that CCE does require to set zigzag or alternate scan order, whereas Tmpgenc does not have this option to check.
Does Tmpgenc "naturally" choose the right scan order?

As far as 2-pass VBR is concerned, I didn't know that my settings could cause a worse quality with reference to CBR.
When I examine my Tmpgenc mpeg files with Bitrate viewer, I find out a strange thing. Let me explain it.
I normally try to set max bitrate higher than avg using VBR, but I get always from 500 to 800 kbps over max bitrate - i.e. Tmpgenc encodes at a bitrate exceeding max bitrate of that value.
I am having the same problem also with CBR encoding - final max bitrates exceeding the set avg bitrate of 500-800kbps.

Do you have similar experiences?
Yours faithfully
Paolo

[mb1]

@ rubic42
Here is the scanning order:
http://www.cs.sunysb.edu/~daniel/mpeg/MPEGinaNutshell_files/image005.gif
(a) is zig zag scan for progressive sources
(b) is alternate scan for interlaced sources

and here if you prefer it in numbered values:
http://www.bretl.com/mpeghtml/zigzag.HTM

Arrange your matrix values in the appropriate order (maybe take a comparing look at my matrix).
But it´s not that simple. Maybe you have to choose slightly different values. That´s a little try and error ... and lots of detailed testings. :rolleyes:

@ ninja.rogue
You´re right, tmpeg does it automatically if you set your output to interlaced.

Bitrate peaks:
Every encoder peaks the set max bitrate - sometimes drastically.
The peaking value (in % of max bitrate) is higher if the bitrate is set lower.
Means:
- set your max bitrate to f.e. 2.520 kbps (for svcd) and you´ll probably get peaks around 2.800 to 3.000 !
But that doesn´t automatically hurt the svcd standard. The vbv buffer weakens these peaks as it stores a certain amount of these picture informations.
Thus the vbv should always be set to the max allowed value (224 KB for SVCD and DVD). So set it up in Tmpeg to 224. CCE does it automatically.

- set your max bitrate to f.e. 8.000 (for dvd) and you`ll probably get peaks around 8.000 to 8.500 !
So the peaks are lower compared to the max bitrate (in %). Again vbv buffer weakens them.

If you can´t live with that peaks :p simply lower the max bitrate around 500 kbps.

cbr does not mean absolutely constant bitrates. There also are peaks and lows but mostly weaked.

And last but not least:
Bitrateviewer shows bitrate values always in seconds (PAL: 25 frames/sec) but encoders calculate their bitrates in GOPs (f.e. 15 pictures per GOP). So the numbers of I,P and B frames differ from second to second.
Remember: I-frames need a lot of space, P-frames some and B-frames only a small amount.

To have representative views in bitrateviewer you have to set a GOP of 5 or 25 (that also prevents that typical zig zag line of tmpeg encodes in bv).

[rubic42]

Wow, that was some good info there. I'll see what I can manage, thanks!

[rubic42]

Here's a quickie perl script I threw together to help analyze and convert the matrices from one scan order to the other. Maybe someone else will find it useful.

Say you have a text file containing two matrices, like this:


Intra10_0=8 13 13 17 17 21 21 28
Intra10_1=13 13 17 17 19 21 23 30
Intra10_2=13 17 19 19 21 23 28 34
Intra10_3=13 17 19 19 21 23 28 48
Intra10_4=17 19 19 19 23 28 34 48
Intra10_5=19 19 23 25 28 32 34 48
Intra10_6=19 21 23 25 28 32 34 48
Intra10_7=21 21 25 25 28 32 34 48
NonIntra10_0=8 11 11 15 15 17 17 24
NonIntra10_1=11 11 15 15 17 17 21 24
NonIntra10_2=13 15 15 17 21 21 26 34
NonIntra10_3=13 17 15 17 21 21 26 48
NonIntra10_4=17 21 21 23 21 30 34 48
NonIntra10_5=17 21 21 23 28 30 34 48
NonIntra10_6=19 19 25 26 28 30 48 48
NonIntra10_7=19 19 25 26 28 30 48 48


Running 'perl dctmat.pl -f file.txt' will print out the same matrix just as it appears.

Running 'perl dctmat.pl -f file.txt -i zigzag' will unzigzag the coefficients and print them out in flat order.

Running 'perl dctmat.pl -f file.txt -i zigzag -o zigzag' will unzigzag the coefficients, rezigzag them and print them out. This will produce the same output as the first command.

Running 'perl dctmat.pl -f file.txt -i zigzag -o alternate' will unzigzag the coefficients, rearrange them into alternate scan order and print them out. This lets you see what your coefficients look like in alternate scan order instead of zigzag.

Other combinations are possible.


sub usage() {
print STDERR <<EOF;

dctmat.pl - this program will unravel and ravel
DCT coefficient matrices automatically

usage: $0 [-h] [-f file] [-i zigzag|alternate] [-o zigzag|alternate] [-p flat]

-h : prints this help screen
-f file : uses file for input, will read any 8 lines of 8 numbers
: that are separated by spaces or commas
-i zigzag|alternate : input matrices are in zigzag or alternate order
: defaults to flat row/col order
-o zigzag|alternate : output matrices are in zigzag or alternate order
: defaults to flat row/col order
-p flat : prints in flat, one row, 64 col format
: defaults to 8x8 matrix

EOF

exit;
}

use Getopt::Std;

@zigzag = (
0, 1, 5, 6, 14, 15, 27, 28,
2, 4, 7, 13, 16, 26, 29, 42,
3, 8, 12, 17, 25, 30, 41, 43,
9, 11, 18, 24, 31, 40, 44, 53,
10, 19, 23, 32, 39, 45, 52, 54,
20, 22, 33, 38, 46, 51, 55, 60,
21, 34, 37, 47, 50, 56, 59, 61,
35, 36, 48, 49, 57, 58, 62, 63
);

@alternate = (
0, 4, 6, 20, 22, 36, 38, 52,
1, 5, 7, 21, 23, 37, 39, 53,
2, 8, 19, 24, 34, 40, 50, 54,
3, 9, 18, 25, 35, 41, 51, 55,
10, 17, 26, 30, 42, 46, 56, 60,
11, 16, 27, 31, 43, 47, 57, 61,
12, 15, 28, 32, 44, 48, 58, 62,
13, 14, 29, 33, 45, 49, 59, 63
);

getopts ("hi:o:f:p:", \my %args);

usage() if ( $args{h} );

@matrices_raw = ();
@matrix = ();
open FILE, "$args{f}" or die "Could not open $args{f} for reading";
while (<FILE>) {
if (/(\d+)[,\s]+(\d+)[,\s]+(\d+)[,\s]+(\d+)[,\s]+(\d+)[,\s]+(\d+)[,\s]+(\d+)[,\s]+(\d+)/) {
push @matrix, ($1,$2,$3,$4,$5,$6,$7,$8);
}
if ( scalar @matrix > 63 ) {
push @matrices_raw, [@matrix];
@matrix = ();
}
}
close FILE;

@matrices_flat = ();
if ( $args{i} eq "zigzag" ) {
for $i ( 0 .. $#matrices_raw ) {
@matrix = ();
for $j ( 0 .. $#{ $matrices_raw[$i] } ) {
$matrix[$zigzag[$j]] = $matrices_raw[$i][$j];
}
push @matrices_flat, [@matrix];
}
}
elsif ( $args{i} eq "alternate" ) {
for $i ( 0 .. $#matrices_raw ) {
@matrix = ();
for $j ( 0 .. $#{ $matrices_raw[$i] } ) {
$matrix[$alternate[$j]] = $matrices_raw[$i][$j];
}
push @matrices_flat, [@matrix];
}
}
else {
@matrices_flat = @matrices_raw;
}

@matrices_out = ();
if ( $args{o} eq "zigzag" ) {
for $i ( 0 .. $#matrices_flat ) {
@matrix = ();
for $j ( 0 .. $#{ $matrices_flat[$i] } ) {
$matrix[$j] = $matrices_flat[$i][$zigzag[$j]];
}
push @matrices_out, [@matrix];
}
}
elsif ( $args{o} eq "alternate" ) {
for $i ( 0 .. $#matrices_flat ) {
@matrix = ();
for $j ( 0 .. $#{ $matrices_flat[$i] } ) {
$matrix[$j] = $matrices_flat[$i][$alternate[$j]];
}
push @matrices_out, [@matrix];
}
}
else {
@matrices_out = @matrices_flat;
}

if ( $args{p} eq "flat" ) {
for $i ( 0 .. $#matrices_out ) {
print "@{$matrices_out[$i]}\n";
}
print "\n";
}
else {
for $i ( 0 .. $#matrices_out ) {
for $j ( 0 .. $#{ $matrices_out[$i] } ) {
printf "%4d", $matrices_out[$i][$j];
if ( $j % 8 == 7 ) { print "\n"; }
}
print "\n";
}
}

[mb1]

Nice one, thanks.

But I have no Perl installed. Have to change that.

Another format in which matrices are stored is this one:

'mb1 interlaced DV'
8,13,13,17,17,21,21,28,
13,13,17,17,19,21,23,30,
13,17,19,19,21,23,28,34,
13,17,19,19,21,23,28,48,
17,19,19,19,23,28,34,48,
19,19,23,25,28,32,34,48,
19,21,23,25,28,32,34,48,
21,21,25,25,28,32,34,48

8,11,11,15,15,17,17,24,
11,11,15,15,17,17,21,24,
13,15,15,17,21,21,26,34,
13,17,15,17,21,21,26,48,
17,21,21,23,21,30,34,48,
17,21,21,23,28,30,34,48,
19,19,25,26,28,30,48,48,
19,19,25,26,28,30,48,48

Think of f.e. the CCE-Patcher and its .ini

Maybe you can expand your script to convert this format as well ...
Would be very helpful for future matrices.

Thanks for that piece of work.

[rubic42]

The script should be able to handle reading in any format as long as it uses commas and/or spaces in any combination, that's the way that regular expression is written.

The next thing would be outputting into different formats, was too lazy to do that yet.

You should try ActiveState's Perl distribution for Windows, I'm running this and it works great for automating tasks that require some computation.