Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 14th November 2013, 09:36   #1  |  Link
xkfz007
Registered User
 
Join Date: Sep 2013
Posts: 38
How to evaluate the VBV-compliant CBR mode of X264?

Hi guys, does anyone know the way to evaluate the VBV-compliant CBR of X264?
For the ABR mode, we can evaluate the method using the difference between the real bitrate and the target bitrate, which is indicated by the parameter --bitrate. However, for VBV, there are three related parameters, --vbv-maxrate, --vbv-bufsize and --bitrate. And in most time, the real bitrate varies a lot with the --bitrate. I am new on rate control of X264. Please help me.
xkfz007 is offline   Reply With Quote
Old 14th November 2013, 14:30   #2  |  Link
handaimaoh
Registered User
 
Join Date: Nov 2013
Posts: 80
Sure, you can use tools like Elecard StreamEye, Neuron2's VBV Checker, etc.

Last edited by handaimaoh; 15th November 2013 at 14:41.
handaimaoh is offline   Reply With Quote
Old 15th November 2013, 00:40   #3  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,867
Yes, to evaluate this in detail you need a stream analyzer. Elecard Buffer analyzer is a relatively inexpensive option - or you can look at more expensive options from Tektronix, Sencore, Interra Systems etc...

neuron2's VBV checker is by far the cheapest and simplest. It's quite useful, but won't be trusted by others used to paying for something like a Sencore CMA-1820 (~ 6 figures last I checked). That being said, the latter is an exhaustive transport stream analyzer that also happens to check buffer compliance of elementary streams.
__________________
These are all my personal statements, not those of my employer :)
Blue_MiSfit is offline   Reply With Quote
Old 15th November 2013, 00:46   #4  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,922
There is also a simple totally free perl script that the x264 guys have promulgated. There is no reason not to trust that script or my GUI tool. I know you are not implying they are untrustworthy but I say this to avoid any misunderstanding.

Last edited by Guest; 15th November 2013 at 00:49.
Guest is offline   Reply With Quote
Old 15th November 2013, 01:29   #5  |  Link
xkfz007
Registered User
 
Join Date: Sep 2013
Posts: 38
Quote:
Originally Posted by neuron2 View Post
There is also a simple totally free perl script that the x264 guys have promulgated. There is no reason not to trust that script or my GUI tool. I know you are not implying they are untrustworthy but I say this to avoid any misunderstanding.
Since I am new, I don't know there is a Perl script. Neither did your software, sorry. Could you tell me where can I find the script and how can I get your soft? Thank you
xkfz007 is offline   Reply With Quote
Old 15th November 2013, 01:40   #6  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,922
For some discussion of the script and where to get it, see here:

http://forum.doom9.org/showthread.php?t=149601

My software is here (scroll to bottom):

http://neuron2.net/dgdecnv/dgdecnv.html

Note that my software requires a $15 donation.

If you tell us more specifically what you want to do, we can probably help you better.
Guest is offline   Reply With Quote
Old 15th November 2013, 07:06   #7  |  Link
drmpeg
Registered User
 
Join Date: Jan 2003
Location: Silicon Valley
Posts: 458
If you're interested in rate control, then you need to know the size of each picture in a H.264 bitstream. Here's some C code to get you started.
Code:
/*
H.264 picture size parser
*/

#include <stdio.h>
#include <stdlib.h>
#include <sys/file.h>

#define    TRUE    1
#define    FALSE   0

int main(int argc, char **argv)
{
    FILE    *fp;
    static unsigned char    buffer[16384];
    int    i, length;
    static int    first = TRUE;
    static unsigned int    parse = 0;
    unsigned int    picture_size = 0;

    if (argc != 2) {
        fprintf(stderr, "usage: bits <infile>\n");
        exit(-1);
    }

    /*--- open binary file (for parsing) ---*/
    fp = fopen(argv[1], "rb");
    if (fp == 0) {
        fprintf(stderr, "Cannot open bitstream file <%s>\n", argv[1]);
        exit(-1);
    }

    while(!feof(fp))  {
        length = fread(&buffer[0], 1, 16384, fp);
        for(i = 0; i < length; i++)  {
            parse = (parse << 8) + buffer[i];
            if ((parse & 0xffffff00) == 0x00000100)  {    /* look for a start code */
                if ((parse & 0xffffff9f) == 0x00000101 || (parse & 0xffffff9f) == 0x00000105)  {    /* look for a coded slice */
                    i++;
                    picture_size++;
                    parse = (parse << 8) + buffer[i];
                    if (parse & 0x00000080)  {    /* make sure it's the first slice of a picture */
                        if (first == TRUE)  {
                            first = FALSE;
                            picture_size = 0;
                        }
                        else  {
                            printf("%8d\n", picture_size * 8);
                            picture_size = 0;
                        }
                    }
                }
            }
            picture_size++;
        }
    }
    fclose(fp);
    return 0;
}
Once you know the size of each picture, it's easy to calculate the VBV occupancy.

1) Initialize the size of the VBV buffer (in bits). Either to full or some percentage of full.

2) Subtract the size of a picture (in bits) from the VBV. Check to see if the VBV buffer fullness is zero or less than zero. If zero or less than zero, it's a VBV underflow.

3) Add the number of bits based on the VBV rate. For example, if the vbv-maxrate is 10,000,000 bits/second and the frame rate is 25 frames/second, then add 400,000 bits. If the stream is really CBR (in x264, you need to explicitly turn on picture stuffing), then you should not exceed the size of the VBV buffer. If you exceed the size of the VBV buffer, it's a VBV overflow. For VBR streams (or CBR stream without picture stuffing), there is no VBV overflow and you just set the VBV level back to full on an overflow.

4) Repeat steps 2 and 3 for the entire bitstream. Note that the number of bits to add on step 3 is based on frame rate. Therefore, VBV doesn't apply to variable frame rate and telecine flags need to be accounted for. Also, if the frame rate is not an integer (like 29.97 fps), then be sure to use floating point math and add fractional bits back to the VBV.

Ron
__________________
HD MPEG-2 Test Patterns http://www.w6rz.net
drmpeg is offline   Reply With Quote
Old 15th November 2013, 11:37   #8  |  Link
xkfz007
Registered User
 
Join Date: Sep 2013
Posts: 38
Quote:
Originally Posted by drmpeg View Post
If you're interested in rate control, then you need to know the size of each picture in a H.264 bitstream. Here's some C code to get you started.
Code:
/*
H.264 picture size parser
*/

#include <stdio.h>
#include <stdlib.h>
#include <sys/file.h>

#define    TRUE    1
#define    FALSE   0

int main(int argc, char **argv)
{
    FILE    *fp;
    static unsigned char    buffer[16384];
    int    i, length;
    static int    first = TRUE;
    static unsigned int    parse = 0;
    unsigned int    picture_size = 0;

    if (argc != 2) {
        fprintf(stderr, "usage: bits <infile>\n");
        exit(-1);
    }

    /*--- open binary file (for parsing) ---*/
    fp = fopen(argv[1], "rb");
    if (fp == 0) {
        fprintf(stderr, "Cannot open bitstream file <%s>\n", argv[1]);
        exit(-1);
    }

    while(!feof(fp))  {
        length = fread(&buffer[0], 1, 16384, fp);
        for(i = 0; i < length; i++)  {
            parse = (parse << 8) + buffer[i];
            if ((parse & 0xffffff00) == 0x00000100)  {    /* look for a start code */
                if ((parse & 0xffffff9f) == 0x00000101 || (parse & 0xffffff9f) == 0x00000105)  {    /* look for a coded slice */
                    i++;
                    picture_size++;
                    parse = (parse << 8) + buffer[i];
                    if (parse & 0x00000080)  {    /* make sure it's the first slice of a picture */
                        if (first == TRUE)  {
                            first = FALSE;
                            picture_size = 0;
                        }
                        else  {
                            printf("%8d\n", picture_size * 8);
                            picture_size = 0;
                        }
                    }
                }
            }
            picture_size++;
        }
    }
    fclose(fp);
    return 0;
}
Once you know the size of each picture, it's easy to calculate the VBV occupancy.

1) Initialize the size of the VBV buffer (in bits). Either to full or some percentage of full.

2) Subtract the size of a picture (in bits) from the VBV. Check to see if the VBV buffer fullness is zero or less than zero. If zero or less than zero, it's a VBV underflow.

3) Add the number of bits based on the VBV rate. For example, if the vbv-maxrate is 10,000,000 bits/second and the frame rate is 25 frames/second, then add 400,000 bits. If the stream is really CBR (in x264, you need to explicitly turn on picture stuffing), then you should not exceed the size of the VBV buffer. If you exceed the size of the VBV buffer, it's a VBV overflow. For VBR streams (or CBR stream without picture stuffing), there is no VBV overflow and you just set the VBV level back to full on an overflow.

4) Repeat steps 2 and 3 for the entire bitstream. Note that the number of bits to add on step 3 is based on frame rate. Therefore, VBV doesn't apply to variable frame rate and telecine flags need to be accounted for. Also, if the frame rate is not an integer (like 29.97 fps), then be sure to use floating point math and add fractional bits back to the VBV.

Ron
Thanks for your detailed reply
xkfz007 is offline   Reply With Quote
Old 15th November 2013, 19:25   #9  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 3,954
Quote:
Originally Posted by xkfz007 View Post
Thanks for your detailed reply
Also, if you can work from how many bits each, you can pull that from a .stats file as long as you archive one from the final encoding pass.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 16th November 2013, 02:34   #10  |  Link
drmpeg
Registered User
 
Join Date: Jan 2003
Location: Silicon Valley
Posts: 458
For analysis, it's very useful to graph things out. Here are some plots from an MPEG-2 encoder I used to work on. The bitstream is true CBR with an IPPPPP GOP. Looking at the running average, it can be seen that the CBR rate control is pretty "tight" (doesn't fluctuate much). The upward spikes are due to I-frame insertion at scene changes.



And a zoom in.



Ron
__________________
HD MPEG-2 Test Patterns http://www.w6rz.net
drmpeg is offline   Reply With Quote
Old 16th November 2013, 02:44   #11  |  Link
drmpeg
Registered User
 
Join Date: Jan 2003
Location: Silicon Valley
Posts: 458
VBV graphs for the same bitstream.



And a zoom in.



Ron
__________________
HD MPEG-2 Test Patterns http://www.w6rz.net
drmpeg is offline   Reply With Quote
Old 18th November 2013, 19:13   #12  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 1,945
Is this calculation correct ?
Code:
// VBV_Check.cpp*: définit le point d'entrée pour l'application console.
//

#include "stdafx.h"

const long double THRESHOLD_LOW=1000.0;

int _tmain(int argc, _TCHAR* argv[])
{
    FILE    *fp;
    uint8_t buffer[16384];
    int i,length;
    bool first=true;
	bool VBV_Underlfow=false,VBV_Low=false;
    uint32_t parse=0,picture_size=0;
	long double VBV_Buffer,VBV_Picture,VBV_Min;
	long double fps,VBV_Init,VBV_Bitrate;
	uint32_t FrameNumber=0,FrameMin=0;
	char *FileName;

    if (argc != 2) {
        fprintf(stderr, "Syntax : VBV_Check <file_name>\n");
        exit(-1);
    }

	length=0;
	while ((argv[1][length]!='\0') && (length<10000)) length++;
	if ((length==10000) || (length==0)) {printf("Error with the file name !\n"); exit(-1);}
	FileName=(char *)malloc(length+1);
	if (FileName==NULL) {printf("Allocation error !\n"); exit(-1);}
	for (i=0; i<length; i++)
		FileName[i]=(char)argv[1][i];
	FileName[length]='\0';

	VBV_Init=0.9;
	fps=23.976;
	VBV_Bitrate=24000000.0;

	VBV_Buffer=VBV_Init*VBV_Bitrate;
	VBV_Picture=VBV_Bitrate/fps;
	VBV_Min=VBV_Bitrate;

    /*--- open binary file (for parsing) ---*/
    fp=fopen(FileName,"rb");
    if (fp==0)
	{
        fprintf(stderr,"Unable to open file <%s>\n",FileName);
		free(FileName);
        exit(-1);
    }
	free(FileName);

    while(!feof(fp))
	{
        length=fread(&buffer[0],1,16384,fp);
        for(i=0; i<length; i++)
		{
            parse=(parse<<8)+buffer[i];
            if ((parse & 0xffffff00)==0x00000100)
			{    /* look for a start code */
				uint32_t detect_start=parse & 0xffffff9f;

                if ( (detect_start==0x00000101) || (detect_start==0x00000105) )
				{    /* look for a coded slice */
					if (i<(length-1)) i++;
					else
					{
						if (!feof(fp))
						{
							length=fread(&buffer[0],1,16384,fp);
							i=0;
						}
						else
						{
							fclose(fp);
							printf("\nMin value of VBV Buffer : %f at frame : %lu\n",VBV_Min,FrameMin);
							return 0;
						}
					}
                    picture_size++;
                    parse=(parse << 8)+buffer[i];
                    if (parse & 0x00000080)
					{    /* make sure it's the first slice of a picture */
                        if (first==true)
						{
                            first=false;
                            picture_size=0;
                        }
                        else
						{
							picture_size<<=3;
                            printf("Frame Number : %lu    Size : %lu",FrameNumber,picture_size);
							printf("   VBV Buffer : %lu/",(uint32_t)floor(VBV_Buffer));
							VBV_Buffer-=picture_size;
							printf("%lu/",(uint32_t)floor(VBV_Buffer));
							if (VBV_Buffer<VBV_Min)
							{
								VBV_Min=VBV_Buffer;
								FrameMin=FrameNumber;
							}
							if (VBV_Buffer<=0.0)
							{
								VBV_Buffer=0.0;
								VBV_Underlfow=true;
							}
							else if (VBV_Buffer<=THRESHOLD_LOW) VBV_Low=true;
							VBV_Buffer+=VBV_Picture;
							if (VBV_Underlfow)
							{
								VBV_Underlfow=false;
								printf("%lu  UNDERFLOW !!!!!!!!!!!!!!!!!!\n",(uint32_t)floor(VBV_Buffer));
							}
							else if (VBV_Low)
							{
								VBV_Low=false;
								printf("%lu  LOW !!!\n",(uint32_t)floor(VBV_Buffer));
							}
							else printf("%lu\n",(uint32_t)floor(VBV_Buffer));
							if (VBV_Buffer>VBV_Bitrate) VBV_Buffer=VBV_Bitrate;
                            picture_size=0;
							FrameNumber++;
                        }
                    }
                }
            }
            picture_size++;
        }
    }
    fclose(fp);
	printf("\nMin value of VBV Buffer : %f at frame : %lu\n",VBV_Min,FrameMin);
    return 0;
}
When muxing for BR with Scenarist, should i remove only picture size or something else also ?
Because a stream i've muxed in Scenarist generated a buffer underflow, but according this calcul method, my min buffer value is 5228639 (Buffer and video bitrate at 24000kb).
But... I indeed find the position of this min value at the position the stream failed in Scenarist.
But this value is... far away from 0, so, either my program is doing something wrong, either... There is something i've misunderstood...
File is 23.976fps viodeo encoded with x264 with --vbv-maxrate 24000 --vbv-bufsize 24000.
According x264 help, default value of init buffer is 0.9 if you don't change this parameter.

Last edited by jpsdr; 18th November 2013 at 19:17.
jpsdr is offline   Reply With Quote
Old 20th November 2013, 20:32   #13  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 1,945
So, is 5228639 supposed to create a buffer underflow ?
jpsdr is offline   Reply With Quote
Old 21st November 2013, 04:40   #14  |  Link
drmpeg
Registered User
 
Join Date: Jan 2003
Location: Silicon Valley
Posts: 458
Your code looks okay. Remember that without HRD, Scenarist may be making entirely different assumptions about the parameters of your bitstream.

Ron
__________________
HD MPEG-2 Test Patterns http://www.w6rz.net
drmpeg is offline   Reply With Quote
Old 21st November 2013, 19:46   #15  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 1,945
I have --nal-hrd "vbr" in my x264 command line.
jpsdr is offline   Reply With Quote
Old 22nd July 2014, 16:00   #16  |  Link
friolator
Registered User
 
Join Date: Jul 2014
Posts: 10
Quote:
Originally Posted by benwaggoner View Post
Also, if you can work from how many bits each, you can pull that from a .stats file as long as you archive one from the final encoding pass.
I'm curious as to how to do this. I'm building a GUI front end for the x264 CLI for our internal use, and I'd like to build a bitrate graph to display after the encode is complete. I'm not seeing any per-frame bitrate information in the stats file. Can that be derived from the various fields displayed in the stats? If so, how?

I know we can get this from a verbose log, but I'd prefer to do it from a stats file if possible. My app's progress indicator during the encode is simply displaying the text sent to stdout (in normal mode, this shows very useful information). Setting the log to verbose messes up the display of the progress indicator while encoding, because it's outputting much more than just the progress. Plus, verbose mode generates a huge file if you're doing a feature film.

If there's a way I can grab the bit rate for each frame from the stats file, I'd love to know how.

Thanks!
friolator is offline   Reply With Quote
Old 22nd July 2014, 21:45   #17  |  Link
fionag
x264 author
 
Join Date: Jul 2014
Posts: 6
Quote:
in:0 out:0 type:I dur:2 cpbdur:2 q:19.83 aq:14.31 tex:1606319 mv:158477 misc:5972 imb:3600 pmb:0 smb:0 d:- ref:;
"tex", "mv", and "misc" (summed up) are the number of bits in the frame, I think.
fionag is offline   Reply With Quote
Old 22nd July 2014, 22:20   #18  |  Link
friolator
Registered User
 
Join Date: Jul 2014
Posts: 10
Quote:
Originally Posted by fionag View Post
"tex", "mv", and "misc" (summed up) are the number of bits in the frame, I think.
Hmm. those numbers aren't matching what I'm getting in a log file (they're close, but not the same values). for example:

Code:
Frame 1 [log: 1645 bytes] [stats: (tex:2555 + mv:3047 + misc:7206) * .125 = 1601 bytes]
Frame 2 [log: 137 bytes]  [stats: (tex:0 + mv:0 + misc:1008) * .125 = 126 bytes]
Frame 3 [log: 133 bytes]  [stats: (tex:0 + mv:0 + misc:944) * .125 = 118 bytes]
If the byte count in the log file is inaccurate and all I need to do is sum these three values, then I'm good. But I want to make sure I'm getting the right numbers. Can anyone confirm that this is the right way?

Thanks!
friolator is offline   Reply With Quote
Old 22nd July 2014, 22:27   #19  |  Link
fionag
x264 author
 
Join Date: Jul 2014
Posts: 6
If you're using 2-pass, the second pass won't update the stats file unless you use --pass 3 instead of --pass 2, so make sure you're not comparing first-pass numbers against second-pass output.
fionag is offline   Reply With Quote
Old 22nd July 2014, 22:30   #20  |  Link
friolator
Registered User
 
Join Date: Jul 2014
Posts: 10
Quote:
Originally Posted by fionag View Post
If you're using 2-pass, the second pass won't update the stats file unless you use --pass 3 instead of --pass 2, so make sure you're not comparing first-pass numbers against second-pass output.
Aha! that must be it. These are 2-pass encodes. I'll mess around with this a bit tomorrow to see if I can get the numbers to match up. Thanks!
friolator is offline   Reply With Quote
Reply

Tags
rate control, x264

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 19:37.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.