Encoding+decoding

[koliva]

Hi all,

I encode my image in the following lines:

//ENCODER
int frame_size=x264_encoder_encode( h, &nal, &i_nal, pic, &pic_out );

for(int nalcounter=0; nalcounter<i_nal; ++nalcounter)
{
//offset is set equal to 4.
memcpy(Framebuffer, nal[nalcounter].p_payload+offset, nal[nalcounter].i_payload-offset);
Framebuffersize = nal[nalcounter].i_payload-offset;
}


then I try to decode the encoded data in these lines:


//DECODER
av_init_packet(&pkt);

pkt.data=Framebuffer;
pkt.size=Framebuffersize;
lenDecoded = avcodec_decode_video2(codecCtx, av_pic, &got_picture, &pkt);


I get an error in decoding part, saying that there is no frame inside the data. Can't I decode the NALs coming from the encoder? Should I wrap the NALs somehow? What is the best way?

Thank you.

[MasterNobody]

//ENCODER
int frame_size=x264_encoder_encode( h, &nal, &i_nal, pic, &pic_out );

for(int nalcounter=0; nalcounter<i_nal; ++nalcounter)
{
//offset is set equal to 4.
memcpy(Framebuffer, nal[nalcounter].p_payload+offset, nal[nalcounter].i_payload-offset);
Framebuffersize = nal[nalcounter].i_payload-offset;
}

This part is wrong. You shouldn't remove anything from NAL-s (also I suppose they are Annex B formatted so 4 is wrong because start codes can be 3-bytes). Also by x264's ABI "the payloads of all output NALs are guaranteed to be sequential in memory" so you can simply memcpy nal[0].p_payload with frame_size (if it is not negative).

[koliva]

This part is wrong. You shouldn't remove anything from NAL-s (also I suppose they are Annex B formatted so 4 is wrong because start codes can be 3-bytes). Also by x264's ABI "the payloads of all output NALs are guaranteed to be sequential in memory" so you can simply memcpy nal[0].p_payload with frame_size (if it is not negative).

So what you are saying is that there will be no for loop, instead just put

memcpy(Framebuffer, nal[0].p_payload, nal[0].i_payload);


Actually I have already tried this by setting the offset equal to zero but I will change this code part as you suggest.
Dark Shikari told me in his email that
"x264 outputs video in annex-B format by default -- you probably have to demux (using libavformat) before sending it to
avcodec_decode_video2. " Do you have any idea how I can demux it?

[Guest]

Earlier versions of libavcodec supported "chunks mode" where you could send NALUs one-by-one into avcodec_decode_video(). Newer versions do not support it anymore and you have to pass a full picture with all the NALUs in one call to avcodec_decode_video2().

You can use libavformat to parse out full pictures (which is what DS meant by "demux") as DS suggested or you can do it yourself, as I did for a private tool I made using a recent libavcodec. Just collect the NALUs delimited by picture boundaries and send them all at once. Finding picture boundaries is covered in the AVC spec.

[MasterNobody]

So what you are saying is that there will be no for loop, instead just put

memcpy(Framebuffer, nal[0].p_payload, nal[0].i_payload);


No. I was saying "memcpy(Framebuffer, nal[0].p_payload, frame_size);"
Dark Shikari told me in his email that
"x264 outputs video in annex-B format by default -- you probably have to demux (using libavformat) before sending it to
avcodec_decode_video2. " Do you have any idea how I can demux it?
You don't need to demux Annex-B (that is what libavcodec expects from you by default). You only need to split send data by full frames but while you don't write more than one frame to your buffer after encoding you don't need this (because they already are).

[koliva]

Earlier versions of libavcodec supported "chunks mode" where you could send NALUs one-by-one into avcodec_decode_video(). Newer versions do not support it anymore and you have to pass a full picture with all the NALUs in one call to avcodec_decode_video2().

You can use libavformat to parse out full pictures (which is what DS meant by "demux") as DS suggested or you can do it yourself, as I did for a private tool I made using a recent libavcodec. Just collect the NALUs delimited by picture boundaries and send them all at once. Finding picture boundaries is covered in the AVC spec.

Thank you very much. I am currently encoding frame by frame. MasterNobody says that I don't need to parse out the full pictures since I am inputting only one frame. Do you have any comment on this?



You don't need to demux Annex-B (that is what libavcodec expects from you by default). You only need to split send data by full frames but while you don't write more than one frame to your buffer after encoding you don't need this (because they already are).

So, according to what you have said, the decoder should be able to decode the data in Framebuffer now with the following code?


//DECODER
av_init_packet(&pkt);

pkt.data=Framebuffer;
pkt.size=frame_size;
lenDecoded = avcodec_decode_video2(codecCtx, av_pic, &got_picture, &pkt);


I am still getting "no frame" error in the avcodec_decode_video2(...) line. got_picture returns always -1 while lenDecoded is 0. What do you think?

[koliva]

I don't know exactly what I did but it works now. I get >0 number in got_picture variable. And the error on the screen says:
non-existing PPS 0 referenced
decode_slice_header error
no frame!

I see the light on the other side of the tunnel now :)

[koliva]

My code works ok now. I can see the decoded images on the screen, which is what I have been trying to do. However, there is still a problem.
What I am doing is, at the encoder, there is an incoming video stream and I am encoding frames grabbed from this stream. Then I send the encoded data to the decoder and decode it and show on the screen. It works perfect for couple of frames then an error appears on the screen. This cycle repeats. I mean it starts showing the decoded images and again this error:

[h264 @ 08d8da80] non-existing PPS 0 referenced
[h264 @ 08d8da80] decode_slice_header error
[h264 @ 08d8da80] no frame!

For example, I have just done a test now. First 3 frames are perfectly encoded->sent->decoded->shown but I got this error for the fourth one.
Of course at this moment lenDecoded returns -1 and got_picture 0.

I guess I am forgetting setting some parameters at the encoder side. Do you have any comment on this issue?

Thank you.

[Guest]

Post a link to an encoded stream that shows this error when you decode it. First we need to see if the problem is in the encoded stream or in the decoder.

[koliva]

Post a link to an encoded stream that shows this error when you decode it. First we need to see if the problem is in the encoded stream or in the decoder.

That's a very good idea. I can normally open and see my frame when I open the encoded bitstream with StreamEye but this one doesn't work. Actually the file size is rather small. Since all frames are encoded with I frame, I expect to see more or less the same file size. Working frames are 216 KB but this one is 95 KB. What is your opinion?

[Guest]

What you posted is a single NALU containing a non-IDR slice.

There is no way that can play in anything. You need to have your SPS and PPS before any pictures and you need to start with an IDR slice.

[koliva]

What you posted is a single NALU containing a non-IDR slice.

There is no way that can play in anything. You need to have your SPS and PPS before any pictures and you need to start with an IDR slice.

I think I couldn't implement the encoding part correctly. Please tell me this. Before calling x264_encoder_encode API, I need to set the parameters for sure. Let say I have IPPPP configuration. These parameters should be set differently for the first frame and different for the second frame, right?

Do you know or have a sample code for SPS and PPS? I have no idea where to get them and how to set.

Thank you.

[Guest]

I don't know anything about the encoder API usage.

The encoder should generate all the required NALUs, including SPS and PPS.

How did you create that sample? Aren't you collecting all of the output of the encoder?

[koliva]

I don't know anything about the encoder API usage.

The encoder should generate all the required NALUs, including SPS and PPS.

How did you create that sample? Aren't you collecting all of the output of the encoder?

I will put here all the encoding code that I have. Perhaps some people in future needing a sample code can use this code. Thanks.



void mpStartEncodingFrame(QVector<const char *> bits, int aWidth, int aHeight, int mvGOPsize, qint32 tstamp)
{
const char **pBits = (const char **)malloc(bits.size()*sizeof(char *));

//Conversion from QVector to pointer array
unsigned int index(0);
foreach(const char *pFrame, bits)
pBits[index++] = pFrame;

//format conversion from RGB24 to YUV420.
sws_scale(convertCtx, (uint8_t**)pBits, &srcstride, 0, aHeight, pic_in.img.plane, pic_in.img.i_stride);
pic_in.i_dts=tstamp;
pic_in.i_type = X264_TYPE_I;
pic_in.i_qpplus1 = 0;

int a = Encode_frame( h , &pic_in );

outStream << (quint64)Framebuffersize;

//Write the encoded data out
}

int Encode_frame( x264_t *h, x264_picture_t *pic )
{
int i_nal, i_nalu_size;
i_nalu_size = 0;

int frame_size=x264_encoder_encode( h, &nal, &i_nal, pic, &pic_out );
if( frame_size < 0 ) { //ERROR return -1; }

memcpy(Framebuffer, nal[0].p_payload, frame_size);
Framebuffersize = frame_size;
return Framebuffersize;
}



void SetParameters(EncoderType et, int aWidth, int aHeight, int quantizationFactor, int gopSize )
{
convertCtx = sws_getContext(aWidth, aHeight, PIX_FMT_RGB24, aWidth, aHeight, PIX_FMT_YUV420P, SWS_FAST_BILINEAR, NULL, NULL, NULL);
x264_picture_alloc(&pic_in, X264_CSP_I420, aWidth, aHeight); //memory allocation for being encoded data.
srcstride = aWidth*3; //RGB stride is just 3*width
Framebuffersize=0;
x264_param_default( &param );
x264_param_default_preset(&param, "veryfast", "zerolatency"); //Default preset.

Framebuffer = (uint8_t*) malloc(MAX_FRAME_SIZE);

mux_buffer_size = 0;
mux_buffer = NULL;

//Set video resolution
param.i_width = aWidth;
param.i_height = aHeight;

//We don't need bframe
param.i_bframe = 0;
//param.i_bframe_adaptive = 0;

//Maximum dimension of the gop
param.i_keyint_min = 2;
param.i_keyint_max = 5;

param.rc.i_lookahead = 0;
param.i_threads = 1;
//param.i_sync_lookahead = 0;

//More high, more compressed but loss quality
quantization_factor = 30;

//Setting CRF
param.rc.i_rc_method = X264_RC_CRF;
//Specify quality of encoding
param.rc.f_rf_constant = quantization_factor;

//only pass1 mode for low latency
param.rc.b_stat_read = 0;
param.rc.b_stat_write = 1;

param.i_threads = 1; //The number of threads that x264 will use
param.i_width = aWidth;
param.i_height = aHeight;
param.i_fps_num = 5; //Apparently GOP size
param.i_fps_den = 1;
// Intra refres:
param.b_intra_refresh = 1;
//Rate control:
param.rc.i_rc_method = X264_RC_CRF;
param.rc.f_rf_constant = 25;
param.rc.f_rf_constant_max = 35;
//For streaming:
param.b_annexb = 1;
param.b_repeat_headers = 1;
//my params:
param.i_log_level = X264_LOG_DEBUG;
//AA added
param.i_bframe_adaptive = 0;
param.rc.i_lookahead = 0;
param.i_bframe = 0;

//Ultrafast encoding params
param->i_frame_reference = 1;
param->i_scenecut_threshold = 0;
param->b_deblocking_filter = 0;
param->b_cabac = 0;
param->i_bframe = 0;
param->analyse.intra = 0;
param->analyse.inter = 0;
param->analyse.b_transform_8x8 = 0;
param->analyse.i_me_method = X264_ME_DIA;
param->analyse.i_subpel_refine = 0;
param->rc.i_aq_mode = 0;
param->analyse.b_mixed_references = 0;
param->analyse.i_trellis = 0;

//baseline profile is selected.
x264_param_apply_profile(&param, "baseline");

//Open the encoder
if( ( h = x264_encoder_open( &param ) ) == NULL )
{
fprintf( stderr, "x264 [error]: x264_encoder_open failed\n" );
exit(1);
}

this->quantization_factor = quantizationFactor;

}

[koliva]

Small update,

I have set:
param.i_keyint_min = 1;
param.i_keyint_max = 1;

and no error appears on the screen. Therefore, I think those frames for which I am getting error are not IDR frames. With this parameter set, the encoder generates only nal.i_type = 7 frames. I think for those frames I need to add some header before sending to the decoder. Any help is appreciated.

[Guest]

As I said, the encoder produces all needed NALUs. The resulting stream should not need to have anything added to it. I asked if you are collecting all the encoder output and how you made the sample but you did not respond.

Maybe you are trying to isolate frames out of the encoded stream and send them alone to a decoder? That won't work. You need to also inject the needed SPS and PPS, and if the frame is not IDR or I, you need to decode previous frames and possibly the previous GOP as well to be able to decode the targeted frame. That is effectively a form of random access, and random access in an AVC stream is not trivial.

You are not telling us what you are doing and why. This makes it hard to help you.

[koliva]

As I said, the encoder produces all needed NALUs. The resulting stream should not need to have anything added to it. I asked if you are collecting all the encoder output and how you made the sample but you did not respond.

Maybe you are trying to isolate frames out of the encoded stream and send them alone to a decoder? That won't work. You need to also inject the needed SPS and PPS, and if the frame is not IDR or I, you need to decode previous frames and possibly the previous GOP as well to be able to decode the targeted frame. That is effectively a form of random access, and random access in an AVC stream is not trivial.

You are not telling us what you are doing and why. This makes it hard to help you.

I put my code so that you can look and see how I am creating the sample. I am not trying to do something strange. I just have frames captured from the screen and I am trying to encode, send to the other pair (which is local at the moment), decode the pictures and show them again on the screen. This is what I am trying to do. I am not collecting the NALUs. Whenever there is a NALU available from the encoder, I send it to the other pair in order to be decoded and shown.

I have solved the apparent problems but it seems that I am having more questions as I go deeper in the encoder+decoder.