Q: Proper way of creating NALs of encoded frame.....

[SmutjeSanji]

Hello,
I hope you can give me some hints. Till yet it was enough to use Google (to search forums etc) for learning but Iam really stuck right now and need your ideas.

-- What iam doing in general and what the problem is

Ive devoloped a server app so far that it is encoding a video with x264 - the video should be a live stream later, now I use videofiles -, packeting the encoded frames in NALs (x264) and sending it to a client who asks for a stream. And all steps happens in real time which means f.e. that after encoding 3-4 frames and packeting in NALs the server sends the NALs immediatly to the client.

The client is developed so far that it gets the stream - in form of NAL packets - and tries to decode it (with ffmpeg) - I give the NALs directly to ffmpeg without modifing them.

The problem is that on client side ffmpeg cant decode the NALs it gets - its telling me that no frame is found inside the NAL packets.

I know that you will answer me just in topics which involves x264, so Iam just asking you about that and not about the other parts of the program.

-- What I want to ask: Do I create NALs correctly?

As codebasis for the x264 encoding part of the server app I ve used x264.c and modified it so that the encoded frames are not written to harddisk anymore. Instead of that I create NAL packets of them and put them in a deque which another thread handles to send the NALs through network.

The modified "Encode_frame()" looks like this:

static int Encode_frame( x264_t *h, x264_picture_t *pic, int64_t *last_dts,
                         std::deque<NALBuffer>* nalQueue, pthread_mutex_t* lockQueue )
{
    x264_picture_t pic_out;
    x264_nal_t *nal;
    int i_nal;
    int i_frame_size = 0;

    i_frame_size = x264_encoder_encode( h, &nal, &i_nal, pic, &pic_out );

    if( i_frame_size < 0 )
    {
        fprintf( stderr, "x264 [error]: x264_encoder_encode failed\n" );
        std::cout << "i_frame_size: " << i_frame_size << std::endl;
        return -1;
    }

    if( i_frame_size )
    {
        for(int i = 0; i < i_nal; i++)
        {

            int long_startcode = !i || nal[i].i_type == NAL_SPS || nal[i].i_type == NAL_PPS;
            NALBuffer nalBuffer;
            nalBuffer.size = x264_nal_encode( h, nalBuffer.buffer, &nal[i], long_startcode );
            nalBuffer.lastNAL = ((i+1) == i_nal) ? true : false;

            // One NAL (of probably more) of a videoframe is created

            pthread_mutex_lock(lockQueue);
            nalQueue->push_front(nalBuffer);
            pthread_mutex_unlock(lockQueue);

            // NAL is added to (sending) queue - another thread is reading the queue and send the NALs
        }

        *last_dts = pic_out.i_dts;
    }
    
    return i_frame_size;
}

Its working. But as I ve said ffmpeg cant decode the NALs by telling me that there are no frames inside it.

The questions are now: Do the NALs are created correctly? Or do I have misconfigurations of ffmpeg on client side which cant decode the NALs? Or do the NALs get corrupt during transport - but right now Iam just testing at localhost, so that this shouldnt be the case?

[onesloth]

try doom10

[Warperus]

1) Client should be able to decode video as far as it gets enough frames to cover the limit for b-frames reference.
As far as I remember b-frame can reference any frame in VBV buffer (before or after it). Frames referenced by b-frame are packed and saved to file before corresponding b-frame.
So if you send not enough frames decoder can't start.

2) NAL packet size is probably much less than even a single frame so I'm nearly sure one NAL is not enough for decoder to start. From your text I can assume you are trying to feed decoder as soon as you get new NAL (it might be incorrect assumption though).
I don't know of ffmpeg ability to handle NAL packets...
If I were you I'd check ffmpeg documentation for possibility and limitations of this method. I'm pretty sure it says all the criteria for decoding your client should comply before calling ffmpeg.

[SmutjeSanji]

Hi, thanks for your hints. Ive posted my problem at doom10 and got some hints. If youre interested in youre welcome: http://doom10.org/index.php?topic=534.0