A rookie question regarding on B frame in AVC

[jasonme]

Hello guys, I have a rookie question on B frame coding in AVC, please help me clarify this, thanks in advance.

In AVC, the parameter NumberOfReferenceFrames defines the maximum number of reference frames an inter frame can use for prediction. But what is the maximum number of reference frames for a B frame? Is it still NumberOfReferenceFrames or 2*NumberOfReferenceFrames?

Since I'm trying to allocate memory pool for reference frames, so this is very important for me. Thank you very much for your kindly answers!

[akupenguin]

Still NumberOfReferenceFrames.
The only difference between P- and B-frames is that in B-frames 1 block can use 2 of the reference frames at once.

[jasonme]

Thank you for clarifying me that.

There is another thing that's confusing me now. How to control play flow of a H.264 bit-stream. For example, If I have a bit-stream of format I P B1 B2 B3 P, then I know that after decoding I frame, it can be displayed. Then the P frame is gonna wait there until next P frame decoded for display. But the display order of B frame is not such a clear case. So along with the decoding flow, how can I control the display flow to display the video in the correct order?

Thank you guys a lot for everything. These are all basic questions, hope you guys don't mind me asking these things here.

[akupenguin]

In each frame header, the encoder writes the frame number in display order.
So the stream goes I0 P4 B1 B2 B3 P8 ..., where the number is not just my annotation but actually in the stream.

[Sergey A. Sablin]

Thank you for clarifying me that.

There is another thing that's confusing me now. How to control play flow of a H.264 bit-stream. For example, If I have a bit-stream of format I P B1 B2 B3 P, then I know that after decoding I frame, it can be displayed. Then the P frame is gonna wait there until next P frame decoded for display. But the display order of B frame is not such a clear case. So along with the decoding flow, how can I control the display flow to display the video in the correct order?

Thank you guys a lot for everything. These are all basic questions, hope you guys don't mind me asking these things here.

check out picture_order_counter (POC) from slice header and dbp_output_delay from PictureTiming SEI message.

[jasonme]

Got it, thanks. This helps a lot with my player. Thank all you guys very much for the help!

[jasonme]

Wait, I still have one more problem.....

In JM software, there is a parameter:"BReferencePictures", which indicates whether B frames can be used as a reference frame or not. But surprisingly, this info is not written to the bit-stream. Then how can I know on the decoder side that whether I should keep decoded B frames in the decoded picture buffer as a reference frame?

Thanks again!

[akupenguin]

There is no "whether B frames can be used as a reference frame or not". Each individual frame can be referenced or not, indicated by the NAL priority. "disposable" = not referenced, the other 3 possible settings are all referenced. Also look at "memory control operations" which can manipulate the DPB beyond just choosing which pictures get added.

[jasonme]

There is no "whether B frames can be used as a reference frame or not". Each individual frame can be referenced or not, indicated by the NAL priority. "disposable" = not referenced, the other 3 possible settings are all referenced. Also look at "memory control operations" which can manipulate the DPB beyond just choosing which pictures get added.

So can I put it this way, if a NAL unit has the property of nal_ref_idc=0, then this frame is a reference frame, but if not, this frame is a non-reference frame. Am I right about this?

[akupenguin]

other way around.

[jasonme]

Ahh, got it, thank you so much!

[jasonme]

As the AVC player process goes on, more rookie questions are on my way. First I would like to thank your patience to let me keep throwing these stuff to your guys. Thank you very much for this.

This time, my question is that if I have a sequence decoded in the following order: I1 P5 B3 B2 B4, the number indicate the display order of the frame. But for B3, the frame is marked disposable. By my earlier questions, this frame should be decoded and discarded not using as a reference frame. But I can't really discard it right? I need to hold the decoded B3 frame a little bit longer to wait for B2 get decoded and displayed. Then I can display B3 and release the memory it occupies. Thus, by defining a frame disposable, it only means that I'll not use it for reference but I can't give the frame up until until it is displayed for as long as it takes.

Am I right about this? Or there is no such way to generate a sequence I described above. To my best memory, SVC has sequences like this I believe. I don't know what about how this thing is in the main profile AVC.

Thank you very much for the help!

[Sergey A. Sablin]

As the AVC player process goes on, more rookie questions are on my way. First I would like to thank your patience to let me keep throwing these stuff to your guys. Thank you very much for this.

This time, my question is that if I have a sequence decoded in the following order: I1 P5 B3 B2 B4, the number indicate the display order of the frame. But for B3, the frame is marked disposable. By my earlier questions, this frame should be decoded and discarded not using as a reference frame. But I can't really discard it right? I need to hold the decoded B3 frame a little bit longer to wait for B2 get decoded and displayed. Then I can display B3 and release the memory it occupies. Thus, by defining a frame disposable, it only means that I'll not use it for reference but I can't give the frame up until until it is displayed for as long as it takes.

Am I right about this? Or there is no such way to generate a sequence I described above. To my best memory, SVC has sequences like this I believe. I don't know what about how this thing is in the main profile AVC.

Thank you very much for the help!

the sequence you shown is similar to hierarchical B-frame coding (or pyramid coding), when B3 was encoded before B2 and B4 and was used as reference to those two. But in this case it should not be disposable neither in AVC nor in SVC as SVC share same features with AVC for base layer coding.

In case B3 isn't used for reference, but there is a gap between P5 and B3 in pic_order_cnt, you should store this non reference B-frame in DPB until it's time to display will come.

[edit] oops, sorry - gap between I1 and B3, not between P5 and B3

[jasonme]

the sequence you shown is similar to hierarchical B-frame coding (or pyramid coding), when B3 was encoded before B2 and B4 and was used as reference to those two. But in this case it should not be disposable neither in AVC nor in SVC as SVC share same features with AVC for base layer coding.

In case B3 isn't used for reference, but there is a gap between P5 and B3 in pic_order_cnt, you should store this non reference B-frame in DPB until it's time to display will come.

[edit] oops, sorry - gap between I1 and B3, not between P5 and B3


So I am right about having to keep disposable frames in some certain circumstances then.

This will lead to other issue. If there is a lot of B frames between I/P frames and they are coded in pyramaid pattern. But the weird thing is that they are all disposable frames. How much memory I have to allocate to deal with this situation? Since we only keep the buffer for display not for reference, so I guess that the more B frames we have in a row, the more buffer we need. This will definitely make the hardware solution harder right? Especially for hardware with limited RAM.

Then how can we avoid this thing form happening? Thanks!

[akupenguin]

See num_reorder_frames which is an optional field in the SPS header. If it is not present, the standard gives a rule to infer an upper bound. And in any case it will never be more than 16 frames.
It's not directly related to disposability: Even if all frames are referenced, they could still be nontrivially reordered.

[jasonme]

So basically, to be safe I better keep a buffer of 16 reference frames and 16 display frames buffer to make sure that all sequences are decodable, right? That's a lot of memory for high definition AVC decoding.

[akupenguin]

Up to 16 frames total, and at any given time some of them will be referenced or in the reorder buffer or both.
The main factor that determines the bound is Level. Each of the Levels puts an upper bound on the amount of memory you have to allocate to decoded frames. So higher resolution means fewer frames at a given Level.

[jasonme]

This is great news for me, thank you very much for your help all the time and also for Sergey A. Sablin. I may have further questions. So I'll paste my other issues here too. Thanks in advance for everything!

[Sergey A. Sablin]

See num_reorder_frames which is an optional field in the SPS header.
max_dec_frame_buffering.

[akupenguin]

num_reorder_frames is how many frames might be reordered, which is the question I answered. max_dec_frame_buffering was the next question, involving amounts of memory.

[jasonme]

Sorry for my poor understanding of the AVC architecture, I have to decide my moves on the buffer control at this point, therefore I have to ask again.

I mean, if I have a bitstream with max_ref_frames of 16, does that means I have to buffer at least 16 frames which is decoded previously? I mean if we consider high profile stream with a high level. Thank you!

[Sergey A. Sablin]

Sorry for my poor understanding of the AVC architecture, I have to decide my moves on the buffer control at this point, therefore I have to ask again.

I mean, if I have a bitstream with max_ref_frames of 16, does that means I have to buffer at least 16 frames which is decoded previously? I mean if we consider high profile stream with a high level. Thank you!

number of reference frames has no influence on buffering. read Annex C - Hypothetical reference decoder. That's the place about buffering and timing. Mind that not every encoder output buffering period and picture timing SEI, which can be used for your purposes.
and btw, there is no high level in avc. if you're in avc - use avc terminology.

[jasonme]

Thank you for replying and thank you again, I'll use avc terminology the best I could. I think I know what you mean now. I'll look deeper into the standard and mind my way out. Thank you!

[Trahald]

I have a question (i jumped onto this thread since it is b-frame centric ) With the little ive been reading about pyramid coding, it seems being able to do --bframes 2 --b-pyramid isnt legal? or actually any non even b count (1 is already excluded ) doesnt seem to be in line with pyramid coding.

[Sergey A. Sablin]

I have a question (i jumped onto this thread since it is b-frame centric ) With the little ive been reading about pyramid coding, it seems being able to do --bframes 2 --b-pyramid isnt legal? or actually any non even b count (1 is already excluded ) doesnt seem to be in line with pyramid coding.

it seems x264 allows to use --b-pyramid with more than 1 b-frame. So IBbP (display order) will be produced in this case.

generally speaking, there is no even such a term as pyramid b-frames. they are called generalized b-frames, which means they may be either reference or not and there may be different ways to exploit this ability to raise the quality a bit.
Structures like IbBP or IBBbBBbP or any other are pretty valid, while one couldn't call them "pyramid". Another question is whether such structures of any use in practice, but nevertheless they are valid.