Random access question for AVC gurus

[Guest]

I'm trying to implement code to start at an arbitrary position in a bitstream and then decode and display the first usable picture. Here is an indexing of the stream:

SPS 0
PPS 30
RPS 38
IDR 56
FRM 2 0 // FRM pic_type POC/2
FRM 0 3
FRM 1 1
FRM 1 2
FRM 0 6
FRM 1 4
FRM 1 5
FRM 0 9
FRM 1 7
FRM 1 8
FRM 0 10
FRM 0 11
RPS 414390
IDR 414408
FRM 2 0
FRM 0 3
FRM 1 1
FRM 1 2
FRM 0 6
FRM 1 4
FRM 1 5
FRM 0 9
FRM 1 7
FRM 1 8
FRM 0 10
FRM 0 11
RPS 834243
IDR 834261
FRM 2 0
FRM 0 3
FRM 1 1
FRM 1 2
FRM 0 6
FRM 1 4
FRM 1 5
FRM 0 9
FRM 1 7
FRM 1 8
FRM 0 10
FRM 0 11
...

Note that there is only one set of SPS/PPS at the beginning. But that's OK, I pick those up when I open the file and activate them after seeking to the random point in the stream. The problem I have is that there are recovery point SEIs immediately preceding IDRs. If I randomly access at a point between the RPS and the IDR, I start decoding at the IDR and I get a corrupted picture. But if I start at the immediately preceding RPS, the picture is fine. In the RPS, we have:

recovery_frame_cnt = 0
exact_match_flag = 0
broken_link_flag = 1
changing_slice_group_idc = 0

What I don't understand is why I can't start at the IDR and get a good picture. And why send a recovery point when an IDR seems sufficient? Any clarification from the gurus will be appreciated.

[mediator]

from my understanding, an IDR is always an recovery point, which would mean that the recovery point SEI is functionality-wise redundant. Probably it would help you understanding what's going on by looking inside the decoder for both cases (starting decoding with and without recoevery point SEI, respectively). Maybe the decoder is not flushed properly prior the random access or it's something else decoder specific...

[Sergey A. Sablin]

could you please paste a link to this file? or upload it somewhere?

[Guest]

The stream is here:

http://neuron2.net/misc/holzi.264

[Sergey A. Sablin]

The stream is here:

http://neuron2.net/misc/holzi.264

Donald, could you please clarify at which IDR at given stream you have corrupted picture?
I checked almost all IDR and all subsequent pictures decoded fine. At least it seems so for me :).

Moreover RPS indicates that all pictures after a picture in an access unit associated with RPS should be decoded correctly (or approximately correctly) if you'll start decoding process from that picture.

[Guest]

Thank you, Sergey, for taking the time to look at it. Just a little while ago, I saw a similar problem in another stream, but it doesn't have the RPSs and has only IDRs. So taking that with your result, it seems that my random access method needs to be checked more carefully. I suppose I just goofed up something and the RPS-IDR thing is a red herring. I'll report back what I find.

Do you agree that the RPSs are redundant for this stream?

[Sergey A. Sablin]

Thank you, Sergey, for taking the time to look at it. Just a little while ago, I saw a similar problem in another stream, but it doesn't have the RPSs and has only IDRs. So taking that with your result, it seems that my random access method needs to be checked more carefully. I suppose I just goofed up something and the RPS-IDR thing is a red herring. I'll report back what I find.

btw, what the numbers right after RPS and IDR mean? they look pretty strange for me...
FRM 0 11
RPS 414390
IDR 414408
FRM 2 0

Do you agree that the RPSs are redundant for this stream?
yes I think so too. But maybe this is requirement of some application or receiving device - who knows.

[Guest]

btw, what the numbers right after RPS and IDR mean? they look pretty strange for me...
FRM 0 11
RPS 414390
IDR 414408
FRM 2 0

For SPS, PPS, RPS, and IDR, the number is the offset into the source file of the location of the NALU. For FRM, the first is the pic_type and the second is the POC/2.

[Guest]

OK, I found the source of the problem. First I thought it was a libavcodec problem. Finally, I found it was caused by the trackbar used to scroll to random access points! It was generating multiple events and kicking off multiple decode threads with the same codec context. Oops. I discovered it by implementing the right arrow to just step to the next access point. That always decoded properly because it kicked off just one thread.

[Sergey A. Sablin]

OK, I found the source of the problem. First I thought it was a libavcodec problem. Finally, I found it was caused by the trackbar used to scroll to random access points! It was generating multiple events and kicking off multiple decode threads with the same codec context. Oops. I discovered it by implementing the right arrow to just step to the next access point. That always decoded properly because it kicked off just one thread.

fine! hope you'll points us soon to the streams which use RPSs for real purpose - not as in given case :)

[Guest]

hope you'll points us soon to the streams which use RPSs for real purpose - not as in given case I have such streams but I am at work now. I will upload some for you tonight.

[Guest]

My theory about multiple threads turned out to be wrong! The thread sync was working fine, so I was left baffled again. Now I have discovered the REAL reason, and it is highly instructive for people interested in random access. :)

My method was to pick up the position in the bitstream to start from based on the GUI slider event. So it could be random as far as the bitstream is concerned. My method was to start parsing NALUs starting at the random position and stop at the first SEI recovery point NALU or IDR NALU and then start decoding from there. I found that if the scan stopped at a recovery point it always displayed OK. But if it started at an IDR it sometimes generated a bad display frame. That was what I posted as my original problem.

Here's how it can go wrong: If the IDR frame is made up of multiple slices, then your random start can be at an IDR slice beyond the first one! It can never go wrong if you start at the recovery point because it precedes all the IDR slices and so you decode them all.

My solution is to check the first_mb_in_slice field of the IDR slice header. It must be macroblock 0 if it is the first slice. So you scan until you see the first slice of an IDR. Yes, I know that slices can be sent out of order, but I have never seen such a stream and I'll cross that bridge when I get to it. :) If anybody knows a better solution, please speak up.

The first_mb_in_slice field is Exponential-Goulomb coded but the code is simple, because for that coding if the field is 0, then the first bit must be set, so we have simply this:

case NALU_TYPE_IDR:
if (p[1] & 0x80)
{
had_start = 1;
break;
}
else
continue;

This was a real PITA to debug. I hope it helps anybody trying to do random access in AVC video.

[Guest]

@Sergey,

Here is a stream with only recovery point SEIs and no IDRs at all. My algorithm works fine with it.

http://neuron2.net/misc/bbc.hd.264

[Sergey A. Sablin]

@Sergey,

Here is a stream with only recovery point SEIs and no IDRs at all. My algorithm works fine with it.

http://neuron2.net/misc/bbc.hd.264

:thanks:

[guada2]

but I have never seen such a stream and I'll cross that bridge when I get to it

You are right, when you see order of I slice, B slice, P slice.
IBBP/BBP/BBP/BB....etc

H264 AUD ??????? what is it exactly?

[Guest]

You are right, when you see order of I slice, B slice, P slice. IBBP/BBP/BBP/BB....etc I was talking about multiple slices for one picture.

H264 AUD ??????? what is it exactly? Assuming you are talking about access unit delimiters, they are perfectly well described in the specification, to which you may refer.

[guada2]

I was talking about multiple slices for one picture.
sorry.

Many thanks neuron2.
Bye :) .

[Guest]

I've got to get the random access working for all my test streams before I can release alpha 4, which will have the DGIndex-like seeking around without an index. But I keep running into weird situations brought about by the power of AVC. Here's the latest snafu.

I have a stream that sends multiple PPS's at each recovery point:

SPS 11423729
PPS 11423772
PPS 11423780
PPS 11423788
RPS 11423796
FRM 2 0
FRM 1 13
FRM 1 14
FRM 1 15
FRM 0 4
FRM 1 1
FRM 1 2
FRM 1 3
FRM 0 8
FRM 1 5
FRM 1 6
FRM 1 7

They have IDs 0, 1, and 2. Apparently they specify different entropy encoding modes so that the pictures can dynamically select the encoding mode.

So when I initially load up the stream, I have to save all the PPS's so that I can inject all of them when I do a random access. Currently I just saved the first one and stopped.

Apparently all this multiple SPS/PPS stuff causes a lot of hassles. Lots of players just ignore it, but I want to do things right. :)

When I fix this, I think alpha 4 will be usable.

----------

http://www.freepatentsonline.com/20060233260.html

Abstract: The presently preferred methods of the present invention take advantage of the syntax structure of H.264 and adaptively switch the entropy coding mode between CABAC and CAVLC. In those profiles of H.264 where CABAC is supported such as the Main profile and the High profiles (HP, Hi10, Hi422, Hi444), the entropy coding mode to be used is specified via a picture parameter set ("PPS"), where a particular PPS is specified for each slice in its header ("slice header"). The encoders of the present invention provide a multiple-PPS for both the CAVLC and CABAC, so that bitstreams are produced by both CAVLC and CABAC. CAVLC, having a simpler algorithm structure, is implemented as a part of the video pipe, where each single video pipe stage is designed to operate within a pre-defined number of clock cycles. The video pipe generates SE at pre-scheduled instances buffered in FIFO. The SE serves as input to CABAC, where CABAC encodes the SE in an event-driven fashion. During this process, CABAC is monitored to determine if the CABAC engine is able to keep up with the video pipe. The video pipe, implemented using CAVLC, by design will complete the encoding process within the allotted time. However, CABAC, although having better encoding efficiency and where its generated output is more desirable, its encoding completion time is uncertain and may require more time than it is allowed. Thus if CABAC cannot complete the encoding process within the allotted time, the bitstream generated by CABAC would not be complete and it cannot be used. If CABAC can complete the encoding process within the allotted time, the CABAC engine would be allowed to complete the encoding process and its generated output would be used as the bitstream of choice.