How to tell if an I frame is seekable?

[neuron2]

I've been looking at some streams recorded by the Hauppage HD PVR. Unless you configure it to use IDRs, it uses no IDRs by default and generates I pictures that apparently are seekable. I say apparently because as an experiment I modified DGAVCDec to treat I frames as IDRs and everything worked fine for these streams. Note that there are no SEIs preceding these I frames to mark them as recovery points.

My understanding is that:

i-frames are not necessarily keyframes because after an i-frame, you can find a p-frame that refers to a frame preceding that i-frame. In order to identify a safe frame to seek to, H.264 defines a new frame type, called the IDR-frame. An IDR-frame is an i-frame that prevents the next p-frames from using any prior frames as their reference. Two consecutive IDR-frames determine the boundaries of a GOP.

So, my question: Is there a syntax element that I can check to see if a given I frame is in fact seekable, i.e., that the following P frames don't refer to frames earlier than the I frame?

[akupenguin]

Encoders that generate seekable non-IDR frames are recommended to insert "recovery point" SEI messages.
But there is no required syntax element that inherently marks every seekable frame.

[neuron2]

Thank you. That's what I was afraid of. Would it be possible to inspect the P frame itself to check its references or something like that?

Failing that, I'll have to add an option "Treat I Frames as IDR Frames" to make it possible to support these streams.

Any other thoughts?

[akupenguin]

For most normal streams, it should be possible to start at an I-frame, assume an empty DPB, run the DPB update algorithm for each subsequent frame, and check whether any frame refers to frames that aren't in your assumed DPB (either explicitly by MMCO, or implicitly by num_ref_idx_l0_active) until the DPB is full.
But even that isn't guaranteed: A sufficiently perverse stream could keep a long-term ref from before the I-frame. That would change the DPB update algorithm without ever being explicitly mentioned again, and you'd never know.

[neuron2]

I'll go with the option then. :)

[gruntster]

Not sure if this is helpful. Handbrake determines if a non-IDR frame is an I-frame using the following code. Are you already doing something similar?


if ( stream->ts_stream_type[0] == 0x1b )
{
// we have an h.264 stream
for (i = 13 + adapt_len; i < 188; i++)
{
strid = (strid << 8) | buf[i];
if ( ( strid >> 8 ) == 1 )
{
// we found a start code - remove the ref_idc from the nal type
uint8_t nal_type = strid & 0x1f;
if ( nal_type == 0x05 )
// h.264 IDR picture start
return 1;

if ( nal_type == 0x01 )
{
// h.264 slice: has to be start MB 0 & type I (2, 4, 7 or 9)
uint8_t id = buf[i+1];
if ( ( id >> 4 ) == 0x0b || ( id >> 2 ) == 0x25 ||
id == 0x88 || id == 0x8a )
{
return 1;
}
}
}
}
// didn't find an I-frame
return 0;
}

[neuron2]

Thanks, but the problem is not recognizing I frames; it is determining whether any following P frames reference frames prior to the I frame, i.e., is the I frame seekable.

[Sergey A. Sablin]

For most normal streams, it should be possible to start at an I-frame, assume an empty DPB, run the DPB update algorithm for each subsequent frame, and check whether any frame refers to frames that aren't in your assumed DPB (either explicitly by MMCO, or implicitly by num_ref_idx_l0_active) until the DPB is full.
But even that isn't guaranteed: A sufficiently perverse stream could keep a long-term ref from before the I-frame. That would change the DPB update algorithm without ever being explicitly mentioned again, and you'd never know.

If I understand correctly Don said he indexing something somehow, well at least he said so about PPS, so I guess it might be possible to keep track of used long term refs, or even better - the oldest ref for each picture or so far. If of course anybody cares so much about such streams.

[neuron2]

If of course anybody cares so much about such streams. These streams come out of the only affordable HD capture box, so there's good reason to care. I have it all working now. For these streams you enable the option called "Treat I as IDR". I assume that if you enable this, you know that all I frames are seekable. Not a perfect solution, but it makes the streams usable without jumping through Graphedit hoops.

[akupenguin]

I guess it might be possible to keep track of used long term refs, or even better - the oldest ref for each picture or so far.
Sure, if you don't mind scanning the file from start to finish before deciding anything.

[LoRd_MuldeR]

Sure, if you don't mind scanning the file from start to finish before deciding anything.

Well, DGAVCIndex does do that anyway when creating the index...

[Sergey A. Sablin]

Sure, if you don't mind scanning the file from start to finish before deciding anything.

well, not to finish, but till required point only :) ... and in general case you gotta do this anyway - mind discussion on PPS with identical IDs and different content.

[bob0r]

I would just like to add, i am really loving how dgavcdec is developing. All these "bugs and errors" i had with H.264 streams now seem to surface one by one.

- Where ffdshow failed
- Where coreavc failed
- Where mplayer failed with .ts files
- Where xmuperpro failed with .ts files
- Where Haali failed with .ts files

They all seem to work or will be working soon in dgavcdec, where libavcodec is the biggest "problem" for now.

Really good job.... and who knows, one day Neuron and Pengvado might join force for the ultimate H.264 decoder?
( The ultimate part would be mostly FREE :D )

[bobololo]

Encoders that generate seekable non-IDR frames are recommended to insert "recovery point" SEI messages.
But there is no required syntax element that inherently marks every seekable frame.

Just my 2c comments.

While the standard indicates indeed that some syntax elements are optional in the elementary stream and are not required to decode properly the pixels, it also mentions that some information must be then conveyed by another mean (the system layer for instance). Otherwise, it would be not easy or even possible for an *application* to deal with such streams. I mean if no information about frame rate, access unit delimiter, recovery point, time stamp, etc. are signaled anywhere (since they are all optional at the elementary layer) how are we supposed to handle them properly?

As a result, I'd be tempted to say that those streams are not practically seekable (unless you parse them so deeply that you almost implement a significant part of a decoder ;)) and I would gently provide that feedback to hauppauge which hopefully would aknowledge the issue and add the recovery_point sei like any serious encoder :).

That said, the "I-frame-is-seekable" option is however a useful work around since it allows the handling of the legacy "borked" files.