Extraction of frames in H.264 encoded data

[medha]

Hi All,
Usually H.264 data comes in NAL packets and each NAL packet consists partial frame data .
Our decoder works on frames basis -- So we need to give the input data in terms of frames but not in the form of NAL units..
How can I get a frame based on NAL units..

Thanks
Medha

[Dust Signs]

The standard will tell you...

7.4.1.2.3 Order of NAL units and coded pictures and association to access units
An access unit consists of one primary coded picture, zero or more corresponding redundant coded pictures, and zero or
more non-VCL NAL units. The association of VCL NAL units to primary or redundant coded pictures is described in
subclause 7.4.1.2.5.
The first of any of the following NAL units after the last VCL NAL unit of a primary coded picture specifies the start of
a new access unit.
- access unit delimiter NAL unit (when present)
- sequence parameter set NAL unit (when present)
- picture parameter set NAL unit (when present)
- SEI NAL unit (when present)
- NAL units with nal_unit_type in the range of 13 to 18, inclusive
- first VCL NAL unit of a primary coded picture (always present)
The constraints for the detection of the first VCL NAL unit of a primary coded picture are specified in subclause
7.4.1.2.4.
[..]

7.4.1.2.4 Detection of the first VCL NAL unit of a primary coded picture
Any coded slice NAL unit or coded slice data partition A NAL unit of the primary coded picture of the current access
unit shall be different from any coded slice NAL unit or coded slice data partition A NAL unit of the primary coded
picture of the previous access unit in one or more of the following ways.
- frame_num differs in value.
- field_pic_flag differs in value.
- bottom_field_flag is present in both and differs in value.
- nal_ref_idc differs in value with one of the nal_ref_idc values being equal to 0.
- frame_num is the same for both and pic_order_cnt_type is equal to 0 for both and either pic_order_cnt_lsb differs in
value, or delta_pic_order_cnt_bottom differs in value.
- frame_num is the same for both and pic_order_cnt_type is equal to 1 for both and either delta_pic_order_cnt[ 0 ]
differs in value, or delta_pic_order_cnt[ 1 ] differs in value.
- nal_unit_type is equal to 5 for both and idr_pic_id differs in value.
NOTE – Some of the VCL NAL units in redundant coded pictures or some non-VCL NAL units (e.g. an access unit delimiter
NAL unit) may also be used for the detection of the boundary between access units, and may therefore aid in the detection of the
start of a new primary coded picture.

Dust Signs

[medha]

Hi Dust Signs,
Thanks for the information..
Please correct me, if i am wrong..
So we need to check for the Access Unit Type (NAl_Type - 9) in-order to get the start of New picture..?

I have few more doubts...
(1) As per the H.264 dump -- Initially there are no SPS/PPS data -- It contains only AUD, SEI only.. Do i need to flush this data, until i get first SPS/PPS data..?
(2) frame_num --> As per the spec, it is in Slice Header ---> But how to recognize the start of a frame..
(3) Below is the sample H.264 Parsed data

00 00 00 01 09 30 00 00 00 01 06 01 02 14 40 80 00 00 00 01 41 9A etc...

If we parse this,

00 00 00 01 09 30 --> Access Unit Delimiter (As NAL_Type - 9)
00 00 00 01 06 01 02 14 40 80 --> SEI Information (As NAL_Type - 6
00 00 00 01 41 9A --> Coded Slice of Non-IDR Picture (As NAL_Type - 1)

So, as you said, i need to accumulate data once NAL_TYpe - 9 is found and accumulate till again another Access unit is found..? rt..?

If so, what about SPS and PPS set..? What is the exact siginificance of those values..?

P.S. Basically my decoder takes the input data in terms of frames (not as per NAL Units), So i need to identify each from among (one or more) NAL units and make it as a frame and send them to decoder.. This is the problem..

Thanks,
Medha

[Dust Signs]

If you already have access unit delimiters in your input data stream you just have to pass the data between two access unit delimiters to your decoder (if this is really how the decoder expects its input). Otherwise, please stick to the mechanism for frame determination described in the two passages I quoted. You may have to read it carefully, as it is a typical "standard" text which may be quite hard to understand.
Regarding your PPS and SPS question, you should also have a look into the standard: 7.4.1.2.1 Order of sequence and picture parameter set RBSPs and their activation
NOTE – The sequence and picture parameter set mechanism decouples the transmission of infrequently changing information
from the transmission of coded macroblock data. Sequence and picture parameter sets may, in some applications, be conveyed
"out-of-band" using a reliable transport mechanism.
A picture parameter set RBSP includes parameters that can be referred to by the coded slice NAL units or coded slice
data partition A NAL units of one or more coded pictures.
When a picture parameter set RBSP (with a particular value of pic_parameter_set_id) is first referred to by a coded slice
NAL unit or coded slice data partition A NAL unit (using that value of pic_parameter_set_id), it is activated. This picture
parameter set RBSP is called the active picture parameter set RBSP until another picture parameter set RBSP is
activated. A picture parameter set RBSP shall be available to the decoding process prior to its activation.
Any picture parameter set NAL unit containing the value of pic_parameter_set_id for an active picture parameter set
RBSP shall have the same content as that of the active picture parameter set RBSP unless it follows the last VCL NAL
unit of a coded picture and precedes the first VCL NAL unit of another coded picture.
A sequence parameter set RBSP includes parameters that can be referred to by one or more picture parameter set RBSPs
or one or more SEI NAL units containing a buffering period SEI message.
When a sequence parameter set RBSP (with a particular value of seq_parameter_set_id) is first referred to by activation
of a picture parameter set RBSP (using that value of seq_parameter_set_id) or is first referred to by an SEI NAL unit
containing a buffering period SEI message (using that value of seq_parameter_set_id), it is activated. This sequence
parameter set RBSP is called the active sequence parameter set RBSP until another sequence parameter set RBSP is
activated. A sequence parameter set RBSP shall be available to the decoding process prior to its activation. An activated
sequence parameter set RBSP shall remain active for the entire coded video sequence.
Any sequence parameter set NAL unit containing the value of seq_parameter_set_id for an active sequence parameter set
RBSP shall have the same content as that of the active sequence parameter set RBSP unless it follows the last access unit
of a coded video sequence and precedes the first VCL NAL unit and the first SEI NAL unit containing a buffering period
SEI message (when present) of another coded video sequence.
NOTE – If picture parameter set RBSP or sequence parameter set RBSP are conveyed within the bitstream, these constraints
impose an order constraint on the NAL units that contain the picture parameter set RBSP or sequence parameter set RBSP,
respectively. Otherwise (picture parameter set RBSP or sequence parameter set RBSP are conveyed by other means not specified
in this Recommendation | International Standard), they shall be available to the decoding process in a timely fashion such that
these constraints are obeyed.
During operation of the decoding process (see clause 8), the values of parameters of the active picture parameter set and
the active sequence parameter set shall be considered in effect. For interpretation of SEI messages, the values of the
parameters of the picture parameter set and sequence parameter set that are active for the operation of the decoding
process for the VCL NAL units of the primary coded picture in the same access unit shall be considered in effect unless
otherwise specified in the SEI message semantics.

Hope this helps
Dust Signs