H.264 : error while decoding first MB of second slice

[Boumcke]

Hi,

I'm a PhD student in a belgian university, currently working on video transmission over mobile networks. I recently encountered a problem while manipulating H.264 sequences, and I'm looking for any kind of advice that could help me understand what I'm doing wrong.

What I'm trying to achieve here is exploit the slices in H.264 to modify the content of a video sequence in the compressed domain, i.e. by changing directly the bitstream of the sequence. My goal is to be able to completely replice one slice of a sequence by a whole different sequence, providing that the dimensions of the discarded slice and those of the new sequence are identical. I've identified several parameters of the slice headers (particularly idr_pic_id) that need to be adapted in order to obtain a H.264 compliant sequence, and I have now a prototype application capable of producing a "composite" stream which is in my opinin compliant with the standard. However, my attempts to decode it fail miserably, as the modified part of the bitstream is decoded erroneously.

Here's a description of the operations my program actually does on the bitstream :
- It first parses the stream of the original sequence, waiting for a header indicating an I-frame.
- When an I-frame is detected, it parses the stream further to detect if the frame is divided in multiple slices.
- As soon as a second slice is identified, it replaces the part of the bitstream correponding to the second slice of the frame by a stream representing the sequence to be inserted in the original one.

The output of the program is then a new stream, where all the "slice 2"-related NALs have been replaced by custom created NALs with different data. As for now I'm only working on 1 I-frame sequences to test the validity of the output stream. I'm using the JM decoder to decoded the output sequence, but every attempt I've done so far fails, every time with the same conclusion : modifying the NALs data cause an error in the decoding process of the first MB of the second slice.

Here's a detailed description of what happens. Suppose we have to sequences : the first one (original sequence) is encoded in two slices (dimensions 11x6 macroblocks and 11x3 macroblocks), the second one (external sequence) is encoded in one slice only (dimensions 11x3 macroblocks). Both sequences only contain one I-frame.
- When I decode only the external sequence, I found that the values immediately following the slice header are '0000000011...', indicating a first macroblock of type I_4x4. That is indeed the type given by the JM logs for this macroblock.
- The original sequence is then passed through my prototype application which replaces the NAL corresponding to the second macroblock by a newly created NAL containing the modified external sequence's slice's header and it's data.
- When I decode this new "composite" sequence, the value immediately following the second slice's header stay unchanged '0000000011', but the JM logs indicate that the first macroblock of this slice is decoded as a I_PCM macroblock.
It's interesting to notice that even when I keep the header of the orignal sequence's second slice (the only modification is then the replacement of the slice data with the external slice data), the same error happens in the decoding process.

My current knowledge of the H.264 standard is not deep enough to allow me to understand what I'm doing wrong in my program, and why the first macroblock of the second slice in the composite sequence is incorrectedly interpreted as an I_PCM macroblock. I know this is a tricky question, as the manipulations I realize are hard to explain clearly, but any piece of advice would be greatly appreciated.

Thanks a lot if you read the whole post, and thanks even more if you're able to give me a little help...

[neuron2]

When you generate your replacement NALU, are you protecting the RBSP data with the emulation prevention bytes as required by the standard?

It would be useful for you to post links to your original and modified streams.

[Boumcke]

I just added the links to the different sequences. As far as I remember, I didn't do anything specific about the emulation prevention bytes. I thought that as I'm just copying an H.264 compliant stream into another one, I shouldn't care about this particular problem, but on second thoughts this maybe just plain stupid...
I just read the description of the emulation prevention bytes in the standard description, and this is how I understand it : if the last two bytes to be inserted in the NAL are 0x0000 and the following byte equals 0x03, then add a byte at the end of the NAL with a value of 0x03. Is this interpretation correct?

Thanks for your reply by the way.

[neuron2]

OK, I just skimmed your first post and didn't notice that your replacement NALU was already emulation prevented.

I can't look at these until later tonight because I can't access storage sites through my corporate firewall. Do you have an FTP site to upload them to?

[Boumcke]

I'm afraid I don't have one. I thought about posting as attachments to this thread, but I don't know how long it'll take until they approve it...

[Dark Shikari]

I'm afraid I don't have one. I thought about posting as attachments to this thread, but I don't know how long it'll take until they approve it...How about Mediafire (www.mediafire.com)?

[neuron2]

I will approve them instantly. :)

@Dark

That's where he did post them, but that site is blocked for me.

Anyway, I came home for lunch and will plop them on a USB stick to examine this afternoon.

[neuron2]

You allude to rewriting some slice header data. What data did you rewrite and are you sure you did the Exp-Golomb stuff correctly?

For example, the original stream has two slices. Each slice header carries the Exp-Golomb-encoded number of the first macroblock in the slice. The first slice has the first_mb_in_slice as 0x0, the second slice has 0x42. Your external stream has one slice with first_mb_in_slice 0x0. You want to put the slice from the external stream in place of the second slice of the original stream. But then you will have two slices starting at the same macroblock. You need to rewrite the slice so that it contains the correct first_mb_in_slice value. That will require Exp-Golomb decoding and reencoding, etc.

You may want to ping akupenguin to this thread, as it is closer to an encoding issue than a decoding issue.

[Boumcke]

I rewrote the first_mb_in_slice and idr_pid_id parameter, and I'm absolutely sure the methods I use to decode/encode Exp-Golomb values is correct. I tested it many times, so there's no doubt about that.
What do you mean by ping-ing akupenguin? Should I contact him directly and tell him to take a look at the thread?

[Dark Shikari]

I rewrote the first_mb_in_slice and idr_pid_id parameter, and I'm absolutely sure the methods I use to decode/encode Exp-Golomb values is correct. I tested it many times, so there's no doubt about that.
What do you mean by ping-ing akupenguin? Should I contact him directly and tell him to take a look at the thread?You could drop by #x264dev and ask.

[Boumcke]

Ok thanks, I'll try that.

[neuron2]

I rewrote the first_mb_in_slice and idr_pid_id parameter, and I'm absolutely sure the methods I use to decode/encode Exp-Golomb values is correct. I tested it many times, so there's no doubt about that. I think there is doubt!

I stepped through the decoding of the second slice header of original.264 versus composite.264. Everything is the same until I reach slice_qp_delta. For original.264 it is read as 0xfffffffb, while for composite.264 it is read as 0xfffffff2. Then the following fields are also read wrong and the total bits read for the slice header is different for the two also.

[Boumcke]

That's strange, I could have sworn my methods were correct. I'll take a look at that right now...

UPDATE : The methods are fine. The problem is that the header I create for the second NAL is not complete. I based my work on an existing code, and I naïvely thought that I wouldn't have to add anything to the NAL parsing process, but things like this don't happen in real life...All I have to do now is make sure that I can parse correctly the rest of the header, and then I should be able to produce a correct composite stream.

Anyway, I really gotta thank you for your help, neuron2. When I'm finished, I'll send you a free copy of the code...:-)

[Boumcke]

Quick update on my current situation :

I completed the header parsing process, and the custom-created header is now correctly decoded by JM software. (For the curious ones, the parsing was stopped right before slice_qp_delta, so the custom header didn't have that parameter, as well as disable_deblocking_filter_idc, slice_alpha_c0_offset_div2 and slice_beta_offset_div2 for the sequences I'm working with).

However, the final mixed sequence is still not readable by JM decoder. I checked the bitstreams immediately following what I identified to be the slice header in both the external and the mixed sequence, and I found them to be identical. Examining the logs generated by JM when decoding the mixed sequence, I found out that tough the type of the second slice's first MB is correctly identified (I_4x4), the following parameters (named intra_pred_mode) are all wrong. How can the same bitstream be decoded differently in the case of the mixed sequence? I read the H.264 specs concerning the slice data syntax and the MB layer, but I haven't been able to find an explanation.

UPDATE : I managed to get the program working. When the slice inserted in the original sequence is in the same position than the replaced slice, everything works fine. In summary : I can replace a slice in the original sequence by a slice from another sequence, providing that the two slices have the same dimensions and are in the same position in the two sequences (same first_mb_in_slice). So, something in the slice data must be dependant of first_mb_in_slice, possibly the intra4x4_pred_mode parameters, but there is nothing about such a thing in the H.264 specs. Anybody has an idea ?

[HKisd]

You are shifting the stream by writing different sized exp-golomb coded first_mb_in_slice, I think you should remove and reapply emulation byte prevention for the replacement slice.

[Boumcke]

If by remove and reapply emulation byte prevention you mean simply check for the presence of 0x000003 sequences in the header (before shifting), remove the 0x03 byte as described in the standard, and add it again if needed after shifting, I don't think it should be necessary. I checked the produced bitstream and the only 0x000003 sequences I could find were in the unmodified parts of the bitstream.

[neuron2]

UPDATE : I managed to get the program working. When the slice inserted in the original sequence is in the same position than the replaced slice, everything works fine. In summary : I can replace a slice in the original sequence by a slice from another sequence, providing that the two slices have the same dimensions and are in the same position in the two sequences (same first_mb_in_slice). So, something in the slice data must be dependant of first_mb_in_slice, possibly the intra4x4_pred_mode parameters, but there is nothing about such a thing in the H.264 specs. Anybody has an idea ? Stepping through both the decodes in the debugger and comparing them should disclose the reason for the divergence. Have you done that? Again, if you want help, you'll have to post the streams.

I would guess that first_mb_in_slice affects the value of CurrMbAddr and that the latter affects the encoding of the slice.

[Boumcke]

You got a point there, but I'm afraid I don't have the right tools to achieve this. Which software are you using to step through the decoding process?
I attached the three files again : original, external and composite sequences. If you can find some time to examine them, great, if you can explain to me an efficient manner to examine the decoding process, even better...

EDIT : I managed to find the origin of the problem. As usual, it wasn't that complicated but I kinda skipped that part of the specs. So, the problem was that I forgot to parse the cabac_alignment bits. As the external slice header and the original one don't have the same length, there has to be a different number of cabac_alignment bits in order for the data to be byte-aligned. I corrected that, and presto! It works!
Anyway thanks for every piece of advice you gave me.

Etienne