libavcodec has parallel h.264 decoding support

[bond]

as the title says, thanks to Andreas Öman libavcodec now can make use of dual core processors when decoding h.264
http://svn.mplayerhq.hu/ffmpeg/trunk/libavcodec/h264.c?sortby=date&view=log

note: depends on having a thread library activated at config time, and on having a source encoded with multiple slices

have fun :)

[Sagekilla]

Wow... I can't wait for someone to release a patched libavcodec! With this, I should be able to decode 1080p streams with no troubles..

[ChronoCross]

doesn't x264 not use slices anymore for parallelism? or perhaps I'm confused.

[akupenguin]

doesn't x264 not use slices anymore for parallelism?
Correct. This patch doesn't work for any stream encoded recently by x264.

[Sagekilla]

Crap, so that means x264 users are left out in the dust?

[ChronoCross]

Correct. This patch doesn't work for any stream encoded recently by x264.

thanks. I like the new method of threading so it didn't matter to me. Does anyone know if a majority of the other H264 encoders use slices?

[Tack]

Baby steps, I suppose. I did ask about this (http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/2007-June/031489.html) a few months ago and Andreas does have some ideas. As always, it's just a question of time and motivation.

[LoRd_MuldeR]

I think I'm having a déjà vu ;) Wasn't that patch released moths ago? :confused:

[akupenguin]

The patch has been in public development since May. The news is that it was finished and applied to svn.

[clsid]

For those who wish to try it, this functionality is present in ffdshow tryouts since revision 1465.

[_xxl]

http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/2007-May/thread.html
what i think is better is frame level parallelism
so that thread 1 decode frame 1 and updates a last_decoded_row variable
and thread 2 decodes frames 2 and checks if the needed reference is available
or if last_decoded_row is still too small in the later case the thread waits
a third thread could now work on frame 3 and so on ...

Well, is it then true, that we can decode CABAC of the next block in
advance and parallel? If we could identify the CABAC coded blocks with
exact length we could defer decoding to a second thread, which would
give the decoded stream as array of bytes. The disadvantage would be
that parallelism is limited to 2 cores, but on the other hand, for HDTV
h264 decoding (which is the most demanding task at the moment) 2 cores
of current CPUs would be enough.
A further advantage would be that the changes to h264 decoder code would
be rather small.

[bond]

pengvado, is there a chance that you readd the functionality to x264 to encode with slices (independant of parallel encoding if thats easier/better)?

[burfadel]

In the revision history for ffdshow tryouts, under rev 1463 (5 Sept 2007) it says:
Add slice-based parallel H.264 decoding

Where does encoding come into this?

[Inventive Software]

Encoding comes into it because quite often x264 and libavcodec are linked programmatically (is that even a word? :D). x264 is the most popular free AVC encoder, libavcodec the most popular free AVC decoder, that's why the encoding part matters.

@akupenguin: when (as in revision) was parallel slice-level encoding removed / changed in x264? Can't find it in the changelog. :(

[foxyshadis]

610.

It is not too short, forum.

[Inventive Software]

It's actually 607. Turns out I can't bloody read. :)

[LoRd_MuldeR]

Where does encoding come into this?

Encoding is important, because the new multi-threaded decoding works only with slices! If the H.264 video was not encoded with slices, the new multi-threaded decoding won't give any speed-up at all! Previously the x264 encoder used slices, but the current revision does not do so any more. Now x264 uses a multi-threading method that works without slices (which is actually better), so the resulting files won't be able to use libavcodec's multi-threaded decoding. At least until they implement a frame-based instead of a slice-based multi-threading method in their H.264 decoder...

[akupenguin]

pengvado, is there a chance that you readd the functionality to x264 to encode with slices?
I'm torn between implementing it to allow people who want slices to benefit from the latest features, and not implementing it to discourage use of slices. But I think laziness will win.

[morph166955]

forgive the newb type question but, what is the problem (minus the fact that it hasnt been implemented) with doing a threaded decode of it in the way that x264 currently handles its threading?

[akupenguin]

Slices are easier to implement, so Andreas took the path of least resistance even though the result is less useful.

[LoRd_MuldeR]

If each frame is divided into several slices, each slice can be decoded on it's own. That makes implementing multi-threaded decoding more easy. In case no slices were used to encode the video, you can't use various threads to work on the same frame. So if you want to implement multi-threaded decoding without slices, you will have to find a way to decode multiple frames in a parallel way, so each thread can work on it's own frame. But decoding multiple frames in a parallel way is extremely difficult for various reasons. For example you can't decode a P-Frame before it's reference frame was decoded first. Multiple-reference frames and B-Frames make things even more complex...

[akupenguin]

For example you can't decode a P-Frame before it's reference frame was decoded first. Multiple-reference frames and B-Frames make things even more complex...
Yes you can. That's exactly what x264 does. x264's method works for encoding or decoding of any video compression format. Multiref doesn't affect it, and B-frames make it easier.

edit: Ok, not quite every format... It wouldn't work with H.264 FMO. But nobody uses FMO. Likewise there might be some complications in Theora because Theora's macroblocks aren't coded in raster order but rather in a hilbert curve.

[LoRd_MuldeR]

That is confusing. How can you decode a P-Frame without knowing the reference frame(s) it depends on ???

[Sergey A. Sablin]

That is confusing. How can you decode a P-Frame without knowing the reference frame(s) it depends on ???

it is definitely easier to show in pictures :)
both frames are encoded/decoded in parallel. The minimum number of mb rows ref goes ahead is a space needed to predict current mb (x), which obviously depends on the length of vertical MV pointing down


ref cur
******** ********
******** ***x....
******** ........
******** ........
*x...... ........
........ ........

(*) - decoded mb, (x) - current, (.) - non decoded yet

[giandrea]

That is confusing. How can you decode a P-Frame without knowing the reference frame(s) it depends on ???

perhaps you don't have the whole reference frame, but a part of it, and you decode the P-frame up to the point where you have decoded the reference frame? just thoughts...

[LoRd_MuldeR]

perhaps you don't have the whole reference frame, but a part of it, and you decode the P-frame up to the point where you have decoded the reference frame? just thoughts...

Your assumption matches Sergey A. Sablin's explanation. So it seems that is the point. Thanks for the info! Still the P-Frame cannot be decoded faster than it's reference frame(s), right?

[bond]

btw if someone wants to test this with x264 encodes, the last x264 revision supporting writing slices is 606:
http://mirror03.x264.nl/x264/revision606/x264.exe

[dragongodz]

or another method, for encoding or decoding, would be to break it down to macroblocks. that is encode or decode multiple blocks at a time.

[Manao]

would be to break it down to macroblocksYou can't do that. There are context dependencies between consecutive macroblocks, so you need to decode the first macroblock ( at least the CABAC/CAVLC information ) before starting the second one. Furthermore, intra prediction is spatial, so you must have reconstructed a macroblock before being able to use it.

[Inventive Software]

What's the other method of multi-threaded decoding apart from slices then?

[LoRd_MuldeR]

What's the other method of multi-threaded decoding apart from slices then?

The one x264 uses now:
Encode/Decode several frames in parallel (see Sergey A. Sablin's post above)

[Gabriel_Bouvigne]

Or use different threads for bistream decoding/frame reconstruction/deblocking

[dragongodz]

You can't do that.

i was sure i had read that it was possible. after some searching i found this.

http://www-itec.uni-klu.ac.at/~klschoef/papers/parallelization.pdf

[akupenguin]

http://www-itec.uni-klu.ac.at/~klschoef/papers/parallelization.pdf
a) They have some really strange cpu distribution numbers, possibly due to their complete lack of SIMD. e.g. 42% of their time is spent in idct making it a prime target for parallelization, whereas libavcodec spends 2% of its time in idct making it not worth bothering.

b) If you decode a decent bitrate cabac stream, cabac will take ~1/3 of the cpu. So their method degenerates to simply running 1 thread for parsing, 1 thread for decoding, and 1 for deblocking. This has been discussed before. Yes that's a possible division of labor, but it can't scale any further. See their results where they claim about a 2x speedup from 4 cpus.

[dragongodz]

my point was that i had read about it before and that it was possible to do it on a macro block level.

i dont expect anyone to add any of those methods but i just thought things like the macro block pipelining was interesting. :)

[foxyshadis]

Wouldn't it be possible to just use more memory to decode cabac from multiple frames at once? ffdshow already queues ahead by default, giving a small buffer, but all lavc users could benefit. It'd be a stopgap until x264-style threading appears, anyway.

[akupenguin]

If you're going to decode multiple frames at once, why go half-assed and only do the cabac part? I'm not sure it would even be any simpler than full x264-style: cabac-only could skip the synchronization, but has to add code to save and restore all the information that's currently used immediately after parsing.