B-frames implementation

[moogie]

I understand the general process of implementing B-frames, however all the literature i have read glosses over the most important part, that is, the actual comparision of the current frame with the previous and forward frame.

My question is how is this done? i.e. are the two frames (prev and next) combined into on frame which is then compared with the current frame or is the the block searching area effectively doubled by firstly comparing with the previous frame then the next frame?

I have also thought of a different way (which is potentially better) to use the B-frames. Instead of each B-frame comparing with the last (I or P frame) and the next (I or P frame) divide the sequence binarily. This in theory should give a better compression as the b-frames are created by comparing frames that are closer to themselves
ie.

a sequence of 5 frames:

12345
IBBBP

would look like:

1
I

15
IP

153 B3 compares with I1 and P5
IPB

1532 B2 compares with I1 and B3
IPBB

15324 B4 compates with B3 and P5
IPBBB

or a 7 frame sequuence:

1234567
IBBBBBP

17
IP

174 B4 compares with I1 and P7
IPB

1742 B2 compares with I1 and B4
IPBB

17423 B3 compares with B2 and B4
IPBBB

174235 B5 compares with B4 and P7
IPBBBB

1742356 B5 compares with B5 and P7
IPBBBBB


From my thinking, only one more frame of extra memory is needed as the frame are created the prevous frame goes from I1 to B2 to B4 to B5 with out visting a frame below the curent previous fram again however the next frame does revisit old-future frames, i.e. from P7 to B4 and back to P7.


obiviously the more B frames you have the more memory is needed for old-future frames. This can be reduced by computing the future frame as needed.

what are peoples thoughts on this?

[Asmodian]

Just a quick point about B-frames with the disclaimer that I am not, in any way, an expert or even very knowledgeable about the actual implementation of video encoding techniques, but I have read a lot here on Doom9's forums.

The main compression gain from using B-frames as of now (and AFAIK) is from using significantly higher quantizers on them. This is safe because there are no frames which use the B-frames to predict off of. Using your idea would require predicting off of B-frames.

I guess this means that it would take much more CPU/Memory to implement your B-frames then they would be worth, given the compression gain from them (in Xvid B-frames don't change the compression much when set to a ratio of 1 with an offset of 0).

Of course someone with more understanding of the internals of MPEG4 compression would be useful in this discussion.

P.S. These B-frames would, of course, break MPEG4 compatibility.

[moogie]

You are probably quite correct in saying that it would break MPEG-4 compatability.

I forgot to say that i am thinking of how to implement bi-directional frames for use in a lossless codec i am creating. Currently most if not all the literature is on how b-frames are impelmented in mpeg-4 which while gives me ideas, does not mean that mpeg-4 b-frames are entirely suitable for use in a lossless codec.

Since the codec is lossless, using b-frames in prediction should not introduce errors as the b-frames themselves are lossless.

[sysKin]

Originally posted by moogie
My question is how is this done? i.e. are the two frames (prev and next) combined into on frame which is then compared with the current frame or is the the block searching area effectively doubled by firstly comparing with the previous frame then the next frame?[/b]The second. Searching area is effectively doubled - you get two macroblock modes, "forward" and "backward" which tell decoder which reference is used in a given macroblock. Motion prediction is, of course, independent for each mode.
Mpeg (1, 2, 4, h264) also have "interpolate" mode. In this mode, there are two vectors, one for forward and one for backward reference, and the actual motion-compensated block is the bilinear interpolation of the two.
Mpeg-4 and h264 also have 4th mode "direct" which effectivly works like interpolate, but the two vectors are created using the future reference's motion vector - it assumes linear motion from past reference to the future. It's just a shortcut which takes very little bits.
I have also thought of a different way (which is potentially better) to use the B-frames. Instead of each B-frame comparing with the last (I or P frame) and the next (I or P frame) divide the sequence binarily. This in theory should give a better compression as the b-frames are created by comparing frames that are closer to themselvesYup, exactly what I had in mind for "kludge" codec. It's a shame it's not used in mpeg-4 (or even h264), it's probably because of memory and CPU conciderations (look at how many b-frames you have to decode to show the first b-frame. then you don't have to decode anything, then many bframes at once, again).

Radek

[moogie]

The second. Searching area is effectively doubled - you get two macroblock modes, "forward" and "backward" which tell decoder which reference is used in a given macroblock. Motion prediction is, of course, independent for each mode.
Mpeg (1, 2, 4, h264) also have "interpolate" mode. In this mode, there are two vectors, one for forward and one for backward reference, and the actual motion-compensated block is the bilinear interpolation of the two.
Mpeg-4 and h264 also have 4th mode "direct" which effectivly works like interpolate, but the two vectors are created using the future reference's motion vector - it assumes linear motion from past reference to the future. It's just a shortcut which takes very little bits.


Ah, Thanks for clearing it up. I thought that doubling the search area was the correct way to implmenent it, but it is nice to have confirmation.

Nick

[moogie]

Originally posted by sysKin
The second. Searching area is effectively doubled - you get two macroblock modes, "forward" and "backward" which tell decoder which reference is used in a given macroblock. Motion prediction is, of course, independent for each mode.
Mpeg (1, 2, 4, h264) also have "interpolate" mode. In this mode, there are two vectors, one for forward and one for backward reference, and the actual motion-compensated block is the bilinear interpolation of the two.
Mpeg-4 and h264 also have 4th mode "direct" which effectivly works like interpolate, but the two vectors are created using the future reference's motion vector - it assumes linear motion from past reference to the future. It's just a shortcut which takes very little bits.



Is the interpolation just a simple addition/subraction of vectors? Also is will the interpolation work when the user wants full seaching of the frame? I can imaging huge vectors resulting from a full search that would make the interoplated frame vastly different to the other frames.



Yup, exactly what I had in mind for "kludge" codec. It's a shame it's not used in mpeg-4 (or even h264), it's probably because of memory and CPU conciderations (look at how many b-frames you have to decode to show the first b-frame. then you don't have to decode anything, then many bframes at once, again).

Radek

great minds think alike eh ;)?

it looks like the same old problem of either more CPU usage or more RAM usage.

what is this kludge codec you speak of?

[Manao]

look at how many b-frames you have to decode to show the first b-frame. then you don't have to decode anything, then many bframes at once, againYou'll only have to decode log2(n) frames, where n is the gap between two I-frames, so it's not that big a deal if n isn't too large.

However, I don't think there is much to gain using such a method :

- If there is a lot of motion, the gap between two keyframes can't be very large ( imho < 16 ), because the middle frame would be 8 frames away from them, which would imply huge motion vectors, and bad compensation.
- If there is a little motion, it may be better, but it's not sure ( there again, the gap can't be too large ).

Of course, we could think it's better because any frame would only be at ln2(n) frames from the closest keyframes ( in term of references ), and that hence the quality of the compensation is better, because the frames from which we compensate looks better.

But, in fact, in XviD, the quality between two keyframes doesn't decrease. Keyframes seems only to be useful at scene changes, and for searching purposes.

[moogie]

Originally posted by Manao
You'll only have to decode log2(n) frames, where n is the gap between two I-frames, so it's not that big a deal if n isn't too large.

However, I don't think there is much to gain using such a method :

- If there is a lot of motion, the gap between two keyframes can't be very large ( imho < 16 ), because the middle frame would be 8 frames away from them, which would imply huge motion vectors, and bad compensation.

- If there is a little motion, it may be better, but it's not sure ( there again, the gap can't be too large ).



Yes, the number of b-frames between I/P frames will be dependant on the magnitude of the change (i.e. motion) However it is assumed that b-frames can be represented sufficently few bits as to more than offset the extra bits needed in P-frames (due to the P frames' distance from the previous I frame)




Of course, we could think it's better because any frame would only be at ln2(n) frames from the closest keyframes ( in term of references ), and that hence the quality of the compensation is better, because the frames from which we compensate looks better.

But, in fact, in XviD, the quality between two keyframes doesn't decrease. Keyframes seems only to be useful at scene changes, and for searching purposes.

That does not make much sense to me... since Xvid is a lossy codec each progressive frame losses a little information and if there are no key frames, then over a few frames, lets say 200, the frames will be markedly different!

Since i am creating a Lossless video codec, quality of each frame type (I,P,B) is not an issue and so the benefits i hope to achieve by using B-frames are to give better matching blocks (i.e. motion estimation) and therefore reduce the number of bits required.

[Manao]

That does not make much sense to me... since Xvid is a lossy codec each progressive frame losses a little information and if there are no key frames, then over a few frames, lets say 200, the frames will be markedly different!No, because when you make the motion compensation, you're using the compressed previous frame, not the uncompressed one. So, at any time, you could have a p-frame that would be identical to the original frame, if you were ready to spend enough bytes to encode the difference between the frame and the previous compressed frame after motion compensation.Since i am creating a Lossless video codec, quality of each frame type (I,P,B) is not an issue and so the benefits i hope to achieve by using B-frames are to give better matching blocks (i.e. motion estimation) and therefore reduce the number of bits required.In that case, it's different, and using b-frames like you described can be a good idea.

[sysKin]

Manao didn't you mix keyframes and reference frames?
The number of frames to be decoded is log(n) but n = number of b-frames in a row, not inter-frames in a row.

Anyway, it *is* a problem for realtime decoding. Just remember that buffering frames is difficult because of a/v synchronization.

[Manao]

Well I thought that p-frames would be useless if n was large enough ( since motion compensation wouldn't be precise ). But yeah, replacing keyframes by reference frames would be clearer.

[johnslion]

Sorry for my bad english, but I really want to know!
1.Did the Bidirectional prediction method save bits only through searching composition blocks in both forward and backward reference plane?

2.How 'Direct Mode' and 'Interpolate' implement?