X264 Compression with No IDR Frames

[rlandle]

Is it possible to set up X264 to encode with no IDR frames?

I only want to use P-Frames, where I schedule or force random MBs of the images to be I-MBs for refresh.

If not, can I at least turn off the scene cut detection?

How does this effect the bit rate controller since there is no definition of GOP?

Thanks

[Mutant_Fruit]

Why would you want no IDR's?

[rlandle]

I'm looking at low bitrate/low latency streaming compression over a network. To handle wireless for instance, I need to keep my bitrate as constant as possible.

Compressed IDR/I frames are like 3 times as large as P-Frames (maybe larger), unless you let the rate control over-compress the I frames so that it matches the P-Frames.
I don't want this.

So I want to avoid the bitrate spikes caused by I Frames.

Is it possible with X264?

If not, any suggestions on what I'd have to change to get this to work?

[morph166955]

well as far as I know you need at least one I/IDR frame for the P and B frames to base their images off of. You could try to increase the min/max gap between them but if you make it too big your going to end up running into HUGE P frames at scene cuts if you don't set it right. Bottom line is you need I frames periodically otherwise your going to run into some huge issues and you wont have anything thats watchable at all. As for the scene cut detection, think about it this way. P frames are based off of previous images, if you do have a drastic change in the image where a scene cut would be and you don't put an I frame in, that P frame is going to be HUGE anyway to account for all of the changes in the image from its predecessor.

If your looking to maintain bitrate as evenly as possible then you could try either crf or cq. What resolution are you using for the image and what are your x264 settings currently? Can you explain more about the system your trying to setup? eg. streaming from what kind of system, to what kind of system, etc...try to give hardware specs too if possible it makes it easier to understand everything. Also try to include what the actual display is going to be (HD LCD, computer LCD/CRT, etc) since that will have a great impact on where you go from here (eg if you have only a non HD TV or something similar you dont necessarily need a higher bitrate HD image getting sent to it, you could scale it down which would save mucho bits/sec). However if you have a nice shiny 720p/1080p set you may want to send "full res" HD content to it since you have the pixels to display it. You may also want to look into the mpeg2 and/or mpeg4 transport stream format which is currently being used almost everywhere to transmit TV (both SDTV and HDTV) to its destination in some form or another.

Also just so you realize, x264 is (for the moment) quite a bit slower in encode speed then xvid or something similar is just because of the nature of it. It IS still very possible to run x264 at a rate that would be streamable but you have to realize that possibility is very dependent on the resolution of your image and the settings you use. Just to give you an example, my encoder is probably the most powerful thing on this forum (2x 2.66ghz quad-core x5355 Xeons), it can do an 848x352 image (source: die another day dvd) at a max of ~200fps...OBVIOUSLY fast enough for you to stream at. However if I want to do a 720p movie and (mostly) retain HD quality, I'm getting somewhere on the order of ~24fps...barely enough to stream it. Sure, I could kick my settings down a bit to increase but since I'm not streaming so I'm not particularly worried about it. The moral of that little blerb is that (if your doing realtime encoding as opposed to streaming already encoded files) what ever you have doing the encode has to be fast enough to encode your movie as well as mux your audio and package it (.ts, some form of matroska maybe?) and then get it out your NIC. If realtime encoding isnt your thing, then dont worry about it but I figured I'd throw that in there in case you were.

If you arent going the realtime route, you should realize that if you are going to make these videos with settings like you had said you wont be retaining the original video in all of its quality. Lack of I/IDR frames will degrade the image over longer periods of time, low cq/crf/bitrate settings will also obviously degrade the image.

As far as the network end of it, you have several options here. First question is what wireless ap do you have? If its a linksys thats third-party flashable (most older WRT54G and WRT54GS as well as a few others but those are the big ones, normally version 6 or lower for those guys) would you be willing to put a third-party firmware on there and kick your transmitter power up to make it a more stable/faster link? Is it possible to run a CAT5 to your destination and not run wireless at all? Then you could run gig-e and avoid all of these problems all together since its more then very possible to stream a full 1080p movie (possibly even in raw i420 format or something similar if you wanted too). Again this is very dependent on your setup so more info on what hardware you have and what the final "goal" is would be great.

In the end, it could take a good while for you to find a "happy medium" for your settings that you feel is fast enough to stream and also high enough quality that you aren't going to be complaining about the image itself so be ready for what could very easily be a few months before your satisfied with your end results.

[rlandle]

morph,
thanks for the detailed response.

I'm trying to put together a system that is similar to a video conferencing app, but my bandwidth requirement is quite low < 1mbit, and unfortunately wireless is a must.

I'm only talking about CIF, maybe VGA resolutions here, not HD. Not at that bandwidth :)

With my settings and hardware, I am able to encode/stream real-time at these resolutions with decent results.

So I am very happy with the performance of X264; but I think I need to tweak it a little, so that it has a chance to perform better in a streaming wireless environment.

Here is some more info...

I am not expecting any drastic scene changes, since the video is from a live source, and the majority of my imagery will be the same from frame to frame.

I also don't expect my P Frames to ever approach the size of an IDR frame, since I'm using a constant bit rate control.

I am going off the hunch that in my environment, IDR frames aren't really that necessary. I agree that P and B MBs need a reference, but they don't necessary have to reference pixels from an IDR picture.

Think of it this way. An IDR picture refreshes the whole image instantaneously, since it has no temporal dependencies.

IDRs are good to start with cause it gives the next frame, a whole image worth of reference area. An IDR frame is really only a whole frame filled with I-MBs.

But what if I can't squeeze this whole frame of I-MBs through my bandwidth, without introducing delay?

Can't we eliminate the IDR frame, and just slowly introduce I-MBs to the image over time?

If I could force I-Mbs in a pattern that would eventually refresh the whole image over 4 or 5 frames say, rather than all at once, I can spread out the hit in bitrate.

The caveat is that I will increase the size of each P-Frame since I've introduced more I-MBs, and at constant bitrate that means lower image quality.

What I want like to do is study how much quality is lost as a result of smoothing out the bitrate in this fashion.

My guess is, at low bitrates, the quality loss is negligible.

How would I go about getting X264 to do this?

Thanks

[reiser]

rlandle, just think what will happen when some packet is lost or scrambled during transfer! The video will be b0rked 'til next I frame. Suppose you don't have I frames at all, the watcher is out of the show.
Now I would suggest to lower the bandwidth and the profile of the encoding (there are profiles in AVC designed for wide area of aplications form mobile devices to HD). Make sure your wireless connection can transfer all packets without loss during spikes and increase the cache on the client side. This way your video as whole will be delayed a little, but will play smoothly. Of course the wireless must be able to sustain the average bandwidth.

[Mutant_Fruit]

Or better yet, use a program that can buffer the stream.

VLC has a configurable buffer, you can buffer N seconds of material. I regularly stream 1000kbps stuff + 128kbps audio over my home connection with barely any skips (i generally forget to turn on the buffering).

I think when i do remember to turn on buffering, i get skip-free playback on everything i've tried.

[burfadel]

If you've mostly got a still image, the bitrate requirement will be quite low anyway. You should take in to account the maximum resolution of the camera you are using, many of them are lower than VGA anyway (webcams and other streaming cams). More along the lines of VCD?

The others are right, you just should run a few second buffer and the scenechanges etc won't affect it. If you need it live such as a video-conferencing system, you either need large bitrate or very low quality, there is no getting around that (practicality wise). In a video-conferencing scenario over wifi you're in trouble no matter what you use, as wifi is only really good for one way traffic.

With wifi you have to take in to account the actual bitrate is always lower than the suggested bitrate. There is a lot of bull with wifi, about actual performance, major patent violations... etc etc lol

But yeah, basically you do need the I-frames, without them would be like a car without an engine, you can't drive it!

[drmpeg]

The typical technique for low-delay is to use progressive intra refresh. You send all P-frames, but in each P-frame some amount of macroblocks are forced to be intra. Each successive P-frame forces a different set of macroblocks to be intra, and after so many frames, the entire image has been refreshed.

Here's an example P-frame from an MPEG-2 low-delay bitstream:

http://img47.imageshack.us/img47/4818/intrarefreshga2.jpg (http://imageshack.us)

Intra blocks are in black, forward predicted blocks in blue and skipped blocks in yellow. Blocks with dots are where the quant has changed. 24 macroblocks are forced intra in each field. Since there's 675 macroblocks per field, the entire image gets refreshed in 675/24 = 28 fields or about every half second.

Ron

[akupenguin]

Is there a reason the forced intra blocks are in a stripe like that? I would have thought that the pattern with least visual impact would be to scatter them pseudo-randomly about the frame, while the pattern that maximizes compression in h264 (with spatial intra prediction restricted to adjacent intra blocks) would be one contiguous region.

[drmpeg]

Is there a reason the forced intra blocks are in a stripe like that? I would have thought that the pattern with least visual impact would be to scatter them pseudo-randomly about the frame, while the pattern that maximizes compression in h264 (with spatial intra prediction restricted to adjacent intra blocks) would be one contiguous region.
I don't think it was super optimized. Looks like it was designed to be easy to calculate the intra positions. Here's a 4-frame sequence from the start of the bitstream (first field is all intra).

http://img337.imageshack.us/img337/7272/intrarefresh4wb3.th.jpg (http://img337.imageshack.us/my.php?image=intrarefresh4wb3.jpg)

Ron

[Sergey A. Sablin]

I don't think it was super optimized. Looks like it was designed to be easy to calculate the intra positions. Here's a 4-frame sequence from the start of the bitstream (first field is all intra).

http://img337.imageshack.us/img337/7272/intrarefresh4wb3.th.jpg (http://img337.imageshack.us/my.php?image=intrarefresh4wb3.jpg)

Ron

nice pictures Ron btw!
but what is the reason for intra refresh? to update picture after corruptions? it hardly helps to mpeg-2 because of MC, including AVC intra-prediction mentioned by Loren and deblocking filter these intra striping would not help a lot imho.
actually I never tried with AVC, but my feelings tell me that's will be useless in most cases. Probably some regions will help more, when intra blocks will take use of uncorrupted pixels.

[drmpeg]

nice pictures Ron btw!
but what is the reason for intra refresh? to update picture after corruptions? it hardly helps to mpeg-2 because of MC, including AVC intra-prediction mentioned by Loren and deblocking filter these intra striping would not help a lot imho.
actually I never tried with AVC, but my feelings tell me that's will be useless in most cases. Probably some regions will help more, when intra blocks will take use of uncorrupted pixels.
Intra refresh is used so that you never have to send an I-frame. This allows for a smaller VBV which helps to reduce the encode to decode delay (the target is usually less than 100 milliseconds).

The I-field shown in my example is only sent at the beginning of the bitstream and may be over quantized to fit in the small VBV. Otherwise, only P-frames are sent and it takes a liitle while for the image to build up when you acquire the stream.

This is a bitstream from an old C-Cube MPEG-2 low-delay encoder. As Loren has pointed out, the striping scheme is not optimum for H.264, but I felt this thread needed some MPEG-2 background to help folks understand low-delay in general.

The images are from the C-Cube MPEG-2 bitstream analyzer. Here's a dump of the motion vectors, showing 16x8 (horizontal line through MB) and dual-prime (vertical line through MB) modes that are pretty much only used in low-delay.

http://img176.imageshack.us/img176/5663/vectorslr6.jpg (http://imageshack.us)

Ron

[Manao]

The purpose is to be able, after a defined number of frame, to match encoders & decoders reconstructed output. For that, you need to refresh some macroblocks ( by using intra macroblocks ), AND an area must reference only those MBs ( or MBs derived by those ).

Randomizing intra macroblocks makes it impossible to efficiently find a growing set of P macroblocks that would reference valid areas.

With a diagonal pattern going from left to right, if you prevent MBs from the left part to reference MBs from the right, you are sure to have a match between encoder and decoder once the intra stripe has crossed the frame. It would also be possible with a vertical slice or horizontal slice. I guess vertical wasn't chosen because horizontal movements tend to be larger than vertical ( thus constraining too much the motion ), while horizontal wasn't chosen because it would make a single intra slice much bigger than the other one ( perhaps disturbing their rate control algorithm ).

[foxyshadis]

That's pretty interesting for very low latency, when you don't want q45+ i-frames. Have you introduced random bit-errors to see how well it recovers? Even if it leaves it a little corrupt, better than very corrupt.

[Sergey A. Sablin]

The purpose is to be able, after a defined number of frame, to match encoders & decoders reconstructed output. For that, you need to refresh some macroblocks ( by using intra macroblocks ), AND an area must reference only those MBs ( or MBs derived by those ).

Randomizing intra macroblocks makes it impossible to efficiently find a growing set of P macroblocks that would reference valid areas.

With a diagonal pattern going from left to right, if you prevent MBs from the left part to reference MBs from the right, you are sure to have a match between encoder and decoder once the intra stripe has crossed the frame. It would also be possible with a vertical slice or horizontal slice. I guess vertical wasn't chosen because horizontal movements tend to be larger than vertical ( thus constraining too much the motion ), while horizontal wasn't chosen because it would make a single intra slice much bigger than the other one ( perhaps disturbing their rate control algorithm ).

this is all clear, but encoder has to be sure, that intra macroblock modes it is using also refer correct value, ie uncorrupted.
So for mpeg-2 encoder can use just intra mbs and then refer to this areas, but referring to this areas will decrease compression efficiency.
In case on AVC, encoder can't be sure that decoder will use exactly uncorrupted data, so it is only a luck if decoder will match encoder output after some frames. Seems this situation has very low probability in case of errors.

And yet again - if this is efficient for mpeg-2 as mpeg-2 don't have bit exact output from decoder and thus some refresh is needed every second or two. For AVC this is not a case and as far as I see the only application of such intra refresh maybe an attempt to reconstruct picture after some losses, but this is not determined because of intra prediction and deblock unfortunately.

[Manao]

as far as I see the only application of such intra refresh maybe an attempt to reconstruct picture after some lossesNo. It acts as a GOP, so it's as useful as a GOP (i.e random access ), but it needs a far smaller CPB ( in CBR, the CPB is mostly used to distribute the bits from the P/B to the I frames ). So it's useful for low latency.

[Sergey A. Sablin]

No. It acts as a GOP, so it's as useful as a GOP (i.e random access ), but it needs a far smaller CPB ( in CBR, the CPB is mostly used to distribute the bits from the P/B to the I frames ). So it's useful for low latency.

ok got it. but in that case decoder will need one or two second to reconstruct whole picture using small buffer, right?

BTW - there is almost no difference between CBR and VBR in case of AVC as you may know. The only difference is that CBR always fills the buffer while VBR fills only on actual data. So it all depends only on buffer size - not on CBR or VBR.

[Manao]

but in that case decoder will need one or two second to reconstruct whole picture using small bufferAs with a GOP. When you connect to a streamed video, a 2 second GOP will have a "connection time" of 1 second in average ( you need to wait for an intra ). With that system, if it cycles completely in 2 seconds, you have a connection time of 2 seconds. If you describe all frames as recovery points with a 2 seconds period to actually recover the video, the decoder should start to show the video only after two seconds, so you'll see an artifact free video.

[Sergey A. Sablin]

As with a GOP. When you connect to a streamed video, a 2 second GOP will have a "connection time" of 1 second in average ( you need to wait for an intra ). With that system, if it cycles completely in 2 seconds, you have a connection time of 2 seconds. If you describe all frames as recovery points with a 2 seconds period to actually recover the video, the decoder should start to show the video only after two seconds, so you'll see an artifact free video.

yes, you're right, thanks.

[foxyshadis]

After considering different possibilities, a full row or two of I-blocks every P frame (and b-ref?) would seem to be both visualing pleasing and seemingly much simpler to code as exclusion bands than staggered blocks. Just set every reference frame as a recovery point and any decoder can start quickly; it would even have an interesting wipe-downward effect as it completed. (Or perhaps 8x8 MB blocks to take more advantage of spatial prediction, but that wouldn't have the spiffy effect.) I really can't figure out what particular benefit the staggered sweep format gives.

[Manao]

while horizontal wasn't chosen because it would make a single intra slice much bigger than the other one ( perhaps disturbing their rate control algorithm ).In MPEG2, a slice = a row. From a rate control point of view, it may be interesting to distribute evenly accross the slices the bitrate implied by Intra MBs.

[drmpeg]

Here's a pic of the startup of the MPEG-2 low-delay stream. It is frames 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, 50 and 60 after a Sequence Header has been encountered. The reconstruction buffer starts at all zeroes (green).

http://img172.imageshack.us/img172/1351/ldstartjn8.jpg (http://imageshack.us)


E:\atsc>intd ld_bits.mpg
Interactive MPEG Decoder intd 0.72A
@ e
Found Sequence Start Code at byte position 0.
* -------- Sequence Start Code at 0 ---------
* -------- GOP Start Code at 22 ---------
* -------- Intra frame at 30 ---------
* -------- Predicted field at 12240 ---------
* -------- Predicted frame at 15696 ---------
* -------- Predicted field at 18216 ---------
* -------- Predicted frame at 21792 ---------
* -------- Predicted field at 25784 ---------
* -------- Predicted frame at 29472 ---------
* -------- Predicted field at 35968 ---------
* -------- Predicted frame at 42424 ---------
* -------- Predicted field at 50160 ---------
* -------- Predicted frame at 59248 ---------
* -------- Predicted field at 67880 ---------
* -------- Predicted frame at 79440 ---------
* -------- Predicted field at 89304 ---------
* -------- Predicted frame at 100296 ---------
* -------- Predicted field at 110384 ---------
* -------- Predicted frame at 119480 ---------
* -------- Predicted field at 131856 ---------
* -------- Predicted frame at 141856 ---------
* -------- Predicted field at 152616 ---------
* -------- Predicted frame at 162952 ---------
* -------- Predicted field at 171600 ---------
* -------- Predicted frame at 183888 ---------
* -------- Predicted field at 194200 ---------
* -------- Predicted frame at 204768 ---------
* -------- Predicted field at 214648 ---------
* -------- Predicted frame at 223832 ---------
* -------- Predicted field at 236120 ---------
* -------- Predicted frame at 246232 ---------
* -------- Predicted field at 256992 ---------
* -------- Predicted frame at 267272 ---------
* -------- Predicted field at 276104 ---------
* -------- Sequence Start Code at 288312 ---------
* -------- Predicted frame at 288334 --------- <------ images start here
* -------- Predicted field at 298176 ---------
* -------- Predicted frame at 309160 ---------
* -------- Predicted field at 319320 ---------
* -------- Predicted frame at 328264 ---------
* -------- Predicted field at 340472 ---------
* -------- Predicted frame at 350192 ---------
* -------- Predicted field at 361728 ---------
* -------- Predicted frame at 371880 ---------
* -------- Predicted field at 380440 ---------
* -------- Predicted frame at 393088 ---------
* -------- Predicted field at 402560 ---------
* -------- Predicted frame at 413736 ---------
* -------- Predicted field at 423672 ---------
* -------- Predicted frame at 432464 ---------
* -------- Predicted field at 444856 ---------
* -------- Predicted frame at 454528 ---------
* -------- Predicted field at 465832 ---------
* -------- Predicted frame at 475872 ---------
* -------- Predicted field at 484488 ---------
* -------- Predicted frame at 496952 ---------
* -------- Predicted field at 507104 ---------
* -------- Predicted frame at 517824 ---------
* -------- Predicted field at 527400 ---------
* -------- Predicted frame at 536656 ---------
* -------- Predicted field at 549024 ---------
* -------- Predicted frame at 558632 ---------
* -------- Predicted field at 569632 ---------
* -------- Predicted frame at 579528 ---------
* -------- Predicted field at 588840 ---------
* -------- Predicted frame at 600736 ---------
* -------- Predicted field at 610848 ---------
* -------- Sequence Start Code at 621696 ---------
* -------- Predicted frame at 621718 ---------
* -------- Predicted field at 631832 ---------
* -------- Predicted frame at 641224 ---------
* -------- Predicted field at 652880 ---------
* -------- Predicted frame at 663160 ---------
* -------- Predicted field at 673768 ---------
* -------- Predicted frame at 683904 ---------
* -------- Predicted field at 693616 ---------
* -------- Predicted frame at 705200 ---------
* -------- Predicted field at 715272 ---------
* -------- Predicted frame at 726224 ---------
* -------- Predicted field at 736208 ---------
* -------- Predicted frame at 745536 ---------
* -------- Predicted field at 757352 ---------
* -------- Predicted frame at 767496 ---------
* -------- Predicted field at 778112 ---------
* -------- Predicted frame at 788320 ---------
* -------- Predicted field at 797816 ---------
* -------- Predicted frame at 809344 ---------
* -------- Predicted field at 819648 ---------
* -------- Predicted frame at 830344 ---------
* -------- Predicted field at 840472 ---------
* -------- Predicted frame at 849800 ---------
* -------- Predicted field at 861584 ---------
* -------- Predicted frame at 871968 ---------
* -------- Predicted field at 882624 ---------
* -------- Predicted frame at 892768 ---------
* -------- Predicted field at 902096 ---------
* -------- Predicted frame at 913904 ---------
* -------- Predicted field at 924208 ---------
* -------- Predicted frame at 935304 ---------
* -------- Predicted field at 945272 ---------
* -------- Sequence Start Code at 954168 ---------
* -------- Predicted frame at 954190 ---------
* -------- Predicted field at 966272 ---------
* -------- Predicted frame at 976160 ---------
* -------- Predicted field at 987120 ---------
* -------- Predicted frame at 997088 ---------
* -------- Predicted field at 1006248 ---------
* -------- Predicted frame at 1018488 ---------
* -------- Predicted field at 1028592 ---------
* -------- Predicted frame at 1039288 ---------
* -------- Predicted field at 1049208 ---------
* -------- Predicted frame at 1058192 ---------
* -------- Predicted field at 1070304 ---------
* -------- Predicted frame at 1080424 ---------
* -------- Predicted field at 1091368 ---------
* -------- Predicted frame at 1101048 ---------
* -------- Predicted field at 1110248 ---------
* -------- Predicted frame at 1122544 ---------
* -------- Predicted field at 1132144 ---------
* -------- Predicted frame at 1143376 ---------
* -------- Predicted field at 1153176 ---------
* -------- Predicted frame at 1162288 ---------
* -------- Predicted field at 1174912 ---------
* -------- Predicted frame at 1185152 ---------
* -------- Predicted field at 1197016 ---------
* -------- Predicted frame at 1205928 ---------
* -------- Predicted field at 1214512 ---------
* -------- Predicted frame at 1227848 ---------
* -------- Predicted field at 1236304 ---------
* -------- Predicted frame at 1248600 ---------
* -------- Predicted field at 1257792 ---------
* -------- Predicted frame at 1266688 ---------
* -------- Predicted field at 1279608 ---------

Ron

[Sergey A. Sablin]

thanks Ron - nice illustration!