low latency real time codec?

[ferrous66]

Which codecs are suitable for real time low latency encoding/streaming (for videoconference for example)?
I know of M-JPEG, are there alternatives that give better compression while remaining low-latency RT?

[temporance]

This is a deceptively complex area, but I will answer your question first...

I know of M-JPEG, are there alternatives that give better compression while remaining low-latency RT?

Any codec that doesn't use B-frames can have more-or-less the same latency as M-JPEG.

B-frames screw things up: a buffering latency is needed at the encoder to reorder frames prior to encoding.

So, in principle, any codec can be used, so long as you don't enable B-frames.

The CPU time needed to encode and decode represents another latency. Since you're operating in real time you must have a CPU that is capable of encoding one frame in 1/30 second (substitute 30 with your actual fps). So processing time is not usually such a great problem for videoconferencing.

The tricky part is when you get to thinking about VBV buffering and rate control. This is especially important if you are operating over a constrained bitrate channel, e.g. the Internet. AFAIK, no codecs discussed here have the disciplined bitrate management needed for operating in this environment with very low latency. (~= true CBR).

[slavickas]

Originally posted by temporance
The tricky part is when you get to thinking about VBV buffering and rate control. This is especially important if you are operating over a constrained bitrate channel, e.g. the Internet. AFAIK, no codecs discussed here have the disciplined bitrate management needed for operating in this environment with very low latency. (~= true CBR).

i think at least RV and VP codecs are designed for such cases, and they can keep cbr unless settings are out of logical ranges

[drmpeg]

MPEG-2 can be very effective for low-delay applications. By using the
following features, delays of less than 100 milliseconds can be
achieved:

1) No B-frames (as already mentioned).

2) Field pictures. By encoding field pictures instead of frame
pictures, the pipeline delay will be cut in half.

3) No I-frames. Instead, a progressive intra block update scheme
is used. This makes for an ugly start-up, but having all P-pictures
allows for a small VBV size to be used (which further reduces the
delay).

4) Since there's no B-frames, the 16x8 and Dual-Prime prediction
modes can be used (which partially offsets the coding inefficiency
of not having B-frames).

When I was at my previous company, I integrated the C-Cube low-delay
MPEG-2 encoder into our product line. I believe that microcode was capable
of about 60 to 70 milliseconds of delay, but the MPEG-1 Layer 2 audio decoder
we were using at the time wasn't quite up to the task and I had to run
the video at 100 milliseconds delay to match the audio latency.

However, at that time (1997-98), the M-JPEG low-delay codecs were
so inexpensive and so well entrenched in the market, the MPEG-2
solutions could not compete.

Ron

[temporance]

Originally posted by slavickas
i think at least RV and VP codecs are designed for such cases, and they can keep cbr unless settings are out of logical ranges
So can DivX 5 with its VBV buffer (at least in theory).

Keeping CBR is not enough for professional videoconferencing apps. All of RV, VP and DivX assume a reasonably large bitstream buffer either end of the restricted bitrate channel. This means that they are good for restricted bandwidth streaming where latency is not so important (think of RV's buffering...) but they are pretty much useless for low latency apps.

[vinouz]

and what about quantizer based 1 pass encodes. As they're not supposed to be CBR they don't have to use any bit reservoir and can have a very low latency (no-bframe).

[temporance]

and what about quantizer based 1 pass encodes

Any rate control mode can be low latency if your video conferencing system has infinte bandwidth. It's more realistic to suppose that video conferencing will be done over a channel restricted to, say, 512kbps. In this case a constant-Q mode will either produce awful quality, or bitrate peaks that saturate the channel and introduce latency.