OnLive CEO discusses video compression

[her34]

http://www.viddler.com/explore/gamertagradio/videos/160/13.496

People here might be interested in this. CEO brushes on video compression technique.

At 7 minute mark, he goes over briefly.
33-42minute, during Q&A he goes over in bit more detail

[Dark Shikari]

AFAIK Onlive is just using H.264, so this doesn't really go in the "new and alternative" category.

Their "new idea" is splitting the stream into 16 rectangular slices, each of which gets its own encoder. This brilliant idea massively reduces compression on the edges between slices when the scene is in motion and lets them brag about latency 16 times lower than what they actually have.

[her34]

AFAIK Onlive is just using H.264, so this doesn't really go in the "new and alternative" category.

Their "new idea" is splitting the stream into 16 rectangular slices, each of which gets its own encoder. This brilliant idea massively reduces compression on the edges between slices when the scene is in motion and lets them brag about latency 16 times lower than what they actually have.

try watching the video...

[Dark Shikari]

try watching the video...I know a lot more about Onlive's algorithms than Onlive is going to admit in a press release.

They've changed their marketing claims so many times in the past year that I think it's pretty much safe to ignore anything that they say.

[Manao]

After having lost 10 minutes watching the Q&A on video compression, I can concur with Dark Shikari. The guy manages to talk the audience's ears off without saying anything at all for 10 minutes. It goes like "blah blah [...] psychostuff [...] blah blah [...] shannon (RIP) wasn't wrong, but he didn't answer the right question [...] blah blah [...] feedback loop [...] blah blah [...]".

Seeing the video game plays was a bit more interesting. It gets blurry at each scene cuts (which luckily hardly happen in video games) and takes a quarter of second to recover - which is normal and not too big a deal. The video game looked generally blurry too, but then, it was a low compression video of a hand-held camera recording of a screen which was showing the video game, so there aren't a lot of conclusions that can be made about that either.

[Dark Shikari]

It gets blurry at each scene cutsYup, which is because they're using periodic intra refresh with constant slice size ;)

[ffmpeg]

This brilliant idea massively reduces compression on the edges between slices when the scene is in motion .

I fail to understand!

why can this method massively reduce compression on the edges between slices when the scene is in motion?

and more, "reduce compression" means "reduce compression complexity" or "enhance compression ratio" in your statement?

Thanks

[Dark Shikari]

I fail to understand!

why can this method massively reduce compression on the edges between slices when the scene is in motion?With slices, each slice can reference data anywhere in the previous frame. This means that if something moves from slice A to slice B, there's no problem: slice A can point to it with a motion vector just as if it didn't cross the edge.

But OnLive isn't using slices: they're encoding the video as a bunch of separate streams. These streams are completely separate, and so each stream cannot reference data from the other streams. So, if something crosses the edge of a slice, it cannot be referenced properly!

This effect is normally rather small, as it only affects the edges of the frame, but with OnLive's method, it affects about 8 times the number of macroblocks as it otherwise would, because it affects those on both sides of each slice boundary as opposed to merely the frame edges.

[Leak]

But OnLive isn't using slices: they're encoding the video as a bunch of separate streams. These streams are completely separate, and so each stream cannot reference data from the other streams. So, if something crosses the edge of a slice, it cannot be referenced properly!
I'd hazard a guess that ffmpeg considered "reduced compression" here to be a good thing (a la "higher bitrate"), i.e. less compression artefacts and such - which of course isn't the case, but only less chance for compression overall due to lack of 15/16th of the whole image for compression.

[ffmpeg]

I'd hazard a guess that ffmpeg considered "reduced compression" here to be a good thing (a la "higher bitrate"), i.e. less compression artefacts and such - which of course isn't the case, but only less chance for compression overall due to lack of 15/16th of the whole image for compression.

Your guess is right!

Now I understand "reduce compression" which is just a bad thing

Thank Dark Shikari's explaination

[easy2Bcheesy]

AFAIK Onlive is just using H.264, so this doesn't really go in the "new and alternative" category.

Their "new idea" is splitting the stream into 16 rectangular slices, each of which gets its own encoder. This brilliant idea massively reduces compression on the edges between slices when the scene is in motion and lets them brag about latency 16 times lower than what they actually have.

Interesting. So in what pixel configuration are the 16 rectangles assuming a 1280x720 source?

The use of h.264 always seemed like a given. So they don't use "group of pictures" and instead use "group of lots of rectangles" instead? I'm assuming that if they can't encode buffer "future" frames, they can still draw on remnants from previous frames?

This also means that the Micro Console is presumably just an off-the-shelf SoC, assuming it's capable of decoding the parallelised streams...

[MfA]

I really don't get why they would do it like this ... it makes it conceptually a little simpler to guarantee a definite limit on error propogation in coding, but simple to understand doesn't mean it's a good way ... which it really really isn't. Guaranteeing limits on error propagation isn't even needed. They have a bidirectional link, they can simply signal dropped blocks reliably to the server ... then the server can put intra blocks wherever needed to get them in sync again.

[Dark Shikari]

I really don't get why they would do it like this ... it makes it conceptually a little simpler to guarantee a definite limit on error propogation in coding, but simple to understand doesn't mean it's a good way ... which it really really isn't. Guaranteeing limits on error propagation isn't even needed. They have a bidirectional link, they can simply signal dropped blocks reliably to the server ... then the server can put intra blocks wherever needed to get them in sync again.The latency introduced by such a thing is probably too high. Even though it's a bidirectional link, it would probably still cost a bit of extra time. Periodic intra refresh (which they are using a form of) is good enough for error resilience anyways.

Another general problem is that the primary cause of dropped packets is connection saturation: which means that if you send a bunch of intra blocks to fix it, you often drop even more packets as a result.

[MfA]

It's the same as the control latency, which they say they keep under 80 msec ... so 5 frames at most. I doubt fixed schedule intra blocks updates are going to resync the codec any faster than that ... that's the equivalent of using max GOP 5 after all.

They don't even have to send an intra block though, if you don't mind making things a little more complex on the encoder side you could decode the stream minus the dropped blocks server side and then use the created references for encoding (which might still lead to extra intra blocks, but only if the error cover up was so bad that it's simply R/D optimal ... sometimes there is just no alternative no matter what your bit budget, just like with scene changes ... crap intra is still better than inter with an almost completely uncorrelated reference).

This is assuming rare drops though, rather than just continuous packet loss ... but the continuous part of the packetloss should be handled with an erasure code (Reed Solomon specifically). The unrecoverable drops should be rare.

[easy2Bcheesy]

In the original presentation (not this one), you can see that the latency is well above 80ms - there are several key moments where you can literally measure the amount of frames between mouse being moved and trigger being pulled and then motion on screen. Even in this presentation, when he's wobbling the mouse about (biggest indication of latency), it looks "off".

The 80ms claim is another piece of Perlman marketing that cannot be taken at face value. Perhaps 80ms in an absolute best case scenario, but bearing in mind that the fastest console games updating at 60Hz give response at 50ms *locally*, the claim is almost certainly BS.