Hardware needed for 1000FPS?

[Mr Alpha]

The announcement of OnLive at GDC has caused a lot of buzz. Bringing a thin client kind of computing to gaming sounds outrageous. You have a tiny cheap console with a mouse, keyboard, screen, speakers and an internet connection. Your inputs are streamed to a data center where the game is run and rendered. The results are then encoded and sent back to the console where the stream is decoded and sent to the screen and speakers.

The technical feasibility of this has been questioned. Latency would be a big problem. The stream is 720p60, and for it a 5Mbps connection is needed. I assume this means a around 4Mbps for the video stream so that there is a little left over for audio and some headroom. They say the encoding latency is 1ms, which would mean they are encoding at 1000FPS. According to people who have seen it, the quality is pretty good with only limited amounts och compression artifacts. I assume it has to be a single pass since it is streamed live. Now, this is presumably done in some huge server farm, but I got to wondering, roughly, what kind of hardware would be needed to encode a stream like this at a 1000FPS.

Now, my experience with encoding is limited but my intuition says that to do 1000FPS, even with modern hardware, would require lots of processors. But you guys have a lot more experience that I do so I figured I would ask you. How much hardware (say, in quad-core processors) do you need to do 1000FPS?

[cyberbeing]

They aren't encoding at 1000fps... They only need to encode at ~64fps+ (16.666ms-1ms=15.666ms) to achieve 60fps with a 1ms latency. At least that is how I read it. If not, then they must have a huge cache of frames and short clips with every possible outcome, or are just overlaying the user interactive components over a pre-encoded video.

In any case, I'm going to withhold judgments on this service until it is actually launched.

[Mug Funky]

yeah, latency is just the delay in the stream hitting you from when it's rendered i guess (or is it between you hitting a key and the response showing on the screen?).

either way, i don't see why it'd be necessary to do things this way with GPUs getting as good as they are. seems the hardware required for the decode would approach that required for the rendering, unless some of the on-chip decoder doobies are used (which would lower latency but possibly require a more modest encoded stream, quality wise).

1000 fps would be nice... there's a camera already doing far in excess of that...
http://www.visionresearch.com/

2.5k at 1400fps... yummy

[Manao]

Actually, it depends how they define latency. if it's time the last byte of the frame is encoded minus the time of the first pixel of the frame is sent to the encoder, then having a latency of 1ms implies encoding at 1000 fps. If it's time of the first encoded byte minus time of the first pixel sent, then latency means nothing in regard to encoding speed.

Regardless of that minor detail, imho the whole project is a huge scam. From my field of expertise (video encoding), 1ms latency is unheard of. We know how to achieve it, but the sacrifices in regards to quality are enormous (basically, mjpeg with constant macroblock size), and that isn't compatible with 720p60 @ 5 mbps.

[Mr Alpha]

The latency that really matters is how long it takes for the input to appear as output, from the gamers perspective. The sequence is this: I move my mouse and the signal is sent via the USB port to the mouse driver.
The driver gives the data of my input to the software, which in turn passes it along to my networking stack.
The networking stack sticks it in a packet and sends it scurrying along the internet tubes to their server farm.
There their networking stack receives it and grabs the data out of the packet and gives it to the game engine.
The game engine uses the data to update the game state, and based on the updated game state is renders a new frame.
The frame is then given to the encoder which encodes it and gives it to the networking stack.
The networking stack sticks the encoded frame in a stream of packets which scurry back along the internet tubes to me.
Here my networking stack receives them and extracts the frame from the packets and gives it to the software.
The software gives the encoded frame to the decoder, which decodes it and then gives the decoded frame to the display driver.
The display driver sends it out to the screen which then does what it does to turn it into a picture on the screen.
This picture on the screen then reflects the fact that I moved the mouse.
The latency gamers care about is how long it takes for the result of a input to appear on the screen. The question here is how much does the encoding process (step 6) add to this latency.

[Manao]

That's a bit more complicated than that.

Firstly, the encoded frame can be sent over the network without being entirely encoded, and the decoding can start without the frame having been completely received.
Secondly, since you send the video at a constant bitrate, if you allow for some bitrate variation (using a VBV), then you have an additional latency induced by the VBV : you have to buffer slightly the video to account for possible bitrate spikes. That has a huge impact on the video quality (my reference to constant macroblock size is what happens when that buffer doesn't exist)

[Mr Alpha]

Firstly, the encoded frame can be sent over the network without being entirely encoded, and the decoding can start without the frame having been completely received.I was wondering about that. If it started sending parts of the frame off before the whole frame was encoded, that would let the decoder start decoding earlier than it otherwise would. But you'd still need the whole frame before you could display it, right? So at some point in the sequence it would have to stop and wait for the rest of the frame to arrive. So this would only help mask decoding latency, but you'd still feel the full effect of the encoding latency.

[Comatose]

Presumably the encoder will encode a few frames in advance (GPUs today already do this - usually limited to 3), so that it's always a little ahead.

[Brother John]

@Mr Alpha
The process doesn’t necessarily have to stop, i.e. the delay could actually be reduced. Look at the crude image I made.

http://img3.imagebanana.com/img/a25eg0yf/delay.png

Strict sequential processing ends at time B. If you start transmitting before the whole frame is encoded and start decoding before the transmission is complete the whole process finishes earlier at A. I.e. you can display the frame earlier.
How much you can gain depends on the speed of encoding, transmitting and decoding.

[Dark Shikari]

From what I hear they are actually using JPEG, which means it is in fact a scam.

[Blue_MiSfit]

...srsly?

They plan on doing MJPG at 5mbps for 720p? That's laughable...

~MiSfit

[squid_80]

From what I hear they are actually using JPEG, which means it is in fact a scam.
That's interesting, I hear they definitely licensed an AVC decoder.

[Dark Shikari]

That's interesting, I hear they definitely licensed an AVC decoder.There are two physically impossible things about using AVC here:

1. Frame encoding time of 1ms: probably completely impossible for HD; even using the world's best ASIC, because the bitstream coding alone takes longer than that (and has to be done linearly).

2. At the latency they're looking at, you basically can't use inter prediction in the normal sense.

You could have a stream with no keyframes using periodic intra refresh, but the efficiency of that would be way too low to be practical.

You could have a stream with one keyframe at the start and no intra refresh, but that wouldn't have the error-resilience properties they claim.

[squid_80]

I wasn't saying AVC is definitely being used for the streaming, I said I hear they've got an AVC decoder, which could mean:

- They are using AVC for streaming, which means quality/playability is far below their claims
- They're not using AVC for streaming but are making a big effort to make it seem like they are

As an Australian I've always thought the whole idea was unfeasible simply because 200ms pings are the norm here and I have enough issues trying to play console games through my capture card, where any latency beyond 120ms starts to be noticeable.

[Manao]

1. Frame encoding time of 1ms: probably completely impossible for HD; even using the world's best ASIC, because the bitstream coding alone takes longer than that (and has to be done linearly).
Once again, you don't have to have the entire frame to start encoding. A 60 fps encoder can encode with a latency of 2 ms if it doesn't wait for the entire frame, but just for a macroblock row.
You could have a stream with no keyframes using periodic intra refresh, but the efficiency of that would be way too low to be practical.You don't know what you're talking about. Periodic intra refresh doesn't reduce coding efficiency in comparison to a short GOP (say, 15 frames)

[Mr Alpha]

Steve Perlman, founder of OnLive, says (http://www.mcvuk.com/news/33788/Perlman-OnLive-critics-are-ignorant) this:"[...]We are not doing video encoding in the conventional sense.

“It has taken tens of thousand of man hours to develop the algorithm, but it’s not perfect. You will sometimes see little artefacts on the screen. Video compression is part science and part art. Every time you present new material to it, you will see something it does not compress so well. We note those and correct the algorithm.”

Perlman explains that over the course of OnLive’s development the company has slowly been able to upscale its compression methods. “First of all it was a postage stamp size screen with no latency over the internet. It looked like the silliest kind of game because the screen size was smaller than a cell phone but nonetheless there was no lag.”

The firm now claims to be able to stream 720p HD visuals at a seemingly astonishing 60fps over a 5MB internet connection with barely noticeable lag.

[Dark Shikari]

Steve Perlman, founder of OnLive, says (http://www.mcvuk.com/news/33788/Perlman-OnLive-critics-are-ignorant) this:They can't even make up stuff very well either, it seems...
Once again, you don't have to have the entire frame to start encoding. A 60 fps encoder can encode with a latency of 2 ms if it doesn't wait for the entire frame, but just for a macroblock row.But that's not what they said. They said it took 1 millisecond to encode the entire frame.You don't know what you're talking about. Periodic intra refresh doesn't reduce coding efficiency in comparison to a short GOP (say, 15 frames)That's not what I said. I said that periodic intra refresh is too expensive coding-wise for what they're doing--which is the same as saying that ultra-short gop is too expensive coding-wise for what they're doing--which is true. And a 15-frame gop is nowhere near good enough for the kind of error resilience properties they claim anyways.

Please stop putting words in my mouth.

[Brother John]

Definitely sounds like press/marketing gibberish. There’s a very simple and reliable method for them to prove their claims though …

[Dark Shikari]

Definitely sounds like press/marketing gibberish. There’s a very simple and reliable method for them to prove their claims though …A large portion of their employees are recent graduates of my college (and I trust them), so I suspect what's going on is one of three things:

1. It'll never work, and they got themselves sucked into a legendary company of suck (http://thedailywtf.com/Articles/Virtudyne_0x3a__The_Founding.aspx) that will never be able to do what they claim, but it pays great and they get lots of free time, so who cares?

2. It does work, but pretty badly. Marketing's absurd claims has gotten everyone's expectations so high that when it actually comes out, everyone will be disappointed. They're cursing marketing under their breath.

3. It doesn't work, and not too badly at all, using the only method that makes any sense (hard-CBR mostly-inter encoding, possibly with slow intra refresh). Marketing is still making crap up, and they're still cursing marketing under their breath, but it's going to work to some extent.

[Manao]

Dark Shikari : Assuming no bframes and an average intra frame size of 4 times the average inter frame size, a 15 frame GOP increase the bitrate by 20% in comparison to an infinite GOP size. 20% isn't a physical impossibility.
But that's not what they said. They said it took 1 millisecond to encode the entire frame.No, they didn't. They said video latency was 1 ms. It's other people relaying the information that claimed 1ms latency == 1000 fps. See this link (http://venturebeat.com/2009/03/30/game-industry-executives-react-to-onlive-video-games-on-demand-announcement/) for a confirmation.

Finally, an intra refresh of 15 frames means that an error is recovered in 0.25 seconds, which isn't bad (especially with a good error resilience algorithm on the decode side).

Anyway, thinking more about it, constant MB-row size with horizontal intra refresh might reduce the VBV latency enough to reach the 1 ms without mjpeg-like efficiency. That's feasible - it would be a wonderful piece of engineering, and I seriously doubt I'd see such a latency in the next 10 years, but I must admit it's feasible.

That doesn't remove all the other causes of latency that makes the project's claims imho more than dubious.

[drmpeg]

I've never heard of these guys, but it seems a fully pipelined encoder is possible.

http://www.ocean-logic.com/pub/OL_H264MCE.pdf

Ron