How much power is needed to encode realtime HDTV?

[drbuzz0]

Okay, first of all, I know this is something of a loaded question because it has some variables in it, but I've been wondering lately about the possibility of capturing uncompressed High definition for the purposes of PVR stuff and video archiving.

I know this is not optimal, because you end up decoding and re-encoding the video content and thus can degrade quality and end up ineffeciently recompressing it at a higher resolution than necessary, but considering how crappy the cable and satellite companies are at allowing for recording it might be worth it.

There are currently some HDMI capture cards which are not so bad in terms of price. In the case of the Decklink one, it can capture to a MJPEG codec or uncompressed or via direcshow. The problem is that the uncompressed HD content is so massive that it would flood most disk interfaces and is just too huge to handle.

Thus, it really needs to be compressed in real time to make it managable. Compression to a high quality Mpeg-4 AVC codec would be optimal, but that's obviously going to be very processor intensive. Compression via mpeg-2 at a reasonable quality would be acceptable though. It could always be retranscoded if it's captured at a high enough bitrate anyway.


Is this something which is reasonable to do with software and a reasonably high end system? Would, for example, a quad core intel processor with two to four gigs of ram and a couple of WD-Raptor hard drives be able to handle encoding in realtime without choking? Is there any example of a system that can do this with software?

[burfadel]

Mpeg-2 is simple in comparison to h.264 etc, and therefore requires much less CPU usage.

I use a leadtek digital tv tuner card for terrestrial digital tv, including HD 1920x1080, and use the hardware encoder via the settings in the winfast pvr programme. In silent record mode (that is, so the video isn't displayed) there's at most 2 percent cpu load on a core 2 duo e6600 (average is less). The cards can even record transport streams, so say a HD channel and a couple of standard definition streams with different content! The HD encode is at true 1920x1080. its not down-sampled. All that on a AUS $45 tv card! - it doesn't seem to choke the bandwidth of the hard drive either, you can still do normal stuff quite easily on it

The CPU is much less effective in terms of compression as its not specifically designed for it, however even a modest CPU should be able to handle encoding hdtv in realtime with the correct software.

[Dark Shikari]

For 1080p/i realtime encoding with x264, you need either a heavily overclocked quadcore or an eight-core Xeon system.

[Blue_MiSfit]

Yeah it's probably a better idea to digitally capture the transport stream.

If you must capture uncompressed, prepare for at least 3 hard drives in RAID 0. You will be able to capture real time in YUY2, but it will start dropping frames as it fills up. Prepare to be very limited.

I think a better solution is capturing to MJPEG. It's not bad, and is a whole lot smaller than uncompressed!


For 1080p/i realtime encoding with x264, you need either a heavily overclocked quadcore or an eight-core Xeon system.


Very true. And you will still have to back off on the settings to manage this.

~MiSfit

[Dark Shikari]

Very true. And you will still have to back off on the settings to manage this.Not too much though; Avail Media uses an 8-core Xeon with x264 for real-time 1080i encoding. I think they're able to at least use a couple refs and partition types. Definitely no RDO though unless you overclock your Xeon system, too :p

[ilovejedd]

Yeah it's probably a better idea to digitally capture the transport stream.

~MiSfit

Hmm, is this possible with satellite and cable in the US? I'm trying to look for cable cards but alas, there seem to be none supported by our cable provider. I was looking into a satellite receiver for the PC but no luck there either.

If you're sticking with MPEG2, I see no reason why you won't be able to encode with a Q6600 or better and maxed out system specs (RAM & 10k/15k HDDs). Hmm, don't know if there are MPEG2 encoders that support SMP and can acquire live feeds from a capture card, though. If you can't find a method, then stick with MJPEG.

Unless you have really big, really fast hard drives in RAID 0, encoding to MPEG2 would probably be easier than encoding raw uncompressed HD. I was doing some calculations for 1080i (raw RGB) and a single frame would be around 6MB or 48 megabits. For one second of video, that's 180MB, so yeah, you'll be needing some major fast hard drive space. Hmm, does anyone know what the burst and sustained write speed is for Raptors?

If you already have a high end system, maybe you can "simulate" the MPEG2 encoding process. Iirc, individual PNG images for frames of Elephants Dream is available for download so try converting that to MPEG2 as a test. Source speed might prove to be a bottleneck in this case, though.

[foxyshadis]

On raptors, sustained read and write are about the same. 70-90 MB/s depending on the model.

It's reasonably simple to set up virtualdub to capture from almost anything to a vfw codec, and you can configure xvid or x264vfw to be fast enough to encode realtime. Just don't use b-frames or cabac or anything like that. Alternately, lagarith.

[squid_80]

Unless you have really big, really fast hard drives in RAID 0, encoding to MPEG2 would probably be easier than encoding raw uncompressed HD. I was doing some calculations for 1080i (raw RGB) and a single frame would be around 6MB or 48 megabits. For one second of video, that's 180MB, so yeah, you'll be needing some major fast hard drive space. Hmm, does anyone know what the burst and sustained write speed is for Raptors?
I can normally capture 1080i(30fps) using huffyuv to a single sata drive (no raid at all). If the source is video there's no problem but if it's something like a pc game there's just not enough compression, the hard drive overloads and it craps out giving dropped frames. Lagarith just doesn't cut it speed wise, nor do any other lossless codecs I've tried.
(This is through DVI/HDMI - would probably be a different story with an analog source.)

[foxyshadis]

Even with 4-core? That's a pity.

[burfadel]

Yeah it's probably a better idea to digitally capture the transport stream.



It probably does just capture the transport stream, and not encode it when capturing transport stream... however even so just encoding HD 1920x1080 has the same cpu usage etc.

Those with Nero recode, just consider how fast than encodes mpeg-2, even though (until recently?) it only used a single core...

Capturing HD at 1920x1080 in mpeg-2, using good but not ridiculous settings should be quite easily capable on just about any recent CPU. h.264/avc/xvid etc isn't as good to work with, as its cpu requirements for reasonable settings as pointed out is very high, and is much harder to work with editing wise. The quality of hd in mpeg-2 format can be very good, say at 6gb/hour for example.

[squid_80]

Even with 4-core? That's a pity.
Even when doing a capture test (doesn't write so hard drive speed is irrelevant) it can't keep up. Same goes for ffdshow's mpeg2 encoder, it hits 400% cpu usage in virtualdub and starts dropping frames. Huffyuv and Cineform are the only ones that keep up.

[drbuzz0]

Mpeg-2 is simple in comparison to h.264 etc, and therefore requires much less CPU usage.

I use a leadtek digital tv tuner card for terrestrial digital tv, including HD 1920x1080, and use the hardware encoder via the settings in the winfast pvr programme. In silent record mode (that is, so the video isn't displayed) there's at most 2 percent cpu load on a core 2 duo e6600 (average is less). The cards can even record transport streams, so say a HD channel and a couple of standard definition streams with different content! The HD encode is at true 1920x1080. its not down-sampled. All that on a AUS $45 tv card! - it doesn't seem to choke the bandwidth of the hard drive either, you can still do normal stuff quite easily on it

The CPU is much less effective in terms of compression as its not specifically designed for it, however even a modest CPU should be able to handle encoding hdtv in realtime with the correct software.



When you record HDTV from a card like that you're not encoding it. It might have a hardware encoder for analog capture but the HD tuners can capture the video because they snag the transport stream. The digital broadcast is compressed to begin with.

I'm talking about something entirely different. With stuff in the US like directv and most cable companies you cannot capture the transport stream because there are no PC-based tuners and the tuners they give you either don't have any output or if they do have firewire out then it's usually encrypted and only for approved devices.

Therefore if you want to capture you'd have to grab the uncompressed video. This is obviously less than ideal.

[unix_sansei]

anyone have some good idea that i wanted to capture 1080p 24fps A/V off of a GiGe? or Firewire?

thanks

[drbuzz0]

anyone have some good idea that i wanted to capture 1080p 24fps A/V off of a GiGe? or Firewire?

thanks

Well that's pretty easy if you have a 1080p source that outputs the stream over firewire of GiGe. Otherwise there's a problem..

[unix_sansei]

Well that's pretty easy if you have a 1080p source that outputs the stream over firewire of GiGe. Otherwise there's a problem..

ok what would you use for acquisition? does Vdub or Flask have any hooks built-in to look at the GiGe pipe? and know the packets coming in are AV?

is there a way to get around having to use the cineform codec?

the potential source is that Red One camera, it puts out 1080p, 2k, and 4k frames at 24fps with wavelet compression. so i think data rate would max out around 140Mb/s

[ilovejedd]

Correct me if I'm wrong but from what I've read on the tech specs page, the Red One camera outputs raw YUY2 video (either compressed or uncompressed). From your comment, it would seem that the CineForm codec supports decoding of the recorded video, but I don't think it's really needed since there are a bunch of free codecs and even programs that can decode/read raw video directly.

I dunno about Flask but VirtualDub has a capture mode. Assuming you have the necessary drivers installed which identify your device as a video capture device, then you should be able to capture using VirtualDub. Albeit, I don't know how well it works for HD videos.

[unix_sansei]

Correct me if I'm wrong but from what I've read on the tech specs page, the Red One camera outputs raw YUY2 video (either compressed or uncompressed). From your comment, it would seem that the CineForm codec supports decoding of the recorded video, but I don't think it's really needed since there are a bunch of free codecs and even programs that can decode/read raw video directly.

I dunno about Flask but VirtualDub has a capture mode. Assuming you have the necessary drivers installed which identify your device as a video capture device, then you should be able to capture using VirtualDub. Albeit, I don't know how well it works for HD videos.

yeah,it's that driver that's the tough part. once you get the huge 4k files to disk, these are monster 10-bit not 8-bit, there is very little software that can edit. even premiere dumps everything down to 8-bit. final cut pro can but i'm not buying a mac for that. after affects can somewhat but that's a kluge.

i don't know if vdub would stand up to a 10-bit file. ah, the data gets stored in 10-bit 3d raster files, r3d

[ilovejedd]

Then it would seem you don't really need drivers. You just need software that can manipulate the video output.

[unix_sansei]

Then it would seem you don't really need drivers. You just need software that can manipulate the video output.

the drivers will dump it into a format where you can only read it with final cut pro, native, or you have to run it through a quicktime compressor.

i'd rather just grab the stream off the gigE port and compress on the fly to something more manageable.

yes the video editing is also an issue. i'd rather use vdub/flask as premier is expensive and bloated or even better setup avisynth and let it crunch away.

http://austinfilmtools.com/pages/gear_red_one/red_one_workflow.htm

here is an what one possible work flow diagram.

[David Newman]

To answer the threads question, the power to encode HDTV. CineForm NEO HDV or NEO HD will encode 1080p24/i60 from the Black Magic Intensity card using a nice 2.33Ghz Core 2 Duo system. The bit-rate won't be in the MPEG2/AVC range, yet they are low enough to be sustains to a single hard drive, no RAID required. 1080i60 from the Intensity card will be between 14-18MBytes/s, as compared to 120MBytes/s for uncompressed 4:2:2.

Now that the thread has moved on to issue of RAW compression. CineForm and Red are the only companies to offer RAW compression solutions today, and CineForm's is only one available to work on any camera through tools Streampix (http://www.norpix.com/supported_hardware/cameras.php), supporting support a wide range of GigE, USB and CameraLink style cameras - DIY-RED. The RAW compression is 12bit, which works fine in Adobe tools (which now supports up to 32-bit float per channel), FCP somewhat and yes VirtualDub is still 8-bit.


David Newman
CTO, CineForm
blog : http://cineform.blogspot.com