Silverlight 3 beta with managed decoders support

[benwaggoner]

Silverlight 3 beta just shipped, with a bunch of new media features:

http://on10.net/blogs/benwagg/Silver...New-for-Media/

I told the H.264 guys about our new MPEG-4/H.264/AAC support, but I also wanted to mention our new Raw AV pipeline for doing video and audio decoders in managed code.

Raw AV Pipeline
MediaStreamSource started with the ability to pass off VC-1 and WMA bitstreams to the decoder from managed code, enabling us to add protocol and format support within Silverlight. Most of Smooth Streaming on the client is implemented in C# as MediaStreamSource code. And we’re extending that to H.264 and AAC-LC. But there’s a huge number of other codecs out there that different markets are using and have asked to have in Silverlight. In fact, far more than we can reasonably build and test, and far more than we’d want to have to make everyone download in the installer.

So, instead we added three very simple but very powerful new outputs to MediaStreamSource: YV12, RGBA, and PCM.

Yep, managed code can now output uncompressed video frames and uncompressed audio straight to the media pipeline.

Most video codecs are going to use YV12, which is a flavor of the a standard Y’CbCr 4:2:0 format used in most video codecs . We also wanted RGBA to enable video with alpha channels, and also any media formats that are natively RGB. And of course PCM is classic uncompressed audio.

Using a managed decoder is transparent to the user; it’s just another managed code .dll included in the .xap file. Since it’s running inside the Silverlight sandbox, there’s no download or other action required by the user to activate it; acts just like any other codec.

So, what’s codec performance like in managed code? It’s actually going to be fine for a variety of tasks. In our internal testing and prototyping, managed code offers about half the speed of a native compiled version. So for older codecs like MPEG-2, a managed decoder should be fine on a wide variety of hardware.

And remember, managed code doesn’t require C#. Existing decoders in C or C++ can be used as the basis for a managed decoder. The main restrictions are that native MMX/SSE and pointers can’t be used.

I can’t wait to see what other kinds of formats and codecs people wind up delivering with the Raw AV Pipeline. We’ve already seen some very neat demos.

So, if there's a protocol, parser, or decoder that has source code available, it can now be supported inside Silverlight.

Given Silverlight's rich rendering model, this could be as basic as MPEG-1, or as rich as something like MKV with Theora and Vorbis as codecs.

[Dark Shikari]

Originally Posted by benwaggoner

The main restrictions are that native MMS/SSE

Typo?

Originally Posted by benwaggoner

and pointers can’t be used.

I'm not entirely sure how you would write anything of note in C without pointers...

[benwaggoner]

Originally Posted by Dark Shikari

Typo?

Thanks, fixed.

I'm not entirely sure how you would write anything of note in C without pointers...

Oh, I'm sure all existing code will need to be modified. But it won't require a full port to C#.

We've done a couple of these internally for testing purposes, and it was only a few day's work for a not-too-complicated codec. I'm glad we don't have to do H.264 as managed, certainly .

[benwaggoner]

And here's a completely crazy use of the Raw AV pipeline: realtime C64 emulator!

The idea was codecs, but I guess we designed it pretty flexibly.

http://community.irritatedvowel.com/...eamSource.aspx

Anyway, it also has some good sample code for how the codec blits and such.