H.264 support coming in Silverlight

[benwaggoner]

There've been some past discussions about this, so I wanted to give everyone a heads up that we announced forthcoming support for H.264 and MPEG-4 in Silverlight.

Details here:
http://on10.net/blogs/benwagg/H264-and-AAC-support-coming-in-Silverlight/

[Dark Shikari]

Edit: I apologize, I apparently misread "MIPS" as "MBPS". Its probably easier to just say "processor power" given that the distinction between the two terms is rather important...

Good news it seems: Flash won't have much over Silverlight for much longer at this rate.

[smok3]

i could write a really nasty post here, but instead i'll just say: good work Ben!

[Atak_Snajpera]

VC-1 only needs about half as many MIPS per pixel for playback.
So what? DXVA solves that problem

[mitsubishi]

Well it's about time, I've seen a lot recently from MS acknowledging the weight of opinion on this, so thought it would be coming. It should go without saying, but this includes the same codec deal for moonlight? Also what is meant by the version after 2.0? 2.1, 2.0SP1 or 3.0? ie is it happening soon.

EDIT: Thanks btw, one of things I was hoping to hear this year.

[benwaggoner]

So what? DXVA solves that problem
Alas, given how Silverlight rendering works, decode is software only for the time being. Plus we need to run on systems without DXVA, and on XP where DXVA support, particularly for H.264 is... unpredictible. Plus we're also on Mac, where there's really no infrastructure for hardware decode acceleration.

However, it's one of those cases that the systems where DXVA isn't a safe option are also the systems where you'd most want it from a perf perspective.

Agreed that the codec perf difference isn't much of a factor when there's some kind of hardware support, but we're some years away from that being a safe assumption for a browser plugin like Silverlight.

[benwaggoner]

Edit: I apologize, I apparently misread "MIPS" as "MBPS". Its probably easier to just say "processor power" given that the distinction between the two terms is rather important...
Yeah, that's a pretty big difference. I imagine I missed reading something very rude while I was off at lunch :).

Processor Power's been confusing for people as well in the past, but I agree MIPS isn't really any better. I'm happy for any suggestions for what clear terminology for this would be.

Good news it seems: Flash won't have much over Silverlight for much longer at this rate.
I sure hope so!

[toytown]

Great news, i just posted on another forum today, saying that the lack of H264 decoding was really one of the sticking points for webmasters who were pushing video content

[Sagekilla]

Twice the performance or speed would work well too if you're running at 1/2 the mips per pixel apparently ;)

[Sharktooth]

thats indeed a really good news and im sure it will help the silverlight adoption a lot...
just out of curiosity, does that also mean expression encoder is going to have h.264/mpeg4 too?

[benwaggoner]

thats indeed a really good news and im sure it will help the silverlight adoption a lot...
just out of curiosity, does that also mean expression encoder is going to have h.264/mpeg4 too?
Yep. From my blog post:

"At IBC 2008 we will be demonstrating a technology preview of H.264 video and Advanced Audio Coding (AAC) audio playback support in Silverlight, and H.264 authoring using Microsoft Expression Encoder and Windows Server 2008 for delivery. "

I'm excited myself, as I haven't actually seen the build with that working yet :).

[Sharktooth]

i missed that part. glad to see some of my suggestions were usefull.

[benwaggoner]

i missed that part. glad to see some of my suggestions were usefull.
We're always listening, even if we can't always do everything everyone wants immediately.

[Schrade]

Ben, thanks for posting this. I'm sure this will be awesome news for our upcoming software at Winnov :) I want to get my hands on that h.264 enabled version of Expression Encoder.

[benwaggoner]

Ben, thanks for posting this. I'm sure this will be awesome news for our upcoming software at Winnov :) I want to get my hands on that h.264 enabled version of Expression Encoder.
Come by IBC and check out what we've got.

Note that we we won't neccessarily have feature parity on H.264 in the initial version of Expression Encoder that supports it. The more important hurdle is feature parity on the decode side, since the key reason to have H.264 in Silverlight is to support existing libraries of H.264 content. When something is being encoded specifically for Silverlight, using VC-1 isn't nearly as big a deal.

[smok3]

shame i'll miss IBC again this year :/

[CruNcher]

Perfect benwaggoner i guess for many this will be the end of Flash if it stays such resource intense as it is currently is, btw why is DXVA via the Silverlight API not possible, i guess you will also License the H.264 core from DivX/Mainconcept right or CoreCodec ? :)

[Quark.Fusion]

I think DXVA via the Silverlight API not possible because video can be overlayed (player controls, etc.) and DXVA requires direct connection to compatible renderer to work. (Same reason why you can't do postprocess with DXVA)

[CruNcher]

Hmm Quark Fusion that's no real reason you can also overlay Player controls over DXVA via the Exlusive VMR9 Rendering Mode tough it's more work you have todo because its Fullscreen and 3D but its possible, so when you switch in Silverlight to Fullscreen it would just need to load this Exclusive mode and change it's interface (tough a interface isn't really needed @ all honestly in Fullscreen it's also up to your Website Design and how you gonna implement Silverlight into it and for which purpose) :)

[Comatose]

[...] and H.264 authoring using Microsoft Expression Encoder [...]
Will you be providing information on how to author externally for Silverlight?
That is, technical requirements, etc.

Also, will you be using a container like MP4 or a proprietary container?

[pandy]

"Comparing VC-1 Advanced Profile to H.264 High Profile with all the bells and whistles turned on, VC-1 only needs about half as many MIPS per pixel for playback."

ooooops - from my observations this claim is untrue - VC-1 is slow, especialy when decoder from MS is used, also seems that present implementations of encoders/decoders are far from h.264...
seems that "superiority" of the VC-1 over the h.264 is like "superiority" Vista over XP.

[Inventive Software]

He's talking with regard to most stand-alone players, which is mostly true. VC-1 is highly asymmetrical in it's performance (very slow encoding, fast decoding), while H.264 can mostly be made symmetrical, but is best asymmetrical.

[pandy]

In stand-alone solutions this is also untrue - hardware accelerated playback is no problem at all - due request of BD or DVB/ATSC h.264 and VC-1 is mandatory for stand-alone players so both codecs are comparable, also from market point of view VC1 is closely related with MS platforms and in long range term much more costly than h.264.

I use WinXP and VC-1 from MS is 1/3 fast as libavcodec implementations (which itself is far from perfect) - so where is "superiority"?

sorry for emotions, i try to be polite :D

[Sharktooth]

please, there are plenty of avc vs vc-1 threads, dont hijack this one into one of them...

[Quark.Fusion]

Hmm Quark Fusion that's no real reason you can also overlay Player controls over DXVA via the Exlusive VMR9 Rendering Mode tough it's more work you have todo because its Fullscreen and 3D but its possible, so when you switch in Silverlight to Fullscreen it would just need to load this Exclusive mode and change it's interface (tough a interface isn't really needed @ all honestly in Fullscreen it's also up to your Website Design and how you gonna implement Silverlight into it and for which purpose) :)

What reason to switch to fullscreen when you can watch streaming in your favorite mediaplayer with your favorite postprocessor? :)
And that doesn't solve non-fullscreen problem. You need DXVA 2.0 here.

[benwaggoner]

I think DXVA via the Silverlight API not possible because video can be overlayed (player controls, etc.) and DXVA requires direct connection to compatible renderer to work. (Same reason why you can't do postprocess with DXVA)
Yep, that's a big part of it.

Also, DXVA is simply not one of those things you can assume is going to work for a typical user. Guys around here, sure, but not on your average corporate desktop with old video card drivers.

[benwaggoner]

I use WinXP and VC-1 from MS is 1/3 fast as libavcodec implementations (which itself is far from perfect) - so where is "superiority"?
I'm assuming equally well optimized implementations of both codecs. The Win XP software decoder is pretty old at this point; there's plenty of optimization we've done since then. Also, that's going to be using postprocessing by default unless you turned it off or you're on a very slow machine, which will add additional CPU utilization.

It's also hard to tease how much of the decode load is from decoder versus the rest of the player when comparing player to player.

This info is from some cases where we've done both decoders on the same platform, like the Xbox 360. CABAC and a loop filter that touch a lot more pixels add quite a bit of overhead for software decode. It's enough that we've had multiple partners contact me to make sure we're NOT dropping VC-1, since they've found H.264 HD decode too expensive on low end machines.

[benwaggoner]

Hmm Quark Fusion that's no real reason you can also overlay Player controls over DXVA via the Exlusive VMR9 Rendering Mode tough it's more work you have todo because its Fullscreen and 3D but its possible, so when you switch in Silverlight to Fullscreen it would just need to load this Exclusive mode and change it's interface (tough a interface isn't really needed @ all honestly in Fullscreen it's also up to your Website Design and how you gonna implement Silverlight into it and for which purpose) :)
But even when we're running Silverlight in Full Screen we still support multiple elements, compositing, etcetera. Also, Silverlight is mostly run in-browser.

[puffpio]

There've been some past discussions about this, so I wanted to give everyone a heads up that we announced forthcoming support for H.264 and MPEG-4 in Silverlight.


...... .. ... ....... YYYEEESSSS!!!!
I remember a talk or webcast you gave when h264 was not even on the radar as a feature.
hopefully your decoder will blow the pants off flash's in terms of cpu usage (altho i hear the next version of flash will get a nice speed boost).

license coreavc maybe? :D

[smok3]

It's enough that we've had multiple partners contact me to make sure we're NOT dropping VC-1, since they've found H.264 HD decode too expensive on low end machines.
how did they benchmark?

[Dark Shikari]

He's talking with regard to most stand-alone players, which is mostly true. VC-1 is highly asymmetrical in it's performance (very slow encoding, fast decoding), while H.264 can mostly be made symmetrical, but is best asymmetrical.VC-1 is only highly asymmetrical because the Microsoft encoding implementation is very inefficient. I suspect it can be made just as symmetric as H.264.

[pandy]

I'm assuming equally well optimized implementations of both codecs. The Win XP software decoder is pretty old at this point; there's plenty of optimization we've done since then. Also, that's going to be using postprocessing by default unless you turned it off or you're on a very slow machine, which will add additional CPU utilization.


Hmmm so You say that for fast VC-1 i need MS Vista which itself need VERY POWERFULL PC due DRM mambos jambos inside system architecture...


It's also hard to tease how much of the decode load is from decoder versus the rest of the player when comparing player to player.


can we assume that simple graphedit can be used as a reference "player"


This info is from some cases where we've done both decoders on the same platform, like the Xbox 360. CABAC and a loop filter that touch a lot more pixels add quite a bit of overhead for software decode. It's enough that we've had multiple partners contact me to make sure we're NOT dropping VC-1, since they've found H.264 HD decode too expensive on low end machines.

Ben... please... on XP is slow so i must install Vista, for Vista i must use quite power full PC due "flaws" in OS architecture... so where is a difference between "slow" h.264 and "fast" VC-1???

LOW END machines definitely will not take Vista... Will You Ben say that MS create alternative to the MS Vista (maybe open source), focused on media market (real time architecture OS, less jittery than present MS OS's?)

PS
This is not another thread about which is better VC-1 or h.264 - I'm a engineer (not so young and definitely I'm not a fanboy) and i expect some facts not mambosjambos about codecs. I appreciate Ben for His hard work and respect Him but i don't understand reason for marketing or advertisement on Doom9 forum (especially when we don't have facts).

[Ranguvar]

*wonders how this turned from Silverlight discussion, to H.264 vs VC-1, to some sort of Windows bashing*

[Inventive Software]

VC-1 is only highly asymmetrical because the Microsoft encoding implementation is very inefficient. I suspect it can be made just as symmetric as H.264.

It can't, not for HD, and for SD you're gonna struggle on anything less than a Core 2. It's just plain slow encoding.

[benwaggoner]

...... .. ... ....... YYYEEESSSS!!!!
I remember a talk or webcast you gave when h264 was not even on the radar as a feature.
hopefully your decoder will blow the pants off flash's in terms of cpu usage (altho i hear the next version of flash will get a nice speed boost).

license coreavc maybe? :D
We're building our own implementation.

Do we know that Flash's H.264 decoder is slow in particular, or just that Flash is slow :). It's hard to tease out where the cycles are going in something like Flash; if someone has, I'd sure love to know what the target we're trying to hit is!

[benwaggoner]

It can't, not for HD, and for SD you're gonna struggle on anything less than a Core 2. It's just plain slow encoding.
For live encoding?

Try Expression Encoder 2's live encoding with "Dynamic Complexity" turned on. It'll scale the encoding complexity to what the local hardware is capable of without dropping frames. Faster machines will offer better quality, of course, and HD requries a lot of horsepower. But my laptop can certainly do live 640x480 reasonably well.

Anyway, encoding time and decoding time are quite orthgonal.

[benwaggoner]

It can't, not for HD, and for SD you're gonna struggle on anything less than a Core 2. It's just plain slow encoding.
For live encoding?

Try Expression Encoder 2's live encoding with "Dynamic Complexity" turned on. It'll scale the encoding complexity to what the local hardware is capable of without dropping frames. Faster machines will offer better quality, of course, and HD requries a lot of horsepower. But my laptop can certainly do live 640x480 reasonably well.

Anyway, encoding time and decoding time are quite orthgonal.

[benwaggoner]

how did they benchmark?
Some tried in Flash, others with other H.264 decoders. I imagine ffmpeg was in there somewhere.

[benwaggoner]

Hmmm so You say that for fast VC-1 i need MS Vista which itself need VERY POWERFULL PC due DRM mambos jambos inside system architecture...
Talking about using WMP/DirectShow/Media Foundation for playback? Yes, Vista will be better if you're using DXVA, but as long as you have WMP 11 installed on XP, it's about the same for software decode. Vista just has a much higher probabilty of using DXVA.

There's no DRM utilization in WMP if the content doesn't have DRM applied. Yes, I know that guy in New Zealand wrote that whitepaper about what he imagined Vista might have been like if we did everything in the worst possible way, but every emperical prediction he made there has been refuted by those actually running Vista :).

can we assume that simple graphedit can be used as a reference "player"Sure. That's DirectShow, and the same decoder (software or DXVA) as you'd get playing back in WMP.

Ben... please... on XP is slow so i must install Vista, for Vista i must use quite power full PC due "flaws" in OS architecture... so where is a difference between "slow" h.264 and "fast" VC-1???
I don't understand your question.

Also Vista runs reasonably well on "older" machines. You won't get Aero Glass on a pre-DX9 GPU, but that's about GPU, not CPU performance.

My point is that some of the more complex features of H.264, particularly a loop filter that touches more pixels with more taps, and CABAC, mean that the worst-case performance for a H.264 High Profile encode is about 2x the worst-case performance for an Advanced Profile VC-1 encode at the same bitrate. It won't matter in lots of cases.

FWIW, the Silverlight 2 VC-1 decoder is currently decoding 720p60 with about 30% CPU load on a Core 2 Duo laptop (not sure what the clock speed was for that test, but nothing unusually high).

[Comatose]

"Comparing VC-1 Advanced Profile to H.264 High Profile with all the bells and whistles turned on, VC-1 only needs about half as many MIPS per pixel for playback."

ooooops - from my observations this claim is untrue - VC-1 is slow, especialy when decoder from MS is used, also seems that present implementations of encoders/decoders are far from h.264...
seems that "superiority" of the VC-1 over the h.264 is like "superiority" Vista over XP.
Especially when decoder from MS is used?

There are some 3D videos available (google stereoscopic 3d player).
Get one that's double width 1080p and try playing it with ffdshow. You drop frames.

The MS decoder is fast enough to decode in realtime.

[CruNcher]

@benwaggoner

There's no DRM utilization in WMP if the content doesn't have DRM applied. Yes, I know that guy in New Zealand wrote that whitepaper about what he imagined Vista might have been like if we did everything in the worst possible way, but every emperical prediction he made there has been refuted by those actually running Vista.

Yes his Predictions where a little heavy tough people believed this and later the bad Performance of Vista proved what he said in those users eyes, they didn't experienced that the Kernel was actually improved and that the Performance problems came mostly from the added complexity of the Layer above the Kernel (all the new services,more indexing, shdaow copying, and more processes i/o access even now faster and more efficient now double the operations of XP) that eat up more of the Performance that have been gained by the improvements themselves (when staying with the current hardware).
Bad Balancing decisions i would say (of course non finished optimized drivers that doesn't taking every of the changes of the new Driver Model into account also ads up to this and bugs like in beginning in the Multimedia Scheduling then don't do good either, especially for a system that calls itself a Multimedia Experience) and so the PR Chaos was Perfect and now Vista is seen as a disaster by a big amount of users and for many experienced like a back step from XP.

[Quark.Fusion]

There's no DRM utilization in WMP if the content doesn't have DRM applied.
WMP always run wmv playback trough protected path. Maybe there is no decryption, but that prohibit using of ffdshow.

[benwaggoner]

WMP always run wmv playback trough protected path. Maybe there is no decryption, but that prohibit using of ffdshow.
I'm not sure what your ffdshow issue is, but WMP doesn't play back non-DRM'ed files via the protected video path. You can take screen shots and everything.

[Golgot13]

There've been some past discussions about this, so I wanted to give everyone a heads up that we announced forthcoming support for H.264 and MPEG-4 in Silverlight.

Details here:
http://on10.net/blogs/benwagg/H264-and-AAC-support-coming-in-Silverlight/


Hi all,

MS understand the market :eek: :eek:
Silverlight is nice, but MP4 format support BIFS (DVD like on MP4 container).
And I saw some MP4 ISMA with BIFS (use by some "cable network").

Ben, there is a specific muxer or SL (SilverLight) decode MP4 and MP4 ISMA.
SL support MP4 container or MPEG2 TS ?


Last Ben, do you think SVG is a good format ? ;)

[benwaggoner]

MS understand the market :eek: :eek:
Silverlight is nice, but MP4 format support BIFS (DVD like on MP4 container).
And I saw some MP4 ISMA with BIFS (use by some "cable network").
Really? I thought BIFS was DOA, once Envivo shifted their business plan to codecs.

I don't think I've seen a single BIFS player or file in the wild for at least five years.

Ben, there is a specific muxer or SL (SilverLight) decode MP4 and MP4 ISMA. SL support MP4 container or MPEG2 TS ?
We'll have a native MPEG-4 parser. We also have the MediaStreamSource feature of Silverlight whcih allows parsers to be built in managed code. So it's something an enterprising devleoper could write for Silverlight without us having to explicitly build it in.

Last Ben, do you think SVG is a good format ? ;)
Good for what? We didn't use it instead of XAML because SVG wasn't designed to be able to build interactive UX. Silverlight can access the browser DOM, so we could control SVG playback in a web page on a browser that supports it.

[pandy]

Talking about using WMP/DirectShow/Media Foundation for playback? Yes, Vista will be better if you're using DXVA, but as long as you have WMP 11 installed on XP, it's about the same for software decode. Vista just has a much higher probabilty of using DXVA.


DXVA means that we use hardware for decoding so due market regulations (BD, HD DVD etc) support for h.264 and vc-1 is mandatory - difference in speed are irrelevant (from hardware point of view).


There's no DRM utilization in WMP if the content doesn't have DRM applied. Yes, I know that guy in New Zealand wrote that whitepaper about what he imagined Vista might have been like if we did everything in the worst possible way, but every emperical prediction he made there has been refuted by those actually running Vista :).


security by obfuscation...For true... Vista is slower than XP


I don't understand your question.

Also Vista runs reasonably well on "older" machines. You won't get Aero Glass on a pre-DX9 GPU, but that's about GPU, not CPU performance.


No, Vista not runs "reasonably well" on "older" hardware.
Maybe it runs with similar performance in office tasks but definitely not in multimedia, especially when we don't use embedded solutions from MS.


My point is that some of the more complex features of H.264, particularly a loop filter that touches more pixels with more taps, and CABAC, mean that the worst-case performance for a H.264 High Profile encode is about 2x the worst-case performance for an Advanced Profile VC-1 encode at the same bitrate. It won't matter in lots of cases.



Hmmm Ben... CABAC can be disabled?, more taps=more quality?


FWIW, the Silverlight 2 VC-1 decoder is currently decoding 720p60 with about 30% CPU load on a Core 2 Duo laptop (not sure what the clock speed was for that test, but nothing unusually high).

From my point of view Core 2 Duo is definitively not a low end machine...

[CruNcher]

@pandy
read what i wrote the slowdown of Vista doesn't come from the Core it comes from the massive amount of stuff that runs above it.
If i refer to Vista or XP i refer to the Core not the gadgetry that comes with it :P
Vista should be a lot more responsive a little more secure and faster if you bring it down to XP complexity (on the same old hardware).
The main Features for me in Vista are only these.

+ More Efficient I/O and Virtual Memory management (dummy pages, more efficient page file use lifted restrictions dynamic decissioning)
+ ASLR + No execute
+ DXVA 2.0
+ EVR
+ GPU Management
+ DirectX 10

And all of this should add up to the Desktop Multimedia Experience even on older Machines (except ASLR + No execute) that it currently doesn't is a Design Flaw imho in the complexity of it for sure Windows 7 will do better here.

[hurry]

Will Silverlight support x264 MP4 video playback with HE-AAC v2 parametric stereo audio with this new addition? Thanks in advance.

[Sharktooth]

hurry: read again what ben said...
forum rule number 1: Read up before asking.

[hurry]

I have read Ben's blog article and all posts in this thread but I can't find explicit mention of x264 H.264 support in Silverlight. Maybe H.264 support means x264 support also since x264 is a H.264 codec, but still I wanted to confirm it more clearly if H.264 Silverlight support includes x264 H.264 support also and if AAC audio support includes Nero HE-AAC and HE-AAC V2 parametric stereo audio support also, since H.264 has many other codecs also like QuickTime, MainConcept, Nero Digital, Elecard etc. Thanks.