Intel QuickSync Decoder - HW accelerated FFDShow decoder with video processing

[egur]

Originally Posted by nevcairiel

@timestamps in general:

I'm sure the MSDK offers an ability to handle timestamps by itself. This will probably be fine with all PTS timestamps. At least thats the case with NVIDIAs API and DXVA2.
Just for DTS timestamps, you need to manually map the incoming times to the outgoing frames. Since the number of frames coming in and going out is usually the same, that shouldn't be much of a problem. I have that implemented in LAV Video using a FIFO buffer for the CUVID decoder (because i don't know its exact processing delay), and a fixed circular buffer in the avcodec decoder (because there i know its exact decoding delay).

First, I really appreciate your help. Thanks.
Is there a 1:1 relation between samples coming in and frames coming out?
e.g. is every media sample a single frame, or a frame can be divided into an arbitrary number of samples. This will make mapping time stamps impossible.

[nevcairiel]

In theory, both is possible, to some extend.

In reality, VC1 and H264 are usually always one sample=one frame, MPEG2 might be split over multiple data samples.

[egur]

New and improved version. Zip files contains installer documentation, please read.

Download version 0.14 alpha:
32 bit http://www.multiupload.com/LPX2JXB06S
64 bit http://www.multiupload.com/8H7ZPV5XKC
Source code http://www.multiupload.com/AXET69LL3P

Revision highlights:
v1.14:
* Created ffdshow installer. Installer will default to enabling the Intel QuickSync decoder on new installations.
* More speed optimizations. CPU is at its lowest frequency during playback with very low utilization. 2-3% on desktop and 5-6% on mobile. Mobile lowest frequency is half of desktop (800/1600).
* Fixed handling of interlaced content which is encoded as progressive frames.
* More robust and faster codec initialization

[Atak_Snajpera]

Are you going to merge your code with official builds? I'm asking because there is zero movement at the moment in ffdshow Only libav is currently being updated. That's all.

[egur]

Originally Posted by Atak_Snajpera

Are you going to merge your code with official builds? I'm asking because there is zero movement at the moment in ffdshow Only libav is currently being updated. That's all.

That's the idea. I wanted to get a minimum feature set working and I'm very close to it. clsid was very skeptic I could pull this off because of the memory copying involved, but this part is now solved
I'll contact clsid again and push things forward.

BTW, did you try my build? It should improve Avisynth performance when ffdshow is used as a decoder.

[millercentral]

I know this effort has been focused around ffdshow, but does anyone know of any similar efforts to enhance the ffmpeg encoding library for QuickSync? I would love to see this for transcoding improvements...

[Blight]

millercentral:
Eric said he'll wrap this up in a nice DLL.
From there, ffmpeg coders should be able to integrate with little effort.

[clsid]

Hi Eric,
I haven't tested your build yet, but here are a few questions:
- How are multiple instances of the decoder handled? I assume the HW can only handle a certain amount of streams at once. Will ffdshow simply fail graph connection for instance N+1, will it fall back to software mode (libavcodec), or will it crash and burn?
- This only work on Sandy Bridge, and future similar CPUs?
- Does it work if a mobo doesn't use the integrated GPU? Or if disabled in BIOS? Or if no monitor is attached?

I can give you SVN access if you want. Then you can commit it when deemed stable and also perform updates and fixes.

[ajp_anton]

How can I make the QS decoder ignore unsupported streams (10bit, 4:4:4 chroma, lossless etc) and let for example LAV take care of it?

[egur]

Originally Posted by clsid

- How are multiple instances of the decoder handled? I assume the HW can only handle a certain amount of streams at once. Will ffdshow simply fail graph connection for instance N+1, will it fall back to software mode (libavcodec), or will it crash and burn?

There should not be a (practical) limit. I can modify ffdshow to revert to libavcodec if initialization fails. Most likely that the platform will run out RAM before this happens.

Originally Posted by clsid

- This only work on Sandy Bridge, and future similar CPUs?

Correct, that’s guarantied. It might work on small core processors in the future (Atom), no promises.
The user has the option to use this decoder or not via the standard ffdshow config. Just like other decoders (e.g. wmv9).

Originally Posted by clsid

- Does it work if a mobo doesn't use the integrated GPU? Or if disabled in BIOS? Or if no monitor is attached?

BIOS must enable the IPG as the driver will not load. IGP driver must be loaded and enabled. This is not a problem. My own system has an AMD 6950 Radeon on an h67 chipset MB.
DirectX will not enumerate the IGP if a screen is not connected to it (it enumerate only the displays).
Current version needs to have either the IGP connected to a screen or have switchable graphics or Lucid Virtu is installed and configured to bind the media players to the IGP.
A VGA dummy can also be used.
I’m in the process of enabling the HW acceleration without the above hacks.

Originally Posted by clsid

I can give you SVN access if you want. Then you can commit it when deemed stable and also perform updates and fixes.

That would be great! Please send me instructions by mail if there’s any kind of procedures I’m required to follow.

[egur]

Originally Posted by ajp_anton

How can I make the QS decoder ignore unsupported streams (10bit, 4:4:4 chroma, lossless etc) and let for example LAV take care of it?

The QS decoder will fail to initialize on such streams. The idea is to have ffdshow fallback to libavcodec and that fails to notify the graph/player on connection failure. The player can then load another filter (e.g. LAV). That's the most elegant way to do it as far as I can see.
Future HW might (or not) support these stream types.
It would help if you pointed me to such streams for testing purposes, or to a guide on how to transcode to these formats.

[gvaley]

Reading the news in AnandTech really raised my hopes but they were to quickly plunge after I read egur's full post.

Not to say this initiative isn't commendable, but what I (and the rest of the world) would like to see is a true open source port of the SDK available to all OSes. This would enable the ffdshow, mplayer, VLC, you name it guys to build both encoding and decoding code into their projects and really put QuickSync to good use.

[jakmal]

Originally Posted by egur

The QS decoder will fail to initialize on such streams. The idea is to have ffdshow fallback to libavcodec and that fails to notify the graph/player on connection failure. The player can then load another filter (e.g. LAV). That's the most elegant way to do it as far as I can see.
Future HW might (or not) support these stream types.
It would help if you pointed me to such streams for testing purposes, or to a guide on how to transcode to these formats.

Eric,

You can find a 10b H264 file at this location:

http://dl.dropbox.com/u/15890479/480p_H264.mkv

Regards
Ganesh

[egur]

Originally Posted by gvaley

Reading the news in AnandTech really raised my hopes but they were to quickly plunge after I read egur's full post.

Not to say this initiative isn't commendable, but what I (and the rest of the world) would like to see is a true open source port of the SDK available to all OSes. This would enable the ffdshow, mplayer, VLC, you name it guys to build both encoding and decoding code into their projects and really put QuickSync to good use.

Agree, but before an open source SDK, you'll need driver support. BTW I'm not an expert on the driver, most of my knowledge is public knowledge.
* Linux has a very basic driver. I'm not aware of a video acceleration API (like DXVA).
* Mac OS X driver is developed by Apple not Intel (with Intel support of course and probably full disclosure of Windows driver code). Apple controls the driver API and functionality, because they own the OS (like Microsoft owns D3D & DXVA APIs).

Making a cross platform video acceleration SDK without a proper VA API is probably possible but would require bypassing the DXVA API and communicating in an alternative way with the driver. This usually requires a consent from the OS vendor (Microsoft & Apple) - not an easy task.
The best solution would be to have an open standard for video acceleration that would be accepted by all OS vendors (like OpenGL, OpenCL).

I believe the media SDK will be ported to other OSes in the future, so my code will be a good start point for understanding how to utilize it. My only "Windows" specific code is dealing with the memory GPU allocations, but this can be abstracted by the SDK.

Having an open source SDK is not really a must. The Intel Media SDK is free to use and you also have a large company backing it up. The SDK abstracts the HW enough so that the code is future proof.

[nevcairiel]

Funny that you would say "VA API"

Linux has a API which is supported by Intel, its called simply "VA API". It was designed by Intel initially, but AFAIK you can get adapters to run hwaccel through VAAPI on other GPUs as well these days.

[egur]

Originally Posted by jakmal

Eric,

You can find a 10b H264 file at this location:

http://dl.dropbox.com/u/15890479/480p_H264.mkv

Regards
Ganesh

Thanks

[nm]

Originally Posted by egur

Agree, but before an open source SDK, you'll need driver support. BTW I'm not an expert on the driver, most of my knowledge is public knowledge.
* Linux has a very basic driver. I'm not aware of a video acceleration API (like DXVA).

The current driver is much more than basic. As nevcairiel pointed out, Intel supports hardware-accelerated decoding on Linux through VAAPI: http://intellinuxgraphics.org/h264.html

There's also initial encoding support through libva, but I haven't heard anyone outside Intel having tested it yet.

[clsid]

Originally Posted by egur

The QS decoder will fail to initialize on such streams. The idea is to have ffdshow fallback to libavcodec and that fails to notify the graph/player on connection failure. The player can then load another filter (e.g. LAV). That's the most elegant way to do it as far as I can see.
Future HW might (or not) support these stream types.
It would help if you pointed me to such streams for testing purposes, or to a guide on how to transcode to these formats.

Have a look at ffglobals.cpp. There is SPS parsing code there that is currently used for checking if a stream is supported. Maybe you can also fix the code to no longer depend on the libavcodec golomb stuff.

Send me a PM with your SourceForge username and I can give you commit rights.

[egur]

Originally Posted by nm

The current driver is much more than basic. As nevcairiel pointed out, Intel supports hardware-accelerated decoding on Linux through VAAPI: http://intellinuxgraphics.org/h264.html

There's also initial encoding support through libva, but I haven't heard anyone outside Intel having tested it yet.

Good to know. I'm currently trying to make a good Windows based decoder that's easy to integrate. Cross platform code will be available once Media SDK supports it.

[CruNcher]

This issue might be not easy to find egur, currently ffdshow-quicksync has a tendency to silently selfdestruct after a lot of streams continuously loaded in MPC-HC it dies and MPC-HC falls @ the next stream then back to another Decoder ffdshow-quicksync wont load anymore after until MPC-HC is restarted it isn't a specific stream it seems already changing streams fast without unloading them can trigger this, like in MPC-HC drag & drop while another stream is still loaded.
Another problem though here is MPC-HC cant cleanly unload dshow filter when a file is closed the filter is still acquired not sure though if that has something todo with ffdshow-quicksync currently braking after a while. Though maybe it's also a memory leak i didn't looked into that yet.


http://www.mediafire.com/download.php?ri4sdpjlafxxrry <- hangs (@ the cat) when the lion scene should follow (with both lav splitter and mpc-hc).

Btw the 4:2:2 Mpeg-2 Fallback already works perfectly in combination with Lav Video
Only High10, High422 and X264 losless fail