Fake Overlay Mixer

[yooyo]

Hi,

Rescently, NVidia disable full screen video option in drivers, so with new board, user cant send overlay to second screen in full screen.
Before, when app open overlay surface it is fullscreen on TV out or second screen. Overlay Mixer Direct Show filter was inserted between video decoder and video renderer and "intercept" decoded stream and render it on overlay surface.

Today seems that overlay is obsolete (not usefull in composite desktop).

I would like to develop Fake Overlay Mixer, which actually create fullscreen window on second display and render video on it. I have some experience in Direct Show development, mostly video renderers, to combine video and 3d graphics.

Im using OpenGL to render video stream.

Video Decoder ---> FakeOverlayMixer ---> Video Renderer
FOM create fullscreen window and opengl context, use async PBO codepath to upload frames without CPU stall.

So far now, I tried develop FakeOverlayMixer based on CTransforormInPlace filter or just by CTransform filter but Im not happy with result.
In CTransInPlace way, my filter automatically inserts itself in graph, but when I try to copy frame to video card buffer it is slow and it drop frames.
In case of CTransform filter, my filter cannot connect to renderer.. Input pin gets connected, but output pin cant be connected!?

So far.. I have several questions:
1. How to allow only one instance of filter in graph?
2. Which base class should I use? CTransform or CTransformInPlace? Which functions to override?
3. Filter should support rgb24, rgb32, yuy2, yuv2. I can make shaders to convert yuv to rgb. How to negotiate "supported pixelformats"?

[roozhou]

IMO you can take a look at mplayer's opengl video output driver. It uses fragment program to do conversion from YV12 to RGB as well as high quality scaling. It renders as fast as ddraw overlay.

[yooyo]

This is not problem.. Im very good with OpenGL. My problem is when graphbuilder inserts my filter in graph in between decoder and VMR decoder outputs frame to video memory surface (probably because VMR allocator wants that). When my filter try to copy incoming frame to opengl texture it is slow process because it have to copy buffer from video memory to sysmem and then back to video mem (in opengl texture). Anyway it is just one memcpy, but source and destination is in video memory and this transfer is SLOW.
This mean I have to write my own allocator, which also mean I have to write my own input and output pins.
Simple task gets complicated.. so.. Im wondering should I start from CTransform filter or from CTransInPlace with custom pins and custom allocators.
I need advice from more expirienced direct show filter developers.

Another question is I want to support all kind of pixelformats and do conversion to RGB using opengl shaders. I cant make my filter inserts automatically to graph. If I set MERIT to preffered+2 I got multiple instances in graph connected between them self. Is there any code snippet to ensure just one instance in graph?