VideoInputSource - grab video frame from video capture card or webcam in real-time

[fieliapm]

It is developed for my work in the beginning. But I think somebody might need this to processing real-time captured video, so I post here and wish I could help anyone who use AviSynth to do some prototyping easily.

One day, I am asked to import and process real-time captured video and merge with video from files in AviSynth for making some prototype.
In the beginning, I grabbed video frame from DirectShowSource() with grf file as input. However, I found that DirectShowSource() won't duplicate or skip frame when some video capture device (ex. webcam) cannot produce next frame ontime. This phenomenon will cause AviSynth's frame feeding slow down while playing AviSynth script.
For this reason, I made a AviSynth source filter which capture video from video capture device. It is based on a simple video capture library named VideoInput.



binary:
VideoInputSource.x86.dll built for x86 32bit AviSynth 2.5.7 & VapourSynth API version 3.6
VideoInputSource.x64.dll built for x86 64bit VapourSynth API version 3.6

source:
VideoInputSource on github



The usage of this source filter is as below:

VideoInputSource(device_id,connection_type,width,height,"fps_numerator","fps_denominator","num_frames","frame_skip")



# device_id: the n-th number of video capture device in your computer

# connection_type: can be these string: "Composite","S_Video","Tuner","USB".
#     I leave this parameter because videoInput library offer this option.

# width, height: the width and height of frames captured from video capture device.

# fps_numerator, fps_denominator: FPS numerator and denominator.
#     Default is 30/1 (30.0fps)

# num_frames: How many frame will be captured from video capture device.
#     This will tell AviSynth the length of this video source.
#     Default it will be the value which makes video source length become 24 hours.

# frame_skip: enable/disable frame skip while next new frame from video capture device is not ready.
#     Default is true.

For example:

VideoInputSource(0,"Composite",1920,1080,24,1)

Will grab video from video capture device #0, 1920x1080, 24.0 fps, 24 hours long.



videoInput always output BGR24 packed pixels. For efficiency, current version of VideoInputSource supports AviSynth RGB24 color format only, and cannot be imported into VapourSynth script yet. Since I sometimes use VapourSynth for video processing, I might focus on compatibility of VapourSynth in the future.

By the way, It can work with MP_Pipeline very well since I often test this plugin in separate process generated by MP_Pipeline. However this plugin exclusively accesses to video capture device, so don't create video source from the same video capture device at the same time.

[althor1138]

Thanks for doing this! It works fantastic for me using a diamond theater750-pcie card. I am running avisynth 2.5.8 atm. Would it be possible to enable support for yuy2 capture?

[StainlessS]

Many thanks kind sir.

[lansing]

I tried to capture with my Logitech C525 webcam, I set compression to use lagarith in virtualdub. But the video came out as it were in super slow-mo. What am I missing?

VideoInputSource(0,"USB",1280,720,30,1)

[Chromix]

I had the same problem when using this source for encoding instead of viewing. What happens is that the clip says "I'm 30 FPS" like you specified in VideoInputSource. That's fine for players, but encoding tools of course try to be faster than the frame rate of the clip. So they just keep requesting frames as fast as possible, and the source just says "well, the frame hasn't changed, but here you go anyway". Thus the encoded clip is slow-mo due to tons of duplicated frames.

I've added a parameter to VideoInputSource: frame_wait (false by default). If you set it to true then VideoInputSource will simply block until there is a new frame available. It's a bit of a hack but works fine for me. The modified source code and new DLL is attached in the zip.

First you need to figure out the frame rate your capture device provides (careful, webcams can change that dynamically under low-light conditions!). Just encode a clip with null output or fast compression, and check for the encoding FPS. VirtualDub e.g. displayed 8.33 FPS for my cam during low-light. So the VideoInputSource setting should be 25/3 in this case. This helps to get the audio sync (almost) right. Sometimes it needs to be tweaked a bit afterwards.

[fieliapm]

The problem which lansing and Chromix met might be due to getPixels() is non-blocking at their environment or usage.
However when I use VirtualDub to encode, with my capture card and Microsoft LifeCam Cinema under Windows 7, GetFrame() always blocks at getPixels() while there is no new frame coming.
To ensure new frame coming by asking function isFrameNew() (as Chromix did in modified version), I refined my original source code and put it here:

binary:
VideoInputSource.dll

source:
VideoInputSource on github

Just make frame_skip = false, it will acts like frame_wait in Chromix's mod.

I also use BitBlt() from AviSynth script env instance plus _aligned_malloc() to accelerate frame data copy, because AviSynth offers BitBlt() which support SSE instruction set when it availabie, it might be more efficient while running on various CPUs.

It is based on old stable release videoInput 0.1995 as previous revision.
However, anyone who want to use newer version (ex. 2014) of videoInput from author Theodore Watson's github (videoInput on github), feel free to switch source code belong to videoInput 0.1995 to newer version.



BTW, let me re-introduce this source plugin again.

The usage of this source filter is as below:

VideoInputSource(device_id,connection_type,width,height,"fps_numerator","fps_denominator","num_frames","frame_skip")

# device_id: the n-th number of video capture device in your computer

# connection_type: can be these string: "Composite","S_Video","Tuner","USB".
#     I leave this parameter because videoInput library offer this option.

# width, height: the width and height of frames captured from video capture device.

# fps_numerator, fps_denominator: FPS numerator and denominator.
#     Default is 30/1 (30.0fps)

# num_frames: How many frame will be captured from video capture device.
#     This will tell AviSynth the length of this video source.
#     Default it will be the value which makes video source length become 24 hours.

# frame_skip: enable/disable frame skip while next new frame from video capture device is not ready.
#     Default is true.

For example:

VideoInputSource(0,"Composite",1920,1080,24,1)

Will grab video from video capture device #0, 1920x1080, 24.0 fps, 24 hours long.

videoInput always output BGR24 packed pixels. For efficiency, current version of VideoInputSource supports AviSynth RGB24 color format only, and cannot be imported into VapourSynth script yet. Since I sometimes use VapourSynth for video processing, I might focus on compatibility of VapourSynth in the future.

By the way, It can work with MP_Pipeline very well since I often test this plugin in separate process generated by MP_Pipeline. However this plugin exclusively accesses to video capture device, so don't create video source from the same video capture device at the same time.

[StainlessS]

Thank you, I've just got it up working on remote machine, and serving to local machine via TcpServer(), Tcpsource(), pair, works fine.

EDIT: As not posted in Avisynth Usage forum, might want to consider posting link to developer forum thread in Avisynth Usage, New Plugins and Filters thread (let people know that its available, not every user visits the Devs forum)..

EDIT: Something on TcpSource and TcpServer here:- https://forum.doom9.org/showthread.php?p=1802408

[fieliapm]

Thanks for your suggestion.
I will announce this plugin for end-user since it seems to be stable now.

[Milardo]

Originally Posted by fieliapm

Thanks for your suggestion.
I will announce this plugin for end-user since it seems to be stable now.

Anyway to get audio from a capture card?

[StainlessS]

Thought I'de post link to the thread in Usage forum(only opening post so far):- https://forum.doom9.org/showthread.php?t=174514

Dont miss Milardo's post above, a little earlier today (I have no idea how to answer Milardo query).

EDIT: fieliapm not on-line since Oct 2018.

[fieliapm]

This summer I promoted doing video processing using AviSynth and VapourSynth and introduced how to write their plugins to my local open source community members.
So I refine VideoInputSource source code, upgrade its dependency videoInput (current latest stable) and recompile it using VS2019 to support VapourSynth x86/x64.

Everyone might try it out.

[StainlessS]

I managed to download a zip of the updated filter and source code but could not figure out how I'de done it.
(there are No Releases on linked github), This is easiest way I figured out how to download as a complete zip.
(I dont use GitHub)

Go here,
https://github.com/fieliapm/himawari_avs_plugin.git
Then click on the [<> Code ] box and select Download Zip

Thanx, I give it a try when I get some time.

Here is a D9 link to a project from CodeProject.com,

CaptureManager SDK - Capturing, Recording and Streaming Video and Audio from Web-Cams
https://forum.doom9.org/showthread.p...59#post1986959

Perhaps of interest to you.

again thanks.

EDIT: list of demo program names using the SDK @ CodeProject link

CaptureManager SDK Demo Programs

WPFHardwareButtonTrigger
WPFWebViewerEVRLog4net
WPFVirtualCamera
WPFStreamingAudioRenderer
AudioCaptureProcessor
WPFGIFtoVideo
WPFMixer
WPFStreamer (live video and audio streaming on Facebook and YouTube sites)
WPFRTSPClient
WPFRtspServer
WPFDirectShowPlayer
DShowPlayer
WPFWebViewerEVRHandler
WPFDeviceInfoViewer
InterProcessRenderer
WPFAreaScreenRecorder
WPFHotRecording
WPFPauseResumeRecording
WPFViewerTrigger
UnityVideoAndAudioRecorder
UnityScreenRecorder
WPFWindowScreenRecorder
WPFScreenRecorder
WPFScreenViewer
WPFMultiSourceRecorder
UnityWebCamViewer
WPFMediaFoundationPlayer
NativeMediaFoundationPlayer
WPFVideoAndAudioRecorder
EVRVieweingAndRecording
WPFIPCameraMJPEGMultiSourceViewer
WPFMultiSourceViewer
WPFViewerEVRDisplay
WPFIPCameraMJPEGViewer
WPFImageViewer
TextInjectorDemo
WaterMarkInjectorDemo
WindowsFormsDemo
CaptureManagerSDKJavaxDemo
CaptureManagerToJavaProxy
WPFWebCamSerialShots
WPFWebCamShot
CaptureManagerSDKPythonDemo
QtMinGWDemo
WPFRecorder
WPFWebViewerEVR
WPFWebViewerCallback
WPFWebViewerCall
WPFSourceInfoViewer
CaptureManagerToCSharpProxy
EVRWebCapViewerViaCOMServer
OpenGLWebCamViewerViaCOMServer

[fieliapm]

Actually this plugin is currently under some experiment, something like reasonable dependencies, runtime settings, dependencies management are too ugly.
You said the key point, I should apply formal release after compiling environment and dependencies are well handled.
However I didn't figure out suitable environment and dependencies yes, sorry for waiting me patiently.

Also I will read the thread of CaptureManager SDK, it seems to be interesting.

BTW, with the current situation, what version of AviSynth and VapourSynth should I bind?
If I want to keep the most compatibility including 32bit/64bit, and use the same version of C runtime library.

And thanks for trying my crap plugin.

[StainlessS]

Originally Posted by fieliapm

Actually this plugin is currently under some experiment, something like reasonable dependencies, runtime settings, dependencies management are too ugly.
You said the key point, I should apply formal release after compiling environment and dependencies are well handled.
However I didn't figure out suitable environment and dependencies yes, sorry for waiting me patiently.
...
BTW, with the current situation, what version of AviSynth and VapourSynth should I bind?
If I want to keep the most compatibility including 32bit/64bit, and use the same version of C runtime library.
...

People usually post separate x86 and x64 versions of a plugin. [I know nothing of VapourSynth].
I have not done much [if any] coding since Covid, no idea how your environment and dependencies may best be solved.

Avisynth binding... I guess that you pick Avisynth+ ver$ from this lot:- https://forum.doom9.org/showthread.php?t=181351
[Maybe this one[Avisynth+ 3.7.3 (20230715)]:- https://github.com/AviSynth/AviSynth...ses/tag/v3.7.3 ]
I would maybe not bother with avisynth 2.6/2.58 standard, just the Avsynth+ version.

I just got a camera thingy today [Wifi 1080p Indoor CCTV Baby/Pet pan/tilt monitor, Amazon delivered about an hour ago]:- https://www.amazon.co.uk/dp/B0CQHY41NY
Comes with Android/iPad view/control app.
[Thread on PC CTRL/View on TP-Link forum:- https://community.tp-link.com/en/sma...m/topic/205220 ]
I guess that when I find time, I'll have a little bit play with your plug, and other things camera related.
Thanx again, have a guddun

EDIT:
I think somewhere in thread [Maybe in Usage forum] you mentioned VirtualDub, most people today use VirtualDub2
VirtualDub2:- https://forum.doom9.org/showthread.php?t=172021
(Although, I think someone else may have compiled an updated version in that thread, Shekh (author) aint been around for a while)
EDIT: Although, I'm still using ver$ 44282 so maybe not updated. [EDIT: 44282 latest on VideoHelp.com]

[StainlessS]

Originally Posted by StainlessS

I think somewhere in thread [Maybe in Usage forum] you mentioned VirtualDub, most people today use VirtualDub2

VirtualDub2 update :- https://forum.doom9.org/showthread.p...78#post2006278
but see LigH post #1390 (in that thread) for caveat.

[Milardo]

Originally Posted by fieliapm

It is developed for my work in the beginning. But I think somebody might need this to processing real-time captured video, so I post here and wish I could help anyone who use AviSynth to do some prototyping easily.

One day, I am asked to import and process real-time captured video and merge with video from files in AviSynth for making some prototype.
In the beginning, I grabbed video frame from DirectShowSource() with grf file as input. However, I found that DirectShowSource() won't duplicate or skip frame when some video capture device (ex. webcam) cannot produce next frame ontime. This phenomenon will cause AviSynth's frame feeding slow down while playing AviSynth script.
For this reason, I made a AviSynth source filter which capture video from video capture device. It is based on a simple video capture library named VideoInput.



binary:
VideoInputSource.x86.dll built for x86 32bit AviSynth 2.5.7 & VapourSynth API version 3.6
VideoInputSource.x64.dll built for x86 64bit VapourSynth API version 3.6

source:
VideoInputSource on github



The usage of this source filter is as below:

VideoInputSource(device_id,connection_type,width,height,"fps_numerator","fps_denominator","num_frames","frame_skip")



# device_id: the n-th number of video capture device in your computer

# connection_type: can be these string: "Composite","S_Video","Tuner","USB".
#     I leave this parameter because videoInput library offer this option.

# width, height: the width and height of frames captured from video capture device.

# fps_numerator, fps_denominator: FPS numerator and denominator.
#     Default is 30/1 (30.0fps)

# num_frames: How many frame will be captured from video capture device.
#     This will tell AviSynth the length of this video source.
#     Default it will be the value which makes video source length become 24 hours.

# frame_skip: enable/disable frame skip while next new frame from video capture device is not ready.
#     Default is true.

For example:

VideoInputSource(0,"Composite",1920,1080,24,1)

Will grab video from video capture device #0, 1920x1080, 24.0 fps, 24 hours long.



videoInput always output BGR24 packed pixels. For efficiency, current version of VideoInputSource supports AviSynth RGB24 color format only, and cannot be imported into VapourSynth script yet. Since I sometimes use VapourSynth for video processing, I might focus on compatibility of VapourSynth in the future.

By the way, It can work with MP_Pipeline very well since I often test this plugin in separate process generated by MP_Pipeline. However this plugin exclusively accesses to video capture device, so don't create video source from the same video capture device at the same time.

Does this support audio yet?

[Mitra]

Originally Posted by Milardo

Does this support audio yet?

No, does not support audio.

[Milardo]

Originally Posted by Mitra

No, does not support audio.

ok.

I noticed some delay/input lag in the video, is there a way to correct this?

[Mitra]

Originally Posted by Milardo

ok.

I noticed some delay/input lag in the video, is there a way to correct this?

If you are using the 64-bit version, there should be no problem. (It works for me without any problem)

What camera or capture device do you use and what type of connection?

[Milardo]

Originally Posted by Mitra

If you are using the 64-bit version, there should be no problem. (It works for me without any problem)

What camera or capture device do you use and what type of connection?

Using the 64 bit version in mpc-be with EVR-CP and avisynth+

an hdmi usb capture device which is usb 3.0.

ExtremeCap UVC - BU110

I noticed some input lag/delay when trying to play video games on a console, controller input is delayed a bit.

In the avs file, I have this:

VideoInputSource(1,"USB",1280,720,24,frame_skip=true)


Anything else I need to add to the avs file?

Or something else?

For now, the audio issue is fine my capture card audio can be enabled from windows sound properties, and it is in sync with video.