Is it possible to live capture SDI/HDMI YUV4:2:2 /RGB with Avisynth?

[Mitra]

I tried VideoInputSource and DirectShowSource to capture a live SDI YUV4:2:2 video coming from a capture card, but they don't work.

I have seen many old threads and tried this (https://forum.doom9.org/showthread.php?t=174514)and this (https://forum.videohelp.com/threads/333466-Conan-O-Brien-mouth-cutout-interviews-help#post2068898).

Is it possible to capture SDI/HDMI YUV4:2:2 /RGB with Avisynth?

[DTL]

AVS is about any processing. If you need simply write file - you simply create and run capture graph using graphedit or graphstudionext.

If you try to make AVS as a API link between DirectShow and some sink application - you create capture graph without final render and save as .grf and try load it into DirectShowSource().

If you need to process with AVS inside DirectShow - you can insert AVS script into ffdshow (but it may support only some old AVS versions).

[Mitra]

AVS is about any processing. If you need simply write file - you simply create and run capture graph using graphedit or graphstudionext.

If you try to make AVS as a API link between DirectShow and some sink application - you create capture graph without final render and save as .grf and try load it into DirectShowSource().

If you need to process with AVS inside DirectShow - you can insert AVS script into ffdshow (but it may support only some old AVS versions).

Thank you for your reply, I need to process with AVS in real-time, so am using AVS by ffmpeg (I can't use ffdshow), it seems the only choice for me is DirectShowSource filter, but it gets this error:
"DirectShowSource: GRF file does not have a compatible open video pin.
Graph must have 1 output pin that will bid RGB24, RGB32, ARGB, YUY2,
YV12, I420, NV12, YV16, YV24, Y41P, Y411 or AYUV
(d.avs, line 2)
d.avs: Unknown error occurred"


my script :
#last=VideoInputSource(2,"HDMI",720,576,25,1)
last=DirectShowSource("cap6.grf", audio=false, fps=25, seek=false, framecount=100000)

[poisondeathray]

I need to process with AVS in real-time, so am using AVS by ffmpeg (I can't use ffdshow), it seems the only choice for me is DirectShowSource filter, but it gets this error:


"DirectShowSource: GRF file does not have a compatible open video pin.



Did you leave the output pin open in the grf ?

Another option would be ffmpeg rawvideo pipe to avisynth using rawsourceplus

What processing do you specifically need in avs ? Maybe alternatives in ffmpeg ?

[DTL]

What SDI capture hardware do you use ? Can you configure its output pin to something like RGB 8bit 4:4:4 ? The 4:2:2 YUV may be not one compatible with simple enough DirectShowSource(). Other solution is to put some format adapter filter after capture filter (so its output pin will be connected to DirectShowSource with compatible format). As a manual force format converter it is again ffdshow may be tested with some settings for output formats (and expect it will accept output format of your capture device driver).

If you need pro-level 10bit RGB/YUV it may be even more complex task. Try to start with 8bit samples first.

"this is the only filter in my graph :"

It may be inserted image - but it is not displayed. Try different image hosting ?

The DirectShow filters configuration (its properties and pins) with .grf files may be not possible - so some solution may be first start graphedit and configure all filters in the graph manually (set YUV or RGB mode of the capture device and 4:4:4 format for output pin) and expect it will keep configuration when AVS will load grf file and construct separate graph to work. So workflow -
1. boot windows
2. open graphedit and construct some sample capture graph manually and confugure its capture filter using GUI controls for filter properties and capture/output pin.
3. close graphedit
4. load AVS with grf file

"Graph must have 1 output pin that will bid RGB24, RGB32, ARGB, YUY2,
YV12, I420, NV12, YV16, YV24, Y41P, Y411 or AYUV"

It looks DirectShowSource is old enough from the days of 8bit AVS and not updated for 10bit formats. So the only soultion with it is use some 8bit input formats. If your SDI capture device can not output 8bit - so the only way is search for 10 to 8 bit conversion DirectShow filter. Start from ffdshow in RAW processing mode and some other possible. May be even some Microsoft colour conversion may work and even in auto-insert mode - try to run 'render pin' command at the output pin of the capture device and look which filters will windows find to feed to display render device. If you luck - you can get some compatible with DirectShowSource format before display renderer and simply delete it and save .grf for AVS.

You need to check media type and media subtype at the output pin of the graph - select pin and see 'properties' in the context menu and switch tabs to some table with format.

Also depending on the driver designer from capture device vendor you may not have capture device output format control via DirechShow standard properties interface at all - it may be controlled by some application from windows control panel. And pass control params via registry or other way.

From the list of supported - 4:2:2 formats are YUY2 (typical for DirectShow interleaved and SDI-interleaved words order- MEDIASUBTYPE_YUY2 at https://gix.github.io/media-types/ ) and YV16 (planar - may be rare for DirectShow). And it is 8bit 4:2:2 - not real 10bit pro-SDI. If possible capture card can truncate 10bit SDI to these 8bit formats in hardware.
RGB are RGB24, RGB32, ARGB - also 8bit RGB.

If you see something like MEDIASUBTYPE_Y210 at the output pin format - it is natural full SDI 10bit 4:2:2 interleaved - but not compatible with DirecthShowSource and you need either reconfigure capture device driver or found and insert format conversion filter to the graph.

[Mitra]

Did you leave the output pin open in the grf ?

Another option would be ffmpeg rawvideo pipe to avisynth using rawsourceplus

What processing do you specifically need in avs ? Maybe alternatives in ffmpeg ?

It's unclear to me and I don't know should i leave the output pin open or put a renderer or a file writer in the graph or not, but I tried all of them without success.

I need to deinterlace by QTGMC and apply some AVS video denoiser (those there are not in ffmpeg) to capture SDI interlaced 576i50 as 576p50.

[Mitra]

my source can be as HDMI RGB or SDI YUV4:2:2 10b

http://media4dco.com/downloads/sderror.png

[Mitra]

What SDI capture hardware do you use ? Can you configure its output pin to something like RGB 8bit 4:4:4 ? The 4:2:2 YUV may be not one compatible with simple enough DirectShowSource(). Other solution is to put some format adapter filter after capture filter (so its output pin will be connected to DirectShowSource with compatible format). As a manual force format converter it is again ffdshow may be tested with some settings for output formats (and expect it will accept output format of your capture device driver).

If you need pro-level 10bit RGB/YUV it may be even more complex task. Try to start with 8bit samples first.

"this is the only filter in my graph :"

It may be inserted image - but it is not displayed. Try different image hosting ?

The DirectShow filters configuration (its properties and pins) with .grf files may be not possible - so some solution may be first start graphedit and configure all filters in the graph manually (set YUV or RGB mode of the capture device and 4:4:4 format for output pin) and expect it will keep configuration when AVS will load grf file and construct separate graph to work. So workflow -
1. boot windows
2. open graphedit and construct some sample capture graph manually and confugure its capture filter using GUI controls for filter properties and capture/output pin.
3. close graphedit
4. load AVS with grf file

"Graph must have 1 output pin that will bid RGB24, RGB32, ARGB, YUY2,
YV12, I420, NV12, YV16, YV24, Y41P, Y411 or AYUV"

It looks DirectShowSource is old enough from the days of 8bit AVS and not updated for 10bit formats. So the only soultion with it is use some 8bit input formats. If your SDI capture device can not output 8bit - so the only way is search for 10 to 8 bit conversion DirectShow filter. Start from ffdshow in RAW processing mode and some other possible. May be even some Microsoft colour conversion may work and even in auto-insert mode - try to run 'render pin' command at the output pin of the capture device and look which filters will windows find to feed to display render device. If you luck - you can get some compatible with DirectShowSource format before display renderer and simply delete it and save .grf for AVS.

You need to check media type and media subtype at the output pin of the graph - select pin and see 'properties' in the context menu and switch tabs to some table with format.

I tried 4:2:0 8bit but it does not work too, but with GraphStudio works.
my graph :
http://media4dco.com/downloads/ds2.png


http://media4dco.com/downloads/ds3.png

[poisondeathray]

Your screenshot shows UYVY - which is 8bit422 . It's stored differently than YUY2 (packed 8bit 422), or YV16 (planar 8bit 422) in avisynth . Notice DirectShowSource does not have a pixel_type="UYVY"

Besides, it's unlikely that you get realtime performance with QTGMC + some denoiser (especially a good tempral one) with a live stream

You need multiple frames read ahead and buffered for temporal filters such as QTMGC + a temporal denoiser . They are not spatial only filters. Likely you will get get dropped frames and other issues .

[Mitra]

@DTL, Thank you very much for your tips and explanations.

I have several years of experience in programming with C language and ffmpeg API, but I have never written a plugin for AVS. Do you think I can write a plugin to do this by fork and changing FFms2 source code?

[DTL]

May be start from inserting ffdshow in RAW conversion mode to YUY2 ?

It is good if you can change DirectShowSource AVS filter to accept UYVY too and make commit to AVS+ but it may be more work. It is not hard - you need to add more supported media subtype and some simple byte swapping function to repack UYVU interleaved into typical YV12 planar.

Also with some quality loss you can try to add color space converter DirectShow filter to unpack UYVY to RGB24 and it will be accepted by current DirectShowSource (or also ffdshow in this conversion mode only). From your graph - check output format (at XForm Out pins) of the already filters in the graph - AVI Decompressor and Color Space Converter - they may be already in some RGB so you need only cut and save graph at this state. Microsoft may be smart enough to save and restore internal DirectShow filters config with .grf file.

As I see Color Space Converter only swaps different RGB formats (?) - so may be AVI Decompressor converts UYVY to some RGB ?

http://media4dco.com/downloads/ds2.png
If AVI Decompressor have some compatible RGB media subtype at output - select and delete last 2 filters and save .grf and try to load with DirectShowSource().

[Mitra]

Your screenshot shows UYVY - which is 8bit422 . It's stored differently than YUY2 (packed 8bit 422), or YV16 (planar 8bit 422) in avisynth . Notice DirectShowSource does not have a pixel_type="UYVY"

Besides, it's unlikely that you get realtime performance with QTGMC + some denoiser (especially a good tempral one) with a live stream

You need multiple frames read ahead and buffered for temporal filters such as QTMGC + a temporal denoiser . They are not spatial only filters. Likely you will get get dropped frames and other issues .

Thank you for your point about pixel_type, So you mean it is not possible with DirectShowSource ?

I want to use QTGMC with "Fast"or "Faster" preset and I think it will work with my i7 13700k machine.

[poisondeathray]

UYVY is supported by RawSourcePlus ,so are many other pixel types including 10bit422 that you send through ffmpeg (yuv422p10le) . It also supports live "pipe" in from ffmpeg through y4m (yuv4mpegpipe)
https://github.com/Asd-g/RawSource_2.6x

RawSourcePlus("-")

(But it's still unlikely that you get problem free results)

[Mitra]

UYVY is supported by RawSourcePlus ,so are many other pixel types including 10bit422 that you send through ffmpeg (yuv422p10le) . It also supports live "pipe" in from ffmpeg through y4m (yuv4mpegpipe)
https://github.com/Asd-g/RawSource_2.6x

RawSourcePlus("-")

(But it's still unlikely that you get problem free results)

Thank you! , I will try it.

[Mitra]

May be start from inserting ffdshow in RAW conversion mode to YUY2 ?

It is good if you can change DirectShowSource AVS filter to accept UYVY too and make commit to AVS+ but it may be more work. It is not hard - you need to add more supported media subtype and some simple byte swapping function to repack UYVU interleaved into typical YV12 planar.

Also with some quality loss you can try to add color space converter DirectShow filter to unpack UYVY to RGB24 and it will be accepted by current DirectShowSource (or also ffdshow in this conversion mode only). From your graph - check output format (at XForm Out pins) of the already filters in the graph - AVI Decompressor and Color Space Converter - they may be already in some RGB so you need only cut and save graph at this state. Microsoft may be smart enough to save and restore internal DirectShow filters config with .grf file.

It's a bit strange to me, can converting UYVY 4:2:2 10b to RGB24 cause quality loss?
(I can understand that converting RGB24 to UYVY will cause a loss in quality)

[poisondeathray]

It's a bit strange to me, can converting UYVY 4:2:2 16b to RGB24 cause quality loss?



UYVY is 8bit422 . Converting to 8bit RGB technically is lossy , because it' s not reversible - there is clipping (negative RGB values produced, clipped to zero) and rounding losses even if you upsampled using nearest neighbor (to prevent chroma up/down sampling lossess)

[DTL]

"It's a bit strange to me, can converting UYVY 4:2:2 16b to RGB24 cause quality loss?"

Yes - there are many issues in this conversion starting from typical clipping of narrow to full levels mapping and continuing with some poor or better quality chroma upsampling (also not specified in the industry standards at all and always more or less lossy). As the more bad things - DirectShow may even not understand new matrix like rec.709 or rec.2020 and use rec.601 always ? Need to read Microsoft docs better but I do not see (manual) matrix controls at both displayed DirectShow filters.

So better to use YUV narrow from blackmagic output and not go into RGB full typically only possible in standard old DirectShow. So as a lossless byte swapper it is idea to use 8bit ffdshow - simply insert it manually in the graph and set output format to YUY2 only. But it is old filter and may not be stable even with such simple operation with new windows.

When you finally get YUY2 inside AVS you can convert to YV16 planar and feed to modern filters with planar processing for better performance. Old YUY2 interleaved is not very friendlt with SIMD.

[Mitra]

UYVY is 8bit422 . Converting to 8bit RGB technically is lossy , because it' s not reversible - there is clipping (negative RGB values produced, clipped to zero) and rounding losses even if you upsampled using nearest neighbor (to prevent chroma up/down sampling lossess)

Thank you for your explanation.

[Mitra]

"It's a bit strange to me, can converting UYVY 4:2:2 16b to RGB24 cause quality loss?"

Yes - there are many issues in this conversion starting from typical clipping of narrow to full levels mapping and continuing with some poor or better quality chroma upsampling (also not specified in the industry standards at all and always more or less lossy). So better to use YUV narrow from blackmagic output and not go into RGB full typically only possible in standard old DirectShow.

Ah, I understand now, Thank you for your explanation.

[Mitra]

May be start from inserting ffdshow in RAW conversion mode to YUY2 ?
........

If AVI Decompressor have some compatible RGB media subtype at output - select and delete last 2 filters and save .grf and try to load with DirectShowSource().

AVI Decompressor has RGB 32b media subtype, I deleted last 2 filters and saved .grf file but does not work.

[DTL]

"RGB 32b media subtype but does not work."

What is the media subtype displayed in the pin properties ? What is the error text in AVS now ?

As I see DirectShowSource expect to accept https://github.com/AviSynth/AviSynthPlus/blob/2b55ba40ec22652d72121fcef56b46da1fc2e427/plugins/DirectShowSource/directshow_source.cpp#L295
MEDIASUBTYPE_RGB32
MEDIASUBTYPE_ARGB32

[Mitra]

"RGB 32b media subtype but does not work."

What is the media subtype displayed in the pin properties ? What is the error text in AVS now ?

As I see DirectShowSource expect to accept https://github.com/AviSynth/AviSynthPlus/blob/2b55ba40ec22652d72121fcef56b46da1fc2e427/plugins/DirectShowSource/directshow_source.cpp#L295
MEDIASUBTYPE_RGB32
MEDIASUBTYPE_ARGB32

You are a great expert, finally with your help the ffdshow raw filter worked thanks a lot, but it's a bit frustrating because some AVS filters need YV12 and some need YUV and these conversions reduce the quality.

[DTL]

Also got an idea - the DirectShowSource final render filter may be not (completely ?) compatible with live sources because of the graph clock and seeking (as realtime source filters do not provide seeking ?). So ffdshow as some intermediate filter may use some form of hacking to workaround this (as it also can execute AVS environment inside itself and work with live sources). Or with live sources may be special settings on the graph clocking also required in both graph config and DirectShowSource config.

Microsoft also writes Live Sources are the special case in DirectShow and not very nice so new API is recommended in new Windows solutions for live sources - https://learn.microsoft.com/en-us/windows/win32/directshow/live-sources .

[DTL]

" can converting UYVY 4:2:2 16b to RGB24 cause quality loss?"

It is sort of wiki-article to AVS: Yes - digital chroma-subsampled YUV formats of the former great civilization of the past (like 197x.. 198x years - the days of the first ITU Rec.601) are some specially designed with sources of irreversible distortions.

There are actually 2 types of RGB data - immediate RGB and flat/final/display_ready RGB. The difference is with UV anti-gibbs conditioning filter at UV upscaler before dematrix of 4:4:4 YUV to 4:4:4 RGB. And Microsoft designers of AVI Decompressor with subsampled-YUV to 4:4:4 RGB may also either not know it or make flat-RGB filtering. As I remember there is no metadata signalling in the DirectShow (and also in AVS too) about spatial properties of the current RGB data domain - either in the immediate-RGB state or flat-RGB.

The hidden feature of all current industry digital moving images formats with chroma subsampling (from rec.601 SD to rec.2020 UHD) is that the UV anti-gibbs conditioning filter must be placed at the end of chain final YUV to RGB transform. And all immediate YUV->RGB->YUV transform _not_ changing frame size in samples must use immediate (UV-unfiltered) transforms. So immediate-RGB is some special form of digital data not ready for scale (display transform also scaling to infinite sample count operation). It is designed to keep chroma sharpness in a long production chain with 4:2:2 SDI as an immediate data transfer domain with subsampled YUV.

The only way to go away from this ugly system of the past is to make all production and data transfer and storage in 4:4:4 RGB domain.

DirectShow initially was designed as a simple enduser side media player environment so its 4:2:x to RGB conversion may make flat-RGB output to feed to (scaling) display renderer. And to use in AVS (in most cases of no-scaling processing) you need immediate-RGB so you can convert it to subsampled YUV again with less distortions (like YV16 or even worse YV12). At each chroma subsampled conversion of YUV<->RGB (with both H and V chroma subsampling) users must track the filtering used (current AVS typically do not pay attention to this).

The difference between immediate-RGB and flat-RGB is not greatly visible (also with typically chroma-soft content) but if many subsampled-YUV<->RGB conversions are chained using UV-filtered flat-RGB users can see how chroma sharpness degrades. Also if immediate-RGB is used with some scaling (typically sinc-kernel based) it will cause chroma ringing distortion (also to see it - the high quality chroma sharp content required).

YUV<->RGB transforms are only lossless if enough precision used (like float) and only with 4:4:4 YUV without chroma subsampling used (because subsampled Y is lost it is impossible to restore 4:4:4 data from 4:2:x YUV even with lower bandwidth chroma with low distortions).

[Mitra]

Also got an idea - the DirectShowSource final render filter may be not (completely ?) compatible with live sources because of the graph clock and seeking (as realtime source filters do not provide seeking ?). So ffdshow as some intermediate filter may use some form of hacking to workaround this (as it also can execute AVS environment inside itself and work with live sources). Or with live sources may be special settings on the graph clocking also required in both graph config and DirectShowSource config.

Microsoft also writes Live Sources are the special case in DirectShow and not very nice so new API is recommended in new Windows solutions for live sources - https://learn.microsoft.com/en-us/windows/win32/directshow/live-sources .

Considering the oldness of DirectShow and ffdshow and the problems you mentioned in some previous posts and the tests I did, it seems practically unusable and unstable, so instead of using DirectShow, I might make a plugin for this using FFmpg (FFMS2 (https://github.com/FFMS/ffms2)).

[Mitra]

It is sort of wiki-article to AVS:
............
YUV<->RGB transforms are only lossless if enough precision used (like float) and only with 4:4:4 YUV without chroma subsampling used (because subsampled Y is lost it is impossible to restore 4:4:4 data from 4:2:x YUV even with lower bandwidth chroma with low distortions).
I learned important and new points (new for me) from your posts, thank you very much.

[DTL]

I might make a plugin for this using FFmpg (FFMS2 (https://github.com/FFMS/ffms2)).

If it can interface with Blackmagic API and provide both 8bit and 10bit YUV capture (may be higher with new Blackmagic cards ?) it may be best solution. Though each hardware vendor like Blackmagic or AJA can provide different API to interface with capture software so plugin need to support many different APIs and may be need to be rebuild with driver versions advances and so on.

The good point of Microsoft APIs is putting some single standard API over different hardware vendors for typical use cases via additional layer like DirectX. So it become easier to program and Microsoft can certify vendor's driver to work with Microsoft requirements for the API to user side. No need of direct interaction of user with hardware vendor support.

[FranceBB]

may be higher with new Blackmagic cards?

My Decklink is still relatively old and it's indeed limited to 4:2:2 YUV 10bit.

https://i.imgur.com/tlua2cV.png

Not that it matters for my use case given that I'm capturing SD 25i TFF from 1 inch type c, BetaCAM, U-Matic and DVCPro, but still...
Does anyone have any new Blackmagic Decklink card so that we can see which recording options it offers in media express?


EDIT:
So... what I posted above was the 4K card, but by googling a bit it actually looks like they released the new DeckLink 8K Pro G2 and they claim to support 4:2:2 YUV 10bit and 12bit.

https://images.blackmagicdesign.com/images/products/decklink/techspecs/hero/decklink-8k-pro-g2-sm.jpg?_v=1715735481

I don't have such a card, so I can't verify their claims, but it looks like 12bit is now a thing.
The only downside? Well, it's gonna set you back by $745. :(

[DTL]

As I remembe AJA made 12bit capture cards with analog inputs too. But for SDI >10bits possibly for some HDR formats only. 12bit looks like was typical for Analog/Digital cards at the end of 200x - https://www.amazon.com/AJA-KONA-LHe-Digital-Capture/dp/B006WUD8Q0 . When BetacamSP was still in use at some places. Though 12bit may be some overkill for typically very noisy BetacamSP recordings. It was simply sort of High-End analog SD capture cards. But it may be 12bit ADC only and processing - it may feed 10bit only data to capture client.

AJA KONA 1 promise 12bit RGB digital for SD/HD up to 60p https://www.aja.com/products/kona-1 .

[Mitra]

If it can interface with Blackmagic API and provide both 8bit and 10bit YUV capture (may be higher with new Blackmagic cards ?) it may be best solution. Though each hardware vendor like Blackmagic or AJA can provide different API to interface with capture software so plugin need to support many different APIs and may be need to be rebuild with driver versions advances and so on.

The good point of Microsoft APIs is putting some single standard API over different hardware vendors for typical use cases via additional layer like DirectX. So it become easier to program and Microsoft can certify vendor's driver to work with Microsoft requirements for the API to user side. No need of direct interaction of user with hardware vendor support.

Thank you for your tips, but FFmpg can also use the features and filters of the directshow, of course, I think it is better to use the Blackmagic API directly, because I have been familiar with the Blackmagic SDK for years and I have created several software, including a free 8-channel standard SD/HD/UHD playout / recorder (http://media4dco.com/downloads/HTPlayoutSetupForAdmin.zip) (not open source but completely free) that all channels can work simultaneously on a computer.


Unlike Blackmagic, the Aja cards SDK is not free and I don't think all its features are accessible via DirectShow, so I think FFmpeg is better choice for me.

[Mitra]

My Decklink is still relatively old and it's indeed limited to 4:2:2 YUV 10bit.
.......
I don't have such a card, so I can't verify their claims, but it looks like 12bit is now a thing.
The only downside? Well, it's gonna set you back by $745. :(

I have an 8K pro (not G2):

http://media4dco.com/downloads/bm.png

[FranceBB]

Oh... so still 4:2:2 YUV 10bit v210 for 8K Pro... Bummer. :(

[Mitra]

My free 8-channel standard SD/HD/UHD playback / recorder supports all old and new blackmagic decklink cards, please try it whenever you need.

Windows only and requires Desktop Video 11.4 or later.

Download link (http://media4dco.com/HTPlayoutSetup.zip)

screenshots :
http://media4dco.com/downloads/playout3.jpg

http://media4dco.com/downloads/playout4.jpg