New DirectX 12 Video APIs

[tormento]

Today DirectX 12 provides APIs to support GPU acceleration for several video applications such as Video Decoding, Video Processing and Motion estimation as detailed in Direct3D 12 Video Overview.

Announcing new DirectX 12 feature – Video Encoding! (https://devblogs.microsoft.com/directx/announcing-new-directx-12-feature-video-encoding/)

Direct3D video motion estimation (https://docs.microsoft.com/en-us/windows/win32/medfound/direct3d-video-motion-estimation)

I hope it can be usefull for some new plugins or to modernize current ones.

[DTL]

Good news. But how to know which video cards or embedded video accelerator support motion estimation ? Before buying it and testing by special software. Even for developing and debugging it hardware is required because no software emulation ?

[tormento]

Good news. But how to know which video cards or embedded video accelerator support motion estimation?
I think that they are the same listed for video encoding API support:


AMD Radeon RX 5000 series or greater, Ryzen 2xxxx series or greater
Intel Tiger Lake, Ice Lake, Alder Lake (from early 2022)
NVIDIA GeForce GTX 10xx and above, GeForce RTX 20xx and above, Quadro RTX NVIDIA RTX

AFAIK HEVC has some sort of motion estimation during encoding.

Beside that, my last programming skills are earlier than Microsoft even invented APIs, on a Bull mainframe and a Fortran77 compiler :)

I think that digging into the APIs, you can find everything.

[DTL]

"NVIDIA GeForce GTX 10xx "

We have GTX 1060 6 GB accesible at work. Can try to make simple test software to check if motion estimation API available. Windows10 system resources monitor shows Video Encoding performance graph.

"HEVC has some sort of motion estimation during encoding."

All MPEGs are based on the motion estimation. Only not any hardware encoder have open API for using it by external software. And may be hardware of motion estimation is not accessible by external data input and output - it may be closed MPEG encoder engine. With only uncompressed frames input and MPEG elementary stream output. It was some work of Microsoft DirectX developers with hardware manufacturers about creating and standartizing some API for motion estimation hardware processing (data format, etc) to be accessible via DirectX API.

[DTL]

The Motion Estimator interface looks like working at GTX 1060 . Size of frame max 4096x4096 and min 32x32. Block size looks like only 8x8 and 16x16.

Testing tool is https://drive.google.com/file/d/1-lKHBp0I4_6JyATl-Py2qIPYPNPTWuxH/view?usp=sharing . It is based on Microsoft D3D12 color triangle 'hello world' sample and try to init motion estimator at startup and emit messageboxes with queried data. 0-result mean S_OK. Works at Win10 build 19043 . Though ID3D12VideoDevice1::CreateVideoMotionEstimator require
Minimum supported client Windows 10 Build 20348
Minimum supported server Windows 10 Build 20348
https://docs.microsoft.com/en-us/windows/win32/api/d3d12video/nf-d3d12video-id3d12videodevice1-createvideomotionestimator
Do not know how it corresponds.

But the output is only vector field without SAD data. So to feed the MDegrainN the SAD calculation will be needed. It need again read src and ref frames from memory to CPU. Though getting motion vectors field already saves about half of MAnalyse current time (at pel=1 and much more time at qpel). So the possible speedup from using hardware motion estimator is not very great. Or it is need to develop (shader-based ?) the SAD computation with given vector field inside GPU and output separate 'texture' as field of SADs and combine it into format for MDegrainN.

So the sort of add-on to mvtools is possible. Need experienced in shader-creation developer ?

Unfortunately I even can not set development enviroment at the system with GTX 1060 for debugging - so the possibe development time is undefined. May be need to found the cheapest possble second-hand PCI-E video card with this support and buy it to home system. GTX 1060 is too expensive. Which may be the cheapest ?

[tormento]

Which may be the cheapest ?
A used one. I don't know where you are from but on eBay you can have one with ~100€.

[StainlessS]

Is Microsoft API the way to go, it kind of cuts out linux a bit.

An Introduction to the NVIDIA Optical Flow SDK:- https://developer.nvidia.com/blog/an-introduction-to-the-nvidia-optical-flow-sdk/
Says implemented for Turing+ cards [whatever that means].

Also, Apparently implemented in OpenCV [see Resources]:- https://developer.nvidia.com/opticalflow-sdk

EDIT: Seems Turing a lot more recent than my 1070:- https://en.wikipedia.org/wiki/Turing_(microarchitecture)
Products using Turing

GeForce 16 series
GeForce GTX 1650
GeForce GTX 1650 (Mobile)
GeForce GTX 1650 Max-Q (Mobile)
GeForce GTX 1650 (GDDR6)
GeForce GTX 1650 Super
GeForce GTX 1650 Ti (Mobile)
GeForce GTX 1660
GeForce GTX 1660 (Mobile)
GeForce GTX 1660 Super
GeForce GTX 1660 Ti
GeForce GTX 1660 Ti (Mobile)
GeForce GTX 1660 Ti Max-Q (Mobile)
GeForce 20 series
GeForce RTX 2060
GeForce RTX 2060 12GB
GeForce RTX 2060 (Mobile)
GeForce RTX 2060 Max-Q (Mobile)
GeForce RTX 2060 Super
GeForce RTX 2060 Super (Mobile)
GeForce RTX 2070
GeForce RTX 2070 (Mobile)
GeForce RTX 2070 Max-Q (Mobile)
GeForce RTX 2070 Max-Q Refresh (Mobile)
GeForce RTX 2070 Super
GeForce RTX 2070 Super (Mobile)
GeForce RTX 2070 Super Max-Q (Mobile)
GeForce RTX 2080
GeForce RTX 2080 (Mobile)
GeForce RTX 2080 Max-Q (Mobile)
GeForce RTX 2080 Super
GeForce RTX 2080 Super (Mobile)
GeForce RTX 2080 Super Max-Q (Mobile)
GeForce RTX 2080 Ti
Titan RTX
Nvidia Quadro
Quadro RTX 3000 (Mobile)
Quadro RTX 4000
Quadro RTX 5000
Quadro RTX 6000
Quadro RTX 8000
Nvidia Tesla
Tesla T4

[kedautinh12]

I think GTX 10xx very cheap now if it's compaire with latest gen of Nvidia GPU. Just compaire powerful and price between them

[DTL]

List of NVIDIas with encoder support - https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new . Still not know which minimum generation of NVenc will have Motion Estimation API.
May be slowest GTX 750 will work too ? But it is used here also about $66 that is too much for experiments. May be something like $10..15 ?

"I don't know where you are "

I from big poor cold country. And now the price of its 'wood money' is low. For the price of used GTX 750 I can buy firewood to heat my living construction about half of winter with 2 wood stoves. And now we have exessively cold beginning of December from the beginning of a century - night is as low as -26C 2 times already. Typically it is about 0 Celsium untill beginning of a January.

" Turing+ cards [whatever that means]"

It looks FAMILY column in that table. But Turing+ mean Ampere only ?

That is what enough for mvtools - "NVIDIA GPUs from Maxwell, Pascal, and Volta generations include one or more video encoder (NVENC) engines which provided a mode called Motion-Estimation-only mode. This mode allowed users to run only motion estimation on NVENC and retrieve the resulting motion vectors (MVs).".

So even Maxwell 1st Gen FAMILIY and 4th Gen NVenc should support Motion Estimation API ? That is GTX 750 card minimum listed in that table.

" it kind of cuts out linux a bit."

I think if NVIDIA cards have driver for Linux it have to expose same Motion Estimation API if hardware support.

"my 1070"

It should support Motion Estimation and have 2 NVENC chips (may be run in parallel and have 2x performance of 1060 ?) . You can run that test software - it have to display 0 return on getting resources (!=0 mean HRESULT error code) and non-zero max/min width/height of frame to process.

That executable part for MotionEstimation check is

ComPtr<IDXGIAdapter1> hardwareAdapter;
GetHardwareAdapter(factory.Get(), &hardwareAdapter);

ThrowIfFailed(D3D12CreateDevice(
hardwareAdapter.Get(),
D3D_FEATURE_LEVEL_11_0,
IID_PPV_ARGS(&m_device)
));


ComPtr<ID3D12VideoDevice> vid_dev;

HRESULT query_device1 = m_device->QueryInterface(IID_PPV_ARGS(&vid_dev));

char str[2048];
sprintf_s(str, "Query ID3D12VideoDevice return %d\n", query_device1);
MessageBoxA(NULL, str, "Test HW motion estimation", MB_OK);

D3D12_FEATURE_DATA_VIDEO_MOTION_ESTIMATOR MotionEstimatorSupport = { 0u, DXGI_FORMAT_NV12 };
HRESULT feature_support = vid_dev->CheckFeatureSupport(D3D12_FEATURE_VIDEO_MOTION_ESTIMATOR, &MotionEstimatorSupport, sizeof(MotionEstimatorSupport));

sprintf_s(str, "CheckFeatureSupport return: %d\n", feature_support);
MessageBoxA(NULL, str, "Test HW motion estimation", MB_OK);

sprintf_s(str, "MEstimator Feature support: DXGI_FORMAT InputFormat %d\n \n", MotionEstimatorSupport.InputFormat);
MessageBoxA(NULL, str, "Test HW motion estimation", MB_OK);

sprintf_s(str, "MEstimator Feature support: D3D12_VIDEO_MOTION_ESTIMATOR_SEARCH_BLOCK_SIZE_FLAGS BlockSizeFlags %d", MotionEstimatorSupport.BlockSizeFlags);
MessageBoxA(NULL, str, "Test HW motion estimation", MB_OK);

sprintf_s(str, "MEstimator Feature support: D3D12_VIDEO_MOTION_ESTIMATOR_VECTOR_PRECISION_FLAGS PrecisionFlags %d", MotionEstimatorSupport.PrecisionFlags);
MessageBoxA(NULL, str, "Test HW motion estimation", MB_OK);

sprintf_s(str, "MEstimator Feature support: D3D12_VIDEO_SIZE_RANGE SizeRange \n MaxW %d MaxH %d MinW %d MinH %d", MotionEstimatorSupport.SizeRange.MaxWidth, \
MotionEstimatorSupport.SizeRange.MaxHeight, MotionEstimatorSupport.SizeRange.MinWidth, MotionEstimatorSupport.SizeRange.MinHeight);
MessageBoxA(NULL, str, "Test HW motion estimation", MB_OK);


ComPtr<ID3D12VideoDevice1> vid_dev1;

HRESULT query_vid_device1 = m_device->QueryInterface(IID_PPV_ARGS(&vid_dev1));

sprintf_s(str, "QueryInterface ID3D12VideoDevice1 return: %d\n", query_vid_device1);
MessageBoxA(NULL, str, "Test HW motion estimation", MB_OK);


D3D12_VIDEO_MOTION_ESTIMATOR_DESC motionEstimatorDesc = {
0, //NodeIndex
DXGI_FORMAT_NV12,
D3D12_VIDEO_MOTION_ESTIMATOR_SEARCH_BLOCK_SIZE_8X8,
D3D12_VIDEO_MOTION_ESTIMATOR_VECTOR_PRECISION_QUARTER_PEL,
{1920, 1080, 1280, 720} // D3D12_VIDEO_SIZE_RANGE
};

ComPtr<ID3D12VideoMotionEstimator> spVideoMotionEstimator;
HRESULT vid_est = vid_dev1->CreateVideoMotionEstimator(
&motionEstimatorDesc,
nullptr,
IID_PPV_ARGS(&spVideoMotionEstimator));

sprintf_s(str, "ID3D12VideoMotionEstimator return: %d\n", vid_est);
MessageBoxA(NULL, str, "Test HW motion estimation", MB_OK);




4096x4096 looks natural limit of 16bit signed output MV format with qpel precision - it is 16384/4. So for pel=1 processing the data need to be /4. Having hardware-accelerated half pel and quaterpel is very valueable because with CPU search they are very and extremely very slow.

[Dogway]

Reading around it seems GTX 1650 Super is the "cheapest" with 7th Gen NVENC. There are some no-SUPER with chips TU116/TU106, but that's hard to know when buying.

[StainlessS]

It looks FAMILY column in that table. But Turing+ mean Ampere only ?
by turing+, I meant turing and later.

I think if NVIDIA cards have driver for Linux it have to expose same Motion Estimation API if hardware support.
Lovely, thanks.

On my 1070 Founders edition, with that Testing Tool, I get


Test HW Motion estimation

Query ID3D12VideoDevice return 0
CheckFeatureSupport return 0
MEstimator Feature support: DXGI_FORMAT InputFormat 103
MEstimator Feature support: D3D12_VIDEO_MOTION_ESTIMATOR_SEARCH_BLOCK_SIZE_FLAGS BlocksizeFlags 3
MEstimator Feature support: D3D12_VIDEO_MOTION_ESTIMATOR_VECTOR_PRECISION_FLAGS PrecisionFlags 1
MEstimator Feature support: D3D12_VIDEO_SIZE_RANGE SizeRange MaxW 4096 MaxH 4096 MinW 32 MinH 32
Queryinterface ID3D12VideoDevice1 return 0
ID3D12VideoMotionEstimator return 0

EDIT: Typo tilde removed from above.

night is as low as -26C 2 times already.
WOW, cold enough to freeze the balls off a brass monkey.

[tormento]

Test HW Motion estimation

Query ID3D12~VideoDevice return 0
CheckFeatureSupport return 0
MEstimator Feature support: DXGI_FORMAT InputFormat 103
MEstimator Feature support: D3D12_VIDEO_MOTION_ESTIMATOR_SEARCH_BLOCK_SIZE_FLAGS BlocksizeFlags 3
MEstimator Feature support: D3D12_VIDEO_MOTION_ESTIMATOR_VECTOR_PRECISION_FLAGS PrecisionFlags 1
MEstimator Feature support: D3D12_VIDEO_SIZE_RANGE SizeRange MaxW 4096 MaxH 4096 MinW 32 MinH 32
Queryinterface ID3D12VideoDevice1 return 0
ID3D12VideoMotionEstimator return 0


Same here on a 1060 3GB.

DTL, perhaps with a CLI version could be easier to paste results. About cheaper cards: the question is not if they exist but if they are supported by drivers.

I think the lower supported driver should have WDDM 2.9: WDDM 2.9 in Windows 10 Insider Preview "Iron" will bring support for GPU hardware acceleration to the Windows Subsystem for Linux 2 (WSL 2) and support for feature level 12_2 and HLSL Shader Model 6.6.3.x.

[DTL]

"MEstimator Feature support: D3D12_VIDEO_SIZE_RANGE SizeRange MaxW 4096 MaxH 4096 MinW 32 MinH 32
Queryinterface ID3D12VideoDevice1 return 0
ID3D12VideoMotionEstimator return 0"

It works OK as expected. We need a hero to try to make version of MAnalyse with this API :) . Also good to found some 'simplest' card Maxwell (1st Gen) and check too.

The announcement of HW encoding of https://devblogs.microsoft.com/directx/announcing-new-directx-12-feature-video-encoding/ looks like about HEVC high quality encoding. But Motion Estimation API is documented at Microsoft site from the middle of 2021 (or may be beginning - Direct3D video motion estimation Article 02/05/2021). Also at the reddit site there is a comments - the very old NVenc were worse in the mpeg encoding quality in compare with newer (Turing and Turing+), but as documented should support Motion Estimating and hopefully not (significantly) worse in compare with newer chips.

" perhaps with a CLI version could be easier to paste results."

Unfortunately printf() function do not output text from that program so the use of MessageBox() was fastest way to create portable tool. Or need to sit and search why printf() do not output text to CLI. May be something like 'console support' not connected/disabled/etc.

The base of the program were taken from 'hello-world d3d12' sample of https://github.com/microsoft/DirectX-Graphics-Samples/tree/master/Samples/Desktop/D3D12HelloWorld/src

[StainlessS]

Query ID3D12~VideoDevice return 0
just small note, that tilde [~] is typo, should not be there.

EDIT:
We need a hero to try to make version of MAnalyse with this API
That would be way too heroic for me I'm afraid, not much of a CPP programmer, and almost no Windows or DirectX experience,
and would have major problems just understanding mvtools code.

[DTL]

"just understanding mvtools code."

The simplest version is easy enough - just take 2 planes (ref and src) at GetFrame of MAnalyse - convert plane to the required DXGI_FORMAT_NV12 (or may be try to feed MAnalyse with this format if Avisynth can do this conversion or may be use some CUDA/DX conversion after loading YV12 or..). upload to GPU mem and load and execute that ME program. Then download (read back to host mem) the result MV buffer - set it as level1 predictor result and perform 'SAD check' only SearchMVs processing in PlaneofBlocks once for level0 plane to get SADs of the MVs do not use any more predictors or refining (close or equal to PredictorType=3 already in new MAnalyse). And return result to MDegrain().

The 'second generation' is faster but more advanced - try to perform SAD check inside DirectX-GPU and load back to host memory 2 arrays - MVs and SADs and interleave to feed back to MDegrainN.

Third-generation - perform MDegrainN inside GPU (possibly no sense because not enough GPU memory typical onboard for large enough tr, though may work for MDegrain1,2,3 (6?) and not large frame sizes).

In 'scalar' SAD function version the PT=3 PseudoEPZ_search is very simple as

template<typename pixel_t>
void PlaneOfBlocks::PseudoEPZSearch_no_refine(WorkingArea& workarea) // no refine - only predictor check
{
typedef typename std::conditional < sizeof(pixel_t) == 1, sad_t, bigsad_t >::type safe_sad_t;

sad_t sad;

if (smallestPlane) // never get here - normal use is sequence of params with 'real' search like optPredictorsType="3,x" where x < 3.
{
workarea.bestMV = zeroMV;
workarea.nMinCost = verybigSAD + 1;
}
else
{
workarea.bestMV = workarea.predictor; // already ClipMV() processed in the search_mv_slice
// only recalculate sad for interpolated predictor to be compatible with old/typical thSAD setting in MDegrain
sad_t sad = LumaSAD<pixel_t>(workarea, GetRefBlock(workarea, workarea.bestMV.x, workarea.bestMV.y));
sad_t saduv = (chroma) ? ScaleSadChroma(SADCHROMA(workarea.pSrc[1], nSrcPitch[1], GetRefBlockU(workarea, workarea.bestMV.x, workarea.bestMV.y), nRefPitch[1])
+ SADCHROMA(workarea.pSrc[2], nSrcPitch[2], GetRefBlockV(workarea, workarea.bestMV.x, workarea.bestMV.y), nRefPitch[2]), effective_chromaSADscale) : 0;
workarea.bestMV.sad = sad + saduv;
}


// we store the result
vectors[workarea.blkIdx].x = workarea.bestMV.x;
vectors[workarea.blkIdx].y = workarea.bestMV.y;
vectors[workarea.blkIdx].sad = workarea.bestMV.sad;

workarea.planeSAD += workarea.bestMV.sad; // for debug, plus fixme outer planeSAD is not used
}


For 'multi-block' SIMD processing (faster) it is a bit more complex - first got length of coherency in vector stream and call the appropriate multi-block SAD function from the list of available for current CPU SIMD architecture or pass a mask of coherent blocks for each vector offset value.

[tormento]

possibly no sense because not enough GPU memory typical onboard for large enough tr, though may work for MDegrain1,2,3 (6?) and not large frame sizes)
When resizable bar will be widespread enough, cpu or gpu memory will have less or no impact at all, given DDR5 bandwidth and lower latencies on transfer between the two kinds.

[DTL]

Well - the better for speed is 'gen 1.5' way: Do not calculate SAD in MAnalyse and output only MVs to re-writtten version of MDegrainN. Because it is faster to check SAD in MDegrainN at time of weight calculating. It will reduce read traffic from memory to about 1/2 and increase speed to about 2x.

But special 'compatibility' mode in MAnalyse with standard mv output (mv + sad) still may be required for other functions using MAnalyse output.

The cheapest possible Maxwell-based card is looks GTX 745 - low profile OEM card with slow memory in compare with GTX 750 and lower performance. I start to bargaining it secondhand from about $33 here. And may be will test for ME feature support next week - one seller provide testing capability before purchasing.

Though it is an idea to test 'remote debugging mode' - if it is not too dangerous to install 'remote debugger' at system with GTX 1060 it will be very helpful. Still not have experience with remote debugging of applications via IP.

[tormento]

And may be will test for ME feature support next week - one seller provide testing capability before purchasing.
If memory helps me, it has CUDA too. I have heard that the CUDA toolkit converts almost flawlessly from C++ to CUDA instruction. Perhaps you could give a look at it too.

[pinterf]

If memory helps me, it has CUDA too. I have heard that the CUDA toolkit converts almost flawlessly from C++ to CUDA instruction. Perhaps you could give a look at it too.

Like here in Nekopanda's CUDA Mvtools light
https://github.com/pinterf/AviSynthCUDAFilters/blob/master/KTGMC/MV.cpp
And the code actually in CUDA
https://github.com/pinterf/AviSynthCUDAFilters/blob/master/KTGMC/MVKernel.cu
with MDegrainN
https://github.com/pinterf/AviSynthCUDAFilters/blob/master/KTGMC/MVKernel.cu#L2600

[DTL]

I see the remote debugging is simple enough and not required to setup awful software like visual studio on remote system. So will try to start adding hardware dx12-me to MAnalyse this week at work.

Anyway even with infinite speed me engine it is required to re-write MDegrainN from scattering received mvs array to supplemental Fake* structure because it uses almost same time as MDegrainN processing at Core2 Duo E7500. Though the first version will mostly probably uses standard old MDegrainN.

It may be inheritance from old past times when MDegrain was interconnected with motion search in one filter and the later developers simply put the received mvs via AVS file to Fake structure to keep working MDegrain without rewriting. And with typically very slow MAnalyse the speed penalty was not very visible. But nowdays with fast MAnalyse modes and hardware-accelerated MAnalyse it really visible speedlimit (in the MVClip::Update()) and need to be removed. MDegrainN processing should access the received MVs array directly.

[DTL]

With multi-PCIe cards setup it is possible to create 'degraining farm' at one host and it may be much cheaper in compare with multi-workers distributed processing with CPU-based workers and IP interconnection.

I see used cheapest Maxwell-based 745 cards from about $50. Though it require some multi-PCIe (x16 ?) motherboard or may be available special 'PCIe-risers' to connect 16x card to 1x slot at slower speed ?

The MDegrainN motion search task is naturally paralleled to 2_x_tr number of src+ref searches so simple tr=12 degraining process can load up to 24 DX12-ME workers (NVIDIA promises to execute >1 ME task per chip at some chips).

The MVs data usage in MDegrainN is already fixed in some way and should be less limiting the speed I hope.

It also interesting question - what is the max vector length search in NVIDIA ME engine ? Do it have any limits ? Currently API do not have any settings about it. Though for typical degraining work the very short vectors like +-several samples are typically enough because it work on slow motion areas and with high speed motion noise is less visible and typically no large high speed areas in the motion pictures content.

[ReinerSchweinlin]

Maybe helpful addition:

While "Turing" refers to the GPU Generation, the included Encoders are not necessarely the same.

A 1650 has the VOLTA Encoders, not being able to produce B-frames with h265.

The 1660 has the b-frame HEVC capability, but only with nvencc is able to produce b-frames, the Nvidia provided encoder (used in A´s Video converter for example) is NOT using b-frames.

ffmpeg with hw-acc encoding also does not support b-frames at the moment.

all cards having "RTX" in theyr name work fine with b-frames HEVC encoding.

some of the quadro and OEM Cards also are missing the newer encoder (while being labeled turing or pascal)...

[DTL]

Development with remote debugging via LAN is started but not very fast because of my rare being at work with GTX1060 card and not very great being after long no sleeping time. Currently DX12-me init is partially finished in MAnalyse - this commit https://github.com/DTL2020/mvtools/commit/cbeb37ae747f11bcc3f5c92695c8404e1ea79769

If someone can try to provide remote debugging environment it may speedup the process. It require public IP connection for debugger and ftp access to the folder to upload dev builds of .dll and may be other required debug .dlls. It not require to install many to system - only unpack archive to some folder with known path from root of disk letter and run remote debugger monitor application. The debug session is protected with login/password of local system user with 'remote debugging' access.

It is not recommended by Microsoft to debug via internet because it may be slow and packet loss and more but it is possibly working solution.

Still do not know if GTX745 card support DX12-ME and not get it to my living place cheap enough. Unfortunately it looks no software emulation of this processing in DX12 available. Not search it very hard though.

[tormento]

If someone can try to provide remote debugging environment it may speedup the process.
Perhaps you can try to contact videoh. He has lot of experience with nvidia cards and CUDA.

[DTL]

Sad news - tested GTX 745 card at Win10 19041 and November 2021 driver and it do not provide HW-ME function. So it is again not known which cheapest card (from Maxwell ?) can provide required service. May be need to e-mail NVIDIA support about this issue.

Hehe - the article about hardware ME acceleration for external software clients https://developer.nvidia.com/blog/an-introduction-to-the-nvidia-optical-flow-sdk/ is dated Feb 2019 - the 2+ years passed already. Have post a question to nvidia developers support forums about selecting DX12-ME capable card.

[DTL]

Possibly more sad news - the GTX750 was tested and looks also not support. Though it is listed as second-generation of Maxwell.

I mostly finished DX12_ME interface part of program. But still need many help of Microsoft developers to get it working. Or may be any working sample of DX12 video encoder (or motion estimator). The current program returns no-error HR-state of most of functions but final mapping of the resulted resource buffer to read MVs in the plugin return 'general error - D3D device disconnected' and error reason 'application have errors and need debug'. Very unclear where may be error. Also the resource state to motionestimator logical device still can not be switched to required state 'motion estimator read source' after writing source and ref data.

[DTL]

With enabling debug layer in DX12 the error messages much more better. Current progress: Possibly close to real processing tech sample for speed test. Still required conversion of input to NV12 format and correct reading of output. Also currently somewhere inside GPU is huge memory leak so it fill 3 GB memory after 500 FullHD frames. Need to found how to free resources. The ME performance in the GTX1060 looks comparable to i5-9600K or may be 2..3 times better. The number of concurrent ME tasks looks like non-limited but may be performance will be degraded after 100% Encoder loading.

Need some thinking how to balance ME parallel tasks per each MDegrainN thread. Currently each MDegrainN thread scan src-ref pairs with MAnalyse one by one (total 2_x_tr pairs per each output frame) and it looks required thinking about Avisynth interfacing with MDegrainN and MAnalyse to allow asking for several src-ref pairs in parallel for load balancing between MDegrainN CPU processing and DX12-ME hardware acceleration. Without balancing currently hardware accelerator is not fully loaded.

May be the only easy solution to use several 'super' clips in one MDegrainN and asks for ME search in several MAnalyse objects in parallel ? Unfortunately current interfacing between MDegrain and MAnalyse via Avisynth layer looks like limit some easy ways of ME-multithreading.

[DTL]

Some more info on NVIDIA: https://www.nvidia.com/en-us/geforce/guides/broadcasting-guide/
One thing that is great about NVENC on the GeForce RTX 20 and 30-series and GeForce GTX 1650 Super and up is that all GPUs have the same NVENC with the same performance and quality, from the RTX 2060 to the RTX 3090.
NVENC can do up to 8K30, so the only way to overload it is to do 2x4K60 streams.

It is not clear how many current+ref pairs do it process for ME in MPEG encoding for each output frame (possibly from 1 to several) but even with 1 pair at 8K 30fps it mean about 16x better at fullHD resolution. I.e. 480 pairs per second.
MAnalyse/MDegrain need to process 2_x_tr pairs per each output frame.

[ReinerSchweinlin]

.....
One thing that is great about NVENC on the GeForce RTX 20 and 30-series and GeForce GTX 1650 Super and up is that all GPUs have the same NVENC with the same performance and quality, from the RTX 2060 to the RTX 3090....


not quite

http://forum.doom9.org/showthread.php?p=1959335#post1959335

[DTL]

It only about b-frames for MPEG. I hope performance of ME-engine of NVENC not degrades greatly at low cost chips. It is an idea using current sources to create small tech test of RAW ME performance of current hardware. With output result in pairs of processed frames per second. Currently I do not have an ideas how to run MAnalyse in solo processing mode without MDegrain as frames-request engine and in 'multi=true' mode. And MDegrain even in overlap=0 mode also a significant part of slow processing now.
With MShow() as data-sink it possibly will request only 1 pair or src+ref frames.

And script with MAnalyse output returns an error like 'no video clip created for output'.

[pinterf]

Had a look at NVidia site;
https://docs.nvidia.com/video-technologies/video-codec-sdk/nvenc-video-encoder-api-prog-guide/#motion-estimation-only-mode
DX12 and NvEnc direct ME mode cannot work at the same time.

[DTL]

It may be limits of old (or current) manufacturer-only API. In current Microsoft Windows API the hardware motion estimator looks do not have limitations and also data movement to and from ME-engine in the NVENC is performed via DX12 resources. Starting from creating D3D12-device.

Or may be NVIDIA have several generations (it mention 'old ME' and 'new Optical Flow') of ME modes and some (old ?) can work via DirectX12 without limitations. But may be lower in quality ?

Anyway the ME via DX12 works via DX12 API (Windows SDK) - not via NVIDIA SDK. And described as part of Windows Media Foundation API. So it should be NVIDIA-independent (as promised to have HWAcc for DX12 from Intel and AMD too). It is good advantage from NVIDIA-SDK only - not connected to one HW manufacturer only. It promised to be much more standard and long-live support in compare with one HW manufacturer only.
I hope in current latest windows the all old DirectShow API is perfectly working. It demonstrates the support for decades.

May be working with ME via NVIDIA SDK provide more settings/modes/params. Currently DX12-ME only support NV12 format as input for example.

[DTL]

Finally some working testbuild of MVtools with DX12_ME search mode for MAnalyse.

https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-dx12_me.a01

Enabling DX12_ME search: optSearchOption=5 (and levels=1). Using levels > 1 or default will possibly overwrite saved vectors with interpolated prediction from level 1. It copy MVs from DX12_ME output to vectors structure of the plane 0 and perform SAD calculation for MDegrainN weighting processing. Chroma looks like supported in both DX12_ME search and SAD. Currently looks like pel=1 only. The pel 2 and 4 is possible but in the future builds.

Only supported block sizes 8x8 and 16x16. Windows 10 build 19041 or newer looks like minimum requirement. When it uses DX12_ME hardware - the resource monitor TaskManager->GPU->Video Encoder shows the load graph and %.

Test script is about:

LoadPlugin("mvtools2.dll")
LoadPlugin("ffms2.dll")

FFMpegSource2("src.mxf")

ConvertToYV12()

tr = 12 # Temporal radius
super = MSuper (mt=false, chroma=true,pel=1, hpad=8, vpad=8)
multi_vec = MAnalyse (super, multi=true, blksize=8, delta=tr, chroma=false, overlap=0, mt=false, optSearchOption=5, levels=1)
MDegrainN (super, multi_vec, tr, thSAD=300, thSAD2=290, mt=false, wpow=4)

Prefetch(6) # set prefetch number to number of host CPU cores


Only hope completely compatible with MDegrainN (may crash other MDegrains and other with outside frame borders vectors sometime returned by DX12_ME). Need to find why they pass the ClipMV() limiting in MAnalyse.
So the performance of MDegrainN blenging engine is a bit limited in this build with additional clipping of block pos inside valid region.
Input format currently limited to YV12 only but may be relaxed to all Mvtools inputs because currently frames are read from loaded structures of MAnalyse.
Frame size tested - 1920x1080. Unfortunately inverlaced with separated fields (to 1920x540 frame size) still return some error from DX subsystem - need to fix in the later builds.

The speed at i5-9600K and with GTX1060 is a bit faster in compare with official 2.7.45 build and a bit slower in compare with SO=2. It looks overlapped modes of MDegrainN is supported (run but not test for quality).

May be better speed will be with SAD calculation also inside HWAcc with DirectX resoures (textures) processing. It will completely avoid the memory read into MAnalyse SAD calculation.

[tormento]

Finally some working testbuild of MVtools with DX12_ME search mode for MAnalyse.
Tried and not working for me. Nvidia 1060 3GB.

Access violation from AVS.

[DTL]

What was the script, frame size, frame format and error message ?

Currently found V-size limitation: It looks required integer number of blocks so block 8x8 work with 1080 height (1080/8=135) frame and not work with SeparatedFields to 540 height (540/8=67.5). Solution - pad vertical size with AddBorders() For example to process interlaced 1080 - pad to 1088. 1088/2/8=68.

[tormento]

What was the script, frame size, frame format and error message?
Really simple SMDegrain call:

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\3_00 wolf of Wall Street, The\wolf.dgi",ct=140,cb=140,cl=0,cr=0)
ConvertBits(16)
SMDegrain (tr=3, thSAD=300, refinemotion=true, contrasharp=false, PreFilter=5, plane=4, chroma=true)
fmtc_bitdepth (bits=8,dmode=8)
Prefetch(6) (tried without Prefetch too)

1920*1080 cropped to 1920*800 by DGSource.

[DTL]

1. MAnalyse with DX12_ME only support 8 bit input (and currently only YV12 format). Not sure if SMDegrain convert internally to 8 for MAnalyse or not.
2. To activate DX12_ME search mode in that build you need to pass option optSearchOption=5 to MAnalyse (and set levels =1 to not overwrite DX12_ME result with level1 search data in the level0 MVs array).

If you use SMDegrain without new options passing - again special build required with hardcoded options SO=5 and levels=1 .

Because speed of ME engine in GTX1060 is not many times faster in compare with i5-9600K currently trying to make Compute Shader SAD search to skip SearchMVs from GroupofPlanes completely (it saves 1 set of memory read operations) and will create fully compatible result of MAnalyse with all other clients filters of mvtools. Also have an idea how to make MDegrain HWAcc based without DX12 resources management in AVS core - to make all required reqources allocation in MAnalyse and pass pointers to resources via existing way of mvtools inbetween filters - via pseudo-audio stream. So in the best future all MAnalyse+MDegrain processing may be created with loaded to HWAcc memory frames once.

Also in current sources already implemented idea of half-sized ME for better speed: The DX12_ME engine currently can only work with qpel precision that is too large for typical pel=1 fastest processing. But with block size 8x8 it allow to make search with half-sized data (level1 from MSuper) and scatter received MVs to 16x16 block sized level0 with half precision truncating (from qpel to half pel). It currently controlled as optSearchOption=6. And now the speed limiting is SAD calculation in MAnalyse - so the next step to compute-shader processing in HWAcc required to get SAD values too.

[magnetite]

I tried this out on my GTX 1080 Ti and initially it threw an error (0xC000374). So I lowered the temporal radius to 2 instead of 12, and it was able to run. Performance was around 150 FPS.

While using the CPU only, performance was around 125 FPS on my i7 6700K.

Source was a DVD.

[ChaosKing]

I also made some quick tests:

Source ntsc DVD
CPU Ryzen 3600, GPU 3070 TI

Test script from here https://forum.doom9.org/showpost.php?p=1960271&postcount=33

But without wpow=4 (mvtools comlained, so I removed it)
+ removed Searchoption in non DX test

original mvtools
without prefetch: 28 fps
with prefetch(6): 130 fps
with prefetch(8): 155 fps
with prefetch(12): 177 fps

DX12 mvtools
without prefetch: 13 fps
with prefetch(6): 86 fps
with prefetch(8): 94 fps
with prefetch(12): 97 fps

[DTL]

" an error (0xC000374)"

This error code looks like something about heap corruption. Still do not have ideas where it can come from. Most of DirectX API calls have checks of HR-return value and if non-S_OK result - writes error message about the tried API call. It is possible to enable DirectX Debug layer with better error messages what is detected to going wrong but need to found way how to pick messages from something like 'debug output stream' in VisualStudio and add to Avisynth environment error output. So it may be special 'DX debug' builds of the plugin with possibly better error-collection from users.

"lowered the temporal radius to 2 instead of 12, and it was able to run"

Is it the max possible value without throwing error or just some low working ?

"DX12 mvtools
without prefetch: 13 fps"

It is with optSearchOption=5 ? The other modes may be somehow slower in compare with 2.7.45 in MDegrain because of added 'anti-bug' check of invalid vectors in that testbuild. In latest sources it looks I found where bad vectors may be passed and this check removed.

"ntsc DVD
CPU Ryzen 3600"

It have 32 MB L3 cache so at small frame size (and low tr) it may be faster to process on CPU and not send data for motion search to HWAccelerator and readback. For that config and frame size the HW processing may be faster only after transferring 'all' processing to DX12 pipeline - both motion search and blocks blending in MDegrain. It still some unknown future.

"with prefetch(12): 177 fps"

In that testbuild fastest working 'onCPU' optSearchOption should be 2. It should be faster in compare with v.2.7.45. Unfortunately SO=3 still not finished in debugging and works only at static colorbars mostly as tech demo of multi-blocks MAnalyse processing.

[ChaosKing]

" an error (0xC000374)"

It is with optSearchOption=5 ?

Yes all DX mvtools runs were tested with optSearchOption=5, for non DX I just removed it bcs it does not know the optSearchOption parameter. => I tested "pinterf mvtools" vs "DX mvtools"


EDIT:

with your mvtools build and optSearchOption=2
I get:
no prefetch() 45 fps
prefetch(6) 220 fps
prefetch(8) 248 fps
prefetch(12) 275 fps


That's quite the improvement! :eek:

[DTL]

"275 fps "

275/177=1.55. Not very great. For a bit faster processing you can also set optPredictorType=1 - though it may lower quality a bit. Default is optPredictorType=0 - should be most close quality to v.2.7.45. Raising PredictorType to 2,3,4 may degrade quality even more - experimental modes. Fastest is optPredictorType=4 (require adjustment of thSAD in MDegrain to lower values).

I expect from moving to HWAcc about 5x better speed. At least for intel-based CPU hosts with small enough cache sizes. The MVtools onCPU is limited mostly by host memory speed at fast CPUs like end-of-201x. Host memory speed is about 50 GB/s nowdays (about 2 channels of DDR4) and at top HWAccelerators reach about 1 TB/s - about 10..20 times faster.

[tormento]

That's quite the improvement! :eek:
I can't understand why there is so little sharing of this thread inside doom9 community. MVTools are one of the slowest and most used filter around and any help would be useful.

DTL is the only one developing this branch with almost no help at all.

[DTL]

It looks community is almost absent nowdays. And development process going to more difficult state: today I make about finished compute shader pipeline for SAD calculation with compute shader at HWAcc and found that remote debugging looks like do not support shader debugging. So the development of shader-based processing looks will be more slow with only way to download ready result from shader and analyse it.
Most of freeware developers were active about decade ago.

Some working tech demo of completely DX12-based (ME motion search + compute shader for SAD calculation) MAnalyse - https://drive.google.com/file/d/1cNsZvjVPEFdHi414-MJH24wwvn0s_x-0/view?usp=sharing . Based on the commit - https://github.com/DTL2020/mvtools/commit/1bfc70162c11b058e6ce5bee6047cba16727e838

It outputs 'standard' MVs pseudo-file (with SAD values) so should be compatible with other clients filters.

Only SO=5 (SO=6 still not finished in this build). Only overlap=0, chroma is used in ME but shader still not finished for addition of chroma SAD. Only 8x8 block size.

It works slower in compare with i5-9600K in RAW mvtools performance (with SO=2) but with x264 encoding it works faster because using HWAcc for MAnalyse saves some (about 30%) of CPU time for MPEG encoding.

Based on the testing of GTX1060 ME-engine performance: It looks RAW performance is about 700..800 pairs of ref+current frames per second with 1080p frame format and qpel precision. So it can significantly outperform i5-9600K only in half-frame size mode that limit block size to 16x16 only. Though max possible transfer of denoising processing to HWAcc will mostly free host CPU for MPEG encoding so total transcoding with denoising will be about twice faster (or limited to current settings x264 encoder execution speed onCPU).

[DTL]

The usage of natural massive-parallel processing on HWAcc makes things much easier:

The SAD compute shader for HWAcc is as small and easy as

RWTexture2D<int>OutputTexture : register(u0);
Texture2D<int>CurrentTexture_Y : register(t0);
Texture2D<int2>CurrentTexture_UV : register(t1);
Texture2D<int>ReferenceTexture_Y : register(t2);
Texture2D<int2>ReferenceTexture_UV : register(t3);
Texture2D<int2>ResolvedMVsTexture : register(t4);

void main(uint3 DTid : SV_DispatchThreadID)
{
int3 i3Coord;

i3Coord.x = DTid.x;
i3Coord.y = DTid.y;
i3Coord.z = 0;

int iBlockSize = 8;

int2 i2MV = ResolvedMVsTexture.Load(i3Coord);
i2MV.r = i2MV.r >> 2; // full frame search qpel/4
i2MV.g = i2MV.g >> 2;

int iYsrc;
int iYref;

int iSAD = 0;

for (int x = 0; x < iBlockSize; x++)
{
for (int y = 0; y < iBlockSize; y++)
{
i3Coord.x = DTid.x * iBlockSize + x;
i3Coord.y = DTid.y * iBlockSize + y;
iYsrc = CurrentTexture_Y.Load(i3Coord).r;

i3Coord.x = DTid.x * iBlockSize + x + i2MV.r;
i3Coord.y = DTid.y * iBlockSize + y + i2MV.g;

iYref = ReferenceTexture_Y.Load(i3Coord).r;

iSAD += abs(iYsrc - iYref);
}
}

OutputTexture[DTid.xy] = iSAD;
}

(still no chroma but add chroma is also simple in the future). And it is executed in the prof-programmers designed massive parallel architecture and software environment of HWAcc. No large hand-made parallel/SIMD program required. The execution speed is very nice. I see almost 0 load of GPU graph.

It is also not suffers from 'out of buffer' memory exceptions with 'invalid' vectors produced sometime by ME-engine. The sampler simply return zeros for out of frame coordinates.

The last core required MDegrainN compute shader is close to this.

And C++ multi-blocks SAD computing on general-purpose CPU with SIMD co-processor is about thousands lines of program and still greatly suffers from low memory speed.

Most of AVS filters may be put to compute shaders and greatly add to performance. But it require addition of DX-resources management in AVS core. Currently it will be only inside 'mvtools-environment'.

To help users it may be designed some helper-filter-pack like 'toDX12' to upload AVS pipeline to HWAcc memory and generate pointers to resources and 'fromDX12' to download back to host CPU memory and to continue AVS pipeline processing. And ability to compile and load for execution compute shaders for processing. It may be performed at user-side at filter load before execution. No special development environment required (though some debugging will be useful).

[ChaosKing]

Maybe you can also get some inspiration from a "cudaSynth" concept from here http://rationalqm.us/board/viewtopic.php?f=14&t=671

[DTL]

I think CUDA is more manufacturer-oriented API and less universal in compare with Windows DX-API. I expect support of DX-API will be not only from NVIDIA but intel and AMD (and some more HWAcc developers). So using natural for Windows API is easier in support and may be development I hope.

It is game-developers API that drives the market of HWAcc goods (for end-users/home entertainment use). CUDA market may be more limited and more for pro-usage (not for poor-people home PCs).

Unfortunately it looks intel with its QSV hardware accelerator may be too slow in driver development for DX12ME or may be old QSV hardware do not support required output by design. But intel with host-CPU built-in accelerator is also band-limited with very poor host RAM performance in compare with separate good accelerator board.

The too outdated intel-architecture of end-users PCs is now divided into separate high-performance islands - inside CPU caches and inside HWAcc board fast RAM. The host RAM is large but service only as very slow cache of data (for HDD and SSD).

[DTL]

Had a look at NVidia site;
https://docs.nvidia.com/video-technologies/video-codec-sdk/nvenc-video-encoder-api-prog-guide/#motion-estimation-only-mode
DX12 and NvEnc direct ME mode cannot work at the same time.

If it is about "Motion-estimation (ME) only mode is not supported if DirectX 12 device is used."

It may mean if DX12 device is used in application (? or the whole system ?) - it takes resources of ME engine to use for DX12-ME clients so it can not be shared with NVIDIA-SDK based applications at the same time.

Current sources shows how DX12-ME server runs with same DX12 device that provide Compute Shader execution (that is enough for other tasks of MAnalyse + MDegrain processing). And CS execution works with same resources in the same format as used by ME-engine - it saves from additional converting/copying/uploading. The loaded for ME processing resources after resolving MVs simply assigned as Shader Resource Views inputs to CS.

Also " For full-pel precision, the client must ignore two LSBs of the motion vector. For sub-pel precision, the two LSBs of the motion vector represent fractional part of the motion vector."

Possibly mean even with NVIDIA-API it is not possible to switch ME engine from qpel to full-pel search mode for better speed. It always works in qpel mode and user must discard 1 or 2 LSBs to get full pel MV value. But it not adds to speed.

I currently think if ME-engine from NVENC is not very great in speed and left most of HWAcc resources free it may be possible to make 'standard' MAnalyse processing on Compute Shaders because at each level blocks processing is mostly independent (only interconnected with FetchPredictors -> getting some surrounding processed blocks MVs as predictors) but it may possibly can also somehow added using ordered (?) processing inside group of threads. Or that additional predictors gathering may be skipped for more or less lower quality. It is now something inbetween optPredictorsType=1 and 2 modes - the usage of zero, interpolated and global predictors is possible with full independent blocks processing.

So users of 'complex' mvtools-based scripts may use different MAnalyse calls with different hardware execution units and get more speed.

[DTL]

New build - https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.06

Some faster in resource uploading and finally make chroma in SAD calculation. With chroma=true it finally visibly faster at GTX1060 in compare with CPU (i5-9600K) processing because optSearchOption=2 (and higher CPU-based) do not have chroma SAD processing now.
Also scaleCSAD is working too and significantly changes output MPEG speed. (Setting to -1 makes MPEG speed lower - either with lowering noise or softening too, -2 not tested yet) The adjusting steps looks even too coarse as integer 0,-1,-2 and may be good to add some finer (trying to keep compatibility - may be use float ?). The speed with executing at accelerator should not be visibly smaller even with fine float adjustment with float multiplication and converting to integer.

It looks scaleCSAD should be adjusted (fine tuned) with thSAD values in MDegrain (and other like scene change and other th-s).

At some quick tests it looks with FullHD frame and 8x8 block size there exist some issue with lowest 3 rows of blocks - may cause visible error blends with 'large' tr like 25. To fix if happens (temporarily ?) - make thSCD1 lower default 400 (like 350..320). It may be some hidden bug in data transfer somewhere - like cut-out some buffer too early. Though it appear rare enough near some scene changes only.

[tormento]

New build
Will try ASAP.

Is HBD a long way to go?