New DirectX 12 Video APIs [Archive]

DTL

11th December 2021, 11:14

The Motion Estimator interface looks like working at GTX 1060 . Size of frame max 4096x4096 and min 32x32. Block size looks like only 8x8 and 16x16.

Testing tool is https://drive.google.com/file/d/1-lKHBp0I4_6JyATl-Py2qIPYPNPTWuxH/view?usp=sharing . It is based on Microsoft D3D12 color triangle 'hello world' sample and try to init motion estimator at startup and emit messageboxes with queried data. 0-result mean S_OK. Works at Win10 build 19043 . Though ID3D12VideoDevice1::CreateVideoMotionEstimator require
Minimum supported client Windows 10 Build 20348
Minimum supported server Windows 10 Build 20348
https://docs.microsoft.com/en-us/windows/win32/api/d3d12video/nf-d3d12video-id3d12videodevice1-createvideomotionestimator
Do not know how it corresponds.

But the output is only vector field without SAD data. So to feed the MDegrainN the SAD calculation will be needed. It need again read src and ref frames from memory to CPU. Though getting motion vectors field already saves about half of MAnalyse current time (at pel=1 and much more time at qpel). So the possible speedup from using hardware motion estimator is not very great. Or it is need to develop (shader-based ?) the SAD computation with given vector field inside GPU and output separate 'texture' as field of SADs and combine it into format for MDegrainN.

So the sort of add-on to mvtools is possible. Need experienced in shader-creation developer ?

Unfortunately I even can not set development enviroment at the system with GTX 1060 for debugging - so the possibe development time is undefined. May be need to found the cheapest possble second-hand PCI-E video card with this support and buy it to home system. GTX 1060 is too expensive. Which may be the cheapest ?

tormento

11th December 2021, 12:50

Which may be the cheapest ?
A used one. I don't know where you are from but on eBay you can have one with ~100€.

StainlessS

11th December 2021, 12:51

Is Microsoft API the way to go, it kind of cuts out linux a bit.

An Introduction to the NVIDIA Optical Flow SDK:- https://developer.nvidia.com/blog/an-introduction-to-the-nvidia-optical-flow-sdk/
Says implemented for Turing+ cards [whatever that means].

Also, Apparently implemented in OpenCV [see Resources]:- https://developer.nvidia.com/opticalflow-sdk

EDIT: Seems Turing a lot more recent than my 1070:- https://en.wikipedia.org/wiki/Turing_(microarchitecture)
Products using Turing

GeForce 16 series
GeForce GTX 1650
GeForce GTX 1650 (Mobile)
GeForce GTX 1650 Max-Q (Mobile)
GeForce GTX 1650 (GDDR6)
GeForce GTX 1650 Super
GeForce GTX 1650 Ti (Mobile)
GeForce GTX 1660
GeForce GTX 1660 (Mobile)
GeForce GTX 1660 Super
GeForce GTX 1660 Ti
GeForce GTX 1660 Ti (Mobile)
GeForce GTX 1660 Ti Max-Q (Mobile)
GeForce 20 series
GeForce RTX 2060
GeForce RTX 2060 12GB
GeForce RTX 2060 (Mobile)
GeForce RTX 2060 Max-Q (Mobile)
GeForce RTX 2060 Super
GeForce RTX 2060 Super (Mobile)
GeForce RTX 2070
GeForce RTX 2070 (Mobile)
GeForce RTX 2070 Max-Q (Mobile)
GeForce RTX 2070 Max-Q Refresh (Mobile)
GeForce RTX 2070 Super
GeForce RTX 2070 Super (Mobile)
GeForce RTX 2070 Super Max-Q (Mobile)
GeForce RTX 2080
GeForce RTX 2080 (Mobile)
GeForce RTX 2080 Max-Q (Mobile)
GeForce RTX 2080 Super
GeForce RTX 2080 Super (Mobile)
GeForce RTX 2080 Super Max-Q (Mobile)
GeForce RTX 2080 Ti
Titan RTX
Nvidia Quadro
Quadro RTX 3000 (Mobile)
Quadro RTX 4000
Quadro RTX 5000
Quadro RTX 6000
Quadro RTX 8000
Nvidia Tesla
Tesla T4

kedautinh12

11th December 2021, 12:55

I think GTX 10xx very cheap now if it's compaire with latest gen of Nvidia GPU. Just compaire powerful and price between them

DTL

11th December 2021, 13:16

List of NVIDIas with encoder support - https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new . Still not know which minimum generation of NVenc will have Motion Estimation API.
May be slowest GTX 750 will work too ? But it is used here also about $66 that is too much for experiments. May be something like $10..15 ?

"I don't know where you are "

I from big poor cold country. And now the price of its 'wood money' is low. For the price of used GTX 750 I can buy firewood to heat my living construction about half of winter with 2 wood stoves. And now we have exessively cold beginning of December from the beginning of a century - night is as low as -26C 2 times already. Typically it is about 0 Celsium untill beginning of a January.

" Turing+ cards [whatever that means]"

It looks FAMILY column in that table. But Turing+ mean Ampere only ?

That is what enough for mvtools - "NVIDIA GPUs from Maxwell, Pascal, and Volta generations include one or more video encoder (NVENC) engines which provided a mode called Motion-Estimation-only mode. This mode allowed users to run only motion estimation on NVENC and retrieve the resulting motion vectors (MVs).".

So even Maxwell 1st Gen FAMILIY and 4th Gen NVenc should support Motion Estimation API ? That is GTX 750 card minimum listed in that table.

" it kind of cuts out linux a bit."

I think if NVIDIA cards have driver for Linux it have to expose same Motion Estimation API if hardware support.

"my 1070"

It should support Motion Estimation and have 2 NVENC chips (may be run in parallel and have 2x performance of 1060 ?) . You can run that test software - it have to display 0 return on getting resources (!=0 mean HRESULT error code) and non-zero max/min width/height of frame to process.

That executable part for MotionEstimation check is

ComPtr<IDXGIAdapter1> hardwareAdapter;
GetHardwareAdapter(factory.Get(), &hardwareAdapter);

ThrowIfFailed(D3D12CreateDevice(
hardwareAdapter.Get(),
D3D_FEATURE_LEVEL_11_0,
IID_PPV_ARGS(&m_device)
));

ComPtr<ID3D12VideoDevice> vid_dev;

HRESULT query_device1 = m_device->QueryInterface(IID_PPV_ARGS(&vid_dev));

char str[2048];
sprintf_s(str, "Query ID3D12VideoDevice return %d\n", query_device1);
MessageBoxA(NULL, str, "Test HW motion estimation", MB_OK);

D3D12_FEATURE_DATA_VIDEO_MOTION_ESTIMATOR MotionEstimatorSupport = { 0u, DXGI_FORMAT_NV12 };
HRESULT feature_support = vid_dev->CheckFeatureSupport(D3D12_FEATURE_VIDEO_MOTION_ESTIMATOR, &MotionEstimatorSupport, sizeof(MotionEstimatorSupport));

sprintf_s(str, "CheckFeatureSupport return: %d\n", feature_support);
MessageBoxA(NULL, str, "Test HW motion estimation", MB_OK);

sprintf_s(str, "MEstimator Feature support: DXGI_FORMAT InputFormat %d\n \n", MotionEstimatorSupport.InputFormat);
MessageBoxA(NULL, str, "Test HW motion estimation", MB_OK);

sprintf_s(str, "MEstimator Feature support: D3D12_VIDEO_MOTION_ESTIMATOR_SEARCH_BLOCK_SIZE_FLAGS BlockSizeFlags %d", MotionEstimatorSupport.BlockSizeFlags);
MessageBoxA(NULL, str, "Test HW motion estimation", MB_OK);

sprintf_s(str, "MEstimator Feature support: D3D12_VIDEO_MOTION_ESTIMATOR_VECTOR_PRECISION_FLAGS PrecisionFlags %d", MotionEstimatorSupport.PrecisionFlags);
MessageBoxA(NULL, str, "Test HW motion estimation", MB_OK);

sprintf_s(str, "MEstimator Feature support: D3D12_VIDEO_SIZE_RANGE SizeRange \n MaxW %d MaxH %d MinW %d MinH %d", MotionEstimatorSupport.SizeRange.MaxWidth, \
MotionEstimatorSupport.SizeRange.MaxHeight, MotionEstimatorSupport.SizeRange.MinWidth, MotionEstimatorSupport.SizeRange.MinHeight);
MessageBoxA(NULL, str, "Test HW motion estimation", MB_OK);

ComPtr<ID3D12VideoDevice1> vid_dev1;

HRESULT query_vid_device1 = m_device->QueryInterface(IID_PPV_ARGS(&vid_dev1));

sprintf_s(str, "QueryInterface ID3D12VideoDevice1 return: %d\n", query_vid_device1);
MessageBoxA(NULL, str, "Test HW motion estimation", MB_OK);

D3D12_VIDEO_MOTION_ESTIMATOR_DESC motionEstimatorDesc = {
0, //NodeIndex
DXGI_FORMAT_NV12,
D3D12_VIDEO_MOTION_ESTIMATOR_SEARCH_BLOCK_SIZE_8X8,
D3D12_VIDEO_MOTION_ESTIMATOR_VECTOR_PRECISION_QUARTER_PEL,
{1920, 1080, 1280, 720} // D3D12_VIDEO_SIZE_RANGE
};

ComPtr<ID3D12VideoMotionEstimator> spVideoMotionEstimator;
HRESULT vid_est = vid_dev1->CreateVideoMotionEstimator(
&motionEstimatorDesc,
nullptr,
IID_PPV_ARGS(&spVideoMotionEstimator));

sprintf_s(str, "ID3D12VideoMotionEstimator return: %d\n", vid_est);
MessageBoxA(NULL, str, "Test HW motion estimation", MB_OK);

4096x4096 looks natural limit of 16bit signed output MV format with qpel precision - it is 16384/4. So for pel=1 processing the data need to be /4. Having hardware-accelerated half pel and quaterpel is very valueable because with CPU search they are very and extremely very slow.

Dogway

11th December 2021, 13:28

Reading around it seems GTX 1650 Super is the "cheapest" with 7th Gen NVENC. There are some no-SUPER with chips TU116/TU106, but that's hard to know when buying.

StainlessS

11th December 2021, 14:27

It looks FAMILY column in that table. But Turing+ mean Ampere only ?
by turing+, I meant turing and later.

I think if NVIDIA cards have driver for Linux it have to expose same Motion Estimation API if hardware support.
Lovely, thanks.

On my 1070 Founders edition, with that Testing Tool, I get

Test HW Motion estimation

Query ID3D12VideoDevice return 0
CheckFeatureSupport return 0
MEstimator Feature support: DXGI_FORMAT InputFormat 103
MEstimator Feature support: D3D12_VIDEO_MOTION_ESTIMATOR_SEARCH_BLOCK_SIZE_FLAGS BlocksizeFlags 3
MEstimator Feature support: D3D12_VIDEO_MOTION_ESTIMATOR_VECTOR_PRECISION_FLAGS PrecisionFlags 1
MEstimator Feature support: D3D12_VIDEO_SIZE_RANGE SizeRange MaxW 4096 MaxH 4096 MinW 32 MinH 32
Queryinterface ID3D12VideoDevice1 return 0
ID3D12VideoMotionEstimator return 0

EDIT: Typo tilde removed from above.

night is as low as -26C 2 times already.
WOW, cold enough to freeze the balls off a brass monkey.

tormento

11th December 2021, 14:56

Test HW Motion estimation

Query ID3D12~VideoDevice return 0
CheckFeatureSupport return 0
MEstimator Feature support: DXGI_FORMAT InputFormat 103
MEstimator Feature support: D3D12_VIDEO_MOTION_ESTIMATOR_SEARCH_BLOCK_SIZE_FLAGS BlocksizeFlags 3
MEstimator Feature support: D3D12_VIDEO_MOTION_ESTIMATOR_VECTOR_PRECISION_FLAGS PrecisionFlags 1
MEstimator Feature support: D3D12_VIDEO_SIZE_RANGE SizeRange MaxW 4096 MaxH 4096 MinW 32 MinH 32
Queryinterface ID3D12VideoDevice1 return 0
ID3D12VideoMotionEstimator return 0

Same here on a 1060 3GB.

DTL, perhaps with a CLI version could be easier to paste results. About cheaper cards: the question is not if they exist but if they are supported by drivers.

I think the lower supported driver should have WDDM 2.9: WDDM 2.9 in Windows 10 Insider Preview "Iron" will bring support for GPU hardware acceleration to the Windows Subsystem for Linux 2 (WSL 2) and support for feature level 12_2 and HLSL Shader Model 6.6.3.x.

DTL

11th December 2021, 15:00

"MEstimator Feature support: D3D12_VIDEO_SIZE_RANGE SizeRange MaxW 4096 MaxH 4096 MinW 32 MinH 32
Queryinterface ID3D12VideoDevice1 return 0
ID3D12VideoMotionEstimator return 0"

It works OK as expected. We need a hero to try to make version of MAnalyse with this API :) . Also good to found some 'simplest' card Maxwell (1st Gen) and check too.

The announcement of HW encoding of https://devblogs.microsoft.com/directx/announcing-new-directx-12-feature-video-encoding/ looks like about HEVC high quality encoding. But Motion Estimation API is documented at Microsoft site from the middle of 2021 (or may be beginning - Direct3D video motion estimation Article 02/05/2021). Also at the reddit site there is a comments - the very old NVenc were worse in the mpeg encoding quality in compare with newer (Turing and Turing+), but as documented should support Motion Estimating and hopefully not (significantly) worse in compare with newer chips.

" perhaps with a CLI version could be easier to paste results."

Unfortunately printf() function do not output text from that program so the use of MessageBox() was fastest way to create portable tool. Or need to sit and search why printf() do not output text to CLI. May be something like 'console support' not connected/disabled/etc.

The base of the program were taken from 'hello-world d3d12' sample of https://github.com/microsoft/DirectX-Graphics-Samples/tree/master/Samples/Desktop/D3D12HelloWorld/src

StainlessS

11th December 2021, 16:01

Query ID3D12~VideoDevice return 0
just small note, that tilde [~] is typo, should not be there.

EDIT:
We need a hero to try to make version of MAnalyse with this API
That would be way too heroic for me I'm afraid, not much of a CPP programmer, and almost no Windows or DirectX experience,
and would have major problems just understanding mvtools code.

DTL

11th December 2021, 16:33

"just understanding mvtools code."

The simplest version is easy enough - just take 2 planes (ref and src) at GetFrame of MAnalyse - convert plane to the required DXGI_FORMAT_NV12 (or may be try to feed MAnalyse with this format if Avisynth can do this conversion or may be use some CUDA/DX conversion after loading YV12 or..). upload to GPU mem and load and execute that ME program. Then download (read back to host mem) the result MV buffer - set it as level1 predictor result and perform 'SAD check' only SearchMVs processing in PlaneofBlocks once for level0 plane to get SADs of the MVs do not use any more predictors or refining (close or equal to PredictorType=3 already in new MAnalyse). And return result to MDegrain().

The 'second generation' is faster but more advanced - try to perform SAD check inside DirectX-GPU and load back to host memory 2 arrays - MVs and SADs and interleave to feed back to MDegrainN.

Third-generation - perform MDegrainN inside GPU (possibly no sense because not enough GPU memory typical onboard for large enough tr, though may work for MDegrain1,2,3 (6?) and not large frame sizes).

In 'scalar' SAD function version the PT=3 PseudoEPZ_search is very simple as

template<typename pixel_t>
void PlaneOfBlocks::PseudoEPZSearch_no_refine(WorkingArea& workarea) // no refine - only predictor check
{
typedef typename std::conditional < sizeof(pixel_t) == 1, sad_t, bigsad_t >::type safe_sad_t;

sad_t sad;

if (smallestPlane) // never get here - normal use is sequence of params with 'real' search like optPredictorsType="3,x" where x < 3.
{
workarea.bestMV = zeroMV;
workarea.nMinCost = verybigSAD + 1;
}
else
{
workarea.bestMV = workarea.predictor; // already ClipMV() processed in the search_mv_slice
// only recalculate sad for interpolated predictor to be compatible with old/typical thSAD setting in MDegrain
sad_t sad = LumaSAD<pixel_t>(workarea, GetRefBlock(workarea, workarea.bestMV.x, workarea.bestMV.y));
sad_t saduv = (chroma) ? ScaleSadChroma(SADCHROMA(workarea.pSrc[1], nSrcPitch[1], GetRefBlockU(workarea, workarea.bestMV.x, workarea.bestMV.y), nRefPitch[1])
+ SADCHROMA(workarea.pSrc[2], nSrcPitch[2], GetRefBlockV(workarea, workarea.bestMV.x, workarea.bestMV.y), nRefPitch[2]), effective_chromaSADscale) : 0;
workarea.bestMV.sad = sad + saduv;
}

// we store the result
vectors[workarea.blkIdx].x = workarea.bestMV.x;
vectors[workarea.blkIdx].y = workarea.bestMV.y;
vectors[workarea.blkIdx].sad = workarea.bestMV.sad;

workarea.planeSAD += workarea.bestMV.sad; // for debug, plus fixme outer planeSAD is not used
}

For 'multi-block' SIMD processing (faster) it is a bit more complex - first got length of coherency in vector stream and call the appropriate multi-block SAD function from the list of available for current CPU SIMD architecture or pass a mask of coherent blocks for each vector offset value.

tormento

11th December 2021, 19:45

possibly no sense because not enough GPU memory typical onboard for large enough tr, though may work for MDegrain1,2,3 (6?) and not large frame sizes)
When resizable bar will be widespread enough, cpu or gpu memory will have less or no impact at all, given DDR5 bandwidth and lower latencies on transfer between the two kinds.

DTL

11th December 2021, 23:32

Well - the better for speed is 'gen 1.5' way: Do not calculate SAD in MAnalyse and output only MVs to re-writtten version of MDegrainN. Because it is faster to check SAD in MDegrainN at time of weight calculating. It will reduce read traffic from memory to about 1/2 and increase speed to about 2x.

But special 'compatibility' mode in MAnalyse with standard mv output (mv + sad) still may be required for other functions using MAnalyse output.

The cheapest possible Maxwell-based card is looks GTX 745 - low profile OEM card with slow memory in compare with GTX 750 and lower performance. I start to bargaining it secondhand from about $33 here. And may be will test for ME feature support next week - one seller provide testing capability before purchasing.

Though it is an idea to test 'remote debugging mode' - if it is not too dangerous to install 'remote debugger' at system with GTX 1060 it will be very helpful. Still not have experience with remote debugging of applications via IP.

tormento

12th December 2021, 09:28

And may be will test for ME feature support next week - one seller provide testing capability before purchasing.
If memory helps me, it has CUDA too. I have heard that the CUDA toolkit converts almost flawlessly from C++ to CUDA instruction. Perhaps you could give a look at it too.

pinterf

13th December 2021, 19:28

If memory helps me, it has CUDA too. I have heard that the CUDA toolkit converts almost flawlessly from C++ to CUDA instruction. Perhaps you could give a look at it too.

Like here in Nekopanda's CUDA Mvtools light
https://github.com/pinterf/AviSynthCUDAFilters/blob/master/KTGMC/MV.cpp
And the code actually in CUDA
https://github.com/pinterf/AviSynthCUDAFilters/blob/master/KTGMC/MVKernel.cu
with MDegrainN
https://github.com/pinterf/AviSynthCUDAFilters/blob/master/KTGMC/MVKernel.cu#L2600

DTL

13th December 2021, 19:37

I see the remote debugging is simple enough and not required to setup awful software like visual studio on remote system. So will try to start adding hardware dx12-me to MAnalyse this week at work.

Anyway even with infinite speed me engine it is required to re-write MDegrainN from scattering received mvs array to supplemental Fake* structure because it uses almost same time as MDegrainN processing at Core2 Duo E7500. Though the first version will mostly probably uses standard old MDegrainN.

It may be inheritance from old past times when MDegrain was interconnected with motion search in one filter and the later developers simply put the received mvs via AVS file to Fake structure to keep working MDegrain without rewriting. And with typically very slow MAnalyse the speed penalty was not very visible. But nowdays with fast MAnalyse modes and hardware-accelerated MAnalyse it really visible speedlimit (in the MVClip::Update()) and need to be removed. MDegrainN processing should access the received MVs array directly.

DTL

15th December 2021, 14:53

With multi-PCIe cards setup it is possible to create 'degraining farm' at one host and it may be much cheaper in compare with multi-workers distributed processing with CPU-based workers and IP interconnection.

I see used cheapest Maxwell-based 745 cards from about $50. Though it require some multi-PCIe (x16 ?) motherboard or may be available special 'PCIe-risers' to connect 16x card to 1x slot at slower speed ?

The MDegrainN motion search task is naturally paralleled to 2_x_tr number of src+ref searches so simple tr=12 degraining process can load up to 24 DX12-ME workers (NVIDIA promises to execute >1 ME task per chip at some chips).

The MVs data usage in MDegrainN is already fixed in some way and should be less limiting the speed I hope.

It also interesting question - what is the max vector length search in NVIDIA ME engine ? Do it have any limits ? Currently API do not have any settings about it. Though for typical degraining work the very short vectors like +-several samples are typically enough because it work on slow motion areas and with high speed motion noise is less visible and typically no large high speed areas in the motion pictures content.

ReinerSchweinlin

15th December 2021, 15:09

Maybe helpful addition:

While "Turing" refers to the GPU Generation, the included Encoders are not necessarely the same.

A 1650 has the VOLTA Encoders, not being able to produce B-frames with h265.

The 1660 has the b-frame HEVC capability, but only with nvencc is able to produce b-frames, the Nvidia provided encoder (used in A´s Video converter for example) is NOT using b-frames.

ffmpeg with hw-acc encoding also does not support b-frames at the moment.

all cards having "RTX" in theyr name work fine with b-frames HEVC encoding.

some of the quadro and OEM Cards also are missing the newer encoder (while being labeled turing or pascal)...

DTL

17th December 2021, 15:04

Development with remote debugging via LAN is started but not very fast because of my rare being at work with GTX1060 card and not very great being after long no sleeping time. Currently DX12-me init is partially finished in MAnalyse - this commit https://github.com/DTL2020/mvtools/commit/cbeb37ae747f11bcc3f5c92695c8404e1ea79769

If someone can try to provide remote debugging environment it may speedup the process. It require public IP connection for debugger and ftp access to the folder to upload dev builds of .dll and may be other required debug .dlls. It not require to install many to system - only unpack archive to some folder with known path from root of disk letter and run remote debugger monitor application. The debug session is protected with login/password of local system user with 'remote debugging' access.

It is not recommended by Microsoft to debug via internet because it may be slow and packet loss and more but it is possibly working solution.

Still do not know if GTX745 card support DX12-ME and not get it to my living place cheap enough. Unfortunately it looks no software emulation of this processing in DX12 available. Not search it very hard though.

tormento

18th December 2021, 12:40

If someone can try to provide remote debugging environment it may speedup the process.
Perhaps you can try to contact videoh. He has lot of experience with nvidia cards and CUDA.

DTL

18th December 2021, 13:41

Sad news - tested GTX 745 card at Win10 19041 and November 2021 driver and it do not provide HW-ME function. So it is again not known which cheapest card (from Maxwell ?) can provide required service. May be need to e-mail NVIDIA support about this issue.

Hehe - the article about hardware ME acceleration for external software clients https://developer.nvidia.com/blog/an-introduction-to-the-nvidia-optical-flow-sdk/ is dated Feb 2019 - the 2+ years passed already. Have post a question to nvidia developers support forums about selecting DX12-ME capable card.

DTL

22nd December 2021, 18:00

Possibly more sad news - the GTX750 was tested and looks also not support. Though it is listed as second-generation of Maxwell.

I mostly finished DX12_ME interface part of program. But still need many help of Microsoft developers to get it working. Or may be any working sample of DX12 video encoder (or motion estimator). The current program returns no-error HR-state of most of functions but final mapping of the resulted resource buffer to read MVs in the plugin return 'general error - D3D device disconnected' and error reason 'application have errors and need debug'. Very unclear where may be error. Also the resource state to motionestimator logical device still can not be switched to required state 'motion estimator read source' after writing source and ref data.

DTL

27th December 2021, 16:21

With enabling debug layer in DX12 the error messages much more better. Current progress: Possibly close to real processing tech sample for speed test. Still required conversion of input to NV12 format and correct reading of output. Also currently somewhere inside GPU is huge memory leak so it fill 3 GB memory after 500 FullHD frames. Need to found how to free resources. The ME performance in the GTX1060 looks comparable to i5-9600K or may be 2..3 times better. The number of concurrent ME tasks looks like non-limited but may be performance will be degraded after 100% Encoder loading.

Need some thinking how to balance ME parallel tasks per each MDegrainN thread. Currently each MDegrainN thread scan src-ref pairs with MAnalyse one by one (total 2_x_tr pairs per each output frame) and it looks required thinking about Avisynth interfacing with MDegrainN and MAnalyse to allow asking for several src-ref pairs in parallel for load balancing between MDegrainN CPU processing and DX12-ME hardware acceleration. Without balancing currently hardware accelerator is not fully loaded.

May be the only easy solution to use several 'super' clips in one MDegrainN and asks for ME search in several MAnalyse objects in parallel ? Unfortunately current interfacing between MDegrain and MAnalyse via Avisynth layer looks like limit some easy ways of ME-multithreading.

DTL

29th December 2021, 13:00

Some more info on NVIDIA: https://www.nvidia.com/en-us/geforce/guides/broadcasting-guide/
One thing that is great about NVENC on the GeForce RTX 20 and 30-series and GeForce GTX 1650 Super and up is that all GPUs have the same NVENC with the same performance and quality, from the RTX 2060 to the RTX 3090.
NVENC can do up to 8K30, so the only way to overload it is to do 2x4K60 streams.

It is not clear how many current+ref pairs do it process for ME in MPEG encoding for each output frame (possibly from 1 to several) but even with 1 pair at 8K 30fps it mean about 16x better at fullHD resolution. I.e. 480 pairs per second.
MAnalyse/MDegrain need to process 2_x_tr pairs per each output frame.

ReinerSchweinlin

30th December 2021, 11:53

.....
One thing that is great about NVENC on the GeForce RTX 20 and 30-series and GeForce GTX 1650 Super and up is that all GPUs have the same NVENC with the same performance and quality, from the RTX 2060 to the RTX 3090....

not quite

http://forum.doom9.org/showthread.php?p=1959335#post1959335

DTL

30th December 2021, 16:01

It only about b-frames for MPEG. I hope performance of ME-engine of NVENC not degrades greatly at low cost chips. It is an idea using current sources to create small tech test of RAW ME performance of current hardware. With output result in pairs of processed frames per second. Currently I do not have an ideas how to run MAnalyse in solo processing mode without MDegrain as frames-request engine and in 'multi=true' mode. And MDegrain even in overlap=0 mode also a significant part of slow processing now.
With MShow() as data-sink it possibly will request only 1 pair or src+ref frames.

And script with MAnalyse output returns an error like 'no video clip created for output'.

pinterf

30th December 2021, 16:15

DTL

30th December 2021, 16:24

It may be limits of old (or current) manufacturer-only API. In current Microsoft Windows API the hardware motion estimator looks do not have limitations and also data movement to and from ME-engine in the NVENC is performed via DX12 resources. Starting from creating D3D12-device.

Or may be NVIDIA have several generations (it mention 'old ME' and 'new Optical Flow') of ME modes and some (old ?) can work via DirectX12 without limitations. But may be lower in quality ?

Anyway the ME via DX12 works via DX12 API (Windows SDK) - not via NVIDIA SDK. And described as part of Windows Media Foundation API. So it should be NVIDIA-independent (as promised to have HWAcc for DX12 from Intel and AMD too). It is good advantage from NVIDIA-SDK only - not connected to one HW manufacturer only. It promised to be much more standard and long-live support in compare with one HW manufacturer only.
I hope in current latest windows the all old DirectShow API is perfectly working. It demonstrates the support for decades.

May be working with ME via NVIDIA SDK provide more settings/modes/params. Currently DX12-ME only support NV12 format as input for example.

DTL

31st December 2021, 14:25

Finally some working testbuild of MVtools with DX12_ME search mode for MAnalyse.

https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-dx12_me.a01

Enabling DX12_ME search: optSearchOption=5 (and levels=1). Using levels > 1 or default will possibly overwrite saved vectors with interpolated prediction from level 1. It copy MVs from DX12_ME output to vectors structure of the plane 0 and perform SAD calculation for MDegrainN weighting processing. Chroma looks like supported in both DX12_ME search and SAD. Currently looks like pel=1 only. The pel 2 and 4 is possible but in the future builds.

Only supported block sizes 8x8 and 16x16. Windows 10 build 19041 or newer looks like minimum requirement. When it uses DX12_ME hardware - the resource monitor TaskManager->GPU->Video Encoder shows the load graph and %.

Test script is about:

LoadPlugin("mvtools2.dll")
LoadPlugin("ffms2.dll")

FFMpegSource2("src.mxf")

ConvertToYV12()

tr = 12 # Temporal radius
super = MSuper (mt=false, chroma=true,pel=1, hpad=8, vpad=8)
multi_vec = MAnalyse (super, multi=true, blksize=8, delta=tr, chroma=false, overlap=0, mt=false, optSearchOption=5, levels=1)
MDegrainN (super, multi_vec, tr, thSAD=300, thSAD2=290, mt=false, wpow=4)

Prefetch(6) # set prefetch number to number of host CPU cores

Only hope completely compatible with MDegrainN (may crash other MDegrains and other with outside frame borders vectors sometime returned by DX12_ME). Need to find why they pass the ClipMV() limiting in MAnalyse.
So the performance of MDegrainN blenging engine is a bit limited in this build with additional clipping of block pos inside valid region.
Input format currently limited to YV12 only but may be relaxed to all Mvtools inputs because currently frames are read from loaded structures of MAnalyse.
Frame size tested - 1920x1080. Unfortunately inverlaced with separated fields (to 1920x540 frame size) still return some error from DX subsystem - need to fix in the later builds.

The speed at i5-9600K and with GTX1060 is a bit faster in compare with official 2.7.45 build and a bit slower in compare with SO=2. It looks overlapped modes of MDegrainN is supported (run but not test for quality).

May be better speed will be with SAD calculation also inside HWAcc with DirectX resoures (textures) processing. It will completely avoid the memory read into MAnalyse SAD calculation.

tormento

7th January 2022, 01:09

Finally some working testbuild of MVtools with DX12_ME search mode for MAnalyse.
Tried and not working for me. Nvidia 1060 3GB.

Access violation from AVS.

DTL

7th January 2022, 20:28

What was the script, frame size, frame format and error message ?

Currently found V-size limitation: It looks required integer number of blocks so block 8x8 work with 1080 height (1080/8=135) frame and not work with SeparatedFields to 540 height (540/8=67.5). Solution - pad vertical size with AddBorders() For example to process interlaced 1080 - pad to 1088. 1088/2/8=68.

tormento

7th January 2022, 23:50

What was the script, frame size, frame format and error message?
Really simple SMDegrain call:

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\3_00 wolf of Wall Street, The\wolf.dgi",ct=140,cb=140,cl=0,cr=0)
ConvertBits(16)
SMDegrain (tr=3, thSAD=300, refinemotion=true, contrasharp=false, PreFilter=5, plane=4, chroma=true)
fmtc_bitdepth (bits=8,dmode=8)
Prefetch(6) (tried without Prefetch too)

1920*1080 cropped to 1920*800 by DGSource.

DTL

8th January 2022, 12:12

1. MAnalyse with DX12_ME only support 8 bit input (and currently only YV12 format). Not sure if SMDegrain convert internally to 8 for MAnalyse or not.
2. To activate DX12_ME search mode in that build you need to pass option optSearchOption=5 to MAnalyse (and set levels =1 to not overwrite DX12_ME result with level1 search data in the level0 MVs array).

If you use SMDegrain without new options passing - again special build required with hardcoded options SO=5 and levels=1 .

Because speed of ME engine in GTX1060 is not many times faster in compare with i5-9600K currently trying to make Compute Shader SAD search to skip SearchMVs from GroupofPlanes completely (it saves 1 set of memory read operations) and will create fully compatible result of MAnalyse with all other clients filters of mvtools. Also have an idea how to make MDegrain HWAcc based without DX12 resources management in AVS core - to make all required reqources allocation in MAnalyse and pass pointers to resources via existing way of mvtools inbetween filters - via pseudo-audio stream. So in the best future all MAnalyse+MDegrain processing may be created with loaded to HWAcc memory frames once.

Also in current sources already implemented idea of half-sized ME for better speed: The DX12_ME engine currently can only work with qpel precision that is too large for typical pel=1 fastest processing. But with block size 8x8 it allow to make search with half-sized data (level1 from MSuper) and scatter received MVs to 16x16 block sized level0 with half precision truncating (from qpel to half pel). It currently controlled as optSearchOption=6. And now the speed limiting is SAD calculation in MAnalyse - so the next step to compute-shader processing in HWAcc required to get SAD values too.

magnetite

9th January 2022, 07:24

I tried this out on my GTX 1080 Ti and initially it threw an error (0xC000374). So I lowered the temporal radius to 2 instead of 12, and it was able to run. Performance was around 150 FPS.

While using the CPU only, performance was around 125 FPS on my i7 6700K.

Source was a DVD.

ChaosKing

9th January 2022, 13:20

I also made some quick tests:

Source ntsc DVD
CPU Ryzen 3600, GPU 3070 TI

Test script from here https://forum.doom9.org/showpost.php?p=1960271&postcount=33

But without wpow=4 (mvtools comlained, so I removed it)
+ removed Searchoption in non DX test

original mvtools
without prefetch: 28 fps
with prefetch(6): 130 fps
with prefetch(8): 155 fps
with prefetch(12): 177 fps

DX12 mvtools
without prefetch: 13 fps
with prefetch(6): 86 fps
with prefetch(8): 94 fps
with prefetch(12): 97 fps

DTL

9th January 2022, 13:41

" an error (0xC000374)"

This error code looks like something about heap corruption. Still do not have ideas where it can come from. Most of DirectX API calls have checks of HR-return value and if non-S_OK result - writes error message about the tried API call. It is possible to enable DirectX Debug layer with better error messages what is detected to going wrong but need to found way how to pick messages from something like 'debug output stream' in VisualStudio and add to Avisynth environment error output. So it may be special 'DX debug' builds of the plugin with possibly better error-collection from users.

"lowered the temporal radius to 2 instead of 12, and it was able to run"

Is it the max possible value without throwing error or just some low working ?

"DX12 mvtools
without prefetch: 13 fps"

It is with optSearchOption=5 ? The other modes may be somehow slower in compare with 2.7.45 in MDegrain because of added 'anti-bug' check of invalid vectors in that testbuild. In latest sources it looks I found where bad vectors may be passed and this check removed.

"ntsc DVD
CPU Ryzen 3600"

It have 32 MB L3 cache so at small frame size (and low tr) it may be faster to process on CPU and not send data for motion search to HWAccelerator and readback. For that config and frame size the HW processing may be faster only after transferring 'all' processing to DX12 pipeline - both motion search and blocks blending in MDegrain. It still some unknown future.

"with prefetch(12): 177 fps"

In that testbuild fastest working 'onCPU' optSearchOption should be 2. It should be faster in compare with v.2.7.45. Unfortunately SO=3 still not finished in debugging and works only at static colorbars mostly as tech demo of multi-blocks MAnalyse processing.

ChaosKing

9th January 2022, 14:08

" an error (0xC000374)"

It is with optSearchOption=5 ?

Yes all DX mvtools runs were tested with optSearchOption=5, for non DX I just removed it bcs it does not know the optSearchOption parameter. => I tested "pinterf mvtools" vs "DX mvtools"

EDIT:

with your mvtools build and optSearchOption=2
I get:
no prefetch() 45 fps
prefetch(6) 220 fps
prefetch(8) 248 fps
prefetch(12) 275 fps

That's quite the improvement! :eek:

DTL

9th January 2022, 14:47

"275 fps "

275/177=1.55. Not very great. For a bit faster processing you can also set optPredictorType=1 - though it may lower quality a bit. Default is optPredictorType=0 - should be most close quality to v.2.7.45. Raising PredictorType to 2,3,4 may degrade quality even more - experimental modes. Fastest is optPredictorType=4 (require adjustment of thSAD in MDegrain to lower values).

I expect from moving to HWAcc about 5x better speed. At least for intel-based CPU hosts with small enough cache sizes. The MVtools onCPU is limited mostly by host memory speed at fast CPUs like end-of-201x. Host memory speed is about 50 GB/s nowdays (about 2 channels of DDR4) and at top HWAccelerators reach about 1 TB/s - about 10..20 times faster.

tormento

10th January 2022, 12:41

That's quite the improvement! :eek:
I can't understand why there is so little sharing of this thread inside doom9 community. MVTools are one of the slowest and most used filter around and any help would be useful.

DTL is the only one developing this branch with almost no help at all.

DTL

10th January 2022, 13:17

It looks community is almost absent nowdays. And development process going to more difficult state: today I make about finished compute shader pipeline for SAD calculation with compute shader at HWAcc and found that remote debugging looks like do not support shader debugging. So the development of shader-based processing looks will be more slow with only way to download ready result from shader and analyse it.
Most of freeware developers were active about decade ago.

Some working tech demo of completely DX12-based (ME motion search + compute shader for SAD calculation) MAnalyse - https://drive.google.com/file/d/1cNsZvjVPEFdHi414-MJH24wwvn0s_x-0/view?usp=sharing . Based on the commit - https://github.com/DTL2020/mvtools/commit/1bfc70162c11b058e6ce5bee6047cba16727e838

It outputs 'standard' MVs pseudo-file (with SAD values) so should be compatible with other clients filters.

Only SO=5 (SO=6 still not finished in this build). Only overlap=0, chroma is used in ME but shader still not finished for addition of chroma SAD. Only 8x8 block size.

It works slower in compare with i5-9600K in RAW mvtools performance (with SO=2) but with x264 encoding it works faster because using HWAcc for MAnalyse saves some (about 30%) of CPU time for MPEG encoding.

Based on the testing of GTX1060 ME-engine performance: It looks RAW performance is about 700..800 pairs of ref+current frames per second with 1080p frame format and qpel precision. So it can significantly outperform i5-9600K only in half-frame size mode that limit block size to 16x16 only. Though max possible transfer of denoising processing to HWAcc will mostly free host CPU for MPEG encoding so total transcoding with denoising will be about twice faster (or limited to current settings x264 encoder execution speed onCPU).

DTL

11th January 2022, 13:59

The usage of natural massive-parallel processing on HWAcc makes things much easier:

The SAD compute shader for HWAcc is as small and easy as

RWTexture2D<int>OutputTexture : register(u0);
Texture2D<int>CurrentTexture_Y : register(t0);
Texture2D<int2>CurrentTexture_UV : register(t1);
Texture2D<int>ReferenceTexture_Y : register(t2);
Texture2D<int2>ReferenceTexture_UV : register(t3);
Texture2D<int2>ResolvedMVsTexture : register(t4);

void main(uint3 DTid : SV_DispatchThreadID)
{
int3 i3Coord;

i3Coord.x = DTid.x;
i3Coord.y = DTid.y;
i3Coord.z = 0;

int iBlockSize = 8;

int2 i2MV = ResolvedMVsTexture.Load(i3Coord);
i2MV.r = i2MV.r >> 2; // full frame search qpel/4
i2MV.g = i2MV.g >> 2;

int iYsrc;
int iYref;

int iSAD = 0;

for (int x = 0; x < iBlockSize; x++)
{
for (int y = 0; y < iBlockSize; y++)
{
i3Coord.x = DTid.x * iBlockSize + x;
i3Coord.y = DTid.y * iBlockSize + y;
iYsrc = CurrentTexture_Y.Load(i3Coord).r;

i3Coord.x = DTid.x * iBlockSize + x + i2MV.r;
i3Coord.y = DTid.y * iBlockSize + y + i2MV.g;

iYref = ReferenceTexture_Y.Load(i3Coord).r;

iSAD += abs(iYsrc - iYref);
}
}

OutputTexture[DTid.xy] = iSAD;
}

(still no chroma but add chroma is also simple in the future). And it is executed in the prof-programmers designed massive parallel architecture and software environment of HWAcc. No large hand-made parallel/SIMD program required. The execution speed is very nice. I see almost 0 load of GPU graph.

It is also not suffers from 'out of buffer' memory exceptions with 'invalid' vectors produced sometime by ME-engine. The sampler simply return zeros for out of frame coordinates.

The last core required MDegrainN compute shader is close to this.

And C++ multi-blocks SAD computing on general-purpose CPU with SIMD co-processor is about thousands lines of program and still greatly suffers from low memory speed.

Most of AVS filters may be put to compute shaders and greatly add to performance. But it require addition of DX-resources management in AVS core. Currently it will be only inside 'mvtools-environment'.

To help users it may be designed some helper-filter-pack like 'toDX12' to upload AVS pipeline to HWAcc memory and generate pointers to resources and 'fromDX12' to download back to host CPU memory and to continue AVS pipeline processing. And ability to compile and load for execution compute shaders for processing. It may be performed at user-side at filter load before execution. No special development environment required (though some debugging will be useful).

ChaosKing

11th January 2022, 14:32

Maybe you can also get some inspiration from a "cudaSynth" concept from here http://rationalqm.us/board/viewtopic.php?f=14&t=671

DTL

11th January 2022, 15:15

I think CUDA is more manufacturer-oriented API and less universal in compare with Windows DX-API. I expect support of DX-API will be not only from NVIDIA but intel and AMD (and some more HWAcc developers). So using natural for Windows API is easier in support and may be development I hope.

It is game-developers API that drives the market of HWAcc goods (for end-users/home entertainment use). CUDA market may be more limited and more for pro-usage (not for poor-people home PCs).

Unfortunately it looks intel with its QSV hardware accelerator may be too slow in driver development for DX12ME or may be old QSV hardware do not support required output by design. But intel with host-CPU built-in accelerator is also band-limited with very poor host RAM performance in compare with separate good accelerator board.

The too outdated intel-architecture of end-users PCs is now divided into separate high-performance islands - inside CPU caches and inside HWAcc board fast RAM. The host RAM is large but service only as very slow cache of data (for HDD and SSD).

DTL

14th January 2022, 13:32

Had a look at NVidia site;
https://docs.nvidia.com/video-technologies/video-codec-sdk/nvenc-video-encoder-api-prog-guide/#motion-estimation-only-mode
DX12 and NvEnc direct ME mode cannot work at the same time.

If it is about "Motion-estimation (ME) only mode is not supported if DirectX 12 device is used."

It may mean if DX12 device is used in application (? or the whole system ?) - it takes resources of ME engine to use for DX12-ME clients so it can not be shared with NVIDIA-SDK based applications at the same time.

Current sources shows how DX12-ME server runs with same DX12 device that provide Compute Shader execution (that is enough for other tasks of MAnalyse + MDegrain processing). And CS execution works with same resources in the same format as used by ME-engine - it saves from additional converting/copying/uploading. The loaded for ME processing resources after resolving MVs simply assigned as Shader Resource Views inputs to CS.

Also " For full-pel precision, the client must ignore two LSBs of the motion vector. For sub-pel precision, the two LSBs of the motion vector represent fractional part of the motion vector."

Possibly mean even with NVIDIA-API it is not possible to switch ME engine from qpel to full-pel search mode for better speed. It always works in qpel mode and user must discard 1 or 2 LSBs to get full pel MV value. But it not adds to speed.

I currently think if ME-engine from NVENC is not very great in speed and left most of HWAcc resources free it may be possible to make 'standard' MAnalyse processing on Compute Shaders because at each level blocks processing is mostly independent (only interconnected with FetchPredictors -> getting some surrounding processed blocks MVs as predictors) but it may possibly can also somehow added using ordered (?) processing inside group of threads. Or that additional predictors gathering may be skipped for more or less lower quality. It is now something inbetween optPredictorsType=1 and 2 modes - the usage of zero, interpolated and global predictors is possible with full independent blocks processing.

So users of 'complex' mvtools-based scripts may use different MAnalyse calls with different hardware execution units and get more speed.

DTL

20th January 2022, 23:19

New build - https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.06

Some faster in resource uploading and finally make chroma in SAD calculation. With chroma=true it finally visibly faster at GTX1060 in compare with CPU (i5-9600K) processing because optSearchOption=2 (and higher CPU-based) do not have chroma SAD processing now.
Also scaleCSAD is working too and significantly changes output MPEG speed. (Setting to -1 makes MPEG speed lower - either with lowering noise or softening too, -2 not tested yet) The adjusting steps looks even too coarse as integer 0,-1,-2 and may be good to add some finer (trying to keep compatibility - may be use float ?). The speed with executing at accelerator should not be visibly smaller even with fine float adjustment with float multiplication and converting to integer.

It looks scaleCSAD should be adjusted (fine tuned) with thSAD values in MDegrain (and other like scene change and other th-s).

At some quick tests it looks with FullHD frame and 8x8 block size there exist some issue with lowest 3 rows of blocks - may cause visible error blends with 'large' tr like 25. To fix if happens (temporarily ?) - make thSCD1 lower default 400 (like 350..320). It may be some hidden bug in data transfer somewhere - like cut-out some buffer too early. Though it appear rare enough near some scene changes only.

tormento

21st January 2022, 14:01

New build
Will try ASAP.

Is HBD a long way to go?

DTL

21st January 2022, 15:16

"Is HBD a long way to go?"

It not supported by DX12_ME for ME work. Only NV12 is the only supported input format currently for ME engine. It is for bitdepth equal to YV12 8bit format (difference is only in interleaving UV planes). So is the current requirement for SO=5 mode of MAnalyse - only YV12 input. It is currently converted to NV12 at the upload procedure - Y plane used directly as first subresource slice and interleaved UV as second subresource slice. In theory it is possible to upload more bits formats and convert to NV12 by compute shader but it is not first place work. Better use ConvertBits(8) or ConvertToYV12() for the MAnalyse source. Though using 8bit only may limit quality of processing HDR sources (at dark areas) - so it may be recommended to convert HDR to SDR 8bit before feeding MAnalyse.

To have some 'more accelerated' HBD - either required to design compute-shader based MAnalyse (SearchMVs() function of PlaneofBlocks) or MDegrain. It is all planned but in some future. I hope compute-shader based MAnalyse will be faster in compare with the dedicated ME engine from encoder. Currently 'GPU load' system stats shows almost 0 load (even with using SAD compute shader). So it is still free compute resources exist.

The only way to increase speed of ME engine of encoder is possibly overclock accelerator (or measure its speed in different chips).
May be chips with more 'encoder units' will have also ability to run more ME threads at the same time. Currently DX12 environment looks like completely not limits number of ME threads (for same process or different processes) so it is hard to find what is the actual limit of performance of current hardware. One possible method is to run MAnalyse + MDegrain with AVSmeter and get close to 100% Video Encode load and check for output fps and tr-value. The performance of ME engine is about fps*(2*tr) pairs of frames per second.

The MDegrainN support HBD input/output already for onCPU processing. So it looks like required to create 2 'super' clips for using in MAnalyse (8bit) and in MDegrain (>8bit).

The addition of compute-shader based MDegrain is in progress. It started from uploading all required resources (2 x tr + 1 number of frames) in HWAcc. It also will make speed of MAnalyse best possible (only 1 new upload frame per 1 output frame of MDegrain). Currently MAnalyse still use partial optimization - 1 src frame is remembered and not uploaded for all src+ref pairs requested by MDegrain for each output frame. So with full optimization the upload traffic will be (2 x tr) times smaller (also leave more host memory speed for other tasks). Currently even making Y-plane not copy in NV12 intermediate structure before upload makes visible speedup.

The expected limiting factor for MT Avisynth - too low onboard memory of accelerator to hold total frames pool of all MAnalyse+MDegrain threads. With current architecture the frames pool can not be shared across different AVS threads. So it possibly will not allow to run script with all possible CPU cores threads number (but it will leave more free cores to MPEG encoder). With shared ME engine for all threads it may be no need to run many threads to get max possible speed. So the Prefetch(N) should be adjusted between accelerator memory load and Video Encode load. If Video Encode is close to 100% load - no need to add more threads.

Current major limitation of usage of DX12_ME engine - it looks overlap processing mode (for MAnalyse) is not possible (keeping speed in good range). Some cosplay of overlap possible only by sending 4 shifted planes to process but it will make speed 4 times slower. So the only way to get less blocky output of MDegrain is new no-overlap blending mode (still work in progress) with smooth transition of blending weight between blocks. It may run more or less slower onCPU but should run good at compute shader based processing. Though with 'half-frame' SO=6 mode the frame size is 4 times smaller and possibly it will be processed 4 times faster by ME engine and allow to run some cosplay of overlap mode with good speed - need testing too. And SO=6 mode currently limits usable block size to 16x16 only.

magnetite

21st January 2022, 20:00

I kept getting this unhandled C++ exception error while trying to run the new version. However, running it with MeGUI's manual mode actually gave me a specific error message, instead of some generic unhandled C++ exception:

Avisynth script error:
Unhandled error: dimzon_avs_init_2

LoadPlugin("C:\MeGUI 64-bit\tools\lsmash\LSMASHSource.dll")
LSMASHVideoSource("E:\Mobile Video\MVI_0564.mp4")
ConvertToYV12()
tr = 2 # Temporal radius
super = MSuper (mt=true, chroma=true,pel=1, hpad=8, vpad=8)
multi_vec = MAnalyse (super, multi=true, blksize=8, delta=tr, chroma=true, overlap=0, mt=true, optSearchOption=5, levels=1)
MDegrainN (super, multi_vec, tr, thSAD=300, thSAD2=290, mt=true, wpow=4)
Prefetch(8)

Removing the optSearchOption=5 and levels=1 fixes it.

DTL

21st January 2022, 20:05

Is it keep to throw error in 1 thread mode ? Without or commented-out Prefetch(8) at the end of script.

I had some Unhandled C++ exception inside Avisynth when trying to use some Graphic lib from DX12TK in multithreaded mode but now it do not use it.

Also try to set mt=false for MSuper (and also for MAnalyse !)- may it somehow conflict with something (if you also have avstp.dll somewhere and avstp-mt is really activated). With SO=5 the processing in MAnalyse should not reach the avstp-mt slicing now but it still may somehow made difference.

As for MDegrainN avstp-mt - not sure how it work - I not use it and not tested. First it is also recommended to set mt=false in all mvtools filters.
Also if you use AVS+ mt with Prefetch(N) at the end - it mostly good to disable any avstp-mt in mvtools because it will makes things much more slow. You need to use either avstp-mt or AVS+ mt or precisely adjust threading of avstp-mt with AVS+ mt if avstp-mt is really faster in your case.

If avstp-mt is activated in your environment with that script it try to create 8+(num of cores ?) threads. avstp-mt in mvtools is now 'shadow of the poor past' and only may be recommended to use with care if AVS+ mt is not possible.

Also you may set levels=1 to
super = MSuper(chroma=true,pel=1, hpad=8, vpad=8, levels=1)

MAnalyse with SO=5 do not use >1 level now so it may make processing a bit faster (no need to generate lower res levels in MSuper). MDegrainN also do not use smaller levels.

magnetite

21st January 2022, 20:26

I don't use avstp. Just use Avisynth+'s MT modes. With MT completely disabled and no prefetech I get the same error. Have to wonder what Dimzon is.

DTL

21st January 2022, 20:51

"I don't use avstp."

But your script shows mt=true in every mvtools call. It is not required for AVS+ mt. It only switch on/off avstp-based internal mt.

"Have to wonder what Dimzon is."

Yes - it is strange error. 'dimzon' token is not found in AVS+ 3.7 sources. Can you run avsmeter from command line ? I test with avsmeter and virtualdub x64 (also encoding with x264 command line - x264 direct reading src.avs script file).

All files from archive is now required - the Compute.cso must be located in the same folder with mvtools2.dll. Its loading engine (some helper function from DX12 toolkit) mostly probably searches for file near mvtools2.dll in the same path only. It is compiled shader file. I do not know how to pack (link ?) in single executable resources if even possible. So all new versions with compute shaders will have a set of .cso files - some like SAD.cso, Mdegrain.cso and may be Msearch.cso and other. I even not sure if it good and possible to pack different compute shaders in the single file. In theory reader loads some bytestream from file so it possible to read some known parts start-end from single binary file but it is additional work and source of errors.

"Removing the optSearchOption=5"

It is disabling all DX12-based hardware accelerated processing and fallback to 'standard' mvtools 2.7.45 processing.

magnetite

21st January 2022, 21:16

Here's a link to the combined (https://i.imgur.com/KTYGIXb.png) error messages. The last DLL that worked for me was on 12/31/2021. The CSO file is in the same folder as the mvtools2 DLL.

DTL

21st January 2022, 21:39

"LSMASHVideoSource("E:\Mobile Video\MVI_0564.mp4")"

What is frame size ? Can you run with FullHD internal source like

ColorBarsHD(1920,1080) ?

Currently it looks the height must be integer divisible to block size so for FullHD progressive it is 1920x1080 and for interlaced - pad to 1920x1088 to get 1920x544 after SeparateFields() and so on.

"link to the combined error messages."

Evaluate: Unhandled C++ exception - it is AVS+ error message. I see you use still not 'final' 3.7.2 AVS+ build.

I see same error message at pinterf's github: https://github.com/pinterf/AviSynthPlus/issues/42 . May be not all more pointing to place of problem messages still displayed by Avisynth correctly ?

magnetite

21st January 2022, 23:07

The frame size of that video source was 1920x1080 progressive.

I still get the same error message with the official (https://github.com/AviSynth/AviSynthPlus/releases/tag/v3.7.1) Avisynth+ build from Github. Is that the final build you're referring to?

With ColorBarsHD as the source, same error message pops up.

DTL

21st January 2022, 23:31

"with the official Avisynth+ build from Github."

Yes - I also setup latest 'release' 3.7.1 at my the only working system with GTX1060. I think of creating debug build also it have enabled DirectX debug layer - it can be installed as part of Windows 10 user-side and may be will shows more detailed errors. It may be something around structured exceptions handling - may be it is disabled in release build so exception do not catched in the underlying layers and passed to AVS+ host.

Here is debug build - https://drive.google.com/file/d/1yENQ-_NN39jPdzo_bFrJJ5UWc__Q_o0U/view?usp=sharing . Also added exception handling around loading of Compute.cso file. It have builds with 2 types of C++ exception handling - EHa and EHsc in VS2019 settings - may it helps.

*updated link to archive with debug MS .dlls included*

takla

23rd January 2022, 15:00

DTL
I get the following error "MAnalyse: Can not load file Compute.cso ReadData"
Any way to manually specify the file path for it? I tried mvtools2_EHa.dll & mvtools2_EHsc.dll but both give me this error.

DTL

23rd January 2022, 15:32

Same was with magnetite user. Recommended: to put all files in the some very simple folder path like
c:\AVS\
(not something like c:\my_regional_letters_path\some folder Name\avisynth+\ )

Last message he report solve the problem but not exactly how.

Simple recommended setup:
put all files (mvtools2.dll, Depan.dll, Depanestimate.dll, Compute.cso, src.avs script, avsmeter64.exe) in path like
c:\AVS\
and run 'avsmeter64 src.avs' command from command line with current/working folder path c:\AVS (I use FAR file manager for 'navigate to some folder and run from it').

I will try to re-write the loading function of shader file in next builds - it looks C++ Microsoft-based sample file reading function have some issues with some paths at some user's Windows setups. There possible tons of different ways of load binary file data from file into RAM buf in C/C++/WinAPI etc.

May be loading issue is because of typically using path (for plugins) like C:\Avisynth+ with "+" sign in path name. I do not use this paths in my development windows so can not see error.

Currently used function is 'microsoft signed' :) https://github.com/microsoft/Xbox-ATG-Samples/blob/9654b6efa4e1db52365fc2ae80386e0e27b9658d/Kits/ATGTK/ReadData.h

Helper for loading binary data files from disk
//
// For Windows desktop apps, it looks for files in the same folder as the running EXE if
// it can't find them in the CWD

CWD is possibly Current Working Directory

May someone see some sources of error to fix in it ? The placement of EXE is looks like not an issue - I can load src.avs script in VirtualDub.exe located from very different path.

Simple try-catch structure looks like can not point to exact line where error happens:

// load compute shader
std::vector<uint8_t> computeShaderBlob;
try {
computeShaderBlob = DX::ReadData(L"Compute.cso");
}
catch (std::exception& e)
{
env->ThrowError(
"MAnalyse: Can not load file Compute.cso %s", e.what()
);
}

It looks e.what() return only 'ReadData' string.

magnetite

23rd January 2022, 16:04

Same was with magnetite user. Recommended: to put all files in the some very simple folder path like
c:\AVS\
(not something like c:\my_regional_letters_path\some folder Name\avisynth+\ )

Last message he report solve the problem but not exactly how.

I uninstalled Avisynth+ from C:\Avisynth+ and reinstalled it to C:\AVS as suggested, then put everything in C:\AVS\plugins64. I deleted the plugins64+ folder just to be safe. VRAM usage was around 4 GB.

DTL

23rd January 2022, 16:07

+ some note: as I later found that builds have 'reversed' scaleCSAD adjustment for DX12 processing (2 used as -2 for 'classic MAnalyse' and so on). The default 0 works as 0.
Missed the lines

// YV12 YV16 YV24
// nLogXRatioUV 1 1 0
// nLogYRatioUV 1 0 0
// effective_chromaSADscales: (shift right chromaSAD)
// chromaSADscale=0 -> 0 1 2 // default. YV12:no change. YV24: chroma SAD is divided by 4 (shift right 2)
// =1 -> -1 0 1 // YV12: shift right -1 (=left 1, =*2) YV24: divide by 2 (shift right 1)
// =2 -> -2 -1 0 // YV12: shift right -2 (=left 2, =*4) YV24: no change
effective_chromaSADscale = (2 - (nLogxRatioUV + nLogyRatioUV));
effective_chromaSADscale -= chromaSADscale; // user parameter to have larger magnitude for chroma SAD
// effective effective_chromaSADscale can be -2..2.
// when chromaSADscale is zero (default), effective_chromaSADscale is 0..2

of PlaneofBlocks.cpp in sending params data to shader.

nLogxRatioUV + nLogyRatioUV for YV12 is 1+1 so
effective_chromaSADscale -= chromaSADscale is
2 - (1+1) = 0 - chromaSADscale - that reverses adjustment.

Will be fixed in the next build.

takla

23rd January 2022, 17:42

DTL

23rd January 2022, 18:22

It can now work with chroma=true.

super = MSuper (pel=1, levels=1, chroma=true)
multi_vec = MAnalyse (super, multi=true, blksize=8, delta=tr, optSearchOption=5, overlap=0, levels=1, chroma=true)

The close 'software mode' is only

super = MSuper (pel=1, levels=0, chroma=true)
multi_vec = MAnalyse (super, multi=true, blksize=8, delta=tr, optSearchOption=1, overlap=0, levels=0, chroma=true)

takla

23rd January 2022, 20:57

DTL
Using the script posted in #64, optSearchOption=5 is about ~3 seconds slower then optSearchOption=1.

DTL

23rd January 2022, 21:11

"script posted in #64,"

Do not uses chroma processing in MAnalyse. That may make quality of motion search worse (and cause visible bugs in some cases - https://forum.doom9.org/showthread.php?p=1783361#post1783361 ) . So enabling chroma in MAnalyse generally increase quality (and recommended to use if speed is acceptable).

It may be interesting to check the speed lost of MAnalyse between

optSearchOption=1 chroma=true/false (onCPU only processing via host RAM)

and

optSearchOption=5 chroma=true/false (DX12_ME + CS via HWAcc RAM)

The ME hardware engine always uses chroma in search and enable/disable in MAnalyse settings only switch on/off UV planes copying from source to upload buffer and chroma SAD enable/disable compute and addition to output SAD in CS. So with optSearchOption=5 enabling chroma is close to free for speed. And with CPU processing enabling chroma in MAnalyse is typically very visible speed impact.

The second totally winner feature of HW ME - it uses always pel=4 search that is awfully slow onCPU. Though its full usage in mvtools still not implemented (may cause also very slow MDegrain onCPU). It will be used in SO=6 mode for increasing speed with blocksize=16. Using pel=4 precision may also makes degraining a bit better. It possibly can be tested with only optSearchOption=0 or may be 1 max.

"GPU usage is around 8%. Very low."

Is it Video Encode load % of 'GPU (total ?) %' ?

takla

23rd January 2022, 21:27

It is total GPU load. And I wanted to compare apples to apples, that is why I had chroma set to false for both gpu and cpu tests.
I'm not too worried about quality at this stage, only wanted to test speed. But if the speed penalty for chroma=true is low with gpu, I'll keep that in mind for the future, thanks.

Edit: Tested chroma=true with GPU & CPU, CPU is still faster.

DTL

24th January 2022, 13:30

"It is total GPU load."

I hope you use for speed-test release build from github ? Not debug build.

"CPU is still faster."

It looks you use AMD CPU with large cache. For intel-users with low L2/L3 cache the MAnalyse may be more slow.

"Tested chroma=true with GPU & CPU"

What is speed difference of chroma=false and chroma=true for MAnalyse with optSearchOption=5 ? And what GPU-card is used ? With chroma=true in MSuper (so it is enabled for MDegrain).

takla

24th January 2022, 18:12

I didn't think debug build would make a difference. My bad. Here are my new tests, again, with the settings used in post #64:

r.2.7.46-a.06
time=15.841s
MSuper & MAnalyse chroma=true
optSearchOption=5

r.2.7.46-a.06
time=15.554s
MSuper & MAnalyse chroma=false
optSearchOption=5

r.2.7.46-a.06
time=4.016s
MSuper & MAnalyse chroma=true
optSearchOption=1

r.2.7.46-a.06
time=3.576s
MSuper & MAnalyse chroma=false
optSearchOption=1

CPU: AMD Ryzen 9 3900x
GPU: AMD Radeon RX 5700

I didn't think the difference would be THAT much. Also, Video Codec usage under GPU in task manager showed 99% usage. So again, my bad on that.

CPU is almost 4 times faster.

DTL

24th January 2022, 20:18

"I didn't think debug build would make a difference."

It is full of software checks and very slow. Only provided to get possibly more pointing error message about that C++ exception issue at user-side.

" AMD Radeon RX 5700"

Thank you for information. It is really the first report of non-NVIDIA accelerator usage.

The good news that AMD also provides drivers for DX12_ME operation. And no need for any software tuning between AMD and NVIDIA products. It was expected. Good work for compatibility from Microsoft. The other news is that Ryzen 9 3900x is about 4x faster. Though I have not any info about expected speed of Video Encoder engine in AMD accelerators (if it even exist there). I think it will be some future products from Intel and AMD that will provide hardware features for DX12_ME operation.

Also as expected the speed difference between chroma true and false with accelerator is small enough - about 2%. With 'onCPU' processing is about 12% (still not very large - but it is AMD large-cache device or too small frame size for this cache size).

Update: Known issue and workaround: Lower up to 8 rows of blocks (with block size 8x8) may be somhow not correctly processed at some scenes. Current solution: pad bottom of 1920x1080 frame with 72 lines with AddBorders() and Crop() after processing.
Example:
Addborders(0,0,0,72)
...msuper, manalyse, mdegrain..
Crop(0,0,0,1080)

takla

25th January 2022, 18:42

Update: Known issue and workaround: Lower up to 8 rows of blocks (with block size 8x8) may be somhow not correctly processed at some scenes. Current solution: pad bottom of 1920x1080 frame with 72 lines with AddBorders() and Crop() after processing.
Example:
Addborders(0,0,0,72)
...msuper, manalyse, mdegrain..
Crop(0,0,0,1080)
Mhh that is messy. Can't you make mvtools do exactly that, but internally?

DTL

25th January 2022, 22:08

It is not long time solution I hope. Just a known way of bugfix *if* it occur. I even not sure where is it come from. Most of buffers copying-processing is totally static but issue happens rarely (may be 1% of frames or less) on some moving patterns near bottom of frame. Like water waves or some camera pan or objects move. May be it even issue of my NVIDIA GTX1060 NVENC (with current driver, etc).
Or it may be SAD compute shader threading dispatch (may not process last threads groups sometime ?) issue. But its Dispatch() call also equal for any frame so the issue should be visible in any frame like no denoising (?) currently if it cause not same blocks blending it may return small or close to zero SAD for several bottom rows of blocks ?

Comment from Microsoft support about used file loading function:

The DX::ReadData helper looks in the current working directory at runtime for the file.

When building for Win32 desktop, if it can't find it there, then it looks in the same folder as the running EXE. If you are a 'plugin' then likely that directory where you have your plug in installed.

Your best option is to use some full path to your "plugin" directory to find your data and not rely on CWD.

So it looks I need to look how 'avstp.dll' is searched for possible paths for more use cases of avisynth + plugins + scripts paths installation and running. It looks ReadData() can not found compute.cso file if mvtools.dll is copied in some path not equal to loaded script path ? I usually put all files in 1 folder (source file, all used plugins, source script) so do not see this issue.

DTL

30th January 2022, 18:03

New update: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.07

Now pel=4 have about usable speed. At i5-9600K + GTX1060 with x264 encoding of 1080i footage SO1 is about 3 times slower in compare with SO5 using pel=4. About 1.26 fps vs 3.76 fps. Pel=1 works with 6.46 fps SO=5.
And it is still onCPU SAD computation for pel 2 and pel 4. Next versions with onHWAcc SAD computation for pel 2 and pel 4 expected to be even faster.
The quality at some quick tested camera pan movements with pel=4 is a bit better in compare with pel=1.

Example of processing script:

tr=25
super=MSuper(last, mt=false, chroma=true, pel=4, hpad=8, vpad=8, levels=1)
multi_vec=MAnalyse (super, multi=true, blksize=8, delta=tr, overlap=0, chroma=true, optSearchOption=5, mt=false, levels=1, scaleCSAD=2)
MDegrainN(last,super, multi_vec, tr, thSAD=250, thSAD2=240, mt=false, wpow=4, thSCD1=350, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16)

MDegrainN new added params:
1. adjSADzeromv (1.0 - default, no op) - possible SAD multiplier for zero-move blocks (before thSAD processing for getting block's weighting value). Float value. Recommended values: 0.9..0.4-. Possible medium values 0.75..0.5. Allow to increase degraining at static areas.

Example: setting 0.5 result thSAD for zero move blocks (static) will be thSAD*2.

2. adjSADcohmv (1.0 - default, no op) - possible SAD multiplier for blocks in coherent moving areas (before thSAD processing). Float value. Recommended values: 0.9..0.4-. Possible medium values 0.75..0.5. Allow to increase degraining at big enough coherent moving areas of much larger 1 block_size size (like camera pan movement over non-changing scene).

3. thCohMV (-1 default, no op) - threshold to detect if block's move vector is equal to surround blocks (top,left,right, down) move vectors. -1 - disables this part of processing (faster), 0 - lowest working value. Recommended values 0..4. Possible range - 0..unlimited int. Too high values will create error-blended blocks (like with too high thSAD value).

The thCohMV is not auto-scaled with setting pel to 2 or 4 and as integer MVs length become 2 and 4 times larger it looks also required correction of thCohMV param about xPel. I do not think internal auto-correction is good because in integer param it will limit range of fine-tuning. Or the integer need to be chenged to float.

Also with pel 2 and pel 4 it looks good to add 1 more param like thZeroMV - threshold to detect if block is close to zero move. With pel > 1 currently even the small sub-pel movements is treated as non-static block. Currently large areas of 'static' blocks also may be adjusted with adjSADcohmv param that can be applied to blocks non-coherency movement (including deltas around zero) defined by thCohMV param.

takla

31st January 2022, 07:28

DTL what source filter do you use? I tried both, LWLibavVideoSource and AviSource but they both crash without error.
The only way I can test optSearchOption=5 is with ColorBars.

Anyways, here are some more benchmarks:

ColorBarsHD(1920, 1080)
ConvertToYV12()
Trim(0, 600)
tr = 3
super = MSuper (pel=1, levels=1, chroma=true)
multi_vec = MAnalyse (super, multi=true, blksize=8, delta=tr, optSearchOption=1, overlap=0, levels=1, chroma=true)
MDegrainN (super, multi_vec, tr, thSAD=150)
Prefetch(12)

r.2.7.46-a.07

ffmpeg -y -benchmark -i TEST.avs -c:v ffv1 TEST.mkv

optSearchOption=5

pel=1 chroma=false
time=14.972s

pel=1 chroma=true
time=15.236s

pel=2 chroma=true
time=15.710s

pel=4 chroma=true
time=15.910s

optSearchOption=1

pel=1 chroma=false
time=3.695s

pel=1 chroma=true
time=4.091s

pel=2 chroma=true
time=4.807s

pel=4 chroma=true
time=5.487s

DTL

31st January 2022, 12:00

I use FFMpegSource2() from ffms2.dll. The current builds very critical to V size of frame. Known working sizes 1920*1080 (progressive). 1920*1088 (interlaced for SeparateFields(), 1920*1152 (1080+72) to workaround possible low rows issues and interlaced frame. Still not search in deep what is limit size or how is workaround it inside. But with non-compatible size it throws error like 'error - commandlist-close' or other text error. It looks typical working height (for interlaced source) is integer divisible to 16.

I see you use tr=3 with low params MDegrainN - that cause fallback to MDegrain3 and not tested. It may not work because now MDegrainN uses different way of loading MVs from 'fake' structures. The default creation of 'Fake*' class is to use old scattering of MVs data into FakeBlock class. And will cause crash with MDegrain(X) trying to load from uninitialized memory.
Also SO=5 is not good compatible with MDegrain(X) because their block-procesing functions do not have additional MVs limiting of invalid vectors (sometime coming from hardware ME). With invalid (far outside frame area) MVs the MDegrain(X) will crash without good error message.
Currently only pel=2 and pel=4 with old MAnalyse SAD computation should be compatible with MDegrain(X). The MAnalyse do additional check and limiting of MVs before SAD computing. May be it is good idea to put this limiting into MAnalyse at MVs data coping function from readback buffer to MAnalyse output buffer.

To use MDegrainN with low (any) tr you need to set both thSAD and thSAD2 to different values like 150 and 150-1. To make editing easier it may be created script user-variable like my_thSAD and set it to MDegrainN as

my_thSAD=150
MDegrainN(thSAD=my_thSAD, thSAD2=my_thSAD-1)

Benchmarks shows your CPU is great with core speed and cache size and may be performance of DX12ME at accelerator is limited to 15-sec with tr=3. With 4k frame and larger tr when frameset with pel4 will not fit in CPU cache the onCPU speed may be lower.

takla

1st February 2022, 02:38

DTL Specifying a value for thSAD2 is what fixed it, thanks.

LWLibavVideoSource("C:\Users\Admin\Documents\01.mkv")
Trim(0, 600)
Addborders(0,0,0,72)
tr = 8
super = MSuper (pel=4, levels=1, chroma=true)
multi_vec = MAnalyse (super, multi=true, blksize=8, delta=tr, optSearchOption=5, overlap=0, levels=1, chroma=true)
MDegrainN (super, multi_vec, tr, thSAD=150, thSAD2=75)
Crop(0,0,0,1080)
Prefetch(12)

r.2.7.46-a.07

optSearchOption=5

pel=1 chroma=false
time=42.293s
time=41.430s
262.419 KB

pel=1 chroma=true
time=41.253s
time=41.285s
246.503 KB

pel=2 chroma=true
time=45.720s
time=44.713s
248.503 KB

pel=4 chroma=true
time=41.868s
time=44.675s
247.812 KB

optSearchOption=1

pel=1 chroma=false
time=8.963s
265.645 KB

pel=1 chroma=true
time=9.085s
249.453 KB

pel=2 chroma=true
time=14.969s
251.571 KB

pel=4 chroma=true
time=21.300s
248.394 KB

File size was just for me to check if settings were used properly.
optSearchOption=5 seems to sometimes fluctuate by 3 seconds.
optSearchOption=1 is still much faster. I'll test some 4K.

Edit: Here is 600 4K frames with TR=6:
optSearchOption=1 time=205.274s
optSearchOption=5 time=211.290s
Or about ~3FPS.

Boulder

1st February 2022, 12:35

I'd like to test the new version, but I'm having a hard time following the complex explanations.

I.e. could someone please simplify things a bit by telling me which settings would make the result as close to a vanilla MVTools2 one as possible, in order to make a fair comparison? My normal chain of operation is MSuper (separate for analysis and MDegrain) - MAnalyse - MRecalculate (possibly two times with halving blocksize with each iteration) - standard MDegrain.

DTL

1st February 2022, 13:53

"4K frames with TR=6:
optSearchOption=1 time=205.274s
optSearchOption=5 time=211.290s"

With TR=12 the speed difference may finally reverse to SO=5 is better.

I got an idea how to make at least MDegrainN onCPU better speed for pel >1 still not going into full onHWAcc processing. So onCPU times of MDegrainN part of processing will be lower.
It is about not reading sub-pel shifted copies of frame from memory but shift single block to sub-pel before averaging inside CPU register file or caches. At least for MDegrainN with 1 src block using only 1 ref shifted-position block it should make better speed. And with SO=5 the MVs creation is now serviced with dedicated accelerator. The idea come from the SAD calculation for pel >1 : I not like an idea to upload to accelerator 4 for pel=2 or 16 for pel=4 copies of the ref frames produced by MSuper(). It is slower with upload via bus and wastes fast but limited in size RAM of accelerator awfully. So I will re-make current SAD compute shader to make sub-pel shift of ref block from single full-size ref frame inside shader before SAD calcultaion. It is natural enough task for data compute accelerator.

" My normal chain of operation is MSuper (separate for analysis and MDegrain) - MAnalyse - MRecalculate (possibly two times with halving blocksize with each iteration) - standard MDegrain."

May be better to post your current script so when someone will have time - will tried to change its MAnalyse processing to SO=5 and look if it work / if it possible. Also the input frame size is required.

Boulder

1st February 2022, 15:38

" My normal chain of operation is MSuper (separate for analysis and MDegrain) - MAnalyse - MRecalculate (possibly two times with halving blocksize with each iteration) - standard MDegrain."

May be better to post your current script so when someone will have time - will tried to change its MAnalyse processing to SO=5 and look if it work / if it possible. Also the input frame size is required.

It's basically something like this. The last MRecalculate iteration is often skipped for 4K so it's just MAnalyse+MRecalculate before MDegrain.

source("whatever.dgi") # 4K or 1080p

blksize = 16 # or 32 for 4K
blksize2 = 8 # or 16 for 4K
blksize3 = 4 # or 8 for 4K
overlap = 8 # or 16 for 4K
overlap2 = 4 # or 8 for 4K
overlap3 = 2 # or 4 for 4K

prefilt = ex_minblur(r=1, uv=3)
superanalyse = prefilt.msuper(pel=4, sharp=1, rfilter=2)
supermdg = msuper(pel=4, levels=1, sharp=1, rfilter=2)

fv1 = manalyse(superanalyse, isb=false, delta=1, blksize=blksize, overlap=overlap, search=4, searchparam=2, pelsearch=2, truemotion=false, dct=5, scalecsad=2, trymany=false, global=true)
bv1 = manalyse(superanalyse, isb=true, delta=1, blksize=blksize, overlap=overlap, search=4, searchparam=2, pelsearch=2, truemotion=false, dct=5, scalecsad=2, trymany=false, global=true)
fv1 = mrecalculate(superanalyse, fv1, thsad=100, blksize=blksize2, overlap=overlap2, search=4, searchparam=2, truemotion=false, dct=5, scalecsad=2)
bv1 = mrecalculate(superanalyse, bv1, thsad=100, blksize=blksize2, overlap=overlap2, search=4, searchparam=2, truemotion=false, dct=5, scalecsad=2)
fv1 = mrecalculate(superanalyse, fv1, thsad=100, blksize=blksize3, overlap=overlap3, search=4, searchparam=2, truemotion=false, dct=5, scalecsad=2)
bv1 = mrecalculate(superanalyse, bv1, thsad=100, blksize=blksize3, overlap=overlap3, search=4, searchparam=2, truemotion=false, dct=5, scalecsad=2)

convertbits(16).mdegrain1(supermdg, bv1, fv1, thsad=150, thsadc=200, thscd1=500, thscd2=90, limit=0.2, limitc=0.45)

takla

1st February 2022, 23:07

With TR=12 the speed difference may finally reverse to SO=5 is better
DTL At 4K and TR=6 around 10 to 12GB of my RAM is being used. But at TR=8 I have to cancel the encoding because 99% of my RAM (32GB) is used. (Possible memory leak? Why does ram usage increase so much?) So any higher TR at 4K is not possible.

Boulder there have been plenty of examples in this thread. See post #77.

DTL

2nd February 2022, 00:40

"Why does ram usage increase so much?"

Pel=4 finest level is currently 16x more RAM in compare with pel=1 for 'super' clip (+lower sized levels) and it is multiplied to number of AVS+ threads and to AVS+ cache system. So it looks old developers not went down to pel=8 with 64x more RAM usage.

Each 'super' frame for pel=4 and 4K is about 2160x17 = about 37000 in height size. You can check it - return 'super' clip and see its frame size.

So may be ask AVS+ support how to decrease cached frames by AVS+ ? I read somewhere about 2-params Prefetch() - like

Prefetch(N, M)

where one value is num of threads and second is cached num frames ? Do not found anything about it in docs. Also there are 2 values of cache control:
SetCacheMode(mode)
AVS+Fine tunes the internal frame caching strategy in AviSynth+.
Available values:
0 or CACHE_FAST_START start up time and size balanced mode (default)
1 or CACHE_OPTIMAL_SIZE slow start up but optimal speed and cache size

May be try to set CACHE_OPTIMAL_SIZE ?

Also http://avisynth.nl/index.php/MT_modes_explained - may be adjusting of MT mode may decrease number of cached frames ?

I run tr=25 with 1080i at 16 GB system with few enough RAM usage - may be about half.

"Possible memory leak?"

It typically increases over time. I run 3 hours transcoding without leakage issues.

The 'fully optimized' mvtools with DX12_ME search and all pel modes on-shader SAD caculation and at-processing sub-shift MDegrainN will use 16x less host RAM for pel=4 processing.

DTL

4th February 2022, 19:48

ColorBarsHD(1920, 1080)
ConvertToYV12()
tr = 3
super = MSuper (pel=1, levels=1, chroma=false)
multi_vec = MAnalyse (super, multi=true, blksize=8, delta=tr, optSearchOption=5, overlap=0, levels=1, chroma=false)
MDegrainN (super, multi_vec, tr, thSAD=150)
Prefetch(12)

Encoded 600 frames in 17.893s (FFV1)
GPU usage is around 8%. Very low.

It looks I found where the DirectCompute load graph is shown: In Win 10 task manager 'GPU' window the hardware load graphs can be switched to Compute_0 and Compute_1. It looks all ComputeShader load is displayed only in these graphs. But some software may display sum load of many graphs (3d+cuda+compute+copy+videoencode+videodecode+...).
I still do not found what difference between Compute_0 and Compute_1 load graphs. At degrain running I see some about equal load in both graphs. But it finally not 0..1% load as displayed in 3D GPU graph.

Currently at some still in-progress version of sub-pel shifting for pel=2 and pel=4 for SAD computing I got about 35..40% load of both Compute_0 and Compute_1 graphs. At GTX1060 card and 1920x1080 interlaced processing (at about 18 fps output to AVSmeter). Not very few but still have some space to add MDegrainN in the future. Also I hope inside HWAcc the shifted (subpel-motion compensated) blocks may be reused in both SAD and MDegrainN computation.

Currently was and idea to download set of sub-shifted-blocks to host memory to send to MDegrainN but it will load memory transfer and still complex enough. It may be better to add onCPU subshift to MDegrainN as intermediate solution before transferring all processing to accelerator.

DTL

9th February 2022, 19:48

Somehow working update https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.08 . Now all supported pel values processed inside accelerator. Not sure if it good for pel 1 and 2 for fast CPUs but still no user-side selection where to process. For testing of onCPU pel 2 and pel 4 processing the previous version may be used.
Not very much tested for quality of SAD generation for pel 2 and 4 and it may more or less be different from onCPU old processing because of using different sub-sample shifting kernel.
It looks making compute shaders at HLSL is not very efficient because 'compute_X' load is significant now at GTX1060 card at relatively simple operation of runtime shifting of blocks with about 8 total samples kernel. Though the performance now is not depend on sub-pel shift value (can support any float shift with equal speed) and depend slightly only on kernel size (with half size of 4 the speed is a bit better, but still no user-control param and only separate build with internal constant define possible for now). So it may be mostly benefitical at pel=4 (depending on balance of CPU and accelerator speed).

magnetite

9th February 2022, 23:26

I think I went from 30% load up to 60% with this new build on my GTX 1080 Ti.

DTL

10th February 2022, 00:25

Some strategic idea: if sub-sample shifting takes significant time (resources) it can be somehow reused for degraining. Either to decrease host memory read traffic or to save time of host CPU from performing same shift operation one more time.

Possible ways:
1. Finish degraining inside accelerator (as planned). The already found limitation (at least for CS 5.1 standards): HLSL compiler reports about 16384 max recommended temp array size in threads group. It looks the limitation of 'register file' size of one core in accelerator (of some generation) ? So in current version of shader the number of threads in group was reduced to 4x4 to stop compiler from warning. The compiler allow to have more buf but warn about degrading performance (it can auto-offload temp array to main memory ?). And each thread currently have only small enough buf about 3x time block size to hold sub-shifted block (H and HV shifted).
But MDegrainN operation require to hold a set of 2_x_tr ref blocks to compute sad -> next compute weights from sads -> normalize weights and use blocks in averaging. Unfortunately it looks additive accumulation of shifted blocks in single temp buf is not possible. Because to get weight of block in the sum - we need to calculate all weights and normalize.
So it looks the sub-shifted blocks can not be stored in on-chip memory and need to be temporarily written to accelerator's main memory (it is typically faster in compare with host but usually not very for medium consumer accelerators). But this approach will limit more available memory in accelerator (need to store both source frames + shifted copies) and limit possible max tr-value.

2. Pack sub-shifted blocks into some framebuf and download to host memory and use as source for MDegrainN (instead of super clip of 4x or 16x time larger for pel 2/4). It will be close to current output of MCompensate I think. Though it will be new datastream for mvtools (like replacing of 'super' input clip in MDegrain arguments). And download operation from accelerator and loading into host cpu may takes some time. It allow to use any tr-value because not need to store all source + shifted frames in the accelerator's limited memory.

Also I still not sure if HLSL compiler make best possible asm program to compile convolution - may be some hand-crafted asm (inline-asm if possible in HLSL ?) may be faster and take less resources of accelerator. Need to read more how compute units in accelerator are designed. May be it sort of SIMD dispatch ports and can compute FMA of several floats per clock. Still not check what current HLSL compiler produce. It can output asm file but I need to read about its syntax and application to execution units in shader compute model.

DTL

10th April 2022, 20:00

Some first working example of low pass motion vectors internal filtering before MDegrainN processing: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.09

It is not final filter and not perfect - just first working example without significant output bugs. Currently only luma SAD is checked after new vectors calculated and compared with thSAD. If new filtered vector have SAD above thSAD - the original vector from MAnalyse is used.

The initial idea and issue about converting of noise (luma+chroma) into spatial (phase) noise at MDegrainN processing with nosied vectors was described at post: https://forum.doom9.org/showthread.php?p=1963966#post1963966

New control params for MDegrainN: MVLPFCutoff, thMVLPFCorr.

MVLPFCutoff: cut off frequency of the low pass filter for motion vector's components (dx,dy) in temporal (tr) axis.
Default 1.0 additional processing disabled.
Valid range 0.0..1.0. Estimated working range when enabled: 0.05 to 0.5. Values below 0.05..0.01 possibly change nothing because internal kernel size of filter is fixed 10 taps now.

thMVLPFCorr: Maximum difference between original and filtered vector's dx,dy components (any of component) for correction. If difference above this value (not internally scaled to pel value) - the original vector from MAnalyse is used.
Value =0 (default) disables correction completely (no LPF-processing effect even with MVLPFCutoff < 1.0). May be useful to fix some bugs at the footage with lots of different movement and noise.
Expected good value: pel*(4..10). It is mostly additional 'fail safe' limit. If no issues found it may be set to infinite (like frame_width * pel) to allow processing of very fast movements. Typical real upper value: about maximum inter-frame shift of moving subjects * 1.5 * pel.

Current production degrain script used for testing (interlaced 1080 source):

SetFilterMTMode("DEFAULT_MT_MODE", 3)

__source_here___

AddBorders(0,0,0,72)
ConvertToYV12(interlaced=true)
SeparateFields()

tr=15
super=MSuper(last, mt=false, chroma=true, pel=4, hpad=8, vpad=8, levels=1)
multi_vec=MAnalyse (super, multi=true, blksize=8, delta=tr, search=3, searchparam=2, overlap=0, chroma=true, optSearchOption=5, mt=false, levels=1, scaleCSAD=0)
MDegrainN(last,super, multi_vec, tr, thSAD=185, thSAD2=170, mt=false, wpow=4, thSCD1=350, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFCutoff=0.1, thMVLPFCorr=50)

Weave()
Crop(0,0,0,1080)

Support of overlap processing using filtered vectors still not implemented. It is not complex but need some time.

Found and fixed some point of memory leak in MDegrainN - may be it adds to the issues on February builds too.

DTL

3rd May 2022, 18:05

Some more idea: typically 'non-simple' degrain scripts use some pre-denoised pre-processed clip as source for MAnalyse (super clip for MAnalyse). Sometime the preprocessing is as simple as some low-pass filtering like blur. So the idea is to add this simple pre-processing into MAnalyse with hardware search options to offload more work to accelerator. The processing may be done with compute shader dispatched with uploaded to accelerator frames before sending to ME engine. It will free more host resources for MPEG encoding.

Dogway

4th May 2022, 16:04

That's a good idea. The 'standard' though is to use MinBlur() which denoises more flat areas and less edge areas. A 'cheap' alternative similar to MinBlur is to use Inter Quartile Median (IQM), that would be easier to implement.

DTL

4th May 2022, 22:11

I see typical 'pre-filter' in QTGMC is

prefilt = last
w = prefilt.width()
h = prefilt.height()
removegrain(12, 12).gaussresize(w, h, 0, 0, w+0.0001, h+0.0001, p=2).mergeluma(prefilt, 0.1)

That is equal in result (may be not speed) to

Blur(1).gaussresize(w, h, 0, 0, w+0.0001, h+0.0001, p=2).mergeluma(prefilt, 0.1)

Where combination of Blur() and gaussresize() is 2 low-pass filters in a sequence (may be merged to single with combined transfer characteristic).
That is all internal AVS+ processing operators. May be simplified to SomeLowPassFilter(args).mergeluma(prefilt, 0.1) that mean mixing input plane with weight 0.1 to low-pass filtered plane.

In SMDegrain script I see much more complex pre-filter processing.

DTL

6th May 2022, 11:45

Trying to simulate 'overlap' processing with non-overlap MAnalyse/MDegrain I try to make 2 processing paths with half-blocksize diagonal shift:

BkSz=8
BkSz_d2=BkSz/2

AddBorders(BkSz_d2,BkSz_d2,BkSz_d2,BkSz_d2)

no_sh=last
sh=Crop(BkSz_d2,BkSz_d2,width-BkSz_d2, height-BkSz_d2).AddBorders(0,0,BkSz_d2,BkSz_d2)

tr = 12 # Temporal radius
super_no_sh = MSuper (no_sh, chroma=true, pel=2)
super_sh = MSuper (sh, chroma=true, pel=2)

multi_vec_no_sh = MAnalyse (super_no_sh, multi=true, chroma=true, overlap=0, search=3, searchparam=2, delta=tr, mt=false, optSearchOption=1)
multi_vec_sh = MAnalyse (super_sh, multi=true, chroma=true, overlap=0, search=3, searchparam=2, delta=tr, mt=false, optSearchOption=1)

no_sh=MDegrainN(super_no_sh, multi_vec_no_sh, tr, thSAD=200, thSAD2=190, mt=false)
sh=MDegrainN (sh, super_sh, multi_vec_sh, tr, thSAD=200, thSAD2=190, mt=false)

#back
sh=AddBorders(sh, BkSz_d2,BkSz_d2,0,0).Crop(0,0,width-BkSz_d2, height-BkSz_d2)

Layer(no_sh, sh, "fast")

Crop(BkSz_d2,BkSz_d2,width-BkSz,height-BkSz)

The result looks really better in compare with no-overlap processing but still not as smooth as overlap=blocksize/2. Also the speed is about 2x better in compare with overlap=blocksize/2 (with CPU only processing).

Edit: may be right solution is not about deblocking of layers but creating correct blending mask. Like rhomb-shaped per each block tiled over all blocks of frame. Will try it with auto-sizing of mask using 'for' loops of AVS+.

DTL

12th May 2022, 08:43

New version: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.10

Added MVLPFGauss MVs low-pass filtering mode to MDegrainN as single control-param adjustment. MVLPF implemented in all processing modes of MDegrainN (chroma enabled and overlap enabled). Default = 0 (disabled), float param. Expected practical adjustment range 0.5..3.0. Too low values like <0.1 will mostly disable procesing, too high like 10 may cause bugs because internal convolution kernel is about 10 samples size and too high sigma values will make kernel not gauss-shaped but rectangular. The old values of 2-params LPF MVLPFCutoff and MVLPFSlope still exist but suspended to future development because simple gauss-kernel for LPF looks produce good results already. The speed of processing should not depend on kernel type. The non-over/undershoot gauss-kernel processing expected to make good results and the development of other LPF with controlled both cut-off frequency and slope with non-over/undershoot performance is more complex. Left for the future versions.

Added usage of scaleCSAD param defined in MAnalyse in the secondary SAD check after MVLPF processing in MDegrainN.

Fixed bug in MAnalyse that cause random processing aborting with error message 'motion vectors clip too short' (in SO=5). It magically works with non-initialized memory long time in the past but start to fail frequently with >1 MAnalyse in the script or testing with single pair of src-ref frame and MShow().
Fixed bugs in SAD computation in shader with luma and chroma with pel=2 and with chroma with pel=4.

When experimenting with shifted layers blending for simulate overlap processing I found the small padding of the single layer with about blocksize/2 also makes output MPEG encoded speed a bit lower. May be it is about interacting of previous MPEG compressed source blocks tesselation with hardware MVs search engine blocks tesselation. So current single layer processing script for 1080i source is:

SetFilterMTMode("DEFAULT_MT_MODE", 3)

__source_here___

AddBorders(0,0,0,72)

SeparateFields()

BkSz=8
BkSz_d2=BkSz/2
AddBorders(BkSz_d2,BkSz_d2,BkSz_d2,BkSz_d2)

tr = 15 # Temporal radius
super = MSuper (chroma=true, pel=4, levels=1)

multi_vec = MAnalyse(super, multi=true, blksize=8, delta=tr, overlap=0, chroma=true, optSearchOption=5, mt=false, levels=1)
MDegrainN(super, multi_vec, tr, thSAD=250, thSAD2=240, mt=false, wpow=4, thSCD1=350, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=100)

Crop(BkSz_d2,BkSz_d2,width-BkSz,height-BkSz)
Weave()
Crop(0,0,0,1080)

It looks current float-based sub-shifting in the shader (for pel=2 and pel=4) implementation not good in speed and need to make some integer-based implementation to see if it will be faster.

Some number of tests with pre-filtering of clip for MAnalyse (with 'simple' processing like from QTGMC) still shows close to no improvement in output MPEG output speed. May be current additional low-pass filtering of MVs in time domain before MDegrain works in the close way to low-pass filtering in spatial domain before MAnalyse.

All new features of MDegrainN also works with software modes of MAnalyse so not require Win10+DX12-ME hardware for usage so applicable to https://forum.doom9.org/showthread.php?t=173356 thread mvtools.

takla

14th May 2022, 05:38

Here are my thoughts so far:

I don't see a point in using GPU acceleration at all (unless you somehow manage to make it SIGNIFICANT faster to what it is now) because the only time GPU is faster right now, is in niche cases like TR=>10 with pel=2 or 4 at 4K. But even in those cases you need over 64GB of system RAM or else you run out of it, as I've pointed out before.

And considering hardware "tiers", when looking at the same price class, a $500 CPU will easily beat a $500 GPU here.

So personally I'd prefer improvements on the CPU side of things, because it makes more sense.

tormento

14th May 2022, 08:44

I don't see a point in using GPU acceleration at all
Because not everybody owns a 16 cores CPU and whatever resource you can free, it can be allocated to other filters or encoding.

takla

14th May 2022, 12:12

Because not everybody owns a 16 cores CPU and whatever resource you can free, it can be allocated to other filters or encoding.

Good point.

DTL

14th May 2022, 15:57

" the only time GPU is faster right now, is in niche cases like TR=>10 with pel=2 or 4 at 4K. But even in those cases you need over 64GB of system RAM or else you run out of it, as I've pointed out before."

At my current work setup of old enough i5-9600K CPU + GTX1060 accelerator and transcoding with x264 with close to 'placebo' settings I got about 3.5 fps with CPU only and about 5+ fps with SO=5 option. So with hardware acceleration I can process more footages per work day. The blockiness artifacts without overlap is very rare at that footage - mostly on large size fire flames or large size smoke. The water looks good.

Also I still not show here the tests but the 'star-like' hyperbolic zoneplate sub-sample moving + noise test shows a bit better motion compensation in V-direction (horizontal part of hyperbolic zoneplate) in compare with internal MAnalyse pel=4 search. Will try to post comparison results next time when will be at work.

The interlaced 1080 runs well with 6 threads at 16 GB Win10 system. Takes about 50% of RAM, so I expect 4K will takes about 32 GB at 6 threads (with 'typical' AVS+ cache control). But if you run at massive multicore CPU with >10 cores it really can overflow 64 GB RAM. So it may be good to limit number of threads per AVS and leave some free cores to MPEG encoder only.

"will easily beat a $500 GPU here."

I expect when prices after mining will drop - the old enough accelerators much cheaper $500 will be good to installed as 1 or 2 (or more) per host to help free CPU resources to MPEG encoding. Also the >1 accelerator I hope can be used for 'overlap' simulation using either internal AVS scripting (with masked Overlay() function) or I found old Fizick's plugin BlockOverlap with internal mask generation and blending 2 half-blocksize diagonally shifter layers. Though it is C-only and may be not as good in speed as possible AVS+ internal Overlay() filter. The mask for Overlay may be loaded from 8x8 BMP file and tiled using scripting to required frame size. Also this method allow to use any handcrafted blocks blending mask in MSPaint or any other pixel-setting editor. Or may be some scripting-way is possible to create clip of typical blocksize 8x8 with perset samples values.

Though currently I have only about 30% of VideoEncoder load - may be it is possible to make some more advanced mod of MAnalyse to send 2 pairs of frames per command sequence and get 2 ME output results for 'special' mode of MAnalyse for alternative motion-clip format for overlap blending (with not same blocks positions as with 'old overlap mode of MAnalyse+MDegrain) and special mode in MDegrainN for alternative overlap blending. It should be best in speed but require more programming work. It may be already tested for quality using AVS scripting.

The diagonal shift of 'shifted' version of clip in MAnalyse to send to ME accelerator is very easy - just shift starting address of buffer reading at creation of 'upload' command (and after typical padding after MSuper it will not buffer-overrun at the lower-right corner of buffer). In scripting it require combination of AddBorders+Crop that may produce much more memory bus traffic.

I make x64 build of Fizick's BlockOverlap plugin for new AVS+, but still not test it yet: https://github.com/DTL2020/BlockOverlap

"prefer improvements on the CPU side of things"

It is also planned. Currently in progress the internal (inside CPU) shifting of blocks for MDegrain for pel > 1. So with SO=5 in MAnalyse it can completely skip larger sub-shifted planes creation and decrease memory bus traffic. Currently the latest build have not finally debugged 'tech speed test demo' of this mode - new option for MSuper(pelrefine=false) to disable pel >1 planes creation and MDegrainN(UseSubShift=1) to enable alternative request of sub-shifted block from Fake* structure. It currenty have only integer AVX2 implementation for block size 8x8 (and luma only) but the AVX512 may be a bit faster. Currently not in production state - create distorted output but perform the full processing. Hope to look into debug of it soon.

It also will greatly decrease amount of memory for 'super' clips - about 15 times less for pel=4. Currently it is not made because it require to change size of 'super' clip and look if it not crash the underlying processing somewhere. Currently only CPU load disabled for pelrefine=false MSuper mode to test speed benefit that is much easier. So after this change will be finished and 'super' clip will be cropped to only 1x frame size we will get large memory saving with pel=2 (about 3 times) and pel=4 (about 15 times) with hardware ME modes. I theory in that case the usage of 'super' clip in that case will be mostly eliminated (may be to store padded 1x frame only with levels=1).

Also using same ideas of internal scaling of patch in CPU register file planned to make MAnalyse search optimized for pel >1 in same way. But it require more complex SIMD programming.

tormento

14th May 2022, 19:02

It currenty have only integer AVX2 implementation for block size 8x8 (and luma only) but the AVX512 may be a bit faster.
Please, keep in mind that someone (me) still has AVX CPUs. :o

DTL

15th May 2022, 01:21

kedautinh12

15th May 2022, 07:25

Hi, but anyone have much money like you :D

tormento

15th May 2022, 07:36

Intel promises AVX1024 in the mid of 202x already.
Perhaps on Xeons. They are disabling AVX512 in consumer CPUs.

Boulder

15th May 2022, 08:01

AVX mean it do not have fast enough integer operations with increased size register file. Also operations are limited to SSE2 128bit integer per op, AVX2 allow 256bit integer ops that is virtually twice faster. It is better to upgrade to AVX2 CPU at least in 202x years. Intel promises AVX1024 in the mid of 202x already.
This.

You can get a very decent second hand AVX2 system on the cheap thanks to AMD. The 5xxx series Ryzens can be used on very old mobos like several X370 or B350 based ones after a BIOS update. The previous generation is also not bad at all, I'm very happy with the 3900X. I'm contemplating on upgrading to 5900X since people seem to be switching to 5800X3D and there's a supply of used CPUs.

DTL

15th May 2022, 10:02

niche cases like TR=>10 with pel=2 or 4 at 4K.

'Big' tr-values really more naturally for 4K and more because with the same amount of light passed via input aperture of camera the samples at 4K tesselation of image receive 4 times less light in compare with 1080. Also if fill-factor of sensors is not great - even less.
So for 4K we need about 4 times more tr to have same relative shot noise in compare with 1080.

Also the new motion vectors low pass processing become more effective with not very small tr values. At least >5..6. So the number of position samples may be covered with impulse kernel of about 10 size. Also if I understand right to make good low-pass filter with cut-off of about 0.1 of the max (Nyquist) band we also need to process not very few samples (may be about 10).

The value noise of samples really cause 'position noise' because the position of image area (patch/block) defined by values of its samples only. So noised values of completely static block start to be translated in the position noise with current motion-search algoriphms (and may be with any motion search algo). After receiving not-zeroed coordinates for completely static block the blending engine start to motion-compensate with errors-noised vectors and it will cause blurring of block. For slow moving blocks it also cause not-perfect motion compensation and blurring. Also blending with shifted blocks with non-zero average value cause some shifting of block in the output frame so performed conversion of value noise in the position noise and MPEG encoder need to find and encode this error-movement too.
The effect of false noise-caused movement may become more visible with decreasing of block size. So the MVLPF processing may be more effective at smaller block sizes (like 8x8 or even 4x4).
So low-pass filtering of motion vectors after search for current pair of src+ref frames in big enough time domain using the nature of real images not have too fast and random moving of blocks can decrease these negative effects. But to perform filtering good enough it need to gather big enough number of motion vectors values of the block in consecutive frames.

The need of creating AVS-interface (script-accessible of samples data for read and write) to the MV data passed from MAnalyse to MDegrain really high. It will allow script-writers to create more advanced algoriphms of intermediate motion search data processing before MDegrain blending in the script form. And after slow experiments the best methods may be encoded into compiled functions of MDegrain or some new compiled filters of mvtools-pack for fast execution.
I think of creating motion-data clip in the RGB16 format with mapping of 16bit (dx,dy,SAD) data to R,G,B channels. But real current motion-data 'frame' have also some header of about 256 bytes inside with analysis data. So need to find if this header can be transfered as metadata unchanged and not in 'active' part of frame.

takla

20th May 2022, 19:38

'Big' tr-values really more naturally for 4K and more because with the same amount of light passed via input aperture of camera the samples at 4K tesselation of image receive 4 times less light in compare with 1080. Also if fill-factor of sensors is not great - even less.
So for 4K we need about 4 times more tr to have same relative shot noise in compare with 1080.

Oh wow. I didn't think about this but it makes perfect sense.

takla

27th May 2022, 05:05

AVX512 is coming you said @DTL?
https://i.imgur.com/kNyftsO.jpg

DTL

27th May 2022, 11:16

AVX1024 is coming in mid of 202x if this civilization will not die too fast. I hope it will have even larger register file size to perform more operations in fastest available memory in chip. AMD chips with at least AVX512 support is also good.

From https://clickthis.blog/en/sluhi-o-proczessorah-intel-xeon-sleduyushhego-pokoleniya-10-nm-emerald-rapids-7-nm-granite-rapids-5-nm-diamond-rapids-do-144-yader-lion-cove-k-2025-godu/
https://i0.wp.com/clickthis.blog/wp-content/uploads/2022/01/intel-xeon-future-table-2.jpg?w=640&ssl=1

Currently expected the most of performance boost from AVX2/AVX512 operations strategic redesign of MVtools are internal sub-sample shift and scale for both MAnalyse (onCPU processing) and MDegrain (onCPU processing) for pel 2 and 4. It will greatly reduce memory requirement for pre-calculated super clip and it looks very important for new massive multicore chips and current cache AVS design. Also as core become more and more faster in compare with main host memory it should finally make internal scale/shift processing faster in compare with pre-calculated upscaled planes for pel 2 and 4.

I also interested when intel hardware will provide DX12-ME API. It looks intel UHD Graphics 750 still do not have either drivers designed or required hardware features to expose this interface from MPEG encoder.

"I didn't think about this but it makes perfect sense."

Same is about HFR feature of new video systems - the max possible frame accumulation time with HFR is lower so number of accumulated photons per frame per object's view also lower.

So MDegrainN works as 'secondary level video camera' allowing to extend photon's flux accumulation time over the frame's exposure time in 'first level time sampling video camera'. It is only about real optical video cameras (not about digitally synth images). The more image data carriers (photons) accumulated per object's view - the more precision got. The object's view less distorted (noised) by natural photon's shot noise. So the possible tr value is in best case unlimited. Practically may be limited to typical cutscene duration that is several seconds minimum and with even old 24/25 fps it is about many 10'ths or 100'ths.

So the host RAM-based solution allow much higher tr values in compare with full onAccelerator processing (the max tr multiplied to threads number is very limited by accelerator's memory size to hold all required frames). Only sending current+ref pairs to accelerator to ME-process as it work now. Actually it now upload to accelerator current frame once and only send new refs per one output frame of MDegrainN.

StainlessS

27th May 2022, 17:03

Methinks you take on monster task DTL, mucho respecto, grassy ass :)

DTL

27th May 2022, 17:15

The more 'monster task' is to add more transforms support in MVtools - like rotate, scale, lighting (additive and multiplied levels change) and may be more. MVtools with only translate transform support are still at the very beginning of development.

guest

28th May 2022, 04:44

New version: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.10

Hi DTL, I had an error with this build with SMDegrain. (In RipBot264)

I can't tell you what the error was other than it was something to do with Line 645, of SMDegrain 3.5.0d

https://github.com/Dogway/Avisynth-Scripts/blob/master/SMDegrain%20v3.5.0d/SMDegrain%20v3.5.0d.avsi#L645

Or maybe this isn't meant to be used with SMDegrain

takla

28th May 2022, 05:31

Same is about HFR feature of new video systems - the max possible frame accumulation time with HFR is lower so number of accumulated photons per frame per object's view also lower.

Yep. But the higher frame rate also allows your brain to process more informations. This basically cancels that downside. (When comparred to the same video with a lower frame rate, anyway)

DTL

28th May 2022, 07:02

"higher frame rate also allows your brain to process more informations. "

Nowdays before viewing we typically need to very highly compress moving picture data to broadcast or stream via IP or torrents. So MDegrain is mostly part of MPEG compression to make the MPEG output rate and filesize as low as possible with given quality level. I now can make rips from HD1080i sources with x264 crf=18 at medium output rate down to about 4000 kBit/s for low-movement documentaries.

"Line 645"

It is just end of large 'messages output'. Need real AVS error message to see what may be wrong. As pinterf ask it should be compatible with all old scripts when no new params are used (internal defaults should be equal to 2.7.45 release). But to use new features it is required to manually edit scripts like SMDegrain to add new params in no-default values.

guest

28th May 2022, 07:50

"Line 645"

It is just end of large 'messages output'. Need real AVS error message to see what may be wrong. As pinterf ask it should be compatible with all old scripts when no new params are used (internal defaults should be equal to 2.7.45 release). But to use new features it is required to manually edit scripts like SMDegrain to add new params in no-default values.

That "message output" IS SMDegrain v3.5.0d !!!

I don't understand what you are saying here, it's a LOT over my head...I would think that Dogway would need to rectify this, so as to use your Mvtools2 build(s).

kedautinh12

28th May 2022, 10:15

You need show more detail about error. That not only said line 645. What is error in line 645?? You need sceenshot when error show with ripbot

DTL

28th May 2022, 10:15

If your error is with command-line AVSmeter: What is the error text ? Where this 645-line data come from ?

The 645 line at github is 2 }} - this either end of "SHOW PANEL" block or end of all SMDegrain() function. Anyway it is not command or filter or function call line so error may be anywhere above this line (or in any function call from above block).

guest

28th May 2022, 12:39

You need show more detail about error. That not only said line 645. What is error in line 645?? You need sceenshot when error show with ripbot

I can't give anymore info, as all AVSMeter displays is the path where SMDegrain v3.5.0d is, and that Line 645 is where the error is, and AVSMeter stops, at that point.

I don't know why it chooses that line which is simply }}, and I guess that's where Dogway needs to have a look.

That's ALL I can do...sorry.

takla

30th May 2022, 05:54

@DTL newest release (https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.10) gives me this error (https://i.imgur.com/DAEJOCt.png) which goes away if I delete line 5. Is dithering no longer valid?

DTL

30th May 2022, 08:48

Oh - I need to check it. Dithering in AVS ConvertBits() from 16 to 8 - is not in MDegrain. But may be output of MDegrain somehow cause this error. If you feed MDegrain with 16bit it should output 16bit too (with right params or defaults ?) and it can not not perform dithering internally. The DX12-ME mode MAnalyse only can use 8bit YV12 now so you need to make conversion in the script EZdenoise internally if you use 16bit input.

Where is the version of EZdenoise that you use in that script ?

takla

30th May 2022, 18:20

@DTL fixed it like this:

function EZdenoise(clip Input, int "thSAD", int "thSADC", int "TR", int "BLKSize", int "Overlap", int "Pel", bool "Chroma")
{
thSAD = default(thSAD, 150)
thSADC = default(thSADC, thSAD)
TR = default(TR, 3)
BLKSize = default(BLKSize, 8)
Overlap = default(Overlap, 4)
Pel = default(Pel, 1)
Chroma = default(Chroma, false)

Super = Input.ConvertBits(16).MSuper(Pel=Pel, Chroma=Chroma)
Multi_Vector = Super.ConvertBits(8).MAnalyse(Multi=True, Delta=TR, BLKSize=BLKSize, Overlap=Overlap, Chroma=Chroma)

Input.ConvertBits(16).MDegrainN(Super, Multi_Vector, TR, thSAD=thSAD, thSAD2=thSAD/2, thSADC=thSADC, thSADC2=thSADC/2)
}

DTL

30th May 2022, 20:27

I catch this crash with 16bit source: It happens only in release build in MAnalyse in SAD 16bit SSE function. Need some time to find what is wrong. I not touch that part from beginning.

Current workaround may be to feed 8bit to MAnalyse and 16bit to MDegrain.

This looks like working:

function EZdenoise(clip Input, int "thSAD", int "thSADC", int "TR", int "BLKSize", int "Overlap", int "Pel", bool "Chroma")
{
thSAD = default(thSAD, 150)
thSADC = default(thSADC, thSAD)
TR = default(TR, 3)
BLKSize = default(BLKSize, 8)
Overlap = default(Overlap, 4)
Pel = default(Pel, 1)
Chroma = default(Chroma, false)

Super = Input.MSuper(Pel=Pel, Chroma=Chroma)
Super8 = Input.ConvertBits(8).MSuper(Pel=Pel, Chroma=Chroma)
Multi_Vector = Super8.MAnalyse(Multi=True, Delta=TR, BLKSize=BLKSize, Overlap=Overlap, Chroma=Chroma)

Input.MDegrainN(Super, Multi_Vector, TR, thSAD=thSAD, thSAD2=thSAD/2, thSADC=thSADC, thSADC2=thSADC/2)
}

takla

30th May 2022, 20:38

@DTL new issues
mvtools2 from printerf MvTools2 2.7.45 and your newest release is no longer bit identical with same settings (post 117)

MvTools2 2.7.45 pinterf
time=21.380s
501.153 KB
ProRes

mvtools2 DTL (https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.10)
time=19.116s
500.510 KB
ProRes

The last build that was bit identical was https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.05

ChaosKing

30th May 2022, 21:50

@DTL new issues
mvtools2 from printerf MvTools2 2.7.45 and your newest release is no longer bit identical with same settings (post 117)

Maybe because some bugs were fixed!?

takla

30th May 2022, 22:24

Maybe because some bugs were fixed!?

Good point. Maybe. I'd like DTL to comment on it in more detail.

One more thing, the script from post #117 has 510.140 KB while the script from post #118 has 510.021 KB. Why is that? Shouldn't they both do the exact same thing? Encoded with ffmpeg -y -benchmark -i 01.avs -c:v ffv1 TEST.mkv

DTL

31st May 2022, 00:28

@DTL fixed it like this:

Super = Input.ConvertBits(16).MSuper(Pel=Pel, Chroma=Chroma)
Multi_Vector = Super.ConvertBits(8).MAnalyse(Multi=True, Delta=TR, BLKSize=BLKSize, Overlap=Overlap, Chroma=Chroma)

I think it should not work at all. The 'super' clip from MSuper is only semi-compatible with AVS so it must be directly feed to other filters of MVtools. If you try to 'convert 16bit super into 8bit' it may be damaged.
MSuper and MAnalyse outputs are semi-clips and I think can not be processed with any other filters (at todays) and designed to be used only as inputs for other MVtools filters. The only MSuper output can be outputted as AVS data and rendered as a viewable images. But I think it is not mean it can be easily 'converted' from 16bit to 8bit.

Same is about question "the script from post #117 has 510.140 KB while the script from post #118 has 510.021 KB. Why is that? Shouldn't they both do the exact same thing?". Yes - I think 'super' 8bit converted from 16bit 'super' is not the same as 'super' clip created from 16to8bit downconverted 'standard AVS clip' with MSuper().

I hope syntax

Super8 = Input.ConvertBits(8).MSuper(Pel=Pel, Chroma=Chroma)
Multi_Vector = Super8.MAnalyse(Multi=True, Delta=TR, BLKSize=BLKSize, Overlap=Overlap, Chroma=Chroma)

is 'auto' compatible with any bitdepth input because ConvertBits(8) will pass 8bit formats and downconvert any >8 to 8. So MAnalyse will always receive 8bit 'super8' semi-clip created from 8bit MSuper() input.

It looks

Multi_Vector = Input.ConvertBits(8).MSuper(Pel=Pel, Chroma=Chroma).MAnalyse(Multi=True, Delta=TR, BLKSize=BLKSize, Overlap=Overlap, Chroma=Chroma)

also works.

DTL

31st May 2022, 01:10

@DTL new issues
mvtools2 from printerf MvTools2 2.7.45 and your newest release is no longer bit identical with same settings (post 117)

And same is with workaround from post 118 ?

Also about >8 bit: It useful only if output of the denoising is also used with increased bitdepth. Like 8bit input and 10bit of more output to appropriate codec with >8bit support. Increasing bitdepth from 8 to 16 before MDegrain and converting back to 8 after MDegrain output possibly makes nothing useful but wastes RAM and speed. MDegrain can use 8bit input naturally and output 16 if it required. Simply set 'out16=true'. I currently do not use HEVC with 10bit for my work so do not use this feature.

takla

31st May 2022, 02:04

And same is with workaround from post 118 ?

Yes. I just double checked:

Note: I used ProRes before, but this here is FFV1.

2.7.46-a.10
510:021 KB

2.7.46-a.05
513.247 KB

2.7.45
513.247 KB

Here are the exact settings:

LWLibavVideoSource("C:\Users\Admin\Documents\01.mkv")
Trim(0, 600)
EZdenoise(HBD=true)
ConvertBits(10, dither=1)
Prefetch(12, 48)

function EZdenoise(clip Input, int "thSAD", int "thSADC", int "TR", int "BLKSize", int "Overlap", int "Pel", bool "Chroma", bool "HBD")
{
thSAD = default(thSAD, 150)
thSADC = default(thSADC, thSAD)
TR = default(TR, 3)
BLKSize = default(BLKSize, 8)
Overlap = default(Overlap, 4)
Pel = default(Pel, 1)
Chroma = default(Chroma, false)
HBD = default(HBD, false)

Super = Input.MSuper(Pel=Pel, Chroma=Chroma)
Super8 = Input.ConvertBits(8).MSuper(Pel=Pel, Chroma=Chroma)
Multi_Vector = Super8.MAnalyse(Multi=True, Delta=TR, BLKSize=BLKSize, Overlap=Overlap, Chroma=Chroma)

Input.MDegrainN(Super, Multi_Vector, TR, thSAD=thSAD, thSAD2=thSAD/2, thSADC=thSADC, thSADC2=thSADC/2, out16=HBD)
}

ffmpeg -y -benchmark -i 01.avs -c:v ffv1 EZ.mkv

And thanks for the explanation on msuper & manalyze. Makes sense that they do not care about bit depth, except for compatibility.

DTL

31st May 2022, 12:06

Here is a test build for testing : (Updated )

It have optimizations disabled for 16bit (>8bit) SAD functions. Currently it is the only fast way to save from crash in release build. Debug build works but too slow and can not show the source of crash. May process slower. I hope no other (8bit) processing speed of MAnalyse is touched. It is fast but not perfect fix. Also as a bonus it have 2 C++ builds for SSE2 and AVX2 CPUs. May be AVX2 is a bit faster at AVX2 CPU.

If testing will be acceptable I will place next pre-release to github.

About not bit-exact results with old 2.7.45 - it may be one of many small changes to MDegrainN finally works a bit different to old release. Will try to look later.

Update: https://drive.google.com/file/d/1lkmkuvP8mncb0CuOwzUq5pAL2KoQlyjq/view?usp=sharing

It is even more strange - the crash happens only with SSE2-targeted build inside SSE2 intrinsic-based function. With AVX2-targeted it looks working with all optimizations enabled. It may be some complex bug inside mvtools or in current used version of MSVS compiler ? So in updated archive link the SSE2 build is with optimizations for SAD >8 bit disabled and AVX2 build is 'normal'.

"thSAD2=thSAD/2"

I do not think it is good internal default. Typically thSAD should be 'just a small above noise level' . So setting thSAD2 too low by default you either lost many useful frames in tr-scope or force user to increase tr-value too high to have more neibour frames being covered with high enough thSAD near current frame (and it will cause slower speed processing). If user will raise thSAD high enough to get thSAD2 not too small - it may cause additional blurring or detail lost. I typically set thSAD2 manually just a few below thSAD. In float math it is about thSAD2=0.9*thSAD. I not test how it processed by AVS+ scripting. May be ToInt(0.9*thSAD) required if MDegrain will not accept float value or something else. Or let user to enter thSAD2 and thSADC2 too. I think the feature to allow lower thSAD at the edges of tr-range is mostly to decrease artifacts if artifacts happens. If not - the best for denoising and speed with current tr-value is thSAD2 close to thSAD.

takla

31st May 2022, 21:22

What do you want me to test with the test build exactly? Here are the encoding times:

AVX2
time=15.412s
SSE2
time=40.264s

Unfortunately thSAD2=0.9*thSAD is not a valid parameter. I changed thSAD2's to 135 manually, which is the same value. It made the encoding faster by 0.5 seconds on a 25 seconds clip (not applied in the encodings measured above) which is nice.

function EZdenoise(clip Input, int "thSAD", int "thSAD2", int "thSADC", int "thSADC2", int "TR", int "BLKSize", int "Overlap", int "Pel", bool "Chroma", bool "HBD")
{
thSAD = default(thSAD, 150)
thSAD2 = default(thSAD, 135)
thSADC = default(thSADC, thSAD)
thSADC2 = default(thSADC, thSAD2)
TR = default(TR, 3)
BLKSize = default(BLKSize, 8)
Overlap = default(Overlap, 4)
Pel = default(Pel, 1)
Chroma = default(Chroma, false)
HBD = default(HBD, false)

Super = Input.MSuper(Pel=Pel, Chroma=Chroma)
Super8 = Input.ConvertBits(8).MSuper(Pel=Pel, Chroma=Chroma)
Multi_Vector = Super8.MAnalyse(Multi=True, Delta=TR, BLKSize=BLKSize, Overlap=Overlap, Chroma=Chroma)

Input.MDegrainN(Super, Multi_Vector, TR, thSAD=thSAD, thSAD2=thSAD2, thSADC=thSADC, thSADC2=thSADC2, out16=HBD)
}

DTL

31st May 2022, 23:00

"What do you want me to test with the test build exactly?"

If it not crash at your system with your 16bit (>8bit) clips (frame size, colour format, etc).

"AVX2
time=15.412s
SSE2
time=40.264s"

Oh - it is so great difference. The non-optimized SSE2 really very slow. I think it will be only a bit slower because it uses SSE2 intrinsics internally. I hope not many users will run nowdays at SSE2 only CPUs with >8bits MAnalyse. To found what is wrong with 'normally optimized' SSE2 builds may take unknown time.

"Unfortunately thSAD2=0.9*thSAD is not a valid parameter. "

I think more comfortable to user to enter short 'far end thSAD multiplier' as script param and it can be applied to both thSAD2 and thSADC2 equally. Something like

function EZdenoise(clip Input, int "thSAD", float "far_thSAD_mul", int "TR", int "thSADC", int "BLKSize", int "Overlap", int "Pel", bool "Chroma", bool "HBD")
{
thSAD = default(thSAD, 150)
thSADC = default(thSADC, thSAD)
far_thSAD_mul = default(far_thSAD_mul, 0.9)
thSAD2 = Int(thSAD * far_thSAD_mul)
thSADC2 = Int(thSADC * far_thSAD_mul)

In real use it is easy to call like EZdenoise(200, 0.8, 10). Not set 4 params of th-s in some fixed ratio between values every time when need to adjust 'base thSAD'.

Name far_thSAD_mul is not short and nice - may be something shorter possible.

Also HBD is not clear about 'internal conversion' of 8bit input to 16bit output. May be better name 'toHBD'.

takla

1st June 2022, 00:11

Thanks. It no longer crashes.
And I take the 0.5seconds speed gain back. Just modern CPU boosting which caused the variance...
On HBD: I only added it because of the crash with my usual settings. But since that is fixed now I'll remove it again.
Also I never wanted to expose thSAD2's anyways, for the same reason you mentioned.

And thank you for showing me how to add the falloff multiplier.

This is what I'm using now:

LWLibavVideoSource("C:\Users\Admin\Documents\01.mkv")
Trim(0, 600)
ConvertBits(16)
EZdenoise()
ConvertBits(10, dither=1)
Prefetch(12, 48)

function EZdenoise(clip Input, int "thSAD", int "thSADC", int "TR", int "BLKSize", int "Overlap", int "Pel", bool "Chroma", float "Falloff")
{
thSAD = default(thSAD, 150)
thSADC = default(thSADC, thSAD)
TR = default(TR, 3)
BLKSize = default(BLKSize, 8)
Overlap = default(Overlap, 4)
Pel = default(Pel, 1)
Chroma = default(Chroma, false)
Falloff = default(Falloff, 0.9)

Super = Input.MSuper(Pel=Pel, Chroma=Chroma)
Multi_Vector = Super.MAnalyse(Multi=True, Delta=TR, BLKSize=BLKSize, Overlap=Overlap, Chroma=Chroma)

Input.MDegrainN(Super, Multi_Vector, TR, thSAD=thSAD, thSAD2=Int(thSAD*Falloff), thSADC=thSADC, thSADC2=Int(thSADC*Falloff))
}

takla

2nd June 2022, 12:52

@DTL
What is your opinion on Vulkan based video processing (https://www.khronos.org/blog/an-introduction-to-vulkan-video)? Are you aware of it? And yes I realize it does not explicitly talk about motion estimation (because it is probably missing?).

My reason for bringing it up is you talked about (https://forum.doom9.org/showpost.php?p=1969840&postcount=51) MVtools based denoising on a capture device and I too thought about this before. The thing is, such a device would probably not support DX12. Realistically, the camera footage is send to a smartphone. And that platform would support Vulkan on Android or Metal on iOS.

There is also vkFFT (https://github.com/DTolm/VkFFT), which could also be used for denoising.

DTL

2nd June 2022, 17:29

"(because it is probably missing?"

Yes - it looks Vulkan developers still not reach the ME-API as a service of hardware video encoder exposed to other applications. It looks very rarely need by anyone even todays. Only Microsoft understand it and add to DirectX API. Also I think no other (using general purpose computation units in accelerator) implementation of ME available as free to use library in Vulkan.

"on a capture device"

Not on a capture hardware interface card directly. But as a feature of live capture host or pass-through host with a function of live MDegrain. Practically its data flow will be from capture interface card to accelerator card and via CPU to output card. Or with industry transition to IP-based streams - from software API to receive stream - to accelerator and back to API so send stream via IP. Using hardware IP adapters.

"that platform would support Vulkan on Android or Metal on iOS."

I think current smartphones uses same methods of noise reduction as mvtools in hardware. But I not read if any API is exposed to user applications. It may be deeply in the hardware units for camera data processing. So in a perfect world it may be even possible to connect set/rig of smartphones as hardware accelerators to some PC host via USB and use as external hardware accelerated denoisers. The most of money now looks put in the quality of hardware accelerated denoising in smartphones so they quickly got nice results. But it may be covered by patents and not exposed as API for external user applications.

In the current phase of dying civilization the end-users home desktop PCs are dying too. And current buyers can simply put money to smartphone with good denoising if required to shoot new footage. The pro broadcast cameras looks also progress with internal denoisers as I see in 2022 from Tallinn Europe Skate championship broadcast. So it looks any investment in hardware accelerated denoising for home desktop PCs is not any profitable nowdays.

takla

3rd June 2022, 02:57

IIRC denoising depends on the camera app itself. There are some apps which let you disable most postprocessing. But yes, it is unclear if they use fixed functions or CPU.

And yes, if you can do good denoising internally, I can see why no one wants to spent dev time for a desktop solution.

DTL

16th June 2022, 12:27

Finally some working tech demo of 2 different processing of MSuper/MDegrainN for pel > 1 : https://drive.google.com/file/d/1E8iAWho9yf9svQq32FzsrPp13mU2Z_Q7/view?usp=sharing . Only for chroma=false processing or RAM usage/speed testing. Chroma=true with UseSubShift>0 in MDegrainN still outputs some buggy blocks sometime. AVX2 build only.

Test scripts :

LoadPlugin("mvtools2.dll")

ColorBars(3840,2160, pixel_type="YV12")

Trim(0,1000)

tr=12
super=MSuper(last,chroma=true, mt=false, pel=4, hpad=16, vpad=16, levels=1, pelrefine=false)
multi_vec=MAnalyse (super, multi=true, blksize=8, delta=tr, overlap=0, chroma=false, mt=false, optSearchOption=5, optPredictorType=0,levels=1)
MDegrainN(last,super, multi_vec, tr, thSAD=175, thSAD2=160, mt=false,wpow=7, UseSubShift=1)

Prefetch(6)

vs

LoadPlugin("mvtools2.dll")

ColorBars(3840,2160, pixel_type="YV12")

Trim(0,1000)

tr=12
super=MSuper(last,chroma=true, mt=false, pel=4, hpad=16, vpad=16, levels=1, pelrefine=true)
multi_vec=MAnalyse (super, multi=true, blksize=8, delta=tr, overlap=0, chroma=false, mt=false, optSearchOption=5, optPredictorType=0,levels=1)
MDegrainN(last,super, multi_vec, tr, thSAD=175, thSAD2=160, mt=false,wpow=7, UseSubShift=0)

Prefetch(6)

At i5-9600K + GTX1060 the second (standard mvtools MSuper/MDegrainN) takes about 7+ GB RAM and runs about 1.5 fps. The new internal sub-shifting method for MDegrainN of single full-size frame takes about 1.8 GB RAM and run at about 7.5 fps. Unfortunately the decreasing of RAM usage with pel=4 is not 16x times lower but only about 4 times. May be super clips are not largest in caching in AVS+.

takla

16th June 2022, 18:08

tested with 720p (I'll test 4K in a bit)

pelrefine=false with UseSubShift=1 looks like this (https://imgsli.com/MTEyNzg2) (after). Uses 1860MB

pelrefine=true with UseSubShift=0 looks normal. Uses 2880MB

And please do not use ColorBars for testing. I'll share a 4K sample in an hour or so...

DTL

16th June 2022, 18:32

"UseSubShift=1 looks like this (after)."

I know the output quality is completely bad. It is only test for RAM usage and speed at the use cases like https://forum.doom9.org/showthread.php?p=1962965#post1962965 . Where it exhaust 32 GB of RAM with tr about 10.

takla

16th June 2022, 18:48

Oh, ok.

Edit:
PelRefine=true & UseSubShift=0 uses 5908MB

PelRefine=false & UseSubShift=1 uses 3731MB

Here is the sample (https://www.mediafire.com/file/pu0tqruqpklqcd4/SAMPLE.mkv/file)

Not full "4K", cropped to 1600 pixels in height, but still.

by the way, can you tell me why my script crashes? I ended up testing with your settings instead.

function EZdenoise(clip Input, int "thSAD", int "thSADC", int "TR", int "BLKSize", int "Overlap", int "Pel")
{
thSAD = default(thSAD, 150)
thSADC = default(thSADC, thSAD)
TR = default(TR, 12)
BLKSize = default(BLKSize, 8)
Overlap = default(Overlap, 0)
Pel = default(Pel, 4)

Super = Input.MSuper(Pel=Pel, Chroma=true, Levels=1, PelRefine=false)
Multi_Vector = Super.MAnalyse(Multi=true, Delta=TR, BLKSize=BLKSize, Overlap=Overlap, Chroma=false, Levels=1, optSearchOption=5, optPredictorType=0)

Input.MDegrainN(Super, Multi_Vector, TR, thSAD=thSAD, thSAD2=int(float(thSAD*0.9)), thSADC=thSADC, thSADC2=int(float(thSADC*0.9)), UseSubShift=1)
}

DTL

16th June 2022, 21:25

"can you tell me why my script crashes? "

That was not good debugged build and real far outside frame motion vectors may cause crash. So the first test was with not-noised colourbars and larger padding of 16 to save from crash better. Though it was not very good test because not-noised colourbars should produce zero motion vectors and sub-sample shifting is not used so speed may be better. Though the test shows significant difference in speed even in with static colourbars (may be 4K pel=4 super shifted planes creation with old method pelrefine=true also loads CPU/memory a lot too).

Here is possibly better protected from that crash version so should run with real content with default padding of 8 -
https://drive.google.com/file/d/1oIyVmqo8gYRsZFeinZ-7l46zwerbw-Hk/view?usp=sharing

"PelRefine=true & UseSubShift=0 uses 5908MB
PelRefine=false & UseSubShift=1 uses 3731MB"

What was the real frame size and threads number ? Can you adjust Prefetch() to match your CPU cores number ? What is the fps difference ?

"Not full "4K", cropped to 1600 pixels in height,"

If you do not have full-frame 4K source you can put simple fast resize like BilinearResize(3840,2160) before degraining.

"Here is the sample"

I test with your source file sample and FFMS2 source with a full script:

LoadPlugin("mvtools2.dll")
LoadPlugin("ffms2.dll")

function EZdenoise(clip Input, int "thSAD", int "thSADC", int "TR", int "BLKSize", int "Overlap", int "Pel")
{
thSAD = default(thSAD, 150)
thSADC = default(thSADC, thSAD)
TR = default(TR, 12)
BLKSize = default(BLKSize, 8)
Overlap = default(Overlap, 0)
Pel = default(Pel, 4)

Super = Input.MSuper(Pel=Pel, Chroma=true, Levels=1, PelRefine=false)
Multi_Vector = Super.MAnalyse(Multi=true, Delta=TR, BLKSize=BLKSize, Overlap=Overlap, Chroma=false, Levels=1, optSearchOption=5, optPredictorType=0)

Input.MDegrainN(Super, Multi_Vector, TR, thSAD=thSAD, thSAD2=int(float(thSAD*0.9)), thSADC=thSADC, thSADC2=int(float(thSADC*0.9)), UseSubShift=1)
}

FFmpegSource2("sample.mkv")

ConvertBits(8)
ConvertToYV12()

EZdenoise(TR=12)

Prefetch(6)

At my CPU i5-9600K with 6 cores with 6 threads it run AVSmeter with about 5600M RAM and about 2.6 fps. Without crash (at least at first about 100 frames) with latest build. The total letterboxed frame size in a sample I see is 3840x2160 that is enough for test.
With pelrefine=true and UseSubShift=0 it looks start to swap taking about 10..11+ GB RAM and so fps drops to about 0.3.

It looks with filesource filter and some simple intermediate like convertbits and converttoyv12 the used RAM for AVS+ caching difference is even smaller. Though still reach about 2x.

I also tried to play with combination of SetCacheMode(1) and different Prefetch(6,N):

With lowest possible Prefetch(6,1) and pelrefine=true usesubshift=0 I got RAM usage about stable 8400M but still low fps about 0.3.

With Prefetch(6,2) and pelrefine=false usesubshift=1 RAM usage is about 4700M and speed about 1.2 fps.
With not-defined N in Prefetch(6,N) the RAM usage is just a bit higher about 5200M and speed quickly reach 2.5+ fps.

takla

17th June 2022, 01:23

No longer crashing, thanks.

With
LWLibavVideoSource("C:\Users\Admin\Downloads\SAMPLE.mkv")
ConvertBits(8)
Crop(0, 280, -0, -280)
BilinearResize(3840, 2160)
EZdenoise()
Prefetch(12, 12)

I get 4082MB & 6293MB (only encoding the first 12 frames in avspmod). During actual encoding, 2.6 FPS and 1.3 FPS respectively. Memory usage is much higher during encoding in ffmpeg, with UseSubShift=0 randomly spiking from 8GB to 22GB.

With SetCacheMode(1) memory usage is halved but so is the speed (FPS)

With just CPU, RAM usage is much more stable and stays below 7GB at all times. FPS is about 2.7

DTL

18th June 2022, 09:38

Thank you for testing and providing results.

There also an idea about simulating of overlap MDegrainN processing with single non-overlap output of MAnalyse - using interpolating of motion vectors with half-block offset. May be simple mean MV of the 4 surrounding MVs and also mean of SAD. It will not be as precise as true shifted motion search but may good enough simulate overlap processing to hide currently sometime visible blockiness while keeping speed at good value and not require to use 2 hardware accelerators to keep speed.

For speed of development the overlapping may be made inside AVS scripting (or BlockOverlap fizik's filter).

The interpolation may be made inside MDegrainN. Possible processing may be like

mvs_clip=MAnalyse(overlap=0)
std=MDegrainN(mvs_clip)
shifted=MDegrainN(mvs_clip, interpolateoverlap=true)
BlockOverlap(std, shifted)

DTL

2nd July 2022, 13:19

anton_foy

8th July 2022, 15:58

New build: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.11 SSE2/AVX2 builds by VisualStudio2019 16.3.

Added UseSubShift param to MDegrainN and pelrefine=true/false to MSuper. Default = 0, set to 1 to enable.

If pelrefine=false in MSuper - all other filters must use UseSubShift=true (or optSearchOption=5 for MAnalyse).

Also redesigned MDegrainN no-overlap processing to single pass YUV formats processing - looks like also added to performance. Old MDegrainN uses each plane separate processing that cause 3 times more DegrainWeight() and norm_weights() calls with equal data. Also separate planes processing not allow to reuse subshifted blocks in both SAD re-check in MVLPF processing followed by MDegrainN processing efficiently

Currently only YV12 format for block size 8x8 (8x8 Y and 4x4 UV) is fully accelerated with AVX2 for sub-shifting. All other will fallback to C-reference that is slow. Also only 8bit is now fully supported.

There is very experimental UseSubShift=1 for MAnalyse to test possible speed (may be unstable and crash with chroma=true , only valid for optPredictorType=1) in 'onCPU' MAnalyse with pel > 1. I still not expect it may be faster in compare with pre-computed planes form MSuper (at least before more fast AVX512 subshift implementations will be designed) but may help to run with larger threads count or tr-value at the limited RAM hosts.

Also to make speed of MVLPF better by reusing of subshifted blocks from MVPlane - the redesigned to single pass processing use_block_yuv() function in MDegrainN is added. It looks helps also in other processing modes performance. Currently only non-overlap single pass colour formats processing in MDegrainN is implemented (no external switches - auto detecting if YUV format input and overlapH=overlapV=0). For overlapped processing single-pass mode is also possible but need more time and more complex design.

If hardware ME accelerator is not available it is possible to enable some 'fastest' mode of MAnalyse to check pel=4 processing without badly limited speed by full-processing mode in MAnalyse:

tr=15
super=MSuper(last, mt=false, chroma=true, pel=4)
multi_vec=MAnalyse(super, multi=true, blksize=8, delta=tr, search=3, searchparam=2, overlap=0, optSearchOption=1, optPredictorType=4, chroma=false, mt=false)
MDegrainN(last,super, multi_vec, tr, thSAD=250, thSAD2=240, mt=false, UseSubShift=1)

At i5-11600 it runs at 15.6 vs 20.4 fps with UseSubShift=0 or 1 for MDegrainN for about FullHD frame size.

Updated: _2 version from 02.07.22 with fixed bug.

What you do with mvtools is very interesting and I love the progress you make, much respect. Yesterday I tried your latest build and it worked quite well with the one clip I tested. Some details were smeared/blurred compared to my tests with to my prefiltered tests with the pinterf mvtools but I only tested one problematic clip yet. Overall I am very positive to this build you made since even without prefiltering the lines and objects in high grain clip did not get the usual dancing/wobblyness that is very annoying. I cannot wait until you release it for HBD and blocksize above 8. The source I have been testing it with is 8-bit slog2 4K (3840x2160). With your script above I get about 0.9fps with my intel i5 3570 3.4ghz, 32gb ram, nvidia geforce GTX 970. Mostly I found with pinterf mvtools to experiment with different prefiltering techniques since I have 4 quite different test clips in 4k that are very hard to denoise with the same script. After many months I have found something after trial and error that seems to work pretty well for everything but with your build I think it will improve alot in the future. Thanks again for your great efforts!

DTL

8th July 2022, 21:14

"I cannot wait until you release it for HBD and blocksize above 8. "

It is only for better speed and lower RAM usage at some use cases. All new quality features already should work with all 2.7.45 bitdepth and blocksize values supported. The DX12-ME mode can not support > 8 bit input because it is not supported by current Microsoft DX12 API. And the only supported input format for hardware ME is NV12 that is internally converted from YV12 AVS format. So to process HBD with hardware ME you need to downconvert source for MAnalyse to YV12. And you can feed 16bit source to MDegrainN using different super clip.

Example was already shown here like

Super = MSuper(levels=1...)
Super8 = ConvertToYV12.MSuper(levels=1...)
Multi_Vector = Super8.MAnalyse(optSearchOption=5, levels=1...)

MDegrainN(Super, Multi_Vector,...)

"the lines and objects in high grain clip did not get the usual dancing/wobblyness that is very annoying."

It is with MVLPF options enabled for MDegrainN ?

DTL

10th July 2022, 19:56

New version: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.12

Added single pass colour overlapped processing in MDegrainN. Fixed regression of not using thSADC/thSADC2 in single pass processing.
Added tweaking param adjSADLPFedmv to MDegrainN to adjust SAD of MVs passed thSAD check after filtering. Float param. Default 1.0 - no correction. Recommended value about 0.8. Typically SAD of the filtered MVs positions is a bit higher in compare with initial after ME processing (so ME engine points to best SAD). So this adjustment allow to add some boost to weighting of blocks after interfiltering of MVs.

Added optSearchOption=6 to MAnalyse. In this mode DX12-ME only used for getting MVs from HW accelerator and SAD calculation performed on host CPU. Compute.cso shader is not used. Also for 8x8 8bit block available UseSubShift=1 for MAnalyse to use sub-shifting (allow to run with pelrefine=false at MSuper and save RAM).
May be faster at some combinations of host/accelerator. Also the SAD calcultation of shader for pel=2 and pel=4 still not completely correct (higher in compare with original mvtools).

So for onCPU SAD calculation (as 'reference' quality mode untill shader not completely fixed):

super=MSuper(mt=false, chroma=true, pel=4, hpad=8, vpad=8, levels=1, pelrefine=false)
multi_vec=MAnalyse (super, multi=true, blksize=8, delta=tr, overlap=0, chroma=true, optSearchOption=6, mt=false, levels=1, UseSubShift=1)
MDegrainN(last,super, multi_vec, tr, thSAD=250, thSAD2=240, mt=false, wpow=4, thSCD1=400, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.8, UseSubShift=1)

On i5-9600K CPU with GTX1060 MAnalyse (SO=6 and USS=1) is still a bit slower with MPEG encoding (about 6 vs 6.7 fps) but produces a bit smaller file.

The RAW performance of MAnalyse with different options of search and SAD calcultaion looks may be tested with AVSmeter without MDegrain like

super=MSuper(mt=false, chroma=true, pel=4, hpad=8, vpad=8, levels=1, pelrefine=false)
multi_vec=MAnalyse (super, blksize=8, chroma=true, optSearchOption=6, mt=false, levels=1, UseSubShift=1)
MStoreVect(multi_vec)

It looks MStoreVect not support multi=true ? So the resulted fps need to be divided to tr_x_2 to estimate real processing speed (without MDegrain) with different tr-values. This script produces performance in pairs frames per second (src+ref).

The 16bit subshifting in MDegrainN still not work completely correctly and still only slow C-reference.

DTL

20th July 2022, 13:45

Finally the most long awaited feature to MDegrainN with hardware acceleration in 2022 - fully internal MDegrainN interpolated overlap mode : First working release - https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.13

Added interpolated overlap mode to MDegrainN. Only 'max' overlap of blocksize/2 mode currently implemented.

New param of MDegrainN: IntOvlp (int).
Values:
0 - standard mode (default).
1 - internally interpolate input MVs to blocksize/2 overlap mode.

Added block size 16x16 for subshifting with AVX2 implementation. Fixed bug with chroma=false in MDegrainN no copy of chroma planes to output from previous release.

The new param is int and not bool because it is planned to test different interpolated overlap modes in the future. Currently it have more fail-safe design with SAD re-check for interpolated MVs to decrease probability of bad blends. But it is slower. It is possible to run interpolation-only faster mode with SAD interpolation too without re-check but it may decrease quality. Also it is possible to move MVLPF processing before interpolation to test speed/quality.

At the i5-9600K with GTX1060 it runs with x264 encoding about 50% slower but the quality is visibly better. No more blockiness on flames/fogs/fades should be. Also small blockiness on moving objects mostly removed.
It also may run with 'onCPU' MAnalyse with no-overlap MVs search to make some performance gain without HW accelerator.
The subshifting may be used in this mode but may or not be faster in compare with 'precalculated' sub planes in MSuper - looks may depend on host CPU. At i5-9600K with IntOvlp=1 a bit faster run with no-use of subshifting feature.

Now about 4 different quality/speed modes avaialable for overlapping:
1. Old onCPU MAnalyse overlap (full true up to 4x blocks number overlapping search) - possibly best 'reference' quality. Slowest mode.
2. 2 separated clips diagonally shifted at half-block sized processed with hardware-accelerated MAnalyse (single or dual accelerators should be supported if available for each MAnalyse) and overlapped in AVS using different internal or external filters. Uses a 2 sets of really analysed full frame MVs. May be a bit lower in quality in compare with 1. Require additional scripting and/or plugins. The Fizick's BlockOverlap pluging is still C-reference only so may be slow. Speed depend on host performance.
3. Hardware MAnalyse (SO=5 or 6) and interpolated overlap in MDegrainN based on single non-overlapped MVs array. Faster but may be lower in quality in compare with 1 and 2. Possibly the most RAM-saving mode (also support minimal RAM usage with UseSubShift option).
4. No overlap processing with hardware accelerated MAnalyse and standard MDegrain in no overlapped mode. Lowest quality - may produce visible blockiness on flames/fogs/fades. Fastest mode.

I hope the speed penalty from no-overlapped MDegrainN with interpolated overlap may be decreased in the future releases - still not look with profilter what may be optimized more. But in the interpolated overlap mode it also processes 4x blocks number so the host CPU load is high.

Some sad news - the block size 16x16 runs unstable at least at some test modes with hardware acceleration at my remote test host and remote debugger can not catch exception about divide by zero. So it may be a NVIDIA driver issue of Windows or this software.

If you like my software - you may donate me or join my team in OZON promo platform to support my growing family with several kids. Write me a private message for details.

mastrboy

21st July 2022, 18:40

Finally the most long awaited feature to MDegrainN with hardware acceleration in 2022 - fully internal MDegrainN interpolated overlap mode : First working release - https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.13

Does it not support YUV420 in 8bit?
I can only get it to work with 10,12,16bit:

Working tests:
ColorBarsHD().crop(4,0,-4,0)
ConvertToYUV420().ConvertBits(16)
#ConvertToYUV420().ConvertBits(12)
#ConvertToYUV420().ConvertBits(10)
tr = 3
super = MSuper ()
multi_vec = MAnalyse (super, multi=true, delta=tr, blksize=16)
MDegrainN (super, multi_vec, tr, thSAD=400, thSAD2=150, IntOvlp=0)

Not working:
ColorBarsHD().crop(4,0,-4,0)
ConvertToYUV420().ConvertBits(8)
#ConvertToYUV420()
tr = 3
super = MSuper ()
multi_vec = MAnalyse (super, multi=true, delta=tr, blksize=16)
MDegrainN (super, multi_vec, tr, thSAD=400, thSAD2=150, IntOvlp=0)

Error I get from AVSmeter on 8bit content:
Exception 0xC0000005 [STATUS_ACCESS_VIOLATION]
Module: C:\Program Files (x86)\AviSynth+\plugins64+\mvtools2.dll
Address: 0x00007FFD41996618

DTL

21st July 2022, 21:52

"Does it not support YUV420 in 8bit?"

Practically the only mostly tested format is the YV12 that is separated planes YUV420 in 8bit. That I typically use at my encodings. And block size of 8x8. If you got crash with block size 16 - try to increase padding in MSuper to 16 or more. I think it was fixed in some old versions (may be in the pinterf 2.7.45 source) but if appear again - the current first workaround to try is to increase padding.

So the better MSuper for blocksize=16 is
MSuper(hpad=16, vpad=16)

I even think of making it something like auto-adjust of padding from block size but unfortunately the data flow is from MSuper to downstream filters so MSuper can not get the block size from MAnalyse (in the easy way of current frames sending via AVS environment). And padding of 8 is internal default in MSuper. May be it can be safely enough increased to 16 or even 32 because current PCs typically have more memory. Will try to do it in next builds.

It looks it is old issue of mvtools so in some scripts I see auto-increasing padding to the block size may be added for fail-safety - it it easy in script but may be not possible in a separated filters execution:

Myblksize = 16
sc=MSuper(hpad=Myblksize, vpad=Myblksize, ...)
MAnalyse(sc, blksize=Myblksize,...)

I know users like block size of 16 because it typically faster onCPU (and with overlap it make not very visible blockiness) but I typically use 8x8 because it give better quality (also as I see 16 is unstable with HW modes at least at my current only avaialble test hardware setup).

Other known issue that SSE2 builds may run unstable with bitdepth >8 on new CPUs. So AVX2 build is recommented where possible.
Also the very few frame sizes were tested - so it is recommended to start from 'standard' of 1920x1080 for FullHD and 3840x2160 for UHD4K. If HW mode will create several buggy blocks lines at the bottom of frame - current workaround is to pad frame from the bottom to several block size lines (I typically use 72 for FullHD and blocksize of 8x8).

The padding is required to keep good quality at the edges of frames because all search, SAD check and blend engines can not operate with parts of blocks (paranoid check of borders will decrease processing speed over all the frame) so for correct and best quality work the padding of at least blocksize size is good. Too large default padding will waste of RAM and may decrease speed. But if 0xC__5 exception occur and increasing padding to some 'large value' like blocksize x10 solves it - it is a mark that some more debug and adjusting of clipping MVs or other bugfix required.

takla

22nd July 2022, 10:28

@DTL
Great update!

LWLibavVideoSource("C:\Users\Admin\Downloads\newSAMPLE.mkv")
Crop(0, 280, -0, -280)
BilinearResize(1920, 1080)
ConvertBits(8, dither=1).ConvertToYV12()
EZdenoise(thSAD=300, TR=8, Chroma=true)
Prefetch(12, 48)

function EZdenoise(clip Input, int "thSAD", int "thSADC", int "TR", int "BLKSize", int "Overlap", int "Pel", bool "Chroma")
{
thSAD = default(thSAD, 150)
thSADC = default(thSADC, thSAD)
TR = default(TR, 3)
BLKSize = default(BLKSize, 8)
Overlap = default(Overlap, 0)
Pel = default(Pel, 1)
Chroma = default(Chroma, false)

Super = Input.MSuper(Pel=Pel, Chroma=Chroma, Levels=1)
Multi_Vector = Super.MAnalyse(Multi=true, Delta=TR, BLKSize=BLKSize, Overlap=Overlap, Chroma=Chroma, Levels=1, optSearchOption=5)

Input.MDegrainN(Super, Multi_Vector, TR, thSAD=thSAD, thSAD2=int(float(thSAD*0.9)), thSADC=thSADC, thSADC2=int(float(thSADC*0.9)), IntOvlp=1)
}

CPU
v13 - Levels=1, Overlap=BLKSize/2, 8-bit & YV12
time=74.898s
time=75.007s
857.119 KB

GPU
v13 -optSearchOption=5 & IntOvlp=1
time=71.913s
time=73.643s
time=72.687s
826.089 KB

GPU
v13 -optSearchOption=5 & IntOvlp=0
time=76.360s
785.041 KB

GPU
v13 -optSearchOption=6 & IntOvlp=1
time=72.494s
824.445 KB

All tested using the previously posted 4K sample, with this command:

ffmpeg -y -benchmark -i 01.avs -c:v prores_ks -qscale:v 4 v13.mkv

Hardware used:
AMD Ryzen 3900X
AMD Radeon RX5700

Quality difference to CPU is now much closer and speed is now faster even at 1080P. I'll most likely post an update for EZdenoise in my thread soon, with some instructions.

DTL

22nd July 2022, 14:52

It is more interesting to test with max quality at pel=4. CPU only vs DX12-ME assisted. Default pel=1 is sort of 'draft' quality only. Same as IntOvlp=0 - fastest but low quality mode.

Also it is good to test if 'large cache' AMD Ryzen will be faster or slower with new interpolated overlap mode for MDegrainN (prefferably with MVLPF enabled also that adds one more SAD re-checking pass and full ref frames reload to dispatch ports of CPU) and UseSubShift true/false. For both 1080p and 4K with pel=4.

So combinations to test:

MSuper(pelrefine=false, pel=4)
MAnalyse(optSearchOption=5) (optSearchOption=6 require UseSubShift=1 in this case)
MDegrainN(MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.8, UseSubShift=1, IntOvlp=1)

and

MSuper(pelrefine=true, pel=4)
MAnalyse(optSearchOption=5 or 6)
MDegrainN(MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.8, UseSubShift=0, IntOvlp=1)

"v13 -optSearchOption=5 & IntOvlp=1
time=71.913s
v13 -optSearchOption=5 & IntOvlp=0
time=76.360s"

It is even strange - at my 'old' intel CPU of 9-series the (interpolated) overlap mode of MDegrainN is about 2 times slower. May be here something else limits speed so results are close or even much more complex overlap processing in MDegrainN even faster ? The overlap processing in MDegrainN is at minimum 2 passes over the frame - first pass accumulates partial weighted blocks (may be even in float or short 16bit at least) and second pass blends and convert to output bitdepth. Though I typically work with pel=4 only so it require either large RAM planes fetching or many sub-sample shift computing in CPU.

"CPU
v13 - Levels=1,"

Running MAnalyse onCPU is better to use all levels (so levels=0). It is typically only a bit slower but may catch good long MVs if there is fast movement in footage.

magnetite

22nd July 2022, 18:19

So I tried this new update with the OnCPU SAD calculation in this post (https://forum.doom9.org/showpost.php?p=1971553&postcount=142), and it still asks me for the Compute.cso shader file. Is that normal, or I thought it was supposed to be CPU only?

DTL

26th July 2022, 19:51

New version: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.14

Added mode 2 for IntOvlp for MDegrainN: It do not check real SAD of the interpolated blocks positions. So it is faster but may be lower in quality.

Fixed buffer overrun bug in InterpolateOverlap in MDegrainN.

Added AVX2 (8 bit output), SSE2 and SSE4 ( >8 bit output) second pass processing to output format into MDegrainN.

Disabled loading of shader file Compute.cso in optSearchOption=6 mode of MAnalyse.

Added different builds - for Win10 and later with DX12, for Win7 and others without DX12. Also some IntelC++ builds available for AVX2 CPUs.

It is possible to move (copy) interpolation of overlap to MAnalyse and also put its mode=1 computing of SAD to accelerator. But as I test with IntOvlp=2 at my CPU without re-check of SAD the speed benefit is small enough (about 12%). And that redesign need more time.

As current profiling shows the most of time for overlap processing in MDegrainN with high pel precision is in ref data fetching from memory (USS=0) or sub-shifting computing (USS=1). At the i5-9600 CPU both processes is about balanced. But at faster chips and AVX512 subshifting may be finally USS=1 mode will be visibly faster. Though it depends on cache size and speed and task size. The overlap blend computing and data conversion/storing is very fast already. So putting of post-overlap 16bit to 8bit conversion from C-ref to AVX2 makes almost zero speed addition. At least at my tested config.

IntelC++ SSE2 builds require some syntax redesign and development time so not included in this release. At i5-9600 the speed decreases in a sequence IntelC AVX2 -> MSVC AVX2 -> MSVC SSE2 as 3.55 -> 3.4 -> 3.2 fps with UseSubShift=1 and IntOvlp=1.

Update: Finally add new options descriptions to documentation. See updated file https://github.com/DTL2020/mvtools/blob/mvtools-pfmod/Documentation/mvtools2.html . Still no documented lots of limitations of new options. Like block size, bitdepth and so on (supported yes/no, SIMD accelerated yes/no). It looks need a table form.

DTL

3rd August 2022, 09:36

New version: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.15

Added diagonal interpolated overlap mode to MDegrainN of 2x blocks number to process. IntOvlp=3 with SAD re-check and IntOvlp=4 with interpolated SAD.
Added more error messages if non-compatible options provided for MSuper/MAnalyse/MDegrainN.
Updated documentation with new options. Updated file is https://github.com/DTL2020/mvtools/blob/mvtools-pfmod/Documentation/mvtools2.html
Added meander scan in the combined luma+chroma overlapped processing - may be better reuse of cached ref planes data.

Now the IntOvlp=3 is the typical everyday usage mode because it is much better in speed and very close to the quality as IntOvlp=1. Only about 30% slower in compare with no-overlap processing at i5-9600 with SO=5. It is close or equal to old BlockOverlap plugin operation.

Now for the future possible to make many combined modes of speed/quality:
1. Diagonal overlap search onCPU in MAnalyse.
2. Diagonal overlap is compatible with DX12ME and can double load of accelerator with 'real' search - may be used when host CPU speed is low and accelerator is underloaded.
Some internal flags may be added to MVs clip to indicate if it contain diagonal overlapped MVs data.
MDegrainN may also accept 2 MVs clips from 2 MAnalyse for original and diagonally shifted blocks seach data using any combinations of onHWAcc (single or dual) or onCPU search. To balance loading between host CPU and a single or multiply accelerators. As I see after-mining secondhand headless cards with may be equal to GTX1060 chip are now avaialble at Aliexpress at about $35..50.
Though the quality between real searched 2 MVs planes for diagonal overlap mode and second interpolated MVs set need to be examined - may be too low difference. May be only worth is host CPU too slow for MDegrainN and many free accerelators resources available.

anton_foy

3rd August 2022, 22:31

DTL

3rd August 2022, 23:08

" prefiltering/auxilary clip will not be required with your build. "

I still not found visible benefit of prefiltering if using 'interfiltering' of MVs inside MDegrainN with MVLPF processing. So it may be new (partial or complete) replacement of the old prefiltering method. I not made much testing. Though the MVLPF is still in the 2 simple implementations available and may be subject of complex (linear and or non linear) development in the future as both internal processing inside mvtools binary or as intermediate scripting (using MStoreVect/MRestoreVect and sample-accessing methods from script of new AVS+).

So if you use old scripts and not enable MVLPF processing - it is mostly probably prefiltering will make a benefit. If you enable MVLPF it is better to make new tests if prefiltering required or not or how much/etc.

"In that case how does it work?"

To enable internal MVLPF you need to adjust either MVLPFCutoff or MVLPFGauss from default values. Defaults are disabled state for compatibility with old scripts. Only one of 2 may work at the same time. It may be not best options naming. May be better to change MVLPF params to 'MVLPF_Type=None/Sinc(?)/Gauss/..." and 'MVLPF_Param1, MVLPF_Param2,...". So MVLPF_Type=None will clearly mean the processing is disabled (or one of possible filters is selected). So the options may be changed in the future in theory.
The best values of there settings may more or less depend on the footage and current user's preference. I currently use MVLPFGauss=0.8 for my encodings of HDTV 1080i documentaries.

Addition:
Old BlockOverlap plugin have additional 'kernel' control param:
http://avisynth.org.ru/blockoverlap/blockoverlap.html
kernel - blending window form (float, from 0.0 (uniform) to 1.0 (cosine kernel), default =0.5).
Blending mode with kernel=0 is the same as Avisynth command Overlay with opacity=0.5. In this mode the filter can not remove all block artifactes, but it halve them, try use some additional deblock filter.
Mode kernel=1.0 effectively smoothes blocks, but can produce some dot (circle) artefactes instead.

In current IntOvlp modes 3 and 4 full cosine-shaped window is used (equal to kernel=1.0 in BlockOverlap plugin). If it not best for some cases - the additional control param may be added. As I see in 2.7.45- versions of overlap windows - only full cosine window used and not some averaging with rectangular window. Though in old BlockOverlap plugin default is a mix of 0.5 cosine and 0.5 rectangular. May be it can be also subject to test by users and/or script developers.

DTL

9th August 2022, 21:11

Trying to use new builds with QTGMC for some processing of interlaced HD in intermediate progressive form I found next issues:
1. DX12_ME modes return some minor error about motion clip is too small. So currently only onCPU with optSearchOption=1 is max possible new processing modes with QTGMC.
2. Using default for QTGMC block size of 16 crashes MAnalyse onCPU with out of frame buffer memory access - may be some more padding may help in MSuper or some more vectors limiting check/add in MAnalyse. Only block size of 8 is working. May be limiting were damaged in some redesign from 2.7.45 version - need more debug.

Also as I see the main processing in QTGMC is based in MCompensate function. The some denoising is based on MDegrainX so it is easy enough to rewrite to MDegrainN (though the too low tr is typically not effective with MVLPF).
After looking into MCompensate of 2.7.45 mvtools I see it is very outdated and some redesign is planned:

1. Currently MCompensate is based on hard thresholding by thSAD (same as new MDegrainN with wpow=7). So it may be good to make some smoother rolloff of weighting of the ref block using same weighting functions as in MDegrain. With same wpow new param as with MDegrainN
2. The motion-adaptive adjustment of weighting depending on MVs length and coherency may be added but need testing if it is good for QTGMC activity.
3. MVLPF for 2 frames MCompensate is about not effective at all (or require some redesign of MCompensate for requesting big enough set of frames from MAnalyse in multi-mode and it will load ME-part of mvtools significantly - though may be only caching of MVs-frames for reusage in both deinterlacing and MDegrainN denoising may help to decrease load).
4. Adding usage of sub-sample shifting is the simplest task and will be quickly.
5. Main addition - to support new interpolated overlap modes with non-overlapped search in MAnalyse the same interpolated overlap processing need to be added to MCompensate. To not make many copy parts of program from MDegrainN to MCompensate it may be good to make some 1 for all usage classes/functions may be. So the architecture of old and simple MCompensate function need to be redesigned to the same as current MDegrainN (with array-based of MVs data storage - not Fake object with accessing of VECTOR data as Block class data).

MysteryX

11th August 2022, 02:40

MVTools2 with hardware acceleration? Great!

Does this implementation have the potential to work on Linux or it's Windows-only? DirectX 12 doesn't look very Linux-friendly... which uses Vulkan. Is Vulkan API suitable for this work? It's generally faster for most tasks.

takla

11th August 2022, 03:44

@MysterX
Read post #129 (https://forum.doom9.org/showthread.php?p=1969845#post1969845) & #130

DTL

11th August 2022, 14:37

AMD provide ME from DX11 and windows 7 via custom LiquidVR API. But it may be still windows only and not Linux compatible. Though DX11 may be easier to emulate at Linux if AMD provide all needed drivers.

In better case Linux community need to define ME API for applications and provide DDK for hardware developers (intel/amd/nvidia) to develop required drivers support for this API. May be it will also helps to UNIX developers of x264 MPEG encoder to have some help from hardware ME accelerator too.

May be Linux developers may add ME (and MPEG encode/decode) to Vulkan API and ask drivers developers to support it.

DTL

24th September 2022, 19:38

New release: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.16

Added SuperCurrent param to MAnalyse, clip param Allow to provide differently processed clip as current source for search. May be useful to use with prefiltering usecases.

Added SearchDirMode param to MAnalyse, int param. 0 - search standard direction (current frame to ref frame MVs). 1 - reverse search.

Fixed bug with not-selecting combined luma+chroma processing modes when thSADC=thSAD (and thSADC2=thSAD2).

Added Multi-Pass Blending mode in MDegrainN. New params:
MPBthSub, int (10), threshold for subtracted blocks.
MPBthAdd, int (20), threshold for std blended blocks (additively).
MPBNumIt, int (0), number of iterations. 0 - MPB processing mode not used.
MPB_SPC, float (1.5), multiplier and divider for weight adjustment at each iteration if SAD of curent blending result vs subtracted or ref block is above threshold.

Currently MPB mode only supported for 8bit formats.
Current typical usage params for MDegrainN (onCPU search for example):

tr=10

super=MSuper(last, mt=false, chroma=true, pel=2)
multi_vec=MAnalyse(super, multi=true, blksize=8, delta=tr, search=3, searchparam=2, overlap=0, optSearchOption=1, optPredictorType=0, chroma=false, mt=false)
MDegrainN(last,super, multi_vec, tr, thSAD=150, thSAD2=140, mt=false, wpow=4, thSCD1=400, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16,
MVLPFGauss=0.9, thMVLPFCorr=50, UseSubShift=1, IntOvlp=3, MPBthSub=10, MPBthAdd=20, MPBNumIt=2, MPB_SPC=1.5)

The MPB params are subject of long experiments for best results (zopti optimizer may be highly required). Too much iterations may quickly decrease 'denoising'. Typically enabling MPB require to set thSAD to a bit higher value to keep denoising at static and flat areas good enough. Example - from 110 to 150 with default MPB params and 2 iterations. So each iterations weight adjustment param MPB_SPC is defaulted currently to high enough value of 1.5. Also MPB mode make high CPU computing load (most data is cached but it require many additional SAD computing and some block subtraction from total blending result). So speed penalty from enabling MPB mode is enough even with 2 iterations. May be lower in the future with AVX2/AVX512 SAD and block subtract future functions to develop. Currently uses only SSE2.
Possibly max quality MPB with luma + chroma SAD usage is only supported in combined luma+chroma processing (if thSADC=thSAD, thSADC2=thSAD2 - default mode). Separated planes processing only use luma or current chroma plane SAD so may give worse results.

MPB mode operation:
For each iteration:
1. Calculate subtracted blocks (current blend result with current blend weights minus single block for speed, should be equal or very close to partial blend with single block excluded from blend, partial blends require more operations in compare with subtraction).
2. Calculate SAD of current blend result vs subtracted blocks array (vector) and input blocks array (src + all refs).
3. Calculate average SAD for subtracted and standard blended blocks arrays.
4. If current block SAD different from average above threshold (MPBthSub and MPBthAdd) - adjust its weight to decrease (for subtracted blocks) or increase in MPB_SPC ratio.

The idea is to create more equal weight field for all blocks in blending pool. So if block too badly contribute to blending average its weight is decreased and if blocks looks more equal to blended average its weight increased. No MVs is analysed here (yet).

MPBthSub and MPBthAdd expected to be about thSAD/10 and MPBthAdd > MPBthSub about twice (depend on tr-param and current degraining between input and output) because SAD of currently degrained block vs single subtracted is significantly lower in compare with degrained block vs input noised ref or src.

Too high values of MPBthSub and MPBthAdd disables weight adjusting in MPB (for subtracted or incoming blocks). So it may be used to check result from separated parts of processing. Setting both MPBthSub and MPBthAdd too high - disables MPB processing practically but still make processing speed lower.

anton_foy

26th September 2022, 14:33

DTL many thanks for this! So many new features and parameters to understand yet.
As I use mostly motioncompensated TemporalSoften rather than mdegrain, my question is if this latest version is suitable for hbd (16bit) using the motion compensation bit? What parameters could you suggest to use and or change in the TemporalSoftenMC (TSMC) script?

####### modded to adj. smaller mrecalculate blocksize
function TSMC(clip input, int "tradius", int "mthresh", int "lumathresh", int "blocksize", int "rBlock",clip "auxclip", bool "pref", int "Y", int "UV")
{

pref = Default(pref, true)
Y = Default(Y, 3)
UV = Default(UV, 2)

t=Defined(tradius)
tradius=t ? tradius : 6
# temporal radius-number of frames analyzed before/after current frame.

m=Defined(mthresh)
mthresh=m ? mthresh : 180
# motion threshold-higher numbers denoise areas with higher motion.
#Anything above this number does not get denoised.

l=Defined(lumathresh)
lumathresh=l ? lumathresh : 255
# luma threshold- Denoise pixels that match in surrounding frames.
#255 is the maximum and default. 0-255 are valid numbers.
#Also adjusts chroma threshold.

b=Defined(blocksize)
blocksize=b ? blocksize : 32
#larger numbers = faster processing times

rBlock = Default(rBlock, 4)

chroma = UV == 3
aux=Defined(auxclip)

w = width(input)
h = height(input)
isUHD = (w > 2599 || h > 1499)
nw = round(w/2.0)
nh = round(h/2.0)
inputA = aux ? auxclip : input
inputA = isUHD ? inputA.ConvertBits(8,dither=-1).BilinearResize(nw+nw%2, nh+nh%2) : inputA

super = MSuper(input, pel=1, hpad = 0, vpad = 0, chroma=true, mt=true, levels=1)
superfilt = MSuper(inputA,pel=1, hpad = 0, vpad = 0, chroma=true, mt=true) # bug can't disable chroma otherwise luma isn't processed

vmulti = Manalyse(superfilt,multi=true,delta=tradius,temporal=true,truemotion=true,blksize=blocksize,overlap=blocksize/2, mt=true, chroma=true)
vmulti2 = Mrecalculate(superfilt,vmulti,thsad=mthresh,truemotion=true,tr=tradius,blksize=rblock,overlap=rblock/2, mt=true, chroma=true)
vmulti2 = isUHD ? vmulti2.MScaleVect() : vmulti2
mocomp = Mcompensate(input,super,vmulti2,thsad=mthresh,tr=tradius,center=true,mt=true) # recursion=50 is bugged
dnmc = mocomp.temporalsoften(tradius,lumathresh,lumathresh,15,2)
dec = selectevery(dnmc,tradius * 2 + 1,tradius)
Y != 3 ? input.mergechroma(dec) : dec }

You mentioned here (https://forum.doom9.org/showthread.php?p=1974137#post1974137) automatic adjustment according to grainlevels/noiselevels.
This I tried with my noise detection to feed to scriptclip that outputs a dynamic mask and also now experimenting using temporalsoften dynamically adjusted (tradius and lumathresh) in ScriptClip for TSMC.

DTL

26th September 2022, 18:13

"if this latest version is suitable for hbd (16bit)"

Currently it is not tested for 16bit. At least the input with MPB-processing enabled should be 8bit (only 8bit component subtraction functions are currently created) and for output only 'lsb' mode compiled (may be old AVS format of 'lsb subplane'). So even for out16 (with AVS+ 16bit plane) it need to be slightly changed and recompiled.
Also the MPB with blocksize luma down to 4x4 and so chroma of 1 sample (blocksize 1x1) for YV12 is not tested - may be better to use YV24 format with both luma and chroma blocksize of 4x4 (it currently work for YV12 and blocksize 8x8 so chroma is 4x4).

About auto-adjust of main params like thSAD - I think about some simple ways to do it inside MDegrain but still not started. For current ideas it may be close to the SCD processing like calculating average of SAD of all blocks in the current frame or all valid blocks in tr-scope and apply some correction multiplier or addition to it (or some more complex function) to calculate thSAD and thSAD2 (and thSCD1) values internally and per each output frame.

"What parameters could you suggest to use and or change in the TemporalSoftenMC (TSMC) script?"

At first it not use MDegrain at all. So MCompensate (also used in widely used QTGMC) is still not changed at all from 2.7.45 version. We have some new features of MDegrain now to move to MCompensate too - at least interpolated overlap modes. May be other new additions too.
As I see it uses MScaleVect for better speed for UHD - it may be not best for quality so you can disable it if required better quality.
The only that can possibly help with speed a bit for this script with current version is using hardware search for MAnalyse (use optSearchOption=5 or 6) and set blocksize to 8 (16 may be still buggy but may be recommended to try for multi-pass search with Mrecalculate later).

The quality difference between degrain approaches of
1. Current MDegrainN
2. MCompensate + temporalsoften() + selectevery()
is good to be evaluated and if approach 2 is better at some scenes I think it may be added as one more mode to MDegrainN (may be something like additional blending mode or blending param for non-linear blending with lumathresh) so you can skip this script (or change sequence of MCompensate +temporalsoften +selectevery to single MDegrainN call) and possibly get better speed with single MDegrainN.

For testing you can try to replace lines

mocomp = Mcompensate(input,super,vmulti2,thsad=mthresh,tr=tradius,center=true,mt=true) # recursion=50 is bugged
dnmc = mocomp.temporalsoften(tradius,lumathresh,lumathresh,15,2)
dec = selectevery(dnmc,tradius * 2 + 1,tradius)

with

dec=MDegrainN(input,super, vmulti2, tr=tradius, thSAD=mthresh, thSAD2=mthresh-10, mt=false, wpow=4, thSCD1=400, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16,
MVLPFGauss=0.9, thMVLPFCorr=50, MPBthSub=10, MPBthAdd=20, MPBNumIt=2, MPB_SPC=1.5)

Though it is not directly compatible for mthresh param and require new adjustment. Also 8bit input/output formats only.

Also for quality it is better to use pel 2 or even 4 in MSuper and set hpad and vpad to blocksize or larger if crashes happen.

anton_foy

26th September 2022, 23:43

Lovely I will try this out tomorrow. I did not have alot of good results with mdegrain but I did see your tests and was impressed! Also mscalevect with blocksize=8 and mrecalculate and "optSearchOption=5" I will try for manalyse/mcompensate. Thanks!

DTL

27th September 2022, 09:39

It looks current 'starting' MPB params of MPBNumIt=2 and MPB_SPC=1.5 may be too high for SD noisy sources like VHS captures or may be other too noisy sources. So it cause significant decreasing of 'denoising' over all frame even with high enough tr and thSAD params. For such sources the MPB_SPC may be as low as 1.02..1.05 (it expected to be > 1.0) and MPBNumIt to 1. It is a bit unexpected low values so may be in future versions the internal math may be changed for more expected adjustment in range of about 1.1..2.0. Though leaving some residual noise may be used instead of 'debanding' after typical too clean degrain.

Also thCohMV param depends on pel about pel*4 so thCohMV=16 is my typical value for pel=4. If you use pel=1 it may be lower to about thCohMV=4.

Addition from 02.10.2022: It was found that using hardware motion search with NVIDIA GTX1060 card looks like sometime produces significant MVs errors and it cause significant details lost with MDegrainN processing. It is more visible if using 'high enough' tr-values like 10. May be other chips of new NVIDIA families or AMD produces better result. So currently for highest quality work only onCPU MAnalyse can be recommended.

DTL

6th October 2022, 18:55

New test release https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.17

Added MPB_PartBlend param to MDegrainN to check real partial blend vs subtraction of block. Default false (use subtraction). If set to true - perform full block blending with test block removed (slower but a bit more accurate in SAD). Subtraction method with integer 8(16 immediate) bit processing may still give up to +-1 error for each sample may be due to integer rounding errors and total SAD difference from real partial blending for even small 8x8 block of 64 samples total may be comparable with low threshold value. So for best possible quality MPB_PartBlend=true is recommended (may be significantly slower).

Separated subtractive and additive coefficients to different params MPB_SPCsub and MPB_SPCadd for better flexibility at experiments of finetuning.

Added isMVsStable function to check if MVs in current tr-pool for current block are enough coherent (stable) - to try to make MPB processing only at areas with stable enough motion search in tr-pool of frames. New MDegrainN param MPBthIVS - threshold to compare current calculated measure of non-stabilily of MVs (sum of accelerations multiplied to sum of vectors angle difference). Param is internally scaled to squared pel value but may significantly depend on tr and other settings. To help adjust this threashold - use IVS-mask display with showIVSmask=true.

Added protection to MDegrainN against too low padding (now need to be at least blocksize in size) and not equal temporal radius param for MAnalyse and MDegrain - display error messages instead of corrupted output.

Added showIVSmask param to MDegrainN to mark blocks detected as stable enough MVs with black. Default false. Black blocks with are detected as ready for MPB processing.

Added mvmultivs param to MDegrainN as option to provide separate MVclip with different search source or options for IVS mask creating. Provided clip must be equal to mvmulti in block number, overlap mode and recommended to use truemotion=false preset of MAnalyse to show noise-moved blocks as best as possible, not recomended to make from prefiltered clip. mvmulti clip may use any required params for best denoising, can be created from prefiltered source and so on.

Example of using mvmultivs clip with separate MAnalyse settings:

tr=10

super=MSuper(last, mt=false, chroma=true, pel=2)

multi_vec=MAnalyse(super, multi=true, blksize=8, delta=tr, search=3, searchparam=2, optSearchOption=1, chroma=true, mt=false)
multi_vec_vs=MAnalyse(super, multi=true, blksize=8, delta=tr, search=3, searchparam=2, optSearchOption=1, truemotion=false, pnew=0, pzero=0, chroma=true, mt=false)

MDegrainN(last,super, multi_vec, tr, thSAD=200, thSAD2=190, mt=false, wpow=4, thSCD1=400, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50,
UseSubShift=1, IntOvlp=3, MPBthSub=5, MPBthAdd=20, MPBNumIt=2, MPB_SPCsub=0.5, MPB_SPCadd=1.5, MPB_PartBlend=false, MPBthIVS=1500, showIVSmask=false, mvmultivs=multi_vec_vs)

Typically onCPU MAnalyse with truemotion=false provide significantly better MVs for IVS-mask generation in compare with hwAcc search with NVIDIA GTX1060 but the processing is slower.

The idea of IVS-masked MPB processing is not to increase noise on non-detailed areas (like out of focus, clear sky and so on). Only enchance details (and also some noise) at areas with at least some details detected by motion search engine and existance of some temporal coherence of the MVs in current tr-scope around current frame. Current IVS-mask generation engine is not final design and may be subject to change in future.

If no mvmultivs clip is provided - single mvmulti is used for all operations (for example in hardware-only search of MVs). Though if MVs clip is created from anti-noise pre-filtered source the quality of IVS-mask may be more degraded.

Addition: The MPB processing in this build also significantly redesigned: Old processing tried to adjust initial weights array from old DegrainWeight function based on ratio of block SAD vs current thSAD. New design trying to create new weight-array from initial equal-weight condition (equal to wpow=7). And after 1 or more iterations of weights aligning it apply finally new calculated weight array to initial using proportional scaling.

Addition2: Currently MPBthIVS param is auto-scaled internally to pel*pel of the 'main working' mvmulti clip because acceleration part of metric depends on square absolute value of MVs coordinates differences and they are scaled with pel value. But to make processing faster if separate MVclip provided for IVS-mask it may be generated from lower pel-mode of MAnalyse (like pel=2 or pel=1) and in the future versions it looks good to make autoscaling to pel value of the mvmultivs clip (if provided). Though it still may significantly depends on other params like tr-value so still not auto-corrected completely anyway and need to be checked before starting of production processing after many other params adjustment.

DTL

9th October 2022, 18:30

New release https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.18

Added SSIM metric in MAnalyse and MPB MDegrainN processing. Now possible dissimilarity metric flags (as bit mask):
bit 0 - SAD,
bit 1 - SSIM luma only,
bit 2 - SSIM contrast and structure.

Examples:
SAD only = 1
SSIM luma only = 2
SSIM contrast and structure = 4
Full (standard) SSIM = 6
SAD + SSIM contrast and structure = 5

Selecting of dissimilarity metric supported only in part of optSearchOptions and optPredictorType of MAnalyse. For example optSearchOption=2 have hardcoded SAD in SIMD and can not be switched.

New param for MDegrainN - MPB_DMFlags. Integer any of dissimilarity metric bitmask, default=1.
New param for MAnalyse - DMFlags Integer any of dissimilarity metric bitmask, default=1.
New param for MRecalculate - DMFlags Integer any of dissimilarity metric bitmask, default=1.

Current release have only C-reference partially float32 SSIM calculation functions (best precision and support all blocksizes and bithdepths) so very slow. For quality check mostly. Current processing speed degradation of MDegrainN MPB using SSIM about 2 times. MAnalyse usiing SSIM about 4 times. Expected good benefit from SIMD versions in the future (for some limited number of block sizes and bitdepths).

Example of interleaved frames with different dissimilarity metric for MPB processing:

ssim=MDegrainN(last,super, multi_vec, tr, thSAD=250, thSAD2=240, thSADC=240, mt=false, wpow=4, thSCD1=400, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50,
UseSubShift=1, IntOvlp=3, MPBthSub=2, MPBthAdd=2000, MPBNumIt=2, MPB_SPCadd=1.5, MPB_SPCsub=0.6, MPBthIVS=1500, showIVSmask=false, MPB_DMFlags=6).Subtitle("SSIM")
sad=MDegrainN(last,super, multi_vec, tr, thSAD=250, thSAD2=240, thSADC=240, mt=false, wpow=4, thSCD1=400, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50,
UseSubShift=1, IntOvlp=3, MPBthSub=2, MPBthAdd=2000, MPBNumIt=2, MPB_SPCadd=1.5, MPB_SPCsub=0.6, MPBthIVS=1500, showIVSmask=false, MPB_DMFlags=1).Subtitle("SAD")
Interleave(ssim, sad)

The SSIM metric produce much more great deviation of metric value so if using in MAnalyse and MDegrain the thSCD1 param of MDegrain must be significantly adjusted to higher value (like from 400 to 15000 or higher). Check MShow mean-SAD output to see expected value. Though thSAD can be about the same as with SAD metric.
Combinations of SAD+SSIM is additive (so thSAD may be about 2 times higher if used).
Combinations of SSIM-luma and SSIM-contrast and structure are multiplied (to form 'standard SSIM'). So enabling standard SSIM may not require significant thSAD adjustment.

Current conversion of SSIM into something close to SAD computing engine:

Dissimilarity metric = (1-SSIM) * maxSAD/2.
Where maxSAD = (3 * nBlkSizeX * nBlkSizeY * (pixelsize == 4 ? 1 : (1 << nBPP)))

So -2 SSIM (totally different blocks) shoud reach maxSAD. Though it looks the intermediate values of SAD and SSIM with some intermediate blocks dissimilarities are very different. May be it good to add some power-function to align a bit better in the future. Though the much higher gain of SSIM around low SAD may helps to MDegrainN and MVs search better.

Expected other metrics for addition in the future like CW-SSIM and others. Currently new intermediate helper class DisMetric is used so addition of new metrics is now much easier. Though the new class add some overhead at calling of metric function and selection so performance of SAD only metric may be visibly degradred.

anton_foy

10th October 2022, 08:05

Thanks DTL! Just a thought here, would it be possible to implement in mvtools a way to analayse and detect the grain/noise amount (using gpu?) for it to automatically and dynamically adjust tr/thsad as an option? I guess it would be a runtime thing that will slow down alot though.

DTL

11th October 2022, 15:42

I still not have good idea how to implement auto-thresholding. Current single mvmulti clip with low tr may be not applicable and/or too slow for something like scene-wide lookahead. So may be add one more analysis mvclip (with single frame temporal stepping) to MDegrainN so it can try to call many next frames fast enough to look from current to next scene-change detect point and calculate something like mean-sad over total scene-wide frame numbers and apply it.

Currently still many more ideas to implement.

It the post about previous version it is not directly mentioned but with optSearchOption=6 it should now able to apply different metric to onHWAcc searched MVs. So it may be real reason why DX12 ME API do not output any metric with MVs - it may be up to the enduser to select and calculate any similarity/dissimilarity metric if required. For other search options the metric is either limited to search metric (like optSearchOption=0 and 1) or limited to SAD only (2,3,4,5). So I think it require more option to MAnalyse to select search and output option to select different metric (the single metric may be not best for search on all cases I think) and MVs consumer filter may be work better with different metric provided. As I see in the article https://videoprocessing.ai/metrics/ways-of-cheating-on-popular-objective-metrics.html there are lots of different metrics available now and good to check how is they work with denoising or other mvtools activity.
Also the computer shader for optSearchOption=5 need to be redesigned to use many more metrics selected by DMFlags (or DMFlagsOutput) option to MAnalyse. So the external to CPU compute accelerator will offload more computing work of computing much more complex metrics in compare with simple SAD metric.

anton_foy

13th October 2022, 16:51

DTL

13th October 2022, 17:16

Currently IVS-mask is to decrease level of details blurring on 'important' areas of frame while keeping degraining at no detailed areas good enough. As I see from latest tests the MAnalyse produces significantly different MVs with SAD and SSIM metrics at the non-detailed areas. So it may be additional method to create IVS-mask from even single pairs of frames search - it is faster in compare with many frames analysis for MVs difference for block. So in the new versions the MDegrainN may have 2 additional inputs of MVs clips and separate tr-value for this part of processing. Like mvmultivs2 and MPB_IVStr params. So the mvmulti clips may be created with much smaller tr and less CPU load, may be down to tr=1. Also it may help to reduce issue of current IVS-mask creation algorithm - after scene change it will not produce any good mask at about tr/2 or more number of frames.

MAnalyse with SSIM metric works about 2 times slower in compare with SAD metric even with AVX2 calculation of SSIM (getting sigmas and means in 16..32 bit integer subtraction/addition/multiplication and final processing in float32 including full precision square root). So it is good to decrease amount of processing with SSIM metric if possible for better performance.

anton_foy

13th October 2022, 21:46

Very interesting with the new IVS-mask. So in the new versions the MDegrainN may have 2 additional inputs of MVs clips and separate tr-value for this part of processing. Do you mean like conditional filtering or a more seamless/smooth transition? I was thinking something like this:

Tr1 = MdegrainN(last,super1,multivec, tr=1, thsad=200...)
Tr6 = MdegrainN(last,super2,multivec, tr=6, thsad=400...)
Mask = 'scriptclip the mask for eg. Blackness() with opacity changing"
Mt_merge(tr6,tr1,mask,luma=true,u=3,v=3)

Changes dynamically between tr1 and tr6 (min/max).

DTL

13th October 2022, 23:49

It is all done inside single blending engine for better speed. Any sequence of 'filters' in AVS takes lots of RAM for caching inbetween filters and for each thread and also the second main reason of single blending engine - it can load once all src and all ref frames blocks data from RAM to CPU L1D cache and make many processing not touching host RAM any more.

If you chain some filters in AVS each filter make full frame scan in some order so data is many time loaded from host RAM into CPU and it is much slower.

So for performance reasons it is better to make all processing for each input/output block in single (may be very complex) blending engine and not write some intermediate results in AVS-clips RAM objects and making final blend/merge in separate objects. Unfortunately AVS-filters can not interoperate image data on small chunks like blocks or samples - only total frames are requested by downstream filters from upstream. And total frame request invalidate most of caches (at least L1D as the fastest but only about <100 kBytes in size).

If IVS-mask is useful for some other scripting it can be special mode of MDegrainN to output this mask in some more nice form like grayscale 256 levels or just 1 black and white. Currently its output is designed only to check of mask placing over the frame data so it is not clean from image data.

I think for better understanding how old and current MDegrainN (MAnalyse + MDegrainN) is working and where many of adjustments params are passing it is good to create some structure scheme. May be will do in some form for documentation.

One sad issue with 'direct output' of IVS-mask: In better quality 'overlapped' mode of MdegrainN it is also generated for each block of 'overlapped blending space' and the blocks count in any overlapped mode are > blocks count in not overlapped mode (simple tesselation of frame to width/blocksizeH and height/blocksizeV number of blocks). So the IVS-mask in simple single frame form can not be outputted for good quality overlapping processing modes. Or some way of output of overlapped mask need to be designed (like doubling or quadrupling of output framerate to output each part of mask in separate frame). It is much more complex. It is also reason of using single blending engine instead of attempt to transfer mask via current AVS scripting interface based mostly on simple 'clips' objects of fixed frame size and may be expecting masks to be produced in same size.

anton_foy

18th October 2022, 15:16

https://forum.doom9.org/showthread.php?p=1962308#post1962308

https://forum.doom9.org/showthread.php?p=1973516#post1973516

Possibly for this thread or not but since it is about improving mvtools I guess it is suitable.
Using RIFE for/in mvtools would be possible?

DTL

19th October 2022, 06:18

Well - RIFE at GTX1060 can run for 1080i degrain at simulating of MDegrain1. Run at about 2 fps at only 3 AVS+ threads fit 6 GB RAM of HWA. 4 and more threads typically throw vkmemoryallocation errors. Also Win10 GPU memory graph hit max value. The most of HWA load is Compute1 about 99..100% in Win10 GPU performance counters.

For 1080i source the script was

FFmpegSource2("src.mxf")

yadifmod2(mode=1)

ConvertToRGB(interlaced=false)

ConvertToPlanarRGB()
ConvertBits(32)
src32=last

even=SelectEvery(2,0)
odd=SelectEvery(2,1)

even_d=RIFE(even)
odd_d=RIFE(odd)

even_d=SelectEvery(even_d,2,1)
odd_d=SelectEvery(odd_d,2,1)

den_int=Interleave(even_d, odd_d).Trim(1,0)

src32_trim=Trim(src32,1,0)

den_int2=Average(den_int, 0.666, src32_trim, 0.3333)
den_int2.ConvertToRGB24().ConvertToYV12()

SeparateFields()
SelectEvery(4,0,3)
Weave()

Prefetch(3)

It is not clean field-based denoise of interlaced (uses intermediate 50fps yadif deinterlaced form) but work as interlaced input and interlaced output. Also it is not clean test for quality because yadif simple deinterlacer adds its own distortions.

The MPEG x264 filesize saving with crf=18 encode is still very visible - the input noised source via same yadif deint->reint path give about 17.5 Mbps and denoised only 7.8 Mbps average.

The only good for users - it do not have any user adjustable params like thSAD and 10..20+ more tweaking params of current mvtools.

Separated RIFE degrain1 function may be like this

function RIFE_degrain1(clip src)
{
ConvertToRGB(src)
ConvertToPlanarRGB()
ConvertBits(32)
src32=last
even=SelectEvery(2,0)
odd=SelectEvery(2,1)
even_d=RIFE(even)
odd_d=RIFE(odd)
even_d=SelectEvery(even_d,2,1)
odd_d=SelectEvery(odd_d,2,1)
den_int=Interleave(even_d, odd_d).Trim(1,0)
src32_trim=Trim(src32,1,0)
den_int2=Average(den_int, 0.666, src32_trim, 0.3333)
return den_int2.ConvertToRGB24().ConvertToYV12()
}

Uses Average plugin - http://avisynth.nl/index.php/Average , may be replaced to something internal AVS+ weighted blending with mask like Layer() or Overlay(). As tr=1 degrain it skips 1st frame of source clip for more simple form of function - output is input-1 frame count.

ChaosKing

19th October 2022, 10:05

Not sure if it was asked here, but would DirectStorage be of any use for mvtools? Or maybe other plugins?

https://devblogs.microsoft.com/directx/directstorage-1-1-coming-soon/

As far as I understand a constant CPU<->GPU transfer can be a big bottleneck. DirectStorage solves this by communication directly to the SSD/NVME storage without touching the cpu.

DTL

19th October 2022, 10:25

DirectStorage is mostly about loading large resources from storage to GPU. Like GPU can issue load commands directly to storage and free CPU cycles to do some useful computing. Most of mvtools processing is about fetching RAM resources and math computing. DirectStorage may be 'environment' feature for AVS (mostly source plugins) because it loads data from storage into AVS environment. But as I see AVS developers not like to make AVS core windows-dependent. So it may be some feature for some source plugin to load file from storage directly to GPU for decompresion and download decompressed frames to host RAM as AVS resources (frames of clip). It may save some CPU cycles but may be not many.

May be some developer can make DirectStorage-using DirectShow plugin so AVS can use it via existing DirectShowSource input.

About RIFE denoising on some long footage test: With default settings it looks have too poor scene change detection and also create flickering on some repeating texture patterns at architecture. May be require model manual tweaking/selecting and params adjusting like scene change.

DTL

24th October 2022, 20:35

New test release: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.19
Added MPB_MVlth param to MDegrainN. Limit allowed length of MV for weight correctionby MPB processing. Can decrease possible 'ghosting' with too extreme MPB_SPCsub/add params values. Not scaled by pel currently. Recommended values - about 2..3 squared pel value. Valid working range from 0 to squared frame size (unlimited). Zero may be disable MPB weight adjusting completely.

Added reduced tr blending mode for MPB controlled currently by MPB_SPCadd > 10.

Added MPBtgtTR param to MDegrainN:
In standard MPB mode controls initial number of weights used for calculate initial blend estimation (may be 0 - only current block used).
Valid range - from 0 to tr.
In reduced weights MPB mode - controls number of ref frames (2 * tr) used for blending without any other weights adjustment by MPB.

Added MPB_DMFlags=64 flag. Uses covariance metric only. Can be used only with MDegrainN.

Added VIF (DWT- based) metric. Controlled by 0x10 (VIF-Approximation) and 0x20 (VIF-Edges) flags. 16 and 32 decimal. Full VIF (VIF-A * VIF-E) is DMFlags=16+32=48. Can be used in both MDegrainN MPB flags and MAnalyse.

Current possibly best settings for processing:

MDegrainN(last,super, multi_vec, tr, thSAD=250, thSAD2=240, thSADC=240, mt=false, wpow=4, thSCD1=400, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9,
thMVLPFCorr=50, UseSubShift=1, IntOvlp=3, MPBthSub=5, MPBthAdd=5, MPBNumIt=3, MPB_SPCadd=3, MPB_SPCsub=0.3, MPBthIVS=1500, showIVSmask=false, MPB_DMFlags=64, MPB_MVlth=8, MPBtgtTR=tr)

Example of reduced tr blending mode without MPB weights adjusting:

MDegrainN(last,super, multi_vec, tr, thSAD=250, thSAD2=240, thSADC=240, mt=false, wpow=4, thSCD1=400, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9,
thMVLPFCorr=50, UseSubShift=1, IntOvlp=3, MPBNumIt=1, MPB_SPCadd=11, MPBthIVS=1500, showIVSmask=false, MPBtgtTR=tr-5)

Blocks with tr cut to tr-5 controlled by IVS mask.

With enough aggressive settings like MPBNumIt=3, MPB_SPCadd=3, MPB_SPCsub=0.3 it is recommended to use low enough MPB_MVlth=8 (about 2 squared pel value) or the small moving objects may start to have 'ghosts' . This 'ghosting' feature of MPG is not perfectly fixed (and even not completely debugged) so added limitation of max MV length for weight adjustment. Current idea of this ghosting: Some blocks far enough from moving objects with errors MVs got some non zero weight in total blending pool of blocks and MPB multi-pass greatly amplificated weight of such blocks as it have some details and they become visible at output. So they looks like exist in standard mvtools/MDegrainN processing but with typical low weight are invisible.

Current MPB_DMFlags=64 uses covariance only metric that is inverted to typical metrics (covariance increases with increasing blocks similarity) and do not have good max expected values to make easy inverting to align with other dis-similarity metrics so it currently implemented outside DisMetric class and uses separate processing function in MDegrainN only. Currently with enough aggressive MPB settings it looks make best result on details keeping (and even look like produce some 'sharpening' effect).

Just a thought here, would it be possible to implement in mvtools a way to analayse and detect the grain/noise amount (using gpu?) for it to automatically and dynamically adjust tr/thsad as an option? I guess it would be a runtime thing that will slow down alot though.

Currently with progress of multi-pass blending and in the future the multi-frame multi-pass blending the importance of thSAD will be less and less important. In this processing the weights of blocks to blend are many times adjusted by many functions. So initial thSAD is for first approximations and to skip too bad blocks. Same as you see the RIFE do not have thSAD or close adjustment and process all blocks similarity automatic.

About offloading of MDegrain MPB and MFMPB to accelerator - it may be possible but in some far future. With the increasing computing complexity of 'self-adjusting blending' the penalty for upload and download result to and from accelerator may be not very big even for per-block processing. Better to upload full set of frames and download resulted frame only but it require more complex compute shader design. Also I do not have good way to debug compute shaders now with remote debug. Though with MDegrainN the host do not need DX12-ME features for run separate compute shader and may be local debug will work on much poor accelerator that exist in some way on my development host with visual studio.

Currently the MPB still not very slow with much more computing in compare with old single pass blending mode with thSAD-defined weights only because it uses very small blocks data once that is good cached to L1D cache of CPU and no more disturbs very slow host RAM. So it good benefits from multi core CPUs and fast cores even with slow enduser RAM with low memory channels and poor performance cache/memory controllers of endusers cheap CPUs.

anton_foy

5th February 2023, 17:46

@DTL, with your versions of mvtools I can run scripts with mdegrain scripts but not motion compensated scripts with MCompensate. It just stops, aborts without error. Do you know why?

Edit: sorry I just read this (https://forum.doom9.org/showthread.php?p=1972963#post1972963)
But does it mean your version does not even support the mcompensate at all?

Also here (https://forum.doom9.org/showthread.php?p=1976228#post1976228) you mention the problem with thSCD but if I do not use scenechange detection at all I put it to zero? Will it make it easier for mvtools to denoise?

For HBD problems mentioned here (https://forum.doom9.org/showthread.php?p=1961840#post1961840) could you just use two MSuper? Does MAnalyse really degrade quality too much if 8-bit yv12?

Something like this:

# HBD source

Sup8 = convertbits(8).converttoyv12()
Sup = source.MSuper(levels=1)
Sup8 = source8.MSuper()
multi_vec_vs = sup8.MAnalyse(multi=true, blksize=8, ...).convertbits(source)
MDegrainN(last, sup, multi_vec_vs, ...)

Selur

5th February 2023, 20:02

Any hope for this getting Vapoursynth support?

DTL

5th February 2023, 22:16

"does it mean your version does not even support the mcompensate at all?"

MCompensate should be the same as in 2.7.45 version from pinterf. But blocksize 16x16 may be broken (or bitdepth >8). Also blocksize of 16x16 do not run well at my GTX1060 with hardware search (in MAnalyse) and cause some crash that remote debugger can not break in. So I use only block size of 8x8. It is possible with scripting to feed half frame sized to MAnalyse with blocksize 8x8 and use MScaleVect to map output MVs it to full frame size with blocks of 16x16 if you want to try. It may be also faster for large frame sizes.

"thSCD but if I do not use scenechange detection at all I put it to zero?"

Disabled thSCD is about maxSAD (and it may be internallly silently clipped to maxSAD). So to disable scenechange detection you need to set thSCD to some verybig value (may be >1000 or >10000).

"could you just use two MSuper? Does MAnalyse really degrade quality too much if 8-bit yv12?"

Yes - you can use 8bit source for MAnalyse and use output MVs clip to any bitdepth MDegrain processing. If source is enough noised it is also enough self-dithered so may be no significant degradation of motion search quality. It is also may be tested with software 16bit MAnalyse.

"Any hope for this getting Vapoursynth support?"

It is a question to developers of Vapoursynth if they can take sources and redesign to Vapoursynth. I do not know how to make it.

anton_foy

5th February 2023, 23:19

MCompensate should be the same as in 2.7.45 version from pinterf. But blocksize 16x16 may be broken (or bitdepth >8). Also blocksize of 16x16 do not run well at my GTX1060 with hardware search (in MAnalyse) and cause some crash that remote debugger can not break in. So I use only block size of 8x8. It is possible with scripting to feed half frame sized to MAnalyse with blocksize 8x8 and use MScaleVect to map output MVs it to full frame size with blocks of 16x16 if you want to try. It may be also faster for large frame sizes.

"thSCD but if I do not use scenechange detection at all I put it to zero?"

Disabled thSCD is about maxSAD (and it may be internallly silently clipped to maxSAD). So to disable scenechange detection you need to set thSCD to some verybig value (may be >1000 or >10000).

"could you just use two MSuper? Does MAnalyse really degrade quality too much if 8-bit yv12?"

Yes - you can use 8bit source for MAnalyse and use output MVs clip to any bitdepth MDegrain processing. If source is enough noised it is also enough self-dithered so may be no significant degradation of motion search quality. It is also may be tested with software 16bit MAnalyse.

Thanks! Then I will try 8x8 blocks with MAnalyse, MCompensate but it will not be any benifits in speed with DX12-ME (GPU) when using MCompensate? Only used for MAnalyse? Sounds a bit similar to SVanalyse then in SVPflow. When using mscalevect I had bad experiences with too soft/blurry results but I will try.

DTL

5th February 2023, 23:32

MCompensate is a client function for MAnalyse MVs server, same as MDegrain and all other from mvtools. MAnalyse can offload MVs search to DX12-ME hardware accelerator.
The MRecalculate is some intermediate - it receives MVs from MAnalyse and make some refinement search (and it can not be offloaded to DX12-ME accelerator because it make only single search from beginning).

In theory with the progress of MDegrainN it also may make some refinement search in multi-pass processing - with CPU only.

MAnalyse with DX12-ME mode should be able to make both single pair of frames search and 'multi' for MDegrainN search (using storage of current frame in accelerator and not upload it for each pair of search in 'multi' mode for better performance). So it should be compatible with all other MVs clients of mvtools.

The hardware ME from MPEG encoder looks not designed for best quality - the software may be better (and with iterative multi pass - more better). So for highest quality work the usage of hardware ME may be only for initial search or may be prefiltering.

anton_foy

7th February 2023, 13:22

#DTL:
pre=convertbits(8,dither=-1).converttoyv12().dipre()
tr=6
super = last.MSuper (pel=4, levels=1, chroma=true)
sup8 = pre.MSuper (pel=4, chroma=true)
#multi_vec = MAnalyse (sup8, multi=true, blksize=8, delta=tr, optSearchOption=5, overlap=0, levels=1, chroma=true).convertbits(16).converttoYUV444()
multi_vec=MAnalyse(sup8, multi=true, blksize=8, delta=tr, search=3, searchparam=2, overlap=0, optSearchOption=1, optPredictorType=4, chroma=false, mt=false, levels=1).convertbits(16).converttoYUV444()
#MDegrainN (super, multi_vec, tr, thSAD=150, thSAD2=250)
Mcompensate(last,super,multi_vec,thsad=thsad,tr=tr,center=true,mt=false)
temporalsoften(6,75,76,255,2)
selectevery(tr*2+1,tr)

I get error "unhandled C++ exception" with this script and if using the mdegrain line I get the same error. If using the first Manalyse line I get this error "MAnalyse: frame width not supported by DX12_ME, max supported width 0".

DTL

7th February 2023, 19:28

If using the first Manalyse line I get this error "MAnalyse: frame width not supported by DX12_ME, max supported width 0".

It looks something wrong with DX12 environment at the execution system - may be driver or device not support motion estimator.

HRESULT feature_support = dev_D3D12VideoDevice->CheckFeatureSupport(D3D12_FEATURE_VIDEO_MOTION_ESTIMATOR, &MotionEstimatorSupport, sizeof(MotionEstimatorSupport));

return no-error, but MotionEstimatorSupport structure looks like not initialized with real working values.

You may try that old DX12-ME checking tool from first posts in this thread - https://forum.doom9.org/showthread.php?p=1959067#post1959067

In correctly working system it should report max frame sizes like 4096x4096 or may be more. Something like in https://forum.doom9.org/showthread.php?p=1959078#post1959078 post.

"multi_vec=MAnalyse(sup8, multi=true, blksize=8, delta=tr, search=3, searchparam=2, overlap=0, optSearchOption=1, optPredictorType=4, chroma=false, mt=false, levels=1).convertbits(16).converttoYUV444()"

output of MAnalyse is special clip for mvtools other members and not need and can not be correctly processed with any other AVS filters. Also it is independent of bitdepth and colour format of processed clip. It only about blocksize/blocknumbers and other analysis params. So it is not correct to attempt to change it with .convertbits(16).converttoYUV444(). Same is for all other lines with MAnalyse. If MVs clip is damaged with any processing it may cause crashes in other mvtools filters because it is not checked for checksum or other ways. It is put to documentation - you can not modify it with almost any tools (except some of mvtools like MScaleVect() and others).

From mvtools2.html :
Technical note: MAnalyse does not generate a regular clip that can be displayed. Don't try to modify its content or it will get corrupted. It is made of a single long line, actually containing binary data (vectors, block SAD, misc. information…) instead of pixel values. It also alters the audio descriptor to pass additional data to other filters before any frame request. Therefore, in the current state, a vector clip cannot be saved to a lossless file and reloaded later for processing. If you want to do so, you have to transcode it first with MStoreVect and MRestoreVect. Furthermore, joining vector clips generated with different parameters may lead to unexpected results, because the aforementioned additional data is global to the whole clip and is not updated on each frame. When the MAnalyse filter is destructed (removed from the memory), the additional data is lost too, and an attempt to using the produced vectors may crash the application or give wrong results. For this reason, avoid using MAnalyse in ScriptClip and other functions of the Avisynth runtime subsystem.

anton_foy

8th February 2023, 03:35

Tried the D3D12HelloTriangle and I got "result: 0". I use the SSE2 version of your mvtools.

DTL

8th February 2023, 06:44

"Tried the D3D12HelloTriangle and I got "result: 0"."

HRESULT of 0 in Microsoft API typically mean S_OK (no-error). But you need to got non-zero (and not very small) MaxH/MaxW values in

MEstimator Feature support: D3D12_VIDEO_SIZE_RANGE SizeRange MaxW 4096 MaxH 4096 MinW 32 MinH 32

It must be more than your frame size.

anton_foy

8th February 2023, 23:47

ReinerSchweinlin

9th February 2023, 09:44

My card does not support it then:
D3D12_VIDEO_SIZE_RANGE SizeRange MaxW 0 MaxH 0 MinW 0 MinH 0
What Card are you using ?

anton_foy

9th February 2023, 11:01

What Card are you using ?

The GTX 970

DTL

9th February 2023, 11:34

It may really not support exposing DX12-ME API (though may supoport hardware MPEG encoding via older version of MPEG encoder ASIC). As I my test shows mostly probably DX12-ME interface is supported from GTX1xxx cards numbers.

ReinerSchweinlin

10th February 2023, 11:50

The GTX 970

Thats Maxwell 2.0 GM204 Chip. AFAIR the Encoder is one generation too old.

DTL

10th February 2023, 12:32

It is also interesting when finally Intel integrated MPEG encoder (or may be separate video boards or internal accelerators) will expose DX12-ME interface. Also at some theoretical point one can collect 3 different manufacturers of hardware MPEG encoders (NVIDIA, AMD and Intel) exposing DX12-ME interface and check its quality relative using single noised footage.

Different ME engines may have different search range and most interesting is quality of MVs at significantly noised source. For example how many definitely false non-zero MVs do ME engine produce at flat static noised only areas. For visual comparison of quality output MVs may be displayed with MShow() filter. If MShow() crashes - it is required to greatly increasing padding in MSuper() because some hardware MEs produces large out of frame MVs and current MShow() do not have protection from running out of buffer at attempt to draw such long invalid MV. onCPU 'standard' MAnalyse do not seach MVs outside given padded buffer or clips MVs to given padded buffer so no additional out of buffer checks in other mvtools filters (except MDegrainN it last builds. So even MCompensate with DX12-ME search in MAnalyse may also crash now. Sorry - it was only tested with MAnalyse+MDegrainN for degrain).

Double check and clipping of MVs in both MAnalyse (received from DX12-ME) and in MDegrainN will slightly degrade performance. So may be additional checking/clipping need to be added to all other consumer filters (or finally move from MDegrainN to MAnalsye if it not cause additional issues).

anton_foy

12th February 2023, 01:01

Thanks Reiner and DTL now I borrowed a graphics card tested with d3d test and it put out 4096x4096 res so it hopefully works. I will report.

EDIT: Nope, "can not load file Compute.cso". I have made a separate folder with only DTL's mvtools2.dll and the .cso-file.

EDIT2: now it works when I change from optSearchOption=5 to optSearchOption=1.

DTL

13th February 2023, 22:34

" "can not load file Compute.cso"."

Compute.cso file need to be in the 'current working directory'. It depends on the settings of your application loading your AVS script. You may start with placing Compute.cso to the directory with .avs script.

"I change from optSearchOption=5 to optSearchOption=1."

DX12_ME is used only in SO=5 or 6. Option 5 uses Compute.cso and SAD computing in accelerator (faster) and option 6 uses onCPU SAD computing (slower but may be a bit better quality with pel 2 or 4).

anton_foy

14th February 2023, 00:14

DTL

14th February 2023, 11:58

If Compute.cso loads OK you should be able run processing mode like:

MSuper(pelrefine=false, pel=4) # do not create refined subplanes for pel > 1
MAnalyse(optSearchOption=5) # use hardware motion search and SAD computing with Compute Shader on the same accelerator
MDegrainN(UseSubShift=1) # use blend-time computing of subsample shifted blocks

Boulder

14th February 2023, 13:02

Can the mod be used just to speed up calculating the MVs by setting optSearchOption=6 in MAnalyse without any ill effects, and use other MVTools functions from the original pinterf build?

DTL

14th February 2023, 16:07

You may try and look if it work or throws some errors (or crashes). optSearchOption=6 also uses DX12-ME for getting MVs from accelerator and only compute dissimilarity metric at CPU (SAD or others). Because DX12-ME not provide SAD data (or other (dis)similarity metric). So either compute shader is used in optSearchOption=5 or old MAnalyse with optSearchOption=6 single pass without search and only with SAD (or other) metric compute.

If some consumer plugin/function do not need SAD data from MAnalyse output - it may be designed one more option with better performance (only output MVs, like optSearchOption=7 for example).

Because mvtools support lots of different bitdepth and blocksize and other processing combinations - lots of different limitations also exists. For example
optSearchOption=5 - support only SAD computing in current shader implementaion.
optSearchOption=6 - can use any dissimalarity metric compute (defined by DMFlags option) like classic SAD or SSIM or VIF (or some combination like SSIM-structure only).

In the future mvtools it is better to rename SAD only dismetric to DM so params naming from thSAD to thDM for example.

Boulder

14th February 2023, 16:17

Well, looks like overlap is not supported so that's a no-no for me :(

DTL

14th February 2023, 16:49

Interpolated overlap currently only in the MDegrainN. It need to be implemented in other functions.

Or may be added to MAnalyse to output 4x-overlap (of the overlap=blocksize/2) in DX12-ME MVs processing mode. Better in performance 'diagonal 2x' overlap need to be implemented in each consumer filter. In both 'real search' and 'interpolated' versions.

Also you may try 'scripted diagonal overlap' with 2 shifted clips and BlockOverlap() plugin. It will be 2 real searches and possibly a bit higher in quality in compare with 1 search and interpolation. But it require to change your processing scripts.

Where do you need overlap ? MCompensate for QTGMC ?

It was expected some programmers may expand existing working solutions like interpolated overlap designed currently in MDegrainN to other filters/functions required by any existing users. But it looks current civilization dies too fast and almost no programmers and no users of avisynth and mvtools left today. And it only 2023 year - next will be even worse.

I not use deinterlacing of interlaced SD/HD and not use QTGMC so not have great need to redesign MCompensate or other parts of mvtools.

anton_foy

17th February 2023, 15:58

Thank you DTL it is working now but I get severe blockiness in high motion areas. Tried many script variants.

DTL

17th February 2023, 19:45

To suppress blocky look (edges of blocks) with single non-overlap search with DX12-ME in MAnalyse it is designed interpolated overlap processing in MDegrainN. To enable it set IntOvlp param of MDegrainN > 0. I typically use IntOvlp=3 (2x diagonal overlap with SAD re-check, better quality and slower is IntOvlp=1 - 4x blocks number equal to mvtools overlap=blksize/2 in both H and V).

So when you use optSearchOption=5 or 6 in MAnalyse you can not set overlap > 0 in MAnalyse and it recommended to use IntOvlp > 0 in MdegrainN (valid values from 1 to 4). Same is for better performance of MAnalyse in all modes - set non-overlapped search in MAnalyse and use interpolated overlap in MDegrainN. Because overlapped search in MAnalyse significantly degrades performance and interpolated overlap (fastest is 2x diagonal) runs much faster with may be still good enough quality.

magnetite

18th February 2023, 04:20

Current possibly best settings for processing:

MDegrainN(last,super, multi_vec, tr, thSAD=250, thSAD2=240, thSADC=240, mt=false, wpow=4, thSCD1=400, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9,
thMVLPFCorr=50, UseSubShift=1, IntOvlp=3, MPBthSub=5, MPBthAdd=5, MPBNumIt=3, MPB_SPCadd=3, MPB_SPCsub=0.3, MPBthIVS=1500, showIVSmask=false, MPB_DMFlags=64, MPB_MVlth=8, MPBtgtTR=tr)

As an end-user, I'm not really sure what all this does aside from some of the basic options like TR, thSAD/C, MT. Is there any way to make a preset quality value so people who are unfamiliar with all the settings can get decent results without all the guesswork? I think QTGMC has a preset quality feature, but that's a script, not a plugin like this.

DTL

18th February 2023, 09:36

I understand having 50+ different adjustments in mvtools for simple denoise work is not very easy for enduser. This is still in development and new complex processing like multi-pass blending (still about half of all existing ideas implemented - only spatial checks only, not temporal yet) require additional (and not very small) set of params. And it can not be separated to external 'filter' because it is all runtime single pass over frame processing to have best performance.

It were expected if some active users may have time to make tests and report some good set of settings for some use cases. But as time shows there are smaller and smaller number of users of AVS so it may take longer and longer time to wait if someone will put some time to make warp script with 'presets' (like SMDegrain). At 202x years the total world situation at this planet and around my living place changes significantly (with much worse perspective) so I currently put more time to prepare for possibly not very nice future and have lower time to put to development of this completely free project. So it was already ideas to write to pinterf about fixing some state of development and list most useful features being implemented more or less completely (not still left in debug state like optSearchOption=3 and 4 - multi-blocks search for onCPU processing in MAnalyse) in 2021..2022 for transfer to his 'main branch' and finally making 'official' releale of version after 2.7.45.

Now as I see some working MC RIFE version for tr up to 12 I have new ideas of adding some more mode to MDegrainN so it can work as blending engine for external source of motion compensated frames providing protection from too bad blends using same SAD (or any other implemented dissimilarity metric) for blocks method of analisys. While performance at GTX1060 card with tr=12 is about 20x times slower in compare with DX12-ME and current MDegrainN.

anton_foy

18th February 2023, 10:48

magnetite

18th February 2023, 20:37

No worries DTL. I appreciate what you do. Your work has helped me a lot.

If you need some testing done, let me know. I'm not super experienced with testing, but I can help with some things.

DTL

27th February 2023, 13:38

Found some VS attempt to use NVOF for motion compensation - https://bitbucket.org/mystery_keeper/vapoursynth-nvof/src/master/readme.txt . Trying to e-mail Asd-g about porting it to AVS for testing (not sure if solution from https://stackoverflow.com/questions/12686545/how-to-leave-a-message-for-a-github-com-user will work) . Not sure what is the minimum chip from NVIDIA is required to NVOF to work.

Support from NVIDIA looks like very limited https://forums.developer.nvidia.com/t/nvidia-optical-flow-vapoursynth-plugin-motion-compensation/218981

anton_foy

27th February 2023, 13:49

kedautinh12

27th February 2023, 13:55

I don't think so. I remember Asd-g's gpu don't support CUDA (only for NVIDIA gpu) so he only releases plugin support Vulkan

DTL

27th February 2023, 14:22

It is not only NVIDIA but may be new enough and expensive only:
https://docs.nvidia.com/video-technologies/optical-flow-sdk/read-me/index.html
NVIDIA Turing and above GPUs

Reel.Deel

27th February 2023, 15:51

Found some VS attempt to use NVOF for motion compensation - https://bitbucket.org/mystery_keeper/vapoursynth-nvof/src/master/readme.txt .

The author of the plugin said it sucks: https://github.com/dubhater/vapoursynth-mvtools/issues/60

DTL

28th February 2023, 19:57

I do not see tests with real world shot content from the author. Only tests with black background at NVIDIA site.

The definitely good side of NVIDIA optical flow motion compensation that it is product of a professional fulltime job development team (I hope) and hardware manufacturer. I expect it to have better quality in compare with 'simple' motion search engine from the also NVIDIA chip MPEG encoder. Also it may use much more compute hardware resources for better quality and may have some settings to tweak.

So the plugin is only short interface from optical flow API to AVS and expected to be simple and stable. Also with update in hardware and drivers it is expected to have better in the future. While still working via the same plugin with AVS.
Though I still not have Turing or later hardware to test.

anton_foy

2nd March 2023, 09:06

I found https://github.com/open-mmlab/mmdetection3d and https://github.com/bharathgs/Awesome-pytorch-list which lists the mm3d optical flow .pth. Maybe something interesting?

Edit: this link to convert to onnx to use for mlrt plugin https://github.com/open-mmlab/mmdetection/blob/master/docs/en/tutorials/pytorch2onnx.md

More lighter (faster?) approach: https://github.com/twhui/LiteFlowNet

DTL

2nd March 2023, 11:48

There is also newer https://github.com/twhui/LiteFlowNet3 .

As I see the main poor point or all these implementations - usage only 2 input images (I1 I2 or I0 I1). It is very poor approach for noise-deformed real images sets we have as input for temporal denoise process. May be many of motion-estimation / optical-flows algorithms were designed to work on clean images.

With 2 only input images the algorithm can not detect real static (or low speed motion) and low contrast low detailed areas deformed by noise only. And start to produce lots of errors motion vectors (of the very large length sometime). The better approach for motion pictures temporal denoising need to take in analysis as large as possible set of input frames to try to understand the shape and position of objecs in a sequence of a frames. May be some enthusiast can compose and send a message (e-mail ?) to many known motion-estimation or optical flow estimation engines designers with a request to make multi-input frames version of engine that can better work with more or less significantly degraded by natural noise image sequence from single scene objects. At github it may be can be made as opening issue with feature-request in each repository. And the training process of the models must be performed at the datasets with added natural (gauss) noise with target result of clean sources before adding noise.

anton_foy

2nd March 2023, 12:32

Here he is asking about more input images using optical flow for video. Don't know if it is corresponding to what you look for though. https://discuss.pytorch.org/t/classifying-spatio-temporal-data-videos/68922

Edit: Maybe still need to prefilter before optical flow then?

DTL

2nd March 2023, 13:23

Edit: Maybe still need to prefilter before optical flow then?

No-no. The motion search must be 'Intelectual' itself. And based only on the true-source non-changed frames (they carry most of useful non-distorted data). Same as started with MPB feature (but it still not go in 'temporal' process - work for single current frame only).

The process is really easy but resources consuming:

Imagine you have some low contrast image in a several printed to paper copies. Each copy have added random gauss-distributed noise with zero mean value across each image point in 'temporal' dimension (in a different copy of printed image).

Now you cut each image to small block (like 8x8 samples with regular grid) with scissors and put each cut pieces in separate box (marked 'pieces of copy 1', pieces of copy2, ..., pieces of copy N). Put same cut picture to the box in random shuffled order like broken mosaic.

Next you call an AI robot with Tera/PetaFLOPS engine with GigaBytes memory to your room and ask to take mostly equal looking pieces from each box and arrange in the same location of arrangement grid.

After solving - ask to calculate Average() of the all pieces in each location of a grid and output resulting denoised image.

For moving scene denoising you print noised sequential frames of a cutscene to different sheets of paper - and cut and ask robot to find equal looking parts and get single denoised frame at the output. To get correct moving picture denoised sequence you keep 'current' frame from messing up into box and put its cut pieces into arrangement grid as a reference and ask robot to find mostly equally looking pieces in all other boxes (before and after frames).

That 'find mostly equally looking to current frame' task is the key of the process. But in real life you do not have the clean not-noised 'current' frame as ideal reference to search. So the better process is iterative in 'time':

Robot create first version of denoised frame using motion compensation to 'current noised frame'. Next create several first-generation denoised frames around current and check for the resulting and used motion vectors used (for real life action the motion vectors are not random like white noise Fourier spectrum in time axis). If MVs are no looks good - the MVs are corrected in some way (like MVLPF in some 'linear processing' way for examlpe). After correction of MVs the new set of frames created:

'frames denoise generation 1'. Next is again for each frame search and apply new MVs using 'frames denoise generation 1' as reference. And use initial source frames as source. Next again analyse if motion looks 'natural' and also all found as similair blocks in a sequence of frames looks mostly equal (so the temporal noise with zero mean is removed).

The total 'esa' exhaustive search/process is really not use search of MVs at all but brute-force try of all possible MVs for current block in a frame (the total count of possible MVs is limited) in degraining (averaging) iterations and analyse target conditions of total cutscene frames pool for best matching.

Target conditions of denoising for frame pool of cutscene (sequential set of frames for single scene or single movie):
1. All MVs look naturally (low freqnency enough main energy components of Fourier spectrum and non-equal energy spectrum like random noise for example).
2. All found objects in a frame pool looks mostly equal (lowest dissimilarity metric like lowest SAD or highest SSIM and so on) in each frame (that mean after backward transforms for each frame).

So it is expected from neural network to perform very multi-pass iterative processing over a supplied input pool of frames (+-tr from current frame for long movie) for creating each denoised or motion compensated frame for output.

The best final MVs for final MC and blending for current frame is a product of very multi-generation search over a large set of frames. Not simple 2 input frames as in typical motion search engines.

Motion search and denoise is iterative multi-generation process: Better denoise provide better MVs and better MVs provide better denoise. But all iterations must use same source set of frames as 'mostly true reference'. To prevent from accumulating of errors in both denoise and MVs in multi-generations.

DTL

2nd March 2023, 19:48

Make a test of multi-generation MAnalyse+MDegrainN search with version from https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.19 release (it have SuperCurrent optional input for MAnalyse):

Script is

tr=6
super=MSuper(last,chroma=true, mt=false, pel=4)
src=last
mv_1_g0=MAnalyse(super, delta=1, search=3, searchparam=2, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=0, pzero=0, levels=4)
show1=MShow(super, mv_1_g0).Subtitle("input MAnalyse")

multi_vec=MAnalyse(super, multi=true, delta=tr, search=3, searchparam=2, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=0, pzero=0, levels=4)
g1=MDegrainN(last,super, multi_vec, tr, thSAD=250, thSAD2=240, mt=false, wpow=4, thSCD1=500, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)

gen1=g1

super_g1=MSuper(gen1,chroma=true, mt=false, pel=4)
multi_vec_g2=MAnalyse (super_g1, SuperCurrent=super, multi=true, delta=tr, search=3, searchparam=2, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=0, pzero=0, levels=4)

mv_1_g1=MAnalyse(super_g1, SuperCurrent=super, delta=1, search=3, searchparam=2, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=0, pzero=0, levels=4)
show2=MShow(super_g1, mv_1_g1).Subtitle("gen1 MAnalyse")

g2=MDegrainN(src,super, multi_vec_g2, tr, thSAD=250, thSAD2=240, mt=false, wpow=4, thSCD1=500, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)

gen2=g2

super_g2=MSuper(gen2,chroma=true, mt=false, pel=4)
multi_vec_g3=MAnalyse(super_g2, SuperCurrent=super, multi=true, delta=tr, search=3, searchparam=2, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=0, pzero=0, levels=4)

mv_1_g2=MAnalyse(super_g2, SuperCurrent=super, delta=1, search=3, searchparam=2, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=0, pzero=0, levels=4)
show3=MShow(super_g2, mv_1_g2).Subtitle("gen2 MAnalyse")

g3=MDegrainN(src,super, multi_vec_g3, tr, thSAD=250, thSAD2=240, mt=false, wpow=4, thSCD1=500, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)

gen3=g3

super_g3=MSuper(gen3,chroma=true, mt=false, pel=4)
multi_vec_g4=MAnalyse(super_g3, SuperCurrent=super, multi=true, delta=tr, search=3, searchparam=2, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=0, pzero=0, levels=4)

mv_1_g3=MAnalyse(super_g3, SuperCurrent=super, delta=1, search=3, searchparam=2, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=0, pzero=0, levels=4)
show4=MShow(super_g3, mv_1_g3).Subtitle("gen3 MAnalyse")

g4=MDegrainN(src,super, multi_vec_g4, tr, thSAD=250, thSAD2=240, mt=false, wpow=4, thSCD1=500, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)

gen4=g4

super_g4=MSuper(gen4,chroma=true, mt=false, pel=4)
multi_vec_g5=MAnalyse(super_g4, SuperCurrent=super, multi=true, delta=tr, search=3, searchparam=2, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=0, pzero=0, levels=4)

mv_1_g4=MAnalyse(super_g4, SuperCurrent=super, delta=1, search=3, searchparam=2, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=0, pzero=0, levels=4)
show5=MShow(super_g4, mv_1_g4).Subtitle("gen4 MAnalyse")

g5=MDegrainN(src,super, multi_vec_g5, tr, thSAD=250, thSAD2=240, mt=false, wpow=4, thSCD1=500, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)

gen5=g5

super_g5=MSuper(gen5,chroma=true, mt=false, pel=4)
#multi_vec_g5=MAnalyse(super_g5, SuperCurrent=super, multi=true, delta=tr, search=3, searchparam=2, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=0, pzero=0, levels=2)

mv_1_g5=MAnalyse(super_g5, SuperCurrent=super, delta=1, search=3, searchparam=2, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=0, pzero=0, levels=4)
show6=MShow(super_g5, mv_1_g5).Subtitle("gen5 MAnalyse")

row1=StackHorizontal(show1, show2)
row2=StackHorizontal(show3, show4)
row3=StackHorizontal(show5, show6)
StackVertical(row1, row2, row3)

https://i.ibb.co/nCxGsdy/6gen-01.png (https://ibb.co/6BjMR9S)

All MDegrainN accepts same input current and super clips and only MAnalyse in each generation uses one source from previous generation MDegrainN and one source from input super clip.

Result shows in several generations of refining of MVs as number of generation increases the number of significantly errorneous MVs at low contrast mostly noised static areas slowly decreases. The mostly visible effect in first 1..2 generations.

anton_foy

3rd March 2023, 08:08

DTL

3rd March 2023, 12:23

"Would the "SuperCurrent" be used to compare against the prior superclip (prefiltered with Mdegrain) to get a better estimation? "

I think no. The total idea of 2-inputs MAnalyse is to make search of 'previous generation degrained' block vs input full-true-non-distorted but +grained block. It should save from quick accumulating errors possible to come from 'prefilters'.

If you like to use MRecalculate - the SuperCurrent input can also be easily added there. But typically MRecalculate is used to refine blocksize or use different search params. For simple multi-generation search the single MAnalyse is enough. Also in each generation the params of search (and intermediate and final MDegrain) may be changed.

Old mvtools allow only search inside single input clip (or you need to try to merge different clips into single input with something like Interleave() and look if it cause correct fetching of 'src' and 'ref' frames of different sources). Search inside single clip cause accumulation of errors after 'prefiltering'.

anton_foy

3rd March 2023, 13:55

The total idea of 2-inputs MAnalyse is to make search of 'previous generation degrained' block vs input full-true-non-distorted but +grained block. It should save from quick accumulating errors possible to come from 'prefilters'.

Sorry I am a little slow. Just understood it as prefiltering with MDegrain and only using that data to compare against the non-altered data using this line:

multi_vec_g2=MAnalyse (super_g1, SuperCurrent=super, multi=true, delta=tr, search=3, searchparam=2, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=0, pzero=0, levels=4)

But anyway Great to see the progress even if I did not understand correctly :)

Edit: Yes I thought wrong about MRecalculate, I think I understand now what you mean with your last line of explanation.

DTL

3rd March 2023, 17:46

Playing with settings of MAnalyse I found disabled by default trymany option. After enabling it it looks like best search mode quickly providing stable enough MVs field in multi-generation search even with thresholding penalties of zero and new predictors are set to zero. Without trymany even in 5 generations there is no any convergence to some stable MVs field observed (may be some quantization noise play role in non-stability).

But it currently work good only with old SAD dismetric and crashes with divide by zero error somwhere with VIF dismetric - need debug and make new version.

Though enabling trymany at MAnalyse will significantly degrades performance because it enable refine search around each predictor (and total number of predictors around 6 or 7). So it looks may be enabled only in highest quality of MVs is required.

Multi-generation search with trymany enabled and default SAD dissimilarity metric for search best matching block:
https://i.ibb.co/5Rbx6fc/fr38-dmflags1-pn0-pz0-gf-tmt.png (https://ibb.co/jrnZT14)

anton_foy

3rd March 2023, 18:01

Wow great find! Have you considered to try Zopti with your version of mvtools? Can't wait to try the new version of yours.

DTL

8th March 2023, 11:41

Some important note for multi-generation MVs refinement: The thSAD for MDegrain need to be significantly reduced after 1st generation of MAnalyse using first generation of MDegrain output. Because SAD of mostly cleaned 'current' block with input noised block become about 2time lower. So thSAD for intermediate generations and last output MDegrain need to be reduced to about 0.5 of initial.

So better multi-generation MVs refinement is some like:

init_thSAD=400

s1=MSuper()
mv1 = MAnalyse(s1)
dg1 = MDegrain(s1, mv1, thSAD=init_thSAD)

1stgen_thSAD = (int)(init_thSAD/1.8) # divisor - subject to Zopti refine ?

s2=MSuper(dg1)
mv2 = MAnalyse(s2, SuperCurrent=s1) # or (s1, SuperCurrent=s2) - may be not visible difference
dg2=MDegrain(s1, mv2, thSAD=1stgen_thSAD)

Also it was found enabling trymany=true in MAnalyse while good refining zero MVs also may add some significantly bad MVs. So it is planned to add flags for predictors used in trymany mode to skip possibly bad predictors and to make performance visibly better.

anton_foy

8th March 2023, 12:29

DTL

8th March 2023, 13:14

Yes - MAnalyse with optional SuperCurrent can be used with any optSearchOption value (so including hardware search options). I even think of using >1 HWacc in the system for better performance in pipelined way. So first HWacc making initial analysis and second make refining step.

Later we will have many cheap secondhand old HWaccs capable of DX12-ME so it may be tested. Currently you can try accnum different for MAnalyses:

init_thSAD=400

s1=MSuper()
mv1 = MAnalyse(s1, optSearchOption=5, accnum=1) # use first DX12-ME accelerator in system or accnum=0 ? need testing
dg1 = MDegrain(s1, mv1, thSAD=init_thSAD)

1stgen_thSAD = (int)(init_thSAD/1.8) # divisor - subject to Zopti refine ?

s2=MSuper(dg1)
mv2 = MAnalyse(s2, SuperCurrent=s1, optSearchOption=5, accnum=2) # use second DX12-ME accelerator in system or accnum=1 - need testing
dg2=MDegrain(s1, mv2, thSAD=1stgen_thSAD)

Currently using 2 MAnalyse with single HWacc will drop performance about 2 times.

Also may be combination of 1 external PCI-board DX12-ME acc and build-in into CPU may be tested where avaiable.

Also as I read some NVIDIA boards/chips have >1 MPEG encoder ASIC (?) so may expose >1 full-speed DX12-ME interfaces for applications.

At https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new
# OF CHIPS

# OF NVENC /CHIP

Total # of NVENC

So GeForce GTX 965M > 980M / 980MX Maxwell (2nd Gen) may have 2 full-speed DX12-ME interfaces ?
Also GeForce GTX 960 Ti / 970 / 980 , GeForce GTX 980 Ti , GeForce GTX Titan X
GeForce GTX 1070M / 1080M , GeForce GTX 1070 / 1070Ti, GeForce GTX 1080 , GeForce GTX 1080 Ti, GeForce GTX Titan X / Titan Xp

Same is GeForce RTX 4080 Laptop , GeForce RTX 4080 16GB , GeForce RTX 4090 Laptop , GeForce RTX 4090 - but much more expensive.

Also Titan V - 3 NVENC.

Dogway have GTX 1070? May be good to try to ask for testing 1 vs 2 MAnalyse performance (also accepting different accnum values >0 or >1).

Addition: I not sure if several MPEG encoder ASICs located in single physical board will be switched as different Direct3D12 devices with 'accnum' param. May be environment will auto-spread motion estimation tasks if single board have several task dispatch resources avaialble. So 2-NVENC boards may be simply allow to run 2 MAnalyse with about equal speed with default accum=0.

DTL

8th March 2023, 20:26

New release: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.20

Fixed possible bug with trymany in MAnalyse.
Added trymany into optPredictorType=1 mode (zero, global and median predictors only).

Added partial fix for 4:2:x formats processing chroma shift issue for MAnalyse, MDegrainN, MCompensate (may also MRecalculate). With the curernt pel-precision from MSuper.

The multi-generations MVs refining looks like also work very visibly against blurring for complex motion like facial animation.

Cleaned from MShow processing script:

my_DMFlags=1
my_thSAD=300
my_thSAD2=250

my_thSAD_mg=150
my_thSAD2_mg=100

my_thSCD=500

my_global=true
my_pzero=10
my_pnew=10
my_pglobal=10

my_pel=2
my_trymany=true

my_oPT=1

tr=6
super=MSuper(last,chroma=true, mt=false, pel=my_pel)

multi_vec=MAnalyse(super, multi=true, delta=tr, search=3, searchparam=2, trymany=my_trymany, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=my_pnew, pzero=my_pzero, pglobal=my_pglobal, global=my_global, levels=4, DMFlags=my_DMFlags, optPredictorType=my_oPT)
g1=MDegrainN(super, multi_vec, tr, thSAD=my_thSAD, thSAD2=my_thSAD2, mt=false, wpow=4, thSCD1=my_thSCD, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)

gen1=g1

super_g1=MSuper(gen1,chroma=true, mt=false, pel=my_pel)
multi_vec_g2=MAnalyse(super_g1, SuperCurrent=super, multi=true, delta=tr, search=3, searchparam=2, trymany=my_trymany, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=my_pnew, pzero=my_pzero, pglobal=my_pglobal, global=my_global, levels=4, DMFlags=my_DMFlags, optPredictorType=my_oPT)

g2=MDegrainN(super, multi_vec_g2, tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, mt=false, wpow=4, thSCD1=my_thSCD, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)

gen2=g2

super_g2=MSuper(gen2,chroma=true, mt=false, pel=my_pel)
multi_vec_g3=MAnalyse(super_g2, SuperCurrent=super, multi=true, delta=tr, search=3, searchparam=2, trymany=my_trymany, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=my_pnew, pzero=my_pzero, pglobal=my_pglobal, global=my_global, levels=4, DMFlags=my_DMFlags, optPredictorType=my_oPT)
g3=MDegrainN(super, multi_vec_g3, tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, mt=false, wpow=4, thSCD1=my_thSCD, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)

gen3=g3

super_g3=MSuper(gen3,chroma=true, mt=false, pel=my_pel)
multi_vec_g4=MAnalyse(super_g3, SuperCurrent=super, multi=true, delta=tr, search=3, searchparam=2, trymany=my_trymany, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=my_pnew, pzero=my_pzero, global=my_global, levels=4, DMFlags=my_DMFlags, optPredictorType=my_oPT)
g4=MDegrainN(super, multi_vec_g4, tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, mt=false, wpow=4, thSCD1=my_thSCD, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)

gen4=g4

super_g4=MSuper(gen4,chroma=true, mt=false, pel=my_pel)
multi_vec_g5=MAnalyse(super_g4, SuperCurrent=super, multi=true, delta=tr, search=3, searchparam=2, trymany=my_trymany, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=my_pnew, pzero=my_pzero, pglobal=my_pglobal, global=my_global, levels=4, DMFlags=my_DMFlags, optPredictorType=my_oPT)
g5=MDegrainN(super, multi_vec_g5, tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, mt=false, wpow=4, thSCD1=my_thSCD, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)

gen5=g5

super_g5=MSuper(gen5,chroma=true, mt=false, pel=my_pel)
multi_vec_g6=MAnalyse(super_g5, SuperCurrent=super, multi=true, delta=tr, search=3, searchparam=2, overlap=0, chroma=true, mt=false, optSearchOption=1, truemotion=false, pnew=my_pnew, pzero=my_pzero, pglobal=my_pglobal, global=my_global, levels=4, DMFlags=my_DMFlags, optPredictorType=my_oPT)
g6=MDegrainN(super, multi_vec_g6, tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, mt=false, wpow=4, thSCD1=my_thSCD, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)

return Interleave(g6.Subtitle("g6"), g2.Subtitle("g2"), g1.Subtitle("g1"))

Generally gen2 is already much sharper at motion in compare with gen1 (standard MDegrainN). Gen6 sometime look a bit better and sometime more blurry. May be some average good number of generations is between 2 and 6 (or some detail-restoration processing may be added to regain details from high-gen if that frame (area of frame) is sharper).

Frames g1, g2 and g6 2x enlarged with BSpline:
https://i.ibb.co/wYfC1Mp/g1.png (https://ibb.co/wYfC1Mp)
https://i.ibb.co/BsSMLfN/g2.jpg (https://ibb.co/BsSMLfN)
https://i.ibb.co/By2ByTj/g6.jpg (https://ibb.co/By2ByTj)

It was non-field separated interlaced so 2 fields present.

May be somehow this many calls to MSuper/MAnalyse/MDegrainN for each generation of MVs refining can be compacted to some AVS function and make script smaller.

imgsli comparisons:
https://imgsli.com/MTYwODgx
https://imgsli.com/MTYwODgw

DTL

9th March 2023, 22:57

Real working script with both accelerator and CPU search and refining functions. For 1920x1080i input.

# Input plugins
LoadPlugin("ffms2.dll")
LoadPlugin("mvtools2.dll")

SetFilterMTMode("DEFAULT_MT_MODE", 3)

my_thSAD=260
my_thSAD2=240

my_thSAD_mg=130
my_thSAD2_mg=120

my_thSCD=500

my_pzero=10
my_pnew=10
my_pglobal=10

my_pel=2
my_thCohMV=5 # 5..8 for pel=2, 10..16 for pel=4 ?
my_trymany=true

my_oPT=1
my_overlap=0
my_IntOvlp=3
my_searchparam=2

my_MPBNumIt=2

my_init_tr=12
my_refine_tr=12

Function RefineMV(clip mvclip, clip super_ref, clip src, int _thSAD, int _thSAD2, int in_tr, int refine_tr, int my_thSCD, int my_pel, bool my_trymany, int my_pnew, int my_pzero, int my_pglobal, \
int my_oPT, int my_overlap, int my_searchparam, int my_IntOvlp, int my_thCohMV)
{
g_next=MDegrainN(src, super_ref, mvclip, in_tr, thSAD=_thSAD, thSAD2=_thSAD2, mt=false, wpow=4, thSCD1=my_thSCD, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=my_thCohMV, \
MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_IntOvlp)
super_g_next=MSuper(g_next,chroma=true, mt=false, pel=my_pel)
return MAnalyse(super_g_next, SuperCurrent=super_ref, multi=true, delta=refine_tr, search=3, searchparam=my_searchparam, trymany=my_trymany, overlap=my_overlap, chroma=true, mt=false,\
optSearchOption=1, truemotion=false, pnew=my_pnew, pzero=my_pzero, pglobal=my_pglobal, global=true, optPredictorType=my_oPT)
}

Function RefineMV_HW(clip mvclip, clip super_ref, clip src, int _thSAD, int _thSAD2, int in_tr, int refine_tr, int my_thSCD, int my_pel, int my_thCohMV)
{
g_next=MDegrainN(src, super_ref, mvclip, in_tr, thSAD=_thSAD, thSAD2=_thSAD2, mt=false, wpow=4, thSCD1=my_thSCD, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=my_thCohMV, \
MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3, UseSubShift=1)
super_g_next=MSuper(g_next,chroma=true, mt=false, pel=my_pel, levels=1, pelrefine=false)
return MAnalyse(super_g_next, SuperCurrent=super_ref, multi=true, delta=refine_tr, chroma=true, mt=false, optSearchOption=5, levels=1)
}

FFmpegSource2("1920x1080i.mp4")

AddBorders(0,0,0,72)

noproc=last

SeparateFields()

super_hwa=MSuper(last, mt=false, chroma=true, pel=my_pel, hpad=8, vpad=8, levels=1, pelrefine=false)
super_cpu=MSuper(last, mt=false, chroma=true, pel=my_pel, hpad=8, vpad=8, levels=0, pelrefine=true)

multi_vec_hwa=MAnalyse(super_hwa, multi=true, blksize=8, delta=my_init_tr, overlap=0, chroma=true, optSearchOption=5, mt=false, levels=1)
multi_vec_cpu=MAnalyse(super_cpu, multi=true, delta=my_init_tr, search=3, searchparam=my_searchparam, trymany=my_trymany, overlap=my_overlap, chroma=true, mt=false, \
optSearchOption=1, truemotion=false, pnew=my_pnew, pzero=my_pzero, pglobal=my_pglobal, global=true, optPredictorType=my_oPT)

multi_vec_cpu2=RefineMV(multi_vec_cpu, super_cpu, last, my_thSAD, my_thSAD2, my_init_tr, my_refine_tr, my_thSCD, my_pel, my_trymany, my_pnew, my_pzero, my_pglobal, my_oPT, \
my_overlap, my_searchparam, my_IntOvlp, my_thCohMV)
multi_vec_hybr2=RefineMV(multi_vec_hwa, super_cpu, last, my_thSAD, my_thSAD2, my_init_tr, my_refine_tr, my_thSCD, my_pel, my_trymany, my_pnew, my_pzero, my_pglobal, my_oPT, \
my_overlap, my_searchparam, my_IntOvlp, my_thCohMV)
multi_vec_hwa2=RefineMV_HW(multi_vec_hwa, super_hwa, last, my_thSAD, my_thSAD2, my_init_tr, my_refine_tr, my_thSCD, my_pel, my_thCohMV)

cpu2=MDegrainN(last,super_cpu, multi_vec_cpu2, my_refine_tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, mt=false, wpow=4, UseSubShift=1, thSCD1=my_thSCD, adjSADzeromv=0.7, \
adjSADcohmv=0.7, thCohMV=my_thCohMV, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_IntOvlp, MPBthSub=5, MPBthAdd=20, MPBNumIt=my_MPBNumIt, \
MPB_SPCsub=0.5, MPB_SPCadd=1.5, MPBthIVS=2200, showIVSmask=false).Weave().Subtitle("cpu2")
hwa2=MDegrainN(last,super_hwa, multi_vec_hwa2, my_refine_tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, mt=false, wpow=4, UseSubShift=1, thSCD1=my_thSCD, adjSADzeromv=0.7, \
adjSADcohmv=0.7, thCohMV=my_thCohMV, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_IntOvlp, MPBthSub=5, MPBthAdd=20, MPBNumIt=my_MPBNumIt, \
MPB_SPCsub=0.5, MPB_SPCadd=1.5, MPBthIVS=2200, showIVSmask=false).Weave().Subtitle("hwa2")
hybr2=MDegrainN(last,super_hwa, multi_vec_hybr2, my_refine_tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, mt=false, wpow=4, UseSubShift=1, thSCD1=my_thSCD, adjSADzeromv=0.7, \
adjSADcohmv=0.7, thCohMV=my_thCohMV, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_IntOvlp, MPBthSub=5, MPBthAdd=20, MPBNumIt=my_MPBNumIt, \
MPB_SPCsub=0.5, MPB_SPCadd=1.5, MPBthIVS=2200, showIVSmask=false).Weave().Subtitle("hybr2")

Interleave(cpu2, hybr2, hwa2, noproc.Subtitle("src"))

#last=hybr2

Crop(0,0,1920,1080)

Prefetch(6)

Examples of both accelerator and CPU search and refining and hybrid mode (accelerator first search and CPU refining). Quality onCPU is a bit better. Full CPU search and refine at i5-9600K and 1920x1080i frame run at about 0.28 fps.
Hybrid mode with GTX1060 and i5-9600K CPU run at about 1.24 fps (pel=4) and 1.75fps (pel=2). Quality is close to full CPU search.
Full accelerator search and refine run only a bit faster (about 1.3 fps with pel=4) and quality is a bit lower of hybrid mode at some scenes.

DTL

15th March 2023, 18:39

New version: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.21
Added new processing mode to MDegrainN: MEL (Most Equal Looking) search mode for TTH (Temporal Thresholding).
New params to MDegrainN:

pmode=0 (default) - standard blending, pmode=1 - MEL search and TTH only.

TTH_DMFlags - dismetric flags for estimating blocks difference at TTH compare. Flags 0x01 to 0x20 valid (except 0x08).

TTH_thUPD (0 default, additional thresholding disabled, 100% linear mode, must be >0 for pmode=1) - integer threshold for selection: keep output old in pmode=0 or 'best' in pmode=1 block from memory or update block in memory and output new block. Typical working values expected to be significantly below thSAD (like thSAD/3.. thSAD/4 and less). Startng from 0. 0 mean no blocks from memory used (standard MDegrainN mode - FIR filter).

TTH_chroma - use chroma in TTH dismetric analysis (slower, better quality) or not (faster).

Fixed performance issue with double processing of chroma planes in combined YUV processing with no overlap.

Current testscript:

tr=10
super=MSuper(last, mt=false, chroma=true, pel=2, hpad=8, vpad=8, levels=0, pelrefine=true)
multi_vec=MAnalyse(super, multi=true, blksize=8, delta=tr, search=3, searchparam=2, overlap=0, optSearchOption=1, optPredictorType=0, chroma=false, mt=false)
ref=MDegrainN(last,super, multi_vec, tr, thSAD=185, thSAD2=170, mt=false, wpow=4, thSCD1=350, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, IntOvlp=3)

super2=MSuper(ref, mt=false, chroma=true, pel=2, hpad=8, vpad=8, levels=0, pelrefine=true)
MDegrainN(ref,super2, multi_vec, tr, thSAD=250, thSAD2=240, mt=false, thSCD1=350, pmode=1, TTH_thUPD=100, IntOvlp=3)

TTH_thUPD may be also enabled in 'standard' blending modes (pmode=0 (default)) too (both overlap and no overlap combined YUV processing). It is (much) faster but may provide somehow lower quality. Currently no motion block tracking available so it is mostly effective for completely static blocks only. Some limited motion tracking expected to be in some future versions.

Complexity of analysis in pmode=1 currently is ~tr^2 so it may use separate tr value (and mvclip created with lower tr value). Quality expected to be ~tr value (probability to found most commonly looking block in the total tr-pool). Param thSAD in pmode=1 also controls initial block skipping when accumulating blocks in analysis pool.

TTH_thUPD is the main param to adjust - the higher its value - the more noise blocks are skipped but too high value may cause 'hanging' blocks visible or motion quality degradation. Setting too high thSAD in pmode=1 also may cause more artifacts.

pmode=1 expected to be 'final cleaning' after initial MDegrainN (also must use new super clip with pre-denoised frames) and if highest quality required. For general everyday encodings may be enough to play with TTH_thUPD param in standard pmode=0.

Last MDegrainN with pmode=1 may or may not use refined mv-clip (for best results best refined mvclip is recommended).

pmode=1 not blend at all - so no degrade details quality with any thSAD. It only additional way to select 'best' looking block in current tr-scope and duplicate it in output frames until visual difference with current frame block is below threshold.

TTH_thUPD may be enabled in any MDegraiN in processing script (in MVs refining and final degrain and final cleanup).
TTH_DMFlags may set any avaialable dismetric for visual difference analysis (SAD - faster, SSIM and VIF - slower) at any MDegrainN with enabled TTH_thUPD or pmode=1.

DTL

30th March 2023, 11:04

Some morning quicky implementation of this year idea about noise bitrate estimation to check the degrain quality.

Release 30.03.2023 - https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.22

Added computing and displaying of residual noise bits count per frame to MCompensate.
Compute sum of log2 of the samples absolute difference between source and motion compensated output frame of MCompensate. For complete static frame sequence RNB=0 bits/frame. For noise bitrate per second - value should be multiplied to frame rate.

New param to MCompensate: showRNB (default = false).

Usage example:
super=MSuper()
mv=MAnalyse(super)
MCompensate(super, mv, showRNB=true)

Currently only for 8bit sources. Need to offset processing function to templated for HBD support. Can process Y only input clip or YUV/RGB (3 planes present). For >1 planes the sum of all planes is displayed.

Computing part:

for (int y = 0; y < nHeight; y++)
{
uint8_t* pDstFrame = pDst[0] + nDstPitches[0] * y;
uint8_t* pSrcFrame = (uint8_t*)pSrc[0] + nSrcPitches[0] * y + nOffset[0];
for (int x = 0; x < nWidth; x++)
{
iSumNzBits += 32 - __lzcnt(SADABS((int)pSrcFrame[x] - (int)pDstFrame[x]));
}
}

Not applicable to float32 samples directly (need convert to some finite precision integer first <32bit).

Usage example to measure denoise process:

SeparateFields()

fields_orig=last

tr=3

super=MSuper(last, mt=false, chroma=true, pel=2, hpad=8, vpad=8, levels=0, pelrefine=true)
multi_vec=MAnalyse(super, multi=true, blksize=8, delta=tr, search=3, searchparam=2, overlap=0, optSearchOption=1, optPredictorType=0, chroma=false, mt=false, DMFlags=1)
ref=MDegrainN(last,super, multi_vec, tr, thSAD=185, thSAD2=170, mt=false, wpow=4, thSCD1=350, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, IntOvlp=3)

super2=MSuper(ref, mt=false, chroma=true, pel=2, hpad=8, vpad=8, levels=0, pelrefine=true)
MDegrainN(ref,super2, multi_vec, tr, thSAD=350, thSAD2=340, mt=false, thSCD1=350, pmode=1, TTH_thUPD=100, IntOvlp=3)

super_ref=MSuper(ref)
mv_ref=MAnalyse(super_ref)
rnb_den_ref=MCompensate(super_ref, mv_ref, showRNB=true)

super2=MSuper()
mv2=MAnalyse(super2)
rnb_den=MCompensate(super2, mv2, showRNB=true)

super_orig=MSuper(fields_orig)
mv_orig=MAnalyse(super_orig)
rnb_orig=MCompensate(super_orig, mv_orig, showRNB=true)

StackHorizontal(rnb_den, rnb_den_ref, rnb_orig)

Weave()

Output sample frame
https://i.ibb.co/LkGQgRw/rnb-examp01.jpg (https://ibb.co/LkGQgRw)

Yes - the fileds are blended not very nice. It shows how for static areas the second MDegraiN(pmode=1) decreases noise bitcount about 10 times. First stage denoise about 2.9 times decrease noise bitrate. Addition of secondary non-linear IIR-type filter with memory decreases nosie bitrate about 30 times from source.

Completely (100%) temporal denoised frame sequence for zero calibration is
Trim(1,1)
Loop()

guest

30th March 2023, 11:49

Some morning quicky implementation of this year about noise bitrate estimation to check the degrain quality.

Release 30.03.2023 - https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.22

Just to confirm, and probably a noobie question, but the 2 variants, DX12 & noDX12, are reliant on Direct-X being installed on the system (or not) ??

DTL

30th March 2023, 14:03

If you have Win10 or later and compatible hardware you can use DX12 build. It will not load at Win7 or other without DX12 installed. If you not use DX12 search modes in MAnalyse you can safely use noDX12 build.

mastrboy

30th March 2023, 18:54

DTL, I can't get your builds to work at all in Windows11, I have tried all 4 different .dll's in the zip file...

AVSmeter just stops at 0 frames forever, until I hit ctrl+c.
AVSMeter64.exe -o d:\test.avs

AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r3973, 3.7, x86_64) (3.7.3.0)

Number of frames: 33304
Length (hh:mm:ss.ms): 00:23:09.054
Frame width: 960
Frame height: 720
Framerate: 23.976 (24000/1001)
Colorspace: YV12

Frame (current | last): 0 | 33303

Virtualdub gives me a cryptic memory violation message:
An out-of-bounds memory access (access violation) occurred in module 'VirtualDub64'...
...reading address FFFFFFFFFFFFFFFF.

Avspmod gives a similear error message:
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 315, in 'calling callback function'
File "avsp.pyo", line 5136, in local_wnd_proc
WindowsError: exception: access violation reading 0xFFFFFFFFFFFFFFFF

I have none of these issues with https://github.com/pinterf/mvtools/releases

I also have no idea how to troubleshoot this other than give you some information about my system and hope you have any idea of what is wrong:
AviSynth+ 3.7.3 (r3973, 3.7, x86_64)
Windows 11 22H2 (22621.1413)

Avisynth script I tested with:
FFVideoSource("test.mkv")
SMDegrain(tr=3, thSAD=400, RefineMotion=false, contrasharp=false, plane=4, prefilter=0, chroma=true)

DTL

30th March 2023, 19:41

Unfortunately my builds may be not compatible with many old scripts (using no-default block size of 16x16 and may be some more not tested options). So it still pre-release demos of some features and mostly tested at the examples scripts provided here and typically block size of 8x8. I even make somehow changed QTGMC to use with my builds when I tested deinterlacing.

So it is no good to put this .dll in 'common' folder and recommended to load with LoadPlugin() from current working folder. It is expected may be in some years (in beginning of 2024 it is expected great all planet celebration of 20 years for mvtools) we will have some features ported to 'more official' pinterf or may be other programmer capable to test and bugfix most of supported modes of mvtools. But it still not happen. I going to make some e-table (may be google web docs ?) of all new features and ideas accumulated and partially implemented for post-2.7.45 version with current 'status' and other data for analyse and creating list of mostly important features to port/bugfix.

Also I not use SMDegrain script and make my own denosie scripts based on mvtools only. So I not know what cause crash there. May be some day I will have time to attempt to install SMDegrain and check it with debugger where may be crash with that settings and if it possible to more or less fast to fix it.

For the very first possible solution it is recommended to test with block size of 8x8 (internal default for mvtools).

Though if you use SMDegrain as I read it still not support any new features of post-2.7.45 mvtools so it may be safely to use old 'stable' 2.7.45 build from pinterf. May be still many years until we will have some more stable post-2.7.45 build fully compatible with 2.7.45 processing with default new settings and Dogway will make changes to SMDegrain to use new features.

takla

31st March 2023, 22:22

@DTL
Have you seen this
https://devblogs.microsoft.com/directx/preview-agility-sdk-1-710-0/
Is it applicable for mvtools?

DTL

31st March 2023, 23:05

I have some strategic idea how may be make current post-2.7.45 version more compatible with old scripts and 2.7.45 build - rename all filters with adding _a to the end (like alpha-state). So it may be possible to load both 2.7.45 and post-2.7.45 mvtools in single AVS environment and only use selected filters from post-2.7.45 if required (also it may be (partially) compatible in between - super and mv clips). Now because of same naming it either not loads or may cause undefined usage of different filters from different .dlls. May be in next build.

"Is it applicable for mvtools?"

About new heaps mode - currently the performance is very few limited by textures upload and backward download of MVs and SAD data is very small in size. About sampling - currently some 'simple' sampling mode used in SAD compute shader (sort of sample(x,y) request as CPU from host RAM do (not possible 'complex 3D' sampling when texture mapping to some virtual triangle or other patch performed). So no update of sampling required and can not help in performance.

DTL

17th June 2023, 21:20

New release: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.23

Added denoise mask clip input into MDegrainN. Work only on block-based mode. Must be Y8 format with frame size equal to blocks number to process (including any used overlap mode).

New param to MDegrainN:
dnmask - clip. 0 is full standard denoise, 255 is no denoise (so positive Y-channel can be used as mask to degrain only low brightness levels).

Example script (for IntOvlp=3):
dn_mask1=ConvertToY8()

blksize=8
#int_ ovlp=3
dn_mask_x=dn_mask1.width/blksize
overlap_size=blksize/2
dn_mask_y=(dn_mask1.height-overlap_size)/(blksize-overlap_size)
dn_mask1=BilinearResize(dn_mask1, dn_mask_x, dn_mask_y)

dn_mask1=Levels(dn_mask1, 0, 1, 100, 0, 255, coring=false)

dn_masked=MDegrainN(.., IntOvlp=3, dnmask=dn_mask1)

Added update MEL memory with best (lowest sum of DM table row) block and memory for sum of current stored in IIR memory block.

Real fast way to get block numbers is to feed any sized mask clip and read error message if size is not correct - it will show current blocks number for H and V directions for current used overlap mode.

Simple BilinearResize do not make perfect mask for any overlap mode because even blocks rows shifted to the right to overlap size (typical half block size with max overlap). So better to separate to odd/even rows - shift even rows to the right and combine to frame back. But any overlap processing modes looks like hide these errors with not too large block sizes.

anton_foy

6th July 2023, 13:08

@DTL

How is it possible to use mdegrain2 with your version, or is it only possible with tr=1 and mdegrain()?

I would like to make my Clay script to work with your version separately to get a speed boost and also a quality boost and yet keep close to the results I get with the current Pinterf version of mvtools.
But I think I have to restructure the script without using mrecalculate and overlap in manalyse.

Edit: maybe you have any further ideas for improvement in both speed and quality when using your version? Need to make it quite simple with for example HQ=true/false or I will have to add many possible parameters.

DTL

6th July 2023, 16:06

"is it only possible with tr=1 and mdegrain()?"

MDegrain2 is tr=2. Yes - all new features only included in MDegrainN.

Also as it was found while testing IIR mode with TTempSmooth - any IIR (with previous frames memory) filter can only run in MT_SERIALIZE AVS+ MT mode correctly. So with any IIR-setting enabled (TTH_thUPD > 0) in current MDegraiN release (up to a.23) also require to manually set MT_SERIALIZE for MDegrainN (with SetFilterMTMode(.., force=true)) and to keep multithreading - only use internal AVSTP-based multithreading (mt=true and use updated avstp.dll from pinterf to save from hangs). Thanks to gods pinterf found and fixed that odd issue in avstp and now mvtools can run again with internal multithreading as it really the only possible MT mode with 'temporal' processing like IIR-filtering enabled. MT_SERIALIZE also auto-activated for MAnalyse if 'temporal' predictor is used for the same reason.

In the next versions MDegrainN will auto-register itself with MT_SERIALIZE if any IIR-setting is activated. Maybe also try to set mt=true too ?

"I would like to make my Clay script to work with your version separately to get a speed boost and also a quality boost and yet keep close to the results I get with the current Pinterf version of mvtools.
But I think I have to restructure the script without using mrecalculate and overlap in manalyse."

Best quality of MVs expected only from multi-generation MVs refining - example also in the
https://forum.doom9.org/showthread.php?p=1987964#post1987964

It is more complex in control because it is required to adjust at least 2 different thSAD for first and next generations. With not very small tr-value for first generation it is expected significant part of noise is removed at first generation so the thSAD for second generation may be about 0.6..0.7 of thSAD of first generation. With perfect noise removed it is expected that the last thSAD is first_thSAD/2. But the best strategy of number of generations of refining and decreasing of thSAD (and may be changing tr-value from lower at first generation to higher at second and next generations) at each generation is subject of research (also may be with Zopti). Also with such research the quality metric better be structure-aware (like SSIM or VIF or other).

Also best quality expected from onCPU MAnalyse and full 4x overlap in both MAnalyse and MDegrainN (it is the slowest mode). So currently very many performance/quality modes are possible.

DTL

30th July 2023, 19:37

New version - https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.24

Added Auto-thSAD for MDegrainN.

New params to MDegrainN:
thSADA_a (float), default = 0. Multiplier proportional to estimated nosie level
thSADA_b (float), default = 0. Offset to calculated Auto-thSAD.

If both thSADA_a and thSADA_b = 0 - Auto-thSAD is disabled.

Used Auto-thSAD is a scaled and offsetted arithmetic mean of blocks SAD values below thSCD1 (noise_estimate). Next is applied adjusting params:
Auto_thSAD = thSADA_a * noise_estimate + thSADA_b

thSAD2, thSADC, thSADC2 calculated proportionally to provided old params values.

For a typical workflow user must provide both non-default thSADA_a and thSADA_b values. If only thSADA_b provided - it will be equal to static thSAD. Expected start values are thSADA_a = 1.0 and thSADA_b = 10.

Setting thSADA_a < 1.0 will make higher denoise on low noise scenes and lower at high nosie scenes.
Setting thSADA_a > 1.0 will make higher denoise on high noise scenes and higher at high noise scenes.

thSADA_b is a simple additive offset (may be negative too).

Initial release of Auto-thSAD feature for testing.

Example:

MDegrainN(last,super, multi_vec, tr, thSAD=135, thSAD2=120, mt=false, wpow=4, thSCD1=450, thSADA_a=1.05, thSADA_b=5, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, IntOvlp=3)

To provide roll-off slope for thSAD if required - user must set thSAD and thSAD2 (also thSADC and thSADC2 for chroma if required). It may be in some abstract units if Auto-thSAD is enabled (only relative ratio is calculated internally). Also users must take care of correct thSCD1 param for medium and high noised scenes. Only blocks with SAD below thSCD1 are used in noise estimation, so too low thSCD1 will cause either too bad estimation or fallback to 'static thSAD' provided as old params. Also may quickly disable any denoising if all frames will be detected as 'scenechange'.

anton_foy

1st August 2023, 00:09

DTL

1st August 2023, 09:22

"Does this new feature slow things down alot?"

I did not take tests of performance. But it is expected to be very fast and not make a visible performance hit. If performance hit is visible - performance may be better in next versions with pre-calculating of tr-weights. Currently each frame tr-weights roll-off (defined by thSAD/thSAD2 difference) calculated using float cos() function.

" optSearchOption=5 and intOvlp=3 gave less denoising and less temporal stability"

In my tests the quality of ME with the GTX1060 card is somehow worse in comparison with onCPU MAnalyse. But acceptable for the denoise of documentaries series with offloading part of work from CPU so total mvtools+x264 encoding run faster. For highest quality denoise work only onCPU MAnalyse is recommended (optSearchOption != 5/6).

Hardware ME from MPEG encoder ASIC is not simply hardware-accelerated MAnalyse but completely different ME engine may be optimized for faster MPEG encoding and not for quality. Also at each version of hardware and each vendor (NVIDIA/AMD/Intel/others ?) it may provide different quality and performance.

Maybe hardware ME can be used to make things faster in multi-generations ME refining as first generation of MAnalyse.

My current test script for 2 generations MVs refining and Auto-thSAD used:

# Input plugins
LoadPlugin("ffms2.dll")
LoadPlugin("mvtools2.dll")

SetFilterMTMode("DEFAULT_MT_MODE", 3)

my_thSADA_a=1.1
my_thSADA_b=50

my_thSAD=250
my_thSAD2=Int(Float(my_thSAD) * 0.8)

my_thSAD_mg=150
my_thSAD2_mg=Int(Float(my_thSAD_mg) * 0.8)

my_thSCD=my_thSAD+200

my_pzero=10
my_pnew=10
my_pglobal=10

my_pel=2
my_thCohMV=5 # 5..8 for pel=2, 10..16 for pel=4 ?
my_trymany=false

my_oPT=1
my_overlap=0
my_IntOvlp=3
my_searchparam=2

my_MPBNumIt=2

my_init_tr=6
my_refine_tr=6

Function RefineMV(clip mvclip, clip super_ref, clip src, int _thSAD, int _thSAD2, int in_tr, int refine_tr, int my_thSCD, int my_pel, bool my_trymany, int my_pnew, int my_pzero, int my_pglobal, \
int my_oPT, int my_overlap, int my_searchparam, int my_IntOvlp, int my_thCohMV)
{
g_next=MDegrainN(src, super_ref, mvclip, in_tr, thSAD=_thSAD, thSAD2=_thSAD2, mt=false, wpow=4, thSCD1=my_thSCD, adjSADzeromv=0.6, adjSADcohmv=0.6, thCohMV=my_thCohMV, \
MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_IntOvlp)
super_g_next=MSuper(g_next,chroma=true, mt=false, pel=my_pel)
return MAnalyse(super_g_next, SuperCurrent=super_ref, multi=true, delta=refine_tr, search=3, searchparam=my_searchparam, trymany=my_trymany, overlap=my_overlap, chroma=true, mt=false,\
optSearchOption=1, truemotion=false, pnew=my_pnew, pzero=my_pzero, pglobal=my_pglobal, global=true, optPredictorType=my_oPT)
}

Function RefineMVa(clip mvclip, clip super_ref, clip src, int _thSAD, int _thSAD2, float _thSADA_a, float _thSADA_b, int in_tr, int refine_tr, int my_thSCD, int my_pel, bool my_trymany, int my_pnew, int my_pzero, int my_pglobal, \
int my_oPT, int my_overlap, int my_searchparam, int my_IntOvlp, int my_thCohMV)
{
g_next=MDegrainN(src, super_ref, mvclip, in_tr, thSAD=_thSAD, thSAD2=_thSAD2, thSADA_a=_thSADA_a, thSADA_b=_thSADA_b, mt=false, wpow=4, thSCD1=my_thSCD, adjSADzeromv=0.6, adjSADcohmv=0.6, thCohMV=my_thCohMV, \
MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_IntOvlp)
super_g_next=MSuper(g_next,chroma=true, mt=false, pel=my_pel)
return MAnalyse(super_g_next, SuperCurrent=super_ref, multi=true, delta=refine_tr, search=3, searchparam=my_searchparam, trymany=my_trymany, overlap=my_overlap, chroma=true, mt=false,\
optSearchOption=1, truemotion=false, pnew=my_pnew, pzero=my_pzero, pglobal=my_pglobal, global=true, optPredictorType=my_oPT)
}

FFmpegSource2("src.mp4")

noproc=last

super_cpu=MSuper(last, mt=false, chroma=true, pel=my_pel, hpad=8, vpad=8, levels=0, pelrefine=true)

multi_vec_cpu=MAnalyse(super_cpu, multi=true, delta=my_init_tr, search=3, searchparam=my_searchparam, trymany=my_trymany, overlap=my_overlap, chroma=true, mt=false, \
optSearchOption=1, truemotion=false, pnew=my_pnew, pzero=my_pzero, pglobal=my_pglobal, global=true, optPredictorType=my_oPT)

multi_vec_cpu2=RefineMV(multi_vec_cpu, super_cpu, last, my_thSAD, my_thSAD2, my_init_tr, my_refine_tr, my_thSCD, my_pel, my_trymany, my_pnew, my_pzero, my_pglobal, my_oPT, \
my_overlap, my_searchparam, my_IntOvlp, my_thCohMV)

multi_vec_cpu2a=RefineMVa(multi_vec_cpu, super_cpu, last, my_thSAD, my_thSAD2, my_thSADA_a, my_thSADA_b, my_init_tr, my_refine_tr, my_thSCD, my_pel, my_trymany, my_pnew, my_pzero, my_pglobal, my_oPT, \
my_overlap, my_searchparam, my_IntOvlp, my_thCohMV)

cpu2=MDegrainN(last,super_cpu, multi_vec_cpu2, my_refine_tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, mt=false, wpow=4, UseSubShift=1, thSCD1=my_thSCD, adjSADzeromv=0.7, \
adjSADcohmv=0.7, thCohMV=my_thCohMV, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_IntOvlp, MPBthSub=5, MPBthAdd=20, MPBNumIt=my_MPBNumIt, \
MPB_SPCsub=0.5, MPB_SPCadd=1.5, MPBthIVS=2200, showIVSmask=false)

cpu2a=MDegrainN(last,super_cpu, multi_vec_cpu2a, my_refine_tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, thSADA_a=my_thSADA_a, thSADA_b=my_thSADA_b, mt=false, wpow=4, UseSubShift=1, thSCD1=my_thSCD, adjSADzeromv=0.7, \
adjSADcohmv=0.7, thCohMV=my_thCohMV, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_IntOvlp, MPBthSub=5, MPBthAdd=20, MPBNumIt=my_MPBNumIt, \
MPB_SPCsub=0.5, MPB_SPCadd=1.5, MPBthIVS=2200, showIVSmask=false)

cpu2a_s=MDegrainN(last,super_cpu, multi_vec_cpu2a, my_refine_tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, thSADA_a=my_thSADA_a, thSADA_b=my_thSADA_b, mt=false, wpow=4, UseSubShift=1, thSCD1=my_thSCD, adjSADzeromv=0.7, \
adjSADcohmv=0.7, thCohMV=my_thCohMV, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_IntOvlp)

cpu_s=MDegrainN(last,super_cpu, multi_vec_cpu, my_init_tr, thSAD=my_thSAD, thSAD2=my_thSAD2, mt=false, thSCD1=my_thSCD, IntOvlp=my_IntOvlp)

Interleave(noproc.Subtitle("src"),cpu2.Subtitle("cpu2"), cpu2a_s.Subtitle("cpu2a_s"), cpu_s.Subtitle("cpu_s"))

Prefetch(..)

Where interleaved output frames
src - input source
cpu2 - 2 generations MVs refined with MPB and static thSAD
cpu2a_s - 2 generations MVs refined without MPB and Auto-thSAD at all generations
cpu_s - single generation MAnalyse and MDegrain with static thSAD (mostly close to 2.7.45 version, only interpolated overlap used for better performance).

Settings for MAnalsye in the script are not the best possible for best quality - I set lower for better performance at my old test CPU of E7500. Better quality expected with
my_pel=4
my_thCohMV=12 # 10..16 for pel=4 ?
my_trymany=true

my_oPT=0 # all predictors used
my_overlap=4 # full 4x real search overlap - slowest
my_IntOvlp=0
my_searchparam=2 # better expected with >2 and also pelsearch > 4 (for pel=4)

MPB processing in MDegrainN still looks not make things visibly better (at least at my grainy test footage) so currently may be disabled for a bit better performance. 2 generations MVs refining sometime reduce search errors also at the borders of objects and dark parts of scenes. Using of Auto-thSAD (with old added SAD-related tweaks for static and moving and 'coherent moving' blocks with adjSADzeromv, adjSADcohmv keeps more details at some areas like moving parts with lower denoising at these areas).

tormento

3rd August 2023, 10:27

I hope to see all those news in SMDegrain soon ;)

DTL

7th August 2023, 11:40

It is not likely to be soon until we get good programmers to fix currently already added bugs. As I found with an attempt to enable internal MT with avstp.dll - both MAnalyse and MDegrain crashes with something like memory corruption. Only works stable with AVS+ global frame-based MT. So it looks like compatibility with internal MT via AVSTP is severely broken. An internal MT in MDegrainN is highly recommended if use IIR-based temporal additional filtering (only works good in MT_SERIALIZED). So I think Dogway does not like to use such not stable versions.

About using hardware ME with very slow filtering - it really greatly helps in 2 generations MVs refining. 2 MAnalyse with 'very' slow settings like pel=4, tr=12, trymany=true close to no-start at all. And with the use of DX12-ME from GTX1060 at first MAnalyse total transcoding runs at about 0.25 fps with i5-9600K CPU.

Current practical script with 'best quality' settings is:

# Input plugins
LoadPlugin("ffms2.dll")
LoadPlugin("mvtools2.dll")

SetMemoryMax(10000)

my_thSADA_a=1.3
my_thSADA_b=80

my_thSAD=250
my_thSAD2=Int(Float(my_thSAD) * 0.8)

my_thSAD_mg=150
my_thSAD2_mg=Int(Float(my_thSAD_mg) * 0.8)

my_thSCD=my_thSAD+200

my_pzero=10
my_pnew=10
my_pglobal=10

my_pel=4
my_thCohMV=4 # 5..8 for pel=2, 10..16 for pel=4 ?
my_trymany=true

my_oPT=0
my_overlap=4
my_IntOvlp=0
my_searchparam=4
my_pelsearchparam=4

my_MPBNumIt=2

my_init_tr=12
my_refine_tr=12

my_MT=false

Function RefineMVa(clip mvclip, clip super_hwa, clip super_ref, clip src, int _thSAD, int _thSAD2, float _thSADA_a, float _thSADA_b, int in_tr, int refine_tr, int my_thSCD, int my_pel, bool my_trymany, int my_pnew, int my_pzero, int my_pglobal, \
int my_oPT, int my_overlap, int my_searchparam, int _pelsearchparam, int my_IntOvlp, int my_thCohMV, bool _my_MT)
{
g_next=MDegrainN(src, super_hwa, mvclip, in_tr, thSAD=_thSAD, thSAD2=_thSAD2, thSADA_a=_thSADA_a, thSADA_b=_thSADA_b, mt=_my_MT, wpow=4, UseSubShift=1, thSCD1=my_thSCD, adjSADzeromv=0.6, adjSADcohmv=0.6, thCohMV=my_thCohMV, \
MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)
super_g_next=MSuper(g_next,chroma=true, mt=_my_MT, pel=my_pel)
return MAnalyse(super_g_next, SuperCurrent=super_ref, multi=true, delta=refine_tr, search=3, searchparam=my_searchparam, pelsearch=_pelsearchparam, trymany=my_trymany, overlap=my_overlap, chroma=true, mt=false,\
optSearchOption=1, truemotion=false, pnew=my_pnew, pzero=my_pzero, pglobal=my_pglobal, global=true, optPredictorType=my_oPT)
}

FFmpegSource2("src.mp4")

super_cpu=MSuper(last, mt=my_MT, chroma=true, pel=my_pel, hpad=8, vpad=8, levels=0, pelrefine=true)
super_hwa=MSuper(last, mt=my_MT, chroma=true, pel=4, hpad=8, vpad=8, levels=1, pelrefine=false)

multi_vec_cpu=MAnalyse(super_cpu, multi=true, delta=my_init_tr, search=3, searchparam=my_searchparam, trymany=my_trymany, overlap=my_overlap, chroma=true, mt=false, \
optSearchOption=1, truemotion=false, pnew=my_pnew, pzero=my_pzero, pglobal=my_pglobal, global=true, optPredictorType=my_oPT)

multi_vec_hwa=MAnalyse(super_hwa, multi=true, delta=my_init_tr, chroma=true, mt=false, \
optSearchOption=5, levels=1)

multi_vec_cpu2a=RefineMVa(multi_vec_hwa, super_hwa, super_cpu, last, my_thSAD, my_thSAD2, my_thSADA_a, my_thSADA_b, my_init_tr, my_refine_tr, my_thSCD, my_pel, my_trymany, my_pnew, my_pzero, my_pglobal, my_oPT, \
my_overlap, my_searchparam, my_pelsearchparam, my_IntOvlp, my_thCohMV, my_MT)

MDegrainN(last,super_cpu, multi_vec_cpu2a, my_refine_tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, thSADA_a=my_thSADA_a, thSADA_b=my_thSADA_b, mt=my_MT, wpow=4, UseSubShift=1, thSCD1=my_thSCD, adjSADzeromv=0.7, \
adjSADcohmv=0.7, thCohMV=my_thCohMV, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_IntOvlp)

Prefetch(..)

But as I see at total transcoding fps about 0.25 and FullHD frame it uses only about 1..2% of hardware encoder. So most of the CPU time looks like sitting in the second MAnalyse with slow settings for best quality.

So I get new ideas about better quality of MVs using still free resources of hardware accelerator: To use extra free resources of hardware ME accelerator (also typically not capable to make overlapping processing in mvtools-order with single search job) send several small steps shifted frames for search MVs with a bit different blocks assignment (like +-1 sample for 4:4:4 formats and +-2 samples for 4:2:0 formats) to generate 4 or 8 additional MVs around 'current' block position and calculate some averaging of these 5 or 9 MVs to get possibly more noise-free MV for current block. Averaging modes may be arithmetic mean or median (or other non-linear filtering of data 1D vector or even 2D array). To make it usable with any MAnalyse mode and any other filter consumer of MVs data - make it finally separated mvtools filter like MVProc() with 5 or 9 possible inputs from several MAnalyse (or in the future 1 input from single MAnalyse in special multi-mode). Also maybe transfer MVLPF (and other possible future MVs data intermediate processing) in this filter so it can be used with any MVs data consumer filter in complex scripting and allow to split its output to different filters using AVS scripting - for example as additional predictor for multi-generation search scripts (see feature 48 also). The number of search positions around the current block may be increased up to filling all possible integer blocks positions. Also maybe subsample shifted positions can be added too (to fill radius of 0.5..0.25 to 1.25 and more around current block position).

Expected new features script is like:

#current block pos
super=MSuper(last)
current_mvclip=MAnalyse(super,..)

#shifted 2 samples up block pos
sh2_up=Crop(0,2,last.width, last.height-2).AddBorders(0,0,2,0)
super_sh2up=MSuper(sh2_up)
sh2_up_mvclip=MAnalyse(super_sh2up,..)

# same here for shifting 2 samples left, down, right

# combine 5 MV clips from current and shifted blocks assignment
mvclip=MVProc(current_mvclip, sh2_up_mvclip, sh2_down_mvclip, sh2_left_mvclip, sh2_right_mvclip, average_mode='median',.., optional MVLPF and other)

MDegrainN(last, super, mvclip,...)

So it still requires some development time to check this idea of MVs refining at the typical complex places like low contrast and heavily noisy areas. Where noise close or above amplitude of texture details so simple MVs search typically greatly fails and it causes details blurring with MDegrain.

kedautinh12

7th August 2023, 12:29

Pinterf was fixed bug from AVSTP
https://github.com/pinterf/AVSTP/releases

DTL

7th August 2023, 17:21

Yes - so it is usable with AVS+ again. But now it only can run with a 2.7.45 build because I did not test it during the years of development of my version and it looks like several crash-bugs with internal MT using AVSTP was accumulated.

Some test script with 5x overlap with hardware ME (and vsttempsmooth pmode=1 as median-like 'best' sample value select engine at the end):

# Input plugins
LoadPlugin("ffms2.dll")
LoadPlugin("mvtools2.dll")
LoadPlugin("vsTTempSmooth.dll")

SetMemoryMax(10000)

my_thSADA_a=1.1
my_thSADA_b=60

my_thSAD=250
my_thSAD2=Int(Float(my_thSAD) * 0.8)

my_thSAD_mg=150
my_thSAD2_mg=Int(Float(my_thSAD_mg) * 0.8)

my_thSCD=my_thSAD+200

my_thCohMV=4

my_refine_tr=12

my_MT=false

FFmpegSource2("src.mp4")

AddBorders(16,16,16,16)

super_hwa_center=MSuper(last, mt=my_MT, chroma=true, pel=4, hpad=8, vpad=8, levels=1, pelrefine=false)

src=last
shift_val=4

src_up=Crop(0,shift_val,src.width-0,src.height-shift_val).AddBorders(0,0,0,shift_val)
src_down=Crop(0,0,src.width-0,src.height-shift_val).AddBorders(0,shift_val,0,0)
src_left=Crop(shift_val,0,src.width-shift_val,src.height-0).AddBorders(0,0,shift_val,0)
src_right=Crop(0,0,src.width-shift_val,src.height-0).AddBorders(shift_val,0,0,0)

super_hwa_up=MSuper(src_up, mt=my_MT, chroma=true, pel=4, hpad=8, vpad=8, levels=1, pelrefine=false)
super_hwa_down=MSuper(src_down, mt=my_MT, chroma=true, pel=4, hpad=8, vpad=8, levels=1, pelrefine=false)
super_hwa_left=MSuper(src_left, mt=my_MT, chroma=true, pel=4, hpad=8, vpad=8, levels=1, pelrefine=false)
super_hwa_right=MSuper(src_right, mt=my_MT, chroma=true, pel=4, hpad=8, vpad=8, levels=1, pelrefine=false)

mv_hwa_center=MAnalyse(super_hwa_center, multi=true, delta=my_init_tr, chroma=true, mt=false, optSearchOption=5, levels=1)
mv_hwa_up=MAnalyse(super_hwa_up, multi=true, delta=my_init_tr, chroma=true, mt=false, optSearchOption=5, levels=1)
mv_hwa_down=MAnalyse(super_hwa_down, multi=true, delta=my_init_tr, chroma=true, mt=false, optSearchOption=5, levels=1)
mv_hwa_left=MAnalyse(super_hwa_left, multi=true, delta=my_init_tr, chroma=true, mt=false, optSearchOption=5, levels=1)
mv_hwa_right=MAnalyse(super_hwa_right, multi=true, delta=my_init_tr, chroma=true, mt=false, optSearchOption=5, levels=1)

dg_center=MDegrainN(src, super_hwa_center, mv_hwa_center, my_refine_tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, thSADA_a=my_thSADA_a, thSADA_b=my_thSADA_b, mt=my_MT, wpow=4, UseSubShift=1, thSCD1=my_thSCD, adjSADzeromv=0.7, \
adjSADcohmv=0.7, thCohMV=my_thCohMV, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)

dg_up=MDegrainN(src_up, super_hwa_up, mv_hwa_up, my_refine_tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, thSADA_a=my_thSADA_a, thSADA_b=my_thSADA_b, mt=my_MT, wpow=4, UseSubShift=1, thSCD1=my_thSCD, adjSADzeromv=0.7, \
adjSADcohmv=0.7, thCohMV=my_thCohMV, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)

dg_down=MDegrainN(src_down, super_hwa_down, mv_hwa_down, my_refine_tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, thSADA_a=my_thSADA_a, thSADA_b=my_thSADA_b, mt=my_MT, wpow=4, UseSubShift=1, thSCD1=my_thSCD, adjSADzeromv=0.7, \
adjSADcohmv=0.7, thCohMV=my_thCohMV, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)

dg_left=MDegrainN(src_left, super_hwa_left, mv_hwa_left, my_refine_tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, thSADA_a=my_thSADA_a, thSADA_b=my_thSADA_b, mt=my_MT, wpow=4, UseSubShift=1, thSCD1=my_thSCD, adjSADzeromv=0.7, \
adjSADcohmv=0.7, thCohMV=my_thCohMV, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)

dg_right=MDegrainN(src_right, super_hwa_right, mv_hwa_right, my_refine_tr, thSAD=my_thSAD_mg, thSAD2=my_thSAD2_mg, thSADA_a=my_thSADA_a, thSADA_b=my_thSADA_b, mt=my_MT, wpow=4, UseSubShift=1, thSCD1=my_thSCD, adjSADzeromv=0.7, \
adjSADcohmv=0.7, thCohMV=my_thCohMV, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=3)

#move shifted back
dg_up=Crop(dg_up, 0,0,dg_up.width-0,dg_up.height-shift_val).AddBorders(0,shift_val,0,0)
dg_down=Crop(dg_down, 0,shift_val,dg_down.width-0,dg_down.height-shift_val).AddBorders(0,0,0,shift_val)
dg_left=Crop(dg_left, 0,0,dg_left.width-shift_val,dg_left.height-0).AddBorders(shift_val,0,0,0)
dg_right=Crop(dg_right, shift_val,0,dg_left.width-shift_val,dg_left.height-0).AddBorders(0,0,shift_val,0)

intrl=Interleave(dg_center, dg_up, dg_down, dg_left, dg_right)
vstt=vsTTempSmooth(intrl, ythresh=200, uthresh=200, vthresh=200, pmode=1, maxr=2)
SelectEvery(vstt, 5,2)

Crop(16,16,width-32, height-32)

Prefetch(..)

Using 5x 'overlapped' processing fixes some small search errors and gives lower overall noise level. In comparison with the 'center' output clip. But still have some general search errors in comparison with onCPU MAnalyse with 'max' settings. It is still not tested as 'prefilter/1st generation' processing in 2 or more generations of MVs refining.

" see all those news in SMDegrain soon"

Maybe the only small and still important features like second input to MAnalyse and auto-thSAD to MDegrain may be ported to yet another 'simple addition to 2.7.45 pinterf version' as mostly safe from bugs changes. And Dogway may test it in the SMDegrain. I hope pinterf may return back in the 2023 and I can ask about making port of some very limited pack of simple new features to make some 'official-post-2.7.45' build of mvtools.

DTL

26th September 2023, 19:59

New release https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.25

Added non-linear Median-like MV filtering mode in addition to linear low-pass filtering to MDegrainN.

New params to MDegrainN:
MVMedF (integer) - default 0 (disabled). Temporal radius of median filtering of temporal MVs sequence. Valid range - from 1 to about 1/3 of tr-value used.
MVMedF_em (integer) - default 0. Edges of MV temporal vector processing modes. Mode 0 - copy non-filtered MVs from input. Mode 1 - invalidate non-filtered frames MVs to save from possible blending of bad MVs. Number of non-filtered frames equal to MVMdeF value.
MVMedF_cm (integer) - default 0. MVs coordinates processing mode. Mode 0 - separated x,y vectors median filtering. Mode 1 - using length of difference vector as dissimilarity metric.

Example:

MDegrainN(last,super, multi_vec, tr, thSAD=135, thSAD2=120, mt=false, wpow=4, thSCD1=450, thSADA_a=my_thA, thSADA_b=my_thB, \
adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, MVMedF=2, MVMedF_cm=0, MVMedF_em=0, IntOvlp=3)

For any used MV filtering mode thMVLPFCorr must be non-zero (to use filtered MVs with coordinates difference from input MVs below this value).
Both linear and non-linear MVs filtering may be enabled in any combination. First executed non-linear filtering and next linear.
MVMedF_em=1 may be used for possible higher quality processing (non-filtered MVs/blocks at the edges of tr excluded from blending). But it eats MVMedF frames from total tr-pool and decreases possible denoise level (so to keep same max denoise level with MVMedF_em=1 tr need to be tr+MVMedF).

MVMedF_cm=1 may produce more shifted areas of moving objects - it is subject of testing and may be fix possible in next releases.

Expected working values for MVMedF temporal radius of non-linear filtering - about 1..3 (may be up to about 1/3 of the tr value used, so for MVMedF=3 recommended tr is about 6..10). Values of 3 and more not yet tested.

Non-linear filtering expected to work better in skipping false long strike MVs with lower tr in compare with linear filtering.

Test script for compare new MVs processing features with old (2022 ?) source from post https://forum.doom9.org/showthread.php?p=1974040#post1974040 :

LoadPlugin("mvtools2_260923.dll")
LoadPlugin("ffms2.dll")

FFmpegSource2("test_org.mkv")

examp=FFMpegSource2("test_enc_thSAD200.mkv").Crop(250,200,500,500).ConvertToYUV420(matrix="Rec709").ConvertBits(8).Subtitle("test_enc")

ConvertToYUV420(matrix="Rec709")

Crop(250,200,500,500)

noproc=last.Subtitle("src")

super_std=MSuper(mt=false, pel=2)

tr=14

my_thA = 1.3
my_thB = 30

multi_vec=MAnalyse(super_std, blksize=8, multi=true, search=3, temporal=false, trymany=true, searchparam=2, chroma=true, delta=tr, truemotion=false,\
pzero=10, pnew=10, pglobal=10, levels=0, mt=false, overlap=4)
new_m2=MDegrainN(last,super_std, multi_vec, tr, thSAD=150, thSAD2=140, thSADA_a=my_thA, thSADA_b=my_thB, mt=false, wpow=4, thSCD1=500, \
adjSADzeromv=0.7, adjSADcohmv=0.7, thCohMV=6, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, MVMedF=2, MVMedF_cm=0, MVMedF_em=1,\
IntOvlp=0).Subtitle("new_m2")
old=MDegrainN(last,super_std, multi_vec, tr, thSAD=200, thSAD2=190, mt=false,IntOvlp=0).Subtitle("old_thSAD200")
ConvertBits(8)

Interleave(examp, new_m2, old, noproc)
Sharpen(0.5)

SincLin2Resize(width*2, height*2)

Prefetch(2)

155 (src) frame of interleaved sequence compare https://imgsli.com/MjA5OTQx/1/3

DTL

28th September 2023, 17:06

New release - https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.26

Added new MVF_fm param to MDegrainN. Fixed MV filtering in the non-YUV-combined processing modes.

MVF_fm (integer), default=0 . Blocks failing mode at the process of MVs filtering. Mode 0 (default) - pass blocks with failed SAD re-check unchanged to blending engine. Mode 1 - fail (invalidate to blending) blocks with failed SAD re-check after filtered MVs coordinates.

In MVF_fm=1 mode it saves more blocks from blurring but it typically cause degradation of denoising at these areas. So cause uneven denosie over total frame area. May be more visible when compare static frames.

DTL

2nd October 2023, 20:19

Some strategic announcement about fully hardware multi-generation MVs refining for noised sources.

I made some onCPU tech tests of multi-generation refining in MDegrainN (using simple ESA search algorithm) - even with a search radius of 4 it is much slower. Though still faster in comparison with script-based refining (and uses about 2 times less RAM). It is expected to be somehow faster after most possible optimizations with SIMD but not very much possible.

So as hardware MVs search ASIC typically shows significant underload - it is possible to put MVs refining into hardware accelerator (new modes for MAnalyse). Major part is development of MDegrainN simple or most featured processing as Compute Shader so it can be dispatched in HWA without downloading current generation MVs for external processing. It not breaks logic of MAnalyse in AVS filterchain - it still outputs single MVs clip with refined MVs completely in the hardware accelerator using hardware MVs search ASIC and hardware universal shaders dispatch units to dispatch SAD and MDegrainN shader to provide filtered frames to next generations of MVs refining by same hardware MVs search ASIC. For boards with 2 or more NVENC ASICs I hope drivers are smart enough to create a full filterchain inside accelerator board without download-upload resources between degrain and MVs search stages and spread load over all available ASICs onboard.

tormento

12th March 2024, 07:49

New release
Maybe this (https://devblogs.microsoft.com/pix/pix-2403/) can be useful.

DTL

31st March 2024, 08:04

New release https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.27

Added more 'area' predictors to MAnalyse. Extended optPredictorType to -1,-2, -3.

Added AreaMode MVs refining to MAnalyse.
New params:
AreaMode (integer), valid values 0,1,2,3,4.
0 - disabled
1 - x5 total positions checks (center + 4 diagonal offsets of +-1)
2 - x9 total positions checks (center + 8 diagonal offsets of +-1 and +-2)
3 - x13 total positions checks (center + 12 diagonal offsets of +-1 and +-2 and +-3)
4 - x17 total positions checks (center + 16 diagonal offsets of +-1 and +-2 and +-3 and +-4)

AMdiffSAD (integer), valid values 0 and positive. Recommended range 1..1000.
Allow to add MVs absolute difference from AreaMode search to the SAD to send additional hints about block MV quality. Multiplier to the mean sum of abs MVs coordinates differences. Values about 1000 and higher totally fail the SAD value of the block.

Example of progressive film processing script with difference control from different AreaMode setting:

tr=12
my_AMDiffSAD=0
my_thSADA_a=1.2

my_intOvlp=0
my_ovlp=4

super=MSuper(last, mt=false, pel=2, hpad=8, vpad=8)

multi_vec_cpu=MAnalyse (super, multi=true, blksize=8, delta=tr, search=3, searchparam=2, truemotion=true, overlap=my_ovlp, chroma=false, optSearchOption=1, optPredictorType=0, mt=false)
multi_vec_am5=MAnalyse (super, multi=true, blksize=8, delta=tr, search=3, searchparam=2, truemotion=true, overlap=my_ovlp, chroma=false, optSearchOption=1, optPredictorType=0, mt=false, AreaMode=1, AMdiffSAD=my_AMDiffSAD)
multi_vec_am9=MAnalyse (super, multi=true, blksize=8, delta=tr, search=3, searchparam=2, truemotion=true, overlap=my_ovlp, chroma=false, optSearchOption=1, optPredictorType=0, mt=false, AreaMode=2, AMdiffSAD=my_AMDiffSAD)
multi_vec_am13=MAnalyse (super, multi=true, blksize=8, delta=tr, search=3, searchparam=2, truemotion=true, overlap=my_ovlp, chroma=false, optSearchOption=1, optPredictorType=0, mt=false, AreaMode=3, AMdiffSAD=my_AMDiffSAD)

ma_cpu=MDegrainN(last,super, multi_vec_cpu, tr, thSADA_a=my_thSADA_a, thSADA_b=50, mt=false, wpow=4, adjSADzeromv=0.8, adjSADcohmv=0.8, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_intOvlp).Subtitle("ma_cpu")
ma_cpu_am5=MDegrainN(last,super, multi_vec_am5, tr, thSADA_a=my_thSADA_a, thSADA_b=50, mt=false, wpow=4, adjSADzeromv=0.8, adjSADcohmv=0.8, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_intOvlp).Subtitle("ma_am5")
ma_cpu_am9=MDegrainN(last,super, multi_vec_am9, tr, thSADA_a=my_thSADA_a, thSADA_b=50, mt=false, wpow=4, adjSADzeromv=0.8, adjSADcohmv=0.8, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_intOvlp).Subtitle("ma_am9")
ma_cpu_am13=MDegrainN(last,super, multi_vec_am13, tr, thSADA_a=my_thSADA_a, thSADA_b=50, mt=false, wpow=4, adjSADzeromv=0.8, adjSADcohmv=0.8, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_intOvlp).Subtitle("ma_am13")

Interleave(ma_cpu, ma_cpu_am5, ma_cpu, ma_cpu_am9, ma_cpu, ma_cpu_am13, Subtract(ma_cpu, ma_cpu_am13).Levels(100,1, 140, 0,255))
Sharpen(1.0)

The benefit from more predictors added is very small but may be tested. They are median (mode ?) of the surround block predictors from previous levels of search. Performance cost is not big (in single predictor refining mode with trymany=false).

The performance cost of AreaMode search is great - it is +4, +8, +12, +16 new full searches runs around current block position and next computing of median (mode ?) of the resulted MVs vector to create output MV (SAD is the max of the selected best dx and dy MVs found). AreaMode=3 (12 additional block searches) runs about 5 times slower in comparison with standard single search per block at i5-9600K CPU.
https://i.postimg.cc/qMkJ1BHn/AM-drawing01.png

Dark film scene compare (gamma=2.0 added to show darks better) https://imgsli.com/MjUxNjY3

Also this new level of processing generates more data for analysis of 'quality of MV' and currently a simple idea is used to additionally signal to the denoising engine: If MVs of the small shifted positions around the current block are not coherent - it means MV estimation may be unstable. So the absolute difference of the MVs coordinates (averaged to the number of MVs in the search pool to make no (less ?) dependence of the AreaMode used) multiplied to the AMdiffSAD param may be added to block SAD. So this block will take less degrain weight in MDegrain. Users can try to control the effect of this setting using MShow for average SAD per frame (showsad=true).

For the areas of stable MVs enabling AreaMode makes close to no difference and great performance hit. For the areas of unstable MVs it adds some more quality in denoising (may be good visible at 600..800% crops). x264 encoding bitrate at fixed CRF=18 is also a bit lower (about 3% at some quick test encodings). In the future I hope to add close to this idea for the DX12 accelerated ME. So the ME ASIC may be finally good loaded with useful work. But I still not yet fully restore my development environment (so currently no DX12-ME build available) and also may need to ask Microsoft support how to fast shift loaded resource in the accelerator to send a queue of searches to ASIC in 1 job list instead of uploading lots of shifted copies of the frames to make searches with shifted block tessellation grid.

The AreaMode level is located over standard levels of search in MAnalyse and compatible with any old mode but currently looks like work only with block size 8x8 because of experimental release not fully debugged. In the future it is planned to move into -e.XX builds expected to be more stable in comparison with -a.XX builds.

Also some observation: enabling truemotion=true also significantly helps to MVs stability in the low contrast low detailed noised areas recommended to be enabled in the high quality use cases.

Addition: Current ideas on performance optimization: Limit AreaMode search to full-pel level only (or to some non-finest level depending on current pel-setting). But the full search algorithm still can fail to 'best SAD' and not really best MV while checking lots of predictors (starting from zero predictor). So for using full search with AreaMode down to full-pel level and only sub-sample refining it may be added some more PredictorsTypes (like optPredictorType=5 or something else). Or as I long time thinking it is better to make many control params of MAnalyse arrays instead of single value for all levels. With arrays user can set much more flexible performance/quality balance on each level. Currently MAnalyse only have separate param like persearch to set different search radius at the sub-pel levels. But it can be expanded to many other search params to adjust performance/quality at user side and not hardcode into MAnalyse as 'hard presets'. But I still not have experience in arrays params for AVS filters. So current more easy ways is to add more params like AMll (for AreaMode level limit) and maybe pelPT (for pel PredictorType selection for fixed sets of predictors used at sub-sample levels of search) and maybe peltrymany (to select usage of refining of all predictors or best only at the sub-sample levels). So user can select more quality search at the levels down to full-sample precision (fast enough) and select only limited search at the slowest pel=2 and pel=4 levels for better performance.

guest

2nd April 2024, 01:23

New release https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.27

I hate to say it DTL, but any build after Release_2.7.46_e.03, causes an error in RipBot264 "Cannot render the file" when previewing script in AVSMeter.

With either DX12 or noDX12, build .03 had AVX options.

DTL

2nd April 2024, 09:34

Oh - a.XX builds are the most featured but unstable for many blocksizes/bitdepths. Typically tested only with YV12 and 8x8 blocksize. So if it crash even with YV12 and 8x8 blocksize - write at least frame size (or better full script) so I can try to look in the debugger. The e.03 build only have very small number of new features after 2.7.45 added but expected to be most stable and support all bitdepths/blocksizes as in 2.7.45.

I plan to add AreaMode search mode into e.XX builds but some time later after finishing most of its settings.

With the very limited developer resources at the residuals of current civilization the new features tested and debugged modes are quickly shrinks to the very limited and mostly used format like YV12 and 8x8 blocksize (like at the very beginning of AVS and mvtools). Though I understand the better quality of MVs is expected with at least dual-blocksize processing like first search with blocksize of 16x16 and next is refine to 8x8 with MRecalculate. So most of new controls to MAnalyse need to be added to MRecalculate too.

It may be also natural limiting to AreaMode finest level (also different area-size) - like first search MAnalyse(AreaMode=3, blksize=16) and next refine with MRecalculate(AreaMode=1, blksize=8) for better performance and maybe quality too because of larger blocksize at the first search.

guest

2nd April 2024, 10:53

Oh - a.XX builds are the most featured but unstable for many blocksizes/bitdepths.

I will double check the script I used, but I generally don't have any blocksize settings, and don't know what the video I'm using is, either.

Will report back....

DTL

2nd April 2024, 13:55

Can you post sample video and script used ?

guest

3rd April 2024, 02:45

Can you post sample video and script used ?

Well, another day, and a different result...

.27 is now working, and all I have changed is to update FFmpeg...

DTL

3rd April 2024, 08:48

The sad truth about complex bugs with 'memory corruption' in the still poorly protected memory in current OS/CPUs design - if some thread writes out of the buffer it may be not any visible while the overwritten memory region not used as some executable page or non-checked data bytes. It may be the main source of 'hidden bugs' accumulation - they are invisible untill damage something else important and cause crash finally. The complexity of search of these bugs is because there is no records who and when made write to damaged memory bytes. Crash happens only when damaged data read and used somewhere. Also because of close to random 4KB RAM pages location in address space - each next OS reboot and each next process run may cause different damaged memory area.

So the simple version: a.XX build with some settings cause some RAM page corruption but not all ffmpeg builds uses same memory allocation scheme so this corruption may remain hidden and not damage any important data. So ffmpeg runs without crash and maybe without image data distortion. But everything may change at any time. This is sad process of constant accumulation of overall non-stability and hard to debug the real place where it starts. There exists some automated software tools for program instrumenting for many (every ?) write operation control to help found these bugs. But they may work with C-writes and not check long SIMD data writes. And mvtools uses many hand coded asm parts and these parts may be not compatible with automated instruments for memory damaging analysis.

tormento

4th April 2024, 13:28

New release 2.7.46-a.27
I can't test on my Sandy Bridge, gives instruction error.

Did you compile for AVX2 only?

DTL

4th April 2024, 20:28

New release: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.28

Added new params to MAnalyse and MRecalculate:

AMstep (integer): default 0 (auto) or 1 and higher. Step of each 4 area search positions offset (diagonal) around center position of block. 0 auto mean scaled to 8x8 block size (auto mean 1 for 8x8 block size, 2 for 16x16 block size and so on).
AMoffset (integer): default 0. Offset from 1 to start of area positions.

New params for MAnalyse:
PTpel (integer): PredictorType at sub-sample levels of search. Default = optPredictorType.
AMpel (integer): AreaMode for sub-sample levels (level 0 for pel=1 or level 0 and level 1 for pel=2). Default = AreaMode.

New params for MRecalculate:
AreeaMode, AMdiffSAD - same as for MAnalyse.

Fixed issue with crash with block size 16x16 (and possibly others) with SIMD instructions enabled (SetMAXCPU > 'none').

Added passing of tr-value to nTrad new member of analysisdata structure to mvmulti clip from MRecalculate (to be compatible with compatibility check in new MDegrainN).

"I can't test on my Sandy Bridge, gives instruction error."

Yes - that was AVX2 only build. I added SSE2 and AVX2 in update to current release a.28 at github.

Example of test script starting with pel=1 and blocksize of 16x16 and MRecalculate to pel=2 and blocksize 8x8:

my_tr=12
my_AMDiffSAD=0
my_thSADA_a=1.2

my_intOvlp=0
my_ovlp=0
my_blksize=16
mymrecthSAD=200

super_p2=MSuper(last, mt=false, pel=2, hpad=32, vpad=32)
super_p1=MSuper(last, mt=false, pel=1, hpad=32, vpad=32)

multi_vec_cpu=MAnalyse (super_p2, multi=true, blksize=8, delta=my_tr, search=3, searchparam=2, truemotion=true, overlap=my_ovlp, chroma=false, optSearchOption=1, optPredictorType=0, mt=false)

vec_am0=MAnalyse (super_p1, multi=true, blksize=my_blksize, delta=my_tr, search=3, searchparam=2, truemotion=false, pnew=0, global=true, overlap=my_ovlp, chroma=true,\
optSearchOption=1, optPredictorType=0, mt=false, AreaMode=0, AMstep=2, AMdiffSAD=my_AMDiffSAD)
vec_am1=MAnalyse (super_p1, multi=true, blksize=my_blksize, delta=my_tr, search=3, searchparam=2, truemotion=false, pnew=0, global=true, overlap=my_ovlp, chroma=true,\
optSearchOption=1, optPredictorType=0, mt=false, AreaMode=1, AMstep=2, AMdiffSAD=my_AMDiffSAD)

multi_vec_mrec_am0=MRecalculate(super_p2, vec_am0, thSAD=mymrecthSAD, blksize=8, search=3, searchparam=4, truemotion=false, pnew=0, chroma=true, overlap=my_ovlp,\
AreaMode=0, tr=my_tr)
multi_vec_mrec_am2=MRecalculate(super_p2, vec_am1, thSAD=mymrecthSAD, blksize=8, search=3, searchparam=4, truemotion=false, pnew=0, chroma=true, overlap=my_ovlp, \
AreaMode=2, tr=my_tr)

ma_cpu=MDegrainN(last,super_p2, multi_vec_cpu, my_tr, thSADA_a=my_thSADA_a, thSADA_b=50, mt=false, wpow=4, adjSADzeromv=0.8, adjSADcohmv=0.8, thCohMV=16,\
MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_intOvlp).Subtitle("ma_cpu")
ma_cpu_mrec_am0=MDegrainN(last,super_p2, multi_vec_mrec_am0, my_tr, thSADA_a=my_thSADA_a, thSADA_b=50, mt=false, wpow=4, adjSADzeromv=0.8, adjSADcohmv=0.8,\
thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_intOvlp).Subtitle("ma_am_mrec_am0")
ma_cpu_mrec_am=MDegrainN(last,super_p2, multi_vec_mrec_am2, my_tr, thSADA_a=my_thSADA_a, thSADA_b=50, mt=false, wpow=4, adjSADzeromv=0.8, adjSADcohmv=0.8,\
thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_intOvlp).Subtitle("ma_am_mrec_am")

Interleave(ma_cpu, ma_cpu_mrec_am0, ma_cpu_mrec_am, Subtract(ma_cpu, ma_cpu_mrec_am0).Levels(100,1, 140, 0,255), Subtract(ma_cpu, ma_cpu_mrec_am).Levels(100,1, 140, 0,255))
Sharpen(1.0)

Still not any best params adjusted - just initial working to first tests.

tormento

5th April 2024, 09:34

Yes - that was AVX2 only build. I added SSE2 and AVX2 in update to current release a.28 at github.
:thanks:

tormento

5th April 2024, 09:35

Yes - that was AVX2 only build. I added SSE2 and AVX2 in update to current release a.28 at github.
I can see AVX2/SSE2 build only.

If you can, please provide AVX version too, so I can report issues if any.

tormento

5th April 2024, 10:29

Benchmark on a simple SMDegrain:

MVTools 2.7.46·e03 (AVX ) 7,93 fps
MVTools 2.7.46·a28 (SSE2) 7,28 fps

Then I checked some metrics between MVTools 2.7.46·e03 and MVTools 2.7.46·a28:

PSNR 52.9363
SSIM 0.9974
VMAF 96.0528

DTL

5th April 2024, 15:48

I updated release from 04.04.24 with DX12 and noDX12 SSE2/AVX/AVX2 builds with Visual Studio 2019.

Unfortunately a.XX versions may be really slower in compare with very close to 2.7.45 version e.03 version because of active usage of many possible dissimilarity metrics (SAD, SSIM, VIF) and runtime sub-sample shifting in MAnalyse. a.XX builds are full featured for MAnalyse but contains lots of conditional jumps. These conditional jumps disturbs out-of-order and branch-prediction units of CPUs and may make visible performance penalty.

To regain some performance you need to enable at least optSearchOption=1.

To somehow fix this new performance penalty it is planned to re-design program text of MAnalyse to templated versions of processing functions (use DMFlags dissimilarity metric control or old SAD only and UseSubShift or not). Manual copy of these options and selection of used function of startup will creates even more scary file PlaneofBlocks.cpp and more complex to future development. But this planned templated re-design only possible in some future versions. Now if testing shows the a.28 version finally start to work with typical still used complex scripts like SMDegrain and QTGMC - I can only ask Asd-g to make LLVM builds (typically somehow faster). And if LLVM compiler is more smart and AI-powered inside as we expect from still not completely died progress of civilization it may auto-optimize these conditional jumps better in comparison with simple VS2019 compiler.

tormento

6th April 2024, 11:03

I updated release from 04.04.24 with DX12 and noDX12 SSE2/AVX/AVX2 builds with Visual Studio 2019.
Thanks. I will try and report ASAP.

I think you should give a try to Intel Compiler. AFAIK it produces the fastest builds, even for AMD.
To regain some performance you need to enable at least optSearchOption=1.
We need to convince Dogway to implement some of your new features in his very good scripts. ;)

DTL

6th April 2024, 12:11

" Intel Compiler. AFAIK it produces the fastest builds, even for AMD."

I tried it in beginning of 202x years and it makes some small visible benefit over VisualStudio compiler. But as we see with Asd-g builds the progress of development of LLVM compilers is good and they makes best binaries. Maybe I also someday will try to install LLVM somehow to my development system.

Current best in quality degrain script with first search with 16x16 blocksize pel=1 and recalculating to 8x8 blocksize pel=2 before MDegrainN (with some not very slow speed):

my_tr=12
my_AMDiffSAD=0
my_thSADA_a=1.2

my_intOvlp=3
my_ovlp=0
my_blksize=16

super_p2=MSuper(last, mt=false, pel=2, hpad=32, vpad=32)
super_p1=MSuper(last, mt=false, pel=1, hpad=32, vpad=32)

vec_am1o4=MAnalyse (super_p1, multi=true, blksize=my_blksize, delta=my_tr, search=3, searchparam=2, truemotion=true, pnew=0, global=true, overlap=my_ovlp, chroma=true,\
optSearchOption=1, optPredictorType=0, mt=false, AreaMode=1, AMoffset=4, AMdiffSAD=my_AMDiffSAD)

multi_vec_mrec_am1o4am2recall=MRecalculate(super_p2, vec_am1o4, thSAD=20, blksize=8, search=3, searchparam=4, truemotion=true, pnew=0, chroma=true, overlap=my_ovlp, \
AreaMode=2, AMoffset=0, tr=my_tr)

MDegrainN(last,super_p2, multi_vec_mrec_am1o4am2recall, my_tr, thSADA_a=my_thSADA_a, thSADA_b=50, mt=false, wpow=4, adjSADzeromv=0.8, adjSADcohmv=0.8,\
thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50, adjSADLPFedmv=0.9, IntOvlp=my_intOvlp)

The thSAD param for MRecalculate is significant quality/performance balancing value:

1. Low or zero thSAD: Perform refining search of all new recalculated (interpolated) MVs. Slowest mode but may give best quality.
2. Default of 200: Depend on noise level and motion can cause more or less refining searches for blocks with SAD > thSAD.
3. Very high value (about thSCD1 ): Only interpolate MVs and re-check SAD for interpolated MV. Fastest mode but may give lower quality (depend on the noise level/profile and may more).

About low AreaMode setting (starting from 1 'layer'): It looks some better quality happen with non-zero AMoffset value and expected good value about blocksize/4. For higher AreaMode settings the best AMstep/AMoffset values depending on blocksize is still subject of many tests.
For blocksize of 16x16 AMoffset=0 and AreaMode=1 cause only additional search with +-1 of block's center position and it is only 1/16 of block size. With AMoffset=4 more surround samples used in lower number of new search calculations and it looks create more benefit in quality per used CPU cycles.
Current equation for offset from center block's position for each AreaMode step/'layer' (i) is int iOffset = (iAMstep * i + iAMoffset + 1); and new 4 searches of MAnalyse performed with 4 search positions of +-iOffset (diagonal, not sides). So for AreaMode=1, AMoffset=0, AMstep=1 4 new searches performed with +-1 offset (in current 'pel' scale - different for each level and pel setting). It may be not effective (for areas with complex motion) to make additional searches with iOffset > blocksize or even blocksize/2.

Also the AreaMode algorithm may have some 'natural' limit of MVs quality increase with increasing of number of new search positions. Currently internal (not checked) limit of search positions vector is limited to 100 (expected too slow and no one used ever). So user can test AreaMode offsets much larger than blocksize as experiments.

Currently compute complexity is of linear scale from AreaMode setting (number of new search positions is linear scale 4 of AreaMode setting). It is possible to add check of all other possible integer positions around center with given radius (including sides positions and all intermediate). But it will make compute complexity up to square of AreaMode setting. May be in some future version may be added at least sides check option (it only increases complexity 2x linear). Maybe as additional AMflags param like:
AMflags 1 - diagonal offsets
AMflags 2 - sides offsets
AMflags 4 - all offsets in defined by AreaMode+AMstep+AMoffset area.
So user can select AMflags 1+2=3 for diagonal+sides offsets for example. For 'big' blocksizes like 16x16 or 32x32 using AreaMode=1, AMoffset=blocksize/4, AMflags=3 may be better in performance/quality balance. For blocksize 8x8 AMFlags=3 equal to all possible positions with AreaMode=1 (4 diagonal and 4 sides) and so on.

guest

13th April 2024, 03:58

I have a question...

In the case of encoding using Distributed Encoding (which can use different CPU's), some are AVX only, some are AVX2, and one has AVX512...

Which "variety" of mvtools would be used (I am yet to be able to test for myself), just curious....

DTL

13th April 2024, 11:02

You do not describe how the encoding is distrubuted - by different movies to different hosts or some cut/interleaved parts of single movie to different hosts or even much more methods possible.

For example I now thinking of distrubuting of MAnalyse analysis for AreaMode of different offsets to different hosts if possible and agregate several MVs frames in the single decision filter like MAvg().

For best performance you can install different builds (AVX, AVX2, AVX512) to different hosts. But in some cases the computing results may be somehow different. To get most equal results as possible it may be recommended to install single (lowest SIMD family) build to all hosts and also use SetMaxCPU() to limit all execution to lowest equal SIMD functions.

guest

13th April 2024, 12:00

You do not describe how the encoding is distributed - by different movies to different hosts or some cut/interleaved parts of single movie to different hosts or even much more methods possible.

For example I now thinking of distributing of MAnalyse analysis for AreaMode of different offsets to different hosts if possible and aggregate several MVs frames in the single decision filter like MAvg().

For best performance you can install different builds (AVX, AVX2, AVX512) to different hosts. But in some cases the computing results may be somehow different. To get most equal results as possible it may be recommended to install single (lowest SIMD family) build to all hosts and also use SetMaxCPU() to limit all execution to lowest equal SIMD functions.

Hi DTL, glad you asked :)

I use RipBot264 (see sig), and it has this unique feature, that AFAIK, no other encoding app has anything like it.

Staxrip comes close with it's chunk encoding, but it can only use the PC it's being used on :(

RipBot264 can encode "chunks" that the main server PC processes, and then the client (up to 15 other PC's, on a LAN) can then encode these chunks speeding up the process enormously, then when all the chunks are encoded, the client PC's idle, until the main server then combines the chunks, and muxes it all into the completed encoded video/audio file, then if there is another job in the queue, it will commence processing & compiling the chunks to start the process again.

So, to my question, say the server/main PC is AVX2 or AVX512, so you use the AVX2 build of your mvtools, however, some of the PC's in the encoding "farm" are only AVX, or lower, would a script using mvtools (SMDegrain) work on the AVX PC's ??

But for example, if I use the x265 AVX512 command line on my Ryzen 7950X, then the others will automatically disable that command, and proceed.

Would mvtools do something similar ??

Quite interested in the SetMaxCPU(), that might be a good option.

DTL

13th April 2024, 14:42

"some of the PC's in the encoding "farm" are only AVX, or lower, would a script using mvtools (SMDegrain) work on the AVX PC's ??"

Script will work if you do not set options with AVX2 or higher requirements (optSearchOption=2 and more). If CPU not supported it must throw error about no AVX2 (or AVX512) present and exit. Also you need to install AVX or lower builds of mvtools2.dll to AVX hosts.

There are 2 different types of SIMD optimizations in current mvtools2 builds:
1. Auto by compiler. These builds can only run at target SIMD architecture (or higher).
2. Manual functions with selectors and architecture check - these are controlled by options (and SetMaxCPU()) and can throw error if options are not compatible with host CPU features present.

Typically performance benefit of auto by compiler optimizations are not big so as safe option you can install SSE2 build to all hosts. And after all scripts testing you can ty to install AVX/AVX2/AVX512 builds to the hosts supported and check the performance difference.

guest

13th April 2024, 14:59

Typically performance benefit of auto by compiler optimizations are not big so as safe option you can install SSE2 build to all hosts. And after all scripts testing you can ty to install AVX/AVX2/AVX512 builds to the hosts supported and check the performance difference.

I think the lowest CPU would be AVX, so if I use the AVX compile, on the main server PC, then that should be suitable for any in the "farm" ??

However, if I don't use the AVX CPU's very often, I could use the AVX2 compile...should work either way :)

FYI, I don't need to install it on the clients, as they are "commanded" by the settings on the main encoding server.

I will do some tests, in the coming days, and see what happens.

Thanks for the feedback.

DTL

13th April 2024, 15:30

"I think the lowest CPU would be AVX, so if I use the AVX compile, on the main server PC, then that should be suitable for any in the "farm" ??"

Yes.

"I don't need to install it on the clients, as they are "commanded" by the settings on the main encoding server."

It can copy all required .dlls on the clients ? If it can not be disabled - you can only use 'lowest' build on the server and all clients.

guest

13th April 2024, 15:40

It can copy all required .dlls on the clients ? If it can not be disabled - you can only use 'lowest' build on the server and all clients.

Not exactly how it does what it does, but it works.

I will test a few different compiles, and let you know how it behaves.

DTL

15th April 2024, 16:12

New release: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.29

New params to MRecalculate:

SuperCurrent clip (same as for MAnalyse).

AMthVSMang (float), default = 10.0f (disabled). Threshold of Vectors Stability Metric to skip current level MV and use previous level hierarchy predictor.
Usable range 0.0f .. 1.0f . 0.0f - skip all MVs, 1.0f - do not skip any. Auto-normalized to AMpoints for any combination of AM-params so expected to be non-
dependent on AM-params (AreaMode and others).

AMflags (integer), default = 1. Any combination of AM positions direction flags per each AMstep: 1 = diagonal positions, 2 = sides positions. Current valid
values 1 (4 diagonal positions only), 2 (4 sides positions only), 3 (4 diagonals and 4 sides positions - 8 positions total, about 2x slower).

AMavg (integer), default = 0. The type of averaging operation of MVs after area gathering searches in the checked area around the current block.
0 - Mode (median ?) of dx and dy separated.
1 - Mean of dx and dy.
2 - Mode (median ?) of MVs angle difference.
3 - Mode (median ?) of MVS difference vector length.
Types 0,2,3 uses some performance optimization - skipping of about half search positions if currently gathered MVs equal to input to AreaMode (original block MVs from initial MAnalyse search). Type 1 always searches all defined by AM options positions so may be up to 2 times slower.

New params to MAnalyse:

AMavg (same as for MRecalculate).

AMpt (integer), default = 0. Same as optPredictorType. PredictorType used in AM searches.
AMst (integer), default = 3. Same as search. Search type used in AM searches.
AMsp (integer), default = 2. Same as searchparam. Search param (radius) used in AM searches.

Added some performance optimization for SAD-only DMFlags used (most typical use case).
Fixed optPredictorType=1 for MAnalyse: Added check of median predictor of current level (now zero, global, hierarchy and median predictors checked before refine).

Next ideas for performance/quality balance is to enable use-configurable ratios of already found equal MVs for fast AM-skip processing. Currently if we measure this ratio in the 0.0 to 1.0f range it is already 0.5 for AMavg 0,2,3. But may be extended to much lower than 0.5. It is motion to AM-adaptive search mode. So for stable enough blocks the number of additional AM-searches may be significantly decreased without quality loss. And for non-stable areas the AM searches may be expanded to all defined by settings number of searches (may be several times slower). This level of adaptive search is only possible with per-block AreaMode with onCPU MAnalyse.

In the next versions expected new param like AMfsRatio of the range 0.0f to 1.0f with default of 0.5f (same as current AMavg 0,2,3, optimizations). If set to below 0.5f (or below 1.0f for AMavg=1) it will cause more early (in the AM search positions checking) call to MVs equality comparison function and if all found MVs equal to input (center of block from standard MAnalyse search) it will stop new AM searches and return (output MV assigned as input).

The new added Averaging types for AM may give different results of MVs and resulted in denoising at different types of content (sort of noise profiles) and also personal user preference. In some tests with scanned film footage AMavg=2 saves more details. AMavg=0 makes a bit lower MPEG output bitrate. AMavg=1 provides significantly better denoising (and MPEG output bitrate) but may also cause more details loss.

I made some tests of denoising quality vs Topaz Artemis model: The Topaz AI generally makes much stable objects views in a sequence of frames (even with fast motion and transformed objects) but can lost more details in some frames of low contrast and low brightness areas. mvtools as linear temporal denoiser (also with low enough thSADA_a values around 1..2) left more noise at the fast motion areas and complex transforms but it means these areas left with less processing and close to original input.

DTL

16th April 2024, 14:06

Good news on mining cards - prices are really nice like $30..$50 per board like p104-100. And a patched driver promises it can have the NVENC feature working https://github.com/dartraiden/NVIDIA-patcher .

p104-100 board in the best case can be equal to the GTX1070 with 2 NVENC cores working at 1200+fps combined performance for h.264 FHD encoding. So ME performance is expected at the close level. But mining cheap boards really do not have any warranty on NVENC present in working state on chip. So each board requires testing. It is limited to 4x PCIe lines typically but for AreaMode with onboard frames shifting it should not be significantly limiting. The RAM bus of 256bit promises good performance for texture copying for sample-shifted resources onboard.

I am trying to ask p104-100 board sellers to make tests if DX12-ME fully works with patched drivers.

Also as I found from old (2016 ?) years NVIDIA developer papers - the ME-only mode looks like it was working long ago via the NVIDIA SDK (vendor-only API). And it also may accept input MVs as hints for refining. So it is possible to make more quality hierarchy search more close to onCPU MAnalyse by providing the same downsized frames for high levels of search and return interpolated MVs back to ME search engine as hints for lower levels of search.

Some users at github report the MVs input hints feature for ME via DX12 looks like it does not work. It looks like it was not implemented via DX12 drivers from the vendor while announced in the DX12 Microsoft API.

It is from https://forums.developer.nvidia.com/t/question-regarding-10-series-nvenc-encoding/64456

You must accept that NVidia is incapable to describe correctly own products. I lost hundreds of $$$ for NVidia official incorrect or intentionally unpublished information.
https://developer.nvidia.com/video-encode-decode-gpu-support-matrix 38 is wrong again. Line "GeForce GTX 1060 - 1070 Ti" should be "GeForce GTX some 1060 / 1070 / 1070 Ti / 1080" eg. GP104 or should be split to two lines if "GTX some 1060 / 1070 / 1070 Ti" have GP104 but only one hwenc (see next point).
But be warned. Some crippled (with HW faults from chip factory) chips have less CUDA (some faulted SMs are disabled) and sometimes one of two hwenc is faulted too (compare Quadro P4000/P5000 - see https://devtalk.nvidia.com/default/topic/1036615/ 6). So, there may be the same problem with GTX 1070/1070Ti (GP104-200-A1/GP104-300-A1 vs. GP104-400-A1). If https://devtalk.nvidia.com/default/topic/987460/nvdec-cuda-nvenc-speed-comparison/ 16 is correct (eg. tested correctly on real card) GTX 1070 has only one hwenc enabled.
Check also https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units 25 for chip marking and how it is crippled. For example check GTX 1060 - it uses 4 different chips. Normal chip (GP106-400-A1), crippled chips (GP106-300-A1 / GP106-350-K3-A1) and super crippled chip (GP104-140-A1 (only 9 from 20 SMs and 192 of 256 bit memory bus width are working)). ... Life with NVidia is like a box of chocolates, you never know what you're gonna get ! :-)

takla

17th April 2024, 11:09

@DTL
https://github.com/ROCm/ROCm/releases/tag/rocm-6.1.0

rocDecode, a new ROCm component that provides high-performance video decode support for
AMD GPUs. With rocDecode, you can decode compressed video streams while keeping the resulting
YUV frames in video memory. With decoded frames in video memory, you can run video
post-processing using ROCm HIP, avoiding unnecessary data copies via the PCIe bus.

Would be nice to have MvTools in HIP

tormento

17th April 2024, 12:21

@DTLWould be nice to have MvTools in HIP
If any rewrite occur for GPU, I will prefer CUDA, OpenCL or Vulkan.

CUDA is the fastest, while OpenCL and Vulkan are compatible with almost anything modern.

AMD is prone to present interesting projects, abandon them and never offer proper support.

DTL

17th April 2024, 12:40

More real addition to current MAnalyse is optional MVs search via NVIDIA SDK (not using DirectX at all ?). I see the ME-only mode of NVENC noted in the old NVIDIA developer documents around mid-201x. And NVENC SDK/API (encode mode ?) can run from DirectX 9. So it expected to work from Win7 without hacks of DX12 for Win7.

Also NVIDIA SDK in C may be also compatible with UNIX/Linux ? So this mode of MAnalyse will be compatible with both Win and UNIX (as todays no-DX12 builds). But in addition to MVs search only in best case it is required some computing (like SAD shader compute) and also for new AreaMode the onboard resources/texture shift to save CPU time and bus transfers. I expect NVIDIA NVENC SDK resources also somehow can be processed with CUDA or other APIs (without reuploading of resources).

Also it looks NVIDIA do not like to provide all possible features like MVs re-using (as hints) via Microsoft DX API (or too lazy to make all required drivers features for Windows for rarely used API). So making some addition to MAnalyse on NVIDIA SDK/API only have some benefits. But it will be completely NVIDIA-only in hardware. This also may be not great for open source developers.

"AMD is prone to present interesting projects, abandon them and never offer proper support."

Microsoft with Windows looks like still some working global force in the residuals of civilization to force many hardware vendors provide some commonly used hardware features (like general purpose computing and Motion Estimation) via single API. But it looks DirectX only now. And even DirectX (ME) with low number of promised features not completely implemented in some drivers and/or for some products.

takla

17th April 2024, 16:54

If any rewrite occur for GPU, I will prefer CUDA, OpenCL or Vulkan.

OpenCL is superseded by Vulkan.

Also
https://i.imgur.com/n0XFBoN.jpeg

tormento

17th April 2024, 17:25

More real addition to current MAnalyse is optional MVs search via NVIDIA SDK
Are you talking about this (https://developer.nvidia.com/optical-flow-sdk)?

DTL

17th April 2024, 23:42

About this: https://developer.nvidia.com/video-codec-sdk

Currently implemented in mvtools2:
DirectX Video

Platform Windows

Benefits
Low Level Control
Native DirectX and Windows Integration
Easy for DirectX developers
Multi-Vendor

Native API interface - D3D11 (Decode only) and D3D12

So it is Windows only and Win10 with DX12 and higher only. But support any (all 2 current) manufacturers of DX12 boards - NVIDIA and AMD and possibly some or some days of intel DX12).

Possible other API:

NVIDIA Video Codec SDK
Platform Windows and Linux

Benefits
High Level Control
Native Integration in custom pipelines
Useful for users with less knowledge of Vulkan and Direct X
Easy for C, C++ developers
Nvidia Proprietary API
Comprehensive feature set

Native API interface
D3D9, D3D10, D3D11, D3D12 (Encode only) CUDA (Encode and decode)

So it is Linux and Windows compatible (more happy opensource developers from the residuals of current White civilization) and Windows 7 compatible with DX9 (more happy users of Win7). Also possible more features available. But main disadvantage: NVIDIA hardware only.

Made some testbuild for possible peak ME performance test for multi-positions search for AreaMode: https://drive.google.com/file/d/1gJsSWLpSrtUtEfJkjtAlaC0sauYkcirL/view?usp=sharing

It have 3 .dlls :
x1me - standard 1 call to motion estimation engine per input pairs of frames
x5me - 5 sequential calls to motion estimation engine per input pairs of frames
x9me - 9 sequential calls to motion estimation engine per input pairs of frames

AreaMode=1 (AMflags=1) will use 5 searches for single pairs of frames and AreaMode=2 (AMflags=1) will use 9 searches.

Test script:

LoadPlugin("mvtools2_x9me.dll")

ColorBars(1920,1080, pixel_type="YV12")
#ColorBars(3840,2160, pixel_type="YV12")

#super = MSuper(mt=false, pel=4, pelrefine=true, chroma=true, levels=0) # onCPU compare
super = MSuper(mt=false, pel=4, pelrefine=false, chroma=true, levels=1) # onHWA
#forward_vec1 = MAnalyse(super, isb = false, delta = 1, search=3, chroma=true, optSearchOption=1, levels=0) #onCPU cpmpare
forward_vec1 = MAnalyse(super, blksize=8, isb = false, delta = 1, chroma=true, optSearchOption=5, levels=1) # onHWA
MStoreVect(forward_vec1)

Prefetch(...)

With Gigabyte GTX1060 card GPU clock max 1911 MHz and 1920x1080 frame:

x1me - 500 fps, 74% load of Video Encoder. Performance somehow limited to data transfer overheads. Bus Load ~30%, GPU chip power ~35W only. Expected ME RAW frames pairs per second performance 500/0.74 = 675 fpps.

x5me - 150 fps, 100% load Video Encoder. Bus load ~8%. ME RAW frames pairs per second performance 150x5 = 750 fpps.

x9me - 84 fps, 100% load Video Encoder. Bus load ~5%. ME RAW frames pairs per second performance 84x9 = 765 fpps.

So single NVENC at Pascal chip as expected provides ME performance about 750 fpps. Full-blood GTX1070 and 1080 boards with Dual NVENC healthy expected about 1500 fpps ME RAW performance.

The i5-9600K provides about 6 times lower (with same pel=4 quality).

Changing blocksize from 8x8 to 16x16 do not change anything - it looks performance of NVENC is only samples limited and not blocks number (per frame or per second). With 3840x2160 frame it is about 4x linearly lower.

Expected performance of AreaMode=1 AMflags=1 for 1920x1080 frame size and tr=12:
tr=12 need 24 frames pairs per output frame and AreaMode=1 AMflags=1 need 4 additional frames pairs search (x5 total) so MDegrainN output frame require 24*5=120 pairs frame search with these settings. 750 fpps ME RAW performance of single Pascal NVENC divided to 120 ~= 6 fps total denoise performance max.

With GTX1070..1080 boards (p104..p102 miners editions too ?) performance expected twice higher.

Table 3 of the document https://developer.download.nvidia.com/designworks/video-codec-sdk/secure/7.1/01/NVENC_DA-06209-001_v08.pdf also lists about 648 fps for Pascal NVENC at h.264 encoding at fast mode with FHD frame.

Another perspective Average function to AMavg planned is geometric median - https://en.wikipedia.org/wiki/Geometric_median . It is not easy to compute so may add more performance penalty for onCPU MAnalyse. But planned to offloading to Compute Shader for onHWA processing in the future.
Valuable property of geometric (2D) median for noised sources is:
The geometric median has a breakdown point of 0.5.[13] That is, up to half of the sample data may be arbitrarily corrupted, and the median of the samples will still provide a robust estimator for the location of the uncorrupted data.

tormento

18th April 2024, 10:17

About this
Have a look at the link I provided. It seems to have everything ready.
Full-blood GTX1070 and 1080 boards with Dual NVENC healthy expected about 1500 fpps ME RAW performance.
I suppose you are aware of NVEnc unlocking patches.
p104..p102 miners editions too
There are patches for them too. :)

DTL

18th April 2024, 11:18

"It seems to have everything ready."

That promo do not list any denoising application. So it may work good only with new cameras at good light with low noise at input. Same as RIFE also fail with noised sources. Also I do not have Turing board to check it.

"I suppose you are aware of NVEnc unlocking patches."

I do not got any out_of_licence or other error. It simply limits by expected max MPEG encoder rate at lowest settings (fastest). So I think it is natural limits of NVENC hardware. Also licence limits MPEG encoding sessions to 2 for end-user drivers. Here we have usage of 1 D3D12 device by 1 process. And even not in MPEG encoding mode. May be it is too rarely used by anyone at this civilization so NVIDIA do not think about putting any limits to this API. It even do not use any MPEG-LA licensed algorithms I think.

guest

18th April 2024, 11:54

Originally Posted by DTL

It can copy all required .dlls on the clients ? If it can not be disabled - you can only use 'lowest' build on the server and all clients.

Not exactly how it does what it does, but it works.

I will test a few different compiles, and let you know how it behaves.

Hi DTL, well, I've finally got around to doing some test's, and most of your current builds work for me, (on the AVX2 CPU's) but as I tried to explain the Distributed Encoding I use, the clients that only have AVX CPU's, do not start encoding, despite what compile I tried, when using an SMDegrain script that calls mvtool2.dll.

Not sure what to do, now :(

DTL

18th April 2024, 12:02

More safely use _e.XX builds for SMDegrain. Latest is https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-e.03 . If it also not work on AVX - the only way is fallback to pinterf' 2.7.45 builds.

guest

19th April 2024, 05:35

More safely use _e.XX builds for SMDegrain. Latest is https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-e.03 . If it also not work on AVX - the only way is fallback to pinterf' 2.7.45 builds.

Hi DTL,

Another day, another lot of tests to figure out what's going on here...

So I thought I'd do it all on the AVX E5-2697 v2 dual CPU system, cut to the source, (so to speak), and I've uncovered several problems :(

I was using AVX2 builds of HDRTools & plugins_jpsdr that were causing unexpected problems, so I have revised that situation, and then I tried your latest compile of mvtools,
and I used the AVX DX12 (I have DX installed on this particular PC), and had no issues, tried noDX12, that worked. (any reason why you've done DX variants ?)

Have not got an AVX PC without DX installed, so I can't check that.

So after all that fiddle, I think I've got it sorted, next test is that it behaves itself using the Distributed Encode function from another AVX2 PC (the Ryzen's), in which I will use the DX12 AVX compile.

DTL

19th April 2024, 13:27

guest

19th April 2024, 13:35

"any reason why you've done DX variants ?"

They are for hosts where DX12 installed and users want to use DX12 features. If DX12 is not installed - they will not loads at all. So for Win7 and old only noDX12 builds are applicable.

All my PC's are running Windows 11.

So noDX12 ?

I do have DX installed, and both variants worked.

DTL

19th April 2024, 13:44

guest

19th April 2024, 14:37

Win10 and later looks like has DX12 embedded from setup - you can use both DX12 and noDX12 builds.

If you have different CPUs in the farm - can you test onCPU MAnalyse performance using this script:

LoadPlugin("mvtools2.dll")

ColorBars(1920,1080, pixel_type="YV12")

super = MSuper(mt=false, pel=4, chroma=true)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, search=3, chroma=true, mt=false)
MStoreVect(forward_vec1)

Prefetch(...)

with AVSmeter ? I hope this can run with any build of mvtools. Its output is RAW ME performance in frames pairs per second (using Expanding search - close to Exhaustive). Expanding (from zero dx, dy) center search with pnew > 0 (better to adjust using MShow or other methods for current noise in the source) expected to be more noise-resistant at the static areas. Because after checking center (not moving point) - the other definitely false positions will be somehow penalty-protected by pnew setting. Though too high pnew param will cause skipping of real motion and blur in the motion areas.

Where Prefetch() is for number of real CPU cores at host.

I have to say that I have no idea how to "run" that script with AVSMeter :(

I am only familiar with how scripts are written for RipBot, so changing that script for RipBot is way over my head, sorry.

Here is a sample of a script, from RipBot:-

#After_Prefetch_Custom
LoadPlugin("%AVISYNTHPLUGINS%\Plugins_JPSDR\Plugins_JPSDR.dll")
LevelLimit=(video.BitsPerComponent==8) ? 255 : 1023
IntensityMask=ConvertToY(video).Levels(0,2,LevelLimit,0,LevelLimit,coring=false)
EdgeMask=aSobel(IntensityMask,chroma=0,thresh=255,SetAffinity=false).invert.Levels(0,2,LevelLimit,0,LevelLimit,coring=false).Blur(1)
SharpMask=Overlay(IntensityMask,EdgeMask,mode="Multiply",opacity=1.0)
SharpenedVideo=Sharpen(video,1)
video=Overlay(video,SharpenedVideo,mask=SharpMask,opacity=1.0)

DTL

19th April 2024, 15:18

"I have to say that I have no idea how to "run" that script with AVSMeter"

1. Enter required number of threads for Prefetch(arg) and save to .avs file like test.avs (you can use Notepad text editor from Windows).

For 2 cores CPU (Core2Duo E7500)

LoadPlugin("mvtools2.dll")

ColorBars(1920,1080, pixel_type="YV12")

super = MSuper(mt=false, pel=4, chroma=true)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, search=3, chroma=true, mt=false)
MStoreVect(forward_vec1)

Prefetch(2)

2. Put all required files to some folder like c:\test :
c:\test\test.avs
c:\test\mvtools2.dll
c:\test\AVSmeter64.exe

3. Run from command prompt

c:\test\AVSmeter64.exe test.avs

It will shows performance metering like

AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r3982, 3.7, x86_64) (3.7.3.0)

Number of frames: 107892
Length (hh:mm:ss.ms): 00:59:59.996
Frame width: 720
Frame height: 180
Framerate: 29.970 (30000/1001)
Colorspace: RGB32

Frame (current | last): 2416 | 107891
FPS (cur | min | max | avg): 6.130 | 3.900 | 317501 | 19.94
Process memory usage: 346 MiB
Thread count: 8
CPU usage (current | average): 100.0% | 93.9%

Better to wait some time for FPS avg value to become stable.

Where required line only
FPS (cur | min | max | avg): 6.130 | 3.900 | 317501 | 19.94

If Prefetch( ) arg set correctly (not very low for current CPU cores present) - AVSmeter must display CPU usage close to 100%.

By placing different mvtools2.dll builds in the folder with script you can also check if any performance difference exist between SSE2/AVX/AVX2 builds at your host. If you need best performance with most common scripts (QTGMC/SMDegrain) without modification - I think -e.XX builds will be faster. For -a.XX builds it is recommended at least add optSearchOption=1 to all MAnalyse calls to enable some SIMD optimizations.

guest

19th April 2024, 15:36

guest

21st April 2024, 06:45

2. Put all required files to some folder like c:\test :
c:\test\test.avs
c:\test\mvtools2.dll
c:\test\AVSmeter64.exe

3. Run from command prompt

c:\test\AVSmeter64.exe test.avs

It will shows performance metering like

AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r3982, 3.7, x86_64) (3.7.3.0)

Number of frames: 107892
Length (hh:mm:ss.ms): 00:59:59.996
Frame width: 720
Frame height: 180
Framerate: 29.970 (30000/1001)
Colorspace: RGB32

Frame (current | last): 2416 | 107891
FPS (cur | min | max | avg): 6.130 | 3.900 | 317501 | 19.94
Process memory usage: 346 MiB
Thread count: 8
CPU usage (current | average): 100.0% | 93.9%

Better to wait some time for FPS avg value to become stable.

Where required line only
FPS (cur | min | max | avg): 6.130 | 3.900 | 317501 | 19.94

So I had the opportunity to do some test today, and they are interesting, almost like a benchmark test for Prefetch.

One thing that was strange, I did many test's with different mvtools.dll, and Prefetch's, and then just to double check,
I did a couple that had good results on the 1st run, but on the 2nd run were quite different, faster.....:confused::confused::confused:

Anyway, these were all done on an AVX CPU system (E5-2697v2)

Is there any reason to test on an AVX2 PC ??

Anyway, here are the results:-

https://www.mediafire.com/file/1igo02cyhuq2j27/results.txt/file

I did some quick tests on the Ryzen 7950X

https://www.mediafire.com/file/m6q07md79n8p4ki/7950X.txt/file

guest

21st April 2024, 07:30

LoadPlugin("mvtools2.dll")

ColorBars(1920,1080, pixel_type="YV12")

super = MSuper(mt=false, pel=4, chroma=true)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, search=3, chroma=true, mt=false)
MStoreVect(forward_vec1)

Prefetch(2)

This is probably a bit off topic, but would there some way to use a similar "test" to benchmark x265 @ 4K, using different Prefetch settings ??

DTL

21st April 2024, 12:23

x265 is completely separate process from AVS+ system process so it can not be controlled by AVS+ threading settings. x265 uses its own methods of multithreading and provides separate controls (limited ?).

guest

21st April 2024, 14:08

DTL

21st April 2024, 15:03

I am missed that previous post. Will look now or later when have time.

Quick answer:

"Is there any reason to test on an AVX2 PC ??"

Yes - the main steps in SIMD designs are SSE2 (or latest 4.2) and next is AVX2 for full-blood integers processing. AVX is only new beginning of 256bit SIMD and mostly limited to rarely used float calculations. Next step to 512bit is AVX512.

So I typically make builds of SSE2/AVX2/(AVX512).

DTL

23rd April 2024, 15:46

Finally some time to type the long post:

Thank you for the provided test results. One valuable result is that even top end-users desktop CPUs are still significantly slower in ME in comparison with 10years old MPEG encoder chips. Also it takes close to 100% CPU time and nothing left on MDegrainN and MPEG encoding (x264/x265).

First the recommended number of Prefetch threads is the number of physical cores, not total logic if HyperThreading is enabled. Though it may greatly depends on each CPU architecture especially in the case of massive-multicores chips with low RAM channels and big enough cache.

I also made some tests with that script and CPUs and NVIDIA chips.

It looks the performance is heavily limited by RAM performance so at some CPUs the total cores and 50% cores performance is very close. It means for a real transcoding process it may be better to finetune threads number on the AVS script used so it can leave more CPU time for MPEG encoder. In complex scripts with mvtools used it may be tested with a different Prefetch() setting for each filter (or groups of filters ?)

Like

_myDegrain_with_mvtools_script(params).Prefetch(K)

other_plugins_calls(params).Prefetch(N)

where K < N.

And adjust N and K numbers for best total transcoding performance at a given host.

i5-9600K (6 cores without HT) example:
Prefetch(6) - 120 fps (97% total CPU load)
Prefetch(8) - 118 fps
Prefetch(4) - 120 fps (66% total CPU load)
Prefetch(3) - 112 fps (50% total CPU load)

The on chip cache size looks like winning the game with AMD 7950X (64 MB cache) even in comparison with Xeon Gold 6134 (8c/16th 32 MB cache) and 6 (?) RAM channels.
Xeon Gold 6134 results vs Prefetch number
Prefetch(8) - 220 fps (49% CPU load)
Prefetch(12) - 248 fps (74% CPU load)
Prefetch(16) - 258 fps (91% CPU load)

While with a fixed AVSTP plugin you can test internal MT in mvtools and it may make more performance benefit because of lower RAM usage and more on-chip cache efficiency. Unfortunately -a.XX builds long ago were not tested with AVSTP and look like they become very unstable in that mode.

Testing of different builds branches (-a.XX and -e.XX) really not completely fair as I remember a.XX have always enabled some simple logical optimization of skipping of already checked predictors and it can make visible performance benefit with static noise-free sources like ColorBars(). An attempt to add noise with AddGrain() will take some CPU time and lower results. 2.7.45 build and -e.03 build do not skip any predictor check and make some redundant work and it causes more or less visible performance penalty (may significantly depend on source).

As I see the AMD 7950X also have some graphic core integrated - you can also test its performance in hardware ME mode (using DX12 builds of a.XX and put Compute.cso with same folder) if it provide this API:

LoadPlugin("mvtools2.dll")

ColorBars(1920,1080, pixel_type="YV12")

super = MSuper(mt=false, pel=4, chroma=true, levels=1, pelrefine=false)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, chroma=true, mt=false, optSearchOption=5, levels=1)
MStoreVect(forward_vec1)

Prefetch(..)

Also found during DX12 tests: Setting MT mode to 3 greatly limits performance of DX12, so default MT mode does not limit performance.

Tested Maxwell 2nd gen - it does not provide DX12ME API. So it looks like everything usable starts only from the Pascal series.

Turing cards with single NVENC are only slightly faster over Pascal 1 NVENC (about 700 vs 600 fpps).

I tried to find any information on possibly applicable AMD cards - found only some Wiki about different generations of MPEG hardware encoding ASIC from 199x to 202x on ATI/AMD cards - it looks like only modern VCE can be usable ?

Next release will have MAverage onCPU script function for performing script-based AreaMode simulation. First tests show with pel=4 precision it runs about 4 times faster (i5-9600K and GTX1060) in comparison with onCPU at AreaMode=1 AMflags=1 search more (+4 additional search positions for block). And still uses only 45% of single NVENC. So with full DX12 onboard frame shifts it is expected to be about 2 times faster. But DX12 onboard frame shifts requires much more DX12 programming. The quality is still not as perfect as expected - need more debugging. Currently SAD (DM) is not updated in simple MAverage function (DM update requires much more programming with import of super clip too). But if using IntOvlp=1 or 3 the SAD must be updated in MDegrainN after MVs interpolation so precheck of SAD in MAverage may be redundant and cause performance penalty. Need to check what is wrong now with current sources (some bad blocks passed to output like SAD not updated for resulting MVs or maybe other bugs still exist).

guest

24th April 2024, 03:19

Finally some time to type the long post:

Hi DTL,

You are quite good at long posts :)

So most of that doesn't mean too much to me, but if you got some of that from my tests, then it was worth it.

I might do the same tests on my 13900KF (no iGPU), and see what that shows with the P & E cores.

The 7950X does have an iGPU, but it's so "weak" I don't enable it, or install the appropriate drivers.

Regards

DTL

24th April 2024, 12:35

"The 7950X does have an iGPU, but it's so "weak" I don't enable it, or install the appropriate drivers."

In the reviews it is named as RDNA2 and can encode 264/265 and DX12_1 (or even DX12_2) compatible - so you can try enable it and check performance if it provides DX12-ME API too.

guest

24th April 2024, 12:55

takla

9th October 2024, 12:25

@DTL
I've been away for a while.
Have you managed to implement adaptive-thSAD ?
I mean thSAD automatically calculating and applying the optimal value on a per-frame basis?

And have you managed to add AVX 512 support?

DTL

9th October 2024, 23:35

It was somehow simply implemented in a.XX and also e.XX builds - https://forum.doom9.org/showthread.php?p=1990642#post1990642 . But adjustment of scale and offset params may significantly depends on both source 'noise type/level' and also user's expectation on thSAD used. So thSADA_a may be tweaked from about 1.0 for medium (or close to minimal already started) noise reduction to 2..3+ for much higher (also may cause details blur as usual with too high thSAD).

About future development - as I read developers of MPEG encoders finally understand benefit of more or less noise reduction and start to integrate spatio-temporal noise reduction in MPEG encoder engines too (in new x265 builds ?). So they can reuse MVs for both noise reduction and MPEG encoding and it is faster. It was planned in old years to feed best and very expensive found MVs from degrainer ME into MPEG encoder engine to save some time and increase quality on MPEG compression. But it looks civilization degrade at too large speed and about zero developers for denoisers left. Last developers of this civilization are around MPEG encoders like x264 and x265 (and possibly some later somehow a bit advanced) and they finally start to do good progress in understanding of natural moving pictures compression. If they like they can reuse all currently implemented (and planned and described) ME features of mvtools project. Though higher quality features like Area Mode search are also very compute extensive and slow.

I do not have AVX512 development and test setup also and our company do not plan to make hardware upgrade in any close years. May be to the end of 202x or somewhere in 203x we will finally got some thrown-away old or very cheap new like $30 chips with AVX512 to continue development. AVX512 can somehow help to get more data at AreaMode several shifted positions search at the same time. But its programming require lots of brain resources and if about no one uses these slow modes so very few reasons to do it. In next years we can test and look for the progress of internal spatial-temporal noise reduction engines in the MPEG encoders and compare with mvtools-based pre-filtering. May be better will be some workload separation between some prefiltering with mvtools and next stage internal temporal noise reduction in MPEG encoder. Also in next years it is possible to see hardware temporal noise reduction in new MPEG encode ASICs and it may be much faster in comparison with mvtools onCPU.

takla

10th October 2024, 00:19

It was somehow simply implemented in a.XX and also e.XX builds - https://forum.doom9.org/showthread.php?p=1990642#post1990642 . But adjustment of scale and offset params may significantly depends on both source 'noise type/level' and also user's expectation on thSAD used. So thSADA_a may be tweaked from about 1.0 for medium (or close to minimal already started) noise reduction to 2..3+ for much higher (also may cause details blur as usual with too high thSAD).

Too bad. I was hoping a simple internal function in your mvtools2 fork, like lets say adaptive-thSAD=true/false
and then maybe two limiters for min-thSAD & max-thSAD.

About future development - as I read developers of MPEG encoders finally understand benefit of more or less noise reduction and start to integrate spatio-temporal noise reduction in MPEG encoder engines too (in new x265 builds ?). So they can reuse MVs for both noise reduction and MPEG encoding and it is faster. It was planned in old years to feed best and very expensive found MVs from degrainer ME into MPEG encoder engine to save some time and increase quality on MPEG compression. But it looks civilization degrade at too large speed and about zero developers for denoisers left. Last developers of this civilization are around MPEG encoders like x264 and x265 (and possibly some later somehow a bit advanced) and they finally start to do good progress in understanding of natural moving pictures compression. If they like they can reuse all currently implemented (and planned and described) ME features of mvtools project. Though higher quality features like Area Mode search are also very compute extensive and slow.

It really is like that. Especially the "understanding of natural moving pictures compression"
Sometimes it feels like a global consciousness, where X amount of people have to spend Y amount of time to "unlock" that knowledge for more people. Sure you can say "thats just what research papers are for" but I feel it is higher than that. Its not even about the information itself but the understanding of information processing.

I do not have AVX512 development and test setup also and our company do not plan to make hardware upgrade in any close years. May be to the end of 202x or somewhere in 203x we will finally got some thrown-away old or very cheap new like $30 chips with AVX512 to continue development. AVX512 can somehow help to get more data at AreaMode several shifted positions search at the same time. But its programming require lots of brain resources and if about no one uses these slow modes so very few reasons to do it. In next years we can test and look for the progress of internal spatial-temporal noise reduction engines in the MPEG encoders and compare with mvtools-based pre-filtering. May be better will be some workload separation between some prefiltering with mvtools and next stage internal temporal noise reduction in MPEG encoder. Also in next years it is possible to see hardware temporal noise reduction in new MPEG encode ASICs and it may be much faster in comparison with mvtools onCPU.

Yeah. Something like saving motion-vectors to a file should have been a thing from the very start.
Just thinking about it, you could have someone do the slowest possible mv settings, and then share that file for people who own the exact same video stream, e.g. Blu-Ray. It would save a lot of energy and time.

Also, thank you for bringing it to my attention that x265 developement has continued (https://x265.readthedocs.io/en/master/releasenotes.html#version-3-6)

DTL

10th October 2024, 12:05

" like lets say adaptive-thSAD=true/false"

You can simply set params to some fixed values like thSADA_a=1.2 and thSADA_b=50. But to have control for thSAD decreasing with more tr from current frame - user still need to control thSAD and thSAD2 values (or left defaults). Also as used thSAD anyway significantly change the 'denoising power' it is required some control over its 'power'. It is no good to set 'full-auto thSAD' - it will cause too low denoising for one user or too high to other and there will be no way to control over it. Now we have most significant control param of thSADA_a of values like:
1.0 - low
2.0 - medium
3.0 - high

With float (close to unlimited) precision for fine-tuning. And also second additive param thSADA_b to shift all values a bit higher or lower unconditionally (you can in theory set thSADA_a=0 and thSADA_b=const to completely disable per-frame thSAD variability and it is equal to const thSAD). So thSADA_a mostly control per-frame variability of thSAD used and also possible to create different versions of auto and fixed.

"maybe two limiters for min-thSAD & max-thSAD."

Not yet implemented. It is simple and good idea but require more params like thSADA_min and thSADA_max. Easy to do. Add limiting to min and max to calculated value at https://github.com/DTL2020/mvtools/blob/9eedb9d0850f638fc43212fb515cf048f9b9a58f/Sources/MDegrainN.cpp#L7440

"Just thinking about it, you could have someone do the slowest possible mv settings, and then share that file for people who own the exact same video stream, e.g. Blu-Ray. It would save a lot of energy and time."

Yes - process of searching for best possible MVs is very compute loaded so can be done as distributed computing (the workunit can be some frames range + tr frames around edges of range). Also it is only translate transtform MVs - no other possible transforms like rotation-lighting-scaling and many others currently in searching. But for best possible MVs it is good to make very many passes analysis over all movie (all typical 130000 frames of 90 min 24fps movie). Because it is iterative process of search and refining (by denoising) of objects patches. So if current scene only have too bad patch view severily distorted by noise (like low-light scene with high ISO/gain) but other scene can have this patch texture much better quality so best search engine need to look over all possible frames. It is very complex and compute-loaded task so very slow. Also currently there is no real 'exhaustive' search - only highly optimized multi-step multi-resolution and it can miss some best MVs too. Full ESA search is sort of radius=frame_size/2 and not simply several samples around current. And it is several decimal orders of magnitude slower too. Also it can be multiplied to complexity of AreaMode like search close to each sample + its surroundings individual search instead of much smaller blocks grid search.

The resulted MVs files are small enough and can be released for each known 'classical/fixed' title and can be refined by many users in parts or complete files as different versions from each user. Though there are many possible 'overlayed' blocks scans (subdivision of frame into blocks) in the mvtools - each overlay combination (block size + V and H overlap) create different MVs file.

" x265 developement has continued"

New feature
Motion-Compensated Spatio-Temporal Filtering.

Yes - it is finally beginning of the complex in design and very compute-complex part of MPEG encoder to skip more random data typically from medium distortions (film grain and photon shot noise and electronic other noises) and to keep and save more real object's data. Its advancement may lasts to the last programmers of current civilization. Though first implementations in MPEG encoder may be as simple as initial mvtools design from 2004.

I hope new internal temporal denoisers will be added someday to x264 encoder too.