New DirectX 12 Video APIs [Archive] - Page 4

anton_foy

3rd August 2022, 22:31

New version: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.15

Added diagonal interpolated overlap mode to MDegrainN of 2x blocks number to process. IntOvlp=3 with SAD re-check and IntOvlp=4 with interpolated SAD.
Added more error messages if non-compatible options provided for MSuper/MAnalyse/MDegrainN.
Updated documentation with new options. Updated file is https://github.com/DTL2020/mvtools/blob/mvtools-pfmod/Documentation/mvtools2.html
Added meander scan in the combined luma+chroma overlapped processing - may be better reuse of cached ref planes data.

Now the IntOvlp=3 is the typical everyday usage mode because it is much better in speed and very close to the quality as IntOvlp=1. Only about 30% slower in compare with no-overlap processing at i5-9600 with SO=5. It is close or equal to old BlockOverlap plugin operation.

Now for the future possible to make many combined modes of speed/quality:
1. Diagonal overlap search onCPU in MAnalyse.
2. Diagonal overlap is compatible with DX12ME and can double load of accelerator with 'real' search - may be used when host CPU speed is low and accelerator is underloaded.
Some internal flags may be added to MVs clip to indicate if it contain diagonal overlapped MVs data.
MDegrainN may also accept 2 MVs clips from 2 MAnalyse for original and diagonally shifted blocks seach data using any combinations of onHWAcc (single or dual) or onCPU search. To balance loading between host CPU and a single or multiply accelerators. As I see after-mining secondhand headless cards with may be equal to GTX1060 chip are now avaialble at Aliexpress at about $35..50.
Though the quality between real searched 2 MVs planes for diagonal overlap mode and second interpolated MVs set need to be examined - may be too low difference. May be only worth is host CPU too slow for MDegrainN and many free accerelators resources available.

Thanks DTL very interesting! I may be a little bit under the sun but I think you mentioned sometime before that prefiltering/auxilary clip will not be required with your build. Is this true? In that case how does it work?

DTL

3rd August 2022, 23:08

" prefiltering/auxilary clip will not be required with your build. "

I still not found visible benefit of prefiltering if using 'interfiltering' of MVs inside MDegrainN with MVLPF processing. So it may be new (partial or complete) replacement of the old prefiltering method. I not made much testing. Though the MVLPF is still in the 2 simple implementations available and may be subject of complex (linear and or non linear) development in the future as both internal processing inside mvtools binary or as intermediate scripting (using MStoreVect/MRestoreVect and sample-accessing methods from script of new AVS+).

So if you use old scripts and not enable MVLPF processing - it is mostly probably prefiltering will make a benefit. If you enable MVLPF it is better to make new tests if prefiltering required or not or how much/etc.

"In that case how does it work?"

To enable internal MVLPF you need to adjust either MVLPFCutoff or MVLPFGauss from default values. Defaults are disabled state for compatibility with old scripts. Only one of 2 may work at the same time. It may be not best options naming. May be better to change MVLPF params to 'MVLPF_Type=None/Sinc(?)/Gauss/..." and 'MVLPF_Param1, MVLPF_Param2,...". So MVLPF_Type=None will clearly mean the processing is disabled (or one of possible filters is selected). So the options may be changed in the future in theory.
The best values of there settings may more or less depend on the footage and current user's preference. I currently use MVLPFGauss=0.8 for my encodings of HDTV 1080i documentaries.

Addition:
Old BlockOverlap plugin have additional 'kernel' control param:
http://avisynth.org.ru/blockoverlap/blockoverlap.html
kernel - blending window form (float, from 0.0 (uniform) to 1.0 (cosine kernel), default =0.5).
Blending mode with kernel=0 is the same as Avisynth command Overlay with opacity=0.5. In this mode the filter can not remove all block artifactes, but it halve them, try use some additional deblock filter.
Mode kernel=1.0 effectively smoothes blocks, but can produce some dot (circle) artefactes instead.

In current IntOvlp modes 3 and 4 full cosine-shaped window is used (equal to kernel=1.0 in BlockOverlap plugin). If it not best for some cases - the additional control param may be added. As I see in 2.7.45- versions of overlap windows - only full cosine window used and not some averaging with rectangular window. Though in old BlockOverlap plugin default is a mix of 0.5 cosine and 0.5 rectangular. May be it can be also subject to test by users and/or script developers.

DTL

9th August 2022, 21:11

Trying to use new builds with QTGMC for some processing of interlaced HD in intermediate progressive form I found next issues:
1. DX12_ME modes return some minor error about motion clip is too small. So currently only onCPU with optSearchOption=1 is max possible new processing modes with QTGMC.
2. Using default for QTGMC block size of 16 crashes MAnalyse onCPU with out of frame buffer memory access - may be some more padding may help in MSuper or some more vectors limiting check/add in MAnalyse. Only block size of 8 is working. May be limiting were damaged in some redesign from 2.7.45 version - need more debug.

Also as I see the main processing in QTGMC is based in MCompensate function. The some denoising is based on MDegrainX so it is easy enough to rewrite to MDegrainN (though the too low tr is typically not effective with MVLPF).
After looking into MCompensate of 2.7.45 mvtools I see it is very outdated and some redesign is planned:

1. Currently MCompensate is based on hard thresholding by thSAD (same as new MDegrainN with wpow=7). So it may be good to make some smoother rolloff of weighting of the ref block using same weighting functions as in MDegrain. With same wpow new param as with MDegrainN
2. The motion-adaptive adjustment of weighting depending on MVs length and coherency may be added but need testing if it is good for QTGMC activity.
3. MVLPF for 2 frames MCompensate is about not effective at all (or require some redesign of MCompensate for requesting big enough set of frames from MAnalyse in multi-mode and it will load ME-part of mvtools significantly - though may be only caching of MVs-frames for reusage in both deinterlacing and MDegrainN denoising may help to decrease load).
4. Adding usage of sub-sample shifting is the simplest task and will be quickly.
5. Main addition - to support new interpolated overlap modes with non-overlapped search in MAnalyse the same interpolated overlap processing need to be added to MCompensate. To not make many copy parts of program from MDegrainN to MCompensate it may be good to make some 1 for all usage classes/functions may be. So the architecture of old and simple MCompensate function need to be redesigned to the same as current MDegrainN (with array-based of MVs data storage - not Fake object with accessing of VECTOR data as Block class data).

MysteryX

11th August 2022, 02:40

MVTools2 with hardware acceleration? Great!

Does this implementation have the potential to work on Linux or it's Windows-only? DirectX 12 doesn't look very Linux-friendly... which uses Vulkan. Is Vulkan API suitable for this work? It's generally faster for most tasks.

takla

11th August 2022, 03:44

@MysterX
Read post #129 (https://forum.doom9.org/showthread.php?p=1969845#post1969845) & #130

DTL

11th August 2022, 14:37

AMD provide ME from DX11 and windows 7 via custom LiquidVR API. But it may be still windows only and not Linux compatible. Though DX11 may be easier to emulate at Linux if AMD provide all needed drivers.

In better case Linux community need to define ME API for applications and provide DDK for hardware developers (intel/amd/nvidia) to develop required drivers support for this API. May be it will also helps to UNIX developers of x264 MPEG encoder to have some help from hardware ME accelerator too.

May be Linux developers may add ME (and MPEG encode/decode) to Vulkan API and ask drivers developers to support it.

DTL

24th September 2022, 19:38

New release: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.16

Added SuperCurrent param to MAnalyse, clip param Allow to provide differently processed clip as current source for search. May be useful to use with prefiltering usecases.

Added SearchDirMode param to MAnalyse, int param. 0 - search standard direction (current frame to ref frame MVs). 1 - reverse search.

Fixed bug with not-selecting combined luma+chroma processing modes when thSADC=thSAD (and thSADC2=thSAD2).

Added Multi-Pass Blending mode in MDegrainN. New params:
MPBthSub, int (10), threshold for subtracted blocks.
MPBthAdd, int (20), threshold for std blended blocks (additively).
MPBNumIt, int (0), number of iterations. 0 - MPB processing mode not used.
MPB_SPC, float (1.5), multiplier and divider for weight adjustment at each iteration if SAD of curent blending result vs subtracted or ref block is above threshold.

Currently MPB mode only supported for 8bit formats.
Current typical usage params for MDegrainN (onCPU search for example):

tr=10

super=MSuper(last, mt=false, chroma=true, pel=2)
multi_vec=MAnalyse(super, multi=true, blksize=8, delta=tr, search=3, searchparam=2, overlap=0, optSearchOption=1, optPredictorType=0, chroma=false, mt=false)
MDegrainN(last,super, multi_vec, tr, thSAD=150, thSAD2=140, mt=false, wpow=4, thSCD1=400, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16,
MVLPFGauss=0.9, thMVLPFCorr=50, UseSubShift=1, IntOvlp=3, MPBthSub=10, MPBthAdd=20, MPBNumIt=2, MPB_SPC=1.5)

The MPB params are subject of long experiments for best results (zopti optimizer may be highly required). Too much iterations may quickly decrease 'denoising'. Typically enabling MPB require to set thSAD to a bit higher value to keep denoising at static and flat areas good enough. Example - from 110 to 150 with default MPB params and 2 iterations. So each iterations weight adjustment param MPB_SPC is defaulted currently to high enough value of 1.5. Also MPB mode make high CPU computing load (most data is cached but it require many additional SAD computing and some block subtraction from total blending result). So speed penalty from enabling MPB mode is enough even with 2 iterations. May be lower in the future with AVX2/AVX512 SAD and block subtract future functions to develop. Currently uses only SSE2.
Possibly max quality MPB with luma + chroma SAD usage is only supported in combined luma+chroma processing (if thSADC=thSAD, thSADC2=thSAD2 - default mode). Separated planes processing only use luma or current chroma plane SAD so may give worse results.

MPB mode operation:
For each iteration:
1. Calculate subtracted blocks (current blend result with current blend weights minus single block for speed, should be equal or very close to partial blend with single block excluded from blend, partial blends require more operations in compare with subtraction).
2. Calculate SAD of current blend result vs subtracted blocks array (vector) and input blocks array (src + all refs).
3. Calculate average SAD for subtracted and standard blended blocks arrays.
4. If current block SAD different from average above threshold (MPBthSub and MPBthAdd) - adjust its weight to decrease (for subtracted blocks) or increase in MPB_SPC ratio.

The idea is to create more equal weight field for all blocks in blending pool. So if block too badly contribute to blending average its weight is decreased and if blocks looks more equal to blended average its weight increased. No MVs is analysed here (yet).

MPBthSub and MPBthAdd expected to be about thSAD/10 and MPBthAdd > MPBthSub about twice (depend on tr-param and current degraining between input and output) because SAD of currently degrained block vs single subtracted is significantly lower in compare with degrained block vs input noised ref or src.

Too high values of MPBthSub and MPBthAdd disables weight adjusting in MPB (for subtracted or incoming blocks). So it may be used to check result from separated parts of processing. Setting both MPBthSub and MPBthAdd too high - disables MPB processing practically but still make processing speed lower.

anton_foy

26th September 2022, 14:33

DTL many thanks for this! So many new features and parameters to understand yet.
As I use mostly motioncompensated TemporalSoften rather than mdegrain, my question is if this latest version is suitable for hbd (16bit) using the motion compensation bit? What parameters could you suggest to use and or change in the TemporalSoftenMC (TSMC) script?

####### modded to adj. smaller mrecalculate blocksize
function TSMC(clip input, int "tradius", int "mthresh", int "lumathresh", int "blocksize", int "rBlock",clip "auxclip", bool "pref", int "Y", int "UV")
{

pref = Default(pref, true)
Y = Default(Y, 3)
UV = Default(UV, 2)

t=Defined(tradius)
tradius=t ? tradius : 6
# temporal radius-number of frames analyzed before/after current frame.

m=Defined(mthresh)
mthresh=m ? mthresh : 180
# motion threshold-higher numbers denoise areas with higher motion.
#Anything above this number does not get denoised.

l=Defined(lumathresh)
lumathresh=l ? lumathresh : 255
# luma threshold- Denoise pixels that match in surrounding frames.
#255 is the maximum and default. 0-255 are valid numbers.
#Also adjusts chroma threshold.

b=Defined(blocksize)
blocksize=b ? blocksize : 32
#larger numbers = faster processing times

rBlock = Default(rBlock, 4)

chroma = UV == 3
aux=Defined(auxclip)

w = width(input)
h = height(input)
isUHD = (w > 2599 || h > 1499)
nw = round(w/2.0)
nh = round(h/2.0)
inputA = aux ? auxclip : input
inputA = isUHD ? inputA.ConvertBits(8,dither=-1).BilinearResize(nw+nw%2, nh+nh%2) : inputA

super = MSuper(input, pel=1, hpad = 0, vpad = 0, chroma=true, mt=true, levels=1)
superfilt = MSuper(inputA,pel=1, hpad = 0, vpad = 0, chroma=true, mt=true) # bug can't disable chroma otherwise luma isn't processed

vmulti = Manalyse(superfilt,multi=true,delta=tradius,temporal=true,truemotion=true,blksize=blocksize,overlap=blocksize/2, mt=true, chroma=true)
vmulti2 = Mrecalculate(superfilt,vmulti,thsad=mthresh,truemotion=true,tr=tradius,blksize=rblock,overlap=rblock/2, mt=true, chroma=true)
vmulti2 = isUHD ? vmulti2.MScaleVect() : vmulti2
mocomp = Mcompensate(input,super,vmulti2,thsad=mthresh,tr=tradius,center=true,mt=true) # recursion=50 is bugged
dnmc = mocomp.temporalsoften(tradius,lumathresh,lumathresh,15,2)
dec = selectevery(dnmc,tradius * 2 + 1,tradius)
Y != 3 ? input.mergechroma(dec) : dec }

You mentioned here (https://forum.doom9.org/showthread.php?p=1974137#post1974137) automatic adjustment according to grainlevels/noiselevels.
This I tried with my noise detection to feed to scriptclip that outputs a dynamic mask and also now experimenting using temporalsoften dynamically adjusted (tradius and lumathresh) in ScriptClip for TSMC.

DTL

26th September 2022, 18:13

"if this latest version is suitable for hbd (16bit)"

Currently it is not tested for 16bit. At least the input with MPB-processing enabled should be 8bit (only 8bit component subtraction functions are currently created) and for output only 'lsb' mode compiled (may be old AVS format of 'lsb subplane'). So even for out16 (with AVS+ 16bit plane) it need to be slightly changed and recompiled.
Also the MPB with blocksize luma down to 4x4 and so chroma of 1 sample (blocksize 1x1) for YV12 is not tested - may be better to use YV24 format with both luma and chroma blocksize of 4x4 (it currently work for YV12 and blocksize 8x8 so chroma is 4x4).

About auto-adjust of main params like thSAD - I think about some simple ways to do it inside MDegrain but still not started. For current ideas it may be close to the SCD processing like calculating average of SAD of all blocks in the current frame or all valid blocks in tr-scope and apply some correction multiplier or addition to it (or some more complex function) to calculate thSAD and thSAD2 (and thSCD1) values internally and per each output frame.

"What parameters could you suggest to use and or change in the TemporalSoftenMC (TSMC) script?"

At first it not use MDegrain at all. So MCompensate (also used in widely used QTGMC) is still not changed at all from 2.7.45 version. We have some new features of MDegrain now to move to MCompensate too - at least interpolated overlap modes. May be other new additions too.
As I see it uses MScaleVect for better speed for UHD - it may be not best for quality so you can disable it if required better quality.
The only that can possibly help with speed a bit for this script with current version is using hardware search for MAnalyse (use optSearchOption=5 or 6) and set blocksize to 8 (16 may be still buggy but may be recommended to try for multi-pass search with Mrecalculate later).

The quality difference between degrain approaches of
1. Current MDegrainN
2. MCompensate + temporalsoften() + selectevery()
is good to be evaluated and if approach 2 is better at some scenes I think it may be added as one more mode to MDegrainN (may be something like additional blending mode or blending param for non-linear blending with lumathresh) so you can skip this script (or change sequence of MCompensate +temporalsoften +selectevery to single MDegrainN call) and possibly get better speed with single MDegrainN.

For testing you can try to replace lines

mocomp = Mcompensate(input,super,vmulti2,thsad=mthresh,tr=tradius,center=true,mt=true) # recursion=50 is bugged
dnmc = mocomp.temporalsoften(tradius,lumathresh,lumathresh,15,2)
dec = selectevery(dnmc,tradius * 2 + 1,tradius)

with

dec=MDegrainN(input,super, vmulti2, tr=tradius, thSAD=mthresh, thSAD2=mthresh-10, mt=false, wpow=4, thSCD1=400, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16,
MVLPFGauss=0.9, thMVLPFCorr=50, MPBthSub=10, MPBthAdd=20, MPBNumIt=2, MPB_SPC=1.5)

Though it is not directly compatible for mthresh param and require new adjustment. Also 8bit input/output formats only.

Also for quality it is better to use pel 2 or even 4 in MSuper and set hpad and vpad to blocksize or larger if crashes happen.

anton_foy

26th September 2022, 23:43

Lovely I will try this out tomorrow. I did not have alot of good results with mdegrain but I did see your tests and was impressed! Also mscalevect with blocksize=8 and mrecalculate and "optSearchOption=5" I will try for manalyse/mcompensate. Thanks!

DTL

27th September 2022, 09:39

It looks current 'starting' MPB params of MPBNumIt=2 and MPB_SPC=1.5 may be too high for SD noisy sources like VHS captures or may be other too noisy sources. So it cause significant decreasing of 'denoising' over all frame even with high enough tr and thSAD params. For such sources the MPB_SPC may be as low as 1.02..1.05 (it expected to be > 1.0) and MPBNumIt to 1. It is a bit unexpected low values so may be in future versions the internal math may be changed for more expected adjustment in range of about 1.1..2.0. Though leaving some residual noise may be used instead of 'debanding' after typical too clean degrain.

Also thCohMV param depends on pel about pel*4 so thCohMV=16 is my typical value for pel=4. If you use pel=1 it may be lower to about thCohMV=4.

Addition from 02.10.2022: It was found that using hardware motion search with NVIDIA GTX1060 card looks like sometime produces significant MVs errors and it cause significant details lost with MDegrainN processing. It is more visible if using 'high enough' tr-values like 10. May be other chips of new NVIDIA families or AMD produces better result. So currently for highest quality work only onCPU MAnalyse can be recommended.

DTL

6th October 2022, 18:55

New test release https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.17

Added MPB_PartBlend param to MDegrainN to check real partial blend vs subtraction of block. Default false (use subtraction). If set to true - perform full block blending with test block removed (slower but a bit more accurate in SAD). Subtraction method with integer 8(16 immediate) bit processing may still give up to +-1 error for each sample may be due to integer rounding errors and total SAD difference from real partial blending for even small 8x8 block of 64 samples total may be comparable with low threshold value. So for best possible quality MPB_PartBlend=true is recommended (may be significantly slower).

Separated subtractive and additive coefficients to different params MPB_SPCsub and MPB_SPCadd for better flexibility at experiments of finetuning.

Added isMVsStable function to check if MVs in current tr-pool for current block are enough coherent (stable) - to try to make MPB processing only at areas with stable enough motion search in tr-pool of frames. New MDegrainN param MPBthIVS - threshold to compare current calculated measure of non-stabilily of MVs (sum of accelerations multiplied to sum of vectors angle difference). Param is internally scaled to squared pel value but may significantly depend on tr and other settings. To help adjust this threashold - use IVS-mask display with showIVSmask=true.

Added protection to MDegrainN against too low padding (now need to be at least blocksize in size) and not equal temporal radius param for MAnalyse and MDegrain - display error messages instead of corrupted output.

Added showIVSmask param to MDegrainN to mark blocks detected as stable enough MVs with black. Default false. Black blocks with are detected as ready for MPB processing.

Added mvmultivs param to MDegrainN as option to provide separate MVclip with different search source or options for IVS mask creating. Provided clip must be equal to mvmulti in block number, overlap mode and recommended to use truemotion=false preset of MAnalyse to show noise-moved blocks as best as possible, not recomended to make from prefiltered clip. mvmulti clip may use any required params for best denoising, can be created from prefiltered source and so on.

Example of using mvmultivs clip with separate MAnalyse settings:

tr=10

super=MSuper(last, mt=false, chroma=true, pel=2)

multi_vec=MAnalyse(super, multi=true, blksize=8, delta=tr, search=3, searchparam=2, optSearchOption=1, chroma=true, mt=false)
multi_vec_vs=MAnalyse(super, multi=true, blksize=8, delta=tr, search=3, searchparam=2, optSearchOption=1, truemotion=false, pnew=0, pzero=0, chroma=true, mt=false)

MDegrainN(last,super, multi_vec, tr, thSAD=200, thSAD2=190, mt=false, wpow=4, thSCD1=400, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50,
UseSubShift=1, IntOvlp=3, MPBthSub=5, MPBthAdd=20, MPBNumIt=2, MPB_SPCsub=0.5, MPB_SPCadd=1.5, MPB_PartBlend=false, MPBthIVS=1500, showIVSmask=false, mvmultivs=multi_vec_vs)

Typically onCPU MAnalyse with truemotion=false provide significantly better MVs for IVS-mask generation in compare with hwAcc search with NVIDIA GTX1060 but the processing is slower.

The idea of IVS-masked MPB processing is not to increase noise on non-detailed areas (like out of focus, clear sky and so on). Only enchance details (and also some noise) at areas with at least some details detected by motion search engine and existance of some temporal coherence of the MVs in current tr-scope around current frame. Current IVS-mask generation engine is not final design and may be subject to change in future.

If no mvmultivs clip is provided - single mvmulti is used for all operations (for example in hardware-only search of MVs). Though if MVs clip is created from anti-noise pre-filtered source the quality of IVS-mask may be more degraded.

Addition: The MPB processing in this build also significantly redesigned: Old processing tried to adjust initial weights array from old DegrainWeight function based on ratio of block SAD vs current thSAD. New design trying to create new weight-array from initial equal-weight condition (equal to wpow=7). And after 1 or more iterations of weights aligning it apply finally new calculated weight array to initial using proportional scaling.

Addition2: Currently MPBthIVS param is auto-scaled internally to pel*pel of the 'main working' mvmulti clip because acceleration part of metric depends on square absolute value of MVs coordinates differences and they are scaled with pel value. But to make processing faster if separate MVclip provided for IVS-mask it may be generated from lower pel-mode of MAnalyse (like pel=2 or pel=1) and in the future versions it looks good to make autoscaling to pel value of the mvmultivs clip (if provided). Though it still may significantly depends on other params like tr-value so still not auto-corrected completely anyway and need to be checked before starting of production processing after many other params adjustment.

DTL

9th October 2022, 18:30

New release https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.18

Added SSIM metric in MAnalyse and MPB MDegrainN processing. Now possible dissimilarity metric flags (as bit mask):
bit 0 - SAD,
bit 1 - SSIM luma only,
bit 2 - SSIM contrast and structure.

Examples:
SAD only = 1
SSIM luma only = 2
SSIM contrast and structure = 4
Full (standard) SSIM = 6
SAD + SSIM contrast and structure = 5

Selecting of dissimilarity metric supported only in part of optSearchOptions and optPredictorType of MAnalyse. For example optSearchOption=2 have hardcoded SAD in SIMD and can not be switched.

New param for MDegrainN - MPB_DMFlags. Integer any of dissimilarity metric bitmask, default=1.
New param for MAnalyse - DMFlags Integer any of dissimilarity metric bitmask, default=1.
New param for MRecalculate - DMFlags Integer any of dissimilarity metric bitmask, default=1.

Current release have only C-reference partially float32 SSIM calculation functions (best precision and support all blocksizes and bithdepths) so very slow. For quality check mostly. Current processing speed degradation of MDegrainN MPB using SSIM about 2 times. MAnalyse usiing SSIM about 4 times. Expected good benefit from SIMD versions in the future (for some limited number of block sizes and bitdepths).

Example of interleaved frames with different dissimilarity metric for MPB processing:

ssim=MDegrainN(last,super, multi_vec, tr, thSAD=250, thSAD2=240, thSADC=240, mt=false, wpow=4, thSCD1=400, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50,
UseSubShift=1, IntOvlp=3, MPBthSub=2, MPBthAdd=2000, MPBNumIt=2, MPB_SPCadd=1.5, MPB_SPCsub=0.6, MPBthIVS=1500, showIVSmask=false, MPB_DMFlags=6).Subtitle("SSIM")
sad=MDegrainN(last,super, multi_vec, tr, thSAD=250, thSAD2=240, thSADC=240, mt=false, wpow=4, thSCD1=400, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9, thMVLPFCorr=50,
UseSubShift=1, IntOvlp=3, MPBthSub=2, MPBthAdd=2000, MPBNumIt=2, MPB_SPCadd=1.5, MPB_SPCsub=0.6, MPBthIVS=1500, showIVSmask=false, MPB_DMFlags=1).Subtitle("SAD")
Interleave(ssim, sad)

The SSIM metric produce much more great deviation of metric value so if using in MAnalyse and MDegrain the thSCD1 param of MDegrain must be significantly adjusted to higher value (like from 400 to 15000 or higher). Check MShow mean-SAD output to see expected value. Though thSAD can be about the same as with SAD metric.
Combinations of SAD+SSIM is additive (so thSAD may be about 2 times higher if used).
Combinations of SSIM-luma and SSIM-contrast and structure are multiplied (to form 'standard SSIM'). So enabling standard SSIM may not require significant thSAD adjustment.

Current conversion of SSIM into something close to SAD computing engine:

Dissimilarity metric = (1-SSIM) * maxSAD/2.
Where maxSAD = (3 * nBlkSizeX * nBlkSizeY * (pixelsize == 4 ? 1 : (1 << nBPP)))

So -2 SSIM (totally different blocks) shoud reach maxSAD. Though it looks the intermediate values of SAD and SSIM with some intermediate blocks dissimilarities are very different. May be it good to add some power-function to align a bit better in the future. Though the much higher gain of SSIM around low SAD may helps to MDegrainN and MVs search better.

Expected other metrics for addition in the future like CW-SSIM and others. Currently new intermediate helper class DisMetric is used so addition of new metrics is now much easier. Though the new class add some overhead at calling of metric function and selection so performance of SAD only metric may be visibly degradred.

anton_foy

10th October 2022, 08:05

Thanks DTL! Just a thought here, would it be possible to implement in mvtools a way to analayse and detect the grain/noise amount (using gpu?) for it to automatically and dynamically adjust tr/thsad as an option? I guess it would be a runtime thing that will slow down alot though.

DTL

11th October 2022, 15:42

I still not have good idea how to implement auto-thresholding. Current single mvmulti clip with low tr may be not applicable and/or too slow for something like scene-wide lookahead. So may be add one more analysis mvclip (with single frame temporal stepping) to MDegrainN so it can try to call many next frames fast enough to look from current to next scene-change detect point and calculate something like mean-sad over total scene-wide frame numbers and apply it.

Currently still many more ideas to implement.

It the post about previous version it is not directly mentioned but with optSearchOption=6 it should now able to apply different metric to onHWAcc searched MVs. So it may be real reason why DX12 ME API do not output any metric with MVs - it may be up to the enduser to select and calculate any similarity/dissimilarity metric if required. For other search options the metric is either limited to search metric (like optSearchOption=0 and 1) or limited to SAD only (2,3,4,5). So I think it require more option to MAnalyse to select search and output option to select different metric (the single metric may be not best for search on all cases I think) and MVs consumer filter may be work better with different metric provided. As I see in the article https://videoprocessing.ai/metrics/ways-of-cheating-on-popular-objective-metrics.html there are lots of different metrics available now and good to check how is they work with denoising or other mvtools activity.
Also the computer shader for optSearchOption=5 need to be redesigned to use many more metrics selected by DMFlags (or DMFlagsOutput) option to MAnalyse. So the external to CPU compute accelerator will offload more computing work of computing much more complex metrics in compare with simple SAD metric.

anton_foy

13th October 2022, 16:51

DTL

13th October 2022, 17:16

Currently IVS-mask is to decrease level of details blurring on 'important' areas of frame while keeping degraining at no detailed areas good enough. As I see from latest tests the MAnalyse produces significantly different MVs with SAD and SSIM metrics at the non-detailed areas. So it may be additional method to create IVS-mask from even single pairs of frames search - it is faster in compare with many frames analysis for MVs difference for block. So in the new versions the MDegrainN may have 2 additional inputs of MVs clips and separate tr-value for this part of processing. Like mvmultivs2 and MPB_IVStr params. So the mvmulti clips may be created with much smaller tr and less CPU load, may be down to tr=1. Also it may help to reduce issue of current IVS-mask creation algorithm - after scene change it will not produce any good mask at about tr/2 or more number of frames.

MAnalyse with SSIM metric works about 2 times slower in compare with SAD metric even with AVX2 calculation of SSIM (getting sigmas and means in 16..32 bit integer subtraction/addition/multiplication and final processing in float32 including full precision square root). So it is good to decrease amount of processing with SSIM metric if possible for better performance.

anton_foy

13th October 2022, 21:46

Very interesting with the new IVS-mask. So in the new versions the MDegrainN may have 2 additional inputs of MVs clips and separate tr-value for this part of processing. Do you mean like conditional filtering or a more seamless/smooth transition? I was thinking something like this:

Tr1 = MdegrainN(last,super1,multivec, tr=1, thsad=200...)
Tr6 = MdegrainN(last,super2,multivec, tr=6, thsad=400...)
Mask = 'scriptclip the mask for eg. Blackness() with opacity changing"
Mt_merge(tr6,tr1,mask,luma=true,u=3,v=3)

Changes dynamically between tr1 and tr6 (min/max).

DTL

13th October 2022, 23:49

It is all done inside single blending engine for better speed. Any sequence of 'filters' in AVS takes lots of RAM for caching inbetween filters and for each thread and also the second main reason of single blending engine - it can load once all src and all ref frames blocks data from RAM to CPU L1D cache and make many processing not touching host RAM any more.

If you chain some filters in AVS each filter make full frame scan in some order so data is many time loaded from host RAM into CPU and it is much slower.

So for performance reasons it is better to make all processing for each input/output block in single (may be very complex) blending engine and not write some intermediate results in AVS-clips RAM objects and making final blend/merge in separate objects. Unfortunately AVS-filters can not interoperate image data on small chunks like blocks or samples - only total frames are requested by downstream filters from upstream. And total frame request invalidate most of caches (at least L1D as the fastest but only about <100 kBytes in size).

If IVS-mask is useful for some other scripting it can be special mode of MDegrainN to output this mask in some more nice form like grayscale 256 levels or just 1 black and white. Currently its output is designed only to check of mask placing over the frame data so it is not clean from image data.

I think for better understanding how old and current MDegrainN (MAnalyse + MDegrainN) is working and where many of adjustments params are passing it is good to create some structure scheme. May be will do in some form for documentation.

One sad issue with 'direct output' of IVS-mask: In better quality 'overlapped' mode of MdegrainN it is also generated for each block of 'overlapped blending space' and the blocks count in any overlapped mode are > blocks count in not overlapped mode (simple tesselation of frame to width/blocksizeH and height/blocksizeV number of blocks). So the IVS-mask in simple single frame form can not be outputted for good quality overlapping processing modes. Or some way of output of overlapped mask need to be designed (like doubling or quadrupling of output framerate to output each part of mask in separate frame). It is much more complex. It is also reason of using single blending engine instead of attempt to transfer mask via current AVS scripting interface based mostly on simple 'clips' objects of fixed frame size and may be expecting masks to be produced in same size.

anton_foy

18th October 2022, 15:16

https://forum.doom9.org/showthread.php?p=1962308#post1962308

https://forum.doom9.org/showthread.php?p=1973516#post1973516

Possibly for this thread or not but since it is about improving mvtools I guess it is suitable.
Using RIFE for/in mvtools would be possible?

DTL

19th October 2022, 06:18

Well - RIFE at GTX1060 can run for 1080i degrain at simulating of MDegrain1. Run at about 2 fps at only 3 AVS+ threads fit 6 GB RAM of HWA. 4 and more threads typically throw vkmemoryallocation errors. Also Win10 GPU memory graph hit max value. The most of HWA load is Compute1 about 99..100% in Win10 GPU performance counters.

For 1080i source the script was

FFmpegSource2("src.mxf")

yadifmod2(mode=1)

ConvertToRGB(interlaced=false)

ConvertToPlanarRGB()
ConvertBits(32)
src32=last

even=SelectEvery(2,0)
odd=SelectEvery(2,1)

even_d=RIFE(even)
odd_d=RIFE(odd)

even_d=SelectEvery(even_d,2,1)
odd_d=SelectEvery(odd_d,2,1)

den_int=Interleave(even_d, odd_d).Trim(1,0)

src32_trim=Trim(src32,1,0)

den_int2=Average(den_int, 0.666, src32_trim, 0.3333)
den_int2.ConvertToRGB24().ConvertToYV12()

SeparateFields()
SelectEvery(4,0,3)
Weave()

Prefetch(3)

It is not clean field-based denoise of interlaced (uses intermediate 50fps yadif deinterlaced form) but work as interlaced input and interlaced output. Also it is not clean test for quality because yadif simple deinterlacer adds its own distortions.

The MPEG x264 filesize saving with crf=18 encode is still very visible - the input noised source via same yadif deint->reint path give about 17.5 Mbps and denoised only 7.8 Mbps average.

The only good for users - it do not have any user adjustable params like thSAD and 10..20+ more tweaking params of current mvtools.

Separated RIFE degrain1 function may be like this

function RIFE_degrain1(clip src)
{
ConvertToRGB(src)
ConvertToPlanarRGB()
ConvertBits(32)
src32=last
even=SelectEvery(2,0)
odd=SelectEvery(2,1)
even_d=RIFE(even)
odd_d=RIFE(odd)
even_d=SelectEvery(even_d,2,1)
odd_d=SelectEvery(odd_d,2,1)
den_int=Interleave(even_d, odd_d).Trim(1,0)
src32_trim=Trim(src32,1,0)
den_int2=Average(den_int, 0.666, src32_trim, 0.3333)
return den_int2.ConvertToRGB24().ConvertToYV12()
}

Uses Average plugin - http://avisynth.nl/index.php/Average , may be replaced to something internal AVS+ weighted blending with mask like Layer() or Overlay(). As tr=1 degrain it skips 1st frame of source clip for more simple form of function - output is input-1 frame count.

ChaosKing

19th October 2022, 10:05

Not sure if it was asked here, but would DirectStorage be of any use for mvtools? Or maybe other plugins?

https://devblogs.microsoft.com/directx/directstorage-1-1-coming-soon/

As far as I understand a constant CPU<->GPU transfer can be a big bottleneck. DirectStorage solves this by communication directly to the SSD/NVME storage without touching the cpu.

DTL

19th October 2022, 10:25

DirectStorage is mostly about loading large resources from storage to GPU. Like GPU can issue load commands directly to storage and free CPU cycles to do some useful computing. Most of mvtools processing is about fetching RAM resources and math computing. DirectStorage may be 'environment' feature for AVS (mostly source plugins) because it loads data from storage into AVS environment. But as I see AVS developers not like to make AVS core windows-dependent. So it may be some feature for some source plugin to load file from storage directly to GPU for decompresion and download decompressed frames to host RAM as AVS resources (frames of clip). It may save some CPU cycles but may be not many.

May be some developer can make DirectStorage-using DirectShow plugin so AVS can use it via existing DirectShowSource input.

About RIFE denoising on some long footage test: With default settings it looks have too poor scene change detection and also create flickering on some repeating texture patterns at architecture. May be require model manual tweaking/selecting and params adjusting like scene change.

DTL

24th October 2022, 20:35

New test release: https://github.com/DTL2020/mvtools/releases/tag/r.2.7.46-a.19
Added MPB_MVlth param to MDegrainN. Limit allowed length of MV for weight correctionby MPB processing. Can decrease possible 'ghosting' with too extreme MPB_SPCsub/add params values. Not scaled by pel currently. Recommended values - about 2..3 squared pel value. Valid working range from 0 to squared frame size (unlimited). Zero may be disable MPB weight adjusting completely.

Added reduced tr blending mode for MPB controlled currently by MPB_SPCadd > 10.

Added MPBtgtTR param to MDegrainN:
In standard MPB mode controls initial number of weights used for calculate initial blend estimation (may be 0 - only current block used).
Valid range - from 0 to tr.
In reduced weights MPB mode - controls number of ref frames (2 * tr) used for blending without any other weights adjustment by MPB.

Added MPB_DMFlags=64 flag. Uses covariance metric only. Can be used only with MDegrainN.

Added VIF (DWT- based) metric. Controlled by 0x10 (VIF-Approximation) and 0x20 (VIF-Edges) flags. 16 and 32 decimal. Full VIF (VIF-A * VIF-E) is DMFlags=16+32=48. Can be used in both MDegrainN MPB flags and MAnalyse.

Current possibly best settings for processing:

MDegrainN(last,super, multi_vec, tr, thSAD=250, thSAD2=240, thSADC=240, mt=false, wpow=4, thSCD1=400, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9,
thMVLPFCorr=50, UseSubShift=1, IntOvlp=3, MPBthSub=5, MPBthAdd=5, MPBNumIt=3, MPB_SPCadd=3, MPB_SPCsub=0.3, MPBthIVS=1500, showIVSmask=false, MPB_DMFlags=64, MPB_MVlth=8, MPBtgtTR=tr)

Example of reduced tr blending mode without MPB weights adjusting:

MDegrainN(last,super, multi_vec, tr, thSAD=250, thSAD2=240, thSADC=240, mt=false, wpow=4, thSCD1=400, adjSADzeromv=0.5, adjSADcohmv=0.5, thCohMV=16, MVLPFGauss=0.9,
thMVLPFCorr=50, UseSubShift=1, IntOvlp=3, MPBNumIt=1, MPB_SPCadd=11, MPBthIVS=1500, showIVSmask=false, MPBtgtTR=tr-5)

Blocks with tr cut to tr-5 controlled by IVS mask.

With enough aggressive settings like MPBNumIt=3, MPB_SPCadd=3, MPB_SPCsub=0.3 it is recommended to use low enough MPB_MVlth=8 (about 2 squared pel value) or the small moving objects may start to have 'ghosts' . This 'ghosting' feature of MPG is not perfectly fixed (and even not completely debugged) so added limitation of max MV length for weight adjustment. Current idea of this ghosting: Some blocks far enough from moving objects with errors MVs got some non zero weight in total blending pool of blocks and MPB multi-pass greatly amplificated weight of such blocks as it have some details and they become visible at output. So they looks like exist in standard mvtools/MDegrainN processing but with typical low weight are invisible.

Current MPB_DMFlags=64 uses covariance only metric that is inverted to typical metrics (covariance increases with increasing blocks similarity) and do not have good max expected values to make easy inverting to align with other dis-similarity metrics so it currently implemented outside DisMetric class and uses separate processing function in MDegrainN only. Currently with enough aggressive MPB settings it looks make best result on details keeping (and even look like produce some 'sharpening' effect).

Just a thought here, would it be possible to implement in mvtools a way to analayse and detect the grain/noise amount (using gpu?) for it to automatically and dynamically adjust tr/thsad as an option? I guess it would be a runtime thing that will slow down alot though.

Currently with progress of multi-pass blending and in the future the multi-frame multi-pass blending the importance of thSAD will be less and less important. In this processing the weights of blocks to blend are many times adjusted by many functions. So initial thSAD is for first approximations and to skip too bad blocks. Same as you see the RIFE do not have thSAD or close adjustment and process all blocks similarity automatic.

About offloading of MDegrain MPB and MFMPB to accelerator - it may be possible but in some far future. With the increasing computing complexity of 'self-adjusting blending' the penalty for upload and download result to and from accelerator may be not very big even for per-block processing. Better to upload full set of frames and download resulted frame only but it require more complex compute shader design. Also I do not have good way to debug compute shaders now with remote debug. Though with MDegrainN the host do not need DX12-ME features for run separate compute shader and may be local debug will work on much poor accelerator that exist in some way on my development host with visual studio.

Currently the MPB still not very slow with much more computing in compare with old single pass blending mode with thSAD-defined weights only because it uses very small blocks data once that is good cached to L1D cache of CPU and no more disturbs very slow host RAM. So it good benefits from multi core CPUs and fast cores even with slow enduser RAM with low memory channels and poor performance cache/memory controllers of endusers cheap CPUs.

anton_foy

5th February 2023, 17:46

@DTL, with your versions of mvtools I can run scripts with mdegrain scripts but not motion compensated scripts with MCompensate. It just stops, aborts without error. Do you know why?

Edit: sorry I just read this (https://forum.doom9.org/showthread.php?p=1972963#post1972963)
But does it mean your version does not even support the mcompensate at all?

Also here (https://forum.doom9.org/showthread.php?p=1976228#post1976228) you mention the problem with thSCD but if I do not use scenechange detection at all I put it to zero? Will it make it easier for mvtools to denoise?

For HBD problems mentioned here (https://forum.doom9.org/showthread.php?p=1961840#post1961840) could you just use two MSuper? Does MAnalyse really degrade quality too much if 8-bit yv12?

Something like this:

# HBD source

Sup8 = convertbits(8).converttoyv12()
Sup = source.MSuper(levels=1)
Sup8 = source8.MSuper()
multi_vec_vs = sup8.MAnalyse(multi=true, blksize=8, ...).convertbits(source)
MDegrainN(last, sup, multi_vec_vs, ...)

Selur

5th February 2023, 20:02

Any hope for this getting Vapoursynth support?

DTL

5th February 2023, 22:16

"does it mean your version does not even support the mcompensate at all?"

MCompensate should be the same as in 2.7.45 version from pinterf. But blocksize 16x16 may be broken (or bitdepth >8). Also blocksize of 16x16 do not run well at my GTX1060 with hardware search (in MAnalyse) and cause some crash that remote debugger can not break in. So I use only block size of 8x8. It is possible with scripting to feed half frame sized to MAnalyse with blocksize 8x8 and use MScaleVect to map output MVs it to full frame size with blocks of 16x16 if you want to try. It may be also faster for large frame sizes.

"thSCD but if I do not use scenechange detection at all I put it to zero?"

Disabled thSCD is about maxSAD (and it may be internallly silently clipped to maxSAD). So to disable scenechange detection you need to set thSCD to some verybig value (may be >1000 or >10000).

"could you just use two MSuper? Does MAnalyse really degrade quality too much if 8-bit yv12?"

Yes - you can use 8bit source for MAnalyse and use output MVs clip to any bitdepth MDegrain processing. If source is enough noised it is also enough self-dithered so may be no significant degradation of motion search quality. It is also may be tested with software 16bit MAnalyse.

"Any hope for this getting Vapoursynth support?"

It is a question to developers of Vapoursynth if they can take sources and redesign to Vapoursynth. I do not know how to make it.

anton_foy

5th February 2023, 23:19

MCompensate should be the same as in 2.7.45 version from pinterf. But blocksize 16x16 may be broken (or bitdepth >8). Also blocksize of 16x16 do not run well at my GTX1060 with hardware search (in MAnalyse) and cause some crash that remote debugger can not break in. So I use only block size of 8x8. It is possible with scripting to feed half frame sized to MAnalyse with blocksize 8x8 and use MScaleVect to map output MVs it to full frame size with blocks of 16x16 if you want to try. It may be also faster for large frame sizes.

"thSCD but if I do not use scenechange detection at all I put it to zero?"

Disabled thSCD is about maxSAD (and it may be internallly silently clipped to maxSAD). So to disable scenechange detection you need to set thSCD to some verybig value (may be >1000 or >10000).

"could you just use two MSuper? Does MAnalyse really degrade quality too much if 8-bit yv12?"

Yes - you can use 8bit source for MAnalyse and use output MVs clip to any bitdepth MDegrain processing. If source is enough noised it is also enough self-dithered so may be no significant degradation of motion search quality. It is also may be tested with software 16bit MAnalyse.

Thanks! Then I will try 8x8 blocks with MAnalyse, MCompensate but it will not be any benifits in speed with DX12-ME (GPU) when using MCompensate? Only used for MAnalyse? Sounds a bit similar to SVanalyse then in SVPflow. When using mscalevect I had bad experiences with too soft/blurry results but I will try.

DTL

5th February 2023, 23:32

MCompensate is a client function for MAnalyse MVs server, same as MDegrain and all other from mvtools. MAnalyse can offload MVs search to DX12-ME hardware accelerator.
The MRecalculate is some intermediate - it receives MVs from MAnalyse and make some refinement search (and it can not be offloaded to DX12-ME accelerator because it make only single search from beginning).

In theory with the progress of MDegrainN it also may make some refinement search in multi-pass processing - with CPU only.

MAnalyse with DX12-ME mode should be able to make both single pair of frames search and 'multi' for MDegrainN search (using storage of current frame in accelerator and not upload it for each pair of search in 'multi' mode for better performance). So it should be compatible with all other MVs clients of mvtools.

The hardware ME from MPEG encoder looks not designed for best quality - the software may be better (and with iterative multi pass - more better). So for highest quality work the usage of hardware ME may be only for initial search or may be prefiltering.

anton_foy

7th February 2023, 13:22

#DTL:
pre=convertbits(8,dither=-1).converttoyv12().dipre()
tr=6
super = last.MSuper (pel=4, levels=1, chroma=true)
sup8 = pre.MSuper (pel=4, chroma=true)
#multi_vec = MAnalyse (sup8, multi=true, blksize=8, delta=tr, optSearchOption=5, overlap=0, levels=1, chroma=true).convertbits(16).converttoYUV444()
multi_vec=MAnalyse(sup8, multi=true, blksize=8, delta=tr, search=3, searchparam=2, overlap=0, optSearchOption=1, optPredictorType=4, chroma=false, mt=false, levels=1).convertbits(16).converttoYUV444()
#MDegrainN (super, multi_vec, tr, thSAD=150, thSAD2=250)
Mcompensate(last,super,multi_vec,thsad=thsad,tr=tr,center=true,mt=false)
temporalsoften(6,75,76,255,2)
selectevery(tr*2+1,tr)

I get error "unhandled C++ exception" with this script and if using the mdegrain line I get the same error. If using the first Manalyse line I get this error "MAnalyse: frame width not supported by DX12_ME, max supported width 0".

DTL

7th February 2023, 19:28

If using the first Manalyse line I get this error "MAnalyse: frame width not supported by DX12_ME, max supported width 0".

It looks something wrong with DX12 environment at the execution system - may be driver or device not support motion estimator.

HRESULT feature_support = dev_D3D12VideoDevice->CheckFeatureSupport(D3D12_FEATURE_VIDEO_MOTION_ESTIMATOR, &MotionEstimatorSupport, sizeof(MotionEstimatorSupport));

return no-error, but MotionEstimatorSupport structure looks like not initialized with real working values.

You may try that old DX12-ME checking tool from first posts in this thread - https://forum.doom9.org/showthread.php?p=1959067#post1959067

In correctly working system it should report max frame sizes like 4096x4096 or may be more. Something like in https://forum.doom9.org/showthread.php?p=1959078#post1959078 post.

"multi_vec=MAnalyse(sup8, multi=true, blksize=8, delta=tr, search=3, searchparam=2, overlap=0, optSearchOption=1, optPredictorType=4, chroma=false, mt=false, levels=1).convertbits(16).converttoYUV444()"

output of MAnalyse is special clip for mvtools other members and not need and can not be correctly processed with any other AVS filters. Also it is independent of bitdepth and colour format of processed clip. It only about blocksize/blocknumbers and other analysis params. So it is not correct to attempt to change it with .convertbits(16).converttoYUV444(). Same is for all other lines with MAnalyse. If MVs clip is damaged with any processing it may cause crashes in other mvtools filters because it is not checked for checksum or other ways. It is put to documentation - you can not modify it with almost any tools (except some of mvtools like MScaleVect() and others).

From mvtools2.html :
Technical note: MAnalyse does not generate a regular clip that can be displayed. Don't try to modify its content or it will get corrupted. It is made of a single long line, actually containing binary data (vectors, block SAD, misc. information…) instead of pixel values. It also alters the audio descriptor to pass additional data to other filters before any frame request. Therefore, in the current state, a vector clip cannot be saved to a lossless file and reloaded later for processing. If you want to do so, you have to transcode it first with MStoreVect and MRestoreVect. Furthermore, joining vector clips generated with different parameters may lead to unexpected results, because the aforementioned additional data is global to the whole clip and is not updated on each frame. When the MAnalyse filter is destructed (removed from the memory), the additional data is lost too, and an attempt to using the produced vectors may crash the application or give wrong results. For this reason, avoid using MAnalyse in ScriptClip and other functions of the Avisynth runtime subsystem.

anton_foy

8th February 2023, 03:35

Tried the D3D12HelloTriangle and I got "result: 0". I use the SSE2 version of your mvtools.

DTL

8th February 2023, 06:44

"Tried the D3D12HelloTriangle and I got "result: 0"."

HRESULT of 0 in Microsoft API typically mean S_OK (no-error). But you need to got non-zero (and not very small) MaxH/MaxW values in

MEstimator Feature support: D3D12_VIDEO_SIZE_RANGE SizeRange MaxW 4096 MaxH 4096 MinW 32 MinH 32

It must be more than your frame size.

anton_foy

8th February 2023, 23:47

ReinerSchweinlin

9th February 2023, 09:44

My card does not support it then:
D3D12_VIDEO_SIZE_RANGE SizeRange MaxW 0 MaxH 0 MinW 0 MinH 0
What Card are you using ?

anton_foy

9th February 2023, 11:01

What Card are you using ?

The GTX 970

DTL

9th February 2023, 11:34

It may really not support exposing DX12-ME API (though may supoport hardware MPEG encoding via older version of MPEG encoder ASIC). As I my test shows mostly probably DX12-ME interface is supported from GTX1xxx cards numbers.

ReinerSchweinlin

10th February 2023, 11:50

The GTX 970

Thats Maxwell 2.0 GM204 Chip. AFAIR the Encoder is one generation too old.

DTL

10th February 2023, 12:32

It is also interesting when finally Intel integrated MPEG encoder (or may be separate video boards or internal accelerators) will expose DX12-ME interface. Also at some theoretical point one can collect 3 different manufacturers of hardware MPEG encoders (NVIDIA, AMD and Intel) exposing DX12-ME interface and check its quality relative using single noised footage.

Different ME engines may have different search range and most interesting is quality of MVs at significantly noised source. For example how many definitely false non-zero MVs do ME engine produce at flat static noised only areas. For visual comparison of quality output MVs may be displayed with MShow() filter. If MShow() crashes - it is required to greatly increasing padding in MSuper() because some hardware MEs produces large out of frame MVs and current MShow() do not have protection from running out of buffer at attempt to draw such long invalid MV. onCPU 'standard' MAnalyse do not seach MVs outside given padded buffer or clips MVs to given padded buffer so no additional out of buffer checks in other mvtools filters (except MDegrainN it last builds. So even MCompensate with DX12-ME search in MAnalyse may also crash now. Sorry - it was only tested with MAnalyse+MDegrainN for degrain).

Double check and clipping of MVs in both MAnalyse (received from DX12-ME) and in MDegrainN will slightly degrade performance. So may be additional checking/clipping need to be added to all other consumer filters (or finally move from MDegrainN to MAnalsye if it not cause additional issues).

anton_foy

12th February 2023, 01:01

Thanks Reiner and DTL now I borrowed a graphics card tested with d3d test and it put out 4096x4096 res so it hopefully works. I will report.

EDIT: Nope, "can not load file Compute.cso". I have made a separate folder with only DTL's mvtools2.dll and the .cso-file.

EDIT2: now it works when I change from optSearchOption=5 to optSearchOption=1.

DTL

13th February 2023, 22:34

" "can not load file Compute.cso"."

Compute.cso file need to be in the 'current working directory'. It depends on the settings of your application loading your AVS script. You may start with placing Compute.cso to the directory with .avs script.

"I change from optSearchOption=5 to optSearchOption=1."

DX12_ME is used only in SO=5 or 6. Option 5 uses Compute.cso and SAD computing in accelerator (faster) and option 6 uses onCPU SAD computing (slower but may be a bit better quality with pel 2 or 4).

anton_foy

14th February 2023, 00:14

DTL

14th February 2023, 11:58

If Compute.cso loads OK you should be able run processing mode like:

MSuper(pelrefine=false, pel=4) # do not create refined subplanes for pel > 1
MAnalyse(optSearchOption=5) # use hardware motion search and SAD computing with Compute Shader on the same accelerator
MDegrainN(UseSubShift=1) # use blend-time computing of subsample shifted blocks

Boulder

14th February 2023, 13:02

Can the mod be used just to speed up calculating the MVs by setting optSearchOption=6 in MAnalyse without any ill effects, and use other MVTools functions from the original pinterf build?

DTL

14th February 2023, 16:07

You may try and look if it work or throws some errors (or crashes). optSearchOption=6 also uses DX12-ME for getting MVs from accelerator and only compute dissimilarity metric at CPU (SAD or others). Because DX12-ME not provide SAD data (or other (dis)similarity metric). So either compute shader is used in optSearchOption=5 or old MAnalyse with optSearchOption=6 single pass without search and only with SAD (or other) metric compute.

If some consumer plugin/function do not need SAD data from MAnalyse output - it may be designed one more option with better performance (only output MVs, like optSearchOption=7 for example).

Because mvtools support lots of different bitdepth and blocksize and other processing combinations - lots of different limitations also exists. For example
optSearchOption=5 - support only SAD computing in current shader implementaion.
optSearchOption=6 - can use any dissimalarity metric compute (defined by DMFlags option) like classic SAD or SSIM or VIF (or some combination like SSIM-structure only).

In the future mvtools it is better to rename SAD only dismetric to DM so params naming from thSAD to thDM for example.

Boulder

14th February 2023, 16:17

Well, looks like overlap is not supported so that's a no-no for me :(

DTL

14th February 2023, 16:49

Interpolated overlap currently only in the MDegrainN. It need to be implemented in other functions.

Or may be added to MAnalyse to output 4x-overlap (of the overlap=blocksize/2) in DX12-ME MVs processing mode. Better in performance 'diagonal 2x' overlap need to be implemented in each consumer filter. In both 'real search' and 'interpolated' versions.

Also you may try 'scripted diagonal overlap' with 2 shifted clips and BlockOverlap() plugin. It will be 2 real searches and possibly a bit higher in quality in compare with 1 search and interpolation. But it require to change your processing scripts.

Where do you need overlap ? MCompensate for QTGMC ?

It was expected some programmers may expand existing working solutions like interpolated overlap designed currently in MDegrainN to other filters/functions required by any existing users. But it looks current civilization dies too fast and almost no programmers and no users of avisynth and mvtools left today. And it only 2023 year - next will be even worse.

I not use deinterlacing of interlaced SD/HD and not use QTGMC so not have great need to redesign MCompensate or other parts of mvtools.