MVTools-pfmod

[DTL]

Heh - with all new features added in end of April (+2 new filters for mvtools) I think to make version numbers more separate: Make next release 2.8.0 tagged instead of planned 2.7.46-a.30. As in x.Y.z system the Y-advancing mean something (significantly) new added.

[Boulder]

Are the two ICL builds meant only for specific CPUs? I was looking to do some comparison between the builds and they refused work on my system (5950X), "Script error: There is no function named 'msuper'" is what I got.

[DTL]

The (partial) fix for unbiased shift-division at UV blocks fetching with chroma-subsampled formats (https://github.com/pinterf/mvtools/issues/59) is not planned to addition to next versions ?

[pinterf]

Are the two ICL builds meant only for specific CPUs? I was looking to do some comparison between the builds and they refused work on my system (5950X), "Script error: There is no function named 'msuper'" is what I got.
If don't have SSE4.2 they won't work. See the notes txt.

[DTL]

AMD 5950X listed as having up to AVX2 https://www.techpowerup.com/cpu-specs/ryzen-9-5950x.c2364 and including SSE 4.2.

It looks I found - ICL builds require intel libmmd.dll. Asd-g include it with some builds.

And also svml_dispmd.dll - they are somewhere in the redistributabe of intel compiler ?

With these (old) versions from 202x years ICL build is working at Core 2 Duo E7500 CPU (no AVX even) - https://drive.google.com/file/d/1eDZ3Zukk6wzaN-vUirvrovYmevcxTMrk/view?usp=sharing . But it may be better to add .dlls from new (?) intel compiler.

Also svml_dispmd.dll and libmmd.dll required to load ICX_LLVM build. But it crash VirtualDub with script of:

LoadPlugin("ffms2.dll")
LoadPlugin("mvtools2.dll")
FFMpegSource2("52.avi")

AssumeBFF()

ConvertToYV12(interlaced=true)

SeparateFields()
tr = 6 # Temporal radius
thCommonSAD=600

super = MSuper (pel=2, chroma=true, levels=0, hpad=64, vpad=64)
multi_vec = MAnalyse (super, multi=true, overlap=0, overlapV=0, blksize=8, search=3, searchparam=2, delta=tr, chroma=true, mt=false, levels=0)

mvlpf=MDegrainN(super, multi_vec, tr, thSAD=thCommonSAD, thSAD2=thCommonSAD-1, mt=false)
last = mvlpf

Weave()

Prefetch(2)


Either intel .dlls are old or not applicable to ICX at all or other reason.

[pinterf]

The (partial) fix for unbiased shift-division at UV blocks fetching with chroma-subsampled formats (https://github.com/pinterf/mvtools/issues/59) is not planned to addition to next versions ?
I checked the open issues but did not find exact and quick code replacement for #59 (given the time frame I wanted to deal with this release).

Actually I didn't even plan this "next version" but the latest issue could be solved like changing a "+" sign to "-". Then I realized that I still had un-uploaded former commits, so I wanted to do a cleanup. Then it turned out that my Intel compiler does not work and there is an Intel 2024 compiler instead of 2021. Downloaded, fixed the old and new warnings and build errors, the XP build did not built either. And then came the documents and change logs and re-tests for all compilers.

So the "oh it's only one character" change turned into a nine-hour process :)

[DTL]

I checked the open issues but did not find exact and quick code replacement for #59 (given the time frame I wanted to deal with this release).


This commit - https://github.com/DTL2020/mvtools/commit/fa7ccc48f0d0a588e2b9e3ec957f8e6d47cef1ef

[pinterf]

This commit - https://github.com/DTL2020/mvtools/commit/fa7ccc48f0d0a588e2b9e3ec957f8e6d47cef1ef
Thanks, this will fit in the weekend I hope

[pinterf]

This commit - https://github.com/DTL2020/mvtools/commit/fa7ccc48f0d0a588e2b9e3ec957f8e6d47cef1ef
Thought this direct link works, but when I add your branch as a remote, I can see a totally different commit under this title, and on that day, with a different commit ID:
https://github.com/DTL2020/mvtools/commit/bfff0f0664fde574e507b4910c3bc3c44d1758cb
I thought I can cherry-pick it but I rather copy-paste the differences unless we figure out what happened with the history of your repo.

[DTL]

I am not understand how github is working and only use web-interface of it (once github-desktop lost one of my commmit so I stop using even this way). So it is better to make copy-paste. There is a few places to change and all changes are equal:
At GetPointer() call we need to add +1 to dx and dy MV coordinates bias before >>LogRatioUV shift-division if LogRatioUV > 0. This makes quantization error distribution symmetrical and decrease max abs error to factor of 2.

if (nLogxRatioUV_super == 1) blx++; // add bias for integer division for 4:2:x formats
if (nLogyRatioUV_super == 1) bly++; // add bias for integer division for 4:2:x formats

Complete fix of this issue requires bigger mvtools redesign - make pel*2 UV super clip part and use real /2 coordinate fetching without any error. It is equal to internal conversion of input 4:2:x to 4:4:4.
For pel=4 it requires to design either pel=8 for UV planes or use runtime sub-shifting of block with pel=8 granularity.

Run-time subshifting is much slower for MAnalyse searches but faster for MDegrain (only single shifted block required).

[pinterf]

I am not understand how github is working and only use web-interface of it (once github-desktop lost one of my commmit so I stop using even this way). So it is better to make copy-paste. There is a few places to change and all changes are equal:
At GetPointer() call we need to add +1 to dx and dy MV coordinates bias before >>LogRatioUV shift-division if LogRatioUV > 0. This makes quantization error distribution symmetrical and decrease max abs error to factor of 2.

if (nLogxRatioUV_super == 1) blx++; // add bias for integer division for 4:2:x formats
if (nLogyRatioUV_super == 1) bly++; // add bias for integer division for 4:2:x formats

Complete fix of this issue requires bigger mvtools redesign - make pel*2 UV super clip part and use real /2 coordinate fetching without any error. It is equal to internal conversion of input 4:2:x to 4:4:4.
For pel=4 it requires to design either pel=8 for UV planes or use runtime sub-shifting of block with pel=8 granularity.

Run-time subshifting is much slower for MAnalyse searches but faster for MDegrain (only single shifted block required).
Visual Studio Community 2022 perfectly integrated git, now with quite a few 'advanced' options are supported, I recommend it.

These changes intend to fix a chroma shift. Does it work indepentenly of the original chroma placement, e.g. top-left or center?

[FranceBB]

Thank you for the SSE4.1 assemblies, they work like a charm on Windows XP.


ConvertBits(10)

super = MSuper(pel=2, sharp=1)
bv1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
fv1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
bv2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
fv2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super,bv1,fv1,bv2,fv2,thSADC=200, thSAD=200)


SSE4.1

AVSMeter 3.0.4.0 (x86), (c) Groucho2004, 2012-2020
AviSynth+ 3.7.3 (r4066, master, i386) (3.7.3.0)

Number of frames: 501
Length (hh:mm:ss.ms): 00:00:16.717
Frame width: 848
Frame height: 480
Framerate: 29.970 (30000/1001)
Colorspace: YUV420P10
Audio channels: 2
Audio bits/sample: 32 (Float)
Audio sample rate: 48000
Audio samples: 802401

Frames processed: 501 (0 - 500)
FPS (min | max | average): 5.856 | 21.34 | 13.31
Process memory usage (max): 438 MiB
Thread count: 26
CPU usage (average): 62.9%

Time (elapsed): 00:00:37.636



Plain C

AVSMeter 3.0.4.0 (x86), (c) Groucho2004, 2012-2020
AviSynth+ 3.7.3 (r4066, master, i386) (3.7.3.0)

Number of frames: 501
Length (hh:mm:ss.ms): 00:00:16.717
Frame width: 848
Frame height: 480
Framerate: 29.970 (30000/1001)
Colorspace: YUV420P10
Audio channels: 2
Audio bits/sample: 32 (Float)
Audio sample rate: 48000
Audio samples: 802401

Frames processed: 501 (0 - 500)
FPS (min | max | average): 3.335 | 12.27 | 8.005
Process memory usage (max): 439 MiB
Thread count: 26
CPU usage (average): 54.2%

Time (elapsed): 00:01:02.584


https://i.imgur.com/RVrjqkN.png

[pinterf]

Thank you for the SSE4.1 assemblies
Are you sure with these performance numbers? Did you compare 2.7.45 and 46? Was it surely both example at 10 bits in the tests? Mdegrain2 was not changed unless I forgot about something I have done in the past theee years :). MdegrainN was affected, when thSAD and thSAD2 is different.

[DTL]

"These changes intend to fix a chroma shift. Does it work indepentenly of the original chroma placement, e.g. top-left or center?I am not sure - it is subject to testing. Currently I am not see in the design of mvtools any respect of chroma-placement. It may simply 'work from 2004 year as is'."

Typically with any noised sources the MV-jitter is much larger in the comparison of differences of sub-sample chroma-placement and after applying MDegrain total jitter of both luma and chroma planes is also much larger. So users may notice only large-areas 'stable' chroma-shift reported in that issue originally.

"Are you sure with these performance numbers? Did you compare 2.7.45 and 46? Was it surely both example at 10 bits in the tests? Mdegrain2 was not changed unless I forgot about something I have done in the past theee years . MdegrainN was affected, when thSAD and thSAD2 is different."

Each build/version of the compiler can create significant differences in the executable performance at some hosts. Also when compiled with some limits of max-SIMD valid. So if even sources do not change - the relative performance of 2.7.45 old builds and all new builds may be different. CPU usage reported is also very different.

Also do you add intel .dlls to run ICL/ICX builds ? I still don't know why got crashed with ICX build and think it requires proper intel .dlls to be used (from build environment). ICX_LLVM build is expected to be fastest at least at intel-CPU hosts.

"Visual Studio Community 2022 perfectly integrated git, now with quite a few 'advanced' options are supported, I recommend it"

After I updated VS2019 from some 'old' to some 'new' version I got lots of compile errors in Microsoft-CPP text for DX12-ME part of mvtools. A natural error was found in Microsoft DX samples working with 'old' VS2019 and stop working with 'new' VS2019. So I need to add new fixes to DX12 MAnalyse to get it compiled with 'new' VS2019. So I am afraid the migration to VS2022 may open even more errors to fix. Now I have working DX12 builds for me.

[FranceBB]

Are you sure with these performance numbers?

Well, both 10bit, one with SetMaxCPU("none") and the other without it, but that's not the problem unfortunately.


Mdegrain2 was not changed unless I forgot about something I have done in the past three years :). MdegrainN was affected

I'm an idiot, when you said "MDegrainN" for some reason my brain thought you meant "MDegrain1", "MDegrain2", "MDegrain3" etc where "N" stands for "number", not the MDegrainN() function hahahahahahahahaha


Anyway new test, this time properly:

#Indexing progressive flagged as interlaced SD source
video=FFVideoSource("test.ts")
audio=FFAudioSource("test.ts")
AudioDub(video, audio)

#From SD anamorphic to SD 1.77 FF 16x9
Spline64Resize(848, 480)

#Bringing everything to 10bit
ConvertBits(10)

#Degrain
Super = MSuper(Pel=1, Chroma=true)
Vector = Super.MAnalyse(Multi=true, Delta=3, BLKSize=8, Overlap=4, Chroma=true)

MDegrainN(Super, Vector, 3, thSAD=150, thSAD2=135, thSADC=150, thSADC2=135)

https://i.imgur.com/pIZubQa.png


Plain C

AVSMeter 3.0.4.0 (x86), (c) Groucho2004, 2012-2020
AviSynth+ 3.7.3 (r4066, master, i386) (3.7.3.0)

Number of frames: 501
Length (hh:mm:ss.ms): 00:00:16.717
Frame width: 848
Frame height: 480
Framerate: 29.970 (30000/1001)
Colorspace: YUV420P10
Audio channels: 2
Audio bits/sample: 32 (Float)
Audio sample rate: 48000
Audio samples: 802401

Frames processed: 501 (0 - 500)
FPS (min | max | average): 2.782 | 11.77 | 8.425
Process memory usage (max): 422 MiB
Thread count: 26
CPU usage (average): 64.2%

Time (elapsed): 00:00:59.466


SSE4.1

AVSMeter 3.0.4.0 (x86), (c) Groucho2004, 2012-2020
AviSynth+ 3.7.3 (r4066, master, i386) (3.7.3.0)

Number of frames: 501
Length (hh:mm:ss.ms): 00:00:16.717
Frame width: 848
Frame height: 480
Framerate: 29.970 (30000/1001)
Colorspace: YUV420P10
Audio channels: 2
Audio bits/sample: 32 (Float)
Audio sample rate: 48000
Audio samples: 802401

Frames processed: 501 (0 - 500)
FPS (min | max | average): 3.920 | 19.29 | 12.10
Process memory usage (max): 421 MiB
Thread count: 26
CPU usage (average): 68.7%

Time (elapsed): 00:00:41.395

[amayra]

does this version support VapourSynth ?

[DTL]

No. VS has even more old and simple version - https://forum.doom9.org/showthread.php?t=171207

[Selur]

Any plans for Vapoursynth support?

[DTL]

mvtools2 is a complex software project (may be the most complex for current AVS+ software). So I see there were designed separate version for VS. It is better to ask current (if any still exist) programmers of VS version of mvtools why a separate version was designed ?

Somewhere I read about performance reasons like VS-mvtools running up to several times faster in VS-environment. It may be because of different memory management ? So VS-mvtools also may be an attempt to re-design of some old mvtools version for better performance and other new nice (programming ?) features. But it looks development of VS-version was stopped even before AVS-version.

As I see porting of mvtools2 to VS started about 2014 - a 10 years after beginning of mvtools2 design. But 201x years were already great end of all freeware video processing software (with end of home video cameras) so the porting was incomplete.

[Selur]

Thanks, so I guess that is a: No :(

[amayra]

it's not possible to port improvements over to vapoursynth version?

[DTL]

Everything is possible - but programmers required. As you see we have close to no programmers for AVS plugins in 202x years from the total planet. Sad enough. I tried to ask Asd-g for some non-free help with mvtools (like bug fixes or feature additions) but no reply.

[Soloiam68]

I have been using MVTools for a while with great results. However, recently I encountered some issues. I encoded a video and noticed some weird horizontal line-type blockings. It’s not banding, but when a cloudy-type object covers a wide area and moves, it looks like multiple horizontal lines of box-type patterns appear and it shows up on multiple parts of the video. It's not present in the source file and isn't coming from any other prefilters used before MDegrain3. When the video goes through MVTools and removes the grain, the issue appears. I am facing this issue recently and have tried everything to solve the problem, but nothing has worked. I also used the latest and many other versions of MVTools, but the issue persists.

My Setup:
AMD Epyc Processor; Windows 10 Server Edition 2019
AviSynth+ 3.7.3
Encoder Used: x264 (veryslow preset)

All the source, avs, test files and images are included.

Files (https://drive.google.com/file/d/17647P0pgmzSZCFe_G4Lomv1X7G6ACBgj/view)

Please someone look into the matter kindly and inform me. My request.

[takla]

FPS (min | max | average): 2.782 | 11.77 | 8.425
FPS (min | max | average): 3.920 | 19.29 | 12.10

Over 40% speedup? Amazing :o

[Selur]

@Solaima68: opened your script and can't the artifacts (I added avsfilter, to the required filters since you didn't include it), you highlighted
Here's what I see in AvsPmod:
https://i.ibb.co/7VZrWsS/grafik.png (https://ibb.co/q9cmsTD)
Here's what I see when I look at your test-output.mp4 in AvsPmod:
https://i.ibb.co/HTJm3dj/grafik.png (https://ibb.co/F4PGCVF)
neither of them seem to show the artifacts in your screenshots.
=> might be a playback issue

[tormento]

From MVTools documentation:

dct: using of block DCT (frequency spectrum) for blocks difference (SAD) calculation. In particular it can improve motion vector estimation at luma flicker and fades.
0 - usual spatial blocks, do not use DCT;
1 - use block DCT instead of spatial data (slow for block size 8x8 and very slow for other sizes);
2 - mixed spatial and DCT data; weight is dependent on mean frame luma difference;
3 - adaptive per-block switching from spatial to equal-weighted mixed mode (experimental, a little faster).
4 - adaptive per-block switching from spatial to mixed mode with more weight of DCT (experimental, a little faster).
added in v1.9.5.3 modes: (require sadx264 0-7, uses fastest SATD function, luma only!)
5 - SATD instead of SAD for luma
6 - same as 2 only use SATD
7 - same as 3 only use SATD
8 - same as 4 only use SATD
9 - similar to 2, use SATD and weight ranges from SAD only to equal SAD & SATD
10 - similar to 3/4,use SATD weight is on SAD, only on strong luma changes
Default = 0.

Can someone explain me some concrete cases where to change them from 0 to other values? What are the benefits and the caveats?

[takla]

Can someone explain me some concrete cases where to change them from 0 to other values?

Example:

Vector = Super.MAnalyse(Multi=true, Delta=3, BLKSize=8, Overlap=4, Chroma=true, dct=1)



What are the benefits
Insignificant quality enhancement. CPU cycles are much better used on a lower BLKSize & higher Delta value.

[tormento]

Example
Thanks for your time but I need case scenarios. Such as for this kind of noise or function, use this or that.
Insignificant quality enhancement. CPU cycles are much better used on a lower BLKSize & higher Delta value.
I disagree. I played a bit yesterday with SMDegrain, setting DCT=10, and the resulting video went from 5 GB to 4GB with no discernible difference.

[takla]

I disagree.
SMDegrain

And there is your problem. Use EZdenoise (https://forum.doom9.org/showthread.php?t=183192) and you'd see for yourself that what I said is true.

Thanks for your time but I need case scenarios.

No you don't.
Use

EZdenoise(BLKSize=4, Overlap=2, tr=8, Chroma=true, out16=true)
ConvertBits(10, dither=1)


For maximum quality and best efficiency of your CPU time.

[tormento]

Use EZdenoise
Tried with the material posted here (https://forum.doom9.org/showthread.php?p=2008439#post2008439).

Nice on light scene but the grain in the dark one is still there, dancing around.

[takla]

Tried with the material posted here (https://forum.doom9.org/showthread.php?p=2008439#post2008439).

Nice on light scene but the grain in the dark one is still there, dancing around.

Sorry, I can't help you further. I would, but my PC is dismantled atm.

[tormento]

Did you compare 2.7.45 and 46?
The Intel_ICL_Classic and Intel_ICX_LLVM builds are not working for me.

They give me the error "MSuper not found".

MSVC and ClangCL ones work ok.

[StvG]

The Intel_ICL_Classic and Intel_ICX_LLVM builds are not working for me.

They give me the error "MSuper not found".

MSVC and ClangCL ones work ok.

You probably need Intel runtimes (https://www.intel.com/content/www/us/en/developer/articles/tool/oneapi-standalone-components.html) (click on left "Runtime Versions"->"Runtime Versions - Microsoft Windows* and macOS* Binary Packages" and download "Intel® oneAPI DPC++/C++ Compiler Runtime for Windows*").

[tormento]

You probably need

Uhm. Will try that, I run many other plugins and x265 compiled with ICC and I had no issues. Perhaps they were statically linked.

[tormento]

Eventually, they are not statically linked. It works with ICC runtimes.

SMDegrain (tr=9, thSAD=900, plane=4, chroma=true, LFR=false, DCTFlicker=false)

MSVC 2.471
Intel_ICL_Classic 2.581 (+4,45%)
ClangCL 2.658 (+7,57%)
Intel_ICX_LLVM 2.737 (+10,76%)

[Adub]

I have a clarifying question I'd like to ask about the behavior of MVTools, specifically the use of MSuper, the concept of "pel" and pixel subsampling as a method of speeding up motion search.

If I'm reading the MVTools code correctly, MSuper creates a new clip from a source clip, with the new clip containing a hierarchy of levels. The top level (index 0, if thinking of the hierarchy as an array) contains the frame data that has been interpolated from the source frame, according to the "pel" parameter. So if "pel = 2", then the top level is twice the size (twice the width, twice the height, so technically 4x) of the source frame. If "pel=4", then the top level is 4x the size (4x width, 4x height, so 16x).

Sticking with the example of "pel = 2", then the next level down in the hierarchy is the original source frame (the equivalent of pel = 1). Continuing, the next level from that is 1/2 the size of the source frame, and so on and so on until some very small fraction of the original source.

This makes sense, as this is the hierarchy that is walked when searching for motion vectors. It starts from the bottom (the smallest version of a frame) (technically there are two hierarchys, one for the current frame and one for another frame, like the previous or next frame), finds the motion vector of a block, and then works its way up the hierarchy refining that motion vector so that it is more and more accurate as we work our way up the hierarchy.

Where I'm a little confused is how MSuper seems to split up these levels of the hierarchy. You can see this if you output the super clip directly. It appears to be creating subsampled versions of each level, and stacking them on top of each other.

Using the "pel=2" example, then the top of the stack is derived from an interpolated frame that's twice the size of the source frame, where the very top contains the information for pixels at index 0, 2, 4, 6, etc. Then the next clip in the stack contains the information for pixels 1, 3, 5, 7 of the interpolated frame. This means, in theory, that the top of the stack actually contains a copy of the originally sized frame, but then the next clip in the stack contains just the interpolated pixels from the 2x upscale. So the source frame is upscaled 2x, and then subsampled into two different clips and stacked on top of each other.

Do I have this correct?

If so, then looking at the motion search code, it selects which clip to search based on the coordinates of the block it is searching for. If the coordinates are (0,0) for example, then it selects the clip at the very top of the stack, which contains the original pixels (and *NONE* of the interpolated pixels) and loads that data into a block. If the block had the coordinates (1,0) instead, then it would select the clip with the interpolated pixels.

This seams to suggest that only some blocks use interpolated pixels, instead of all blocks, as I would have originally expected.

What exactly is the purpose of this approach?

It kind of seems like MSuper interpolates all of these pixels, but then MAnalyse turns right around and inverts the process for some blocks, begging the question - if you're going to subsample, then why interpolate all of that pixel data in the first place, as you're just going to skip over it?

I hope this question makes sense. I'm really just trying to learn how MVtools is operating.

[DTL]

For pel >1 super clip contain upsampled version of a frame. For pel=2 2x upsampled and for pel=4 4x upsampled.
The only difference with standard upsampled frame - it is stored as separated shifted versions of a frame. It is done to make MAnalyse and MDegrain faster on old CPUs - to make sub-sample precision search it do not shifts or upsample frame for each operation but fetch shifted versions of input frame from MSuper clip (so it create large read traffic and may cause slower performance on new many-cores CPUs and narrow slow memory bus). To partially fix it at MDegrainN version was designed runtime sub-sample shifting of real size input frame block. Also available for MAnalyse. But it do not make performance of MAnalyse faster (on current non-AVX512 and slow chips) - only save RAM because you can always set pel=1 for MSuper and it create lower-sized clip.

The reason of storage of 2x or 4x upsampled frame as a set of shifted sub-planes was for more simple design of MAnalyse search as I remember. But this also cause very hard to make pel>1 search versions on SIMD because it can not simply compare upsampled version of source with integer-shifted upsampled version of ref frame. It always compare of 1x size source block with sub-sample shifted version of ref block - but sub-sample shifted is located in different part of RAM and can not be read in register file fast enough.

You can try to make same super-clip by separating 2x or 4x upsampled frame to 1x sized shifted sub-planes. It is also good example of digital interpolated frame view versions - it either

1. interleaved sub-sample shifted copies (standard upsampled frame storage) as we got from
XResize(width*2, height*2)

or
2. separated set of sub-sample shifted 1x versions of frame as we got from (as current MSuper do and store)
XResize(width, height, force=X, sub_x=-0.5)
XResize(width, height, force=X, sub_x=+0.5)
XResize(width, height, force=X, sub_y=-0.5)
XResize(width, height, force=X, sub_y=+0.5)

"Do I have this correct?"

Yes.

"This seams to suggest that only some blocks use interpolated pixels, instead of all blocks, as I would have originally expected."

Yes - for best performance it may use real input frame as (0.0f, 0.0f) sub-shifted version and create 3 more versions (for pel=2 and 2x upsample) as
0,0=src
1,0=XResize(width, height, force=X, sub_x=+0.5)
0,1=XResize(width, height, force=X, sub_y=+0.5)
1,1=XResize(width, height, force=X, sub_x=+0.5, sub_y=+0.5)

This really cause some distortions and non-equal for 0,0 and sub-shifted versions because for many source no-way of zero-distortion interpolation exist. Internally MSuper uses some sort of Spline (?) interpolation kernel (for sharp=2 and it is not equal to Lanczos with taps=2 - mostly looks like Spline or even different kernel) and it works not bad even for non-sinc conditioned sources (low ringing). But it also means it can make less sharp result for good sinc-conditioned source. This is because MSuper allows user to provide external upsized/interpolated clop for sub-shifted separation only internally. If user knows input source better and can provide better upsized version (like SincResize(taps=16) for good sinc-conditioned sources).

As I remember the sharp=2 interpolation is simple and so fast and create nice non-ringing (or at least very low ringing) result at even completely non-conditioned for sinc resizers sources. And I had an ideas to make that kernel as additional AVS resize kernel to use at some 'unknown' sources. But it may be close to Spline (?).
In sources it is called: https://github.com/pinterf/mvtools/blob/a488b095c4bdc8d81abfd952ab534c015a9d45b7/Sources/Interpolation.cpp#L1879
// so called Wiener interpolation. (sharp, similar to Lanczos ?)
// invarint simplified, 6 taps. Weights: (1, -5, 20, 20, -5, 1)/32 - added by Fizick

So can be named WienerResize(). Though its kernel weights also looks close to some weighted-sinc of low (support and kernel size) size. But only kernel for fixed 2x or 4x is available and no generic f(x) kernel function in the sources so it can not be directly applied to current AVS resize fiilter. Need to find the full f(x) equation for float x argument for that Wiener kernel (if it exist at all).

" if you're going to subsample, then why interpolate all of that pixel data in the first place, as you're just going to skip over it?"

MAnalyse with full exhaustive search always checks all sub-shifted positions in the given radius. But MDegrain (and any other motion-compensated) function only fetch the sub-position pointed by (fractional last 1 or 2 bits) motion vector. So for motion-compensation if CPU is fast and RAM (bus) is slow it is possible to compute sub-shifted position of block inside CPU (caches) instead of fetching it from slow host RAM (as designed in other branch of mvtools).

"What exactly is the purpose of this approach?"

It is both for performance and logic of motion search:

1. For performance - for 2x upsample we need to compute only +3 new sub-shifted planes and can copy source plane as 4th sub-part.
2. For motion vector search - the zero-motion and MV (0,0) mean src block equal to ref block and the SAD of src block and not_sub_shifted ref block is minimal (or = 0). So when MAnalyse make search using blocks dissimilarity metric - it is very natural to compare 1x src block with 1x ref block and get lowest SAD (or other dissimilarity metric) and produce output MV of (0,0). Also this cause the lowest possible interpolation error for zero MVs for any kind of sources (with more or less good interpolate-able). Any interpolation for natural (noised too) sources adds some error and it cause non-zero MVs to have less quality (also may have more errors). Also that (may) mean the error distribution for zero-MVs and non-zero may be not equal. I.e. probability of bad zero-MV is lower in comparison with the probability of bad non-zero MV. This also can be used somehow in the statistical averaging of MVs for denoising.

[Adub]

Thank you very much for the detailed response! It confirms some of my suspicions and expands my understanding of the approach.

Let me restate the method to make sure I properly understood your answer.

Specifically, there is definitely utility in both upsampling (2x or 4x) as well as sub-sampling (selecting every 2nd or 4th pixel) when conducting motion estimation. Upsampling provides more spatial resolution to conduct a search at the finest levels of a hierarchy, allowing for more accurate motion vectors. Sub-sampling provides two benefits: 1) improved performance by reducing the number of pixels considered and 2) the slight inaccuracies induced by using sub-sampling adds a small amount of "jitter" (for lack of a better term) to the motion vectors produced. This has a beneficial effect if motion vectors are averaged together, because less zero-motion vectors will exist, thereby smoothing out the resulting average.

Example from averaging three motion vectors together: without the sub-sample jitter, we might have (0 + 0 + 10) / 3 = 3.3333, but with the sub-sample jitter, we have something more like (1 + 2 + 10) = 5.3333.

I think that all makes sense to me.

With respect to the specifics of how MSuper currently creates sub-sample shifted 1x versions instead of a "normal" interleaved sub-sample - I agree, the current approach seems like a nightmare for cache locality on modern processors. Granted, this impact is reduced if one version is used more than the others, which seems to be the case. I'd be curious to see if modern processors can better handle sub-sampling with the "normal" interleaved sub-sample approach, particularly because you get the benefit of better cache locality and SIMD "gather" instructions now exist. That locality might make up for the cost of the strided sub-sampling access and still come out faster.

So would it be safe to say that a more modern approach to MSuper might be to avoid the use of sub-sample shifted 1x versions and instead use a "normal" interleaved approach instead?

[DTL]

" there is definitely utility in both upsampling (2x or 4x) as well as sub-sampling (selecting every 2nd or 4th pixel) when conducting motion estimation."

It is better to name interpolation for getting a sub-sample level of precision. Yes - the MSuper performs interpolation-upsampling. But there is no sub-sampling (downscaling) in MAnalyse - it simply fetches the sub-shifted blocks (see GetRefBlock() function) from super-clip. SAD (and other dissimilarity) calculation functions always run with 1x sized current and ref blocks - the only differences at 0.5 and 0.25 levels precision is that ref blocks are sub-sample coordinates shifted from original position in a frame.

For pel=2 MAnalyse simply runs +1 level of 0.5 precision search.
For pel=4 MAnalyse runs +2 levels of 0.5 and 0.25 precision search.

MRecalculate always runs only single level (really not full level search but refining only).

"Sub-sampling provides two benefits: 1) improved performance by reducing the number of pixels considered"

At MAnalyse it does not really reduce - it only allows to run 1x SAD calculation functions at any level (both reduced size initial levels and 1x and below 1x). And for motion-compensation it allows to fetch ready to use sub-shifted ref blocks instead of new calculation of shifted blocks from 1x sized plane. This makes more host RAM traffic but may be faster at slow CPUs (was faster with old CPUs). Currently we have runtime sub-sample shifting implementation in MDegrainN in other branch of 2.7.45 mvtools and it runs not any slower (it is required for pel=1 mode of MSuper when hardware ME used because hardware ME do not need interpolated clip for any pel precision).

For example with DX12-ME (or any other hardware or software ME engine) there is no sub-sample shifted/interpolated frames available (like in super-clip in mvtools) and for MVs with 0.5 or 0.25 precision consumer need to calculate the sub-shifted blocks again to do motion compensation.

"2) the slight inaccuracies induced by using sub-sampling adds a small amount of "jitter" "

Simple natural noise causes lots of random jitter in MVs (at least at the flat areas with low signal to noise ratio).

"I'd be curious to see if modern processors can better handle sub-sampling with the "normal" interleaved sub-sample approach, particularly because you get the benefit of better cache locality and SIMD "gather" instructions now exist."

There are some SIMD-accelerated exhaustive search examples (it make several MVs SAD scan at the single refine function call and selects MV with lowest SAD at the end) in second-branch of 2.7.45 mvtools - but it runs only with 1x (pel=1). But to make pel=2 or pel=4 precision it is possible simply upsize 2x or 4x input clip and run mvtools with pel=1 and make re-calculation of MVs to 1x size if needed. But the 2x or 4x upsizing also causes more host RAM traffic because of current AVS+ (and mvtools architecture). For best performance with current machines architectures it is better to do local area (or even frame) upsizing-interpolation in the CPU caches and make fine search inside CPU only without keeping an upsized-interpolated frame in main host RAM. It is the same way hardware ME engines run - they get only 1x current and ref frames and do all required sub-sample precision interpolation inside hardware units and output fine 0.25 precision MVs.

"would it be safe to say that a more modern approach to MSuper might be to avoid the use of sub-sample shifted 1x versions and instead use a "normal" interleaved approach instead?"

It will not decrease required amount of RAM to store same sized interpolated super clip. But will make SIMD functions of search design somehow easier (also will cause search of 2x vs 2x upsized blocks at pel=2 and 4x vs 4x at pel=4). Though because current RAM bus is still very slow relative to cores performance it may not make big performance benefit in multithreading execution. When 4..8..16+ threads will frequently fetch large 2x or 4x interpolated frames from main host RAM via narrow and slow bus.
May be better to run with 1x super clip always (no pel=2 and pel=4 interpolated buffers at all) and make all required interpolation inside CPU core (better inside register file of enough size like AVX512).

[Adub]

Got it, thanks again for your detailed answer, DTL.

I agree with you - a modern approach seems to be handling all reduction and interpolation (what MVTools calls "refining" internally) inside the CPU cache, instead of via a Super clip. It might even be possible to eliminate the use of Super entirely, although it might have some benefits with padding.

I really like the idea of handling all reduction and interpolation in a "local area", just before motion search. This would significantly minimize memory usage (particularly for larger frame sizes like 4K YUV 4:4:4), keeping as much of the pertinent pixel data in the L1 cache as possible.

I might see if I can produce some toy code to benchmark a few different approaches, as a learning exercise.

[DTL]

To make performance of MAnalyse somehow better we have some more SIMD-friendly algoriphm - to make (refining) search of several blocks with single SIMD search process and it valid for many blocks because for static frame areas many nearby blocks have completely zero MVs (or very slightly distorted by noise) and for large coherently moving areas (also global motion/camera pans/dolly) the MVs of the several blocks are coherent. This search may be started is all predictors for a group ob blocks also coherent (or in some not very big offset - but it will cause SIMD src block loading more complex - may be permutex2var with AVX512 can help too ?).
This will significantly improves performance of small sized and high number blocks search cases. It is partially started in implementation in the
https://github.com/DTL2020/mvtools/blob/9eedb9d0850f638fc43212fb515cf048f9b9a58f/Sources/PlaneOfBlocks.cpp#L8910 (AVX2 also available for lower blocksize and search radius)

but it works only for pel=1 in that implementation because of the sub-shifted placement in current super-clip. Also AVX512 with larger register file will allow to search more blocks or of larger size or with larger search radius.

The design idea of that implementation - start multi-block search and continue until the MVs are (enough) coherent. If MVs lost coherency to too much range (for current multi-block search implementation in SIMD) - make fallback to single block search (with unlimited possible MV length).

[Selur]

btw. a Vapoursynth port would still be nice :)

[DTL]

Some new ideas how to make search performance for pel=2 and pel=4 precision better with AVX2/AVX512:


super_p4=MSuper(pel=4, hpad=8, vpad=8)

cl4x=LanczosResize(width*4, height*4, taps=16)
super_4x=MSuper(cl4x, pel=1, hpad=32, vpad=32)

#standard pel=4 search with 16 sub planes
mv_p4=MAnalyse(super_p4, blksize=8, search=3, searchparam=4, overlap=4)
mvrf_p4=MRecalculate(super_p4, mv_p4, thSAD=0, blksize=4, search=3, searchparam=16, overlap=2)

#upscaled single plane search
mv_4x=MAnalyse(super_4x, blksize=32, search=3, searchparam=4, overlap=16)
mvrf_4x=MRecalculate(super_4x, mv_4x, thSAD=0, blksize=16, search=3, searchparam=16, overlap=8)
mvrf_4x_p4=MScaleVect(mvrf_4x, scale=4, mode=1, adjustSubPel=true)



Searching with pel=1 looks more easy to make with instructions like mpsadbw and also loading from memory a single plane expected to use much less RAM read streams. With pel=4 and separated 16 sub-shifted planes each new search position requires reading a new block from the different sub-plane. With a single 4x sized plane all rows of the area to search may be loaded with a single read stream. So the number of large strided reads is about 16 times less for pel=4 and 4 times less for pel=2. For AVX512 with the possibility to load 64 of 8bits samples in a register it is expected to be possible to make a function of 16x16 block search inside a register file with radius about 8. This will emulate 8x8 block search with pel=2. Or 4x4 block with pel=4. Also for refining with MRecalculate typically required blocksize/2 and it is the slow operation now and most wanted to be optimized with SIMD.

Performance tests also show even with standard esa search without additional SIMD optimizations some performance boost with the 4x upsized version. Also the quality of MVs is different. It is still subject to research why quality changed. Maybe some processing at pel >1 levels is not performed (or performed in a different way) and this causes quality change.

[real.finder]

Also the quality of MVs is different. It is still subject to research why quality changed. Maybe some processing at pel >1 levels is not performed (or performed in a different way) and this causes quality change.

maybe you should use left-top/top-left aligned, in mvtools2 manual

pelclip clip =
Optional upsampled source clip for using instead of internal subpixel interpolation (for pel>1).
Pixels at rows and colunms positions multiple to pel (0,2,4,... for pel=2) (without padding) must be original source pixels, other pixels must be interpolated.

Example for pel=2: LanczosResize(width*2,height*2,src_left=0.25, src_top=0.25).
so in your case
cl4x=LanczosResize(width*4, height*4, 0.375, 0.375 (https://forum.doom9.org/showthread.php?t=147285), taps=16)

[DTL]

Yes - very many things may cause more or less significant changes in MVs at the MAnalyse output. The simple LanczosResize is just a first attempt to try. The internal MSuper interpolation is separate from all other AVS+ resizers kernel (Wiener ?) and it may need some research on how external resizers can emulate it with best possible precision (in both kernel shape and position/offset).

Other possible solutions for testing:
1. Export kernel from MSuper and add a new named resizer to AVS+ core (or external filter like ResampleMT).
2. Make some script based or compiled filter more or less slow to repack sub-sample shifted parts of upsampled plane from usual MSuper output into single frame/plane and feed it again at the MSuper input to create lower sized levels.

Though it is only important if users want to attempt to simulate standard output of MAnalyse/MRecalculate as best as possible. I am still not sure if it is completely possible because of possible other changes in the processing of sub-1-sized levels (pel-leveled).

Also there expected some difference in the SIMD implementation of the esa search (may be added in the C and other search methods too): The currently implemented (all ?) stepping searches (refining small algorithms) are made with checking of penalty at each step. This causes anisotropic results (the first MVs in searching area scan order are not examined with the same conditions as last in the search area scan). This causes result dependency from scan order of all positions. This may or may not be good for some search methods. At least this is not a single way of making esa search with penalty enabled (pNew > 0). Another way of applying the penalty is to check all positions with a given searching area first and apply the penalty only to the best (lowest SAD/dissimilarity metric) MV. This may be expected from documentation algorithm description but real implementations are not completely this way designed. The SIMD massive-multithreading implementation may get more or less significant performance penalty if making application of the penalty in that way (checking SAD one by one and checking condition at each step). One of SIMD implementations uses hardware instruction to get lowest value from several inputs and it is faster (warps many comparison operations in one) but this does not allow to completely simulate the stepping search with pNew at each step. This means faster SIMD-esa search will produce also a bit different results in comparison with C-esa implementation (with single SAD computing optimization with SIMD as designed today in pinterf branch). Complete SIMD simulation of currently implemented C-esa search is also possible but will have worse performance.

Natural anisotropy esa scan order + penalty > 0 at each step can cause missing of better MVs:
Scan order from left to right:
https://github.com/pinterf/mvtools/blob/a488b095c4bdc8d81abfd952ab534c015a9d45b7/Sources/PlaneOfBlocks.cpp#L1327
So this search causes some skew to the fist scanned positions if even they produce worse SAD.

Also current esa search is really expanding and will skip better MVs with larger length from starting position with pNew>0. Depending on the scene motion this may or may not help to the quality. This expects the probability of the fast motion may be lower (more likely caused by noise distortion - with any noisy frames sequence best SAD may also be created by noise difference between current and ref frame but it will not be the real best MV).

Possible partial workaround to break spatial anisotropy is to apply pNew only to the end of each round of the expanding search.
So there may exist at least 3 different versions of esa search:
1. Old current version of the expanding search with pNew check at each step (with worst spatial anisotropy).
2. Updated version of expanding search with pNew check after each full round (or even double check: 2.1 - after sides of a square and 2.2 - after corners of a square) - with spatial skew removed.
3. Full esa search with pNew checked once at the end (most SIMD friendly and of the highest performance).

"LanczosResize(width*2,height*2,src_left=0.25, src_top=0.25)."

With this correction total search difference with internal pel>1 looks like become lower. Though Lanczos kernel anyway different from internal shifter/upsampler and some difference expected to be anyway.

Some new test build https://github.com/DTL2020/mvtools/releases/tag/post_2.7.46_808 based on r808 by pinterf with added search=8 'more isotrorpic' exhaustive search. Also replaced default esa (old search=3) search for coarse levels to this search method. pnew now applied once after full border of expanding search is checked (but MotionDistortion penalty still applied at each search position - to adjist it user need to adjust lambda param). New ExpandingSearch2 function is https://github.com/DTL2020/mvtools/blob/b9b7a241c3d28c0dfe89b0dc69f019a4e51817b1/Sources/PlaneOfBlocks.cpp#L1350

[tormento]

I was looking at the Super Resolution (http://www.infognition.com/super_resolution_avisynth/) plugin.

Unfortunately, source hasn't been released and the development seems to have stopped.

The interesting part is not, for me, the resize part but the high frequency recovering from the temporal part of a video.

It states: Internally this plugin uses quarter-pixel motion compensation and a fusion process to combine motion-compensated previously upsized frames with spatially resampled current frame.

Would, thus, be possible to recreate the frequency recovery part by a MVTools script?

[DTL]

In the good conditioned against aliasing and ringing content any frame from a scene (sequence) has the very same view of a texture. It may be shifted relative to sampling grid with camera of object motion. But interpolated (upscaled) view is about the same (if no aliasing present). It is one of the main requirement for high quality digital moving pictures - no temporal aliasing. This will keep objects views undistorded at any speed of motion. No additional details can be gathered from some frames. If it is possible it mostly means there is some aliasing present and the view of an object is not stable at the motion (so temporal aliasing present and this makes fine details flicker and other temporal defects).

The standard way with noised footage is simply denoise and pass to the 'super-resolution' engines like AI/NN with an attempt to regenerate more detailed textures from local texture libraries from the engine. The texture library _may_ be updated from different footage frames. For example if some scene has a close up shot of a talent it may record high resolution face texture and apply it to blurry frames or with small size of the head and same for any other objects from total footage. But this is much more complex processing in comparison with current mvtools abilities. And this needs to analyse lots of frames, not only a few surrounding about equal frames.

[tormento]

No additional details can be gathered from some frames.
I was thinking about anime, where usually a single frame is repeated at least twice for characters, while background sometimes reaches real fps. It would be nice to recover additional information to recreate missing frequencies and/or recover blocking from starved sources.

[DTL]

If some block is static in a sequence of frames it does not carry additional (new) information in a sequence of frames. Averaging of clean static blocks with its copies does not change output. But with real noised frames and very low contrast texture (texture contrast below or about equal to the noise contrast) MAnalyse can give many false MVs (non-zeroed) for really static blocks and it will cause bad blending at denoise stage and additionally blurs the textures. It is not an error of the algorithm but because current block + noise distortion may better match other block + noise distortion.
Possible solutions in old mvtools - increase pzero (need manual adjustment and quality check for each footage with its textures contrast and noise levels) and other 'truemotion' parameters like lambda. This is expected to decrease probability of single false non-zero MV generation for each block (pzero) and skip possible false non-zero MVs predictors from surrounding blocks (lambda). Though too much pzero and lambda values may cause distortion at real moving areas of the frame.

[tormento]

Have a look at this RIFE-MVTool crossover (https://github.com/n00mkrad/VapourSynth-RIFE-ncnn-Vulkan/blob/master/RIFE/docs/readme.md).