SVPflow motion interpolation (Avisynth + Vapoursynth, Linux, Mac?)

[chainik_svp]

Short version: SVPflow is a successor to MVTools2, providing more speed and more quality. Now available for Windows/Linux/Mac in both 32 and 64-bit versions.

Long story
With all respect to Fizick MVTools is old and there’s almost no progress (except for bug fixes) for several years now. Obviously MVTools isn’t perfect so someone should do something with it And there comes another problem: MVTools is big and fat. There’re tons of dead, commented and copy-pasted code in it. For example all “client” functions (MFlow, MFlowFps, MBlockFps, …) are identical in at least 50% of code which is bad for further development.
And there’s SVP project that started in 2008 on biggest Russian IT forum (iXBT.com). Main goal was to provide some user-friendly controls over all that MVTools-based scripts inside ffdshow for real-time smooth video playback. They’ve experimented with scripts, tuned GUI and so on.
At some time I’ve noticed that MVTools performance is too low on AMD CPU’s, so I’ve made a patch that increased MFlow / MFlowFps speeds up to 50% on all Athlons/Phenoms. That was the beginning – SVP 2.0
The second step was creating MSmoothFps function that incorporates all “client” MVTools function in one with GPU accelerated rendering via OpenCL and some more stuff and it was SVP 3.0. The output of this function was per-pixel the same as output of original MFlowXXX. In latter versions we’ve made some speed optimizations (20-50%) in motion vectors search.
But all of that was only a warming-up before the real job – to give MVTools a second life.


So, SVPflow consists of two parts: GPLed motion vectors search and closed-source frame rendering.

Downloads.
The description and downloads are available there.
Also many of you’re familiar with Interframe script

Disclaimer.
SVPflow isn’t a complete replacement for MVTools cause it’s missing most of additional functions. As of now the main goal of project is to provide fast and high quality real-time video conversion, so all functions unnecessary for this task was cut off and sources were cleared and refactored.

Also there’re a lot of new features added and I could tell about them if someone is interested in

----------
Version 4.0.0.128 (02/22/16)

= bump versions to match SVP 4 multi-OS release
+ Vaporsynth/Windows 32-bit/64-bit builds
+ Vaporsynth/Linux 32-bit/64-bit builds
+ all filters now compatible with MT mode 1

Version 1.1.16 (07/08/15)

= fixed random crash in svpflow1.dll found by SVP 3.1.7 users

Version 1.1.15a (05/15/15)

+ 64-bit build added

Version 1.1.15 (05/13/15)

= fixed performance issues with the default main.search.coarse.width value (introduced in 1.1.12)

Version 1.1.14 (04/24/15)

= up to 15% faster
= 2nd attempt to fix broken SVConvert function NOT fixed in 1.1.13

Version 1.1.13 (03/09/15)

= fixed broken SVConvert function

Version 1.1.12 (03/04/15)

= up to 20% faster
+ "64-bit ready" - assembler code (still not all of them) ported/rewritten in intrinsics, target compiler is VC++2013
+ new option in SVAnalyse - search.main.coarse.width - to get more than one "finest" levels
+ SATD for 32-x blocks
= reduced memory usage for large number of small blocks
= refactoring - removed svpflow_cpu/gpu modules - merged with svpflow2

Version 1.0.11 (05/04/14)

= Fixed crash in CPU rendering mode with pel>1 and blend=true
= Fixed rounding error in GPU rendering mode

Version 1.0.10 (04/01/14)

+ GPU rendering in linear light by default
= SAD functions updated to the latest x264 code

Version 1.0.9 (07/17/13)

+ SVSmoothFps switches to 13th SVP shader if scene quality is below scene.m1 value
= GPU rendering was very slow for some "prime" frame sizes (like 1916x1076) on some video cards
- removed workaround for a bug in early IvyBridge's drivers

Version 1.0.8 (02/03/13)

= fixed regression from 1.0.5 (low rendering precision in GPU mode)
= improved lighting quality (reduced color banding)

Version 1.0.7 (01/23/13)

+ improved scene change detection
= fixed crash after SVConvert, adjusted SAD values in MVTools' vectors field

Version 1.0.6 (12/23/12)

+ super clip size reduced by 4 for pel=1
('full' param in SVSuper and 'src' param in SVAnalyse)
= added workaround for OpenCL rendering on Intel's IvyBridge GPUs
+ "Ambilight" can now produce "glow" effect from all sides of the frame
('light.zoom' param in SVSmoothFps)
= several small bugs fixed:
== crash with frame rate ratio < 2 (50->60, 60->50 and so on)
== arts near screen edges with CPU rendering and some frame / block sizes combination
== green lines with 21th shader and arts masking on CPU

Version 1.0.5 (05/03/12)

= added compatibility with GTX680 (and may be with all other Kepler GPUs) - thanks to flagger

Version 1.0.4 (05/02/12)

+ added special.delta param

Version 1.0.3 (04/16/12)

= fixed random crash in MT mode
= fixed usage of invalid predictors from reverse vectors (thanks -Vit-)

Version 1.0.2 (04/04/12)

= support for multiply SVSmoothFps instances needed for proper 3D handling
+ SVConvert function for integration with MVTools-based scripts
+ GPU selection for rendering
= updated SAD/SATD x264 code

[aegisofrime]

Hmm I wonder if Q/TGMC could be updated to make use of SVPflow? Or are the functions used by Q/TGMC not present in SVPflow yet?

[Zerofool]

Originally Posted by chainik_svp

Also there’re a lot of new features added and I could tell about them if someone is interested in

Yes, please do.
And also, if I may ask you to consider implementing support for YV24 colorspace in a future version, that would be great.
Thank you for your work!

[chainik_svp]

aegisofrime
No it couldn't. It uses MDegrain, MCompensate and MFlowBlur.
All I could do is providing some kind of converter from SVAnalyse vectors to MAnalyse output format o_O

Zerofool
consider implementing support for YV24 colorspace in a future version

There's no real visual difference in real-time between YV12 and YUY2, but YUY2 processing is 30% slower.
That's the reason why SVPflow supports only YV12 now.
YV24 will be 2 times slower.
So why?

[-Vit-]

Originally Posted by chainik_svp

All I could do is providing some kind of converter from SVAnalyse vectors to MAnalyse output format o_O

Even that would be very interesting and potentially provide quality improvements to QTGMC. Are there any major differences in format?

[chainik_svp]

Originally Posted by -Vit-

Are there any major differences in format?

Nope, just forward and backward vectors together plus additional integer for each block (it's average luma value).

[lansing]

so so far SVPflow can't be used for denoising?

[-Vit-]

My edit to my last post crossed with your reply so you may have missed it, I had some questions about the new penalty settings:

What does pnbour do exactly? It seems related to field coherence, but I can't quite tell from the docs.

How exactly does penalty.prev work - does it assume that forward vectors at frame 1 (from the frame 0 to frame 1) are expected to be the opposite of backward vectors at frame 1 (from frame 2 to frame 1)? Or does it have a temporal aspect, forward vectors at frame 1 are expected to be the reverse of backward vectors at frame 0 (I think you probably know what I mean), or is it something different again...
Also are there downsides to penalty.prev? Objects appearing or disappearing would strongly affect it, but that's a problem case already.

Thanks for you work BTW

[henryho_hk]

Originally Posted by chainik_svp

aegisofrime
All I could do is providing some kind of converter from SVAnalyse vectors to MAnalyse output format o_O

Yes, please.

And from your experience how promising is GPU based rendering? I am replacing my display card (ATI 4850). Now choosing between 5850 (if i could find it), 6850 and GTX 560SE (336 CUDA core).

[chainik_svp]

-Vit-

In MVTools there're up to 5 predictors for each block:
- zero vector (with pzero penalty)
- global (with pglobal penalty)
- interpolated from previous level (w/o any penalty)
- left, top, right (interpolated) and their median (with lambda penalty)

In SVPflow:
- zero (pzero)
- global (pglobal) (but global MV is average between forward global MV and backward global MV)
- from previous level (w/o penalty)
- from previous level in opposite direction (*) (if exists, with prev penalty)
- vectors of up to 8 already processed neighbours (**) (with pnbour penalty)

(*) Forward (from frame 0 to frame 1) and backward (from 1 to 0) vectors are finding at the same time, like this: "fwd on smallest level 5", "bwrd on smallest level 5", "fwd on level 4", "bwrd on level 4", ..., "fwd on finest level 0", "bwrd on finest level 0".
So when processing some block in "bwrd level 5" we can take already founded MV from "fwd level 5" that points to this block (if such MV exists) as a predictor.
And for block in "fwd level 4" we can take MV interpolated from "bwrd level 5" that points to this block.

(**)
In MVTools blocks are processed in strict order row by row. In SVPflow order of processing is defined by block's SAD values - blocks with less SAD are processing first. So for each block its already processed neighbours (if any) are always have better quality. BTW the next good idea is to sort blocks not by SAD but by its contrast value...

[chainik_svp]

henryho_hk
how promising is GPU based rendering?

1. It's fast

2. It provides much more precision cause all processing on GPU done in floats.
What's the point of hpel and qpel? First: you can find vectors with non-integer coordinates up to 0.5 or 0.25 of pixel. Second: it makes additional interpolated planes in super clip that are used by CPU rendering to fetch sub-pixels.
What if you need to render frame at the time moment = 0.2, for example? You'll need not the original MVs but forward MV/5 and backward MV*4/5. Obviously there're no exactly those sub-pixel with neither hpel nor qpel on CPU.
But GPU can easily fetch the exact sample you need, it's like "infinite" pel level Also with high precision rendering there's less need for high precision in vectors search.

3. It can do bicubic interpolation instead of bilinear (even on low-end cards) which is definitely visible with naked eye.

I dunno why but I can't upload any attachments images

--------
Upd: some images.

[Didée]

Originally Posted by chainik_svp

BTW the next good idea is to sort blocks not by SAD but by its contrast value...

Finally! That sounds like ... something.

When you bring "block contrast" into the game, consider making various SAD thresholds RELATIVE to block contrast. The current system with absolute values surely was easy to implement, but really, it is quite dumb. (An object is sold for $50. Is this a good price, or is it expensive? Well, it depends on the kind of object ... obviously. Now, is SAD=50 a good threshold, or not? Well, it depends...)

[chainik_svp]

Didée
Yeah, I know that it was your ideas about contrast

Right now block contrast is used for "adaptive" search distance - large distance for more contrast blocks, zero distance for uniform ones. Which is good for eliminating "window blinds" artifacts BTW.
And relative (or "adjusted") SAD is used in scene change detection, but it's adjusted not by contrast but by avg. luma.

[Terka]

It is great news! Thank you, chainik_svp!

[Bloax]

Originally Posted by chainik_svp

I dunno why but I can't upload any attachments images

http://imgur.com/

[chainik_svp]

Bloax
That's not exactly what I've looked for... but thanks

Some synthetic images with GPU rendering. Big blue thing with red edge moves on some background.
All images with pel=1!

[Wilbert]

So, SVPflow consists of two parts: GPLed motion vectors search and closed-source frame rendering.

Would you be willing the release the frame rendering under GPL too?

[henryho_hk]

Originally Posted by chainik_svp

henryho_hk
1. It's fast
some images.

No flame .... but NVidia or ATI better?

[gyth]

Originally Posted by chainik_svp

YV24 will be 2 times slower.

IMO it'd be like subpixel estimation, but with real data instead of interpolated.

Zerofool, you could do something like this. I'm sure there are some gotchas, like chroma placement.

w=width
h=heigth
resize (2 * w, 2 * h)
converttoYV12

#motion stuff

converttoYV24
resize(w, h)

[chainik_svp]

Originally Posted by Wilbert

Would you be willing the release the frame rendering under GPL too?

No, but "never say never" (c)

Originally Posted by henryho_hk

NVidia or ATI better?

As of now - the same s..t
But on NV you can do real HW deinterlacing with LAV @CUDA before motion interpolation.