SVPflow motion interpolation (Avisynth + Vapoursynth, Linux, Mac?)

[mastrboy]

what is the performance penalty of using SVConvert?

[chainik_svp]

I'm sure it's about zero

But the important thing is: SVAnalyse vectors can point outside of frame while MAnalyse ones can not (at least with zero MSuper padding), so I have to clip all such vectors.

[-Vit-]

Thanks for the SVConvert functionality. Are the forward vectors already shifted back one frame to match the forward vectors we would expect from MVAnalyse?

A couple of other things I noticed:
- The vectors returned seem to be slightly non-deterministic when using SetMTMode(2) and many threads. Is that a known or a concern?
- The default for SVSuper scale.down is 2 not 4 as stated in the docs (I know to check now... )

[chainik_svp]

Originally Posted by -Vit-

The vectors returned seem to be slightly non-deterministic when using SetMTMode(2)

It can't be. What's the source filter? The one and only non-deterministic behavior that I know was caused by DirectShowSource().

The default for SVSuper scale.down is 2 not 4 as stated in the docs (I know to check now... )

Thanks. I think I should exchange "2" for "rfilter" with "4" for "sharp" in sources...

Are the forward vectors already shifted back one frame

I hope so

[-Vit-]

I have been using DGDecode_mpeg2source. I tried a number of .d2v sources, several (but not all) demonstrated slight non-determinism on many frames using the scripts below.

I've been testing TGMC adapted to use SVPflow. TGMC blends motion compensated neighbor frames with the current to remove bob-shimmer and enhance detail. This requires accurate vectors or it produces blurring and ghosting. Initial tests show that the svpflow vectors are fine much of the time. But they are rather less accurate than the TGMC MVTools settings in a number of situations and produce more ghosting/blurring overall. However, I've only experimented briefly, using the default settings mostly, so I hope I can find better tailored settings. I also want to be sure I'm doing things correctly.

I'd particularly like to check how I implemented delta for SVAnalyse. The script uses vectors of delta 1 and 2. To get delta 2 I did this:

vecs2e = super.SelectEven().SVAnalyse( svAnalyse )
vecs2o = super.SelectOdd(). SVAnalyse( svAnalyse )
fVec2 = Interleave(vecs2e.SVConvert( isb=false ), vecs2o.SVConvert( isb=false ))
bVec2 = Interleave(vecs2e.SVConvert( isb=true ),  vecs2o.SVConvert( isb=true  ))

The scripts in full. Outer script was standard (script is deterministic if SVP=false):

SetMTMode(3,8)
DGDecode_mpeg2source("some.d2v")
SetMTMode(2)
TGMC_SVP_Test(SVP=true)
Distributor() # or not

Main script function. The outer code is TGMC. The motion code is highlighted in the middle - it's written to make settings adjustments and comparisons simple:

function TGMC_SVP_Test( clip Input, bool "SVP" )
{
	SVP = default( SVP, true )
	
	Prefilter  = 2    # Prefilter for search clip: 0 = none, 1 = Gauss blur + slight back blend, 2 = Full TGMC pre-filter
	Sharpness  = 1.5  # TGMC range not QTGMC range
	EdiThreads = 1

	w = Input.Width()
	h = Input.Height()
	epsilon = 0.0001

	bobbed = Input.bob(0,0.5)

	ts1       = bobbed.TemporalSoften( 1, 255,255, 28, 2 )
	ts2       = bobbed.TemporalSoften( 2, 255,255, 28, 2 )
	binomial0 = ts1.Merge( ts2, 0.357 ).Merge( bobbed, 0.125 )

	repair0 = binomial0.DeshimmerRepair( bobbed )

	spatialBlur = Prefilter > 0 ? repair0.RemoveGrain( 12,12 ).GaussResize( w,h, 0,0, w+epsilon,h+epsilon, p=2 ).MergeLuma( repair0, 0.1 ) : repair0
	tweaked     = Prefilter > 1 ? mt_lutxy( repair0, bobbed, "x 3 + y < x 3 + x 3 - y > x 3 - y ? ?", U=3,V=3 ) : NOP()
	srchClip    = Prefilter > 1 ? spatialBlur.mt_lutxy( tweaked, "x 7 + y < x 2 + x 7 - y > x 2 - x 51 * y 49 * + 100 / ? ?", U=3,V=3 ) : spatialBlur

	
	#----------MOTION ANALYSIS - MVTools and/or SVPflow-------------

	# Initial test - have only matched the TGMC overlap to the SVPflow setting, all other settings are the defaults for TGMC or SVPflow
	
	# SVPflow settings for SVSuper and SVAnalyse
	svSuper   = "{ gpu:0, pel:2, scale:{up:2, down:2} }"
	svBlock   = "w:16, overlap:2"
	svSearch  = "type:4, distance:-4, satd:false, coarse:{type:4, distance:-10, satd:true}"
	svPenalty = "lambda:10.0, plevel:1.5, lsad:8000, pnew:50, pglobal:50, pzero:100, pnbour:50, prev:0"
	svAnalyse = "{ block:{" + svBlock + "}, main:{ search:{" + svSearch + "}, penalty:{" + svPenalty + "}}}"

	# MVTools settings for MVSuper and MVAnalyse
	mvSuper   = "pel=2, hpad=8, vpad=8"
	mvBlock   = "blksize=16, overlap=4"
	mvSearch  = "search=4, searchparam=2, pelsearch=2"
	mvPenalty = "truemotion=false, lambda=100*(16*16)/(8*8), lsad=400, pnew=25, plevel=0, global=true, DCT=0"    # (16*16)/(8*8) is (blksize/8)^2
	mvAnalyse = mvBlock + ", " + mvSearch + ", " + mvPenalty
	
	# Calculate forward and backward vectors from search clip using either MVTools2 or SVPflow
	srchSuper = SVP ? srchClip.SVSuper( svSuper ) : Eval("srchClip.MSuper(" + mvSuper + ")")
	vecs1  = SVP ? srchSuper.SVAnalyse( svAnalyse ) : NOP()
	vecs2e = SVP ? srchSuper.SelectEven().SVAnalyse( svAnalyse ) : NOP()
	vecs2o = SVP ? srchSuper.SelectOdd(). SVAnalyse( svAnalyse ) : NOP()
	fVec1 = SVP ? vecs1.SVConvert( isb=false ) : Eval("srchSuper.MAnalyse( isb=false, delta=1, " + mvAnalyse + ")")
	bVec1 = SVP ? vecs1.SVConvert( isb=true  ) : Eval("srchSuper.MAnalyse( isb=true,  delta=1, " + mvAnalyse + ")")
	fVec2 = SVP ? Interleave(vecs2e.SVConvert( isb=false ), vecs2o.SVConvert( isb=false )) : Eval("srchSuper.MAnalyse( isb=false, delta=2, " + mvAnalyse + ")")
	bVec2 = SVP ? Interleave(vecs2e.SVConvert( isb=true  ), vecs2o.SVConvert( isb=true  )) : Eval("srchSuper.MAnalyse( isb=true,  delta=2, " + mvAnalyse + ")")

	pad = SVP ? 0 : 8 # Padding requirement for SVP
	
	#-----------------------------------------------------------

	
	edi = Input.NNEDI3( field=-2, nsize=1, nns=1, qual=1, threads=EdiThreads, U=true,V=true )

	ediSuper = edi.MSuper( pel=2, levels=1, hpad=pad, vpad=pad )
	bComp1   = edi.MCompensate( ediSuper, bVec1, thSCD1=180,thSCD2=98 )
	fComp1   = edi.MCompensate( ediSuper, fVec1, thSCD1=180,thSCD2=98 )
	tMax     = edi.mt_logic( fComp1, "max", U=3,V=3 ).mt_logic( bComp1, "max", U=3,V=3 )
	tMin     = edi.mt_logic( fComp1, "min", U=3,V=3 ).mt_logic( bComp1, "min", U=3,V=3 )

	degrain1  = edi.MDegrain1( ediSuper, bVec1,fVec1, thSAD=10*(8*8), thSCD1=180,thSCD2=98 )
	degrain2  = edi.MDegrain1( ediSuper, bVec2,fVec2, thSAD=10*(8*8), thSCD1=180,thSCD2=98 )
	binomial1 = degrain1.Merge( degrain2, 0.2 ).Merge( edi, 0.0625 )

	vresharp = Merge( binomial1.mt_expand( mode="vertical", U=3,V=3 ), binomial1.mt_inpand( mode="vertical", U=3,V=3 ) )
	resharp  = binomial1.mt_lutxy( vresharp.RemoveGrain(12), "x x y - "+ string(Sharpness) + " * +", U=3,V=3 )

	backBlend1 = resharp.mt_makediff( mt_makediff( resharp, binomial1, U=1,V=1 ).RemoveGrain( 12, -1 ).GaussResize( w,h, 0,0, w+epsilon,h+epsilon, p=5 ), U=2,V=2 )

	sharpLimit1 = backBlend1.mt_clamp( tMax,tMin, 0,0, U=3,V=3 )

	stableSuper = sharpLimit1.MSuper( pel=2, levels=1, hpad=pad, vpad=pad )
	stable      = sharpLimit1.MDegrain1( stableSuper, bVec1,fVec1, thSAD=4*(8*8), thSCD1=180,thSCD2=98 )

	repair2 = stable.DeshimmerRepair( edi )

	return repair2	
}
	
function DeshimmerRepair( clip Input, clip Ref )
{
	bobdiff   = mt_makediff( Ref, Input, U=3,V=3 )
	choke1    = bobdiff.mt_inpand( mode="vertical", U=3,V=3 ).mt_inpand( mode="vertical", U=3,V=3 ).mt_deflate( U=3,V=3 )
	choke1    = choke1.mt_expand( mode="vertical", U=3,V=3 ).mt_expand( mode="vertical", U=3,V=3 )
	choke2    = bobdiff.mt_expand( mode="vertical", U=3,V=3 ).mt_expand( mode="vertical", U=3,V=3 ).mt_inflate( U=3,V=3 )
	choke2    = choke2.mt_inpand( mode="vertical", U=3,V=3 ).mt_inpand( mode="vertical", U=3,V=3 )
	bobrepair = bobdiff.mt_lutxy( choke1, "x 129 < x y 128 < 128 y ? ?", U=3,V=3 ).mt_lutxy( choke2, "x 127 > x y 128 > 128 y ? ?", U=3,V=3 )
	return      Input.mt_adddiff( bobrepair, U=3,V=3 )
}

[Andrey /MAG/]

Originally Posted by -Vit-

I tried a number of .d2v sources, several (but not all) demonstrated slight non-determinism

Do you ever see such non-determinism when using MAnalyse function?
I sow different results earler on MAnayse. And I can't understand what the cause of results difference from run to run.

I have checked source codes. And no "random" function there.

[-Vit-]

The script I provided above uses MAnalyse if SVP=false and SVAnalyse if SVP=true. It's always deterministic with MAnalyse and with single-threaded SVAnalyse. However, with multi-threaded SVAnalyse (SetMTMode) it is fairly frequently non-deterministic on some sources. So the issue is with threading: thread-unsafe code, a dependency on the order of frame processing, or something similar...

Edit: <snip> Removed a mistaken crash report - I'd missed out an MSuper. Must remember that a super clip from SVSuper can't be used with MVTools functions. The TGMC script does that properly though.

[chainik_svp]

non-deterministic

Found it, it's caused by error in reverse vectors handling.
Also found bug in svpflow2 that could lead to crash in MT mode

[chainik_svp]

OK, ver.1.0.3

= fixed random crash in MT mode
= fixed usage of invalid predictors from reverse vectors

[Bernardd]

Hello,
With ver 1.0.3, in CPU mode, i have not the same output (very small difference) with refine: [ { thsad: 200 } ] or refine: [ ]. It is like thsad = 200 is not default value for SV Analyse refine params_string.
I do not understand why.
Thank.
Bernard

[chainik_svp]

Bernardd
What difference did you expect? I believe it works as described.

[Bernardd]

Hi Chainik

I want to write a script for SVSmooth with only the usefull params_string for my clip (restoring old Super 8 films). It seem for me that Refine params_string tunning is not necessary. Perhaps is wrong. For example i have changed refine: thsad value without observe frame picture change.

So in my script i have deleted all Refine params_strings with default values. This line " refine: [ { thsad: 200, search: { type: 4, distance: 4, satd: false }, penalty: { pnew: 50 } } ]" has been deleted. I expect no frame picture change, but i have a little change.

I hope you understand.

Thank

Bernard

[chainik_svp]

I hope you understand.

I'm trying So you think that "refine: []" should produce the same output as "refine: [ {thsad:200} ]", right?
This's wrong cause "[]" is just an empty array, nothing to do here. For the refinement there should be at least one element in the array, like this: "[ {} ]".

[Bernardd]

Ok, for this,
but i want to say that the not written line "refine: [{.............}]" (all line is deleted) do not produce the same output as "refine: [ {thsad:200} ]".

[Yellow_]

hi, any advice on parameters to minimize shape shifting Preditor style artifacts in movement, simple normal people movement. As a rough start point I tried defaults :-) and then this, I'm starting with 24p source to 120fps and I'm not expecting miracles just minimise movement artifacts however long it takes to analyse and encode. No doubt I'm using the functions badly:

super_params="{pel:1}"
analyse_params="{ algo: 2, block:{w:8} }"

super = SVSuper(super_params)
vectors = SVAnalyse(super, analyse_params)

forward_mv = SVConvert(vectors, false)
backward_mv = SVConvert(vectors, true)

super_mv = MSuper(pel=1, hpad=0, vpad=0) #padding should be zero here!
MFlowFps(super_mv, backward_mv, forward_mv, num=120, den=1)

Many thanks

[chainik_svp]

Yellow_

Why MFlowFps? It has no advantages over SVSmoothFps.
The main weapon to kill that arts around moving objects is cover/uncover masking (called "occlusion mask" in MVTools documentation) but this mask should be much more correct in SVPflow. So try to use SVSmoothFps with "algo:21", GPU acceleration and bicubic vectors/masks interpolation turned on ("cubic:1") and play with "mask.cover" value.

[Yellow_]

Thanks for the reply, unfortunatley I'm using this on Linux via Wine so haven't been able to get GPU stuff working even if my NV card is supported. My problem. :-)

So I'm here:

super = SVSuper("{pel:1}")
vectors = SVAnalyse(super, "{ algo: 21, block:{w:8} }")

SVSmoothFps(super, vectors, "{ num:120, den:1, algo:21, cubic:0 }", url="www.svp-team.com")

Looking at mask stuff now. Cheers.

**EDIT**

Vast improvement over first attempt. :-)

Added a mask cover of 50 to start, what I'm seeing is a jagged edge trailing movement, ie: outline of where the moving object was previous frame.

[chainik_svp]

Originally Posted by Yellow_

haven't been able to get GPU stuff working

Have you seen that?

BTW I've read that Wine now supports OpenCL calls but I'm too lazy to check it

---------
With smaller blocks - "{block:{w:8}, refine: [{thsad:-1}] }" - mask should be more accurate.

[Yellow_]

Cheers, I'll look into getting GPU stuff going via Wine. I get overlap error due to CPU frame comp with thsad:-1 :-(

[Mug Funky]

just dropping by to say thanks to all involved in the SVP project.

my canon workflow owes a lot to it

what i'm doing:

- shoot at as high a framerate as i can manage (720p60), at as high a shutterspeed as i can get a good exposure on (i like to shoot wide open, so it's no problem to get 1/250 or so). also, high fps means less rolling-shutter effect than if i'd shot at my target frame rate. less jelly = better.

- depanstabilize. goes without saying if i'm not willing to give up coffee.

- SVPfps to my output framerate*samples. samples working in the same way as in a graphics package.

- temporalsoften to blend only frames that correspond to my desired shutter angle wrt the output frame rate

- selectevery(samples, 0)

i'm able to output 2400/1001 with motion blur as if it was shot that way at 180 degrees

the interpolation quality is such that when it fails, i don't see weird things. usually just a little ghosting.

me happy