Motion Compensated FFT3DGPU

[tormento]

I am trying to find a good and fast filter for general usage.

Lurking around, I found this:

bl=8
ov=4
sh=2
vf=noisy.mvanalyse(pel=2,blksize=bl,isb=false,idx=1,overlap=ov,sharp=sh,truemotion=true)
vb=noisy.mvanalyse(pel=2,blksize=bl,isb=true,idx=1,overlap=ov,sharp=sh,truemotion=true)
vf2=noisy.mvanalyse(pel=2,blksize=bl,isb=false,idx=1,delta=2,overlap=ov,sharp=sh,truemotion=true)
interleave(\
mvcompensate(noisy,vf2,idx=1,thSCD1=600)\
, mvcompensate(noisy,vf,idx=1,thSCD1=600)\
, noisy\
, mvcompensate(noisy,vb,idx=1,thSCD1=600))
FFT3DFilter(sigma=4, bt=4, bw=16, bh=16, ow=8, oh=8)
denoised=selectevery(4,2)

It uses MVTools1 and FFT3D not GPU.

Do you think it could be worth to try to port it to MVTools2 and FFT3DGPU? Do you have some coding hints?

[onesloth]

AVISource("c:\test.avi") # or MPEG2Source, DirectShowSource, some previous filter, etc
super = MSuper()
backward_vectors = MAnalyse(super, isb = true)
forward_vectors = MAnalyse(super, isb = false)
forward_compensation = MFlow(super, forward_vectors, thSCD1=500) # or use MCompensate
backward_compensation = MFlow(super, backward_vectors, thSCD1=500)
# create interleaved 3 frames sequences
interleave(forward_compensation, last, backward_compensation)

FFT3DGPU(sigma=4, plane=4) # place your preferred temporal (spatial-temporal) denoiser here

selectevery(3,1) # return filtered central (not-compensated) frames only

This is the example from the MVTools docs with FFT3DGPU added.

The question is: Is there a way make FFT3DGPU process the interleaved clip in temporal blocks centered on the original frames only. I believe this is how dfttest(tmode=1) works. In this example FFT3DGPU denoises the compensated frames even though you are just going to throw them away. The final output would be the same either way, just faster without the unnecessary processing.

[um3k]

If I'm not mistaken, once you decimate the video after filtering, Avisynth is smart enough to only process the frames that remain.

[onesloth]

If I'm not mistaken, once you decimate the video after filtering, Avisynth is smart enough to only process the frames that remain.
Yep, that makes sense. The filter chain is evaluated in reverse.

[tormento]

The "problem" with dfttest is that it is not GPU aware ;)

Onesloth: could you provide some code from your thoughts?

[onesloth]

Sorry, no. I was mistaken about the functioning of the script I posted. As um3k pointed out, the code I showed works exactly as I said I wanted it to. It should suit your needs just fine.

[tormento]

I have tried your script and I get ~1.5 fps. Either there is something wrong or my 9800GT is not enough to provide the calculation.

With a simple fft3dgpu, I get ~ 11 fps when encoding 1080p to x264 -slow.

Any guinea pig but me?

[onesloth]

MAnalyse is a really slow function. It isn't GPU accelerated or anything so 1.5 fps could be entirely reasonable depending on your setup. What resolution is your source? What CPU do you have? Are you running AviSynth multithreaded? How many threads?

[tormento]

Source is 1080p, I7 920 @ 3.6GHz. fft3dgpu doesn't like MT as far as I know.

[onesloth]

With that CPU you should be able to do a lot better than 1.5 fps for two MAnalyse calls at 1080p. If you have AviSynthMT use this script:
SetMTMode(2,0)
AVISource("c:\test.avi") # or MPEG2Source, DirectShowSource, some previous filter, etc
super = MSuper()
backward_vectors = MAnalyse(super, isb = true)
forward_vectors = MAnalyse(super, isb = false)
forward_compensation = MFlow(super, forward_vectors, thSCD1=500) # or use MCompensate
backward_compensation = MFlow(super, backward_vectors, thSCD1=500)
# create interleaved 3 frames sequences
interleave(forward_compensation, last, backward_compensation)
SetMTMode(5)
FFT3DGPU(sigma=4, plane=4) # place your preferred temporal (spatial-temporal) denoiser here
SetMTMode(2)
selectevery(3,1) # return filtered central (not-compensated) frames only
FFT3DGPU works fine for me in SetMTMode(5) and the other filters all usually work fine in SetMTMode(2).

[Didée]

There's more "speed tricks" you can play to optimize MVTools + FFT3D/GPU.

The point is that you can use very fast settings for the MVTools part. Fact is that FFT3D & Co. don't *ulimately need* mocomp-assistance in the first place. They work quite good without that assistance, and also *with* that assistance, they benefit only very little in areas without motion, or where motion is very little. Where they *do* benefit from motion compensation is: a) areas where motion distance is rather large (when, say, bigger than 50% of the windowsize of the fft filter, and ultimately when it's bigger than the windowsize), and b) on scenechanges, especially one low-contrast scene switches to another low-contrast scene.

Long story short: to quickly assist FFT3DFilter or similar filters with MVTools, try e.g. "pel=1, (truemotion=false,) blksize=16, overlap=2 (or 4)". That still gives like 95% of the benefit that FFT3DFilter can only get from MVTools, but doesn't "waste" time on hi-Q MVTools settings that aren't really needed in this case.

Try it yourself, compare such "simple" MVTools settings with the usual more exhaustive settings, and judge the speed/quality ration for yourself. You might get surprised. ;)


(Edit: and really, do use MCompensate. Not MFlow.)

[Gavino]

FFT3DGPU works fine for me in SetMTMode(5) and the other filters all usually work fine in SetMTMode(2).
However, using SetMTMode(5) on the last filter in a script effectively makes the whole script single threaded, so multithreading has no effect (or even slows it down).

[tormento]

(Edit: and really, do use MCompensate. Not MFlow.)

I am a coding donkey. Please... :p

Oh and MVTools for me work much faster with MT(Thread=4) than SetMTMode.. Don't know why.

[onesloth]

(truemotion=false,)

I've read some of your thoughts on truemotion in other threads. With the parens, are you implying YMMV with the setting of this parameter?

[onesloth]

MVTools for me work much faster with MT(Thread=4) than SetMTMode..
It's not surprising that it works faster. But for motion compensation its not *better*. Rather worse. Using MT()'s overlap parameter may prevent artifacts at the slice boundaries or it may not, regardless of how high you set it.

[CruNcher]

With that CPU you should be able to do a lot better than 1.5 fps for two MAnalyse calls at 1080p. If you have AviSynthMT use this script:
SetMTMode(2,0)
AVISource("c:\test.avi") # or MPEG2Source, DirectShowSource, some previous filter, etc
super = MSuper()
backward_vectors = MAnalyse(super, isb = true)
forward_vectors = MAnalyse(super, isb = false)
forward_compensation = MFlow(super, forward_vectors, thSCD1=500) # or use MCompensate
backward_compensation = MFlow(super, backward_vectors, thSCD1=500)
# create interleaved 3 frames sequences
interleave(forward_compensation, last, backward_compensation)
SetMTMode(5)
FFT3DGPU(sigma=4, plane=4) # place your preferred temporal (spatial-temporal) denoiser here
SetMTMode(2)
selectevery(3,1) # return filtered central (not-compensated) frames only
FFT3DGPU works fine for me in SetMTMode(5) and the other filters all usually work fine in SetMTMode(2).

In my experience SetMTMode(6) should be used with FFT3DGPU everything else either results in no Multithreading like Gavino said or heavy visual bugs.

[Didée]

I've read some of your thoughts on truemotion in other threads. With the parens, are you implying YMMV with the setting of this parameter?
Yes. MVTools has a bazillion of knobs to turn, and for each single one, the story is "YMMV". TrueMotion is just another knob.

I'm not a too big fan of TrueMotion generally. There are scenarios where TM works better, but usually I like it more when deactivated. In particular for denoising. And even more so when big blocksizes are used.
(I'd prefer to have it defaulted to 'false', and activate it only when wished.)

But then, YMMV. :D


Oh and MVTools for me work much faster with MT(Thread=4) than SetMTMode.. Don't know why.
I don't know either. Quickly set up a script to test various candidates against each other:
(Sorry I used normal FFT3DFilter ... I'm still setting things up here, and FFT3DGPU isn't in, yet.) (tells me some darn DLL is missing?!?)

# 32bit Avisynth ....

# setmtmode(5,4) # ,8)

mpeg2source("PAL_720x576.d2v")

# setmtmode(2)

# MT("""

o = last
osup = o.msuper(pel=1)

fv2 = osup.manalyse(isb=false,delta=2,truemotion=false,blksize=16,overlap=2,search=4)
fv1 = osup.manalyse(isb=false,delta=1,truemotion=false,blksize=16,overlap=2,search=4)
bv1 = osup.manalyse(isb=true, delta=1,truemotion=false,blksize=16,overlap=2,search=4)

fc2 = o.MCompensate(osup,fv2)
fc1 = o.MCompensate(osup,fv1)
bc1 = o.MCompensate(osup,bv1)

interleave(fc2,fc1,o,bc1)
fft3dfilter(sigma=8,bw=16,bh=16,ow=8,oh=8,bt=4,plane=4)
selectevery(4,2)

# """,4,4)

return(last)

Results:
singlethread: 6:04 7.62 fps
==============================
MTmode(2,4): 2:15 20.56 fps
------------------------------
mt(4,16) : 2:59 15.51 fps
mt(4, 8) : 2:46 16.72 fps
mt(4, 4) : 2:31 18.38 fps
mt(4, 0) : 2:26 19.01 fps
==============================
MTmode(2,8): 1:51 25.01 fps
------------------------------
mt(8,16) : 2:51 16.23 fps
mt(8, 8) : 2:27 18.88 fps
mt(8, 4) : 2:15 20.56 fps
mt(8, 0) : 2:06 22.03 fps

Not only that MT() necessarily has lower quality with MVTools ... for me, worse, it's always slower than MTmode(2).

But of course, YMMV. :D

[Gavino]

In my experience SetMTMode(6) should be used with FFT3DGPU everything else either results in no Multithreading like Gavino said or heavy visual bugs.
The MT documentation is not very clear, but doesn't mode 6 behave just like mode 5 in this repect, so still no multithreading for any preceding filters?
However, using SetMTMode(5) on the last filter in a script effectively makes the whole script single threaded, so multithreading has no effect (or even slows it down).
To expand on this, note that if you have filters after FFT3DGPU, then you can usefully use SetMTMode(2) for them and get at least some multithreading.

[Didée]

This would be an "ideal" case to use ThreadRequest. Alas, I don't get it to work with MVTools.

[tormento]

I have found a little source to get a fftw.dll with cuda. Any guinea pig willing to compile? ;)

[Boulder]

The MT documentation is not very clear, but doesn't mode 6 behave just like mode 5 in this repect, so still no multithreading for any preceding filters?

To expand on this, note that if you have filters after FFT3DGPU, then you can usefully use SetMTMode(2) for them and get at least some multithreading.

According the docs in tsp's MT package, SetMTMode is applied to filters after the call. See the example from the document:
SetMTMode(2,0) #enables multihreading using thread = to the number of available processors and mode 2
LoadPlugin("...\LoadPluginEX.dll") #needed to load avisynth 2.0 plugins
LoadPlugin("...\DustV5.dll") #Loads Pixiedust
import("limitedsharpen.avs")
src=AVIsource("test.avi")
SetMTMode(5) #change the mode to 5 for the lines below
src=src.converttoyuy2().PixieDust()#Pixiedust needs mode 5 to function.
SetMTMode(2) #change the mode back to 2
src.LimitedSharpen() #because LimitedSharpen works well with mode 2
subtitle("Number of threads used: "+string(GetMTMode(true))+" Current MT Mode: "+string(GetMTMode())) #display mode and number of threads in use

[Gavino]

According the docs in tsp's MT package, SetMTMode is applied to filters after the call.
Yes, SetMTMode sets the mode to be used for the filters following. However, my understanding is that filters in modes 5 and 6 are run single-threaded. Therefore, only one thread at a time can be fetching a frame from such a filter and hence from any of its predecessors in the filter chain. In effect, those predecessors have to be single-threaded too.

I may be wrong on this, and I can't run any experiments to test it as I don't have a multi-core machine. I would be grateful for a confirmation, either from someone who really knows how it works internally or just with some real performance results.

[BigDid]

...
Results:
singlethread: 6:04 7.62 fps
==============================
MTmode(2,4): 2:15 20.56 fps
------------------------------
mt(4,16) : 2:59 15.51 fps
mt(4, 8) : 2:46 16.72 fps
mt(4, 4) : 2:31 18.38 fps
mt(4, 0) : 2:26 19.01 fps
==============================
MTmode(2,8): 1:51 25.01 fps
------------------------------
mt(8,16) : 2:51 16.23 fps
mt(8, 8) : 2:27 18.88 fps
mt(8, 4) : 2:15 20.56 fps
mt(8, 0) : 2:06 22.03 fps

Not only that MT() necessarily has lower quality with MVTools ... for me, worse, it's always slower than MTmode(2).


Hi Didée,

I was surprised of your tests cause for me MT is quicker than setMT.

So I took your script literally and results were comparable to yours. After a while I figured out the difference:
no setmemorymax at the beginning of the script = no MT optimizing!
My value is setmemorymax(192) for my 6400 quad-core. *

Incidently I could not find an optimum value of setmemory for SetMT; best shot being setmemorymax(320).

I dont have time atm to make real tests with averaging but if someone is willing to try, please do :)

Did

* Edit when using setmemorymax(192) for MT, the 2 SetMTmode lines are disabled:
# setmtmode(5,4) # ,8)
# setmtmode(2,4)

[Gavino]

filters in modes 5 and 6 are run single-threaded. Therefore, only one thread at a time can be fetching a frame from such a filter and hence from any of its predecessors in the filter chain. In effect, those predecessors have to be single-threaded too.
If what I said is true (and I'm still waiting for a confirmation or refutal), a solution is to call Distributor() before SetMTMode(5). This runs the preceding filters multithreaded in what is effectively an independent chain feeding the rest of the script.

[Boulder]

I did a quick test to verify things.

6.57 fps
SetMTMode(5)
AVISource("w:\nimetön 1.avi") # or MPEG2Source, DirectShowSource, some previous filter, etc
SetMTMode(2)
super = MSuper()
backward_vectors = MAnalyse(super, isb = true)
forward_vectors = MAnalyse(super, isb = false)
forward_compensation = MFlow(super, forward_vectors, thSCD1=500) # or use MCompensate
backward_compensation = MFlow(super, backward_vectors, thSCD1=500)
# create interleaved 3 frames sequences
interleave(forward_compensation, last, backward_compensation)
#Distributor()
SetMTMode(5)
FFT3DGPU(sigma=4, plane=4) # place your preferred temporal (spatial-temporal) denoiser here
SetMTMode(2)
selectevery(3,1) # return filtered central (not-compensated) frames only


6.53 fps
SetMTMode(5)
AVISource("w:\nimetön 1.avi") # or MPEG2Source, DirectShowSource, some previous filter, etc
SetMTMode(2)
super = MSuper()
backward_vectors = MAnalyse(super, isb = true)
forward_vectors = MAnalyse(super, isb = false)
forward_compensation = MFlow(super, forward_vectors, thSCD1=500) # or use MCompensate
backward_compensation = MFlow(super, backward_vectors, thSCD1=500)
# create interleaved 3 frames sequences
interleave(forward_compensation, last, backward_compensation)
Distributor()
SetMTMode(5)
FFT3DGPU(sigma=4, plane=4) # place your preferred temporal (spatial-temporal) denoiser here
SetMTMode(2)
selectevery(3,1) # return filtered central (not-compensated) frames only


6.33 fps
SetMTMode(5)
AVISource("w:\nimetön 1.avi") # or MPEG2Source, DirectShowSource, some previous filter, etc
SetMTMode(2)
super = MSuper()
backward_vectors = MAnalyse(super, isb = true)
forward_vectors = MAnalyse(super, isb = false)
forward_compensation = MFlow(super, forward_vectors, thSCD1=500) # or use MCompensate
backward_compensation = MFlow(super, backward_vectors, thSCD1=500)
# create interleaved 3 frames sequences
interleave(forward_compensation, last, backward_compensation)
Distributor()
SetMTMode(5)
FFT3DGPU(sigma=4, plane=4) # place your preferred temporal (spatial-temporal) denoiser here
#SetMTMode(2)
selectevery(3,1) # return filtered central (not-compensated) frames only

Based on these, the "regular" way is the fastest.

[onesloth]

However, using SetMTMode(5) on the last filter in a script effectively makes the whole script single threaded, so multithreading has no effect (or even slows it down).
It appears as though this is not the case.

I ran some tests. I'm using a Q6600 and 8800GT with JoshyD's AviSynthMT 2.58 64-bit and a 1920x1040 1084 frame source with these scripts:

No MT
AVISource("video.avi")
super = MSuper(pel=1)
backward_vectors = MAnalyse(super, isb = true, truemotion=false, blksize=16, overlap=4)
forward_vectors = MAnalyse(super, isb = false, truemotion=false, blksize=16, overlap=4)
forward_compensation = MCompensate(super, forward_vectors, thSCD1=500)
backward_compensation = MCompensate(super, backward_vectors, thSCD1=500)
interleave(forward_compensation, last, backward_compensation)
FFT3DGPU(sigma=4, plane=4)
selectevery(3,1)

MT
SetMTMode(2,0)
AVISource("video.avi")
super = MSuper(pel=1)
backward_vectors = MAnalyse(super, isb = true, truemotion=false, blksize=16, overlap=4)
forward_vectors = MAnalyse(super, isb = false, truemotion=false, blksize=16, overlap=4)
forward_compensation = MCompensate(super, forward_vectors, thSCD1=500)
backward_compensation = MCompensate(super, backward_vectors, thSCD1=500)
interleave(forward_compensation, last, backward_compensation)
SetMTMode(5)
FFT3DGPU(sigma=4, plane=4)
SetMTMode(2)
selectevery(3,1)

I tried using SetMTMode(6) but it crashed (which IIRC is the reason I don't use 6).
Running each script a few times in an 'analysis pass' in VirtualDub gave these results:

No MT
test 1: 5.02 fps
test 2: 4.97 fps
test 3: 4.93 fps
Average: 4.97 fps

MT
test 1: 9.03 fps
test 2: 9.19 fps
test 3: 9.03 fps
Average: 9.08 fps

I find it difficult to believe that just multi-treading SelectEvery() results in this difference.
Of course, the difference is much less significant when x264 is added:

x264 --crf 23 --profile high --preset slow

No MT
test 1: encoded 1084 frames, 3.91 fps, 4337.23 kb/s
test 2: encoded 1084 frames, 4.54 fps, 4337.70 kb/s
test 3: encoded 1084 frames, 4.15 fps, 4335.23 kb/s
Average 4.20 fps

MT
test 1: encoded 1084 frames, 4.34 fps, 4337.13 kb/s
test 2: encoded 1084 frames, 5.04 fps, 4337.21 kb/s
test 3: encoded 1084 frames, 4.27 fps, 4339.22 kb/s
Average 4.55 fps

[PetitDragon]

This would be an "ideal" case to use ThreadRequest. Alas, I don't get it to work with MVTools.

l have test the following script for realtime use in ffdshow for my 1080/24p anime:

---------------------
SetMemoryMax(1536)
cache=30
queue=5
warmingup=1
RemoveGrain(mode=1).ThreadRequest(cache,queue,warmingup)
FFT3DGPU(sigma=1.5, bt=4, mode=1, degrid=1).ThreadRequest(cache,queue,warmingup)
GradFun2DB().ThreadRequest(cache,queue,warmingup)
super=MSuper(hpad=16, vpad=8, pel=1, levels=2, rfilter=3, isse=true).ThreadRequest(cache,queue,warmingup)
backward_1=MAnalyse(super, blksize=32, blksizev=16, levels=2, search=3, searchparam=2, isb=true, chroma=true, plevel=0, badrange=(-24)).ThreadRequest(cache,queue,warmingup)
forward_1=MAnalyse(super, blksize=32, blksizev=16, levels=2, search=3, searchparam=2, isb=false, chroma=true, plevel=0, badrange=(-24)).ThreadRequest(cache,queue,warmingup)
backward_2=MRecalculate(super, backward_1, blksize=16, blksizev=8, search=3, searchparam=1, chroma=true).ThreadRequest(cache,queue,warmingup)
forward_2=MRecalculate(super, forward_1, blksize=16, blksizev=8, search=3, searchparam=1, chroma=true).ThreadRequest(cache,queue,warmingup)
MBlockFps(super, backward_2, forward_2, num=50, den=1, mode=0)
---------------------

it seems running very well without error.:p

[Didée]

it seems running very well without error.:p
Tongue is inappropriate.

script's longest lifetime - ~4000 frames to crash
script's shortest lifetime - ~100 frames to crash

[Gavino]

@Boulder, onesloth: Thanks for the test results, which show I was wrong in my assumption about how mt mode 5 works.

Digging around the source code, it appears that extra worker threads are created to handle the input(s) to a 'mode 5' filter (basically, something like I was trying to achieve with my Distributor() suggestion), so there is still some multithreading going on for previous filters.

I'm sorry I got this wrong. Unfortunately, the technical explanation of the modes MT_modes_explained (http://avisynth.org/mediawiki/MT_modes_explained) only covers modes 1 and 2, and the exact details of the other modes is not explained.

[Boulder]

Hey, no problem. At least we got some verification done regarding this issue :) I wonder if the handling is any different in the Avisynth 2.6 implementation (whenever it goes official)?

[Wilbert]

Digging around the source code, it appears that extra worker threads are created to handle the input(s) to a 'mode 5' filter (basically, something like I was trying to achieve with my Distributor() suggestion), so there is still some multithreading going on for previous filters.

I'm sorry I got this wrong. Unfortunately, the technical explanation of the modes MT_modes_explained only covers modes 1 and 2, and the exact details of the other modes is not explained.
Is someone up to this task?

[tormento]

Is someone up to this task?
Nobody, I suppose ;)