How do temporal avisynth filters work?

[Tobytl]

What I mean is, how does say a temporal smoother actually go about getting the various frames that it averages together? Does it actually ask for frame n, n-1 and n+1 for every frame? Or is it like a delayed write, like say it keeps a buffer going and only applys its magic and outputs a frame once n+1 arrives, so the filter is constantly 1 frame (or however many it processes) behind its input? I could probably check myself, if I could find up to date source code, but can't find that either I'm afraid (for fluxsmooth).

The whole reason behind this is that I am using avisynth to filter while capturing using ffdshow and I don't know how the script reacts if say fluxsmoothst() asks for the next frame when it hasn't been broadcast to me yet. I tried it, and as far as I can tell it appears to be working like a charm (and still using only 70%-80% cpu without dropping frames). But I don't know if there is some kind of unkown harm I am doing? And I am just plain curious...

It is entirely possible that I am just being retarded here, so if this is a stupid question I apologize.

[hartford]

Avisynth has an internal filter named TemporalSoften().

The source of Avisynth is available here: http://sourceforge.net/project/showfiles.php?group_id=57023

[Tobytl]

I'm not sure how the chaching works in avisynth, maybe setting up the cache hints like it does make the parent filter (or avisynth itself) not ask for a frame until the frames you need as defined by the SetCacheHints() call are, well, cached.

But either way, I ran a simple test and I would say it works the way I had hoped (that or fluxsmooth is borked). I used this script inside the image preprocessing of the ffdshow compressor setup to place two versions of the same input side by side with one filtered while capturing and one being the source without filtering: HorizontalReduceBy2()
Clip1=last
LoadPlugin("C:\AviSynthPlugins\FluxSmooth.dll")
FluxSmoothST(10,15)
StackHorizontal(last,Clip1)
(this is saved as a lossless huffyuv in ffdshow in case someone was wondering)

Then opened this script in mpc: (cap.avi is the capture from above)AVISource("cap.avi")
src=last
clip1=src.crop(0,0,320,480)
clip2=src.crop(320,0,320,480)
LoadPlugin("C:\AviSynthPlugins\FluxSmooth.dll")
clip2=clip2.FluxSmoothST(10,15)
Subtract(clip1,clip2).Levels(120,1,132,0,255)

And it was pure grey. With the FluxSmoothST line commented out you see the noisy ghost like images you would expect from clips that aren't quite the same. Even if you add in "Compare(clip1,clip2)" instead of the subtract line I still get 0 deviation, so they really are the same.

I guess I answered my own question, at least for my purpouses. Thanks hartford for pointing that out, I completely forgot that temporalsoften2 was added to the avisynth codebase. :) It was an interesting look into how temporal filters collect frames at least. If anyone else is wondering the code is in the 'focus.cpp' file.

Sorry for wasting your time everyone, don't know why I didn't think to run that simple test earlier :o...

[hartford]

I'm no expert on scripting, but if I'm not mistaken, the "subtract" filter works on luma only. That said, (someone correct me here) I _think_ that chroma must be merged into to the clip or you will only see "gray."

(foot in mouth, again).

[Tobytl]

Appearantly subtract works on chroma information as well. I just ran this script through mpc:AviSource("cap.avi")
subtract(last,last.swapuv())
SwapUV just exchanges the two chroma channels and does nothing to the luma channel (at least as far as I know it doesn't touch Y). And you get a rather interesting picture, so with Y (luma) being equal in both clips and the only difference being in the chroma channels, we can infer that chroma is being compared as well.

Oh, and subtract starts with grey (half the scale) then adds the difference of the pixels of the two clips. So if it stays grey they should be the exact same clips (I think :scared: ).

[Wilbert]

SwapUV just exchanges the two chroma channels and does nothing to the luma channel (at least as far as I know it doesn't touch Y). And you get a rather interesting picture, so with Y (luma) being equal in both clips and the only difference being in the chroma channels, we can infer that chroma is being compared as well.
Yup.

Oh, and subtract starts with grey (half the scale) then adds the difference of the pixels of the two clips. So if it stays grey they should be the exact same clips (I think)
Yup (grey: Y=126,U=V=128).

[IanB]

Tobytl,

Most temporal filters request all the frames required (...,n-2,n-1,n,n+1,n+2,...) for frame N from the cache immediatly below it. All frame number requests less than 0 are given frame 0, all frame number requests more than the number of frames are given the last frame, M. When the cache is working correctly the filter below that cache instance simply sees a single request for each frame, repeated requests for the same frame are serviced from the previously fetched frames stored in the cache.

Thus T.filter asks for [-2,-1,0,1,2], [-1,0,1,2,3], [0,1,2,3,4], ..., [m-2,m-1,m,m+1,m+2].
Cache asks for [0,1,2], [3], [4], ..., [m], [], []. Note the last 2 requests to T.filter are fully serviced by the cache and result in no frame requests to the filter below the cache. So in effect the cache introduces a overall 2 frame lag between the top of the T.filter and the bottom of it's associated cache instance.

IanB

[videoFred]

The whole reason behind this is that I am using avisynth to filter while capturing using ffdshow

This is very interesting news for me! Is this possible?
What do I need for this? Just ffdshow and configure it?
And does it matter witch program you use for capturing?
I guess you must configure the capture program to use ffdshow, right?

Another question: what is ffdshow actualy and what are the benefits of it? :thanks:

Fred.

[adamhj]

i am wondering how to capture video with ffdshow...

and i think the reading method of a temporal filter all depends on the filter itself, but in my opinion, it doesn't matter at all. when a filter ask for the next frame, avisynth would pause until the next frame is receive.

but, how would the framenumber variable in vi be set if the total frame number is uncertain?

[Tobytl]

@ IanB

Thank you. Thats exactly what I wanted to know, and from the way I understand your explanation, it works the way I had hoped.

@ videoFred

Yes, you can do real time avisynth filtering using ffdshow. I don't know if it's a new feature or not as I am using a relatively recent ffdshow build (03.03.2005). You can use it in any capturing application that lets you choose the compression codec (VFW or DirectShow, it doesn't matter). I'm using virtualdub, but like i said, you can use anything. You just choose ffdshow from the codec list, click configure, use the drop down boxes on the top to choose the actual encoding format (huffyuv for lossless, though the mjpeg at 100 constant quality looks good and compresses 2-5 times better). On the left side click on 'input', make sure 'Enable image processing' is checked and click the configure button. In the left pane of this new window is the avisynth setup option, make sure 'avisynth' and 'Add ffdshow video source' is checked, then just put in your desired script in the text editor. Just make sure your computer is capable of delivering that script in real time, otherwise you will drop frames during capture...

And ffdshow is just a directshow (and vfw) wrapper for ffmpeg/libavcodec/libmpeg2 and a few others. That statement is kind of simplistic, but mostly accurate anyway...

@ adamhj

The num_frames property does not matter as long as the temporal filter in question behaves itself. From the sources I have seen, this property only matters when it is less than the requested future frame (so the filter knows not to average frames that don't exist). But IanB above has pointed out that the temporal filter doesn't even process anything until it gets all the frames that are needed to average whatever frame you are on. And by the time it does that, the source filter (the one ffdshow adds for you) has already changed vi.num_frames to be whatever frame its sending out.

Uh, yea I can't explain things very well. Basically the total number of frames is being added to ahead of the temporal filter, so it doesn't have to worry about it until you stop capturing (when the number of frames is no longer being added to).