Motion Compensation

[mf]

Manao has made the bold step of starting motion compensation filters. I would like to use this thread for two things, for users like me to present ideas, and for developers like Manao to explain the practical side, what is possible, how hard it is, etc. And of course, for developers to share ideas on the implementation of these filters.

This morning I have started to play with MVTools, and am currently encoding a supersampled (sorry, couldn't help myself :rolleyes:) motion blur, to see how it looks in realtime. I am impressed by the results so far.

[Wilbert]

Manao has made the bold step of starting motion compensation filters.
Small question. I don't have the time to look at it yet, but maybe someone of you can say something about it. Is it possible to do the following: hide someone's face with blocks, and keep it hided while he moves using this motion compensation stuff?

[Abond]

The problem was disscused before, maybe in the usage forum. Found it: http://forum.doom9.org/showthread.php?threadid=72895
If I remember the aim was different, but the approach...

[Mug Funky]

i'm attempting a motion-compensated deinterlacer a-la tomsmocomp.

not getting good results yet, but there's a glimmer of nice coming (pans are leaving no combing or stairstepping after just 4 mins of playing)

[edit]

argh! where's my brain? this is tough.

[scharfis_brain]

Do you mean this:

function mvdeint(clip x, float "moblur")
{
mbl=default(moblur,0.1)

x.separatefields()

evn0=selecteven().converttoyv12()
odd0=selectodd().converttoyv12()

# compensate the even field to the temporal position of the odd field
evn1a=evn0.mvinterpolate(nb = 5,bl = 0.5-mbl, el = 0.5+mbl, wf = "hat")
evn1b=evn0.duplicateframe(1).reverse().mvinterpolate(nb = 5,bl = 0.5-mbl, el = 0.5+mbl, wf = "hat").reverse()
evn1=overlay(evn1a,evn1b,opacity=0.5)

# compensate the odd field to the temporal position of the even field
odd1a=odd0.mvinterpolate(nb = 5,bl = 0.5-mbl, el = 0.5+mbl, wf = "hat")
odd1b=odd0.duplicateframe(1).reverse().mvinterpolate(nb = 5,bl = 0.5-mbl, el = 0.5+mbl, wf = "hat").reverse()
odd1=overlay(odd1a,odd1b,opacity=0.5)

# chain original and compensated field
evn2=interleave(evn1,evn0).trim(1,0)
odd2=interleave(odd1,odd0)

# reinterlace the strea,
interleave(evn2,odd2)
weave()
}

[Manao]

Wilbert : it can be done, but it's not very pratical with Avisynth ( no easy way to define where is the face at the beginning, since there is no GUI )

@all : I'm waiting for your ideas, mainly on how to treat uncovering of areas, how to use motion vectors to decide whether there is a scene change ( I have access to their length and the SAD assiociated with them ).

Right now, vectors have to point inside the frame ( which means, that a block that move toward the border gets a wrong motion vector ). I'll try to change that, but that's not an easy thing.

[scharfis_brain]

I'm waiting for your ideas, mainly on how to treat uncovering of areas

Ideas working out hidden areas.

Try to imagine a static background. Maybe a landscape.

now small object drives between camera and background from left to right

everyframe, the object moves from left to right, you have to replace the leaved space with the background (this would the way, our brain is working)

you could do that with a motionmask, that says: "this area was static a frame before / is static the next frame)"

now imagin the background is panning (direction irrelevant, for now)
a normal motionmask will fail, BUT a motioncompensated motionmask will work!

compensate, lets say 5 frames, to the same temporal postion, build a motionmask, and figure out the areas, gone static within those 5 frames. Those pseudostatic areas can then be used to fill the uncovered areas.

I hope that wasn't to irritating, trying to make the best out of my crappy english knowledge

[Mug Funky]

Scharfi:

that's very cool indeed. bit of a dud on scenechanges i'm afraid :)

i get 2 combed frames for 1 uncombed scenechange.

[scharfis_brain]

this mvdeint puts out the naked uncorrected motioncompensated video.
correcting that will require using masks. A lot of masks.

But I do not really like the output. even a whole frame isn't compensated correctly. there is almost stairstepping.
much more than with kerneldeint.

[Wilbert]

Wilbert : it can be done, but it's not very pratical with Avisynth ( no easy way to define where is the face at the beginning, since there is no GUI )
Can't you draw a circle in which the face is located (given that the user provides the location of the circle)? Or do you need the location of the edge of the face for this to work?

[Manao]

Stair stepping is to be expected : the motion during three consecutive frames isn't exactly constant, so the interpolation created from frame 1 and 3 won't be exactly aligned with frame 2.

The method should be the following :
- fieldseparate
- aligning fields ( moving the second field half a pixel up or down, if needed )
- compute motion
- motion compensating odd frames on even frames, and merging a spatial interpolation with the result of the motion compensation.

It can be done using only avisynth scripting, but I can't implement it today.

Wilbert : I do not need the edges, a circle may be enough. But it will have to fit rather well ( ratio area of the face / area of the disk close to 1 ), in order to follow closely the face ( it will even be able to discard blocks inside the circle that don't belong to the face :) ). I provide the source code with the filter, there is a class GenericMotionFilter from with all my filters inherit. You could code it yourself, if you can make my code to compile properly. Fetching the motion vectors is quite easy ( look at MVBlur, for example ), and then the algorithm would be to make a mean of the motion vectors inside the circle, and then decide that all blocks whose vector is close to the mean are inside the face.

You know where these blocks went since you have their motion vectors. So now, you have a set of blocks which should overlay the face on the next frame. You repeat the algorithm and it should work.

[scharfis_brain]

Oh, I think, scripting this within avisynth on my 500MHz Machine will be pain...
I do not have acces to the faster one next days :(

but: what about a motioncompensated 60i to 24p conversion?


function mvconvert60ito24p(clip x, int "mode")
{

mode = default(mode,2)
mbl=0.1

ya=x.mvinterpolate(nb = 4,bl = 0.5-mbl, el = 0.5+mbl, wf = "hat")
yb=x.duplicateframe(1).reverse().mvinterpolate(nb = 4,bl = 0.5-mbl, el = 0.5+mbl, wf = "hat").reverse()
y=overlay(Ya,Yb,opacity=0.5)
interleave(y,x)
mode0=selectevery(5,2)
mode1=overlay(selectevery(5,3),selectevery(5,2),opacity=0.5)
mode2=overlay(overlay(selectevery(5,1),selectevery(5,3),opacity=0.5),selectevery(5,2),opacity=0.3)
mode3=overlay(overlay(selectevery(5,0),selectevery(5,3),opacity=0.5),overlay(selectevery(5,1),selectevery(5,2),opacity=0.5),opacity=0.5)

(mode==0) ? mode0 : (mode==1) ? mode1 : (mode==2) ? mode2 : mode3

}

if the video has a shutter speed of 1/60 sec the modes result in following simulated 24p shutter speeds:

mode0 1/60 sec
mode1 1/40 sec
mode2 1/30 sec
mode3 1/24 sec

If 1/120 sec shutter has been used while shooting, you'll get those simulated shutter speeds:

mode0 1/120 sec
mode1 1/60 sec
mode2 1/40 sec
mode3 1/30 sec

the higher the mode (max 3), the lesser the mv-artifacts

have fun.

[DDogg]

scharfis_brain, the script above would require kernelbob(7), correct? Like -

avisource("G:\final.avi").converttoyv12(interlaced=true)
kernelbob(7)
mvconvert60ito24p()

/Add: Maybe you could start a thread in Avisynth user to discuss?

[scharfis_brain]

jep. exactly this.

[Manao]

Scharfi : concerning the process of uncovering areas, I would prefer if it wasn't involving more than two frames ( because searching for motion vectors is a rather slow process yet, and I would prefer if the filter was running at a few fps, not some mfps :) ).

Now, for the MVConvertFPS, I got an idea which works quite nicely. As you have notice, the interpolated frames have got quite a lot of artifacts, because the moving blocks don't pave well the frame. However, if I compute motion vectors on 8x8 blocks, nothing forbid me from moving 12x12 blocks, or even 16x16 one :). That trick performs very well, and allows some artifacts removal.

It now needs the SAD decision, as well as vectors able to point outside of the frame, and it will become usable.

Wilbert : I gave a try for face tracking. It doesn't work well on close up, because the face then doesn't have a uniform movement. I'll try it on other things, it may prove better.

@all : I'll try to make a good documentation on how to use the API I made for fetching the motion vectors. I don't know what you need in order to make a filter without having to compile my code : obj files, headers, and what else ?

[scharfis_brain]

Manao: what about a server/client structure?

a filter analyes the stream and gives back a data stream with all motionvectors and blocksizes needed to do a compensation.

the other filter will to the compensation then.

this would allow us, to write our own postprcessing routines using avisynth's masking...

I am thinking about a structure similar to fizicks depan().

[Manao]

Scharfi : whatever the method ( client / server or using the API ), you still have to write a filter in order to do use the motion vectors. Then, the postprocessing needs to know what the filter did, so it has to be included with the filter.

It will require for an outside programmer the same amount of work to use the API, or to fetch the stream of vectors. So for the moment, it will stay like this. Later, if I see that some filters could be chained without interfering, and that they both would use the motion vectors from the original clip, I will implement the client / server solution.

[Fizick]

Manao:
Can we use VC6 or free VCToolkit2003 for compiling?

[Manao]

I use the VCtoolkit2003 + Nasm. If you don't succeed, I'll put the .obj files online ( I don't know if it's enough, but I guess it is ).

[Fizick]

Why you remove Compensation function from new version?

[Manao]

Fizick : Which compensation functions ?

In the class BlockData, there is still MotionCompensateBlock(). It copies the block pointed by the vector in the array given in argument.

But I don't use it in MVBlur() and in MVConvertFPS, because in these filters, I move the block which possess the vector along this vector.

Scharfi : I encountered your bug ( choppy motion with high fps conversion ). It's mainly because - for the moment -, I compensate with a pel precision only. It happens also because when I compute by how much I have to move the block, I round a float number with flooring instead of rounding. I'll change that, which will partly solve the problem.

Later, I'll be able to move a block along non integer vectors.

[scharfis_brain]

nice to hear, You've found it, too!

what happens on a Zoom? will your fiter 'only' move the blocks or will they be zoomed, too?

the same on rotations: will it move the blocks or are they rotated, too?

[Manao]

Blocks are not rotated nor zoomed. I only search the translation that minimizes the distorsion. Taking into account rotation and zoom would mean minimizing a function of 5 parameters instead of 2, which is by itself a lot more complex. Moreover, trying such a transformation implies computation far more complex.

That means of course that you'll see artifacts if there is a fast zoom or a fast rotation. But its unavoidable for the moment ( maybe not in 5 or 6 years ).

For filling uncompensated areas, I just tried another trick which performs interestingly ( better results but segfault :) ) : I want to interpolate a frame between frame n and frame n+1 :
- I compute vectors between frame n-1 and frame n, frame n and frame n+1, frame n+1 and frame n, and finally frame n+1 and frame n+2. Vectors from decond and third computation are used in priority. Vectors from first and fourth are inversed, areas filled by these compensation are added to the already filled areas. I also increase the size of the block moved ( making the added borders less prioritary than the core )
- Finally, if there is still some areas unfilled, they are filled with a blend of frame n and frame n+1. I'll try later to fill them spatially, it will be better.

[scharfis_brain]

Another Idea:
what about using a global motion compensation, like fizick depan for moving/rotating/zooming large uniform areas and afterwards compute the remaining diferential motion on this global compensated frame.

Could this improve precision on moving objects and global rotation and zooms (oh-oh zooms ARE global, hehe)?

is this implementable?

I hope, that I don't bother you with my thoughts.
I cannot estimate the programming work your doing. I assume it is much...

[Manao]

It would be possible, but I don't think if it would worth the amount of work required to implement it. It will not be me who would implement that algorithm, because there is still a lot of work on other parts of the motion engine, and on other filters ( MVDenoise, mainly, because a good denoising would be really great )
I hope, that I don't bother you with my thoughts.Not at all, I asked for ideas and I'm glad you're exposing yours.I cannot estimate the programming work your doing. I assume it is much...I'm still a student, so I have plenty of time.

[Fizick]

I want to try to use your plugin similar as Scharfis spoke,
to make local motion compensation after global compensation (by my DePan plugin).

In previous versions, the ShowMotion function had option to make such local compensation.
But now it is removed. Can you restore it? It is very general function, anybody could use it for denoising with any temporal filter.

Yes, I found 2 internal functions in source code:
void BlockData::MotionCompensateBlock(unsigned char *block)
void BlockData::MotionCompensateBlock(unsigned char *block, int dpitch)
(what is correct ?) But right now I can not modify and compile your plugin.

[Manao]

Both were correct a long time ago. But since I changed a lot of things inbetween, the second one (( block, pitch )) is now useless.

I'll will put back the motion compensation part of MVShow.

There will be another release during the week, with updated MVBlur / MVinterpolate / MVConvertFPS / MVShow.

[Fizick]

Thanks for answer.
I hope, you will include both forward and backward compensation (option).
May I ask for make file for project?

[violao]

I managed to compile mvtools using VisualC++ 6.0 and nasm (GPL) and right now I'm trying to create a filter using motion vectors from SearchMVs (like in MVShow). The problems are comming mostly from occlusion areas, mainly uncovering regions around moving object and edges. It appears that SearchMVs produces 'spurious' motion vectors in and around these regions and all further actions are worthless since just by looking at vector we cannot know if it's spurious or not. Having those false vectors around we also cannot properly detect moving edges in order to provide for different treatment for occlusion areas. OTOH this should be possible if we had 'zero' vectors in uncovering regions or anything else that matches the movement of surrounding not-occlusion area. If we find a way to produce 'better' matches for those vectors, then most, if not all further filtering will be better.

Luckily, there seems to be a way to approximate those vectors and I would like to kindly ask Manao, the author, to consider implementing this as either an extension to SearchMVs, or even better a separate function for calculating motion vectors. This should be very similar to existing function, but would need to work on 3 frames: previous, current and next. While searching for vectors it should look in BOTH previous and next frames and choose whatever displacement in either previous or next frame delivers the lowest SAD. This way if we have uncovering region in previous->current frames it will (hopefully) become fully visible region in current->next frames so it's motion vectors will likely be similar to neigbouring not occluded area.

This should probably be sufficient *i think* for various filtering purposes, but even for interpolation this may eventually solve 'edge problems'. Since we should now have completely qualified (even not the best) vectors for the whole frame we might try to detect occlusion areas by finding horizontal and vertical moving edges. Then we may regionalize the frame and apply different interpolation strategies to different regions, for example we may try to interpolate in uncovering regions from next frame only, hopefully producing less artifacts in those regions.

[Manao]

New version : MVTools v0.9.2 (http://jourdan.madism.org/~manao/MVTools-v0.9.2.zip).

Changelog : * MVInterpolate doesn't exist anymore.
* MVBlur and MVConvertFPS have been improved. They also have got new parameters, have a look at the documentation.
* MVShow gets back its compensate mode ( MVShow(cm = true) )
@violao : I'm glad to hear you manage to compile my code, expecially with MSVC 6.0.

Your idea is indeed a good one, but you have to slightly extend it since a covering / uncovering happens between two frames, not exactly on a frame. Hence, you have to fetch motion vectors between frames n-1, n, n+1 and n+2.

But your idea amounts to make a postprocessing on motion vectors, and such processing must not occurs in SearchMV, but elsewhere. Moreover, with the source code I provided with the latest version, you can implement it quite easily. Look at the code in MVConvertFPS to see how to search for vectors between different couples of frames. Once the search is made, it's up to you to find and apply a good algorithm related to what you want to do with motion vectors.

I may write some utilitarian functions to ease that process, but, at the end, the process is too dependant of the use of the motion vectors, so there will not be a function 'MakeMotionVectorsGood'.

@all : you'll see that MVConvertFPS works better, especially on slow motion scenes ( closeup ). There should not be anymore slutering when trying to slow down a lot the video.

But, on high motion scenes, don't expect wonders. The concept of motion vectors shows clearly its limits in such scenes.

When scenechange is detected ( either real scenechange, or too much motion for the filter to cope with ) the frame which is built is not interpolated. Hence, on really high motion scene where everything is detected as scenechange, motion will slutter.

Covering / uncovering is slightly better interpolated, except when the covering object is the border of the frame ( that's the first place to search for artifacts ).

[scharfis_brain]

* MVInterpolate doesn't exist anymore.

why this?

I hoped you had made some improvements to it, too. But now you've removed it :(

mvinterpolate was very good for:
- repairing dropped & duped frames (it worked very cool)
- trying to build some motioncompensated deinterlacing (that was working too :) but with the underlying issue, i mentioned earlier)
- building a high configurable 60i to 24p conversion.
- and lot more per-frame repairing stuff

could you Pleeeeeeeze reinclude it?

a function that does custom motioncompensation (with your el &bl params) is really important, IMO....

[violao]

Originally posted by Manao
Your idea is indeed a good one, but you have to slightly extend it since a covering / uncovering happens between two frames, not exactly on a frame. Hence, you have to fetch motion vectors between frames n-1, n, n+1 and n+2.
I don't really get this. The idea of covering/uncovering is based on a premise that uncovered regions in (n-1, n) sequence will be fully visible in (n, n+1) sequence. Since they are already fully visible in the latter case, we may rely on vectors produced by (n, n+1) and I fail to see how can additional (n+1, n+2) vectors help in resolving original problem introduced in (n-1, n). Searching in (n-2, n-1) also wouldn't help since if something was occluded in n-1 it has probably been occluded in n-2 too.
But your idea amounts to make a postprocessing on motion vectors, and such processing must not occurs in SearchMV, but elsewhere.
Of course, but I was not talking about post-processing, but about the search itself. Post-processing cannot help solving original problem. Just by comparing 2, 3 or more vectors we cannot tell which one is 'false'. What I proposed was a modification of search scheme, just for this purpose, of course, not generally. The idea is that if we have covered->uncovered->visible sequence in 3 frames, then min SAD search will likely find better match for uncovered region in the next frame. That motion vector (n+1, n) is in no correlation with previous (n-1, n) so simple comparison or other operations with them are useless. Instead we need to use (n-1, n),(n, n+1) SAD as a criterion, not the final vectors from (n-1, n), (n, n+1) or whatever. At least I think so.

I'll check on your latest sources later. BTW, thanks for your great work.

[Manao]

violao : I grouped covering / uncovering together, one of which happened between frames n and n+1, that's why I spoke of frames n-1 to n+2.

And for implementing the algorithm outside SearchMV : don't forget that you have access to a lot of things once SearchMV has been call :
- GetFinalMV() gives you the motion vector
- GetFinalSAD() gives its SAD
- GetBestMV(int i) gives you the ith best motion vector found, SADwise. ( GetBestMV(0) <> GetFinalMV() if fth > 0 )
- GetBestSAD(int i)

So with all that, I think you can implement your algorithm.

scharfi : I thought it was useless with MVConvertFPS, but if you need it, I will make it come back.

@all : the last frames of MVConvertFPS are not returned. I'll correct it.

[violao]

Thanks. Somehow I missed the SAD part od blockdata :(

[Manao]

Alright, new version : MVTools 0.9.2.1 (http://jourdan.madism.org/~manao/MVTools-v0.9.2.1.zip)

Changelog : * MVInterpolate makes its come back.
* MVConvertFPS should work on the last few frames of the clip

violao : I'll try to make a good documentation on the API, with doxygen. It will be easier for you.

[scharfis_brain]

Many thanks!

I will test that out asap.

[Leak]

Originally posted by Manao
MVTools 0.9.2.1 (http://jourdan.madism.org/~manao/MVTools-v0.9.2.1.zip)

Thanks for the new version - I just hope you won't add another number at the end with each one from now on, that could get a bit unwieldy... ;)

When playing around with MVConvertFPS I've found that actually setting wf to "hat" produces an image only consisting of garbage blocks in red, yellow, green, blue, violet and cyan instead of the expected picture, which probably isn't what you intended... :)

I'm just curious why it seems to work if I don't specify wf at all, since "hat" is the default - but for generic functions not being able to specify "hat" even if it works as the default might be a bit problematic...

np: Autechre - C/Pach (Tri Repetae)

[Manao]

Well spotted, Leak. It seems that "hat" in fact never worked well. Hopefully, default was changed ( not intentionnally ) to "uniform", and I forgot to change documentation. Both will be updated ASAP, "hat" in order to work, documentation in order to report well default values.

So here is MVTools 0.9.2.2 (http://jourdan.madism.org/~manao/MVTools-v0.9.2.2.zip)

[scharfis_brain]

manao: the default should stay on uniform blending. THis is the natural way.

look at moblurred Film or Video. you'll only see uniform motionblur. no hat or other ramps there.

[Manao]

It stays with "uniform" as default, don't worry. I may have made me misunderstood in the previous post ( "not intentionnally" was meaning not related to the bug ).

[avih]

Manao, this is sick :D

as some of u might know, i much prefere 50fps playback over 25fps when it comes to video (i.e. live sport events, etc). i usually capture <something>x576 interlaced with ffdshow with some processing thrown in, and i don't re-process my caps as i mainly use it as pvr (playback with ffdshow, deinterlacing using a dscaler filter).

now, i tried taking one of my encoded clips (400x576), take one field only, and MVConvertFPS (0.9.2.2) to 50fps, and it's awesome :) i reduces the threshold a bit (quite a bit), and i truely get a 'video-like' expreience :) still few glitches here and there, but generally, it rocks.

i'm now thinking of using ffdshow to capture, with deinterlacing filter (encode 25fps 'progressive'), and then use MVConvertFPS during playback to record full res at 25fps and playback interpolated 50fps. might save quite some bits during encoding i think.

although so far, MVConvertFPS doesn't play good in zoomplayer (both direct avs and through ffdshow with ebmedded avisynth script). vdub plays it well. i don't know what's the cause yet. i'll try to update the thread if i have new conclusions.

cheers Manao, great work :)

[violao]

While I was playing with motion vectors calculated by SearchMVs I noticed the following problem that sometime makes vectors found in a certain area useless. Suppose you have an object moving in front and near the edge of a very dark static background area. Now suppose that beside that edge there is another static area with the brightness level similar to that of the moving object and that the distance between the object and this similar area is less than displacement of the object between n and n-1 frames. It appears that the vector search algorithm finds this closer background area more similar to the original object than the object itself in n-1 frame so the resulting vectors point from surrounding background to the object, instead from object in n-1 to object in n. Is it possible to do something about it? Perhaps increasing vector search space?

Another question, how is it possible to fetch vectors from higher levels? I'm familiar now to using BlockData class after SearchMVs, but I fail to see how to get field of blocks for higher levels. Manao?

EDIT: Fetching higher level vectors solved (I think).

[Manao]

First, the easy question :
Another question, how is it possible to fetch vectors from higher levels? I'm familiar now to using BlockData class after SearchMVs, but I fail to see how to get field of blocks for higher levels.You access them by using a higher PlaneOfBlock. If GOP is a GroupOfPlane, GOP[x] return the xth PlaneOfBlock, counting from the lowest, and GOP[x][i] then return ith BlockData of the xth PlaneOfBlock.

Now, for the other one. There are several reasons that may lead to such results, could you provide a screenshot, it would help determine whether it's a failure of the search algorithm, or something inherent to block matching.

For the principle of the search algorithm, there is no such thing as "space search". The multilevel analysis allows to fetch long vectors, while the use of a non exhaustive recursive search ( One Time Search, Diamond Search ) tries to find the local minimum around the best predictor ( either the one from the multilevel analysis, or the motion vector from surrounding blocks ).

It you want an exhaustive search, in PlaneOfBlocks::SearchMVs(), replace blocks[i]->SearchMV(predictedMotionVectors[i],1,1,ONETIME, CURRENT);byblocks[i]->SearchMV(predictedMotionVectors[i],1,radius,EXHAUSTIVE, CURRENT);Where radius is the half-width of the search square. Also, set 'fth' to zero. But it will be very slow then. You'll then be able to know if the 'wrong' vector you're getting is due to the search algorithm or to the block matching itself.

avih : thank you for the kind words. I don't know what's wrong with ZoomPlayer, but if I had to make a guess, I would say it's the DirectShow interface.

[violao]

Originally posted by Manao
It you want an exhaustive search, in PlaneOfBlocks::SearchMVs(), replace by...Where radius is the half-width of the search square. Also, set 'fth' to zero. But it will be very slow then. You'll then be able to know if the 'wrong' vector you're getting is due to the search algorithm or to the block matching itself.[/B]
Thanks. I'll try various search algorithms and will let you know what happens. Radius is given in block units, I suppose?

[Manao]

No, it's given in pixel unit. The search is made for all the vectors whose coordinates don't differ from the best predictor by more than 'radius'.

But be warned, a radius as small as 5 will imply 121 SAD computations by blocks, so it will be slow.

[violao]

Originally posted by Manao
But be warned, a radius as small as 5 will imply 121 SAD computations by blocks, so it will be slow. [/B]
OK, I tried with radius up to 16 with no success. Other search methods give similar results. I suppose this clip I'm playing with violates basic assumption that is 'intensity consistency hypotesis'. It seems that whenever an object moves between different illumination conditions (various shades and similar) there are some funny vectors around. Another complication seems to be the bluring of faster moving objects. Bluring is itself a blending of an moving object with static background, that results in changing both objects shape and it's luminance.

Therefore all I need/can do is to supress the filter processing in the area of 'invalid' vectors. Only problem is - how to tell what vectors are invalid?

[Manao]

Only problem is - how to tell what vectors are invalid?If that answer was simple...

I have implemented two measurement of validity for a vector : the SAD, and what I call 'DifferenceFromNeighbours', which is basically by how much the motion vector is different from motion vectors of the surrounding blocks.

So you have to use both of them, but it won't be easy.

[violao]

Originally posted by Manao
'DifferenceFromNeighbours'...
"Sum of quadratic differences between the motion vector of the block and all its surrounding blocks' motion vectors."

8 surrounding blocks? Is this a difference between MV length?

[Manao]

No, not exactly. If the value is high, it means that the motion vector is a singularity ( it doesn't belong here ), because it isn't homogenous with its neighbours. If it was a mere difference of MVLengths, vectors could be very different and the value still be low.

To compute it, you compute the difference between the motion vector and one of its neighbours ( hence you obtain a vector ), you consider its length, and you sum the square of the lengths of difference obtained for all the surrounding blocks.

For the moment, I detect scenechanges using only that value, and it works ( not perfectly, I will later use a combination of SAD and this value ).

[violao]

Originally posted by Manao
To compute it, you compute the difference between the motion vector and one of its neighbours ( hence you obtain a vector ), you consider its length, and you sum the square of the lengths of difference obtained for all the surrounding blocks.
Are you assuming here that all neighbours are 'valid'? Wouldn't a 'distance from median' be more appropriate here? It would protect from possible outlier neighbours.

[scharfis_brain]

manao, is it possible, to create a function, that returns a float value of the currently present motion?

i.e. summarizing the vectors lenghts of the largest, moving object?

this would help, detecting framedrops, skips and duplications.

it also may help setting the fieldorder automatically...

[Manao]

violao : you're right, but since it took more time to code, I prefered implement the easy version, i.e. the mean. I'll put that on my TODO list.

scharfi : yes, it's easy, but what do you want exactly ? The mean of the length of the motion vectors ? The median ? If not, what else ?

[scharfis_brain]

I'll try to explain

imagine a constant horizontal scrolling video.

origonal
Frames: A B C D E F G H I J K L

now, frame F gets dropped:

Frames: A B C D E E G H I J K L

this means, the (simplified) output of the wished function should be:

Frames: A B C D E E G H I J K L
output: 1 1 1 1 1 0 1 1 1 1 1 1

another more weird sample:

a frame gets dropped, but isn't replaced with is predecessor. It is replaced with any of the previous frames:

Frames: A B C D E A G H I J K L
output: 1 1 1 1 1-5 1 1 1 1 1 1

as you can see, the dropped frame F gets replaced with A, this means, the output of my wished function should be the (at least) inverse of the previous value.

the calculation method of the length of those vectors, seems to be irrelevant for this kind of detection, I think....

What do you think?

[Manao]

Alright, so you need the vector, not its length. It's not hard to do, it will be in the next release. But I wonder if this can't already be done with the Global Motion Compensation tools made by Fizick.

[vinetu]

An idea for 2 pass motion compensation process.

Lets assume the source is progressive.

1.upsize the source x2 using some advanced resizer (Smart_resize_filter or Didee's iiP or similar)

2.aplly the best denoiser on the upsized source (don't know which exactly)
-the targed is almost perfectly denoised and stabilized (without flickering) picture

3.apply some advanced sharpener filter (VirtualDub MSU Smart Sharpen Filter for example)
-the target is to produce an almost extremely contrast picture (looking like synthetic one)

4.run MVTools over that "synthetic" source,calculate needed data and save it in a "1-pass.log" file

5. 2-pass process - run it again over original (not enhanced) source,taking in mind scaling x2

Is it possible in theory ?


P.S. I'm not familiar with MVTools yet... :)

Best Regards!

[Manao]

It's possible in theory. No need to upscale I think ( anyway, MVTools can work at subpel precision ). But it would need be able to export motion vectors to a file, which isn't yet in the TODO list ( but since it interests a lot of people, it will soon get there I think ).

Now, I don't think results will be far better than without preprocessing.

[vinetu]

Now, I don't think results will be far better than without preprocessing.

Let's hope it could be :)

Thank You!

[Fizick]

Vinetu:
For noisy source (also luma flicker, blotch, etc), pre-cleaning must give more regular vectors. But why to write vectors to file?

Manao:
I stil hope that you will release not only forward, but full backward compensation as general using function.

[Manao]

I think you can have it by using the following trick : source.duplicateframe(1).reverse().mvshow(cm = true).reverse()(I didn't invert it, I first saw it on a scharfi's script)

Hence, you'll be able to wait the next release in which it'll be possible.

Edit : oh, and the "file" idea : most codec still need two passes to work, that's why I was speaking of outputting it to a file. Anyway, as soon as I defined a structure to store the motion vectors data, it won't matter whether I output it in a file or in an array in order other filters to use it (i.e. client / server as you're doing with depan )

[vinetu]

...But why to write vectors to file?

Finaly I think that the only necessary thingy to
improve MVectors detection is to split the input video
stream in 2 -one for proccesing and another for MVectors analysis.
This second stream could be resized,denoised,sharpened,luma-chroma stabilized
and even slightly blured ...I'll try soon if this "preprocessing" can help.

So generally there is no need for 2-passes...

But now I see another usefull trick :
at 1-pass you can load a completely different "reference" file and save MVectors.log and then
at 2-pass load the "video" file and procces it in relation to "reference" file...

I don't know yet where this could be used,but it's sound intriguing for SpecialFX compositing...

Best Regards!

[Fizick]

Vinetu:
Yes, 2 streams (clips)!

[vinetu]

Fizick you win! for now the count is 2:0 :)

But here is come the next situation:
1.you made a 1-pass metrics and save them in a "YourFavoritMovie.MVectors"
2.open that file in a MVectorsEditor.exe (a dreamed application) and correct some MVectors here and there...
3.watch the movie in 100Hz (another dreamed thing- a DirectShow filter which will read the values from *.MVectors
and process (interpolate) the picture in real time) ...

too many dreams here? ;)

Best Regards!

[Fizick]

As I know (and even tried ones upon a time :devil: ),
1 and 2 is realized in Retimer
http://www.realviz.com/products/rtpro/index.php
But manual editing is not very interesting work.

About 3. Seems, MC is realized in modern TV (by hardware).

But what if I say, that good old film restoration (with MC, filtering, etc) is produced in about 1 fps in software (by 1 CPU).
Probably, we also need in some hardware signal processing chips.
May be, videocard chip?

(It is not my dreams, but some dilettant's thinking).

Best regards!

[scharfis_brain]

I found out, that mvinterpolate(el=1,bl=1) is a simply null transform.

and mvinterpolate (el=0.75,bl=0.75) doesn't do 0.75 fwd compensation. it does a 0.25 back compensation instead.

for the things, I want to do, I need a full compensated frame.
Only the underlying frame may be the frame towards is compensated to.

this means, el=bl=1 will shift the movie temporally by 1 frame, but all frames are compensated fully by one frame.

[Manao]

I found out, that mvinterpolate(el=1,bl=1) is a simply null transform.

and mvinterpolate (el=0.75,bl=0.75) doesn't do 0.75 fwd compensation. it does a 0.25 back compensation instead.Yes, but that's the wanted behavior. It isn't meant to motion compensate, it is meant to interpolate. So the compensation of each frame is weighted by the distance ( in time ) from the interpolation to the frame.

Hence, if you do 'el = bl = 1.0', the filter will build 2 frames, but when merging, they'll be weighted by 0 and 1 respectively, so only one will be kept.

Also, the filter interpolates between the current and the previous frame. So there's no forward compensation possible. ( except reversing the clip ). And 'bl = el = 0.75' does a 0.75 backward interpolation.

To motion compensate, use MVShow(mc = true) instead.

@all : I'm currently working on the core of the set of filters. I've cleaned the code, I'm now using optimized copy functions ( from avisynth's code ), the type of search can now be chose ( four methods implemented : one time search, N step search, logarithmic search ( aka diamond ) and exhaustive search ).

[violao]

Originally posted by Manao
@all : I'm currently working on the core of the set of filters. I've cleaned the code, I'm now using optimized copy functions ( from avisynth's code ), the type of search can now be chose ( four methods implemented : one time search, N step search, logarithmic search ( aka diamond ) and exhaustive search ).

Manao, have you seen this: Correlation Based Search Algorithms for Motion Estimation (http://amp.ece.cmu.edu/Publication/Deepak/Pcs99.pdf)

Might give you some new ideas :cool:

[scharfis_brain]

I tried to use mvshow(...) for full compensation, but its output was (who wonders) crappy

Could you code something like this ? :

http://home.arcor.de/scharfis_brain/ffc.png

That should handle hidden areas fairly well in theory

[Tuesday]

Hey, I'm not an experianced programmer or anything...but doesn't xvid have a rather well developed motion seach/detection/prediction engine ticking over somewhere in its core?

maybe some of their code would be useful, being open-source an all :)

if im entirely wring then just ignore me and i apologise for meddling in matters beyond my understanding :)

[Fizick]

I try to use motion compensation. It work quite good!
But there are often many blocks on picture.
Probably, the better will be compensate not block, but pixels (every or may be only from such bad blocks),
with motion vectors calculated by some interpolation data from current and neighbors blocks.

[Manao]

@violao : thank you, however, the algorithms described is this article are slower than the one I use, and not really better I guess.

@Scharfi : what do you want to do ? Compensating or interpolating ? Hidden areas while compensating aren't an issue, whereas while interpolating they are.

Your example describes the behavior of MVDenoise ( without the missing area filling, since there are none to do ).

The result of full motion compensation is indeed crappy ( well, mostly blocky in fact ), but it was meant to be that way. I'll add parameters in order it to be tweaked ( the way Fizick is describing )

@Tuesday : indeed, XviD ME is good, that's why I used its principle in the first place ( diamond search, and spatial prediction ).

@Fizick : that may be done, I'll add it to the to do list.

@all : the client / server part is working now. I'm now looking how to write vectors to a file, which means I'll have to compress them ( you wouldn't like a vector file weighting 15 GB :rolleyes: ). I've coded a huffman tree, which would reduce it to 500 MB - 1 GB. The funny part is, if I ever add a DCT / iDCT code somewhere, I'll almost make a codec out of it :D

Without kidding now, the client / server was a good idea. It makes the job for people who want to write filters a lot easier. They should have anymore to compile my whole code.

I'll try to release it during the week ( I still have to update documentation ) but without the 'saving vectors to file' part ( that one will definitely be tricky )

[scharfis_brain]

I try to build a full motion compensated deinterlacer.
Everything I tried so far has big issues.

fieldsequence:
A C E G I <-top field
b d f h j <-bottom field

1st method:
mvinterpolate between A & C to recreate B
mvinterpolate between b & d to recreate c
mvinterpolate between C & E to recreate D
mvinterpolate between d & f to recreate E
.
.
.

this works absolutely correct if there is only linear motion present.
(it completely restores the progressive frame!)

but on acceleration etc. it completely fails, because el=bl=0.5 do not fit anymore due to the non-linear motion (acceleration)

2nd method
- do some kind of full frame deinterlacing with good interpolation (edge directed is cool)
- I'll name those interpolated fields with '

then compensate as follows to recreate the missing fields
mvshow(cm=true) from A to B' and from C to B' to create B
mvshow(cm=true) from b to c' and from d to c' to create c
mvshow(cm=true) from C to D' and from E to D' to create D
mvshow(cm=true) from d to e' and from f to e' to create e
.
.
.
.

this compensation nearly ALWAYS fits the motion, what results in very less remaining combing, BUT mvshow(cm=true) tries to recreate field' and not the missed field. this means it has no advantage in resolution over the previously applyed full frame deinterlacer.

thus I need some kind of mix of both methods.

- measure the moving distance from A to b and C to b with mvtools
- set el=bl to this moving distance.

example linear motion:
A to b = 1.5 (value doesn't matter, it is just choosen to show my thoughts)
C to b = -1.5
==> el=bl=0.5

acceleration
A to b = 1
C to b = -2
==> el=bl=0.3333

slowdown
A to b = 2
C to b = -1
==> el=bl=0.6666

I hope this all wasn't too confusing...

[scharfis_brain]

If someone is interested in a completely naked uncorrected motion compensated bobber, here it is:


function mvbob(clip i)
{
# fieldseparation
j=i.separatefields()

# settings for motion compensation
pel1=4
nb1=4
mbl=0.05
wf1="uniform"
mpt1=10

# generate missing odd (even) field
x4= getparity(i) ? j.selectodd() : j.selecteven()
x5=x4.mvinterpolate(nb=nb1, bl=0.5-mbl, el=0.5+mbl, pel=pel1, wf=wf1, mpt=mpt1)
x6=interleave(x5,x4).duplicateframe(0)

# generate missing even (odd) field
y4= getparity(i) ? j.selecteven() : j.selectodd()
y5=y4.mvinterpolate(nb=nb1, bl=0.5-mbl, el=0.5+mbl, pel=pel1, wf=wf1, mpt=mpt1)
y6=interleave(y5,y4)

# weave them back together
getparity(i) ? interleave(x6,y6) : interleave(x6.trim(1,0),y6.duplicateframe(0))
assumefieldbased()
weave()
}

in most cases it fails matching the fields together, but if it matches motion, the results are impressive!

issues when motion isn't matched:
- jaggy diagonals
- weird combing
- blockyness
- scenechanges always are broken into two combed frames :(

so a lot of postprocessing has has to be done to knock out its artifacts

I think on scrolling interlaced end titles, this deinterlacing should work really good.

a sample pic, where mvbob matched most of the motion:
(look closer at the grey background!)
http://home.arcor.de/scharfis_brain/deint.jpg

[Fizick]

To Manao:
1. As I see in source, MVSHOW (in compensate mode) firstly copies current frame to destination, yes? That is why there are no black borders after compensation. I.e. the "compensation" is not honest! :)
2. At last I build makefile (by hand), and now can compile MVTOOLS with VC++ Toolkit 2003. But linker writes some warning about defaultibrary and LIBC. I correct it with this setting:

CXX=cl.exe
CXXFLAGS= /LD /Ox /O2 /G7 /D "NDEBUG" /D "WIN32" /D "_USRDLL" /D "_WINDOWS"
LINK32=link.exe
LINK32FLAGS= /dll /nologo /nodefaultlib:libc
ASM=nasmw.exe
ASMFLAGS= -f win32 -DPREFIX

If these settings is correct, why my MVTOOLS.DLL size is 148k (while your is 66k) ?

[Manao]

Hi,

First, a new version : MVTools-v0.9.3 (http://jourdan.madism.org/~manao/MVTools-v0.9.3.zip)

Changelog :

* Last cleanings in the search of the motion vectors. It should be slightly faster
* More search parameters can be set by the user, especially the search algorithm. See the documentation
* Server / client implemented. You now first have to use MVAnalyse, and then the filter you want. Look at the documentation and at the examples I'll give alter.
* MVCompensate is separated from MVShow ( it's more logic that way ). For the moment, it doesn't move the chroma ( same behavior as MVShow in the latest releases )
* Some cleaning in MVBlur / MVInterpolate / MVConvertFPS, but still some work to do. Now, MVBlur blurs around the frame, not between the frame and the previous one.
* Half of the work is done for writing vectors to a file. But the resulting file will be large ( around 500 MB - 1 GB I guess ).
* MVDenoise is slightly faster ( at least it should )
* Copies are optimized inside the filter, thanks to avisynth's copy functions.
* MVShow can display the mean SAD of the compensation ( using showsad = true )

----------

Now, some examples :

To show the motion vectors ( forward ):vectors = source.MVAnalyse(isb = false)
return source.MVshow(vectors)To show the backward one :vectors = source.MVAnalyse(isb = true)
return source.MVshow(vectors)To use MVMask :vectors = source.MVAnalyse(isb = false)
return source.MVMask(vectors)To use MVBlur :vectorsfw = source.MVAnalyse(isb = false)
vectorsbw = source.MVAnalyse(isb = true)
return source.MVBlur(vectorsbw, vectorsfw)It's the same for MVInterpolate, MVDenoise and MVConvertFPS.

One warning : do not use MVAnalyse on clips that have different resolutions inside the same script, it would cause issue ( I'm desperatly waiting for Avisynth 3.0 and it's ability to attach values to a clip... )

Another warning : pel > 1 is quite useless for the moment, since no filter have a use for nth pel interpolation.

@Fizick : in compensate mode, the result doesn't have any black borders, even if I don't do the copy. In this mode, I move the blocks pointed by the MV from the previous frame to the current one, where the blocks are not overlapping.

I don't have any warning with the libc.

My final size is shorter only because I use upx ( http://upx.sourceforge.net/ )

@Scharfi : I see your point. I'll see what I can do.

@all : I strongly recommend for the parameters of MVAnalyse fth = 40, it's really efficient for copping with noise. But now, you can also try de denoise the picture before giving it to the analyser.

[Boulder]

Manao,

scharfis_brain's excellent MVConvert60ito24p() function on the first page doesn't work with MVTools v0.9.3 (works with v0.9.2.x). What would I have to change in the function to make it work?

The error message is "invalid arguments to function "mvinterpolate" ".

[Manao]

function mvconvert60ito24p(clip x, int "mode")
{

mode = default(mode,2)
mbl=0.1

vectorsforward = x.mvanalyse(isb = false)
vectorsbackward = x.mvanalyse(isb = true)

y = x.mvinterpolate(vectorsbackward, vectorsforward, nb = 4, bl = 0.5 - mbl, el = 0.5 + mbl, wf = "hat")

interleave(y,x)
mode0=selectevery(5,2)
mode1=overlay(selectevery(5,3),selectevery(5,2),opacity=0.5)
mode2=overlay(overlay(selectevery(5,1),selectevery(5,3),opacity=0.5),selectevery(5,2),opacity=0.3)
mode3=overlay(overlay(selectevery(5,0),selectevery(5,3),opacity=0.5),overlay(selectevery(5,1),selectevery(5,2),opacity=0.5),opacity=0.5)

(mode==0) ? mode0 : (mode==1) ? mode1 : (mode==2) ? mode2 : mode3

}

[Boulder]

Thanks a lot:)

[Fizick]

It is possible to interpolate not bidirectional, not only forward or backward (use onny one vector set)?
Or maybe not interpolate, but do partial (not full) compensation?
I want try do motion compensation from frame n+2 and frame n-2 to frame n, and use these 3 frames for spot removing. In this case bloches will not corrupt the motion vector in its places.

[violao]

Fizick, I tried this, but it seems that the risk of "wrong" vectors is much higher when you skip frames in vector search. Initially I tried to estimate unreliable vectors (under blotches) by using 3rd SearchMVs between (n-1) and (n+1), but it appears from my initial test that better estimation is obtained by interpolating from healthy neighbourhood vectors.

[Manao]

New version : MVTools 0.9.4 (http://jourdan.madism.org/~manao/MVTools-v0.9.4.zip)

Changelog :

* Vectors can be saved to a file. In order to do so, add the parameter filename="C:\foo.bar" to the filter MVAnalyse. If the file doesn't exist, vectors will be saved into it. If it exists, vectors will be read from it. But, be warned :
- The file for a whole movie will be around 500 MB
- Saving / reading from a file need for the moment a linear access to the frames, so it has to be used only when encoding the movie, not when doing random access in it.
- The speed gain is not as great as one may hope, because SADs can't be saved ( it would take too much space ) and so have to be recomputed.
* The filter MVDenoise now works on 5 frames, and its parameters are now "thT" and "sadT" ( have a look in the documentation to see how they work ). It works nice ( very good for heavy denoising )
* The scene change detection thresholds have slightly changed. Now, a block has changed if its SAD it over thSCD1. The default for thSCD1 is 300, and for thSCD2 it is 130. It orks well ( better than the previous SCD engine ).

I'll have to implement random access with writing / reading from a file.

I haven't got time to look at your different propositions. However, I think I'll try to add a parameter to MVAnalyse, in order to change the gap between the two frames that are analysed ( thus allowing what Fizick is needing ).

Also, if somebody has a useful motion vectors compression scheme to propose, I'm listening, because I think a 500 MB file is still to much. Right now, I'm using an adaptive huffman coding, with prediction from the previous block. With that, I get an entropy around 1.6 bits / vector coordinate ( and the mean deepness of the huffman tree that is built is 1.65, so an arithmetic coding would not improve things by much ).

The specification for the scheme are :
* higher compression ratio than my method
* high speed, both to code and to decode

[scharfis_brain]

I have lots of problems with mvtools:

- v0.9.4. doesn't work at all.
it always claims : genericmotionfilter needs yv12 (at the line with mvinterpolate, mvblur, mvconvertfps or whatever), but the source is already yv12

- v0.9.2.x the nb parameter is useless, I do not see any difference.
the motionblur isn't getting better with higher nb
this means, on high motion, the motionblur is always ghosted instead of being one connected smoothed area..


btw. IMO it would give a MUCH better motionblur, if the image gets blurred directionally using 'simple' pixel mixing instead of block moving.
I mean, if you could just blur the pixels underlying a vector(field) into its direction, it should be THE ultimative mvblurrer...

[Manao]

Scharfi : can you provide the scripts you used ? I think you're messing up with the vector's stream.
I mean, if you could just blur the pixels underlying a vector(field) into its direction, it should be THE ultimative mvblurrer...I don't think so. First, you would need to interpolate the vector's field for all pixels, which would be slow. Then, you would have to copy & move a pixel along its vector, which would mean interpolate its position for each time we copy it. Slow again. And finally, if the vector's field was diverging, you would get strange results ( blocks tend to lessen that issue, they don't solve it all )

[mg262]

Manao - thanks for the great tools.
_______________________

I'm trying to use the motion compensation to set up the context for a temporal blur as follows:

Get a frame order like this:

10, 8, 10, 9, 10, 10, 10, 11, 10, 12,
11, 9, 11, 10, 11, 11, 11, 12, 11, 13,
...
n, n-2, n, n-1, n, n, n, n+1, n, n+2

Then forward search + MVCompensate, and select the odd frames so you get

8, 9, 10, 11, 12 (all mocomped to match 10 as closely as possible)
9, 10, 11, 12, 13 (all mocomped to match 11 as closely as possible)
etc.

____________________________________

(Then you can apply any blur with radius two and throw away 4/5 of the frames.)

Unfortunately the results aren't great... looking at each run (e.g. 8,9,10,11,12), there doesn't seem to be less motion than in the original. The script is below and if anyone has any insights into what's going wrong I'd appreciate them...

(I have tried playing with the fth parameter. Also, the frame numbers coming out of the output are correct.) Have I understood forwards/backwards correctly? Flipping to backwards doesn't help...
________________________________________

o = last

Interleave( o, blankclip(o, 2) + o, o, blankclip(o, 1) + o, o, o, o, trim(1,0), o, trim(2,0))#, o, blankclip(o))

v = MVAnalyse(isb=false, fth=0 )
MVCompensate(last, v)

selectodd()

_____________________________________

By the way, I'm wasn't sure if this should be a new thread, but all the information on these filters seems to be here, so I've stuck with it...

Mohan

[sh0dan]

Wow - this is probably the best stuff I've seen released for ages.

It has a LOT of potential for framerate conversion, deinterlacing and denoising!

Wow! Great work!

@scharfi: You need to explicitly give the first video parameter, etc:
vec = mvanalyse(fth=30, isb=true)
mvshow(last, vec)
This is because the vector clip is optional, and therefore your vector information is assumed to be the clip you'd like to apply your motion stuff on.

A bit about the implementation (feel free to ignore)

- I really like the idea of having motion information as a clip.

- Forward backward considerations. It seems a bit strange you need to do both backward and forward prediction. Why not simply alsways do forward prediction, and request the previous (cached) motion frame?

- Binary format. A separate index file, and a motion vector file would probably be the nicest implementation. When frame 'n' is requested:

1) Check if present in index. If yes, read MV's from file.
2) Calculate MV's.
3) Append MV information to binary file.
4) Write index of frame 'n' to index file.

That way you only need to keep the index (two integers per frame) in memory, and you get random access.

[MfA]

Motion compensating using the per block forward motion vectors (in terms of the direction in time) from the last frame leaves gaps in the present frame if you try to reuse it ... which complicates things.

[sh0dan]

Oh, I see - it makes sense using current frame as reference to the previous.

I noticed a very slow, but also very good motion blur by using:
vec_back=mvanalyse(fth=30, isb=true)
vec_forw=mvanalyse(fth=30, isb=false)

MVConvertFPS(last, vec_back, vec_forw, fps=100, fbw=2, thSCD2=255, thSCD1=200)
temporalsoften(2, 255, 255, 0, 2)
selectevery(4,0)
Assuming 25fps material. Works very well for my CG material.

[scharfis_brain]

and again,a feature request:

I want to repair framedrops using this script: http://forum.doom9.org/showthread.php?s=&threadid=79771

but, mvinterpolate returns garbage when trying to interpolate the missing frame.

I am going this way:

for progressive Video:

normal Sequence:

A B C D E F G

framedrop at frame D

A B C C E F G

my function interpolates the 50:50 position between C & E, but somehow the C C E F sequence messes up mvinterpolate because C is a dupe.

the same with laced video:

normal Sequence:

Aa Bb Cc Cc Ee Ff Gg

here, I interpolate between c and E to get two frames with the weights
c*0.66+E*0.33 = D
and
c*0.33+E*0.66 = d

but like said before the dupes (or with interleced: forward-backwards stutter) makes mvinterpolate going messy.

would there be any solution for that?

maybe included in the plugin itself (dropped frames are almost indentcal, so detection is very easy!)

the function I posted in the thread works for some low motion video, but completely fails on high motion / short shutter speed video...

also burst framedrops (several framedrops in one sequence: A B C C C C G H) could be handled, too.

what do you think manao?

[Manao]

Detecting dup is fairly easy with the motion vectors and the sad. However, the rest of the processing would take a long time to code ( not complicated however ), and I don't have a much time as before ( internship, not a student anymore :( ).

I'm still working on the motion estimation ( I'm adding a padding of the reference frame, to better catch objects that come into the frame, as it is done in MPEG-4 ). Halfpel works again, it is faster than before ( fast enough to make useless the saving to file feature ), but no filter is using it yet.

[morsa]

Can this be thought in a more general way?
I mean, thinking about interpolating dropped frames, I got to the idea of interpolating missing frames from old films.
You know those portions of a film where 3, 5, 10, frames have been lost.In this case whe don't have a dupe frame, So I guess we should give the filter parameters like first frame, last frame and how many frames to be interpolated.
Is this posible?

At this moment we have:
DeScratch
Despot
Equlines
RemoveGrain
RemoveDirt

So what we lack is what Scharfis is asking (sorry Scharfis if I miss something)

[Manao]

morsa : it's already possible ( just not quite straightforward ) :
function MVInterpolateBetween(clip c, int "first", int "last", int "number")
{
number = Default(number, last - first)
beginbase = (first > 0 ) ? first - 1 : 0
endbase = ( last < framecount(c) ) ? last + 1 : framecount(c)

newfps = framerate(c) * number

beginning = c.trim(0, first)
ending = c.trim(last, 0)
base = c.trim(beginbase, first) + c.trim(last, endbase)

mvf = base.MVAnalyse(isb = false, lambda = 2000)
mvb = base.MVAnalyse(isb = true, lambda = 2000)

reconstructed = base.MVConvertFPS(mvb, mvf, fps = newfps)
\.trim(number + 1, 2 * number - 1).AssumeFPS(framerate(c))

return beginning + reconstructed + ending
}

Edited : first version was not working correctly.

[Dee]

to Manao: do you plan to release sources of most recent version (and future versions)?

[Wilbert]

What do you mean? The sources are included in 0.9.4.

[Manao]

I've released a testing version because E-Male was making a test between several temporal denoisers ( http://forum.doom9.org/showthread.php?s=&threadid=79706 ). But the code was being heavily modified at that time, so I decided not to release the source for that particular un'official' version. The next 'real' release will have the source code, of course.