SlowMo with Motion Interpolation (MSU FRC and MVFlowFps) experiment & help... [Archive]

View Full Version : SlowMo with Motion Interpolation (MSU FRC and MVFlowFps) experiment & help...

Undead Sega

23rd August 2010, 05:00

As the title says, recently I was experimenting on making some slow motion scenes with a method that I mentioned here and there and also that has been used by some others with those who dont actually own a high speed camera but something like a Canon HV30 (which I own :D).

A often used example is the flow of water being poured or splashed, this was the case with me. I shot the necessary footage and I actually first tried it out with MSU's Framerate converter and saw the results were quite good after that I would search for my previous threads which we discussed about frame interpolation was given a certain script as well using MVTool's MVFlowFps as I asumed was a better interpolator than MSU's filter, this is what I was using:

MSU Framefrate converter:
MSU_FRC(10, "slow")

MVFlowFps:
a=assumefps(50)
vf=a.mvanalyse(isb=false, overlap=4, search=3, pel=4)
vb=a.mvanalyse(isb=true, overlap=4, search=3, pel=4)
mvflowfps(last,vb,vf, num=500)

my workflow was taking 50i footage and bob deinterlacing it to 50p, that progressive clip is then inputted to the formentioned filters and given a 10x the amount of frames = 500fps. From there, before it is saved as a lossless avi, the speed is adjusted in Virtualdub to 25p.

My question to all of this is, after looking at the latter filter's results, while I believe the interpolation's quality is better, it did not behave or move very linear, meaning that the motion from one frame to the next few looked like if it morphed suddenly and constantly, this gives a weird and unrealistic flow to the bubbles in the water and etc.

I have made and uploaded a split screen comparision video of the results, both on sync in everyway:

http://www.megaupload.com/?d=P5Q2EOX5

Therefore, I was wondering if there is a way to resolve this look or its just the nature of the MVTools filter I'm using?

I hope you understand what I'm talking about, I look forward hearing back. Thanks.

EDIT: For those who can't wait for the download, please watch it in 1080p here: http://www.youtube.com/watch?v=4VlFaZcahh8

mgh

23rd August 2010, 05:59

Emulgator

23rd August 2010, 15:06

Undead Sega:
I found the same when using MFlowInter interpolating more than 1 interframe with steadily increasing "time" parameter.
The first interframes advance too slow to fit into linear motion.
So from the last interframe to the next real frame is almost always a step.
Not sure about the reason yet, though...

Undead Sega

23rd August 2010, 15:09

this is what i use

Function slowdown( clip clip, int c ) {
fn=FramerateNumerator (clip)
fd=FramerateDenominator(clip)
b=fd*c
source=assumefps(clip,fn,b).killaudio()
backward_vec = source.MVAnalyse(isb = true, truemotion=true, pel=2, idx=1)
forward_vec = source.MVAnalyse(isb = false, truemotion=true, pel=2, idx=1)
cropped = source.crop(4,4,-4,-4) # by half of block size 8
backward_vec2 = cropped.MVAnalyse(isb = true, truemotion=true, pel=2, idx=2)
forward_vec2 = cropped.MVAnalyse(isb = false, truemotion=true, pel=2, idx=2)
return source.MVFlowFps2(backward_vec,forward_vec,backward_vec2,forward_vec2,num=fn,den=fd,idx=1,idx2=2)
}

How does this differ from the original script I am using?

Undead sega:
I found the same when using MFlowInter interpolating more than 1 interframe with steadily increasing "time" parameter.
The first interframes advance too slow to fit into linear motion.
So from the last interframe to the next real frame is almost always a step.
Not sure about the reason yet, though...

Ahhh okay, so it looks like i am nto the only one who thinks of this. How can I resolve this script in giving me a linear motion?

Emulgator

23rd August 2010, 15:16

mgh said MVFlowfps2.
2 vector clips normal, 2 vector clips shifted.
Is marked by Fizick as superceded by MVFlowfps, so obsolete, but if it works better, why not ?
Have not yet tried that one.

um3k

23rd August 2010, 15:19

You should probably look into updating to MVTools 2, which prefixes its filters with "M" instead of "MV" (I.E. MFlowFPS). Not sure how much the quality will improve, but I imagine it'd be worth a try.

Emulgator

23rd August 2010, 15:27

I am mainly working with MVTools 2.5.10.1, the behaviour is the same there.

Undead Sega

24th August 2010, 05:08

Dam its the same with MVTools 2??? Surely there must me a way to tweak the script to get more of that linear motion look resembling the MSU's filter?

poisondeathray

24th August 2010, 05:42

Didée

24th August 2010, 07:20

The most obvious observation is that both tools fail at doing their intended job. The motion in the original source is too chaotic, both tools do not manage to actually find the correct motion. For the most part, you're just seeing how the tools are doing frame blending.
Also, related to the not-found motion, there is this "bing!" or "switch"-effect everytime when a new original source frame is reached. It seems to me that this effect is more pronounced with the MSU tool.

Undead Sega

24th August 2010, 18:21

I see what you're saying, I think it's more evident near the end of the clip , when all the water is splashing around in the glass. The generated frames don't seem to be evenly sampled or spaced in time, so it gives the appearance of more choppy motion.

Just curious - What makes you say that you "believe the interpolation quality is better" with mvtools ? Is it this clip or something else ?

I think I understand what you mean, and it would be more evident near the end of the clip, however i was focusing more of when the water splashes inside the glass forming the bubbles, there is what i am talking about, if you see it closely compared to MSU's, the bubbles just morphs from one frame to the other, like it shrinks and expands constantly in shapes instead of giving a continuous flow similar to MSU's.

And reason I said that (silly) was I would assume that the MVTool's algorithm in motion interpolation would be superior as its one that's constantly tweakable and developing.

The most obvious observation is that both tools fail at doing their intended job. The motion in the original source is too chaotic, both tools do not manage to actually find the correct motion. For the most part, you're just seeing how the tools are doing frame blending.
Also, related to the not-found motion, there is this "bing!" or "switch"-effect everytime when a new original source frame is reached. It seems to me that this effect is more pronounced with the MSU tool.

I know the motion in the footage is 'chaotic' as you described, and wasnt looking for perfect results either, however its just the way its doing its job and givining me this weird motion. Isnt there a way to input a deeper analysis onto the footage to get closer to the correct motion?

Also, im dont entirely understand to know this "bing!" or "switch"-effect, could this be with what I've described or mentioned above?

Didée

24th August 2010, 18:59

Also, im dont entirely understand to know this "bing!" or "switch"-effect, could this be with what I've described or mentioned above?
Dunno how to explain it much better ... but if anything, then it's the exact opposite of what you are issuing. When watching your clip at its native speed, I do not notice any "uneven" spacing in MVTools' interpolation. But when I concentrate on either clip, then the MSU result gives me this ... switch! ...aaand... switch! ...aaand... switch! ..... experience. It's also present in MVTools' interpolation, but there it's less noticeable to me. In fact, all-in-all the MVTools interpolation seems more smooth to me, probably because the stronger switch-effect of MSU is more disturbing to me.

That's my purely subjective rating. Other viewers might have a completely different experience with this!

(I'll see if I can get my wife as a test viewer when she's home again. Be guaranteed, she has not the slightest clue about what "MSU" or "MVFlowFPS" possibly could be.) ;)

Didée

25th August 2010, 01:33

"Darling, have a look at this left/right comparison. Do you see any difference between them? Does either one seem better or worse than the other?"

"Hmmmhhhh" ... "the left one is brighter, the right one is darker!"

"What? Oh, you're right. Okay, forget about that. Pay attention to how things are moving. Any difference?"

"Okay..." - [watches for some seconds] - "hmmhh, the left is more choppy, isn't it? The one to the right is smoother!"

The conservation was in German instead of English, and slightly more elaborative ... but in essence, it was exactly like this. Apart from drawing her attention from colors to motion, I did not give any further hints.

Undead Sega

25th August 2010, 03:54

Hmmm that is very interesting, because they are exactly the same footage so therefore one cant be brighter than the other (unless the interpolation messes with the colours???)

Okay, the one on the left might be 'choppy' as one would describe, thats probably because its doing more frame blending than interpolating (or vice-versa) but this is where it can get abit confusing, I will post another upload of the same video but a zommed in version of the glass, hopefully the results will be more noticable.

Take alook from 1minute of the footage (I hope you are watching it in 1080p at fullscreen and running it on a up to date PC hardware-wise :D), and focus on the bubbles, forget about the interpolation on a serious note and hopefully you wil see that the bubbles morph from one to another which doesnt look like as if they were flowing naturally. Ok maybe all that may have sounded abit silly however imagine watching it in realtime, you'll probably see the bubbles shifting from one area to another weirdly.

Once again, in relation to my script, even after reading up the documentation files for MVTools, i still dont quite understand its function or its tweakable settings.

Also, what is the function 'MVFlowInter'? it is described as the Motion interpolation function like 'MVFlowFps', are there multiple motion interpolation functions in MVTool??? :D

poisondeathray

25th August 2010, 04:11

Also, what is the function 'MVFlowInter'? it is described as the Motion interpolation function like 'MVFlowFps', are there multiple motion interpolation functions in MVTool??? :D

mflowinter (or mvflowinter) is usually used for replacing bad frames with motion interploated ones . But in general context it's just a motion interpolation function. According to the documentation, mflowfps, mflowinter, mflow (all from mvtools2) all use the same pixel based motion compensation.

I think they both look odd , both "bad" or unnatural, but in different ways

If you use other material , you can see similar morphing artifacts at the edges from motion prediciton errors - it's actually more easy to see the errors on "plain" scenes, like a car driving or person running

I've never seen super duper clean , nearly perfect interpolation, except on very slow clips where there is low motion. They all have weird artifacts, and it's worse when you interploate more frames (you're doing 10x)

If you look at other motion interpolation solutions, they usually use foreground and background mattes to improve the interpolation (to reduce the weird artifacts)

Undead Sega

25th August 2010, 05:44

Upload of zoomed in version of the original upload:

http://www.megaupload.com/?d=COSYL529

For those who cant wait, youtube version:

http://www.youtube.com/watch?v=Kf7kZ6iqRd4

It is almost 6am here, i will do a full reply when i get some sleep and after breaking my Fast as well. In the meantime, check them out (as I havent yet properly due to the current time) and I look forward hearing back.

Emulgator

25th August 2010, 18:29

poisondeathray

25th August 2010, 18:41

another option is to use an external clip for the motion estimation ; mvtools2 does support this

some sources are problematic , e.g. might be too noisy, or levels washed out etc... that interferes with the motion estimation. So you would filter the clip (e.g. adjust levels, denoise etc...) and use that subclip as the detection clip , and apply those calculations to the original clip

I'm guessing your source might have problems with the translucence nature of the water droplets, so you might filter it in some fashion (maybe crush the levels so they are more opaque?) or do some other filtering such that the droplets are less translucent for the detection

Undead Sega

27th August 2010, 12:08

Undead Sega, you may play around with
blksize (4 gives more disturbed motion, 8 is default, 16 is smoother),
search (effort for searching vectors, I use 5)
ml (exclusion mask, 100 blends more, is softer, lower values exclude more)
dct (it may be better to read the doc, it is too much to describe here)
thSCD1 (threshold that decides from which SAD on a block is considered changed)
thSCD2 (threshold that decides from which percentage of changed blocks per frame the frame is considered a scenechange,
127 being 50% and 255 being 100%)
The latter two decide when a scenechange is thrown, which will be trigger a blend if bld=true (what we don't need here).
MAnalyse has lots of parameters, the documentation that came with mvtools has them all.

Additionally you may comment in either

#MShow(super, forward_vectors)
or
#MShow(super, backward_vectors)

These show you what vectors referencing the shown frame have been calculated.

When for a particular frame no motion vectors within the given parameter range could be found,
the vector overlay will be empty, meaning this frame is interpreted as a scenechange.
Now you may tweak until vectors are there again.

Took me quite a while to find a setting that would suit my needs for interpolating missing/damaged frames.

Thanks very much for this, i will definately give it a try, however will it give the same result that you and me had described in the earlier posts?

Either way, I'll give it a try, but which filter is this exactly for? MVFlowfps2 or something perhaps?

another option is to use an external clip for the motion estimation ; mvtools2 does support this

some sources are problematic , e.g. might be too noisy, or levels washed out etc... that interferes with the motion estimation. So you would filter the clip (e.g. adjust levels, denoise etc...) and use that subclip as the detection clip , and apply those calculations to the original clip

I'm guessing your source might have problems with the translucence nature of the water droplets, so you might filter it in some fashion (maybe crush the levels so they are more opaque?) or do some other filtering such that the droplets are less translucent for the detection

I filmed this as clean as possible even though it was filmed with a shutter speed of 1/2000, it was filmed outdoors where there wa enough light and thus grain wsant too visible. In addition, it was bob deinterlaced with TGMC at a low denoising setting.

In regards to the transparancy of the water droplets, if making them more opaque will probably improve detection is it possible to use that information of interpolation and process it onto the original footage?

poisondeathray

27th August 2010, 14:48

I filmed this as clean as possible even though it was filmed with a shutter speed of 1/2000, it was filmed outdoors where there wa enough light and thus grain wsant too visible. In addition, it was bob deinterlaced with TGMC at a low denoising setting.

I wasn't saying your footage was noisy. I was giving textbook examples where people use an external clip fo the interpolation

I'm guessing because you have translucence and specular highlights, the algorithm can't detect the edges correctly. As a result you get "edge dragging", and morphing.

In regards to the transparancy of the water droplets, if making them more opaque will probably improve detection is it possible to use that information of interpolation and process it onto the original footage?

That was the whole point of the post :) mvtools2 supports using an external clip for the interpolation

Think about it: if it was a pure black and white image or "drops" or whatever you're trying to track, there would be less difficulty finding edges , so you wouldn't get as much morphing around the edges

You might be able to do some other manipulation to clear up the interpolation with a subclip, I was just thowing out changing levels or somehow altering it for the external clip

The other technique people use is mattes for foreground and background. This is especially useful when objects cross each other (the algorithm can't tell what object is what when objects overlap) . But mvtools2 doesn't support this feature AFAIK

Undead Sega

28th August 2010, 16:03

I wasn't saying your footage was noisy. I was giving textbook examples where people use an external clip fo the interpolation

I'm guessing because you have translucence and specular highlights, the algorithm can't detect the edges correctly. As a result you get "edge dragging", and morphing.

That was the whole point of the post :) mvtools2 supports using an external clip for the interpolation

Think about it: if it was a pure black and white image or "drops" or whatever you're trying to track, there would be less difficulty finding edges , so you wouldn't get as much morphing around the edges

You might be able to do some other manipulation to clear up the interpolation with a subclip, I was just thowing out changing levels or somehow altering it for the external clip

The other technique people use is mattes for foreground and background. This is especially useful when objects cross each other (the algorithm can't tell what object is what when objects overlap) . But mvtools2 doesn't support this feature AFAIK

Ohhh right, okay I get what you mean, hahaha sory about that, but yeah I can understand what you mean, because of the randomness of the water flow and droplets (in addition its tansparency), it makes it harder for the interpolation when there is no edges, thus the artifacts?

...and oh wow, i did not know that MVTools2 was capable of working with an external clip (hmmm I must be abit behind the times here :D) but i suppose thats great then, however i would like to ask, if an external clip was used, would MVTools2 fully depend on that clip? or using it as support thus it analyzes both the clips for interpolation?

In this case to try and improve with the current clip(s) I have in order to get more define edges is proving abit tricky because of the semi-transparency of the water, I do not know wexactly what to do except I brought up the shadows abit so it exposes more of the flowing water from the top. I know that isnt enough but i could do something bizarre by adding a photoshop filter where I can 'cartoonize' it where it will have flat colours and black lines on the edges (but hardly on the pouring water). What do you think? :D

Original frame:

http://img837.imageshack.us/img837/1865/slowmowateroriginalfram.png

Shadow enhanced frame:

http://img716.imageshack.us/img716/7947/slowmowatershadowenhanc.png

poisondeathray

28th August 2010, 16:21

That's just my guess as to why the results aren't "great" . A solid object with clear borders moving slowly , linearly, and predictably would likely have fewer problems - it makes sense to me.

I don't know , try it out on a sample. I've never done water before , or used mvtools with external clip method, I was just browsing the documentation the other day and surprised to see that it did.

Those techniques (mattes, external clips, point guiding) definitely helps with other tools to get more accurate results, especially with edge dragging and morphing around edges - which is the #1 problem with any motion estimation. With the other tools, it mimizes the ME errors around the edges, so I suspect that's what mvtools2 would do too.

But I'm not sure how you would treat water drops specifically - I was just throwing ideas out for you to try . Translucency , specular highlights, complex rapid motion (like water drops) are mentioned as being problematic for motion estimation in other documenatation (e.g. twixtor's manual).

Undead Sega

28th August 2010, 16:52

I will definately give it a try, just need to see if i have all the right plugins for the filter, especially for that recommended script a few posts earlier on.

I am myself browsing quickly through the MVTools documentation, I cannot quite find where it is mentioned about external or subclip, can you point it out for me if thats okay? :D

I sort of gave it a try in making a matte but because of the water it doesnt exactly work in giving me a matte of what i would want i.e. the water being most likely white.

Another frame example (dont know if this one's acceptable :D :

http://img832.imageshack.us/img832/4711/slowmowaterbizarreframe.png

poisondeathray

28th August 2010, 16:58

there's some examples at the bottom , look for prefiltered clip with denoising example or external interploation clip example. I think you would just modify it a bit , but as I said earlier I haven't done this method with mvtools2 so I don't know the specifics

Undead Sega

28th August 2010, 17:33

To use prefiltered clip for more reliable motion estimation, but compensate motion of not-prefiltered clip (denoising example)

AVISource("c:\test.avi") # or MPEG2Source, DirectShowSource, some previous filter, etc
# Use some denoiser (blur) or deflicker for prefiltering
prefiltered = blur(1.0)
super = MSuper(levels=1) # one level is enough for MCompensate
superfilt = MSuper(prefiltered) # all levels for MAnalyse
backward_vectors = MAnalyse(superfilt, isb = true)
forward_vectors = MAnalyse(superfilt, isb = false)
# use not-prefiltered (super) clip for motion compensation
forward_compensation = MCompensate(super, forward_vectors)
backward_compensation = MCompensate(super, backward_vectors)
# create interleaved 3 frames sequences
interleave(forward_compensation, last, backward_compensation)

DeGrainMedian() # place your preferred temporal (spatial-temporal) denoiser here

selectevery(3,1) # return filtered central (not-compensated) frames only

Could it be something like this? I did another quick search through the MVTools documentation.

Therefore im trying to figure out where would i place the external or 'prefiltered' clip in the script.

Undead Sega

30th August 2010, 04:42

anyone may i ask?

again, this was an example from the MVTools documentation.

pbristow

30th August 2010, 17:17

You see the line saying "prefiltered = blur..." ?

Just replace blur with whatever function(s) you want to use.

Undead Sega

30th August 2010, 19:01

You see the line saying "prefiltered = blur..." ?

Just replace blur with whatever function(s) you want to use.

Oh okay then, but is this what i was looking for?

and what kind of functions can you replace on there?

pbristow

30th August 2010, 22:13

"Oh okay then, but is this what I was looking for?"

Only you can answer that. =:o}

"and what kind of functions can you replace on there? "

Whatever you think might do what you want.

I have to say, the problem you're trying to address here may be fundamentally insoluble. Think about it: when water moves across a backround of other objects/textures, the movement of the details of what's seen *through* the water is not very simply related to the movement of the water itself. There are also *reflections* off the surface of the water which are rapidly appearing, warping, and disappearing. To fully analyse the motion going on here you'd need a processs that can see at parse-out at least two layers of motion information, maybe three.

Myself, I've only got about halfway to figuring a technique for two-layer movement analysis with MVtools, which I wanted for scenes where smoke is drifting across the main action. The approach I tried is to initially treat the smoke as noise, thus persuade MVtools to lock onto the "solid" details and track those, and thus use MDegrain to eliminate the smoke (almost completely) from the picture. That was the "main" layer. Then, by subtracting this smokeless version from the original, I got a separate layer which was just the smoke (and any actual video noise). Those two separate clips got fed through MVtools to interpolate their motions, then recombined.

The result was... well... promising, but not terrific.

And that was just a layer of translucent stuff drifting in front of otherwise unaffected background. I didn't have reflections and refractions to take into account.

In order of increasing difficulty, I would say these are the problems taht existing motion-detection algorithms aren't really equipped to handle:

1. translucent, non-reflective, steadily moving material (such as non-turbulent smoke) moving in front of main action.

2. translucent, non-reflective, warping material (such as turbulent smoke) moving in front of main action.

3. Transparent but reflective sheet of solid material moving in front of main action (e.g. a glass door closing between the camera and the background). This creates a layer of reflections, variably translucent, moving in a consistent fashion across the background.

4. Transparent but reflective solid material of variable thickness/curvature, moving in front of main action.
Same problems as 3, but with added problem of refraction (distortion) of the background as the glass moves.

5. Transparent put reflective fluid moving in front of main actions. Same probelms a four, but with more complex/fluctuating movement/warping of both reflections and refractions.

The problem you're trying to tackle, moving water, is the toughest of these five!

There's another fundamental problem that occurs whenever a textured surface moves further between frames than the average spacing of the details that make up the texture: It becomes impossible for the motion estimator to determine exactly which feature of the texture in frame one corresponds to which feature in frame 2, etc., unless it somehow understands the boundaries of the *object* that the texture is part of.

When water is pouring, for example falling down a cliff, its surface becomes textured in way that can raise this problem... but with the added complication that the "object" - the stream of water - is constantly changing shape.

So all in all, um... Get stuck in to studying the docs for all the filters you think might help, and good luck! :) You'll need to be prepared to really get to grips with the parameters of the various MVtools functions, why the're there and what they do.