Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > VapourSynth

Reply
 
Thread Tools Search this Thread Display Modes
Old 28th April 2017, 23:35   #1  |  Link
Joachim Buambeki
Registered User
 
Join Date: May 2010
Location: Germany, Munich
Posts: 49
Looking for help with script for weighted averaging of frames for smooth motion blur

Hello,

It's been almost ten years since I last used Avisynth and so far I am still trying to figure out how to run Vapoursynth(VS) on OSX, so please excuse my beginner level questions.

I would like to process 360 shutter high frame rate sources into standard frame rate clips by averaging multiple frames to get smooth motion blur. The attached image represents what I would like to do in VS:
  • each blue column represents one frame
  • height of column represents weight
  • negative height means the image gets subtracted with the weight of its height
  • the number of blue bars in the yellow, dotted lines represents the speedup or timebase conversion factor (5 by default but can be everything)
By counting pixels I came up with the (approximated) weights for each of the 19 example frames in the attachment:
-.01, -.03, -.03, -.06, 0, .17, .42, .69, .91, 1.0, .91., .69, .42, .17, 0, -.06, -.03, -.03, -.01



I understand averaging the five frames within the yellow, dotted verticals can be done like this with Avisynth (AS) but that is as far as I've just gotten.
Code:
c1=selectevery(last, 4, 0)
c2=selectevery(last, 4, 1)
c3=selectevery(last, 4, 2)
c4=selectevery(last, 4, 3)
c5=selectevery(last, 4, 4)

Average (c1, .69, c2, .91, c3, 1, c4, .91, c5, .69)
I assume that something like:
Code:
SelectRangeEvery(clip, 5, 19, x) #19 calls of SelectRangeEvery where x=0 to 19
#and
Average(c1, -.01,c2, -.03, c3, -.03, c4, -.06, c10, 1.0, ... c18, -.03, c19, -.01)
would do the trick in Avisynth but I can't wrap my head around how do this in VS (nor how to do exactly in AS).

To take this concept one step further - how to describe the frame weighting as a function, so faster speedup factors, different kinds of artificial motion blur (like asymmetrical functions or simulated shutter speeds of >360) and keyframes are possible?
Unfortunately on top of my lack of advanced scripting skills, the math (I guess AV's "Apply()" would be appropriate for this?) needed for functions is not my strong suit. :-(

Additionally, since sometimes it is technically not possible to shoot 360 shutter but only around 180-270, is there an elegant way to pad this to 360 motion blur with MVtools, then apply the frame averaging (which should eliminate most if not all artefacts introduced by the fake 360 shutter motion blur)?

I hope it became apparent what I am trying to achieve, if not - feel free to ask questions and I will do my best to answer them.
Looking forward to what you guys can come up with and thanks in advance for looking into this. :-)

JB

Last edited by Joachim Buambeki; 30th April 2017 at 20:10.
Joachim Buambeki is offline   Reply With Quote
Old 29th April 2017, 02:42   #2  |  Link
Joachim Buambeki
Registered User
 
Join Date: May 2010
Location: Germany, Munich
Posts: 49
reserved
Joachim Buambeki is offline   Reply With Quote
Old 29th April 2017, 09:33   #3  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 7,097
Some AVS links to threads here.
https://forum.doom9.org/showthread.p...ight=ClipBlend
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 30th April 2017, 16:42   #4  |  Link
Joachim Buambeki
Registered User
 
Join Date: May 2010
Location: Germany, Munich
Posts: 49
Quote:
Originally Posted by StainlessS View Post
Thanks for the links StainlessS. I read all of the subsequent ones but I am still having a hard time trying to figure out how to do it in AVS, let alone VS.
All those examples apply equal weight to the blending except this one, don't they?

Can I do weighted blending with your plugin?

Last edited by Joachim Buambeki; 30th April 2017 at 17:00.
Joachim Buambeki is offline   Reply With Quote
Old 30th April 2017, 17:00   #5  |  Link
raffriff42
Retried Guesser
 
raffriff42's Avatar
 
Join Date: Jun 2012
Posts: 1,377
Quote:
Originally Posted by Joachim Buambeki View Post
All those examples apply equal weight to the blending except this one, don't they?
I wasn't gonna post that because it's so hacky. It does work; I still use it occasionally. There's an improved version here with variable decay rate.

You might like this tip: I tuned the weighting with ShowFrameNumber(scroll=true) to identify all recent frames and judge the amount they are showing through.
raffriff42 is offline   Reply With Quote
Old 30th April 2017, 17:15   #6  |  Link
Joachim Buambeki
Registered User
 
Join Date: May 2010
Location: Germany, Munich
Posts: 49
I just remembered that attachments take forever to get approved and uploaded the image to an imagehoster.

Quote:
Originally Posted by raffriff42 View Post
I wasn't gonna post that because it's so hacky. It does work; I still use it occasionally. There's an improved version here with variable decay rate.
Thanks for chiming in. Can you make the script dynamic so one can specify the speedup and it internally calculates the necessary values? Otherwise a separate script is needed for each speedup factor. This then would also allow to use different functions.

Quote:
Originally Posted by raffriff42 View Post
You might like this tip: I tuned the weighting with ShowFrameNumber(scroll=true) to identify all recent frames and judge the amount they are showing through.
My idea was to later add a debugging mode that generate white bars or circles on a grey (RGB=127) background paired with numbers (ranging from 0 to the amount of frames used for the used blend mode and speedup factor) to illustrate how the blending works, this would make it easy to see how the positive and negative weights are applied to each frame. What do think about this?

Also, what about precision? If you call overlay(), merge() or similar again and again, doesn't that hurt precision? If the weighted merging would be done in a separate plugin like StainlessS's Clipblend() with 32bit internal precision, wouldn't that be favourable?

Last edited by Joachim Buambeki; 30th April 2017 at 17:20.
Joachim Buambeki is offline   Reply With Quote
Old 30th April 2017, 17:20   #7  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Ikea Chair
Posts: 2,002
Quote:
Originally Posted by Joachim Buambeki View Post
I just remembered that attachments take forever to get approved and uploaded the image to an imagehoster.

Thanks for chiming in. Can you make the script dynamic so one can specify the speedup and it internally calculates the necessary values? Otherwise a separate script is needed for each speedup factor. This then would also allow to use different functions.

Also, what about precision? If you call overlay(), merge() or similar again and again, doesn't that hurt precision? If the weighted merging would be done in a separate plugin like StainlessS's Clipblend() with 32bit internal precision, wouldn't that be favourable?
c.misc.AverageFrames(clip, [-1, -3, -3, -6, 0, 17, 42, ...], scale=0.01)

I guess this is what you want. And if it isn't then I'm not sure just how you want to combine those frames.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 30th April 2017, 17:45   #8  |  Link
Joachim Buambeki
Registered User
 
Join Date: May 2010
Location: Germany, Munich
Posts: 49
Quote:
Originally Posted by Myrsloik View Post
c.misc.AverageFrames(clip, [-1, -3, -3, -6, 0, 17, 42, ...], scale=0.01)

I guess this is what you want. And if it isn't then I'm not sure just how you want to combine those frames.
Thanks for commenting Myrsloik.
Yes, that is part of what I want, the problem is that with your proposition I merge the current and 9 adjacent frames into one, then merge the next 19 and so on, leaving me with a speedup factor of 19.
What I want to do is take merge x frames (the number of frames "x" represents depends on the speedup factor and what kind of function is called - as said above, asymmetrical functions are also possible). Then move on by y frames (y is the speedup factor and always <x) and repeat.
I assume the only thing missing to achieve what describe in my first post is a helper script that calculates the weighting based on a given function and the desired speedup factor. I know you can just put variables (that get calculated and served by the helper script) into the "AverageFrames()" call but how to do you implement varying frame amount? Would you just add a very high amount of variables into the AverageFrames()" call, where the first and last variables are zeros (if the function works with less frames) or is there a more elegant way to do this?

To put it another way, I want to speed up the footage by a factor of y but use x amount of frames for each blend (where x>y), so the motion blur (trails) overlap from frame to frame.

I hope it became clear what I am trying to do, please bear with me and my limited abilities to explain what my goal is.

Last edited by Joachim Buambeki; 30th April 2017 at 17:54.
Joachim Buambeki is offline   Reply With Quote
Old 30th April 2017, 18:14   #9  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Ikea Chair
Posts: 2,002
Quote:
Originally Posted by Joachim Buambeki View Post
Thanks for commenting Myrsloik.
Yes, that is part of what I want, the problem is that with your proposition I merge the current and 9 adjacent frames into one, then merge the next 19 and so on, leaving me with a speedup factor of 19.
What I want to do is take merge x frames (the number of frames "x" represents depends on the speedup factor and what kind of function is called - as said above, asymmetrical functions are also possible). Then move on by y frames (y is the speedup factor and always <x) and repeat.
I assume the only thing missing to achieve what describe in my first post is a helper script that calculates the weighting based on a given function and the desired speedup factor. I know you can just put variables (that get calculated and served by the helper script) into the "AverageFrames()" call but how to do you implement varying frame amount? Would you just add a very high amount of variables into the AverageFrames()" call, where the first and last variables are zeros (if the function works with less frames) or is there a more elegant way to do this?

To put it another way, I want to speed up the footage by a factor of y but use x amount of frames for each blend (where x>y), so the motion blur (trails) overlap from frame to frame.

I hope it became clear what I am trying to do, please bear with me and my limited abilities to explain what my goal is.
Simply add c.std.SelectEvery(clip, cycle=y, offsets=0) assuming you only want to keep 1 in y frames. You can pass any vector with up to 25 values to AverageFrames. There's no need to add zeroes at the ends.
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 30th April 2017, 18:43   #10  |  Link
Joachim Buambeki
Registered User
 
Join Date: May 2010
Location: Germany, Munich
Posts: 49
Thanks.

Quote:
Originally Posted by Myrsloik View Post
Simply add c.std.SelectEvery(clip, cycle=y, offsets=0) assuming you only want to keep 1 in y frames.
So my script would look like this?
Code:
Helperscript() #outputs weight (w1, w2, w3,...,w99) for each frame
c.misc.AverageFrames(clip, [w1, w2, w4, ...,w99]) 
c.std.SelectEvery(clip, cycle=y, offsets=0)
If yes, wouldn't that average x/y more frames than necessary if they are discarded afterwards anyway?
Quote:
Originally Posted by Myrsloik View Post
Simply add c.std.SelectEvery(clip, cycle=y, offsets=0) assuming you only want to keep 1 in y frames. You can pass any vector with up to 25 values to AverageFrames. There's no need to add zeroes at the ends.
Does that mean "AverageFrames()" can only process 25 frames? If yes, could this be increased or do I need to call it multiple times and merge them afterwards?

TIA!

Last edited by Joachim Buambeki; 30th April 2017 at 20:56.
Joachim Buambeki is offline   Reply With Quote
Old 30th April 2017, 19:16   #11  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 7,097
Quote:
Originally Posted by Joachim Buambeki View Post
Can I do weighted blending with your plugin?
Sorry, no. and I gave up on trying to do weighted blend as I could not figure out a fast method to do it.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 30th April 2017, 20:34   #12  |  Link
Joachim Buambeki
Registered User
 
Join Date: May 2010
Location: Germany, Munich
Posts: 49
Quote:
Originally Posted by StainlessS View Post
Sorry, no. and I gave up on trying to do weighted blend as I could not figure out a fast method to do it.
Too bad.

@myrsloik, does AverageFrames() have any disadvantages to Clipblend() regarding precision?

That image I posted is a polynomial of degree 4, right?! What would be an approximate function to describe it and how do I use this function to derive the weight of each frame (depending on the speedup rate)?

Last edited by Joachim Buambeki; 30th April 2017 at 21:06.
Joachim Buambeki is offline   Reply With Quote
Old 30th April 2017, 20:46   #13  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 7,097
Clipblend is max 1/2 bit in error for 8 bit result. ClipBlend16, is again max 1/2 bit in error for STACK16 16 bit result.
(no dithering)
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 30th April 2017, 21:36   #14  |  Link
Myrsloik
Professional Code Monkey
 
Myrsloik's Avatar
 
Join Date: Jun 2003
Location: Ikea Chair
Posts: 2,002
Quote:
Originally Posted by Joachim Buambeki View Post
Thanks.

So my script would look like this?
Code:
Helperscript() #outputs weight (w1, w2, w3,...,w99) for each frame
c.misc.AverageFrames(clip, [w1, w2, w4, ...,w99]) 
c.std.SelectEvery(clip, cycle=y, offsets=0)
If yes, wouldn't that average x/y more frames than necessary if they are discarded afterwards anyway?
Does that mean "AverageFrames()" can only process 25 frames? If yes, could this be increased or do I need to call it multiple times and merge them afterwards?

TIA!
I'll increase it to 63 for the next version. But be aware that once a frames only changes +-1% it probably won't have a visible impact at all. All calculations are done with no loss of precision.

Exactly how did you get those numbers anyway? How accurate does it have to be? Chopped off sinc not good enough?
__________________
VapourSynth - proving that scripting languages and video processing isn't dead yet
Myrsloik is offline   Reply With Quote
Old 30th April 2017, 22:35   #15  |  Link
Joachim Buambeki
Registered User
 
Join Date: May 2010
Location: Germany, Munich
Posts: 49
Quote:
Originally Posted by Myrsloik View Post
I'll increase it to 63 for the next version. But be aware that once a frames only changes +-1% it probably won't have a visible impact at all. All calculations are done with no loss of precision.
Great, thanks - would it be possible to get even more? I am aiming for at least a 30x speedup, which would mean 30*3,8 (19/5) frames (if the function above is used to determine the weight for each frame) need to be processed.


Quote:
Originally Posted by Myrsloik View Post
How accurate does it have to be? Chopped off sinc not good enough?
I replicated the effect in Photoshop and a few percent off never made a visible difference. If possible, I would still prefer a fair degree of precision but in the end, if I can load a ProRes444 (12bit) file and render to EXR or DPX without any degradation I am fine.

Here is a sketch for the script, obviously this is in AVS style and even there I don't really have an idea how to do it but it should give an idea what I have in mind. If you have any ideas what to improve conceptually or how to actually execute it, feel free to post it. :-)

Code:
function haventcomeupwithacoolnameyet(parameters)

{
speedup = default( speedup, 5 ) # speedup rate, float
pre_speedup = default( pre_speedup, 0 ) # if this offers no speed gain, use it to determine the threshold where more samples don't make the result any smoother

shutter = default(shutter, 1 )  # 1 = symmetrical blend (the one posted)
                                # 2 = asymmetrical blend
                                # 3 = square shutter, can be >360, rounded to nearest possible value
                                # 4 = experimental idea, after first pass, pixels with higher local contrast are given more weight in second pass

cutoff = default(false)         # cutoff negative weights 

pad = default(false)            # optional padding to 360 shutter for >360 shutter sources with MVtools before frame averaging, optional denoising

warnings = default(true)        # show only warnings
OSD = default(false)            # displays parameters, frame count pre/post and warnings
bars = default(false)           # visualises how blending works by generating blank clip with white bars on grey background with clip properties (optionally show video as pip in corner)

# generate_frame_weights() outputs weight for each frame based on the selected blend type
# For square shutters, round to multiple of 360/speedup

generate_frame_weights(speedup, shutter) #outputs weight for each frame based on the selected shutter type
{however that might work...}    # generates w1, w2, w4, ...,w99

c.misc.AverageFrames(clip, [w1, w2, w4, ...,w99])
}

Last edited by Joachim Buambeki; 5th May 2017 at 16:55.
Joachim Buambeki is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 02:40.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.