Fusion - pyramidal image processing for video [experimental]

[davidhorman]

Even more updated!

The latest version (a rewrite which only implements fuse and fusemask) is at http://horman.net/fusion2.zip


____________________________________________________________________________________________________________

Updated: see here (http://forum.doom9.org/showthread.php?p=1384755#post1384755)

Download fusion.zip (http://horman.net/fusion.zip) here

This is Fusion, a set (if you can call "two" a set) of filters that use image pyramids to blend clips together (more commonly used in the creation of HDR images and image stitching).

I'm not sure it's all that useful for video, but I've written the library now and AviSynth seemed like an interesting way to play with it. As opposed to a normal mix of clips with a mask, the filter fuse(clip1,clip2,mask) will "decompose" frames with a high pass filter, making a stack of different levels of detail. These levels are then blended separately before being recomposed into a final image.

A picture paints a thousand words, so:

http://img94.imageshack.us/img94/7629/fruit.jpg

On the left is a "naive" blend with a widely feathered mask. Notice that the stalk is semi-transparent, and the feather line is a little textureless. With multi-level blending, on the right (as output by fuse) textures are preserved and the blending of underlying colour is much wider, yet still preserves features such as the stalk. It fudges up the background - not sure what that's about but I suspect it's just way it works.

A possibly more useful application than creating hybrid fruit is in boosting the levels of underexposed video:

http://img687.imageshack.us/img687/434/fuse.jpg

The mask is a threshold of a blurred copy of the brightened image (top right), and fuse uses this to blend the two different exposures without (or at least, with very little) haloing.

The two filters in the current .dll are:


# NB: all input clips must be RGB32

fuse(clip1, clip2, mask)
# mask should be black/white only, unless extra blending of fine details is required
# in the mask, black represents clip1, white represents clip2

undersharpen(image, level, amount)
# this will sharpen/blur only one level of an image
# level=0 is the top level, i.e. finest detail
# useful values for amount range from -0.33 to +1.00

undersharpen doesn't seem to do anything very useful, but I didn't know that when I wrote it so I've left it in.

I'm not yet sure what other applications image fusion may have with video, so if you know of any please let me know and I'll try to implement them. There might (or might not) be good reasons to try extending this to include the temporal dimension...

David

[davidhorman]

After some more experimentation, it might almost have possibilities for chromakeying... or perhaps not:

http://img204.imageshack.us/img204/4073/chromakey.jpg

David

[tin3tin]

Looks very interesting. I guess with animate this could be used for making some nice mask-transitions too. I will give it a try when I get the time.

[davidhorman]

I've updated the filter (http://horman.net/fusion.zip) to include a new parameter, limit, for the fuse filter. This limits the number of levels of pyramid generated and should result in more pleasing wipes. The effective "width" of the wipe, assuming you start with a hard mask, will be approximately 2^limit, so limit=6 will make your wipe about 64 pixels wide, though that only applies to underlying colour, not the details (which is the whole point of the filter).

David

[davidhorman]

By tweaking the filter, I've been able to improve the green screen effect to conserve either the background, or the foreground, but unfortunately not both, yet (it may not be possible with this filter):

http://img191.imageshack.us/img191/6466/greenw.jpg

David

[davidhorman]

I've updated fusion.dll (http://horman.net/fusion.zip) to fix a couple of bugs, but mainly to include a new filter, fusemask.

Usage:

fusemask(a,b,exp=0.5)

This creates a binary mask to pass to fuse and can be used to merge two differently exposed images of the same scene, for example:

http://img404.imageshack.us/img404/2307/fusemask.jpg

The parameter exp is a target exposure for the mask, and takes values between 0 and 1 (with 0.5 as the default).

Some of the "harshness" that is evident in fusion's image can be mitigated slightly by blurring the fusemask.

David

[markanini]

Great work!

[travolter]

(newbie here)
Plz, could you write an script example of how to use fuse to boost the levels of underexposed video like in the images of your first post?

Im interested in create HDR vids. Currently Im using the contrast mask technique.. and it provides better results than HDRAGC

[redfordxx]

Hi, can you please more exactly or mathematically express, what exactly fuse() does, because it is not clear to me.

Because afaik the point of creating HDR images is having multiple images of one target at different exposures. Which is in case of video hardly achievable.

[Yellow_]

Because afaik the point of creating HDR images is having multiple images of one target at different exposures. Which is in case of video hardly achievable.

From post #1

I'm not sure it's all that useful for video, but I've written the library now and AviSynth seemed like an interesting way to play with it

Magic Lantern Firmware. Not HDR or multiple exposures of same frame but I'd be interested if you have any suggestions on approach.
http://forum.doom9.org/showthread.php?p=1547089#post1547089

[tin3tin]

A testing script for hdr video shot with the new Magic Lantern EOS hack, with both overlay/blend and Fuse:

loadplugin("C:\Users\tin2tin\Desktop\Fotos 60D\fusion.dll")
loadplugin("C:\Program Files\DVD slideshow GUI\FFmpegSource\ffms2.dll")
import("C:\Program Files\DVD slideshow GUI\FFmpegSource\ffms2.avs")

v=ffmpegSource2("C:\Users\tin2tin\Desktop\Fotos 60D\MVI_3769.MOV").converttorgb32().ReduceBy2().ReduceBy2()
a=selecteven(v)
b=selectodd(v)

mymask=greyscale(overlay(a,b)).levels(50,1,256,0,256,true)
c=overlay(b,a,mask=mymask,opacity=0.5,greymask=true,mode="blend",pc_range=true)

Stackvertical(a.subtitle("Dark"),c.subtitle("Blend"),fuse(a,b,mymask.invert()).subtitle("Fuse"),b.subtitle("Light"),mymask.subtitle("Mask"))#, fusemask(a,b,exp=0.3)

The footage used: minus.com/mo8xQ556o#1 (http://minus.com/mo8xQ556o#1) by DavidJFulde (http://vimeo.com/davidfulde)

http://download.videohelp.com/tin2tin/HDR_fuse_test.png

[Yellow_]

How are you dealing with motion?.

[PALpilot]

Results are pretty nice. I didn't know this fusion filter even exited. Great!

Combined with the upcoming interframe 2.0 we have everything in one script.

Bart

======= from interframe site ==============

So now interframe interpolates the missing frames from separate even and odd streams.
What would be great is that interframe would use the motionfactors from the opposite frame and tries to recreate that frame as good as possible. I think it could improve quality a lot! Especially on locating walking leg positions or stuff like that.

Please tell me your thoughts on this.
Bart van den Boogaard
Redkitemedia

-------------
SubJunk
December 23rd, 2011 - 11:19

@Bart, it looks great.
Your suggestion is good and it has been developed, along with other upcoming improvements. It is in the private beta stage right now and will be supported in InterFrame 2.0.
I guess it will be ready in a month – or two considering the busy time of year.
Feel free to link back to this blog anywhere you want, traffic is nice :)

[PALpilot]

UPDATE

Just tried it with both blend and fusion options.
This script is FAST and with little movement very usable WOW!

I went from half an hour workflow when I started to 1 minute now!

[travolter]

plz anyone testing HDR video can post his script? palpilot and others :)

[Yellow_]

http://blendervse.wordpress.com/2011/12/24/canon-magic-lantern-hdr-feature-to-10bit-lossless-h264/

[PALpilot]

I use just this at 720p

v=FFVideoSource("MVI_7980.MOV").converttorgb32().spline36resize(1280,720)
a=selecteven(v)
b=selectodd(v)
mymask=greyscale(overlay(a,b)).levels(50,1,256,0,256,true)
c=overlay(b,a,mask=mymask,opacity=0.5,greymask=true,mode="blend",pc_range=true)

return c
#fuse(a,b,mymask.invert())

=====================
return c = for blend
fuse(a,b,mymask.invert()) is for fusing
switch them by add or remove the # (comment symbol)
Your video has to start with the light frame. Sometimes I need a trim to achieve this.
like v=FFVideoSource("MVI_7980.MOV").trim(0,1).converttorgb32().spline36resize(1280,720)

[Gavino]

mymask=greyscale(overlay(a,b)).levels(50,1,256,0,256,true)
Is this right?
Since you are using the default mask and opacity here, doesn't overlay(a,b) just give you b?

[travolter]

thanks a lot for these scripts examples.. Im anxious to check. I always used contrastmask to create HDR like videos..

http://forum.doom9.org/showthread.php?t=161986

[PALpilot]

Is this right?
Since you are using the default mask and opacity here, doesn't overlay(a,b) just give you b?

I'm just using the script from tin3tin. I don't understand it completely. I'm just trying it on some footage I shot.

If anyone want some additional HDR footage from Magic Lantern 600d. I can share.

[jmac698]

You can do the same kinda things with CHDK, and even stock firmwares in newer models now.

[Yellow_]

You can do the same kinda things with CHDK, and even stock firmwares in newer models now.

hey jmac, with video? I think not. :-) Unless it's RED HDRx ;-)

Yes, bracketed exposures and HDR shooting in camera for stills is common, not video. :-)

[davidhorman]

A testing script for hdr video shot with the new Magic Lantern EOS hack, with both overlay/blend and Fuse:

fuse(a,b,mymask.invert())

Hi tin3tin - it's nice to see Fusion is finally finding some use, and those example images you posted are a good example of what it can do. Just a note for anyone else reading, the algorithm behind fuse doesn't require a greyscale mask - a hard black/white mask usually produces good results as well.

I think I may dig out the old code and rewrite it. I've recently written a still image blender and learned a lot in the process, so a new version will probably be a lot quicker.

David

[Yellow_]

With regard to color space is fusion a function that is best done in RGB? Is there possibility of native YCbCr function for video?

Would it work within Dither Tools stacked 16bit functions?

http://forum.doom9.org/showthread.php?p=1386559#post1386559

[tin3tin]

@ Gavino You're right as always :). Anyway making the mask as a mix of the two frames will not work with things in motion. The actual mask gotta be created out of a frame which is exactly similar to the frame it is overlayed motionwise. So either the overexposed or the underexposed frame must be turned into an inbetween(with interframe) before it is overlayed/fusioned with a mask based on it.

@DavidHorman It is a great plugin you have scripted. I wonder how it deals with the blacks and whites - does it strech the spectrum after overlaying the two images? The blacks seems a bit crushed?

[smok3]

and there is no Magic Lantern for 7D?

[Yellow_]

@ Gavino You're right as always :). Anyway making the mask as a mix of the two frames will not work with things in motion. The actual mask gotta be created out of a frame which is exactly similar to the frame it is overlayed motionwise. So either the overexposed or the underexposed frame must be turned into an inbetween(with interframe) before it is overlayed/fusioned with a mask based on it.

I think that depends on the shot, the assumption is that these 'HDR' videos are made of two 12.5fps streams that need the motion interpolating, in reality all the motion is there at 25P or whatever, it's the exposure that jumps and, as an example the seascape video has so little movement, that to produce interframe output seems unnecessary when something like 90% of the content is stationary and no hand held motion to complicate from a locked off tripod shot.

Something like masktools2 mt_motion could possibly be used to isolate and 'remove' from processing the motion, blend the 90% static that's left and reverse the mask to add back the motion and adjust to blend in.

In theory. :-) I've been trying mt_motion out with some success but am a total novice with the functions. :-)

[davidhorman]

With regard to color space is fusion a function that is best done in RGB? Is there possibility of native YCbCr function for video?

Would it work within Dither Tools stacked 16bit functions?

I don't know anything about Dither Tools - does it create some kind of fake RGB64?

@DavidHorman It is a great plugin you have scripted. I wonder how it deals with the blacks and whites - does it strech the spectrum after overlaying the two images? The blacks seems a bit crushed?

It doesn't do anything to the spectrum - I think that's just the way it works. I might include some parameters for adjusting levels before output, because it might be possible that some parts get clipped and some detail is lost.

David

[Yellow_]

hi David, yes Dither Tools uses a denoiser to try and recover high bit depth data from 8bit sources and then stacks that back in LSB/MSB.

http://forum.doom9.org/showthread.php?p=1386559#post1386559

Then using a tool like Avs2yuv or pipemod to handle the avs script it can pipe the data to 10bit x264 encoder or as rgb48 to Imagemagick amongst other tools, IM can write hdri formats like ppm, tiff, exr, hdr as well as a lot of 8bit LDR formats.

Or there are various dither functions including encoder friendly ones to dither back to 8bit if preferred.

But I think what I'm driving at is if there is possibility to add a function using the fusion type principle that rebuilds the details levels / gradients from the data in the two 8bit exposures (rather than denoising as dither tools) into a wider dynamic range 16bit file rather than just fusing two exposures into 8bit which with two exposures only feels more akin to dodging and burning rather than HDR.

The floating point imagemagick output could then be tonemapped as part of the 'Post' grading process. Although I guess the whole point of fusion is to avoid this. :-) But on the other hand it is also making assumptions / weighting on what image data to keep and throw away?

Not sure if you've seen this, maybe useful generally: http://www.tawbaware.com/tufuse.htm

One last element of this is the practicality of editing flickering video straight off the camera, so as fusion does now merging the exposures into a file for editing is useful too albeit including conversion to RGB and back to YCbCr.

[davidhorman]

I've wondered about applying Fusion to the smoothing problem, but I'm not really sure how to go about it, if it's possible - as far as fuse is concerned, a step in a gradient is a detail like any other. I know about tufuse - I'm on the Tawbaware forums and I designed the PTA icon :) - Fusion works on the same principles and I took some hints from tufuse when I wrote it.

But on the other hand it is also making assumptions / weighting on what image data to keep and throw away?

The decision is made solely on the content of the mask, if that's what you mean... if you had two differently focused videos of the same scene, for example, and could construct the appropriate mask, it could do focus blending as well.

David

[davidhorman]

I've rewritten the filter:

http://horman.net/fusion2.zip

It should definitely be faster and might give better results.

As before, the syntax is:

fuse(a,b,m) # fuse two videos, a and b, using the mask m (green channel)
fusemask(a,b[,exp]) # create a mask to pass to fuse, for a target exposure of exp (0-1, defaults to 0.5)


David

[tin3tin]

Your new release works well. Thank you! :)

[PALpilot]

Your new release works well. Thank you! :)

Hi tin3tin,

Could you post your script that's working with the new version?

[tin3tin]

Something like this, and then tinker with the exp value:
loadplugin("C:\Users\tin2tin\Desktop\Fotos 60D\fusion.dll")
loadplugin("C:\Program Files\DVD slideshow GUI\FFmpegSource\ffms2.dll")
import("C:\Program Files\DVD slideshow GUI\FFmpegSource\ffms2.avs")

v=ffmpegSource2("C:\Users\tin2tin\Desktop\Fotos 60D\MVI_3769.MOV").converttorgb32()
a=selecteven(v)
b=selectodd(v)

fuse(a,b,fusemask(a,b,exp=0.5))

I've noticed that my 60D sometimes starts with the underexposed image and sometimes with the overexposed image, so you'll have to swap the a and b values.

[PALpilot]

Thank you.

But I get an error on fusion.dll (v2)

=========
Avisynth open failure:
LoadPlugin: unable to load "c:\Program Files (x86)\avisynth 2.5\plugins\fusion.dll", error=0x7e

and without loadplugin it doesn't recognise the fusemask command. Maybe it has to do with windows 7 x64? The old version works with the same script. It's in the DLL.

[davidhorman]

I've noticed that my 60D sometimes starts with the underexposed image and sometimes with the overexposed image, so you'll have to swap the a and b values.

Actually you won't have to - you're swapping the parameters for fusemask as well, so they cancel each other out.

Avisynth open failure:
LoadPlugin: unable to load "c:\Program Files (x86)\avisynth 2.5\plugins\fusion.dll", error=0x7e


Just a guess off the top of my head, but do you have a non-SSE2 CPU?

David

[PALpilot]

Just a guess off the top of my head, but do you have a non-SSE2 CPU?
David

I have Intel Core2Quad 8200
http://ark.intel.com/products/36547/Intel-Core2-Quad-Processor-Q8200-%284M-Cache-2_33-GHz-1333-MHz-FSB%29

[kemuri-_9]

The 0x7e error indicates that one or more dependent dlls are missing from the system.

make sure you have the Visual Studio 2010 runtime (http://www.microsoft.com/download/en/details.aspx?id=5555) installed, as the plugin requires it.

[PALpilot]

The 0x7e error indicates that one or more dependent dlls are missing from the system.

make sure you have the Visual Studio 2010 runtime (http://www.microsoft.com/download/en/details.aspx?id=5555) installed, as the plugin requires it.

Nice one. That helped. First I switched to the x64 version but that didn't work. The x86 version is the one you need on x64 windows7 systems.
:thanks:

And I uploaded another test scene for people to try out.
MVI_7951.MOV 92.2 MB
http://www.megaupload.com/?d=5B5CUV3S

[TheoRI]

Dear videoFred and other member,

I am using a modified projector with a 5MP industrial CMOS camera to capture our super 8 film. As part of the development process it is evident that the camera does not have the range to capture all the detail available on the film. So I am contemplating running 2 capturing instances at different exposure - light intensities and then "combining the frames" using the fusion script:
http://forum.doom9.org/showthread.php?t=152109

I capture a Bright and a Dark sequence and then would like to join them together.
The avisynth scrip I use below.

The problem I have is that due to small position difference each frame is not aligned in the x-y as they are captured at different times. This results in some "ghosting" around objects. Is it possible to align the frames using an avisynth script before "fusing" them? IF so how? Please update the script below.

The output of this script will then be fed into the script that videoFred developed for super 8 restoration (Thanks a million!)
http://forum.doom9.org/showthread.php?t=144271

Kind Regards
Theo

# HDR script using Fusion plugin
# http://forum.doom9.org/showthread.php?t=152109
#
#=============================================================================================


Dark="e:\telecine\Dark input.avi" # source clip, please specify the full path here
Bright="e:\telecine\Bright input.avi" # source clip, please specify the full path here

exposure = 0.5

#----------------------------------------------
loadplugin("fusion.dll")
SetMemoryMax(1024)

a= avisource(dark).converttorgb32().trim(0,0)
b= avisource(bright).converttorgb32().trim(0,0)

# horizontal stack of Dark, Fuse, Light
# Stackhorizontal(a.subtitle("Dark"),fuse(a,b,fusemask(a,b,exp=exposure)).subtitle("Fuse"),b.subtitle("Bright"))

fuse(a,b,fusemask(a,b,exp=exposure))

[joka]

TheoRi, I like this idea. Maybe you should open a separate thread in Avisynth usage section.

However, nobody seems to have a solution "ready to use" for your problem. Maybe this can help - not tested.

a = your first clip
b = your second clip


c = Interleave(a, b)

# tweak the darker and the brighter clip (play with a_cont, b_cont, may be different thresholds)
# so, that the results are similar as possible
# (special clip borrowed from Fred)
# maybe you need .converttoYv12() before tweak
a_ref = a.tweak(cont=a_cont).MT_binarize(threshold=80).greyscale().invert()
b_ref = b.tweak(cont=b_cont).MT_binarize(threshold=80).greyscale().invert()
c_ref = Interleave(a_ref, b_ref).crop(32, 24, -32, -24)


# calculate stabilization data
mdata = DePanEstimate(c_ref,trust=1.0,dxmax=20,dymax=20)


# stabilize
c_stab = DePanInterleave(c, data=mdata)


# Select the required frames
# may be you have to crop the borders (consider aspect ration)
b_stab = c_stab.SelectEvery(6, 2)
a_stab = a


Good Luck! If it works show a sample.