Hello, is there a frame difference filter?

[DrPhill]

Hello all, I am Phill. I have wriggled my way through your vetting procedures and can now post to the forum.

First let me say that AVISynthis just what I needed, and a superb piece of software. Thanks to all who have contributed. I already use a lot of open-source software (Eclipse, Audacity and similar), and have made some small contributions to SourceForge myself. I would also like to apologise now if I have chosen the wrong forum for this post. ;-)

I have a private project that I am tackling, and I think AVISynth will enable me to achieve it, but I am one piece short of a solution to the jigsaw. I am hoping you folk could help me solve this. My project is to take a series of videos of the same river view at several times during the year, and merge them together so that the river is seen flowing in real time, but the rest of the image is a time-lapsed year. I think this will make a pleasant backdrop to a whistle video....

I have in AVISynth standard filters all that I need to achieve this if I use simple video clips. However my video camera gives much better stills than videos, so I would like to be able to merge a high resolution still background with the lower resolution moving river. I can convert my stills to a 'video' with standard filters, and could use overlay to combine the these with the moving river, but the result is not to my liking. What I really want is a mask that allows the video of the moving river to overwrite only the river portions of the still scene. I realise that the image quality will differ between the two areas, but I am not sure that this will be too detrimental to the effect.

The river height changes during the year, as do the shapes of th gravel banks, so for each still/video pair I would need to generate a separate mask. I do not want to do this by hand....... I think it should be technically feasible to write a filter that creates a mask from only those pixels that change during a video. This mask could then be used during the overlay to achieve my desired end. Actually, two filters would probably be better and more reusable. The first to record the difference between each consequtive pair of frames, and the second to total the results from the first.

So my questions to this community are:
- Am I right in believing that this is technically feasible?
- Has it already been done?
- If not, then could I have some help please in developing the required filter?

I have set up and verified the compilation environment for the example filter. I think I understand the example code. I am a competent programmer, though it is a few years since I last used C (I mostly use Java these days). I also have an outline for an algorithm for the task (clumbsy and slow but straightforward). It relies on maintaining a copy of the previous clip(s) and creating a mask clip (which could be returned so that progress can be seen).

I am not sure of the environmental constraints on such persistent data (for example, I imagine that the clip memory will be freed outside my control, forcing me to take copies). I am not sure that I could develop the filter to your high distribution standards either - there are probably a host of gotchas hidden in the various video formats. I am working in MPEG, and so my motivation to develop a format independent solution may be a little lacking.

Thank you in advance for any help or guidance that you can give.

Phill

[Minor formatting edit]

Find me on SourceForge (http://phillvanleersum.users.sourceforge.net/)
Find me on YouTube (http://www.youtube.com/user/PhillvanLeersum)

[smok3]

so your hoping to get single image mask out of this? You will still have manual painting to do to make it perfect anyway (like making borders properly hard or soft), so why not hand-paint it from scratch (thats just one image as i understand)? (I would always go with a visual-feedback tool for stuff like this, think after effects or gimp for making a mask if you insist on OS)

[DrPhill]

so your hoping to get single image mask out of this? You will still have manual painting to do to make it perfect anyway (like making borders properly hard or soft), so why not hand-paint it from scratch (thats just one image as i understand)? (I would always go with a visual-feedback tool for stuff like this, think after effects or gimp for making a mask if you insist on OS)

Thank you for responding. I am hoping to create a mask for each of a dozen or two still/video pairs (one taken every fortnight or so), so manual painting would be a chore (yes, I am lazy that is exactly why I became a programmer;)).

I am not sure what you mean by 'making the borders properly hard or soft' - probably because I have not attempted this before. The still scene and the video scene can be superimposed, and the the boundaries between the river and the banks will not suffer from a little ambiguity. I am open to the possibility that I am missing something important here, though, as I am new to video processing.

Also, If the mask is defined dynamically, I may be able to include our dogs moving between the water and the shore. I would not want to hand paint a mask for each video frame. The mask may not work as expected in this case though - one dog is largely black and so I may only detect the edges of the colour patches leading to a see-through dog.

Phill

[smok3]

ok, then it makes some sense, you did not mention dogs in your first post. I did actually saw a filter like that somewhere (maybe it was autodesk smoke), i think it was simply called "difference", but the results were not really usable directly.

[DrPhill]

ok, then it makes some sense, you did not mention dogs in your first post. I did actually saw a filter like that somewhere (maybe it was autodesk smoke), i think it was simply called "difference", but the results were not really usable directly.

Ok, smok3, thanks for that. How would I find 'difference'? Or as a second course of action is there another filter (with source code) that does something similar that I could modify. All I need to do is cache the previous frame, so that I can compare with the current frame pixel by pixel, and set the new frames pixel accordingly. I realise that there are format variations to account for, but I am happy to limit the formats my filter works in.

I had a go at modifying Demosaic, but I am not sure that that is a good starting place. The original pluggin seems to achieve very little when I invoke it in an AVS script, so I do not even know if it is working. I can compile the source to a dll which AVIScript seems happy with, and I can tell my changes are taking effect as I can change the parameter value requirement and get an appropriate error. But all that it seems to do is convert the image to monochrome (or is that all that it should do?).

I would have thought that this should be easier than I seem to be making it - am I barking up the wrong kettle of fish?

Any help you can give would be gratefully appreciated.

Phill

[Youka]

FLuaG (http://forum.doom9.org/showthread.php?t=161852) would be an option to use your algorithm, but i don't know if it would be worth because of the familiarization first. Else you could write it in C++ with the Avisynth SDK.

[DrPhill]

FLuaG (http://forum.doom9.org/showthread.php?t=161852) would be an option to use your algorithm, but i don't know if it would be worth because of the familiarization first. Else you could write it in C++ with the Avisynth SDK.

Thanks Youka. I think I am floundering a bit here, and at risk of annoying you all. I do hope not. I would rather 'retire hurt' than get a bad rep.

I am happy to develop my own solution in C or C++ (I have CodeBlocks set up [for C only??]). What I need is either clear documentation, or a good example of a pluggin (pref simple) that will do something noticeable to an MPEG loaded with DirectShowSource. Even better, the example of the pluggin would implement a function that demonstrates scanning ( as in x,y) the source and destination clips simultaneously. From there I would have a good chance of success (or so I optimistically believe ;-). I have been looking at various pluggins but cannot find one that meets these criteria.

[Gavino]

What I need is either clear documentation, or a good example of a pluggin (pref simple) that will do something noticeable to an MPEG loaded with DirectShowSource. Even better, the example of the pluggin would implement a function that demonstrates scanning ( as in x,y) the source and destination clips simultaneously. From there I would have a good chance of success (or so I optimistically believe ;-). I have been looking at various pluggins but cannot find one that meets these criteria.
Not sure what you mean by scanning the destination clip, but almost any plugin simultaneously scans the input clip and writes the destination clip, including the SimpleSample (http://avisynth.org/mediawiki/Filter_SDK/Simple_sample) example from the Filter SDK.

Also not sure (must be the New Year hangover :)) exactly what you want your plugin to do - can you describe what the input is and what the output should be in terms of that input?

Note that the input and output are always uncompressed video - a plugin neither knows or cares whether the source was MPEG via DirectShowSource or whatever (although the uncompressed format may be one of several supported by Avisynth).

Depending on what you are trying to do, some existing filters might be of use: the built-in Subtract (http://avisynth.org/mediawiki/Subtract)(), or the MaskTools (http://avisynth.org/mediawiki/MaskTools2) functions mt_makediff and mt_motion.

For example, Subtract(Loop(2,0,0)) will show the pixels in each frame which differ from the preceding frame.

[DrPhill]

Not sure what you mean by scanning the destination clip, but almost any plugin simultaneously scans the input clip and writes the destination clip, including the SimpleSample (http://avisynth.org/mediawiki/Filter_SDK/Simple_sample) example from the Filter SDK.

Also not sure (must be the New Year hangover :)) exactly what you want your plugin to do - can you describe what the input is and what the output should be in terms of that input?

Note that the input and output are always uncompressed video - a plugin neither knows or cares whether the source was MPEG via DirectShowSource or whatever (although the uncompressed format may be one of several supported by Avisynth).

Depending on what you are trying to do, some existing filters might be of use: the built-in Subtract (http://avisynth.org/mediawiki/Subtract)(), or the MaskTools (http://avisynth.org/mediawiki/MaskTools) functions mt_makediff and mt_motion.

For example, Subtract(Loop(2,0,0)) will show the pixels in each frame which differ from the preceding frame.

Hi Gavino - either your New Year Hangover (I hope it was well earnt) or my lack of sleep (up at 04:30hrs) making me incomprehensible. I would bet on the latter. But thanks for answering. I am probably guilty of 'solutionising' when I should be specifying requirements.

Filter 1: Each frame shows the pixels that have changed since the last frame. These should be shown in some distinct colour that makes the frame usable as a mask in Overlay (0 or 255). Call it Diff, but it could be as simple as your Subtract(Loop(2,0,0))

Filter 2: Each frame shows all the pixels that have changed since the start of the video. This was my first guess at a solution, but I actually think Filter 1 will do the whole job.

So in pseudo-script

mask_clip = Diff(original_video_clip)
Overlay(still_video_clip, original video_clip, mask=mask_clip)

ie use Diff to find the changing pixels, and only allow those pixels to overwrite the higher resolution clip.

Am I making any more sense now? I apologise if my lack of familiarity with the tool/terminology is confusing - I have been on the other end of this kind of situation and it can be very irritating.

I am currently trying to modify the Invert example (http://kevin.atkinson.dhs.org/avisynth_c/example.html) to do the job, but would prefer a higher level solution (less likely to break something).

[Gavino]

Filter 1: Each frame shows the pixels that have changed since the last frame. These should be shown in some distinct colour that makes the frame usable as a mask in Overlay (0 or 255). Call it Diff, but it could be as simple as your Subtract(Loop(2,0,0))

Filter 2: Each frame shows all the pixels that have changed since the start of the video. This was my first guess at a solution, but I actually think Filter 1 will do the whole job.

So in pseudo-script

mask_clip = Diff(original_video_clip)
Overlay(still_video_clip, original video_clip, mask=mask_clip)

ie use Diff to find the changing pixels, and only allow those pixels to overwrite the higher resolution clip.
OK, I think I understand now what you want.
Try this for your Diff function:
function Diff(clip c) {
mt_lutxy(c, c.Loop(2,0,0), expr="x y = 0 255 ?", chroma="-128")
}
(requires MaskTools and YV12 (or other planar) input)

[redfordxx]

Maybe I am not exactly clear what you are about to achieve, but I don't think you need new plugin.
You should be able to this with existing filters.

Use trim(1,0) for example to shift clip and make difference or whatever op to next frame.

Use masktools, Dither tools, subtract, overlay, merge, RedAverage or I don't know what to create mask and merge.

Be sure to learn something about PCscale and TVscale and which filters expect which input, because that could seriously mix up with your results.
I personally don't like the built in filters because they I think use the TVscale and I think the results are then different than I expected... but thats long time ago already so I cannot tel for sure.
If you wanna be really precise in merging look into my RedAverage thread for some info about maskmerging with mask in range (0,255)

[DrPhill]

OK, I think I understand now what you want.
Try this for your Diff function:
function Diff(clip c) {
mt_lutxy(c, c.Loop(2,0,0), expr="x y = 0 255 ?", chroma="-128")
}
(requires MaskTools and YV12 (or other planar) input)

That looks completely incomprehensible - so it must be worth trying. Thanks. I will give it a try tomorrow when my brain is less fuddled. If I can do this with built-in functions I will be much happier.

As an aside, I am making progress with a hand-crafted plugin/filter - I can detect difference and flag it, but I need to define what I mean by different. The background fluctuates a bit so I need to build in a better test than simple inequality. I think I need to test all bytes of a pixel for a change greater than a threshold. It is harder to remember my C than I thought, but I am getting there.

I will let you know how I get on.

Again, thanks.

Phill

[Gavino]

I can detect difference and flag it, but I need to define what I mean by different. The background fluctuates a bit so I need to build in a better test than simple inequality. I think I need to test all bytes of a pixel for a change greater than a threshold.
That is a flaw I was aware of with my Diff function too - it is only suitable for a fixed camera shot, eg locked tripod. Pans or even simple shaking will be seen as differences from one frame to the next.

However, doesn't your whole idea depend on having a fixed background in the (moving) video?
If the river is not in exactly the same place in each frame, how can you overlay it onto the still video correctly?

EDIT: Ah, you probably mean the background fluctuates in brightness rather than moving. That can be fixed by changing the function to:
function Diff(clip c, int thresh) {
t = string(thresh)
mt_lutxy(c, c.Loop(2,0,0), expr="x y - abs "+t+" < 0 255 ?", chroma="-128")
}

[DrPhill]

That is a flaw I was aware of with my Diff function too - it is only suitable for a fixed camera shot, eg locked tripod. Pans or even simple shaking will be seen as differences from one frame to the next.

However, doesn't your whole idea depend on having a fixed background in the (moving) video?
If the river is not in exactly the same place in each frame, how can you overlay it onto the still video correctly?

EDIT: Ah, you probably mean the background fluctuates in brightness rather than moving. That can be fixed by changing the function to:
function Diff(clip c, int thresh) {
t = string(thresh)
mt_lutxy(c, c.Loop(2,0,0), expr="x y - abs "+t+" < 0 255 ?", chroma="-128")
}

Yes, I had assumed a brightness fluctuation - but it is good to have confirmation. Because I do not really understand the more complex colour codings I changed my script to supply RGB to my hand crafted filter. I am going to test for a change greater than a threshold in any of the (four?) colour values to determine the colour of the entire pixel (black/white).

BTW, to clarify: I will use this filter/function to extract the interesting movement from a video and paste it onto a different video. That way I can use the movement from a low res video with the detail of a high res 'still'. Both the still and the videa are taken from exactly the same fixed position.

Where can I find info on mt_lutxy()? I think it will help me greatly to understand how your solution works.

Continued gratitude.

Phill

[Gavino]

Where can I find info on mt_lutxy()?
See http://avisynth.org/mediawiki/MaskTools2.
Further details are available in the documentation (mt_masktools.html) in the MaskTools2 download linked on that page.

(Note - in post #8, I inadvertently linked to the page describing the earlier MaskTools instead of the current one. I'll correct that now.)

[DrPhill]

See http://avisynth.org/mediawiki/MaskTools2.
Further details are available in the documentation (mt_masktools.html) in the MaskTools2 download linked on that page.

(Note - in post #8, I inadvertently linked to the page describing the earlier MaskTools instead of the current one. I'll correct that now.)

:thanks:

[DrPhill]

Well I had little success with mt_lutxy - probably something I did.

I did however get my version of the difference filter working, and the result is pretty good for a still and a video taken one after t'other. If you look carefully you can see that the ripples in the stream 'float' over the background image, but it is a good effect.

I now need to find a way to encode the avs to mpg or similar.

[Didée]

a flaw I was aware of with my Diff function
There is a syntax/semantic flaw, too. It was very old masktools where you could use the equal operator "=" to check for identity. Nowadays, you're required to use the identity operator "==".

"x y == 0 255 ?"

[Gavino]

There is a syntax/semantic flaw, too. It was very old masktools where you could use the equal operator "=" to check for identity. Nowadays, you're required to use the identity operator "==".

"x y == 0 255 ?"
It worked for me when I tested it.
And the documentation for MaskTools 2.0a48 says:
We can create boolean values with the following comparaison [sic] operators : "<", ">", "<=", ">=", "!=", "==", "=".
Looking through the change history, it seems "=" was added in v2.0a31.
Or does "=" mean something different from "=="?

Well I had little success with mt_lutxy - probably something I did.
As I said, it worked for me when I tested my Diff function. I used
mask = Diff(10)
Note that the input must be YV12 (or any planar format in Avisynth 2.60).

I now need to find a way to encode the avs to mpg or similar.
Many MPEG2 encoders accept avs files as input.
A good free one is HCenc (http://hank315.nl).

[Didée]

Oh, indeed, it works ... again. I do remember that scripts using "=" (from MaskTools v1 era) didn't work correctly anymore in MaskTools v2, because "==" was required then. Probably Manao has re-added "=" at some point, and it slipped my attention.

Sorry for the buzz. :)

(But "==" still is the better syntax.)

[DrPhill]

I remember RPN from my first calculator. Seems an extravagant way to avoid implementing parentheses. My problems with the filter seemed deeper rooted - but never mind, I am on a roll now.

Thanks for the pointer to HCEnc - works well and is a good price.

I have made my first attempt available for the interested (http://phillvanleersum.users.sourceforge.net/clips/Horner.2012-01-02.m2v). I need to create a (static) mask to limit the dynamic mask, as video fluctuations outside the stream are causing flicker. I plan to sum the number differences into frame zero, so this will end up a grey-scale of amount of change. I can then either convert this to B&W or use as is to ignore changes in areas with few changes.

So much for my quick foray into video processing.

[Gavino]

I have made my first attempt available for the interested (http://phillvanleersum.users.sourceforge.net/clips/Horner.2012-01-02.m2v).
Looks good.
I look forward to seeing the changing seasons.

[-Vit-]

I needed something like this once. Had a stable background with a rather slow moving foreground I wanted to mask. It might work for your task too...
Rather than a single frame it compares several frames forwards and backwards and optionally the first frame too. Also post-processes the mask for a cleaner result. It can be successful given stable footage, but tweaking the settings is very important. Particularly important are LocalThresh and Radius, but tbh most of the settings have quite an impact.

Hard to guess how it will work for you without having seen the original footage. You may need to increase LocalThresh to eliminate small light changes / noise (my footage was clean). Probably worth experimenting with Radius, Uniformity and the mask post-processing settings too. I just ran your result footage through it and setting RemoveSpots to about 4 or so cleared up the speckles... Add a last line "return mask" to see the mask being generated, which helps tweak settings.

I've commented it up and tailored it to suit what I think your requirements are (the source loading and the final resizing before the merge - alter if necessary).
Needs GScript (http://forum.doom9.org/showthread.php?t=147846), masktools V2 (http://avisynth.org/mediawiki/MaskTools2) and RemoveGrain (http://home.arcor.de/kassandro/RemoveGrain/RemoveGrain.rar)

Radius = 5 # Number of frames before and after current used for comparison (sensible values 1 to 100+, depends on source)
# Larger (slower) values detect more gradual change (which may be a good or bad thing). Related to Uniformity below.
Step = 1 # Step to speed up larger radius values (e.g. 2, 3...)
MinDiff = 1 # Pixel differences of this value or less are ignored
Uniformity = 0.7 # Balance between "max" (0.0) and "avg" (1.0) when combining the differences found locally, range 0.0 to 1.0
# Most suitable value depends on Radius and type of change in the source - must experiment
LocalThresh = 4 # Difference between current frame and local frames that counts as an 'actual' difference
# Important to tweak to suit your source (values 2 to 16+)
RemoveSpots = 1 # Mask post-process - remove small spots of difference. May help clean up mis-detections, increase for stronger effect
RemoveHoles = 4 # Mask post-process - remove gaps in the mask. Increase to fill in larger holes
Soften = 2 # Mask post-process - soften mask edges
FirstFrameMask = false # Additionally compare current frame with first frame - useful given *very* stable background
FirstThresh = 8 # Difference between current frame and first frame that counts as an 'actual' difference
# Tweak to adjust area detected as different

# Load sources
video = WhateverSource( "path\to\your\video" ).ConvertToYV12() # Possibly supports YUY2 or YV16 - haven't tried it
fc = video.FrameCount()
background = ImageSource("path\to\your\background", end=fc-1 ).ConvertToYV12() # --"--

# Compare current frame with first frame
initDiff = FirstFrameMask ? mt_lutxy( video, video.FreezeFrame(1,fc-1,0), "x y - abs " + String(minDiff) + " < 0 x y - abs", U=3,V=3 ) : 0
initMask = FirstFrameMask ? initDiff.mt_binarize( FirstThresh, U=3,V=3 ) : 0

# Compare current frame with local frames
v1 = video.Loop(Radius,0,0)
avgDiff = BlankClip( video, color_yuv=$000000 )
maxDiff = BlankClip( video, color_yuv=$000000 )
GScript("""
for ( i = 0, Radius * 2, Step ) {
if (i != Radius) {
diff = mt_lutxy( video, v1.trim(i,0), "x y - abs", U=3,V=3 )
avgDiff = mt_lutxy( avgDiff, diff, "y " + String(minDiff) + " < x x y + ?", U=3,V=3 )
maxDiff = mt_logic( maxDiff, diff, "max", U=3,V=3 )
}
}
""")
numDiffs = (Radius * 2 / Step + 1) - ((Radius % Step == 0) ? 1 : 0)
avgDiff = avgDiff.mt_lut( "x " + String(numDiffs) + " /", U=3,V=3 )
localMask = Merge( maxDiff, avgDiff, Uniformity ).mt_binarize( LocalThresh, U=3,V=3 )

# Combine local and first-frame masks, combine Y,U & V masks and post-process
mask = FirstFrameMask ? mt_logic( initMask, localMask, "max", U=3,V=3 ) : localMask
chrmask = mask.UToY()
umask = chrmask.Spline36Resize(mask.Width(),mask.Height())
vmask = mask.VToY().Spline36Resize(mask.Width(),mask.Height())
mask = mask.mt_logic( umask, "max", U=1,V=1 ).mt_logic( vmask, "max", U=-128,V=-128 )
GScript("""
for ( i = 0, RemoveSpots ) { mask = mask.mt_deflate(U=1,V=1) }
for ( i = 0, RemoveSpots ) { mask = mask.mt_inflate(U=1,V=1) }
mask = RemoveSpots > 0 ? mask.mt_binarize( 128, U=1,V=1 ) : mask
for ( i = 0, RemoveHoles ) { mask = mask.mt_inflate(U=1,V=1) }
for ( i = 0, RemoveHoles ) { mask = mask.mt_expand(U=1,V=1) }
multiplier = RemoveHoles == 0 ? 0 : 1 + 1 / (1 + Pow( Log(RemoveHoles) / Log(10), 3)) # lol...
for ( i = 0, Int(RemoveHoles*multiplier) ) { mask = mask.mt_inpand(U=1,V=1) }
mask = RemoveHoles > 0 ? mask.mt_binarize( 0, U=1,V=1 ) : mask
for ( i = 0, Soften ) { mask = mask.RemoveGrain(12) }
""")
chrmask = mask.Spline36Resize(chrmask.Width(),chrmask.Height())
mask = YToUV(chrmask, chrmask).MergeLuma(mask)

# Resize video and mask to suit background, then merge
video = video.Spline36Resize(background.Width(),background.Height())
mask = mask.Spline36Resize(background.Width(),background.Height())
mt_merge( background, video, mask, U=3,V=3 )

You can also take an MVTools (http://avisynth.org.ru/mvtools/mvtools2.html#download) (motion analysis) approach too: blocks that are moving or have a poor motion match are counted as different. However, I don't think that will work so well for this example.

[DrPhill]

Wow, that looks complex...... scarily so. I am going to ponder the techniques over a cuppa. The radius ideas may be useful, though the flowing water benefits from sharp detailed movement.

What I have done is add another function to my filter. This mallocs a data array (4 bytes per pixel) and inits these to 255;
During image processing differences per pixel from previous frame are detected
diff = abs(r1 - r0) + abs(g1 - g0) + abs(b1 - b0) + abs(a1 - a0);
and if this exceeds a threshold the appropriate value in the data array is decremented (but not below zero).
If the value falls below a trigger level then the matching pixel in the output image is set to 0, else it is set to 255.

The result is a slowly built up uber-mask where 'significant' changes occur. The final frame should be the finished uber-mask if I can work out how to dump it reliably from the script. I will then apply that mask to the video to screen out unwanted changes.

I may need to apply a slightly different algorithm. On frame(0) do a scan of all frames to detect differences and generate the uber-mask. Then on frames 1-n I can apply the mask. This may cause a delay at the start of processing........

[-Vit-]

The radius ideas may be useful, though the flowing water benefits from sharp detailed movement.
The radius is only used to build a difference mask, it doesn't blur the actual content.

The result is a slowly built up uber-mask where 'significant' changes occur.
An accumulated difference can be done in script, but unfortunately I don't know of a way to do it that doesn't use more memory with every frame.
Here's a difference accumulator. This one abuses ScriptClip so it needs to be run linearly from start to finish, no multithreading - probably better done recursively. Needs masktools:
Thresh = 4 # Minimum pixel difference counting as a change

clip = WhateverSource( "path\to\your\video" ).ConvertToYV12() # Or other colorspaces supported by masktools

# Frame differences >= than threshold
diff = mt_lutxy( clip, clip.Loop(2,0,0), "x y - abs " + String(Thresh) + " < 0 x y - abs", U=3,V=3 )

# Accumulate differences - will slowly exhaust memory :(
accumdiff = BlankClip( clip, length=1, color_yuv=0 )
mask = clip.ScriptClip("""
accumdiff = mt_lutxy( accumdiff, diff.trim( current_frame,current_frame ), "x y +", U=3,V=3 )
accumdiff
""")

return mask # Separate mask in Y,U & V. Combine with code from my previous script if necessary, or change above to U=-128,V=-128

[DrPhill]

I have done the building an uber-mask before creating the dynamic mask. So my filter will first check for the areas where change is common and only output changes in those areas to the destination video. This removes a lot of the speckles (esp the annoying ones in the upper left).

Here is the result if you are interested (http://phillvanleersum.users.sourceforge.net/clips/Horner.2002-01-02.Script.mpg).

I am quite impressed how easy the combination of AVISynth and Filter writing makes this kind of effort.

Edit: Here is my source and stuff (http://phillvanleersum.users.sourceforge.net/AVISynth/AVISynth.html). It is early days but eventually I will release it under a suitably open license.

[Bloax]

Say, what happens if you run ReduceFluctuations(feedmeinteger) on the mask?

[DrPhill]

Say, what happens if you run ReduceFluctuations(feedmeinteger) on the mask?

Short answer: I am not sure......

I have not encountered ReduceFluctuations().

It also depends upon which mask you mean. Internally my filter generates a mask from all the changes between consequtive pairs of frames, with mechanisms to remove from consideration those areas with little change. This coarse mask is static - it does not change during the processing cycle - so there are no temporal fluctuations to reduce.

The second mask is generated from those significant differences that occur within the area allowed by the static mask. In this case the fluctuations are all required.

[Gavino]

Here's a difference accumulator. ...
...
accumdiff = BlankClip( clip, length=0 )
mask = clip.ScriptClip("""
accumdiff = mt_lutxy( accumdiff, diff.trim( current_frame,current_frame ), "x y +", U=3,V=3 )
accumdiff
""")
Since the clip is YV12, a default BlankClip will wrongly start off accumdiff with Y=16, not Y=0.
It will also have U=V=128, which is wrong if you want to accumulate chroma differences too.
Use color_yuv=0 (or $008080, depending on what you want to do with chroma).
Also, it would be 'cleaner' to use length=1, as the result using length=0 depends on unspecified behaviour of the filters concerned (eg ScriptClip will ask mt_lutxy for frame -1).

[-Vit-]

Thanks, that latter script was just a quick hack job to illustrate the idea. That approach is not generally useful because of the memory issue. First script is more careful. In general is there *any* way to abuse avisynth to get accumulating results without running out of memory?

[smok3]

I have done the building an uber-mask before creating the dynamic mask. So my filter will first check for the areas where change is common and only output changes in those areas to the destination video. This removes a lot of the speckles (esp the annoying ones in the upper left).

Here is the result if you are interested (http://phillvanleersum.users.sourceforge.net/clips/Horner.2002-01-02.Script.mpg).


Totally useless so far (a 2 minute mask done in gimp should look way better) and where is the dog? :)
Can you post the original snap and video to play around?

[DrPhill]

Totally useless so far (a 2 minute mask done in gimp should look way better) and where is the dog? :)
Can you post the original snap and video to play around?

I have the stuff documented here (http://phillvanleersum.users.sourceforge.net/ExtractMotion/ExtractMotion.html). This includes the intermediate stages, though the quality is lower because I used a shorter clip, meaning less accurate noise filtering. I put the original mpg up there - but it is a bit large. I tried shortening it, but kept getting problems with DirectShowSource when reloading the shortened version. I have no idea why.

Anyway, the code, scripts, dll, and usage is documented. Let me know what you think (but remember I only stumbled on AVISynth about a fortnight ago:o))


I will be back when I have some seasonal changes to merge together.

[-Vit-]

Here's a much nicer version of the river (http://www.mediafire.com/?e8zz2di7391298x) done with a refined version of my elaborate script above. I added denoising (that clip really needs it), some temporal smoothing and simple feathering, and tweaked the radius averaging a little. Also allows you to supply a rough mask to limit changes to an area. I used a rough mask for this clip (in the linked zip) - but it almost isn't needed as the script almost nails the mask anyway - except for one moving fern. I also adjusted the background image as it was not exactly aligned to the video (also in the zip), and trimmed off the first and last few frames as they were unstable/dupes. The script is probably overkill for this simple scene, but may not be complex enough to deal with moving objects entering the shot. The script settings would probably need to be changed for that situation. Edit: Added Opacity setting that lets the overlaid river be a little transparent to benefit a little from the increased resolution below.

Revised script below, needs fft3dfilter (http://avisynth.org.ru/fft3dfilter/fft3dfilter.html) now. It uses DirectShowSource and gets the audio from there. That's not ideal in general, but it's what DrPhill is using. Script is not fast, but I have a reputation to maintain:

VideoSource = "path\to\example.mpg"
BackgroundImage = "path\to\exampleAdjust.jpg"
RoughMaskImage = "path\to\exampleMask.png" # Only affect areas within this rough (b/w) mask, type "" if none

Denoise = 6.0 # How much to denoise video for analysis, can improve mask - does not affect video used in final merge
Radius = 5 # Number of frames before and after current used for comparison (sensible values 1 to 100+, depends on source)
# Larger (slower) values detect more gradual change (which may be a good or bad thing). Related to Uniformity below.
Step = 1 # Step to speed up larger radius values (e.g. 2, 3...)
MinDiff = 1 # Pixel differences of this value or less are ignored
Uniformity = 0.7 # Balance between "max" (0.0) and "avg" (1.0) when combining the differences found locally, range 0.0 to 1.0
# Most suitable value depends on Radius and type of change in the source - must experiment
LocalThresh = 12 # Difference between current frame and local frames that counts as an 'actual' difference
# Important to tweak to suit your source (values 2 to 16+)
RemoveSpots = 1 # Mask post-process - remove small spots of difference. May help clean up mis-detections, increase for stronger effect
RemoveHoles = 0 # Mask post-process - remove gaps in the mask. Increase to fill in larger holes
Smooth = 0 # Mask post-process - smooth mask edges
Soften = 2 # Mask post-process - soften (feather) mask edges
TempSoften = 1 # Mask post-process - temporal soften of mask (radius), may not work well with moving objects
Opacity = 192 # Opacity of overlaid video in final merge (0-255), decrease to make video more transparent
FirstFrameMask = false # Additionally compare current frame with first frame - useful given *very* stable background
FirstThresh = 8 # Difference between current frame and first frame that counts as an 'actual' difference
# Tweak to adjust area detected as different

# Load sources
videoOrig = DirectShowSource( VideoSource ).ConvertToYV12()
video = videoOrig.fft3dfilter( sigma=Denoise )
fc = video.FrameCount()
background = ImageSource( BackgroundImage, end=fc-1 ).ConvertToYV12()
roughMask = RoughMaskImage != "" ? ImageSource( RoughMaskImage, end=fc-1 ).ConvertToYV12() : NOP()

# Compare current frame with first frame
initDiff = FirstFrameMask ? mt_lutxy( video, video.FreezeFrame(1,fc-1,0), "x y - abs " + String(minDiff) + " < 0 x y - abs", U=3,V=3 ) : 0
initMask = FirstFrameMask ? initDiff.mt_binarize( FirstThresh, U=3,V=3 ) : 0

# Compare current frame with local frames
v1 = video.trim(1,Radius).Reverse() + video + video.trim(fc-Radius-1,fc-2).Reverse()
avgDiff = BlankClip( video, color_yuv=$000000 )
maxDiff = BlankClip( video, color_yuv=$000000 )
GScript("""
for ( i = 0, Radius * 2, Step ) {
if (i != Radius) {
diff = mt_lutxy( video, v1.trim(i,0), "x y - abs", U=3,V=3 )
avgDiff = mt_lutxy( avgDiff, diff, "y " + String(minDiff) + " < x x y + ?", U=3,V=3 )
maxDiff = mt_logic( maxDiff, diff, "max", U=3,V=3 )
}
}
""")
numDiffs = (Radius * 2 / Step + 1) - ((Radius % Step == 0) ? 1 : 0)
avgDiff = avgDiff.mt_lut( "x " + String(numDiffs) + " /", U=3,V=3 )
localMask = Merge( maxDiff, avgDiff, Uniformity ).mt_binarize( LocalThresh, U=3,V=3 )

# Combine local and first-frame masks, combine Y,U & V masks and post-process
mask = FirstFrameMask ? mt_logic( initMask, localMask, "max", U=3,V=3 ) : localMask
chrmask = mask.UToY()
umask = chrmask.Spline36Resize(mask.Width(),mask.Height())
vmask = mask.VToY().Spline36Resize(mask.Width(),mask.Height())
mask = mask.mt_logic( umask, "max", U=1,V=1 ).mt_logic( vmask, "max", U=-128,V=-128 )
mask = IsClip(roughMask) ? mask.mt_logic( roughMask, "min", U=1,V=1 ) : mask
GScript("""
for ( i = 0, RemoveSpots ) { mask = mask.mt_deflate(U=1,V=1) }
for ( i = 0, RemoveSpots ) { mask = mask.mt_inflate(U=1,V=1) }
mask = RemoveSpots > 0 ? mask.mt_binarize( 128, U=1,V=1 ) : mask
for ( i = 0, RemoveHoles ) { mask = mask.mt_inflate(U=1,V=1) }
for ( i = 0, RemoveHoles ) { mask = mask.mt_expand(U=1,V=1) }
multiplier = RemoveHoles == 0 ? 0 : 1 + 1 / (1 + Pow( Log(RemoveHoles) / Log(10), 3))
for ( i = 0, Int(RemoveHoles*multiplier) ) { mask = mask.mt_inpand(U=1,V=1) }
mask = RemoveHoles > 0 ? mask.mt_binarize( 0, U=1,V=1 ) : mask
for ( i = 0, Smooth ) { mask = mask.RemoveGrain(12) }
for ( i = 0, Soften) { mask = mask.Blur(1.58) }
""")
mask = mask.TemporalSoften( TempSoften, 255,0, 28, 2 )
mask = mask.mt_binarize( Opacity, mode="t x" )
chrmask = mask.Spline36Resize(chrmask.Width(),chrmask.Height())
mask = YToUV(chrmask, chrmask).MergeLuma(mask)

# Resize video and mask to suit background, then merge
videoOrig = videoOrig.Spline36Resize(background.Width(),background.Height())
mask = mask.Spline36Resize(background.Width(),background.Height())
mt_merge( background, videoOrig, mask, U=3,V=3 ).AssumeFPS(videoOrig.FrameRate())

# Add audio from input file (won't work with video-only sources, but using DirectShowSource here for simplicity)
AudioDub( videoOrig )

#return mask # Uncomment this line to see mask

[DrPhill]

I get an error at line 26

26: video = videoOrig.fft3dfilter( sigma=Denoise )

my AVISynth cannot find 'fft3dfilter'.

What am I doing wrong?

[ajk]

Did you install it as per the link in -Vit-'s post?

[DrPhill]

Did you install it as per the link in -Vit-'s post?

Doh, my bad.

I have managed to load all the prereqs apart from RemoveGrain:


loadplugin("c:\users\phill\VideoProject\fromDoom9\RemoveGrain\RemoveGrain.dll")

elicits an 'unable to load.......

[-Vit-]

Ah, RemoveGrain giving problems again, it's a troublesome filter to use. Try the RemoveGrainSSE2 version in this zip file (http://home.arcor.de/kassandro/prerelease/RemoveGrain-1.0.rar).
However, you don't need RemoveGrain for these particular settings. It is only needed if Smooth is set > 0, used to simplify the mask shape. Wanted to capture the edge detail for this river, so I found Smooth = 0 best here.

[DrPhill]

That works, thanks. The results are impressive - definitely superior to my results.

Now I need to try to understand the steps that you use to get this result. I also need to try to understand the clip.

Any hints that you can give me would be gratefully received - remember I have only been video processing for a couple of weeks........

.... but you have already been immensely helpful. I will be trying to repeat this process on five other scenes, captured on several occasions during the year.

As an aside
(1) is there a discussion of which input methods to use? It was implied above that DirectShowSource was not the best, but it was the first I stumbled across in the documentation. I do not want to keep the audio from the video - I will be supplying that later.

(2) is there any documentation of plugin management. I placed my plugins in the AVISynth instal\plugins, and added the recommended gorp to my registry, but that made no difference - AVISynth still failed to find my plugins hence the long absolute paths to them.....

[TheProfileth]

Hmm I tried to adapt -Vit-'s script to be used as a temporal mask of sorts for standard denoising, by replacing the background with videoOrig.filterxyz it works, but I think that a pixel weighting system would probably be better adapted for something like that, on the same idea, using a dynamic pixel weighting system over a dynamic mask could yield a more natural look because the mask tends to either overshoot or undershoot areas by a large amount, but then again I could be wrong.