Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
3rd January 2012, 11:05 | #21 | Link |
Registered User
Join Date: Dec 2011
Location: West Somerset, UK
Posts: 130
|
I remember RPN from my first calculator. Seems an extravagant way to avoid implementing parentheses. My problems with the filter seemed deeper rooted - but never mind, I am on a roll now.
Thanks for the pointer to HCEnc - works well and is a good price. I have made my first attempt available for the interested. I need to create a (static) mask to limit the dynamic mask, as video fluctuations outside the stream are causing flicker. I plan to sum the number differences into frame zero, so this will end up a grey-scale of amount of change. I can then either convert this to B&W or use as is to ignore changes in areas with few changes. So much for my quick foray into video processing. |
3rd January 2012, 15:58 | #23 | Link |
Registered User
Join Date: Jul 2010
Posts: 448
|
I needed something like this once. Had a stable background with a rather slow moving foreground I wanted to mask. It might work for your task too...
Rather than a single frame it compares several frames forwards and backwards and optionally the first frame too. Also post-processes the mask for a cleaner result. It can be successful given stable footage, but tweaking the settings is very important. Particularly important are LocalThresh and Radius, but tbh most of the settings have quite an impact. Hard to guess how it will work for you without having seen the original footage. You may need to increase LocalThresh to eliminate small light changes / noise (my footage was clean). Probably worth experimenting with Radius, Uniformity and the mask post-processing settings too. I just ran your result footage through it and setting RemoveSpots to about 4 or so cleared up the speckles... Add a last line "return mask" to see the mask being generated, which helps tweak settings. I've commented it up and tailored it to suit what I think your requirements are (the source loading and the final resizing before the merge - alter if necessary). Needs GScript, masktools V2 and RemoveGrain Code:
Radius = 5 # Number of frames before and after current used for comparison (sensible values 1 to 100+, depends on source) # Larger (slower) values detect more gradual change (which may be a good or bad thing). Related to Uniformity below. Step = 1 # Step to speed up larger radius values (e.g. 2, 3...) MinDiff = 1 # Pixel differences of this value or less are ignored Uniformity = 0.7 # Balance between "max" (0.0) and "avg" (1.0) when combining the differences found locally, range 0.0 to 1.0 # Most suitable value depends on Radius and type of change in the source - must experiment LocalThresh = 4 # Difference between current frame and local frames that counts as an 'actual' difference # Important to tweak to suit your source (values 2 to 16+) RemoveSpots = 1 # Mask post-process - remove small spots of difference. May help clean up mis-detections, increase for stronger effect RemoveHoles = 4 # Mask post-process - remove gaps in the mask. Increase to fill in larger holes Soften = 2 # Mask post-process - soften mask edges FirstFrameMask = false # Additionally compare current frame with first frame - useful given *very* stable background FirstThresh = 8 # Difference between current frame and first frame that counts as an 'actual' difference # Tweak to adjust area detected as different # Load sources video = WhateverSource( "path\to\your\video" ).ConvertToYV12() # Possibly supports YUY2 or YV16 - haven't tried it fc = video.FrameCount() background = ImageSource("path\to\your\background", end=fc-1 ).ConvertToYV12() # --"-- # Compare current frame with first frame initDiff = FirstFrameMask ? mt_lutxy( video, video.FreezeFrame(1,fc-1,0), "x y - abs " + String(minDiff) + " < 0 x y - abs", U=3,V=3 ) : 0 initMask = FirstFrameMask ? initDiff.mt_binarize( FirstThresh, U=3,V=3 ) : 0 # Compare current frame with local frames v1 = video.Loop(Radius,0,0) avgDiff = BlankClip( video, color_yuv=$000000 ) maxDiff = BlankClip( video, color_yuv=$000000 ) GScript(""" for ( i = 0, Radius * 2, Step ) { if (i != Radius) { diff = mt_lutxy( video, v1.trim(i,0), "x y - abs", U=3,V=3 ) avgDiff = mt_lutxy( avgDiff, diff, "y " + String(minDiff) + " < x x y + ?", U=3,V=3 ) maxDiff = mt_logic( maxDiff, diff, "max", U=3,V=3 ) } } """) numDiffs = (Radius * 2 / Step + 1) - ((Radius % Step == 0) ? 1 : 0) avgDiff = avgDiff.mt_lut( "x " + String(numDiffs) + " /", U=3,V=3 ) localMask = Merge( maxDiff, avgDiff, Uniformity ).mt_binarize( LocalThresh, U=3,V=3 ) # Combine local and first-frame masks, combine Y,U & V masks and post-process mask = FirstFrameMask ? mt_logic( initMask, localMask, "max", U=3,V=3 ) : localMask chrmask = mask.UToY() umask = chrmask.Spline36Resize(mask.Width(),mask.Height()) vmask = mask.VToY().Spline36Resize(mask.Width(),mask.Height()) mask = mask.mt_logic( umask, "max", U=1,V=1 ).mt_logic( vmask, "max", U=-128,V=-128 ) GScript(""" for ( i = 0, RemoveSpots ) { mask = mask.mt_deflate(U=1,V=1) } for ( i = 0, RemoveSpots ) { mask = mask.mt_inflate(U=1,V=1) } mask = RemoveSpots > 0 ? mask.mt_binarize( 128, U=1,V=1 ) : mask for ( i = 0, RemoveHoles ) { mask = mask.mt_inflate(U=1,V=1) } for ( i = 0, RemoveHoles ) { mask = mask.mt_expand(U=1,V=1) } multiplier = RemoveHoles == 0 ? 0 : 1 + 1 / (1 + Pow( Log(RemoveHoles) / Log(10), 3)) # lol... for ( i = 0, Int(RemoveHoles*multiplier) ) { mask = mask.mt_inpand(U=1,V=1) } mask = RemoveHoles > 0 ? mask.mt_binarize( 0, U=1,V=1 ) : mask for ( i = 0, Soften ) { mask = mask.RemoveGrain(12) } """) chrmask = mask.Spline36Resize(chrmask.Width(),chrmask.Height()) mask = YToUV(chrmask, chrmask).MergeLuma(mask) # Resize video and mask to suit background, then merge video = video.Spline36Resize(background.Width(),background.Height()) mask = mask.Spline36Resize(background.Width(),background.Height()) mt_merge( background, video, mask, U=3,V=3 ) You can also take an MVTools (motion analysis) approach too: blocks that are moving or have a poor motion match are counted as different. However, I don't think that will work so well for this example. Last edited by -Vit-; 3rd January 2012 at 16:02. |
3rd January 2012, 17:07 | #24 | Link |
Registered User
Join Date: Dec 2011
Location: West Somerset, UK
Posts: 130
|
Wow, that looks complex...... scarily so. I am going to ponder the techniques over a cuppa. The radius ideas may be useful, though the flowing water benefits from sharp detailed movement.
What I have done is add another function to my filter. This mallocs a data array (4 bytes per pixel) and inits these to 255; During image processing differences per pixel from previous frame are detected Code:
diff = abs(r1 - r0) + abs(g1 - g0) + abs(b1 - b0) + abs(a1 - a0); If the value falls below a trigger level then the matching pixel in the output image is set to 0, else it is set to 255. The result is a slowly built up uber-mask where 'significant' changes occur. The final frame should be the finished uber-mask if I can work out how to dump it reliably from the script. I will then apply that mask to the video to screen out unwanted changes. I may need to apply a slightly different algorithm. On frame(0) do a scan of all frames to detect differences and generate the uber-mask. Then on frames 1-n I can apply the mask. This may cause a delay at the start of processing........ |
3rd January 2012, 17:57 | #25 | Link | ||
Registered User
Join Date: Jul 2010
Posts: 448
|
Quote:
Quote:
Here's a difference accumulator. This one abuses ScriptClip so it needs to be run linearly from start to finish, no multithreading - probably better done recursively. Needs masktools: Code:
Thresh = 4 # Minimum pixel difference counting as a change clip = WhateverSource( "path\to\your\video" ).ConvertToYV12() # Or other colorspaces supported by masktools # Frame differences >= than threshold diff = mt_lutxy( clip, clip.Loop(2,0,0), "x y - abs " + String(Thresh) + " < 0 x y - abs", U=3,V=3 ) # Accumulate differences - will slowly exhaust memory :( accumdiff = BlankClip( clip, length=1, color_yuv=0 ) mask = clip.ScriptClip(""" accumdiff = mt_lutxy( accumdiff, diff.trim( current_frame,current_frame ), "x y +", U=3,V=3 ) accumdiff """) return mask # Separate mask in Y,U & V. Combine with code from my previous script if necessary, or change above to U=-128,V=-128 Last edited by -Vit-; 3rd January 2012 at 20:52. Reason: fix |
||
3rd January 2012, 18:28 | #26 | Link |
Registered User
Join Date: Dec 2011
Location: West Somerset, UK
Posts: 130
|
I have done the building an uber-mask before creating the dynamic mask. So my filter will first check for the areas where change is common and only output changes in those areas to the destination video. This removes a lot of the speckles (esp the annoying ones in the upper left).
Here is the result if you are interested. I am quite impressed how easy the combination of AVISynth and Filter writing makes this kind of effort. Edit: Here is my source and stuff. It is early days but eventually I will release it under a suitably open license. Last edited by DrPhill; 3rd January 2012 at 18:46. |
3rd January 2012, 19:29 | #28 | Link | |
Registered User
Join Date: Dec 2011
Location: West Somerset, UK
Posts: 130
|
Quote:
I have not encountered ReduceFluctuations(). It also depends upon which mask you mean. Internally my filter generates a mask from all the changes between consequtive pairs of frames, with mechanisms to remove from consideration those areas with little change. This coarse mask is static - it does not change during the processing cycle - so there are no temporal fluctuations to reduce. The second mask is generated from those significant differences that occur within the area allowed by the static mask. In this case the fluctuations are all required. |
|
3rd January 2012, 20:05 | #29 | Link | |
Avisynth language lover
Join Date: Dec 2007
Location: Spain
Posts: 3,431
|
Quote:
It will also have U=V=128, which is wrong if you want to accumulate chroma differences too. Use color_yuv=0 (or $008080, depending on what you want to do with chroma). Also, it would be 'cleaner' to use length=1, as the result using length=0 depends on unspecified behaviour of the filters concerned (eg ScriptClip will ask mt_lutxy for frame -1). |
|
3rd January 2012, 20:56 | #30 | Link |
Registered User
Join Date: Jul 2010
Posts: 448
|
Thanks, that latter script was just a quick hack job to illustrate the idea. That approach is not generally useful because of the memory issue. First script is more careful. In general is there *any* way to abuse avisynth to get accumulating results without running out of memory?
|
3rd January 2012, 21:51 | #31 | Link | |
brontosaurusrex
Join Date: Oct 2001
Posts: 2,392
|
Quote:
Can you post the original snap and video to play around?
__________________
certain other member Last edited by smok3; 3rd January 2012 at 21:54. |
|
4th January 2012, 20:33 | #32 | Link | |
Registered User
Join Date: Dec 2011
Location: West Somerset, UK
Posts: 130
|
Quote:
Anyway, the code, scripts, dll, and usage is documented. Let me know what you think (but remember I only stumbled on AVISynth about a fortnight ago)) I will be back when I have some seasonal changes to merge together. |
|
5th January 2012, 02:28 | #33 | Link |
Registered User
Join Date: Jul 2010
Posts: 448
|
Here's a much nicer version of the river done with a refined version of my elaborate script above. I added denoising (that clip really needs it), some temporal smoothing and simple feathering, and tweaked the radius averaging a little. Also allows you to supply a rough mask to limit changes to an area. I used a rough mask for this clip (in the linked zip) - but it almost isn't needed as the script almost nails the mask anyway - except for one moving fern. I also adjusted the background image as it was not exactly aligned to the video (also in the zip), and trimmed off the first and last few frames as they were unstable/dupes. The script is probably overkill for this simple scene, but may not be complex enough to deal with moving objects entering the shot. The script settings would probably need to be changed for that situation. Edit: Added Opacity setting that lets the overlaid river be a little transparent to benefit a little from the increased resolution below.
Revised script below, needs fft3dfilter now. It uses DirectShowSource and gets the audio from there. That's not ideal in general, but it's what DrPhill is using. Script is not fast, but I have a reputation to maintain: Code:
VideoSource = "path\to\example.mpg" BackgroundImage = "path\to\exampleAdjust.jpg" RoughMaskImage = "path\to\exampleMask.png" # Only affect areas within this rough (b/w) mask, type "" if none Denoise = 6.0 # How much to denoise video for analysis, can improve mask - does not affect video used in final merge Radius = 5 # Number of frames before and after current used for comparison (sensible values 1 to 100+, depends on source) # Larger (slower) values detect more gradual change (which may be a good or bad thing). Related to Uniformity below. Step = 1 # Step to speed up larger radius values (e.g. 2, 3...) MinDiff = 1 # Pixel differences of this value or less are ignored Uniformity = 0.7 # Balance between "max" (0.0) and "avg" (1.0) when combining the differences found locally, range 0.0 to 1.0 # Most suitable value depends on Radius and type of change in the source - must experiment LocalThresh = 12 # Difference between current frame and local frames that counts as an 'actual' difference # Important to tweak to suit your source (values 2 to 16+) RemoveSpots = 1 # Mask post-process - remove small spots of difference. May help clean up mis-detections, increase for stronger effect RemoveHoles = 0 # Mask post-process - remove gaps in the mask. Increase to fill in larger holes Smooth = 0 # Mask post-process - smooth mask edges Soften = 2 # Mask post-process - soften (feather) mask edges TempSoften = 1 # Mask post-process - temporal soften of mask (radius), may not work well with moving objects Opacity = 192 # Opacity of overlaid video in final merge (0-255), decrease to make video more transparent FirstFrameMask = false # Additionally compare current frame with first frame - useful given *very* stable background FirstThresh = 8 # Difference between current frame and first frame that counts as an 'actual' difference # Tweak to adjust area detected as different # Load sources videoOrig = DirectShowSource( VideoSource ).ConvertToYV12() video = videoOrig.fft3dfilter( sigma=Denoise ) fc = video.FrameCount() background = ImageSource( BackgroundImage, end=fc-1 ).ConvertToYV12() roughMask = RoughMaskImage != "" ? ImageSource( RoughMaskImage, end=fc-1 ).ConvertToYV12() : NOP() # Compare current frame with first frame initDiff = FirstFrameMask ? mt_lutxy( video, video.FreezeFrame(1,fc-1,0), "x y - abs " + String(minDiff) + " < 0 x y - abs", U=3,V=3 ) : 0 initMask = FirstFrameMask ? initDiff.mt_binarize( FirstThresh, U=3,V=3 ) : 0 # Compare current frame with local frames v1 = video.trim(1,Radius).Reverse() + video + video.trim(fc-Radius-1,fc-2).Reverse() avgDiff = BlankClip( video, color_yuv=$000000 ) maxDiff = BlankClip( video, color_yuv=$000000 ) GScript(""" for ( i = 0, Radius * 2, Step ) { if (i != Radius) { diff = mt_lutxy( video, v1.trim(i,0), "x y - abs", U=3,V=3 ) avgDiff = mt_lutxy( avgDiff, diff, "y " + String(minDiff) + " < x x y + ?", U=3,V=3 ) maxDiff = mt_logic( maxDiff, diff, "max", U=3,V=3 ) } } """) numDiffs = (Radius * 2 / Step + 1) - ((Radius % Step == 0) ? 1 : 0) avgDiff = avgDiff.mt_lut( "x " + String(numDiffs) + " /", U=3,V=3 ) localMask = Merge( maxDiff, avgDiff, Uniformity ).mt_binarize( LocalThresh, U=3,V=3 ) # Combine local and first-frame masks, combine Y,U & V masks and post-process mask = FirstFrameMask ? mt_logic( initMask, localMask, "max", U=3,V=3 ) : localMask chrmask = mask.UToY() umask = chrmask.Spline36Resize(mask.Width(),mask.Height()) vmask = mask.VToY().Spline36Resize(mask.Width(),mask.Height()) mask = mask.mt_logic( umask, "max", U=1,V=1 ).mt_logic( vmask, "max", U=-128,V=-128 ) mask = IsClip(roughMask) ? mask.mt_logic( roughMask, "min", U=1,V=1 ) : mask GScript(""" for ( i = 0, RemoveSpots ) { mask = mask.mt_deflate(U=1,V=1) } for ( i = 0, RemoveSpots ) { mask = mask.mt_inflate(U=1,V=1) } mask = RemoveSpots > 0 ? mask.mt_binarize( 128, U=1,V=1 ) : mask for ( i = 0, RemoveHoles ) { mask = mask.mt_inflate(U=1,V=1) } for ( i = 0, RemoveHoles ) { mask = mask.mt_expand(U=1,V=1) } multiplier = RemoveHoles == 0 ? 0 : 1 + 1 / (1 + Pow( Log(RemoveHoles) / Log(10), 3)) for ( i = 0, Int(RemoveHoles*multiplier) ) { mask = mask.mt_inpand(U=1,V=1) } mask = RemoveHoles > 0 ? mask.mt_binarize( 0, U=1,V=1 ) : mask for ( i = 0, Smooth ) { mask = mask.RemoveGrain(12) } for ( i = 0, Soften) { mask = mask.Blur(1.58) } """) mask = mask.TemporalSoften( TempSoften, 255,0, 28, 2 ) mask = mask.mt_binarize( Opacity, mode="t x" ) chrmask = mask.Spline36Resize(chrmask.Width(),chrmask.Height()) mask = YToUV(chrmask, chrmask).MergeLuma(mask) # Resize video and mask to suit background, then merge videoOrig = videoOrig.Spline36Resize(background.Width(),background.Height()) mask = mask.Spline36Resize(background.Width(),background.Height()) mt_merge( background, videoOrig, mask, U=3,V=3 ).AssumeFPS(videoOrig.FrameRate()) # Add audio from input file (won't work with video-only sources, but using DirectShowSource here for simplicity) AudioDub( videoOrig ) #return mask # Uncomment this line to see mask Last edited by -Vit-; 5th January 2012 at 06:30. Reason: opacity improvement |
5th January 2012, 16:49 | #37 | Link |
Registered User
Join Date: Jul 2010
Posts: 448
|
Ah, RemoveGrain giving problems again, it's a troublesome filter to use. Try the RemoveGrainSSE2 version in this zip file.
However, you don't need RemoveGrain for these particular settings. It is only needed if Smooth is set > 0, used to simplify the mask shape. Wanted to capture the edge detail for this river, so I found Smooth = 0 best here. |
5th January 2012, 17:17 | #38 | Link |
Registered User
Join Date: Dec 2011
Location: West Somerset, UK
Posts: 130
|
That works, thanks. The results are impressive - definitely superior to my results.
Now I need to try to understand the steps that you use to get this result. I also need to try to understand the clip. Any hints that you can give me would be gratefully received - remember I have only been video processing for a couple of weeks........ .... but you have already been immensely helpful. I will be trying to repeat this process on five other scenes, captured on several occasions during the year. As an aside (1) is there a discussion of which input methods to use? It was implied above that DirectShowSource was not the best, but it was the first I stumbled across in the documentation. I do not want to keep the audio from the video - I will be supplying that later. (2) is there any documentation of plugin management. I placed my plugins in the AVISynth instal\plugins, and added the recommended gorp to my registry, but that made no difference - AVISynth still failed to find my plugins hence the long absolute paths to them..... |
7th January 2012, 01:35 | #39 | Link |
Leader of Dual-Duality
Join Date: Aug 2010
Location: America
Posts: 134
|
Hmm I tried to adapt -Vit-'s script to be used as a temporal mask of sorts for standard denoising, by replacing the background with videoOrig.filterxyz it works, but I think that a pixel weighting system would probably be better adapted for something like that, on the same idea, using a dynamic pixel weighting system over a dynamic mask could yield a more natural look because the mask tends to either overshoot or undershoot areas by a large amount, but then again I could be wrong.
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood My New filter is in the works, and will be out soon |
|
|