Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 1st July 2015, 19:32   #1  |  Link
joj
Registered User
 
Join Date: Jan 2013
Posts: 6
ShapeMask - Dynamically mask out bright shapes

If you're watching a lot of screen casts and recorded talks, and are as sick as I am of the over-use of white (console backgrounds, web pages, projections ...), then you may want to try out a plugin I created to mitigate that health hazard.

A compiled binary is available here.
sha1: 5af5727b572b5e14bd32d1f8ee71d73522d41286

Source: github and there you also find the most readable version of the documentation, including example and screenshots.

The plugin is a bit on the large side since it includes parts of OpenCV, a library that is ... challenging to prune.

A note on the implementation. It could probably be optimized a bit more with AVX and some more SSE2, but as it stands it is sufficiently fast for my needs at least (getting about 350 fps with avsmeter).
It does require a CPU that has SSE2 (I believe, don't have one without) as the CV bits were configured for utilizing that. Hopefully that won't be an issue as even the cheapest laptop has that instruction set nowadays..

Let me know if it crashes on you. If it does, then I've probably stripped CV down too much..

Last edited by joj; 1st July 2015 at 19:35.
joj is offline   Reply With Quote
Old 3rd July 2015, 23:09   #2  |  Link
joj
Registered User
 
Join Date: Jan 2013
Posts: 6
Issues

The envisioned benefit of this plugin over ordinary binarize masks is that it eliminates a lot of noise. However, being currently a purely spatial filter it has the drawback of not being able to differentiate background from foreground and this poses a problem in certain cases. Specifically it tends to perform badly when a shape changes form in particular ways, such as a subject connecting different body parts or when the camera captures only parts of objects, which may cause background to shine through "small holes",

What happens in those cases is that "new shapes" / objects are being created, and if they're too small for the specified threshold, then they're excluded from masking consideration.

Example:


The examples were created as follows:
Code:
Overlay(Invert, mask=Binarize(190, upper=false))
Overlay(Invert, mask=ShapeMask(190, minarea=0.02))
Some thoughts I've had to improve the performance of the filter are:
  • Consider temporal information as well. For example, track if new shapes were previously part of other shapes. That is, deemed connected in the previous frame. Then carry that information forward until the shape vanishes.
  • Leverage motion vector information. E.g. if a shape is more or less static, then override the area based constraint and consider the shape for inclusion anyway, provided it meets the luma threshold of course. Or perhaps compare the motion with other shapes, and if it's different "enough" to that of the "others", then flag it as potentially a new shape. Other would need to be defined somehow, such as 1 or 2 degrees removed but it seems this would provide some additional information for decision making regardless of specifics.
There are still issues I can think of that these new tools can't solve. The rotating subject described above, a white shirt on a white background would still be hard to differentiate since the only indication that something has happened is that the surrounding shape has deformed (expanded) in the location where the shirt would be at say a 90 degree turn from facing the background. How would the boundary be identified at this point?

If someone has any other ideas on how to deal with the shape distinction issue, I'd be glad to hear it.

Last edited by joj; 4th July 2015 at 07:38.
joj is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 08:18.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.