Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 7th November 2002, 05:31   #1  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
NoMoSmooth (motion-adaptive spatio-temporal smoother)

While playing with code I wrote to apply 3x3 matrices temporally to 9 consecutive frames I was struck with an idea for a temporal smoother that might not be as suceptible to ghosting as others.

Edit 3: See this post and on for a rewritten version implementing motion-adaptive spatial-temporal smoothing.

Here's my first attempt at implementing it. It's proof of concept code, and as such is very (I mean extremely) slow and might not even do anything on your system other than crash. That said, it works ok on mine .

See the docs for more information on the algorithm.

As I said this is not intended to be a proper "release" of a filter. I'm more interested in receiving as much feedback as possible (positive or negative) than putting out something you can actually use in your encodes.

If people think this filter shows promise I'll fix it up.

Have fun and don't blame me .

Edit: Yes, I had some difficulty coming up with a good name. The one I chose was inspired by TomsMoComp and I think is fairly appropriate. Feel free to suggest another .

Edit 2: Removed links to old version.
__________________
Regards,
SansGrip

Last edited by SansGrip; 9th November 2002 at 19:21.
SansGrip is offline   Reply With Quote
Old 7th November 2002, 07:13   #2  |  Link
High Speed Dubb
Registered User
 
Join Date: Jan 2002
Posts: 283
It’s working well here — no crashes, and no artifacts.

I like the aspect that you give up averaging beyond a certain time if that timepoint shows evidence for motion. It would be interesting to see the same trick used for spatial smoothing, in which an intervening pixel of a different color would prevent averaging with a pixel further away.

The threshold of 2 is probably a little too low to make this effective. At that value, even very light noise is going to cut the averaging range short. At least for over the air stuff, you can expect a mean difference of about 4 to 7 (for luma and chroma combined) even in stationary areas. (Thanks go to folks over in the video capture forum for that number.)

If you decide to put more effort into this sequential approach, you might eventually want to look up hidden Markov models (HMMs). You could use an HMM to infer the probability that motion began at any particular time in the sequence.
__________________
9:) Lindsey Dubb
High Speed Dubb is offline   Reply With Quote
Old 7th November 2002, 13:28   #3  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
The threshold of 2 is probably a little too low to make this effective.

That's just the default I settled for on for my test material, which is a slightly noisy DVD. I didn't try on captured material yet.

You could use an HMM to infer the probability that motion began at any particular time in the sequence.

Sounds interesting. I'll ask Dr Google .
__________________
Regards,
SansGrip
SansGrip is offline   Reply With Quote
Old 7th November 2002, 17:12   #4  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
New version

Since my specified minimum of one person gave some positive feedback (and that person happened to be someone whose opinion I respect ) I rewrote the filter and optimized the algorithm.

In the process I improved performance somewhat (it's now only pretty darn slow ), discovered I could eliminate one of the two thresholds, and unfortunately had to lose "show" mode. Shouldn't matter too much though as it was mainly for debugging.

With some more positive feedback I'll go for a first release with (hopefully) better performance and any nice changes people might suggest.

Edit: Removed links to, er, redundant version .
__________________
Regards,
SansGrip

Last edited by SansGrip; 8th November 2002 at 00:40.
SansGrip is offline   Reply With Quote
Old 7th November 2002, 18:15   #5  |  Link
sh0dan
Retired AviSynth Dev ;)
 
sh0dan's Avatar
 
Join Date: Nov 2001
Location: Dark Side of the Moon
Posts: 3,480
I'm sorry, but what's the difference between this filter and C3D, except the bigger radius and slower processing?
__________________
Regards, sh0dan // VoxPod
sh0dan is offline   Reply With Quote
Old 7th November 2002, 18:22   #6  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
I'm sorry, but what's the difference between this filter and C3D, except the bigger radius and slower processing?

C3D does both spatial and temporal smoothing, this does only temporal. And the ability to use a larger radius without ghosting makes the averaging more accurate.
__________________
Regards,
SansGrip
SansGrip is offline   Reply With Quote
Old 7th November 2002, 18:46   #7  |  Link
sh0dan
Retired AviSynth Dev ;)
 
sh0dan's Avatar
 
Join Date: Nov 2001
Location: Dark Side of the Moon
Posts: 3,480
Oh - I see - thanks for the clarification

Edit: btw, the results looks good - smooths temporal noise very well, without any ghosting, and the speed is very acceptable.
__________________
Regards, sh0dan // VoxPod

Last edited by sh0dan; 7th November 2002 at 18:59.
sh0dan is offline   Reply With Quote
Old 7th November 2002, 19:18   #8  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
the results looks good - smooths temporal noise very well

I'm testing the code with radii up to 15 -- obviously unrealistic -- with good results (except the speed ). Though when the radius gets very large there's more danger of artifacts from scene changes, though theoretically these shouldn't be too noticible. I'm contemplating adding an optional scene-change detector in there.

At the moment I'm pondering an interleave scheme such as in TemporalSoften which should improve performance greatly (e.g. incrementing one pointer instead of one for each frame in the radius).

and the speed is very acceptable.

Either you're being kind or you've only tried low radii (3-5) or you're running it on a quad Xeon system . I don't think the speed is acceptable at the moment, though as I said interleaving will help that. I fear any improvements beyond that will have to wait until I've learned assembler (though this gives me good incentive to actually do it ).
__________________
Regards,
SansGrip
SansGrip is offline   Reply With Quote
Old 7th November 2002, 19:51   #9  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
I like the aspect that you give up averaging beyond a certain time if that timepoint shows evidence for motion.

That's gone (at least for the moment) because I find it works just as well, if not better, if I just skip that value and continue. Edit: This seems to make it identical to TemporalSoften. Note to self: Read Avisynth source properly before coding an idea .

It would be interesting to see the same trick used for spatial smoothing, in which an intervening pixel of a different color would prevent averaging with a pixel further away.

I'm coding it right now . Edit: No i'm not.
__________________
Regards,
SansGrip

Last edited by SansGrip; 8th November 2002 at 00:34.
SansGrip is offline   Reply With Quote
Old 7th November 2002, 21:05   #10  |  Link
sh0dan
Retired AviSynth Dev ;)
 
sh0dan's Avatar
 
Join Date: Nov 2001
Location: Dark Side of the Moon
Posts: 3,480
Quote:
Originally posted by SansGrip
[i]Either you're being kind or you've only tried low radii (3-5) or you're running it on a quad Xeon system .
Running it on a TBird 1.2Ghz, and I'm comparing the speed to temporalsoften. Speed is relative to the complexity to me.

Quote:
It would be interesting to see the same trick used for spatial smoothing, in which an intervening pixel of a different color would prevent averaging with a pixel further away.

I'm coding it right now
This is one of the tricks SmoothHiQ uses - the main problem is that you very seldom have straight lines to the center pixel, except when x or y is the same. I did a test before v1.0, where I actually tested all pixels inbetween pixel x (in the window around the pixel), based on a line drawn from it. It was _painfully_ (and I mean painfully) slow, and gave a very minimal visual improvement (if any).

Testing straight lines (with same x and y) is a posibility, but I don't think it'll give you much. Perhaps as gradiant improver on anim-e?

You should rather try different methods of weighing, and experiment with "frames further away have less influence", and "frames with pixels with bigger difference to the current has less impact on it".

(Example is in "float", but translates to fixed ints)
For the first you could create a table, where:
weight_frame[i] = 1/i, where i is your frame offset from the current frame. Then normalize the values in weight_frame[], so they add up to 1 (or a fixed point value, 65536 for example) .
Use this as a weight table. So you have:

c_frame = current frame number
c_ip = input pixel from frame c
ap = accumulated weighed pixels
aw = accumulated weights
ipw = inverse pixel weight
ip = input pixel (from the current frame 0)
op = output pixel

then for a passed pixels:

ap += weight_table[c_frame] * c_ip
aw += weight_table[c_frame]

For non-passed pixels you do nothing. To calculate the final pixel use:

ipw = 1 - aw
op = (ap + (ipv * ip)) / 1

Divide by one is to show that if you use fixed point you must divide with the accumulated weight value. This is a constant, so there is no divide in the algorithm (use >>16 for example).

So the fewer of the neighboring frames pass, the less the current pixel will be blurred. You can also do a "weigh with difference", where you have another table, where:

diff_table[i] = max(0,(1/threshold)*(threshold-i)). Then calculate ap and aw as:

weight = weight_table[c_frame] * difftable[abs(c_ip-ip)]
ap += weight* c_ip
aw += weight


- This is basicly what I use in SmoothHiQ for better averaging in "Weighed Average" mode.
__________________
Regards, sh0dan // VoxPod
sh0dan is offline   Reply With Quote
Old 7th November 2002, 23:08   #11  |  Link
neily
Registered User
 
Join Date: Nov 2001
Location: London, England
Posts: 157
SansGrip,

If you want someone to agree with you, yes it is very slow However I was more concerned about the lack of effect at the ends of a clip. The first frames included in the radius do not seem to get much change. I would have thought that they should at least by influenced by half the value of the radius frames rather than none at all.
neily is offline   Reply With Quote
Old 7th November 2002, 23:18   #12  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
The first frames included in the radius do not seem to get much change.

At the moment the first radius / 2 frames don't get any change.

I would have thought that they should at least by influenced by half the value of the radius frames rather than none at all.

They will, if this filter turns out to be useful enough to continue development on . Edit: Seems it is useful, so useful in fact that it's been in the core for ages . I'm somewhat red-faced now. I'll keep working on it and try to come up with something that makes it different . Hey -- at least I know I'm thinking the right things .
__________________
Regards,
SansGrip

Last edited by SansGrip; 8th November 2002 at 00:37.
SansGrip is offline   Reply With Quote
Old 8th November 2002, 03:24   #13  |  Link
High Speed Dubb
Registered User
 
Join Date: Jan 2002
Posts: 283
Sh0dan,

Very cool — That sounds like a very nice design. I really strongly favor the weighted average approach. I even have a convoluted, approximate justification for why that (threshold - difference)/threshold weighting should be fairly close to optimal.

SansGrip,

I did warn you that HMMs were my pet method. And I promise to do my best to unearn whatever respect you might have for my opinion.

The sequential approach does have some merit to it. For the most part, it shouldn’t make any difference. After all, if there’s a little evidence for motion at t = -1, then there will probably be more at t = -2. That’s probably why you didn’t see much effect when you removed it.

But there’s a very important exception. Motion over textures will often have a pixel value wavering around some value. In that case, it’s common for older pixel values to be closer than recent pixel values — but you still don’t want to average with those older values.

That may seem a little esoteric, but motion over low contrast texture is usually the hardest case for a temporal smoother, so it makes some sense to concentrate on it. (I usually set my smoothers to the point that they no longer mangle faces. So a small improvement with faces means I can use more aggressive settings for everything else.)

In other words, I’d take a close look at slowly moving smooth facial features, and see if the sequential version was able to reduce blending.


So thinking about Sh0dan’s description of SmoothHiQ, here’s another possible sequential method.

Starting with the current pixel (t=0), travel forward in time pixel by pixel. Probability(t=0 is stationary relative to t=0) = 1. Then let

Probability(t=n is stationary relative to t=0) =
Probability(t=n-1 is stationary relative to t=0)*Probability(t=n is stationary, given that t=0 through t=n-1 are stationary)

where
Probability(t=n is stationary, given that t=0 through t=n-1 are stationary) = (threshold - |Y(t = n) - Y(mean of 0 through n-1)|)/threshold (but limited to a minimum of 0)

(That last formula is a fudge, but is probably a good enough estimate to work with.)

So you can get Prob(t=1 is stationary) based on Prob(t=0 is stationary), get Prob(t=2 is stationary) based on Prob(t=1 is stationary), and so on. The exact same process can be run backward in time.

Then the final pixel value would be
Sum_over_pixels( pixel_color*Prob(pixel_is_stationary) ) / Sum_over_pixels( Prob(pixel_is_stationary) )
__________________
9:) Lindsey Dubb
High Speed Dubb is offline   Reply With Quote
Old 8th November 2002, 09:07   #14  |  Link
LigH
German doom9/Gleitz SuMo
 
LigH's Avatar
 
Join Date: Oct 2001
Location: Germany, rural Altmark
Posts: 4,738
@ SansGrip:

Seems that you suddenly removed it from your page; I hope I'll find your idea soon in another one of your projects when it is getting ready...
LigH is offline   Reply With Quote
Old 8th November 2002, 14:40   #15  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
Seems that you suddenly removed it from your page; I hope I'll find your idea soon in another one of your projects when it is getting ready...

That particular version was redundant. I'll post 0.0c if there's merit in the tweaks I'm making now .

@sh0dan & HSD:

Thanks for the suggestions, I was thinking about them last night and am currently playing .
__________________
Regards,
SansGrip

Last edited by SansGrip; 8th November 2002 at 14:43.
SansGrip is offline   Reply With Quote
Old 9th November 2002, 00:07   #16  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
Motion detector

I've been watching variously smoothed clips very carefully and have come to the conclusion (that of course many before me have reached) that detecting motion is the key to retaining detail.

To that end I've been playing with various algorithms and have come up with one that seems to work quite well. It effectively differentiates between motion and moderate noise (between 1-3). Since posting grabs would be pointless, I've made a 2.62mb 27-second clip to demonstrate. I know it's bad quality, but it is small .

The detection was done at 720x480 and then it was simple-resized down to its current resolution.

Anyway, before I start attempting to make it do something useful like smoothing, I thought I'd run it by people here to see if they can spot any obvious problems with the output. The threshold for this clip might have been a little on the sensitive side, but it's pretty finely tunable.

I'd appreciate any comments .
__________________
Regards,
SansGrip
SansGrip is offline   Reply With Quote
Old 9th November 2002, 02:07   #17  |  Link
High Speed Dubb
Registered User
 
Join Date: Jan 2002
Posts: 283
That looks reasonably good. It definitely gets the moving edges. Depending on the use for the motion estimate, that could be all you need.

The hard part for motion detection is identifying the insides of objects/areas in motion as moving. The green areas are missing the lower contrast stuff in faces and in the grass (toward the end of the clip). That’s very difficult to manage — I’m still looking for a better way to do it. But it can affect picture quality, since incorrectly smoothing/weaving inside textures doesn’t look so good.
__________________
9:) Lindsey Dubb
High Speed Dubb is offline   Reply With Quote
Old 9th November 2002, 05:37   #18  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
New version

Yes, it's time again to put my head on the block and see what comes a-swingin' .

Here's 0.0c, still proof of concept (source available here). It incorporates the abovementioned motion detector, with temporal smoothing applied to static areas and spatial smoothing to areas in motion.

While the motion detection algorithm is optimized (runs near real-time at 720x480 on my system) the smoothers aren't (and don't), with the spatial smoother being especially slow. This will be fixed if there's interest in me doing so.

Other caveats are it passes through pixels within spatial_radius of the edges and frames within temporal_radius of the ends of the clip.

See the documentation for more information and usage.

Here we go again...

Edit: Removed links to old version.
__________________
Regards,
SansGrip

Last edited by SansGrip; 16th November 2002 at 22:23.
SansGrip is offline   Reply With Quote
Old 10th November 2002, 00:45   #19  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
My observations so far:

It seems more tolerant of higher temporal settings (both radius and threshold) than spatial. High spatial settings (particularly threshold) seem to produce much more noticible artifacts.

Increasing caching with SetMemoryMax seems to increase speed, especially with a higher temporal radius.

Comments and suggestions always welcome .
__________________
Regards,
SansGrip
SansGrip is offline   Reply With Quote
Old 10th November 2002, 23:18   #20  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
2.5 YV12 version of 0.0c

In order to teach myself YV12 I decided to practice on my newest filter.

To that end, here is 0.0c built for Avisynth 2.5. Note: At the moment it only works in YV12. Also note it's not necessarily faster because it's totally unoptimized.

Currently the YV12 version only processes luma. When I've got a better handle on things I'll reintroduce chroma smoothing, hopefully in the next couple of days.

This was just a quick release (I didn't update the docs yet) to make sure that it works for people other than myself .

I'd be extremely grateful for feedback from those testing 2.5.
__________________
Regards,
SansGrip

Last edited by SansGrip; 11th November 2002 at 00:07.
SansGrip is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 16:48.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.