bidirectional temporal filter?

[goemon]

Anyone written a bidirectional temporal filter?

Problem with all the temporal filters I have seen is that they have problems with scene changes, it takes a couple frames after a scene change to accumulate pixels for noise reduction.

Smarter filter would note scene changes, then do bidirectional temporal filter within the scene from start to end, and end to start, that way there is no lag on scene changes.

Note: 3d filter is not equivalent to this since it's a sliding window of limited number of frames, travelling in 1 temporal direction.

[sh0dan]

Temporalsoften is bidirectional, for instance. It detects scenechange compared to the current frame, so scenechanges are not compared fram to frame, but current frame to tested frames.

Im not sure what your point is?

[goemon]

temporalsoften is a sliding window, and isnt at all what i'm talking about.

what i'm talking about is a filter which detects entire scene boundaries from start to end, then does forward and reverse temporal filter entire scene start to end, so there's no problem with scene change delay and no limitation of sliding window.

i'm not sure if such a filter is even possible though, i am thinking this is something an external program would have to do.

[temporance]

You have to remember that it is not just scene changes that are your enemy, it's any uncovered texture, for example, when a car moves along a street revealing houses behind it. There will be no accumulated texture info for the houses so they'll be noisier than they should be for a few frames. One technique is to combine temporal and spatial filtering so that the spatial filtering is turned on in the areas where there is no good temporal information (i.e. after scene changes and on revealed texture).

[goemon]

thats where bidirectional filtering could help. if you temporal filter backwards as well as forwards, youll get noisereduction for the hidden textures.

[sh0dan]

what i'm talking about is a filter which detects entire scene boundaries from start to end, then does forward and reverse temporal filter entire scene start to end, so there's no problem with scene change delay and no limitation of sliding window.

The sliding window is adjusted per frame by scenechange detection.

Temporalsoften was rewritten for AviSynth 2.5 - you might be thinking about earlier versions. In my experience there is not much to be gained by max radius>5.

Beside that Dup 2.20 offers blending of very similar frames, with a maximum of 25 frames.

[temporance]

Originally posted by goemon
thats where bidirectional filtering could help. if you temporal filter backwards as well as forwards, youll get noisereduction for the hidden textures. True - the temporal+spatial idea was for a zero delay filter.

I think whole-frame scene-change detection is a bad idea for a noise filter. It could cause non-linear behavior, especially during fades and motion that is detected as a scene change.

[goemon]

Originally posted by temporance
True - the temporal+spatial idea was for a zero delay filter.

I think whole-frame scene-change detection is a bad idea for a noise filter. It could cause non-linear behavior, especially during fades and motion that is detected as a scene change.

i dont see it would be any worse than any other filter with scene-change detection.

but i think such a filter would have to be a standalone program anyway, so you could put in interface for user to doublecheck scene boundaries and adjust them if the auto-detect gets them wrong. and deselect filtering on fades, etc.

[temporance]

No, the whole concept of a whole-frame scene change detection to control temporal filtering is a bad one, IMHO.

It's better to detect scene changes locally, on a pixel-by-pixel basis. And a soft decision is better than a hard decision. That's my last word on the subject :)

[ edit: it's just free advise for someone who spent years working on professional compression prefilters :D ]

[goemon]

you give no justification for either claim, so i discard them :D

[mf]

Originally posted by temporance
[ edit: it's just free advise for someone who spent years working on professional compression prefilters :D ]
You know NTSC->PAL conversions? Those were done "professionally" too :D. And broadcasts too, and cable digitizing too, and anime translation("licensing") too, etc etc etc. I think the term "professional" is losing its meaning in A/V stuff. Nothing personal against you, but professional tools are just as much in their infancy as free tools, and I see no reason why just anybody couldn't come up with a better idea than you.

[sh0dan]

Stay on topic, and keep the tone, or the thread will be closed.

[temporance]

Nothing personal against you, but professional tools are just as much in their infancy as free tools, and I see no reason why just anybody couldn't come up with a better idea than you.Not taken personally, mf. ;).

I agree, professional tools on PCs are in their infancy and are often equalled or bettered by open-source / amateur tools. However, almost all the algorithms and techniques that are now only just finding their way into PC software have been around for years or even decades in custom hardware used by broadcasters. So grizzled old engineers like me sometimes have a comfortable familiar feeling when people describe something as "new" that has actually been around for a long time in academia and hardware. A "bidirectional temporal filter" is one example (could also be called a non-causal temporal filter as it effectively looks into the future).

I have a huge amount of respect for people that come up with better ideas than me, infact for everyone that implements a cool idea with a passion. My replies on this thread weren't meant to criticize others' ideas, only to throw my experience into the discussion to help make better ideas overall.

An off-the-wall thought on the difference between professional and amateur: It's easy to make an algorithm that works for 99.99% of video content out there. But when you make an algorithm for a paying customer, he will always find that 0.01% of video that breaks your algorithm and you will be told your algo is no good.

And back on-topic: best approach would be soft-decision regional discontiunity detection (based on local luma SAD). This can be made to do everything whole-frame SCD can do. It will also handle part-frame scene changes, texture reveals and texture conceals. And it wouldn't produce the artifacts caused by false positives and false negatives of hard-decision whole-frame SCD. The downside is that it's more complex.

[lex3]

@goemon

I've just done a quick test:
- Temporalsoften does filtering of the keyframes (exactly as sh0dan said).
- Dust does this too.

Enjoy them :)
Or try the others :)

[mf]

Originally posted by temporance
I agree, professional tools on PCs are in their infancy and are often equalled or bettered by open-source / amateur tools. However, almost all the algorithms and techniques that are now only just finding their way into PC software have been around for years or even decades in custom hardware used by broadcasters. So grizzled old engineers like me sometimes have a comfortable familiar feeling when people describe something as "new" that has actually been around for a long time in academia and hardware. A "bidirectional temporal filter" is one example (could also be called a non-causal temporal filter as it effectively looks into the future).
Sorry for straying offtopic! I still think this is *kinda* ontopic, but I will stop after this. Please do not strike me, We're only following a small side-path on discussion here.
What I wanted to ask is: Still, how do you explain standards conversion? I don't think there is *any* excuse for that :p.


On the topic, I'd like to say I'd really like to see motion-compensated 3d smoothers. Simple pans can mess up a 3d smoother, and I think that's pretty weak. ;/

[temporance]

Still, how do you explain standards conversion? I don't think there is *any* excuse for that.

I'm not sure I understand what you're saying.

On the topic, I'd like to say I'd really like to see motion-compensated 3d smoothers. Simple pans can mess up a 3d smoother, and I think that's pretty weak. ;/

Yes, this would be nice to see, but it would probably work best with completely different ME to that used in video codecs. You really need pixel-resolution optical flow.

There is a problem though. When you have a whole-sceen pan right you can't filter the new texture that's revealed on the right-hand edge of the screen. So again you run into problems of different regions having different strength filtering.

This is why I think a hybrid temporal/spatial filter would be good (motion compensated or not). One in which the spatial filtering is only enabled in parts of the image where the temporal filter has no good historical (or future) data. You may say "bad idea, spatial filtering makes blurring" but think about it... the areas which are spatially filtered have just been revealed and the eye hasn't had a chance yet to focus on them.

[dokworm]

So has anyone written one yet?

[Mug Funky]

there's building-block plugins for avisynth that allow you to construct extremely complex and effective noise reduction.

the original idea of this thread is sort of whimsical - whole-scene temporal smoothing is pointless for all but the most boring scenes. reason: things don't only change at scenechanges. the last frame before a cut will probably as much in common with the first frame as any scenechange. there's simply nothing that any reference frame can "learn" from a frame so far away from it, unless the scene is completely static.

however, luma compensation can be very useful for dust+scratch removal, and creating more robust motion-compensation. i've experimented with it a tad, but it's still very much on the drawing board for now.

the trick with temporal smoothing is getting as many similar frames together as possible and performing some kind of average/median/etc on it. but the frames used must be as similar as possible, or the smoothing will spoil motion as well (and hence detail will be lost).

[AVIL]

Hi all,

Backward and forward temporal denoising could be a good idea but (i think) with a poor performance. An approximation could be done with a temporal filter passed two times. One with the footage in normal direction (first scene first ) and the other with the footage reversed (last scene first). After that, a "magic" process select the best pixels from the two filtered videos. The magic process could be an average (weighted or not).

Certainly, some more noise are eliminated but at the cost of multiply process time by 3.