A "psychovisual" approach to noise reduction

[SansGrip]

I've been using Avisynth (and lurking in this forum) for quite a while, and experimenting with the various noise reduction filters, during which time some thoughts have occurred to me. I don't claim these are original or even totally on-topic (my excuse being that I've been testing various algorithms as Avisynth filters), but here they are anyway.

From my tests (mainly DVD rips but also some captures of varying quality VHS tapes) I've gradually settled on Convolution3D as a good all-round noise reduction filter, especially when applied prior to resizing and with higher values than the recommended 8 for chroma. A 3x3 average blur seems to be the most effective at increasing compressibility (or, in my case, decreasing the Q level of a CBR MPEG-1), but obviously is also the most destructive.

Selective filters such as msmooth do a good job, and I believe are, theoretically speaking, on the right track as far as noise reduction goes. That said, however, smoothing between edges is a fairly simplistic approach - again, from a conceptual and not a practical point of view - and most of my thoughts have been towards other factors besides edges that could be used to influence a smoother.

Other factors I've identified might be:

1) Detection of "focus." From my tests with various home-grown algorithms this sounds a lot simpler than it is. While SLR cameras are able to auto-focus very effectively by, apparently, comparing tonal contrast levels at various hotspots in the frame, I'm not sure how easily this translates to a flat image (more sophisticated cameras use distance-detection to aid in determining focus, something that obviously cannot be done with a 2D array of pixels). It would seem to me that objects out of focus can receive more aggressive noise reduction. Perhaps tonal contrast might be combined with judging the "sharpness" of existing edges. From what I've read this has something to do with the high frequency parts of the picture, but I've yet to find locally a book on image processing so my knowledge ends here. I'm hoping someone can pick this up and run with it, or explain to me why this is an exercise in futility

2) Detection of motion. I've dabbled with absolute luminance difference which does a fairly good job at finding parts of the frame in motion, but seems to be attracted to the edges of moving objects rather than the entire object. My thinking here is that once moving objects are identified, anything else must be fairly stationary and can perhaps receive slightly more aggressive smoothing. It also might be the case that scenes with very low motion could handle stronger temporal smoothing than, for example, pans, which tend to produce "trails" with over-aggressive temporal blurring.

3) Framing. While not always (or even mostly) true, the centre of the frame tends to contain the most important information, and is certainly where the eyes seem to "return" upon a scene change. This would imply that smoothing can often be weighted towards the edges of the frame, when other factors (such as focus) determine nothing important resides there.

4) Scene changes. Since it takes a finite - possibly relatively substantial - amount of time for the brain to register a scene change, and there is a momentary "confusion" while you reorientate yourself to the new angle, etc., could the frame or two following a scene change also be candidates for more aggressive smoothing?

5) Dissolves. Somewhat related to the previous idea, during the course of a dissolve stronger blurring might be less noticable to the viewer.

6) Intrinsic detail levels. Possibly very difficult to measure, it might make sense that areas containing relatively little detail compared to the rest of the frame (e.g. an out-of-focus background to a headshot) could receive more attention from a smoothing filter.

6) Edge detection. While on its own edge detection can lead to over-smoothing of important details (and, I would submit, is more suited to anime), when combined with other "psychovisual" factors it may be more useful for live-action footage. For example, an out-of-focus background could be further blurred between edges, as could areas already identified as slow-moving.

While each of these individually might have only a little effect on compressibility, perhaps in combination they could provide a fairly significany improvement. Whether the processing time involved would make it unrealistic is another matter.

I apologise for the length of this post (I had no idea it was going to turn out this long), but I've yet to find any other outlet for my ideas - certainly none with such knowledgable participants. Waiting five days to post after registering was very frustrating!

Please feel free to do with these ideas what you will. For all I know I may just be rehashing old and rejected concepts, but a fairly exhaustive reading of the Doom9 forums turned up very few posts regarding psychovisual processing (that is, unrelated to DivX 5), except for the smooth-edges-of-frame idea which has been briefly discussed before, but with no real conclusion.

Also feel free to roll your eyes and point me in the direction of reading material that will set me straight. Image processing is a topic somewhat neglected by my local bookstores, and math class for me was mainly a time to fantasize about the teacher

Thanks for your attention.

Regards,
SansGrip