Adaptive Lens Blur Repair

[fvisagie]

Hi All,

This function is for adaptively repairing video damaged by lens blur, using a frame-adaptive repair mask, a selection of sharpeners and multi-stage motion-compensated artifact removal. It also provides blanket sharpening and denoising-only modes, the latter mainly for applying an equivalent amount of denoising to footage that did not need repairing. A sample of the type of footage this was created for: http://www.mediafire.com/file/0lym5r...ing_sample.wmv

Edit: updated for version 1.0.2

ALBR-1.0.2.zip

#############################################
#                                           #
#      Adaptive Lens Blur Repair 1.0.2      #
#            by Francois Visagie            #
#                                           #
#  Selective motion-compensated deblurring  #
#   using frame-adaptive repair mask and    #
#       multi-stage artifact removal        #
#                                           #
#############################################


#-------------------------------------------
# Rationale
#
#   With heavily and/or variably blurred (e.g. progressive centre-to-edge) video, Unsharp Mask sharpening usually has the following limitations:
#       *   The very first thing it does is further blurring, destroying already fragile motion vectors and creating distorted motion
#       *   It sharpens the whole frame equally much with unattractive results for variably blurred video
#   Variable blur is an especially big problem when it also moves around temporally, e.g. luma flares caused by bright objects in the frame.
#   In such cases even a selective sharpening mask is inadequate because it remains static between frames.
#
#   In an attempt to address these issues, this script
#       *   Works with motion vectors carefully, saving them before anything else
#       *   Uses edge detection to create a frame-adaptive repair mask for identifying flat vs. detailed areas
#       *   Provides a choice of sharpeners
#       *   Inserts motion compensation/denoising between repair stages using motion vectors refined with MRecalculate()
#
#   Various operating modes are available
#       *   Adaptive MC sharpening as described
#       *   Blanket (repair mask inactive) MC sharpening with denoising
#       *   MC temporal denoising only
#
#   To assist with tuning motion vector detection and repair mask parameters, various output modes are available.
#   Settings can be displayed in each output mode.
#
#
# Prototype, parameters and defaults
#
#   ALBR(clip a, string "mode", \
#        int "rc", int "thshlow", int "thshhi", float "vmdb", \
#        string "sharpnr", bool "shrpchr", int "rshy", int "rshc", float "strsh", int "thL", int "thU", int "type", int "map", int "lim", \
#        float "hue", float "sat", float "bright", float "cont", \
#        int "degrain", int "hpad", int "vpad", int "pel", int "rfilter", \
#        int "blksizA", int "ovrlapA", bool "temporl", int "lambda", int "pnew", \
#        int "blksizR", int "ovrlapR", int "thSADR", \
#        string "output", int "mvscale", bool "shosets")
#
#   Mode Setting
#   ------------
#
#   Available modes:
#       "AdSharp"   :   adaptive repair mask sharpening with denoising, all functionality and settings active
#       "Sharpen"   :   blanket sharpening (no repair mask) and denoise, Repair Mask Settings inactive
#       "Denoise"   :   temporally denoise only, Repair Mask and Sharpening Settings inactive
#
# mode    = Default(mode,   "AdSharp")
#
#   Repair Mask Settings
#   --------------------
#
# rc      = Default(rc,     50        )   # Radius for consolidating repair mask detailed and flat areas
# thshlow = Default(thshlow,22        )   # Repair mask low threshold - sharpen all detail BELOW this level
# thshhi  = Default(thshhi, 34        )   # Repair mask high threshold - sharpen no detail ABOVE this level
#                                         # - intermediate levels are sharpened proportionately
# vmdb    = Default(vmdb,   16.0      )   # Blur variance for debanding repair mask
#
#   Sharpening Settings
#   -------------------
#
#   Available sharpeners:
#       "Unsharpen" :   unsharpen() from VariableBlur.txt, almost identical to next, 3.1 fps
#       "Unsharp"   :   unsharp() from VariableBlur.txt, almost identical to previous, 3.1 fps
#       "SMSSQME"   :   Based on simplified multi-scale sampled quasi-median enhancer at http://forum.doom9.org/showthread.php?p=1595531#post1595531,
#                       less good than unsharp/en() on badly blurred source, 2.8 fps
#       "QMENoTopo" :   Above without topological reconstruction, slightly sharper on badly blurred source, 2.8 fps
#       "TUnSharp"  :   Does not work well for badly blurred source, 2.8 fps
#
# sharpnr = Default(sharpnr,"Unsharp" )   # Sharpener to use
# shrpchr = Default(shrpchr, true)        # Sharpen also chroma (true) or luma only (false), ignored by TUnSharp
#     Sharpener presets:                     Unsharpen   Unsharp     SMSSQME     QMENoTopo   TUnSharp
#         (NOTE: radius settings are internally converted to variance values for functions using those)
# rshy    = Default(rshy,   Select(shrpnum,  31,         31,         12,         12,         3        ))  # Unsharp mask Y radius/TUnSharp radius
# rshc    = Default(rshc,   Select(shrpnum,  31,         31,         8,          8,          0        ))  # Unsharp mask C radius, unused by TUnSharp
# strsh   = Default(strsh,  Select(shrpnum,  1.13,       1.13,       1.62,       1.62,       100      ))  # Sharpening strength
# thL     = Default(thL,    Select(shrpnum,  0,          0,          0,          0,          1        ))  # TUnSharp lower edge magnitude threshold,
#                                                                                                         # unused by others
# thU     = Default(thU,    Select(shrpnum,  0,          0,          0,          0,          5        ))  # TUnSharp upper edge magnitude threshold,
#                                                                                                         # unused by others
# type    = Default(type,   Select(shrpnum,  0,          0,          0,          0,          1        ))  # TUnSharp sharpening/unsharpmasking operator,
#                                                                                                         # unused by others
# map     = Default(map,    Select(shrpnum,  0,          0,          0,          0,          0        ))  # TUnSharp output map type, unused by others
# lim     = Default(lim,    Select(shrpnum,  0,          0,          0,          0,          2        ))  # TUnSharp sharpening limit, unused by others
# hue     = Default(hue,    Select(shrpnum,  0.0,        0.0,        0.0,        0.0,        0.0      ))  # Post-sharpening hue normalisation
# sat     = Default(sat,    Select(shrpnum,  0.65,       0.65,       1.0,        1.0,        1.16     ))  # Post-sharpening saturation normalisation
# bright  = Default(bright, Select(shrpnum,  35.3,       35.3,       23.5,       23.5,       0.0      ))  # Post-sharpening brightness normalisation
# cont    = Default(cont,   Select(shrpnum,  0.7,        0.7,        0.8,        0.8,        1.0      ))  # Post-sharpening contrast normalisation
#
#   Motion Vector Settings
#   ----------------------
#
# degrain = Default(degrain,3         )   # temporal radius for denoising/structural consolidation
# hpad    = Default(hpad,   16        )   # Need hpad and vpad this size for blksizA below
# vpad    = Default(vpad,   16        )
# pel     = Default(pel,    2         )   # MSuper() motion estimation accuracy
# rfilter = Default(rfilter,4         )   # Gives better results than default of 2
# blksizA = Default(blksizA,16        )   # Block size for analysing MVs
# ovrlapA = Default(ovrlapA,blksizA/2 )   # Overlap for analysing MVs
# temporl = Default(temporl,false     )   # Seems not to affect output, incompatible with SetMTMode when true
# lambda  = Default(lambda, 500       )   # For blurry source choose lambda and pnew values that slightly relax coherence,
# pnew    = Default(pnew,   25        )   # to prevent too many MVs being discarded and causing motion distortion
# blksizR = Default(blksizR,8         )   # Block size for refining MVs
# ovrlapR = Default(ovrlapR,blksizR/2 )   # Overlap for refining MVs
# thSADR  = Default(thSADR, 400       )   # Ditto relax re-estimation threshold
#
#   Tuning Aids
#   -----------
#
#   Available outputs:
#       "Result"    :   script result
#       "MVs"       :   motion vector overlay
#       "Mask"      :   repair mask
#       
# output  = Default(output, "Result"  )   # What to output
# mvscale = Default(mvscale,2         )   # Scale at which to display motion vectors
# shosets = Default(shosets,false     )   # Shows settings when true, also shows pixel values for output == "Result"
#
#
# Usage
#
#   ALBR(output="MVs", shosets=true)                      # Display motion vectors and settings for tuning purposes
#   ALBR(output="Mask", shosets=true)                     # Display repair mask and settings for tuning purposes
#   ALBR(sharpnr="TUnSharp", map=2, shosets=true)         # Display TUnSharp sharpening map (1/2) for tuning purposes
#   ALBR(shosets=true)                                    # Display result and settings for sharpener tuning
#   a=ALBR()                                              # Adaptively MC-sharpen with default sharpener and settings,
#   b=ALBR(mode="Denoise")                                #   denoise original by same amount,
#   splicelist="75:25, 150:25, 200:25, 400:25, 426:25"    #   specify splices (blends in this case) as needed,
#   SpliceFiltered(b, a, splicelist)                      #   selectively blend in sharpened material
#   ALBR(mode="Sharpen", sharpnr="SMSSQME", strsh=0.5)    # Blanket MC-sharpen cleaner source with high-quality enhancer
#
#
# Limitations
#
#   Repair mask -   although edge detection is much better than nothing, it also includes actual detail-less areas for sharpening
#   Not intended for sharpening motion blur
#
#
# Requirements
#
#   YV12 colourspace
#   From http://avisynth.org/mediawiki/External_filters:
#       MaskTools
#       MedianBlur
#       MVTools v2
#       RemoveGrain
#       TUnSharp
#       VariableBlur
#   Optional
#       SpliceFiltered from http://forum.doom9.org/showthread.php?t=166627
#
#-------------------------------------------

# Version history
#
#   1.0.2
#       Sharpening artifact removal now skipped when only denoising
#       MV recalculation between stages now disabled when only denoising
#
#   1.0.1
#       Expanded usage scenarios
#
#   1.0.0
#       Wrapped in function
#       Documented parameters and use
#       Added additional operating modes for blanket MC sharpening as well as MC temporal denoising
#
#   0.2.3
#       Added QMENoTopo sharpener
#       Added soure frame plus frame pixel values to Result settings display
#       Completed evaluation & characterisation of provided sharpeners & updated sharpener defaults accordingly
#       Various small improvements
#
#   0.1.1
#       Initial version
#

Toggling between the repair mask tuning and source displays:



Sample before-and-after screenshots:



Hopefully this is enough to get going anyone who wants to play with it, but shout if you have any questions.

All input, comments and suggestions gratefully welcomed. Many thanks to the collective assistance from this forum so far, in particular to poisondeathray who gave me direction initially.

Thanks and kind regards,
Francois

[TheProfileth]

Very interesting will take a look when I get time and give feedback.

[fvisagie]

First post updated for version 0.2.3.

[fvisagie]

First post updated for version 1.0.0:

# Version history
#
# 1.0.0
# Wrapped in function
# Documented parameters and use
# Added additional operating modes for blanket MC sharpening as well as MC temporal denoising

[fvisagie]

First post updated for version 1.0.2:

# 1.0.2
# Sharpening artifact removal now skipped when only denoising
# MV recalculation between stages now disabled when only denoising

[TheProfileth]

Oooh now that it is a function and with your new updates I will definitely try this out and possibly see if I can make any helpful additions.
BTW I suggest using something like mediafire so that you don't have to wait for approval.

[Forensic]

@fvisagie

Your script supports five sharpeners (Unsharpen, Unsharp, SMSSQME, QMENoTopo, and TUnSharp). What is your opinion of utilizing unsharpHQ (http://forum.doom9.org/showthread.php?t=159637) within your script? I tried to include it, but only ended up hopelessly breaking your script.

[fvisagie]

Originally Posted by Forensic

What is your opinion of utilizing unsharpHQ (http://forum.doom9.org/showthread.php?t=159637) within your script?

I did evaluate it, but comparatively speaking its results on the heavily lens-blurred footage I was testing against didn't really warrant including it. Why do you ask?

[Forensic]

It has been my personal favorite, but now you have me wondering if I should revisit the Sharpeners that you use.

[fvisagie]

You mileage may vary . The problem I faced with this project was that I couldn't find a dedicated lens deblurrer that really works and sharpeners were the next closest thing. However, it seems that a good sharpener does not necessarily make a good deblurrer and vice versa. To give you an idea, people set unsharp mask radius to say <= 4 when sharpening (some sharpeners even allow sub-pixel radius), but for deblurring where you're trying to reconstitute structure radius is often in the range 16 - 64.

PS. Depending on what you want to do, the approach used by this script also may or may not be the best idea. It does selective sharpening using a repair mask (which admittedly you can turn off with mode="Sharpen"), and it also does built-in temporal denoising to prevent temporal noise from interfering with detail detection and reconstruction. In other words, if all you want to do is sharpening you may get better results using a discrete sharpener as opposed to using this script.

On the other hand, if you do need the approach used by this script but it doesn't support a particular tool that gives you better deblurring, let me know and I'll see if I can add it.

[Forensic]

The best answer might be to implement a Laplacian tool, but I cannot find a Virtualdub/Avisynth version. Thanks for your ALBR tool. I will be watching for possible future releases.

[fvisagie]

Originally Posted by Forensic

The best answer might be to implement a Laplacian tool

Thanks, it's on the to-do list already .

[GMJCZP]

I'm no expert on this matter, but to improve the function might be helpful to look at this:

Here

vcmohan have other filters on their website that focus the image. Good luck.