Bye bye blurred frames

[martin53]

I tried to automate the task of replacing bad frames in a clip by other frames. The result is the attached filter function FSubstitute().

Edit: you can find an abstract some posts below.

The filter uses MVTools2's MCompensate and MFlowInter functions to reconstruct frames, as well as Freezeframe.

Target are
- blurred frames because of rapid camera movement
- super8 digitized clips with some bad frames from broken machinery or cutting
- frames with compression artefacts or blur from compression bitrate shortage

Usage is documented in the script header.

For the list of plugins and their versions see bottom of script.

The filter is slow when effective. It is possible to bypass good frames with appropriate parameter settings and thus get acceptable overall speed. I get about 3FPS with thoroughly processed 640x480 footage on a Core i5/750.

When you give it a try, watch the memory usage. I process clips in slices of max. 10000 frames.
Sorry I failed writing in a tidy style. It is commented extensively.
It it is surely not bug free. Comments are appreciated! But please have patience.
Change log see end of script.

Edit 13/31/2013: see change log at script bottom

[lisztfr9]

Let me just jump in to remember how the old Nikon 950 selected the best frame inside of 8, IIR the BBS setting : Probably by retaining the heaviest one which holds most of details. So the good images weight more and reversely the blurred images are simply lighter than average. Is there a way to retrieve the size of an image and compare it to, huh, the average of a window.. sequence, etc. ?

Thanks, L

[martin53]

The script does not use compressibility, if that's what you call weight?

The amount of detail is determined basically with a function that only honors edges, ignoring average luma. You find it under the name cHighpass() in the script. Of its output, the 'weight' of a frame is simply its AverageLuma.

Tailoring this function will guide the frame estimation/election process fundamentally. The actual implementation was a balance between quality and complexity.

[lisztfr9]

@martin53

With weight i was meaning the size in kb.

In your second § also i fear there is a contradiction with subject luma, which is ignored, but finally outputted... ?

[StainlessS]

@Moderator, can attachment be approved please. :thanks:

[martin53]

@martin53

With weight i was meaning the size in kb.

In your second § also i fear there is a contradiction with subject luma, which is ignored, but finally outputted... ?

Size means little compresssibility. That is not what I use.

The average luma of the original picture says nothing about the amount of detail in it. After applying a function that makes pixels at edges bright, and pixels in even areas dark (basically any edge detection), the picture will be brighter if there are more edges.
Now, the runtime function AverageLuma() will return a measure for the amount of edges in the original picture.
Got it?

[Didée]

Now, the runtime function AverageLuma() will return a measure for the amount of edges in the original picture.
Hehe, 've been there too, back in the dark ages when Restore24() was made. It kinda works, but you cannot reliably distinguish between the cases

a) more edges with less local contrast,
and
b) less edges with more local contrast.

Or, put as simple as possible, "20 edges with value 100" is the same as "200 edges with value 10", when just using AverageLuma.

For that distinction, you need to also take the "area %age covered by edges" into account. :)

[martin53]

If a task needs the distinction between few strong and many weak edges, RT_YInRange() can maybe help by now.

For the script I made, i felt no need to make that distinction. Where do you think that from two adjacent frames with no scene change in between, one could have few concise edges, while being blurred with subtle edges, and the other vice versa?

[Didée]

Where do you think that ...
I've no idea. Frankly, I have no clue what at all you are doing there in that script. :)

I was just going by your comment about "measuring the amount of edges" via AverageLuma. That reminded me of my similar problem in the past, and I thought it wouldn't hurt to mention that specific complicacy.

[lisztfr9]

Where do you think that from two adjacent frames with no scene change in between...

I got your point, but mine is valid too :

1) Nikon Corp implemented it probably in it's 950 Coolpix, because this early camera didn't feature edge detection algorithm.

2) Exactly as you say, from 2 adjacent frames ... - if one makes a jump in size (kb) then there is an image quality change, - toward blurriness if the image is lighter.

[kurish]

Intriguing function but I'm having difficulties with it. You posted some impressive results in another thread (http://forum.doom9.org/showthread.php?p=1593626#post1593626):

http://i.imgur.com/ISBhU.jpg

I'm no expert so bear with me. I gathered all the plugins you mentioned and tried a simple call:

avisource("test.avi")
FSubstitute()

Avisynth then threw the following error:

Avisynth open failure:
rangeF must be 0...5, is -1
(_FSubstitute.avsi, line 964)
(_FSubstitute.avsi, line 1407)

OK, that's strange since line 20 of the _FSubstitute.avsi seems to indicate a default value of zero for rangeF, but let's try again:

avisource("test.avi")
FSubstitute(rangeF=5)

This time I get the following:

Avisynth open failure:
Script error: MAnalyse does not have a named argument "level"
([GScript], line 4)
([GScript], line 15)
(_FSubstitute.avsi, line 203)
(_FSubstitute.avsi, line 978)
(_FSubstitute.avsi, line 1407)

Hmm. Well MAnalyse (http://avisynth.org.ru/mvtools/mvtools2.html) does have levels with an s on the end. Not sure what else to do and probably making a mistake, I added the s to the MAnalyse calls in _FSubstitute.avsi:

Line 191
cVectors = cSuper.MAnalyse(isb=(iDelta>0?true:false), delta=abs(iDelta), dct=dct, levels=2, [etc.])

Line 197
cVectors = cSuper.MAnalyse(isb=(iDelta>0?true:false), delta=abs(iDelta), dct=dct, levels=2, [etc.])

And now my script opens, but text is overlayed intermittently:

http://i.imgur.com/DCDHE.jpg

Flipping that image vertically, the text reads:

MVTools: vector clip is too small (corrupted?)
([GScript], line 208)
([GScript], line 211)
([GScript], line 330)
([ScriptClip], line 332)

And that's as far as I took it. VirtualDub does seem to run somewhat slowly as I step through the frames, so something seems to be happening although I'm not seeing any difference from my source. Admittedly, the only source I have available right now has already been put through QTGMC and is thus highly sharpened, so maybe FSubstitute is analyzing and deciding against making changes? I'll happily dig up some other sources once I'm confident that FSubstitute is running correctly...

Some questions for you:

Your thoughts on the above?
Can you provide more details (and perhaps examples) of the ideal target material? I suppose that "bad frames from broken machinery or cutting" is somewhat clear, but greater specificity would be helpful. "Compression artefacts from excessive compression" interests me the most, however, but exactly what you have in mind is quite unclear. Before/after examples would be greatly appreciated, if possible, and I imagine would do a lot to generate interest in your function.
Can you provide some examples of specific FSubstitute calls/scripts that you've find useful? There's good parameter documentation in the avsi already, but it's difficult (for me at least) to understand the relative importance of each component. Some examples to play with would be very helpful.

Thanks for reading and for the hard work putting this function together. I'm really looking forward to working out the kinks and seeing what it can do.

[martin53]

Intriguing function but I'm having difficulties with it.

First of all, thanks that you gave it a try.
I really appreciate your feedback, since I regret any inconvenience with the script, and the feedback helps me to reduce these inconveniences!

Since I packed so many features into the script, I am unable now to manage to test them all after I did improvements to one part, so I experienced that other parts suffered from bugs afterwards, and that's what you experienced.

It seems that with your version, I introduced 'forced freeze', and I controlled that with a negative rangeF setting. It is a separate forceF now.
The docomentation is at the start of the script. The use of some parameters evolved unfortunately in a non backward compatible manner, but I hope the recent version is in good state!

My MAnalyse does have the level parameter and always worked. I checked it is V2.3.1 - it's outdated. I'll update to 2.5.11.3, but then I'll need to publish another updated version of my script (haven't got the time at this moment).

The flipped display was a neccessary consequence of a tweak I used to enhance motion estimation. I found a better way in the meantime, so errors will display better now (for more details see 'orientation' parameter).

And the last problem you mention - the wrong vector clip size - should no more be present. I used a reduced dimensions clip for some preprocessing for speedup, but that too is done in a different manner now, only little MVTools usage left and no more need for a small helper clip.

I'll give some hints about what the script is supposed to do and how to start with it later today or tomorrow. The image you found basically shows it - it's about clips with short blurred sequences because of rapid camera motion etc.

You could start with setting info=4 to see thumbs of the three substitution candidates and some helpful(?) figures.

Please give me the dimensions of your clip, I worked mostly with 640x480 clips so far, and it's quite probable that some bugs only show with certain dimensions.

[martin53]

Can you provide more details (and perhaps examples) of the ideal target material?

I'll try my best to summarize :)
FSubstitute is an environment for the MVTools2 functions 'MCompensate' and 'MFlowInter'. It is tedious to use these functions on themselves because a very specialized script is needed with individual commands for each frame to process, after manual decision on each frame, without knowing in advance how to best configure the functions.

How It Works
So the first key element of FSubstitute is a filter function I first called 'Highpass', but just renamed to 'Quality' for the next version. The luminosity of the output is used to distinguish good frames from frames to process. The MVTools2 functions are automatically applied on bad frames, you just enter some parameters to guide the script.

You might call the automated search for best replacement source frames the next key element. This one has to deal with the trade-off between proximity and sharpness of the possible sources.

The third key elements are the mentioned functions, accompanied by FreezeFrame. Because I learned that the functions have poor performance when bad detail and much global motion come together, FSubstitute does something unique that really boosted the quality of MCompensate and MFlowInter's output in my experiments: First, the chosen sources are globally motion compensated to best match the frame under processing. This is done completely inside FSubstitute, again and again for each frame under processing. Only this allows to use FreezeFrame as a mature third method besides the other two. Brightness is also adjusted by default.

Finally a filter function fGetError tries with a sophisticated approach, to distinguish between the desired effect and bad motion compensation / missing objects in the difference between the function's outputs and the original frame under processing.


What It Can Do, And What Not
FSubstitute is good for clips that have short quality fall-offs (up to 10 frames in a sequence by now). It is not useful for enhancing footage with a constant quality over time. It is not 'yet another sharpener'. It does no sharpening at all, just frame cloning.
One example of such a fall-off is the motion blur from moving a camera because of shutter time. This is usually not obvious from the start, but annoying after stabilizing. Also, I have some WMV clips here (copyrighted, so I cannot publish examples), which develop sharpness only slowly, with visible delay after each pan.

Howto And Examples
For a start, you might set info=4 to see the figures that your footage creates, especially dThres to avoid processing of good frames. Find a frame range where you identify a quality fall-off and compare the outputs of Compensate, Freeze and Interpolate. Set rangeC, rangeF and rangeI accordingly to the length of bad sequences. Use info=3 or info=2 together with dbgView to find out more about the things that happen. Together with the explanation in the script header and throughout the script, I hope that can clarify things.

I'll publish the example sequence shortly.

[Bernardd]

@martin53

I have tried your script on one old Super 8 film. The capture is 720 by 576 pixels.

With 11-12-2012 script version i and info = 4, the result for two frames, first good and second blurred is shown with picture frame 002685.jpg and frame 002686.jpg (pictures below).

With 11-18-2012 script version i and info = 4, the result for the same two frames, first good and second blurred is shown with picture frame 002685.jpg and frame 002686.jpg (pictures below).

With the last version, a green mask appears, why ?

[martin53]

@Bernardd,
on top left, you see the sharpness figures for the frame group.

The figure for the blurred frame is below its neighbours - good.
on bottom left, you see the 'MCompensate' candidate. The 11-12-2012 reading (center image) says that the script tries to compensate from frame with delta=-1 and achieves an 'error' value of 0.617 (the white areas), which is below the default limit of maxErrC (1.0). Because also the sharpness of 10.58 is above the actual 8.70, this frame will be the substitution source and is marked with a '*'.

Edit: Another compatibility issue, now between my MaskTools v1.4.16 and actual v1.5.8.
I checked all versions of my plugins with the available ones now and wrote a version list in the change history at the end of the script.

[martin53]

So, here is a real - extreme - example. :)
Original clip (http://www.mediafire.com/download.php?dk61uy6kww52tp6)
Stabilized clip (http://www.mediafire.com/download.php?909niw28wxwn8wt) (with these (http://forum.doom9.org/showthread.php?p=1593555#post1593555) deshaker scripts, enlarged by 1.05)
Processed clip (http://www.mediafire.com/download.php?bx37ik2xchoaz24) (rangeC=0,rangeF=5,rangeI=0,maxErrF=2)

It is all done only with FreezeFrame, since I disabled MCompensate and MFlowInter. Because only still life was filmed, I notice the small 'movements' that MCompensate and MFlowInter leave after compensation. The clip looks a bit stuttering, but I feel that it is harder to concentrate on single objects or details in the original because they become blurred.

While the flat objects just look better, the extreme 3-dimensional sequences do not move smoothly.

[Bernardd]

@martin53

With November 18 version of your script, the green mask appears on the corrected frames. I have made your corrections without visual picture change. Only debbug text has a little rewritted.

I use MaskTools v1.5.8, but i try the MaskTool v1.4.16 and it is the same with green mask.

With November 12 version of your script, no problem with green mask.

I have processed my 3 mn long clip with the November 12 version. After one hour, my clip have no more blurred frames. But some frames was changed, without neccessary for a human read.

It is not easy to control the clip frame by frame. A list of changed frames should be welcome.

In November 18 version script, i see new params "framelist" and "framefile". Thus i wait and hope.

This script is a impressive job. With the November 12 version, i see smart substitutions where i cannot make better with manual work.

[lisztfr9]

I'm a bit frighten by a 1400 lines script, because i wrote a 2684 lines program, and i know what it means to debug code.

[martin53]

@martin53

With November 18 version of your script, the green mask appears on the corrected frames. I have made your corrections without visual picture change. Only debbug text has a little rewritted.

I use MaskTools v1.5.8, but i try the MaskTool v1.4.16 and it is the same with green mask.

With November 12 version of your script, no problem with green mask.

In function cTransformFrame(), there is a block if (bright) {...}

- Please try the script with parameter bright=false. I had the same symptom immediately after I updated to MaskTools v1.5.8 (but not with v1.4.16 all the while). Bright=false cured the green symptom at my place. My suspicion is, that something in the YV12LUT function changed between v1.4.16 and v1.5.8. The other places in the script where YV12LUT is used did not make a difference when I changed to the alternative processing without YV12LUT (usually, these commands are in the script, but commented out for quick comparisons). So, in the 11/19/2012 version, I commented out the line #crgb = crgb.YV12LUT... and re-activated the line before, crgb = crgb.Tweak(coring=false, bright=Averageluma(c)-Averageluma(crgb)).
Then the green was gone even with bright=true. Please comment that.

What is 'bright' for: The script tries to adjust brightness and contrast of the substitute to those of the original. But because just RT_Averageluma and RT_YPlaneStdev are used to derive brightness and contrast, the algorithm might fail for some footage. I think is is useful for correcting fades. With bright=false, simply this feature remains unused.


I have processed my 3 mn long clip with the November 12 version. After one hour, my clip have no more blurred frames. But some frames was changed, without neccessary for a human read.

It is not easy to control the clip frame by frame. A list of changed frames should be welcome.

In dbgView, with info=1 and info=2, the action on each frame is written, among other data. Please try filtering in dbgView.


In November 18 version script, i see new params "framelist" and "framefile". Thus i wait and hope.

These parameters are already working. You may either enter a space separated list of frames to be processed directly in 'framelist', or a name of a text file with this information in 'framefile'. If you enter both, 'framefile' wins. Line breaks are internally converted to spaces.
You can only enter additional frames to be processed. But because you can easily set dThres or sThres to high values, you can practically disable automatic frame evaluation.

It is better to use the 11/19/2012 version, see change history at script end.
:)

[martin53]

I have processed my 3 mn long clip with the November 12 version. After one hour, my clip have no more blurred frames. But some frames was changed, without neccessary for a human read.


The default for sThres, 1.05, is quite low. Usually, substitutions achieved a sharpness result 1.05 below the sharpness of the substitution source in my experiments, so i set the default to that value.
With higher sThres, the script tries only substitutions if frames with an even higher sharpness factor than the current frame are in the substitution range.
With lower rangeC (or rangeF, rangeI), a smaller neighbourhood is evaluated. This is useful if the blurred sequences are short, e.g. only single frames.
With lower qThres, more frames are over the 'substitution need' limit and left unprocessed. This is the main speed control parameter, too. There should be no default for qThres. But then, you would have encountered one more barrier to try the script. Look at the figures in the original with info=4 and set qThres to the value shown in brackets of an appropriate frame.

If your footage contains many small objects with individual motion - like the skiers, in contrast to a close-up situation in my example clip, then you should try much lower values for lsad and lambda, if you're not satisfied with the default output. High values help MVTools2 to avoid decomposition of distant local sharp parts of the same object. Low values give MVTools2 the chance to adapt to different independent local motions. More about that in the MVTools2 doc.

[Bernardd]

@martin53

You have right, with 18 nov version of your script, on my computer with param "bright=true", green mask appears and with this param at false no green mask with the two Masktools versions.

But well done, with 19 nov version of your script, none green mask, with bright default valor (true) for the two Masktools versions.

Thanks

I try now your last version. I exam your param mod proposals and report you later.

[Bernardd]

@martin53

Nice script.

I understand the symbol * for substitution picture, but can you explain why the text colors change on this pictures.
two with substitution
13101
13102
two without substitution
13103
13104

[Bernardd]

Second part and End

To forbidden Fsubstitute process on some frames not blurred for a human read, sThres and dThres param are very effiencies. Today, i prefers to use dThres, because :

sThres is not accurate, so for this picture you can see the result for sThres = 1.05 and 1.25. I must choose 1.45 to forbidden subsitution
13106
13107

the Info 4 display give good valors to choose a best level for dThres. As we can see on this three example pictures.
13108
13109
13110

I try filtering in dbgView and OK, no problem, i can list frames processed and write a bookmarks file for AvsPmod.

[martin53]

but can you explain why the text colors change on this pictures.

The subtitle of candidates is green if the candidate meets all requirements to replace the original, red if it's not. Only the winning candidate (always green of course) has an asterisk.

The original clip's top line subtitle is white if it does not need substitution by thresholds, pink if it needs substitution but no candidate meets all requirements, green if substitution will be made.

maxErrC etc are not exact values: the script loosens this limit when the current frame is very blurred. Maybe you wondered about the fQ figure.
fQ is the quality factor between the current frame and the best frame in the (-5...5) range. That means: if the current frame only has half the 'quality' than the best frame in this range, all maxErr limits are multiplied by 2. This helps to keep maxErr small, yet allow substitution of frames that drop significantly in their neighbourhood.

And then forceF is another tool for very bad frames. If a frame's quality is below the average of its two direct neighbours and if still no substitution candidate meets the limits, another try is made with FreezeFrame of the better of the two direct neighbours. This candidate profits from a much higher maxErr. The idea behind it is, that the eye will not so much notice that the frames is just copied, but it will notice single very bad frames.

[Wilbert]

So which version of which plugins do I need in order to use this plugin?

[martin53]

Wilbert,
welcome!

Please have a look at the end of the script in the change log. The plugins with their tested versions are documented there since the 11/19/2012 version of the script.

Today I updated to MaskTools v2, but did not release the new script yet.

[matfra]

Hi,
I download the script and all the plugins. I tried the command line with no parameter. There is no change in the video. Can you give me some exemple how to call the script. Do I need to specify some frame to work on?

[martin53]

matfra,
the script is guided by a value every frame produces - the sharpness, or quality how I'm going to call it more and more.

The actual value depends strongly on your given clip. If your clip produces higher numbers than the threshold default that comes with the script, there will be no processing at first.

I suggest that you call it with info=4. You will then see the quality values of some preceding, the actual and some following frames close to the top left.

Then call it with qThres=the_value_you_read_with_a_known_good_frame. That will activate processing for all frames with a lower value.

Info=4 will also show you then if some frames close to inferior frames could be used to enhance them.

[martin53]

Current version from 11/22/2012 onwards now allows the full restoration with interpolation or freeze of up to 5 (interpolation), or 10 (freeze, allows scene change in gap) consecutive frames that are completely unusable.
It is a quick hack compared to the rest of the script, so please don't be upset, but inform me about malfunctions.

Example see clip from this post (http://forum.doom9.org/showthread.php?p=1593680#post1593680)

>Your Source filter here to open the file<
crop(10,0,-326,0).addborders(10,0,326,0) #because the file already shows before and after, remove after version
DuplicateFrame(10,10,10,10) # just to make it a little worse around frame 10
FSubstitute(mode=2,info=3) # mode=2 settings were prepared for this example

[martin53]

Frame replacing is also thoroughly dealt with in this thread (http://forum.doom9.org/showthread.php?t=152758).

However, one question is how to *detect* a frame that needs replacing. ... The problem here is that there is no generalised meaning of the phrase "bad frame"

So far, I am aware of the follwing types of frame damage

blur from e.g. camera motion
artefacts from compression (e.g. content change with constant bit rate) - Typical case: motion start leads to camera blur, which in turn makes the codec remove sharp small structures. Structures recover only after some time with free bandwith.
VHS tape or head noise, black lines
Super8 duct-tape, scratch
Super8 film transport, big dust
duplicate frames from unreliable capture system

While problems #1-3 generally leave the global frame position in a usable state, problems #5-6 contain frame content that is not wanted. These latter defects would need FSubstitute(...create=true...).

The function cQuality() as published with the script takes the amount of edges in a frame as the quality measure, and the parameter qThres designates all frames below that threshold as 'bad' frames.
That should work from the start with problems #1, #2, maybe #3 and maybe #4, with the example of my previous post also with problem #5.

Problem #5 might also be detectable with RT_AverageLuma(c).

Problem #6 can probably be detectable with function 2-RT_LumaCorrelation(c,c,delta=-1) inside fQuality(). Bad frames should have a quality of exactly one.

Yet I feel that with problems #3-4 there might be no general answer.

Your contributions are welcome.

[Wilbert]

How do i disable the interpolating and compensating? These are way too slow for the clips i want to try. I want know which frames are detected as being too blurry and they should be replaced by freezing (at least if there is a good replacement).

[martin53]

Please have patience one or two more days.
With the published version, set rangeC=0 and rangeI=0.
But because a lot of the time is consumed in the global motion compensation preprocessing, that will not speed up the script much.
In a newer version, I added a parameter PreMoCo that can be set to false, if you just want 1:1 freezeframe copies. This does indeed boost speed.
I just do not want to publish that today, because something else is wrong in it and I don't want to spread a buggy version.

[martin53]

Wilbert,
you may now try the 11/30/12 version with preMoCo=false.
I also suffer from the ultra low speed (down to 0.4 FPS when most frames need processing).
But I noticed that MFLowInter and MCompensate make significantly improved frames when they get them with as little global offset as possible. So it really is a quality vs. speed issue.
This might be a situation where a new plugin could boost speed. I have no idea how I could speed up the script inside the AviSynth limitations. A lot of tweaks are already built in to avoid redundant processing.

I also think of preparing a motion log file with MDepan and preconvert that with an external tool, because the script needs the relative motion between the current frame and all frames in the +-5 ranges. The MDepan log writes the motion between adjacent frames, so FSubstitute would need to read 5 lines and add them - not effective. I do not like multi pass approaches, too.

But that would only save time consumed for motion estimation (since it would have to be done once, not again and again with every frame). Much time is also used for global frame transformation with quad().

EDIT: version 12/15/2012 is again optimized and faster