Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 11th November 2012, 17:29   #1  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
Bye bye blurred frames

I tried to automate the task of replacing bad frames in a clip by other frames. The result is the attached filter function FSubstitute().

Edit: you can find an abstract some posts below.

The filter uses MVTools2's MCompensate and MFlowInter functions to reconstruct frames, as well as Freezeframe.

Target are
- blurred frames because of rapid camera movement
- super8 digitized clips with some bad frames from broken machinery or cutting
- frames with compression artefacts or blur from compression bitrate shortage

Usage is documented in the script header.

For the list of plugins and their versions see bottom of script.

The filter is slow when effective. It is possible to bypass good frames with appropriate parameter settings and thus get acceptable overall speed. I get about 3FPS with thoroughly processed 640x480 footage on a Core i5/750.

When you give it a try, watch the memory usage. I process clips in slices of max. 10000 frames.
Sorry I failed writing in a tidy style. It is commented extensively.
It it is surely not bug free. Comments are appreciated! But please have patience.
Change log see end of script.

Edit 13/31/2013: see change log at script bottom
Attached Files
File Type: zip _FSubstitute_121223.zip (28.1 KB, 128 views)
File Type: zip _FSubstitute_130131.zip (29.3 KB, 191 views)

Last edited by martin53; 31st January 2013 at 21:43. Reason: new version
martin53 is offline   Reply With Quote
Old 11th November 2012, 17:47   #2  |  Link
lisztfr9
Registered User
 
Join Date: Apr 2010
Posts: 175
Let me just jump in to remember how the old Nikon 950 selected the best frame inside of 8, IIR the BBS setting : Probably by retaining the heaviest one which holds most of details. So the good images weight more and reversely the blurred images are simply lighter than average. Is there a way to retrieve the size of an image and compare it to, huh, the average of a window.. sequence, etc. ?

Thanks, L
lisztfr9 is offline   Reply With Quote
Old 11th November 2012, 21:32   #3  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
The script does not use compressibility, if that's what you call weight?

The amount of detail is determined basically with a function that only honors edges, ignoring average luma. You find it under the name cHighpass() in the script. Of its output, the 'weight' of a frame is simply its AverageLuma.

Tailoring this function will guide the frame estimation/election process fundamentally. The actual implementation was a balance between quality and complexity.
martin53 is offline   Reply With Quote
Old 11th November 2012, 21:51   #4  |  Link
lisztfr9
Registered User
 
Join Date: Apr 2010
Posts: 175
@martin53

With weight i was meaning the size in kb.

In your second § also i fear there is a contradiction with subject luma, which is ignored, but finally outputted... ?
lisztfr9 is offline   Reply With Quote
Old 12th November 2012, 05:31   #5  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
@Moderator, can attachment be approved please.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 12th November 2012, 18:55   #6  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
Quote:
Originally Posted by lisztfr9 View Post
@martin53

With weight i was meaning the size in kb.

In your second § also i fear there is a contradiction with subject luma, which is ignored, but finally outputted... ?
Size means little compresssibility. That is not what I use.

The average luma of the original picture says nothing about the amount of detail in it. After applying a function that makes pixels at edges bright, and pixels in even areas dark (basically any edge detection), the picture will be brighter if there are more edges.
Now, the runtime function AverageLuma() will return a measure for the amount of edges in the original picture.
Got it?
martin53 is offline   Reply With Quote
Old 12th November 2012, 19:44   #7  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,389
Quote:
Originally Posted by martin53 View Post
Now, the runtime function AverageLuma() will return a measure for the amount of edges in the original picture.
Hehe, 've been there too, back in the dark ages when Restore24() was made. It kinda works, but you cannot reliably distinguish between the cases

a) more edges with less local contrast,
and
b) less edges with more local contrast.

Or, put as simple as possible, "20 edges with value 100" is the same as "200 edges with value 10", when just using AverageLuma.

For that distinction, you need to also take the "area %age covered by edges" into account.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 12th November 2012, 22:04   #8  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
If a task needs the distinction between few strong and many weak edges, RT_YInRange() can maybe help by now.

For the script I made, i felt no need to make that distinction. Where do you think that from two adjacent frames with no scene change in between, one could have few concise edges, while being blurred with subtle edges, and the other vice versa?
martin53 is offline   Reply With Quote
Old 12th November 2012, 22:19   #9  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,389
Quote:
Originally Posted by martin53 View Post
Where do you think that ...
I've no idea. Frankly, I have no clue what at all you are doing there in that script.

I was just going by your comment about "measuring the amount of edges" via AverageLuma. That reminded me of my similar problem in the past, and I thought it wouldn't hurt to mention that specific complicacy.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 13th November 2012, 09:53   #10  |  Link
lisztfr9
Registered User
 
Join Date: Apr 2010
Posts: 175
Quote:
Originally Posted by martin53 View Post

Where do you think that from two adjacent frames with no scene change in between...
I got your point, but mine is valid too :

1) Nikon Corp implemented it probably in it's 950 Coolpix, because this early camera didn't feature edge detection algorithm.

2) Exactly as you say, from 2 adjacent frames ... - if one makes a jump in size (kb) then there is an image quality change, - toward blurriness if the image is lighter.

Last edited by lisztfr9; 13th November 2012 at 11:13.
lisztfr9 is offline   Reply With Quote
Old 17th November 2012, 23:34   #11  |  Link
kurish
Registered User
 
Join Date: Nov 2006
Posts: 19
Intriguing function but I'm having difficulties with it. You posted some impressive results in another thread:



I'm no expert so bear with me. I gathered all the plugins you mentioned and tried a simple call:

Code:
avisource("test.avi")
FSubstitute()
Avisynth then threw the following error:

Code:
Avisynth open failure:
rangeF must be 0...5, is -1
(_FSubstitute.avsi, line 964)
(_FSubstitute.avsi, line 1407)
OK, that's strange since line 20 of the _FSubstitute.avsi seems to indicate a default value of zero for rangeF, but let's try again:

Code:
avisource("test.avi")
FSubstitute(rangeF=5)
This time I get the following:

Code:
Avisynth open failure:
Script error: MAnalyse does not have a named argument "level"
([GScript], line 4)
([GScript], line 15)
(_FSubstitute.avsi, line 203)
(_FSubstitute.avsi, line 978)
(_FSubstitute.avsi, line 1407)
Hmm. Well MAnalyse does have levels with an s on the end. Not sure what else to do and probably making a mistake, I added the s to the MAnalyse calls in _FSubstitute.avsi:

Code:
Line 191
cVectors = cSuper.MAnalyse(isb=(iDelta>0?true:false), delta=abs(iDelta), dct=dct, levels=2, [etc.])

Line 197
cVectors = cSuper.MAnalyse(isb=(iDelta>0?true:false), delta=abs(iDelta), dct=dct, levels=2, [etc.])
And now my script opens, but text is overlayed intermittently:



Flipping that image vertically, the text reads:

Code:
MVTools: vector clip is too small (corrupted?)
([GScript], line 208)
([GScript], line 211)
([GScript], line 330)
([ScriptClip], line 332)
And that's as far as I took it. VirtualDub does seem to run somewhat slowly as I step through the frames, so something seems to be happening although I'm not seeing any difference from my source. Admittedly, the only source I have available right now has already been put through QTGMC and is thus highly sharpened, so maybe FSubstitute is analyzing and deciding against making changes? I'll happily dig up some other sources once I'm confident that FSubstitute is running correctly...

Some questions for you:
  1. Your thoughts on the above?
  2. Can you provide more details (and perhaps examples) of the ideal target material? I suppose that "bad frames from broken machinery or cutting" is somewhat clear, but greater specificity would be helpful. "Compression artefacts from excessive compression" interests me the most, however, but exactly what you have in mind is quite unclear. Before/after examples would be greatly appreciated, if possible, and I imagine would do a lot to generate interest in your function.
  3. Can you provide some examples of specific FSubstitute calls/scripts that you've find useful? There's good parameter documentation in the avsi already, but it's difficult (for me at least) to understand the relative importance of each component. Some examples to play with would be very helpful.
Thanks for reading and for the hard work putting this function together. I'm really looking forward to working out the kinks and seeing what it can do.

Last edited by kurish; 18th November 2012 at 00:02. Reason: typos
kurish is offline   Reply With Quote
Old 18th November 2012, 17:47   #12  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
Quote:
Originally Posted by kurish View Post
Intriguing function but I'm having difficulties with it.
First of all, thanks that you gave it a try.
I really appreciate your feedback, since I regret any inconvenience with the script, and the feedback helps me to reduce these inconveniences!

Since I packed so many features into the script, I am unable now to manage to test them all after I did improvements to one part, so I experienced that other parts suffered from bugs afterwards, and that's what you experienced.

It seems that with your version, I introduced 'forced freeze', and I controlled that with a negative rangeF setting. It is a separate forceF now.
The docomentation is at the start of the script. The use of some parameters evolved unfortunately in a non backward compatible manner, but I hope the recent version is in good state!

My MAnalyse does have the level parameter and always worked. I checked it is V2.3.1 - it's outdated. I'll update to 2.5.11.3, but then I'll need to publish another updated version of my script (haven't got the time at this moment).

The flipped display was a neccessary consequence of a tweak I used to enhance motion estimation. I found a better way in the meantime, so errors will display better now (for more details see 'orientation' parameter).

And the last problem you mention - the wrong vector clip size - should no more be present. I used a reduced dimensions clip for some preprocessing for speedup, but that too is done in a different manner now, only little MVTools usage left and no more need for a small helper clip.

I'll give some hints about what the script is supposed to do and how to start with it later today or tomorrow. The image you found basically shows it - it's about clips with short blurred sequences because of rapid camera motion etc.

You could start with setting info=4 to see thumbs of the three substitution candidates and some helpful(?) figures.

Please give me the dimensions of your clip, I worked mostly with 640x480 clips so far, and it's quite probable that some bugs only show with certain dimensions.
martin53 is offline   Reply With Quote
Old 18th November 2012, 22:50   #13  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
Quote:
Originally Posted by kurish View Post
Can you provide more details (and perhaps examples) of the ideal target material?
I'll try my best to summarize
FSubstitute is an environment for the MVTools2 functions 'MCompensate' and 'MFlowInter'. It is tedious to use these functions on themselves because a very specialized script is needed with individual commands for each frame to process, after manual decision on each frame, without knowing in advance how to best configure the functions.

How It Works
  • So the first key element of FSubstitute is a filter function I first called 'Highpass', but just renamed to 'Quality' for the next version. The luminosity of the output is used to distinguish good frames from frames to process. The MVTools2 functions are automatically applied on bad frames, you just enter some parameters to guide the script.
  • You might call the automated search for best replacement source frames the next key element. This one has to deal with the trade-off between proximity and sharpness of the possible sources.
  • The third key elements are the mentioned functions, accompanied by FreezeFrame. Because I learned that the functions have poor performance when bad detail and much global motion come together, FSubstitute does something unique that really boosted the quality of MCompensate and MFlowInter's output in my experiments: First, the chosen sources are globally motion compensated to best match the frame under processing. This is done completely inside FSubstitute, again and again for each frame under processing. Only this allows to use FreezeFrame as a mature third method besides the other two. Brightness is also adjusted by default.
  • Finally a filter function fGetError tries with a sophisticated approach, to distinguish between the desired effect and bad motion compensation / missing objects in the difference between the function's outputs and the original frame under processing.

What It Can Do, And What Not
FSubstitute is good for clips that have short quality fall-offs (up to 10 frames in a sequence by now). It is not useful for enhancing footage with a constant quality over time. It is not 'yet another sharpener'. It does no sharpening at all, just frame cloning.
One example of such a fall-off is the motion blur from moving a camera because of shutter time. This is usually not obvious from the start, but annoying after stabilizing. Also, I have some WMV clips here (copyrighted, so I cannot publish examples), which develop sharpness only slowly, with visible delay after each pan.

Howto And Examples
For a start, you might set info=4 to see the figures that your footage creates, especially dThres to avoid processing of good frames. Find a frame range where you identify a quality fall-off and compare the outputs of Compensate, Freeze and Interpolate. Set rangeC, rangeF and rangeI accordingly to the length of bad sequences. Use info=3 or info=2 together with dbgView to find out more about the things that happen. Together with the explanation in the script header and throughout the script, I hope that can clarify things.

I'll publish the example sequence shortly.
martin53 is offline   Reply With Quote
Old 19th November 2012, 13:47   #14  |  Link
Bernardd
Registered User
 
Join Date: Jan 2012
Location: Toulon France
Posts: 248
@martin53

I have tried your script on one old Super 8 film. The capture is 720 by 576 pixels.

With 11-12-2012 script version i and info = 4, the result for two frames, first good and second blurred is shown with picture frame 002685.jpg and frame 002686.jpg (pictures below).

With 11-18-2012 script version i and info = 4, the result for the same two frames, first good and second blurred is shown with picture frame 002685.jpg and frame 002686.jpg (pictures below).

With the last version, a green mask appears, why ?
Attached Images
    

Last edited by Bernardd; 19th November 2012 at 15:15. Reason: amend
Bernardd is offline   Reply With Quote
Old 19th November 2012, 18:50   #15  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
@Bernardd,
on top left, you see the sharpness figures for the frame group.
  • The figure for the blurred frame is below its neighbours - good.
  • on bottom left, you see the 'MCompensate' candidate. The 11-12-2012 reading (center image) says that the script tries to compensate from frame with delta=-1 and achieves an 'error' value of 0.617 (the white areas), which is below the default limit of maxErrC (1.0). Because also the sharpness of 10.58 is above the actual 8.70, this frame will be the substitution source and is marked with a '*'.
Edit: Another compatibility issue, now between my MaskTools v1.4.16 and actual v1.5.8.
I checked all versions of my plugins with the available ones now and wrote a version list in the change history at the end of the script.

Last edited by martin53; 19th November 2012 at 21:16.
martin53 is offline   Reply With Quote
Old 19th November 2012, 22:11   #16  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
So, here is a real - extreme - example.
Original clip
Stabilized clip (with these deshaker scripts, enlarged by 1.05)
Processed clip (rangeC=0,rangeF=5,rangeI=0,maxErrF=2)

It is all done only with FreezeFrame, since I disabled MCompensate and MFlowInter. Because only still life was filmed, I notice the small 'movements' that MCompensate and MFlowInter leave after compensation. The clip looks a bit stuttering, but I feel that it is harder to concentrate on single objects or details in the original because they become blurred.

While the flat objects just look better, the extreme 3-dimensional sequences do not move smoothly.
martin53 is offline   Reply With Quote
Old 19th November 2012, 22:12   #17  |  Link
Bernardd
Registered User
 
Join Date: Jan 2012
Location: Toulon France
Posts: 248
@martin53

With November 18 version of your script, the green mask appears on the corrected frames. I have made your corrections without visual picture change. Only debbug text has a little rewritted.

I use MaskTools v1.5.8, but i try the MaskTool v1.4.16 and it is the same with green mask.

With November 12 version of your script, no problem with green mask.

I have processed my 3 mn long clip with the November 12 version. After one hour, my clip have no more blurred frames. But some frames was changed, without neccessary for a human read.

It is not easy to control the clip frame by frame. A list of changed frames should be welcome.

In November 18 version script, i see new params "framelist" and "framefile". Thus i wait and hope.

This script is a impressive job. With the November 12 version, i see smart substitutions where i cannot make better with manual work.

Last edited by Bernardd; 19th November 2012 at 22:19.
Bernardd is offline   Reply With Quote
Old 19th November 2012, 23:16   #18  |  Link
lisztfr9
Registered User
 
Join Date: Apr 2010
Posts: 175
I'm a bit frighten by a 1400 lines script, because i wrote a 2684 lines program, and i know what it means to debug code.

Last edited by lisztfr9; 19th November 2012 at 23:21.
lisztfr9 is offline   Reply With Quote
Old 20th November 2012, 18:18   #19  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
Quote:
Originally Posted by Bernardd View Post
@martin53

With November 18 version of your script, the green mask appears on the corrected frames. I have made your corrections without visual picture change. Only debbug text has a little rewritted.

I use MaskTools v1.5.8, but i try the MaskTool v1.4.16 and it is the same with green mask.

With November 12 version of your script, no problem with green mask.
In function cTransformFrame(), there is a block if (bright) {...}

- Please try the script with parameter bright=false. I had the same symptom immediately after I updated to MaskTools v1.5.8 (but not with v1.4.16 all the while). Bright=false cured the green symptom at my place. My suspicion is, that something in the YV12LUT function changed between v1.4.16 and v1.5.8. The other places in the script where YV12LUT is used did not make a difference when I changed to the alternative processing without YV12LUT (usually, these commands are in the script, but commented out for quick comparisons). So, in the 11/19/2012 version, I commented out the line #crgb = crgb.YV12LUT... and re-activated the line before, crgb = crgb.Tweak(coring=false, bright=Averageluma(c)-Averageluma(crgb)).
Then the green was gone even with bright=true. Please comment that.

What is 'bright' for: The script tries to adjust brightness and contrast of the substitute to those of the original. But because just RT_Averageluma and RT_YPlaneStdev are used to derive brightness and contrast, the algorithm might fail for some footage. I think is is useful for correcting fades. With bright=false, simply this feature remains unused.

Quote:
Originally Posted by Bernardd View Post
I have processed my 3 mn long clip with the November 12 version. After one hour, my clip have no more blurred frames. But some frames was changed, without neccessary for a human read.

It is not easy to control the clip frame by frame. A list of changed frames should be welcome.
In dbgView, with info=1 and info=2, the action on each frame is written, among other data. Please try filtering in dbgView.

Quote:
Originally Posted by Bernardd View Post
In November 18 version script, i see new params "framelist" and "framefile". Thus i wait and hope.
These parameters are already working. You may either enter a space separated list of frames to be processed directly in 'framelist', or a name of a text file with this information in 'framefile'. If you enter both, 'framefile' wins. Line breaks are internally converted to spaces.
You can only enter additional frames to be processed. But because you can easily set dThres or sThres to high values, you can practically disable automatic frame evaluation.

It is better to use the 11/19/2012 version, see change history at script end.
martin53 is offline   Reply With Quote
Old 20th November 2012, 18:39   #20  |  Link
martin53
Registered User
 
Join Date: Mar 2007
Posts: 407
Quote:
Originally Posted by Bernardd View Post
I have processed my 3 mn long clip with the November 12 version. After one hour, my clip have no more blurred frames. But some frames was changed, without neccessary for a human read.
The default for sThres, 1.05, is quite low. Usually, substitutions achieved a sharpness result 1.05 below the sharpness of the substitution source in my experiments, so i set the default to that value.
With higher sThres, the script tries only substitutions if frames with an even higher sharpness factor than the current frame are in the substitution range.
With lower rangeC (or rangeF, rangeI), a smaller neighbourhood is evaluated. This is useful if the blurred sequences are short, e.g. only single frames.
With lower qThres, more frames are over the 'substitution need' limit and left unprocessed. This is the main speed control parameter, too. There should be no default for qThres. But then, you would have encountered one more barrier to try the script. Look at the figures in the original with info=4 and set qThres to the value shown in brackets of an appropriate frame.

If your footage contains many small objects with individual motion - like the skiers, in contrast to a close-up situation in my example clip, then you should try much lower values for lsad and lambda, if you're not satisfied with the default output. High values help MVTools2 to avoid decomposition of distant local sharp parts of the same object. Low values give MVTools2 the chance to adapt to different independent local motions. More about that in the MVTools2 doc.

Last edited by martin53; 15th December 2012 at 16:40. Reason: typos
martin53 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 07:34.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.