Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 24th September 2012, 07:40   #1  |  Link
videoFred
Registered User
 
videoFred's Avatar
 
Join Date: Dec 2004
Location: Terneuzen, Zeeland, the Netherlands, Europe, Earth, Milky Way,Universe
Posts: 689
Capturing and restoring old 8mm films

Hello everybody,

Original thread:
http://forum.doom9.org/showthread.php?t=144271

The original thread is getting a bit heavy, so I tought to start up a new thread. In this thread we can perhaps discuss the used Avisynth restoring methods some more in detail.

I have recently upgraded my 8mm film transfer system. A new and better IMI machine vision camera and a custom made RGB Led backlightsource and capture software, both developed and build by Frank Vine.
www.cine2digits.co.uk

For the restoring I have used my film restoring script but I had to modify it for this specific source. For example I have used some hue color corrections. You can see the effect in the blue sky, the skintones and in the vegetation.

Example clip:


Your feedback is welcome,
many greetings from Belgium,

Fred.
__________________
About 8mm film:
http://www.super-8.be
Film Transfer Tutorial and example clips:
https://www.youtube.com/watch?v=W4QBsWXKuV8
More Example clips:
http://www.vimeo.com/user678523/videos/sort:newest

Last edited by videoFred; 24th September 2012 at 07:42.
videoFred is offline   Reply With Quote
Old 24th September 2012, 17:29   #2  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
Fred,

I think it is wonderful that you are continuing to improve your transfer system and also improve the post-production techniques, including your script. Many people abandon these projects after their initial effort.

Since you are starting a new thread, and since you want to discuss possible areas for improvement in the script, let me make a few comments about the before/after video you posted. As you know, I have used your script, with a few variations of my own, for several years now. I mention this because what I am about to write below could be seen as negative, but I think you will understand that my intention is to be constructive, and to point out those areas that I still think need improvement.

1. Water. One of the problems in both the MDegrain and also RemoveDirt sections of the script is that they deconstruct waves, flowing water, and other similar "watery" structures. If you go to the 0:20 mark in your video and look at the middle upper part of the little waterfall (concentrate on the bright upper part of the middle section), not only is there a green shift, but there is a loss of a lot of subtle structure in the water, most notable in the whitest parts of the water. I have seen this so often that I no longer use your original script on any scenes that contain open or running water.

2. Blue Sky. I have pretty much given up using the color correction in your original script (the "coloryuv(autowhite=true)" setting), although once in awhile it does help correct color casts that I cannot correct any other way. If you look at the blue skies in the various scenes in the clip, you will see a definite turquoise (blue-green) shift. The original blue colors appear much closer to correct. There is also a shift towards green in some of the white highlights in all the scenes in the clip. I use the Spyder colorimeter to color calibrate my display, so I think I am viewing the colors correctly, although as most people know, color is extremely difficult to get right.

3. Gamma correction. It appears that you may have improved the gamma correction algorithms. The street scene taken from overhead (at about 0:14) has wonderful improvement in the shadows. I will be interested to know what you have done to get that improvement. As anyone who has transferred film knows, this is one of the toughest things to get to look good, especially if you use a stock video camera for capturing.

4. Clouds. This is actually probably the same issue as what I described with the water, but if you look at the clouds at the 0:24 mark, you will see that a lot of subtle structure and detail in the dark part of the clouds has disappeared. One of the issues here is grain removal. I have done a lot of noise reduction over the past ten years, and one thing I have noticed is that noise (and grain) impart a certain artificial sense of detail, even where none existed. Part of this is psychological, but part of it may be due to the fact that all these "dancing artifacts" do tend to form real details via their movement, and eliminating that movement reduces the details. Once again, the problem is most evident in "objects" that have very little fine structure or detail. On the other hand, objects with lots of detail -- like buildings, flowers, people's faces -- look spectacularly better after treatment with the script. The flowers at 0:28 look unbelievably good. Nice job!

5. Streets. This is similar to the water and clouds, but it happens so often in my film transfers that I have noted the problem many times. You'll see it in the scene starting around 1:09. At first, all most people will see is the remarkable improvement on the external structure around the building in the background. Good stuff! But then, if you start to look at the road in front of the car, you'll notice that almost all the detail and "depth" has been eliminated. It almost looks like what happens if you take a little water and smear it over a watercolor painting, blending all the details together.

The scene that follows, with the tops of the railway cars, shows both the positives and negatives: at times the details are improved, and at other times, especially on the tops of other cars, the details seem to be obscured. This scene actually may hold some clues as to how we might build a better script because the detail on the tops of most of the cars seems to have been improved, whereas the details on one or two seem to have been diminished.

6. Motion stabilization edge loss. The depan motion stabilization works exceedingly well. I own Mercalli and have written extensively about Deshaker, so I have had a lot of experience using these tools. You have done a great job discovering the right settings for Depan. However, one area of improvement would be to find a way to use Deshaker or some other tool that preserves the edges so that we don't lose so much of the frame due to the zooming needed to hide the moving black edges introduced by the motion stabilization. I have justified this because the motion stabilization, IMHO, is one of the two biggest improvements the script provides (the other being dirt removal), and because my capture systems (like yours) capture the entire frame of film (the gate has been removed) so all that is being discarded is the part of the film that probably would have never been seen during normal projection. Still, it is sad to see all this information discarded ...

So, in summary, the areas of improvement I would like to work on would include (more or less in order of importance):

1. Better color correction.
2. More natural (or less aggressive) grain removal that doesn't harm detail in water, roads, and clouds.
3. Better gamma correction (which you seem to have already achieved -- I just want to know how you did it!)
4. Deshaking that preserves borders.

I don't have any ideas for #1. I think #2 may mostly be a matter of finding better settings for the existing plugins. #3 you have already achieved, but I hope you can explain how you did it. And, #4 will probably mean incorporating Deshaker, something that may be more bother than it is worth, given the two-pass nature of the approach that program takes.
johnmeyer is offline   Reply With Quote
Old 24th September 2012, 18:04   #3  |  Link
RCBasher
Registered User
 
Join Date: Jun 2010
Location: UK
Posts: 2
Hi John,

The better gamma is down to the larger pixels in the IMI camera (6.45um) and my RGB lighting system which optimises the full well depth of the sensor.

Hope this helps to explain.

Frank

EDIT Should also add that in-camera gamma is done before dropping to 8-bit output.

Last edited by RCBasher; 24th September 2012 at 18:16.
RCBasher is offline   Reply With Quote
Old 24th September 2012, 18:28   #4  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
Quote:
Originally Posted by RCBasher View Post
The better gamma is down to the larger pixels in the IMI camera (6.45um) and my RGB lighting system which optimises the full well depth of the sensor.
I'm sure that the camera is doing a great job, and having direct control over the sensor makes a huge difference.

However, my comments were based on looking on the differences between the left (before the script changes) and right (after the script changes) images in the video Fred linked to. The left is the capture from your camera, and it looks pretty good, but the right image has much better gamma, with far brighter midtones and a little more brightness in the deep shadows. It is this change, which was done in the script and not in the camera, that I was referring to. I'd love to know what plugin and settings Fred used to achieve these differences.

Now, if you are saying that the image on the right side was taken with a different camera than those on the left side, then I did not understand what Fred was trying to show, and I should probably delete my entire previous post.
johnmeyer is offline   Reply With Quote
Old 24th September 2012, 18:49   #5  |  Link
videoFred
Registered User
 
videoFred's Avatar
 
Join Date: Dec 2004
Location: Terneuzen, Zeeland, the Netherlands, Europe, Earth, Milky Way,Universe
Posts: 689
Quote:
Originally Posted by johnmeyer View Post
Now, if you are saying that the image on the right side was taken with a different camera than those on the left side, then I did not understand what Fred was trying to show, and I should probably delete my entire previous post.
Please leave your interesting post.

The right side was not taken with a different camera, I can assure you. The trick is to have max. information in the original even when it appears to look dark on the first sight.

Fred.
__________________
About 8mm film:
http://www.super-8.be
Film Transfer Tutorial and example clips:
https://www.youtube.com/watch?v=W4QBsWXKuV8
More Example clips:
http://www.vimeo.com/user678523/videos/sort:newest
videoFred is offline   Reply With Quote
Old 24th September 2012, 18:44   #6  |  Link
videoFred
Registered User
 
videoFred's Avatar
 
Join Date: Dec 2004
Location: Terneuzen, Zeeland, the Netherlands, Europe, Earth, Milky Way,Universe
Posts: 689
Hi John,

Thank you for examining my work in detail!

- Water - clouds- streets: I agree about the loss of detail caused by the heavy grain reduction. Actualy I have already made a few scripts without MVDegrain. But this specific footage needed a lot of additional sharpening. And with this amount of sharpening, the grain especialy in the blue sky became annoying.

- Gamma: like Frank says: this camera/lightsource setup is capable to get nearly everyting out of the film. Even when captured relative dark, the information is still there and it's easy to make it more visible with levels(0,1.2,255,0,255) for example. But I have used a little trick here: levels(0,1.6,255,0,255) followed by autolevels(output_low=6, output_high=250) This gives almost a HDR effect on some scenes.

- Color correction: this is a tricky one because personal taste can play here and all computer systems are different. So I do not know if you see the same colors as I do.

I have used this for the color correction:
Code:
.tweak(sat=1.1, hue=0, starthue=10, endhue=169)\
.tweak(sat=1.0, hue=0, starthue=20, endhue=104)\

.tweak(sat=1.1, hue=0, starthue=170, endhue=280)\
.tweak(sat=1.1, hue=0, starthue=170, endhue=229)\

.tweak(sat=1.3, hue=-4, starthue=280, endhue=360)\
.tweak(sat=1.1, hue=-3, starthue=288, endhue=308)
With other sat and hue values we can make a complete different picture. So it's realy a matter of personal taste.

- Deshaking and cropping the borders: Personal I do not mind so much about that. Of cource I try to keep this as low as possible.

Fred.
__________________
About 8mm film:
http://www.super-8.be
Film Transfer Tutorial and example clips:
https://www.youtube.com/watch?v=W4QBsWXKuV8
More Example clips:
http://www.vimeo.com/user678523/videos/sort:newest
videoFred is offline   Reply With Quote
Old 24th September 2012, 19:16   #7  |  Link
RCBasher
Registered User
 
Join Date: Jun 2010
Location: UK
Posts: 2
Quote:
Originally Posted by johnmeyer View Post
However, my comments were based on looking on the differences between the left (before the script changes) and right (after the script changes) images in the video Fred linked to.
Quote:
Originally Posted by videoFred View Post
Even when captured relative dark, the information is still there and it's easy to make it more visible with levels(0,1.2,255,0,255) for example.
It is also worth to keep in mind that the left hand image is not the actual original, it has been downsized and compressed for showing on Vimeo.
RCBasher is offline   Reply With Quote
Old 24th September 2012, 20:30   #8  |  Link
videoFred
Registered User
 
videoFred's Avatar
 
Join Date: Dec 2004
Location: Terneuzen, Zeeland, the Netherlands, Europe, Earth, Milky Way,Universe
Posts: 689
Quote:
Originally Posted by RCBasher View Post
It is also worth to keep in mind that the left hand image is not the actual original, it has been downsized and compressed for showing on Vimeo.
I had to downsize and compress it for Vimeo of cource, but there's another reason for the downsizing too: VDub did not accept the stackhorizontal() comparison clip in full 2776 x 1036 size.

Fred.
__________________
About 8mm film:
http://www.super-8.be
Film Transfer Tutorial and example clips:
https://www.youtube.com/watch?v=W4QBsWXKuV8
More Example clips:
http://www.vimeo.com/user678523/videos/sort:newest
videoFred is offline   Reply With Quote
Old 24th September 2012, 20:36   #9  |  Link
videoFred
Registered User
 
videoFred's Avatar
 
Join Date: Dec 2004
Location: Terneuzen, Zeeland, the Netherlands, Europe, Earth, Milky Way,Universe
Posts: 689
Here it is John:

http://forum.doom9.org/showthread.php?t=164154

Fred.
__________________
About 8mm film:
http://www.super-8.be
Film Transfer Tutorial and example clips:
https://www.youtube.com/watch?v=W4QBsWXKuV8
More Example clips:
http://www.vimeo.com/user678523/videos/sort:newest
videoFred is offline   Reply With Quote
Old 24th September 2012, 19:00   #10  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
Fred,

Thanks for sharing the settings for gamma correction and color correction. I have a big transfer project which I will start within the next week, as soon as the film arrives. I'll use your ideas to tweak my version of your script and report back on any improvements I am able to make.
johnmeyer is offline   Reply With Quote
Old 24th September 2012, 19:04   #11  |  Link
videoFred
Registered User
 
videoFred's Avatar
 
Join Date: Dec 2004
Location: Terneuzen, Zeeland, the Netherlands, Europe, Earth, Milky Way,Universe
Posts: 689
Yes, please report back John, we are all learning from each other. Someone has posted different versions of RemoveDirtMC() in the other thread. Have you tried them?

Fred.
__________________
About 8mm film:
http://www.super-8.be
Film Transfer Tutorial and example clips:
https://www.youtube.com/watch?v=W4QBsWXKuV8
More Example clips:
http://www.vimeo.com/user678523/videos/sort:newest
videoFred is offline   Reply With Quote
Old 24th September 2012, 19:12   #12  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
Quote:
Originally Posted by videoFred View Post
Someone has posted different versions of RemoveDirtMC() in the other thread. Have you tried them?
No, I missed that thread. I'll see if I can find it.
johnmeyer is offline   Reply With Quote
Old 24th September 2012, 20:29   #13  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
Well, I searched but could not find the thread talking about improvements to the RemoveDirt function. Hopefully my film restoration skills are better than my forum searching abilities

John
johnmeyer is offline   Reply With Quote
Old 24th September 2012, 20:50   #14  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
Thanks! I'll see what I can do with that information in my upcoming projects.
johnmeyer is offline   Reply With Quote
Old 12th October 2012, 22:22   #15  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
Two years ago, I posted an alternative to VideoFred's wonderful film restoration script:

Alternative Script

My main goal was to improve performance by using MVTools2 instead of MVTools, and also by using multi-threading. I made many other changes as well, all of which are documented in that original script, which you can read by clicking the link above.

Since that time, Fred has provided many updates and improvements to his script, but I haven't shared any of my changes. However, now that Fred has started this new thread, I thought it might be useful to share my latest script. All of my work is based on what he has done, but I continue to focus on performance, as well as to improve dirt removal which, along with motion stabilization, is one of the most important restoration steps in improving old amateur film.

Most of the changes are documented in the comments at the beginning of the script (see my next post), but let me provide a few additional details.

Performance

I took Fred's latest script, and set both input and output speed to 16 fps; set dirt removal to 23; and set block size to 4 (you need this small block size to avoid having small "structure" disappear). I used the Result=4 mode, which does not provide any autolevel or autocolor correction (more on this below).

I set my new and old scripts to the same settings, and then fed the same clip to Fred's script, to my original script, and to my new script. I then ran the "video analysis pass" in VirtualDub and recorded the average fps shown in the progress dialog. Here are the results on my 3.2 GHz Intel i7-965 computer (I used six threads in my multi-threaded scripts):

Code:
Fred's latest script: 3.3 fps
My original script:   8.5 fps
My latest script:    15.2 fps
Dirt Removal

In addition to being twice as fast as what I previously posted, my newer script does a much better job of removing dirt because it uses the more sophisticated two-step motion estimation process which has been posted many times in this forum by some of the doom9 "gurus." I spent quite a bit of time (many hours) comparing various pre-filters, and also working on all the other settings. I also used the following test clip to make sure that fine details were not being removed:

Fine Detail Test Clip

It is a real challenge to remove dirt and reduce grain, without also removing the wires that suspend the riders on this old spinning amusement park ride. It is a very useful test clip.

Auto Correction

Fred has made many changes to improve autolevels, autocolor, and motion stabilization. I tried his auto correction improvements, but did not get good results with my film transfers. One reason for this is that Fred uses an amazing custom camera that gives him much different gamma and color than what I get using my Sony FX-1 HDV camcorder as the capture device. Also, I have found that every attempt at doing these corrections automatically almost never works across the huge variations that I get in really old amateur film, all of which suffers from poor development, bad storage (which results in non-uniform fading of each of the three color layers), fogging (from inserting or removing the film cartridge in direct daylight), and many other similar mishaps. I have found, unfortunately, no substitute for manual correction inside my video editing program (Sony Vegas).

So, in my script, I still incorporate the old levels and color correction, but I seldom actually use them.

Also, I did not get much improvement with Fred's newer motion stabilization parameters. I have actually reduced the amount of Depan that I use because if I really need to do serious stabilization, I do that in Vegas using either Deshaker or Mercalli. Therefore, my use of Depan is only to remove film gate weave and minor camera movement. I find limiting Depan to small corrections lets me keep more of the original frame, and when I really need stabilization, both Deshaker and (to a lesser extent) Mercalli provide edge restoration so that no border cropping is done.

Performance "Secrets"

How did I achieve better performance, while still improving quality? Here are the changes (see notes in the script for version numbers of various plugins):

1. Use newer version of MVTools2 that was compiled for the SVP project.

2. Use better MT version of AVISynth (2.60, Aug 28, 2012, Ben Rudiak-Gould)

3. Change motion compensation to two-step process, with first step using large block size. This improves quality, while also providing a slight improvement in performance.

4. Use SSE versions of RemoveDirt and RemoveGrain (I don't know why I didn't do this before ...)

5. Moved the autolevels calls inside of a non-multithreading section, and made sure they were near the end of the script (thanks to Didée for this insight).

Read the script changelog below for the full list.

Frame Interpolation

I have provided a different approach to creating interpolated frames, for those who want to change 16, 18, or 24 fps progressive film to 25p, 30p, 50i, 60i, or 60p video. I do this via SubJunk's excellent InterFrame2.avsi script which in turn uses SVP's SVSmoothFps function. This provides both performance and quality advantages compared to using the old MVTools and BlendFPS technology. I personally don't use this function very often, so I only include it as a commented-out section of code, rather than taking the time to provide a more elegant method of enabling/disabling this feature. However, it does work, and works well.

Bad Frame Removal

I also included the "filldrops" function which was first created by MugFunky many years ago. I use this in conjunction with my editing program. In my editing program, when I have a bad frame (from a projector burn, bad splice, sprocket jump, etc.) I use a macro to replace that frame with a duplicate of the previous frame. I then enable this function (remove the comment). The function passes through all normal frames, but when it detects a duplicate, it replaces that with a motion-estimated frame. Here is an example of what can be done:

Restoration of Jumpy Film

It is a labor intensive process, but the results are pure magic.

Issues

Many of the plugins used pre-date the SetMTMode() approach to multi-threading in AVISynth. It is actually remarkable that they work at all and, unfortunately, while they do work, stability can be an issue. I have found that if I need to jump ahead in VirtualDub or seek backwards, that I often get crashes. The "solution" is to temporarily comment out the SetMTMode() calls.

I have tried to figure out if a "Distributor()" call might help, but I cannot for the life of me figure out where or how that is supposed to be used, and was not able to fix stability problems by adding that call. I also have tried various SetMemoryMax values, and sometimes that will change the point at which instability occurs, but it really doesn't solve anything permanently.

So, multi-threading in AVISynth is what it is, and this script does work for me, but YMMV, and you should be prepared to change the "threads" variable and also the MemoryMax setting.

Summary

So, as you can tell, I made many of these changes because of a difference in equipment, difference in workflow, and different set of criteria than what Fred had in his work. I think I have achieved better performance and better dirt removal, but other than that, I make no claims that my work is "better" than what Fred has done. We are each dealing with different source material, and each of us has a different aesthetic we are trying to achieve: for instance, I want to retain more of the look of film, so I do less grain reduction and sharpening, and don't usually do motion-estimated frame interpolation.

I offer this to Fred and to everyone else in the hope that perhaps we can collectively move forward and continue to improve transfer and restoration of old movie film stock.

Many thanks not only to Fred, but to all the other people in this forum who have provided the tools and techniques that make all of this possible.

The script follows in the next post.

Last edited by johnmeyer; 13th October 2012 at 01:38. Reason: Add AVISynth version information
johnmeyer is offline   Reply With Quote
Old 12th October 2012, 22:23   #16  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
Code:
# film restoration script, based on work by videoFred.
# denoising, resizing, stabilising, sharpening, auto-levels and auto-white balance.
#
# Changes made by John Meyer on October 10, 2012
#
#   1. Modified RemoveDirtMC function to do estimation in two steps. This improves performance and quality.
#   2. Modified MDegrain to do estimation in two steps. This provides roughly the same quality with blocksize=8
#      as what I was getting with blocksize=4, but with virtually no performance penalty
#   3. Updated to use special build of MVTools2 that works with SVP.
#   4. Moved Autolevels inside special multi-threading section in order to improve performance with autolevels.
#   5. Removed all trim commands because I do all editing outside the script (in Vegas).
#   6. Included InterFrame. The InterFrame script is better than MFlowXXX.
#   7. Reduced default horizontal depan setting to 5. If "real" stabilization is need for really shaky film, then I do
#      stabilization prior to using the script, using either Deshaker or Mercalli. Depan is great for small camera shakes and most importantly
#      for removing "gate weave" inherent in film. However, I don't think it works too well as a general-purpose stabilizer.
#   8. Still to do: Improve multi-threading stability (change memorymax and reduce threads variable if script crashes). If you can't goto 
#      a specific frame without crashes, then comment out all SetMTMode() commands. A "distributor()" statement might improve stability
#      when scrubbing, but I doubt it. The problem is that too many of these plugins were not built with MT in mind. The script seems
#      to perform very well, however, if you just load and then only go forward (or render). The problems general happen only when "jumping"
#      ahead in large intervals.
#
# Changes made by Meyer (June 8, 2010):
#   1. Replaced the MVDegrainMulti function, which was part of the original MVTools
#      with the equivalent function using the newer and faster MVTools2.
#   2. Added RemoveDirt function prior to Degrain in order to eliminate large dust spots. This
#      substantially improves the dirt removal capabilities of this script. The user will need to 
#      download this plugin at:
#          http://www.removedirt.de.tf/
#   3. Eliminated a great deal of sharpening. The original script did sharpening in at least four places.
#      With grainy film stock, this sometimes created objectionable grain. Also, the limitedSharpenFaster
#      function, while excellent for video, is uncessarily slow, and isn't really needed for this low 
#      resolution source. Too much sharpening can make the film look too much like video.
#   4. Added multithreading. I was able to roughly triple the script performance. It may be possible to
#      substantially increase this, perhaps as much as 12x instead of 3x. However, the autolevels function
#      would have to be replaced (see notes above, for changes made in 2012).
#   5. Fixed several things I found. The result3 option didn't have the manual color correction code, so I 
#      added that back in. All the numbered "stab" variables (stab1, stab2, etc.) aren't needed except 
#      for providing a test function for stabilization. I eliminated all of this to streamline the script.
#   6. Deflicker seemed redundant, given all the averaging that takes place with MDegrain, and also the averaging
#      that is done when the autolevels outputs are selected. I include that as a "commented out" option.
#   7. I reduced the number of frames used for averaging autolevels from the default (which is 5) to 2.
#      I probably should add a variable in the header so the user can change this. Something else to do
#      in the future ...
#   8. I added yet another set of crop parameters. I did this because both my capture and my output are 
#      done using NTSC DV AVI which is 720x480. However, 8mm film is almost exactly square, so the captured
#      720x480 video has black bars on the side. These need to be cropped off prior to doing motion 
#      stabilization, but then added back prior to the final output, which must still be 720x480 with the
#      black bars on the side.
#   9. I reduced the default depan settings to 20, which is what was recommended in the original script. I
#      also reduced the post-depan cropping. I did this so I could keep as much of the original frame
#      as possible.
#  10. I removed the second denoising and sharpening function. It just seemed to be too much, and made the 
#      result too artificial
#  11. I removed the MVFLowFPS interpolation. I did this during one of dozens and dozens of attempts
#      to improve the speed of the script. I should probably add this back, but if I do so, I also 
#      need to make it work correctly for interlaced output. If the goal is to show this on an NTSC or PAL
#      television set, then it is not correct to convert from the film fps to 25 fps progressive (PAL) or
#      29.97 fps progressive (NTSC). Instead, this should be done as follows (example given is NTSC):
#
#         MFlowFPS(source,super,backward_vec, forward_vec, num=60000, den=1001,ml=200)
#         SeparateFields()
#         SelectEvery(4, 0, 3)
#         Weave()
#
#      This yields interlaced 29.97, which has twice the temporal resolution as 29.97 progressive, and will
#      therefore look correct on a TV set. I have done a lot of this as part of my Kinescope to video
#      conversion scripts. For those scripts, the goal is to make the filmed version of a TV show look
#      like it was actually videotaped. However, for something that originated on film, this "does violence"
#      to the original feel of the media. It is true that it can make horizontal pans less "juddery," but
#      it won't feel like film anymore. Also, this technique does break down, espcially with fast motion
#      in the foreground.
#  12. Removed the unecessary "coloryuv(off_U=blue,off_V=red)" statement from the denoising section.
#  13. Added killaudio() statement to prevent lockups when using SetMTMode().
#
#====================================================================================

Last edited by johnmeyer; 12th October 2012 at 22:51. Reason: Added additional attribution statements
johnmeyer is offline   Reply With Quote
Old 12th October 2012, 22:23   #17  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
Code:
#VIDEO FILE
#----------------------------------------------------------------------------------------------------------------------------
#Change the following line to point to your video file
film="e:\fs.avi"


#GENERAL PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
result="result4"                                               #specify the wanted output here; valid results are "result1" through "result4" and "resultS1" through "resultS4" for before/after
play_speed=16                                                  #play speed (8mm=16; Super8=18; 16mm sound=24)


#COLOR AND LEVELS PARAMATERS
#----------------------------------------------------------------------------------------------------------------------------
saturation=1.0                                                  #for all outputs
gamma=1.0                                                       #for all outputs 
blue= -0  red=-0                                                #manual color adjustment, when returning result3 & result4. Values can be positive or negative
black_level=0  white_level=255 output_black=0  output_white=255 #manual levels, when returning result2 & result4
 

#SIZE, CROP AND BORDERS PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
CLeft=16  CTop=16  CRight=16  CBottom=16                         #crop values after Depan and before final resizing 
W=720  H=480                                                     #final size after cropping 
bord_left=0        bord_top=0     bord_right=0     bord_bot=0    #720p= borders 150
in_bord_left=0  in_bord_top=0  in_bord_right=0  in_bord_bot=0    #Borders around input that must be removed


#STABILISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
maxstabH=05                                                      #maximum values for the stabiliser (in pixels) 20 is a good start value 
maxstabV=20
est_left=40  est_top=40  est_right=40  est_bottom=40             #crop and contast values for special Estimate clip
est_cont=1.6

#DENOISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
#Fastest parameters
#denoising_strength=600                     #denoising level of first denoiser: MVDegrain() 
#denoising_strength=300                     #denoising level of first denoiser: MVDegrain() 
#block_size=   16                           #block size of MVDegrain
#block_size_v= 16
#block_over=    8                           #block overlapping of MVDegrainMulti()

#Best compromise between speed and quality
denoising_strength= 600                     #denoising level of first denoiser: MDegrain() 
block_size=    8                            #block size of MVDegrain
block_size_v=  8
block_over=    4                            #block overlapping of MVDegrainMulti()

dirt_strength=23                            #sets amount of dirt removal (big spots)


#FOUR STEP SHARPENING PARAMETERS
#--------------------------------------------------------------------------------------------------------------------------------
PRE_sharp_ness= 120   PRE_radi_us= 3              #presharpening (UnsharpMask) just after first denoising
Sharp_Strength= 0.1


#AUTO LEVELS PARAMETER
#--------------------------------------------------------------------------------------------------------------------------------
X=4   #X  is a special parameter for reducing the autolevels effect on the whites
X2=4  #X2 is a special parameter for reducing the autolevels effect on the blacks 


#NUMBER OF THREADS
#--------------------------------------------------------------------------------------------------------------------------------
threads=5


# END VARIABLES, BEGIN SCRIPT
#=================================================================================================================================

# Using AVISynth version 2.60, build August 28, 2012 (Ben-Rudiak-Gould, etc.)
# Using VirtualDub 1.8.6

#Change the following (lower or higher) if you have stability problems with multi-threading
SetMemoryMax(800)

#Load plugins explicitly
LoadPlugin ("mvtools2.dll")        #Version 2.5.11.9    2/24/2012
LoadPlugin("autolevels.dll")       #Version 0.6.0.0     1/09/2011
LoadPlugin("Deflicker.dll")        #Version 0.4.0.0     8/16/2004
Loadplugin("Depan.dll")            #Version 1.10.0.0    4/09/2007
LoadPlugin("DepanEstimate.dll")    #Version 1.9.2.0     3/25/2007
Loadplugin("mt_masktools.dll")     #Version 2.0.23.0    3/14/2008
loadplugin("RemoveDirtSSE2.dll")   #Version 0.9         5/05/2005    
Loadplugin("RemoveGrainSSE2.dll")  #Version 0.9         5/01/2005
Loadplugin("warpsharp.dll")        #                    4/05/2010

#Use the following for alternative frame interpolation
#loadplugin("svpflow1.dll")        #Version 1.2.1.0     5/29/2012
#loadplugin("svpflow2.dll")        #Version 1.3.1.0     6/02/2012 
#Import("InterFrame2.avsi")        #Version 2.1.0       6/27/2012

#Open and crop the video
#Remove all setmtmode statements (there are four in this script) if not using multi-threaded (MT) AVISynth
setmtmode(5,threads)
source1= Avisource(film).killaudio().assumefps(play_speed).converttoYV12()           #killaudio() improves stability (in my experience -- others disagree)
cropped_source=source1.crop(in_bord_left,in_bord_top,-in_bord_right,-in_bord_bot)    #temporarily remove any black borders on input video
setmtmode(2)

#cropped_source=filldrops(cropped_source)     #Use this when removing bad frames that have been removed by duplicating previous frame
                                              #(see notes in function at end of script)

#STABILIZING
#....................................................................................................................................................................
stab_reference= cropped_source.crop(est_left,est_top,-est_right,-est_bottom).tweak(cont=est_cont).MT_binarize(threshold=80).greyscale().invert()
mdata=DePanEstimate(stab_reference,trust=1.0,dxmax=maxstabH,dymax=maxstabV)
stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15)

#Alternative line that includes deflicker. I find that this usually is not needed.
#stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15).deflicker()


#DENOISING
#...................................................................................................................................................................

#Remove dirt
input_to_removedirt=stab.crop(CLeft,CTop,-CRight,-CBottom)
stabcrop=RemoveDirtMC(input_to_removedirt,dirt_strength,false)

#Reduce grain
prefiltered = RemoveGrain(stabcrop,2)
superfilt =   MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super=        MSuper(stabcrop, hpad=32, vpad=32,pel=2)

halfblksize= (block_size>4)   ? block_size/2 : 4
halfoverlap= (block_over>2)   ? block_over/2 : 2

bvec1 =  MAnalyse(superfilt, isb = true,  delta = 1, blksize=block_size, overlap=block_over,dct=0)
bvec1 =  MRecalculate(super, bvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100) 

fvec1 =  MAnalyse(superfilt, isb = false, delta = 1, blksize=block_size, overlap=block_over,dct=0)
fvec1 =  MRecalculate(super, fvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

bvec2 =  MAnalyse(superfilt, isb = true,  delta = 2, blksize=block_size, overlap=block_over,dct=0)
bvec2 =  MRecalculate(super, bvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec2 =  MAnalyse(superfilt, isb = false, delta = 2, blksize=block_size, overlap=block_over,dct=0)
fvec2 =  MRecalculate(super, fvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

denoised=stabcrop.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength).levels(0,gamma,255,0,255).tweak(sat=saturation).unsharpmask(PRE_sharp_ness,PRE_radi_us,0)


#SHARPENING
#...................................................................................................................................................................
sharp1=denoised.sharpen(Sharp_Strength)	
PreBorderFrame = sharp1.Lanczos4Resize(W - bord_left - in_bord_left - bord_right - in_bord_right, H - bord_top - in_bord_top - bord_bot - in_bord_bot)


#FRAME INTERPOLATION (optional)
#...................................................................................................................................................................

#Interpolate frames
#Use this for progressive output -- example shows 29.97 progressive
#PreBorderFrame=InterFrame(NewNum=30000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads)

#Use this for interlaced output -- example shows 29.97 interlaced
#PreBorderFrame=InterFrame(NewNum=60000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads).SeparateFields().SelectEvery(4, 0, 3).Weave()


#RESULT1: AUTOLEVELS,AUTOWHITE
#......................................................................................................................................................................
SetMTMode(5)         #Turn off SetMTMode for Autolevels
result1= PreBorderFrame.coloryuv(autowhite=true).addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT3: AUTOLEVELS, MANUAL COLOR CORRECTIONS
#.....................................................................................................................................................................
result3= PreBorderFrame.coloryuv(off_U=blue,off_V=red).addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)
SetMTMode(2)        #Re-enable SetMTMode after Autolevels

#RESULT2: MANUAL LEVELS, AUTOWHITE
#......................................................................................................................................................................
result2= PreBorderFrame.levels(black_level,gamma,white_level,0,255).coloryuv(autowhite=true).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT4: MANUAL LEVELS, MANUAL COLOR CORRECTIONS
#.....................................................................................................................................................................
result4= PreBorderFrame.coloryuv(off_U=blue,off_V=red).levels(black_level,gamma,white_level,0,255).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)


#PARAMETERS FOR THE COMPARISONS
#.....................................................................................................................................................................
W2= W+bord_left+bord_right
H2= H+bord_top+bord_bot
source4=Lanczos4Resize(source1,W2,H2)


#COMPARISONS: ORIGINAL VS RESULTS
#......................................................................................................................................................................
resultS1= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result1,"autolevels, autowhite",size=28,align=2))
resultS2= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result2,"autowhite, manual levels correction",size=28,align=2))
resultS3= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result3,"autolevels, manual color correction",size=28,align=2))
resultS4= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result4,"manual colors and levels correction",size=28,align=2))

#Finish and go home
Eval(result)


# END SCRIPT, BEGIN FUNCTIONS
#=================================================================================================================================


#REMOVE DIRT FUNCTION
#......................................................................................................................................................................
function RemoveDirt(clip input, int "limit", bool _grey)
{
  clensed=input.Clense(grey=_grey, cache=4)
  alt=input.RemoveGrain(2)
  return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,dmode=2,debug=false,noise=limit,noisy=4, grey=_grey)

  # Alternative settings
  # return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=true)
  # return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,tolerance= 12,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=false)
}

#This function provides motion compensation for the remove dirt function, thus improving quality.
function RemoveDirtMC(clip,int "limit", bool "_grey")
{
  _grey=default(_grey, false)
  limit = default(limit,6)
 
  prefiltered = RemoveGrain(clip,2)
  superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
  super=MSuper(clip, hpad=32, vpad=32,pel=2)

  bvec = MAnalyse(superfilt,isb=true,  blksize=16, overlap=2,delta=1, truemotion=true)
  fvec = MAnalyse(superfilt,isb=false, blksize=16, overlap=2,delta=1, truemotion=true)

  bvec_re = Mrecalculate(super,bvec,blksize=8, overlap=0,thSAD=100)
  fvec_re = Mrecalculate(super,fvec,blksize=8, overlap=0,thSAD=100)

  backw = MFlow(clip,super,bvec_re)
  forw  = MFlow(clip,super,fvec_re)

  clp=interleave(forw,clip,backw)
  clp=clp.RemoveDirt(limit,_grey)
  clp=clp.SelectEvery(3,1)
  return clp
}

# The following function will remove near duplicates ("0.1") or exact duplicates (change to "0.0"). 
# It replaces these duplicates with a motion estimated frame. It works really well (IMHO).
#
# Use: In your video editor, replace any single bad frame (burned frame, jump, missing frame from splice, etc.) with a duplicate of the 
# previous frame. Then, include a call to this function (see script above, where it is commented out).
#
# Here's an example of how this function can be use:
#
# http://www.youtube.com/watch?v=uzMFodrGHDs
#

function filldrops (clip c)
{
  super=MSuper(c,pel=2)
  vfe=manalyse(super,truemotion=true,isb=false,delta=1)
  vbe=manalyse(super,truemotion=true,isb=true,delta=1)
  filldrops = mflowinter(c,super,vbe,vfe,time=50)
  fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")
  return fixed
}

Last edited by johnmeyer; 12th July 2021 at 17:16. Reason: Formatting; added "result=" comment; 2015 correct interleave order; 2021 fixed MRecalculate errors
johnmeyer is offline   Reply With Quote
Old 13th October 2012, 13:38   #18  |  Link
fpp
Midnight Artist
 
fpp's Avatar
 
Join Date: Apr 2011
Location: Paris, France
Posts: 82
The sum total of know-how and experience contained in this handful of posts is incredible, and so are the results. Thanks to all for sharing them !

I must definitely make the trip to Belgium someday, to see Fred's new wonder machine :-)
fpp is offline   Reply With Quote
Old 15th October 2012, 06:47   #19  |  Link
videoFred
Registered User
 
videoFred's Avatar
 
Join Date: Dec 2004
Location: Terneuzen, Zeeland, the Netherlands, Europe, Earth, Milky Way,Universe
Posts: 689
Hi John,

Thank you for all the work and for sharing.

But you have used an old version of my script! Both stabilizing and sharpening are different in my latest script. I'm also using the modified Autolevels() by Frustum, and this one is much better than the old one.

Anyhow, you have some very interesting ideas about speeding up everything. At this moment, I'm working on a faster script myself, without heavy degraining, but still with RemovedirtMC().

It seems that the botteneck for speed is RemoverdirtMC(). Can you give me some tips to speed up RemoverdirtMC()? Are the SSE2 versions from the dll's realy that much faster?

many greetings,
Fred.
__________________
About 8mm film:
http://www.super-8.be
Film Transfer Tutorial and example clips:
https://www.youtube.com/watch?v=W4QBsWXKuV8
More Example clips:
http://www.vimeo.com/user678523/videos/sort:newest

Last edited by videoFred; 15th October 2012 at 12:14.
videoFred is offline   Reply With Quote
Old 15th October 2012, 18:39   #20  |  Link
johnmeyer
Registered User
 
Join Date: Feb 2002
Location: California
Posts: 2,691
Quote:
Originally Posted by videoFred View Post
But you have used an old version of my script! Both stabilizing and sharpening are different in my latest script. I'm also using the modified Autolevels() by Frustum, and this one is much better than the old one.
One problem in making such a long post (so long that it took three posts), is that many points get lost. I did try the new autolevels, and also tried the different approach to using that DLL that you have in your latest script (using the Low/High parameters), but I ended up with all sorts of levels "pulsing" where the levels would shift violently from frame to frame. I'm pretty sure it is some sort of scene detection issue and probably could have been easily solved. However, as I said in my last post, both the auto levels and auto color correction (not just in your script, but in all scripts I have tried) just can't cope with the huge variation in problems that I find in old amateur film. So, I have to do those manually, and therefore didn't want to spend a lot of time trying to tweak these settings.

I also didn't use your code because your newer approach involves two more color space conversions (converttoRGB24() followed immediately by a converttoYV12() ), and those always have a pretty significant performance penalty.

As for the Depan improvements, I did try them, but didn't see much difference (I did temporarily bolt your new code into my script). Also, I realized, as I was testing, that unless there is a performance improvement or a borders improvement, I'm not too interested because for "big" motion stabilization, I always use either Deshaker or Mercalli inside of Vegas prior to frameserving into your script (or my version of your script).

However, if you think I should re-visit your autolevels, autocolor, and Depan improvements, I'll be happy to go back and make the changes. It is only a few minutes of work to put them back.

Quote:
Originally Posted by videoFred View Post
It seems that the botteneck for speed is RemoverdirtMC(). Can you give me some tips to speed up RemoverdirtMC()? Are the SSE2 versions from the dll's realy that much faster?
The SSE2 versions only account for a very small portion of the improvement, but there is zero downside to using them, and they are optimized to use faster instructions available in my processor.

Here are, I think, the major reasons why my script is faster, and therefore these are the things you should try when making revisions to your work:

1. Get rid of the original MVTools. MVTools2 is, I think, better in every way, but the most important improvement is that it was rebuilt to be compatible with AVISynth multi-threading. Multi-threading is essential for speed improvement. Once you get rid of the original MVTools, you should be able to use the SetMTMode() statements, and once you do that, you can get integer multiple speed improvements (2x-4x).

2. Use the two-step motion estimation which involves using MRecalculate. This takes very little additional coding. The basic idea is to do an initial motion estimation using a somewhat denoised clip. fft3dfilter is often used, but I found that RemoveGrain is faster and more importantly, results in more dirt removal and fewer false positives. This initial motion estimation is done using really large block sizes, and therefore is quite fast. Then, a second motion estimation is done using the original unfiltered clip, but using the estimation vectors generated from the denoised clip. This second motion estimation is done using a smaller block size. You end up with better motion estimation and, even though two estimations are done, the overall speed is actually faster than if you did one estimation using really small block size.

3. Do the same two-step estimation for the MDegrain section of the script.

4. Try different versions of AVISynth. This is really important when doing multi-threading, both because of stability and also performance.

I do have plans for possible further improvement. As you know, there is another branch of the MVTools development that resulted in a real-time motion estimation tool called SVP. This is designed primarily for creating smooth motion on 24p material by creating interpolated frames. This is actually built into many modern TV sets (the "soap opera effect"). The SVP tool can use GPU acceleration and can be amazingly fast. Unfortunately, it was not created to produce vectors for MDegrain or other plugins like RemoveDirt. However, the authors did respond to requests for a "hack" to let the SVP vectors get passed to these other plugins. I did try this some months ago, and it did work. I just didn't have time to put it into the script and debug it. This potentially could provide another big increase in speed.

SVP is included as part of the InterFrame script for doing motion-estimated frame rate conversion.

Last edited by johnmeyer; 15th October 2012 at 19:09. Reason: clarified MVTools vs. MVTools 2 confusion in original post; added InterFrame statement at end of post
johnmeyer is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 08:48.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.