The power of Avisynth: salvaging "botched" transfers of old 8mm films to DVD.

[fpp]

This is my "Thank You" to the Doom9 Avisynth community, for all the help I've received here.


#0 - TL;DR ? Impatient, in a hurry, or in doubt ? Just read this first :-)

Maybe the title of this topic caught your eye because you were looking for something... but you'd just like to know if it can be of help before you read the entire saga ?

Here's the fastest possible way to make sure with least hassle:

• First, quickly scan parts #2 and 3 below, to see if your situation and goals are close enough to mine,
  
• Next, jump straight to part #21 for the "fast-track" procedure to test Super8Salvage on a sample of your own source !

If it seems worthwhile, then come back and read the rest :-)


#0.0 - Downloads

Full package (10 MB)
With the directory tree, the Avisynth & Python scripts and five supporting apps pre-installed:
AvsPmod, DGMPGDec, DGpulldown, HCenc, Mpg2Cut2
You will need to install Avisynth and its plugins separately.

Minimal package (only 10 KB)
With only the directory tree and the Avisynth & Python scripts.
You will need to install the supporting apps separately.


#0.00 - Change Log

2011-06-13 : updated both packages (had forgotten to actually use the source_fps and target_fps vars in the python scripts, sorry !)
2011-06-05 : added a side-by-side, before/after comparison between the two clips
2011-06-04 : updated both packages (had forgotten to comment out the "slide frame" stuff, sorry !)
2011-06-03 : added package downloads
2011-06-03 : added important note to parts #10, 19 and 20.
2011-06-03 : initial post (21 parts+1)

[fpp]

This topic is about transferring old 8mm/Super 8 films to digital media, but only indirectly : it supposes that the transfer has already happened (in this case, to DVD format), but that it was so badly done as to be painful to look at (especially considering the price paid). This, unfortunately, seems to be a not uncommon occurrence.

We'll discuss ways to employ the magic of Avisynth in such a case, working from the DVD material, in the hope to make a new one that is at least watchable.

The thread is intentionally named similarly to VideoFred's epic saga :
The power of Avisynth: restoring old 8mm films.

This is where I learnt what little I know about the subject, and you should definitely read it if you're interested in film-to-digital transfers.

VideoFred shares his experience in working from a high-quality digital source (he and many others do their transfers themselves, using home-built equipment!), and using Avisynth wizardry to make the results near-perfect, in many ways superior to the original.

Starting from a bad-quality DVD source to make it somewhat passable relies on the same basic principles, but it turns out to be a significantly different exercise in practice.
Moreover, when you start with no previous knowledge in the field, there is a huge learning curve : most of what you need to understand is not well documented, or well hidden, or both.

Doom9 is clearly the place to be in such a predicament, however.
People in VideoFred's thread have been hugely supportive and patient with this newbie and his not-quite-on-topic questions.
Thanks to their help, I went from feeling totally nonplussed to building a workable process in about two weeks (spare time), and that is entirely due to this community's exceptional mindset.

In return, I will try to sum up what I've learnt here, just in case another lost soul wanders by with the same problem I had :-)

[fpp]

My father bought himself Super-8 camera in 1969, which he used until 1980 or so. Then all his equipment was stolen during a break-in, so he had only the reels left, which he couldn't look at.

Recently he decided (without consulting me first :-) to have them transferred to DVD by a local "professional" (we live 500 km apart).
As it often happens, it seems, the guy turned out to be a fraud, and the result was a disaster.

Understandably my dad was quite dispirited about the whole business.
I fully understood that the rational approach would be to bite the bullet, clean up the old reels and have them transferred again, in DV format, by a more competent operator.

This may happen at some point, but in the meantime I still wanted to try and salvage some of the material : make it at least watchable (which it wasn't), remount it in narrative order with subtitles, author a proper DVD with menu, etc.
Additionally, if I could do it in two month's time, it would make a nice present for his next birthday...

Blundering into a field you know nothing about, in a worst-case scenario, and under a deadline : what could possibly go wrong ? :-)

[fpp]

There are quite a few ways to transfer film to digital media. An interesting summmary, which also serves as a good introduction to Avisynth, can be found here :
http://www.thebattles.net/video/8mm_restoration.html

Executive digest : the simplest method, projecting the film on a screen and capturing that with a digital camera, is also the worst, even when done well.
Professional outfits are supposed to use dedicated equipment to achieve professional results, close to the film's original quality.

Of course, people like my father, getting sentimental about their film reels, tend to be old.
And there's nothing to stop highway bandits from taking advantage of their ignorance, charging professional prices for a quick kitchen-table hack I could have done better myself...

Here is a clip from the opening scene, straight from the DVD :



Enjoy the splendid stroboscopic effect, typical of image-per-second mismatch between film projector and video camera. Also, the replay speed is visibly off, too.

Even worse is what you don't clearly see until you happen upon a "white" frame (no film).
Here is a still that shows what the film was projected through/on :

http://dl.dropbox.com/u/10354034/Super8/background.jpg

Classic uneven lighting, but also : lopsided projector or camera; jagged edges; dirty screen or lens... Done right ? :-)

And I had over two hours of this crap, spanning a decade...
(Oh, and I'll spare you the random elevator music :-)

[fpp]

Okay, okay, so this is what I ended up with:

(end result)

(added a side-by-side, before/after comparison)

Of course it is by no means perfect : with infinite time and patience, more tinkering and tweaking would improve it further.

Unfortunately I had neither, so when I reached a state that looked "good enough", I just called it a day (night, actually) and stopped there.

Hint: the final step, authoring a new DVD, always takes longer than anticipated :-)

In between lies the "encoding" step, where you run your entire DVD material through the Avisynth script then re-compress it.
The time it takes depends on how much power you apply through Avisynth, and how fast your PC runs this kind of task.
In my case, as we will see, it was an important factor in deciding what went into the script.

[fpp]

I only wish... but all the Doom9 wizards will tell you that, even under "normal" circumstances, there is no "magic script" that will yield best results for any kind of film, scene, source, defect... they are but templates that need tweaking accordingly.

In our case, there are enough technical differences that even the templates won't work out of the box, and need more than tweaking :

a) AVI vs. MPEG
Most (all ?) high-quality transfers produce uncompressed, or lightly compressed AVI files, in progressive format (meaning: "non interlaced" -- don't get me started ! :-).
All we have is the VOB files from the DVD, from which we extract MPEG video, which is interlaced. MPEG support is not native in Avisynth.
This hurdle trips the beginner at step 1, and it is not trivial.

b) frames per second : and then there were three...
Most (all ?) high-quality transfers rely on a tightly-controlled capture speed. No such luck here !
Super 8mm film is (normally) captured and projected at 18 frames per second (fps).
MPEG from PAL video runs at 25 fps.
The stroboscopic effect is a complicated optical/mechanical/electronic result of the difference.
But wait, notice also the "Charlie Chaplin effect" in the way people walk ?
This means the old film projector was not actually running at 18fps : maybe there was some problem with it, or maybe the operator had changed the speed to try and dampen the strobing (in which case he wasn't very successful :-).
In any case, we need to try our best to guess what that speed was... and then deal with it.

c) fixed "dirt"
See all those spots and grain on the white frame still up there ? That means they will be in the background of each and every frame of the source.
Avisynth has very efficient "filters" to deal with "dirt" -- transient dirt, that is : spots, specks, cuts that happen on a single film frame or a few consecutive ones.
Basically they work by comparing each frame with a few frames just before and just after it, deciding if a given pixel "belongs" there, and smoothing it out if it doesn't.
Obviously this doesn't work very well when the "dirt" was on the screen or lens, and is always present. Some filters (like sharpening) even make it more prominent and visible !

d) output
AVI sources are generally output to similar AVI files to maintain quality, compression only happening at the authoring stage.
Here we already have heavily compressed MPEG sources to start with.
Avisynth will decompress them for processing, but it makes no sense to save them to a high-quality format since we will be turning them back to DVD-compliant MPEG anyway.
So we re-compress the Avisynth output to MPEG directly, and save time during authoring (which will mostly "direct-copy" the video streams).

I may have forgotten some minor niggles already, but these are the biggies.

[fpp]

Oooh, lots and lots of things... I built an incredible collection of new (free) software in the course of this project, enough that it became difficult to remember which one did what :-)
Note that I may have gone overboard in this matter, and there may well be simpler ways to achieve the same goals.
Being both ignorant and in a hurry, I mostly just grabbed the first tool that came recommended and seemed up to the job.
So take this as my personal experience, YMMV...

First, obviously, you need Avisynth.
You may already know, however, that Avisynth itself has no user interface.
It is basically just a bunch of Windows DLLs acting as a "frame server", reading an existing media file, processing each image according to a script, and spitting out the results to whatever "client" software happens to be interested.
These "consumers" are the actual user interface, and there are many : VirtualDub, VirtualDubMod, AviDemux, some media players, encoders and whatnot.
Some have a GUI, some are command-line driven, some are both.

The very next thing you should do to keep your sanity is to install AvsPmod.
At first sight it is just a competent multi-tabbed code editor : Avisynth scripts (.avs and .avsi) *are* source code after all, much like simplified Python.
What makes it invaluable is its built-in frame-by-frame media player, which lets you see in real time the results of your code, and helps finding bugs by displaying helpful error messages.
There are other such tools, but this one seems to be the only one currently maintained.

Then when you really start tinkering, you will need additional "plugins".
Mostly these are just Avisynth-specific Windows DLLs, created by community wizards to add more powerful functions to Avisynth's internal ones.
This is where the fun starts, double-time:

First, you have to discover that they exist, then track them down.
Not always easy, even if the Avisynth Wiki is a good starting point : they're all over the place.
Some have a Web page (which is good), some have several (which is less good), others still are hidden deep in the bowels of a forgotten Doom9 discussion.
Here you will find the result of my haphazard gathering, but I certainly don't claim it to be extensive or even adequate :-)

Second, "Windows DLL" is usually associated with "DLL Hell", with good reason.
I can guarantee you will pull out your hair at some point, trying to determine which is the latest version, which goes with your version of Windows, which version of X is compatible with Y, all the usual...

One infamous example of this is "Avisynth MT", the multi-threaded variant of the original Avisynth.
Basically it is "just" a modified version of the engine, that you drop into your Windows System folder in replacement of the old one.
Many people (including me) seem to have trouble finding the right version of this DLL, among all those floating around, that will work on their system (trial and error works best :-).
It is worthwhile to go through that trouble though, because it will pay off handsomely when you get to the heavy-duty filtering and re-encoding of your media.
Even if you have a lowly old Core2Duo as I do, number crunching time will be much reduced. Even better if you have a recent AMD or Intel multi-core, of course.
Fortunately, determining whether it works is simple : run the one-line script "SetMTmode(1,0)" in AvsPmod, and keep insisting until it doesn't throw the "No such method" error :-)

Now that the basics are in place (hopefully), we need to turn to another matter : extracting the crappy video from our botched DVD, then turning it into suitable material for use with Avisynth, re-encoding, and finally authoring of the new DVD.

We'll take it step-by-step from there, introducing new tools as required for each task.

Note: grabbing VideoFred's complete package, from the first post in his thread, will give you most of the plugins needed here. I'll mention the others when they come up.

[fpp]

So we have a DVD with a TS_VIDEO folder with a bunch of .VOB files inside, holding our video in custody. What now ?

My first move was to extract the video in MPEG format, with one file per title in the DVD.
Maybe the titles make some sense (like one per reel of film), maybe they don't, but at least it's a step forward.
The easiest and fastest way I've found of doing this is to use the free version of VOB2MPG 3 : http://www.svcd2dvd.com/VOB2MPG/default.aspx
You just point it at the TS_VIDEO folder, check all the titles, tell it where to put the result, and click "Start" : a while later you will find a bunch of files named Title_1.mpg, Title_2.mpg, etc. in the target folder.

Note: the free version of VOB2MPG does not let you change the default file naming method.
This is important, because the automation scripts I will show you later on depend on this naming convention (and others), and may not work if you change it.

Speaking of which, it is time we start paying attention to how we lay these things out on our disk.
Let's say we are working on great, big drive X:, in a folder we named "Super8Salvage"
Under this we created a "source" subfolder and used it as the target for VOB2MPG, so now we have something like this :

X:\Super8Salvage
<DIR> source
Title_1.mpg
Title_2.mpg
Title_3.mpg
...

Better already !
However, the titles probably contain several different clips or scenes, maybe bad frames or uninteresting parts too.
For authoring, you ideally want to start from "rushes", i.e. small, elementary clips, so you can order and assemble things just the way you want.
So we need a quick 'n easy way to cut up our titles into smaller sequences, eliminating any dead wood in the process... that would be Mpg2Cut2 : http://rocketjet4.tripod.com/Mpg2Cut2.htm
This little app is perfect for us : it deals only with MPEG2 (what we have), and its viewer lets you mark as many start/end point pairs as you want.
Then the File/Save Parts menu will save each part to its own individual file, in the same folder as the original.

Again, it is important to accept the default naming convention offered by Mpg2Cut2, which is to add _00X to the original file name.
So if we load Title_1.mpg into Mpg2Cut2, and cut it up into three parts, we end up with this :

X:\Super8Salvage
<DIR> source
Title_1.mpg
Title_1_001.mpg
Title_1_002.mpg
Title_1_003.mpg
Title_2.mpg
Title_3.mpg
...

After doing this for each title, we now have the raw material to work on, and we can start trying to improve it with Avisynth.

There is one little problem left, however : by itself, Avisynth is *really* bad at MPEG files...

[fpp]

Well, it's called "AVI"-synth, right ?

Try this : open a blank script in AvsPmod, then drag-and-drop a random AVI file onto it. This will add a single line to the script :

AVISource("randomfile.avi")

All well and good : AVisynth opened the media file on its own, and you can start navigating it in AvsPmod's viewer.

Now do the same with Title_1.mpg. It appears to work the same way, but it actually doesn't, because the command is now :

DirectShowSource("Title_1.mpg")

This is Avisynth's way of dealing with files it doesn't know how to handle : it just delegates the work to the underlying Windows graphics system, and blindly accepts whatever comes back from it.
In a way it does work, in that it loads the file and lets you play around with it; but it is useless for the kind of work we want to do, because we cannot expect any kind of frame-by-frame accuracy from this stuff. So we need something better.

That something would be DGMPGDec : http://neuron2.net/dgmpgdec/dgmpgdec.html

As usual, DGMPGDec is mostly a Windows DLL that you copy to the Avisynth "Plugins" subfolder.
It adds a new command to the scripting language, named "MPEG2Source", which will let you work on MPEG2 media files with great precision and control. Yay !

So, are we ready to roll then ? Well, not quite...

You see, MPEG2Source, unlike AVISource or DirectShowSource, does not directly load an MPEG media file by its name.
What it wants instead is a much smaller file containing some text and numbers, and ending with ".d2v".
This is actually an "index file" for the real MPEG file, which is referenced inside.
Right, but where does this .d2v companion file come from, and how ?

This is where DGIndex comes in.
DGIndex comes with the DGMPGDec package. If you just run it you will see yet another preview window : load "Title_1.mpg" in it (only one file at a time!), then press F4 (or File/Save project).
By default DGIndex will create the .d2v file with the same name as the .mpg file, which is fine.
It can also do many other nifty things, but we don't need them now, so we're done.

Now let's try the drag-and-drop trick in AvsPmod with our new .d2v file : this time the new line reads,

MPEG2Source("Title_1.mpg.d2v")

And it opens the real .mpg file which is displayed in the preview pane. Whew ! :-)

[fpp]

Yes, Virginia, and we'll need to do it in the proper order, which means a mix of simple and complicated steps.

Let's start with simple : if you go back to post #5, you'll see we have cleared the first hurdle in part a), as we do have our MPEG file loaded in Avisynth and displayed in AvsPmod.

It is still in "interlaced" format however, as you can clearly see if you step through it frame-by-frame : that is where the "striping" effect comes from, that you see from one frame to the other.

Fortunately, basic deinterlacing is not too hard, and doesn't need any new plugin, as it's built into Avisynth. Just add one line after loading the file :

MPEG2Source("Title_1.mpg.d2v")
# remove combed frames by Didée
Bob(-.2,.6)

...and Bob's your uncle. Congratulations, you just achieved your first working, two-line Avisynth script !

The one thing you should NOT do right now, is ask: "But why Bob, -.2 and .6 and not, say, Tom, -42 and .666 ?"

Do it 'cause I said so, just as I did it 'cause a kind soul (Didée) told me to, and it works, so just take it on faith right now.
If you want to understand, you'll need to read the Avisynth Wiki and countless forum threads on Doom9 and elsewhere... don't go there if you're on a deadline ! :-)

Okay, on to more hardcore stuff now.

[fpp]

If you step through your newly "bobbed" clip frame-by-frame, you will see that the "striping" effect of the interlaced original source is mostly gone. Great.

But if it's anything like my sample, and you pay close attention, you will also see that:

• there are a lot of duplicate frames : that's when the image changes only after you've hit the right-arrow twice or more
  
• there is the occasional "combed" frame, which still shows a striping effect as if it was interlaced.

That's because of part b) in post #5 : PAL MPEG plays at 25 fps, but the Super 8 film was shot at 18.

Fixing this is another one-liner, that looks deceptively simple, but wields some heavy black magic :

MPEG2Source("Title_1.mpg.d2v")
Bob(-.2,.6)
SRestore(frate=18)

For this to work you will first have to install SRestore (itself a script acting as a plugin, with the .avsi extension) and its required DLLs (MaskTools2 and TIVTC) : http://avisynth.org/mediawiki/Srestore

Are things better now ? You should see a lot LESS duplicate and combed frames... but maybe not NONE.

Remember the "Chaplin effect" ? Probably the film projector itself was not actually running at 18fps during the transfer... SRestore works its magic best when the "frate" parameter is closest to the actual value in the source.
But how do we find out ?

Sadly, there is no magic here, just trial-and-error and a lot of patience... Thanks to John Meyer for the Sherlock Holmes crash course !

What you do is count the frequency of duplicated/combed frames for a given frate; then add 0.5 or 1 fps, and start again.

Gradually both frequency counts should decrease. The nearer you get to the real value, the more frames you need to step through before seeing a duplicate or combed one ("several dozen" was the advice I got, possibly over a hundred). If the counts start going up again, you've overshot : a classic dichotomy search candidate.

So, clearly, your patience and persistence will define the level of precision you achieve. When you get to well below one rogue frame in a hundred, it will not be noticeable in normal viewing conditions, and speed of movement will feel natural. In my case that level was met at frate=19.75, so I left it at that.

Whew, that was a biggie, but at least we've cleared problem b) : our MPEG source is loaded and browsable; it has been de-interlaced (made progressive), which is necessary for most Avisynth filters to work well; and it's "running" at the correct speed, even if it's a totally non-PAL-compliant one... We'll deal with that later : now at last, we can really start trying to improve the picture quality, in earnest!

IMPORTANT NOTE:
This value, obviously, will be specific to your particular source. You WILL need to change it at the very start of the 04_Salvage.avs script to get the correct result.

[fpp]

Avisynth wizards usually apply changes in specific order, and the first thing they do, if needed, is "stabilize".

Amateur Super 8 movies are often shaky and jerky, which can be painful to watch.
Powerful filters (plugins) can attenuate or even remove those defects, and make the main image much more enjoyable, even if they sometimes create artifacts at the edges.

There are several ways to go about it. Some are more efficient than others.
The code below is lifted, as-is, from VideoFred's "Restore_film" mega-script : this is heavy metal, people, so just trust the wizards...
You're supposed to tweak the default parameter values for best results on your movie, but that gobbles time like mad, so I didn't.

# stabilize by VideoFred (but without deflicker)
requestlinear()
o=last
maxstabH=20 maxstabV=20 # maximum values for the stabiliser (in pixels) - 20 is a good start value
est_left=40 est_top=40 est_right=40 est_bottom=40 est_cont=1.4 #crop and contast values for special Estimate clip
stab_reference= o.crop(est_left,est_top,-est_right,-est_bottom).tweak(cont=est_cont).MT_binarize(threshold=80).greyscale().invert()
mdata=DePanEstimate(stab_reference,trust=1.0,dxmax=maxstabH,dymax=maxstabV)
DePanStabilize(o,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=0,mirror=15)#.deflicker() <-- twice the encoding time on my machine !

Notice that I commented out the last call on the last line, after finding out that it doubled the processing time with no visible improvement... Probably a local glitch on my system, though.

Note: from here on you can still step through the script frame-by-frame, if you're patient, but not scroll it at anything resembling real-time, not on my hardware, anyway :-)
While in the experimenting stage, you may want to skip it and do the following filters first, some of which are quite fast and change the image appearance a lot. I added it at the very end myself, just before final encoding.

[fpp]

One of the most obnoxious defects in the original DVD transfer is that terrible flicker/strobe effect.
There are a number of tools to correct this with Avisynth, but the recipe Didée gave me in the other thread is incredibly efficient, and faster than most.
It even surprised some seasoned veterans out there :

# remove flicker by Didée
rest=last
calm = rest.temporalsoften(1,255,255,32,2).merge(rest,0.25)#.repair(rest,1).removegrain(11)
calm = calm.temporalsoften(1,255,255,16,2).merge(calm,0.25)
LOP = calm.bicubicresize(180,144).bicubicresize(720,576,1,0)
HIP = rest.bicubicresize(180,144).bicubicresize(720,576,1,0)
HIP = mt_makediff(rest,HIP)
mix = LOP.mt_adddiff(HIP,U=2,V=2)
sup1=rest.msuper(levels=1)
sup2=mix.removegrain(11).msuper()
bv2=sup2.manalyse(isb=true, delta=2,blksize=16,overlap=8,DCT=5)
bv1=sup2.manalyse(isb=true, delta=1,blksize=16,overlap=8,DCT=5)
fv1=sup2.manalyse(isb=false,delta=1,blksize=16,overlap=8,DCT=5)
fv2=sup2.manalyse(isb=false,delta=2,blksize=16,overlap=8,DCT=5)
rest.mdegrain2(sup1,bv1,fv1,bv2,fv2,thSAD=640)

This, and SRestore, are the two most complex filters that are specific to working from a bad transfer. Thanks Didée !

[fpp]

The traditional before-last step is trying to "clean up" the picture : noise, grain, sharpness...

VideoFred's script does this in a lot of careful, gradual steps, with the impressive results you can see on his site.

Unfortunately, this works well on the sort of ultra-clean transfers he does himself, but not on the crappy, compressed material we have here.

When I tried it on a sample it actually came out worse, because the "static" noise in the background was sharpened and brought to the foreground, making even more prominent.
Moreover, it added a lot of computing overhead.
Lacking the time and understanding to peek and poke, I took the easy way out and pilfered just two lines that gave "good enough" results :

* RemoveDirtMC is a function "re-discovered" and suggested by John Meyer, and used by VideoFred in his scripts. It's very effective on non-static "dirt" :

RemoveDirtMC(40,false)

* I also used VideoFred's very last step, skipping the rest :

sharpen(0.4).sharpen(0.4).blur(0.2)

Sure, this is all guesswork and hunches, but it makes a real enough difference.

[fpp]

The last steps in the script are mostly cosmetic, only the last two are really important.

First I corrected the lopsided and jagged edges of the screen/projector by cutting most of them out, and adding equivalent black borders to keep the same image size:

Crop(22, 30, -24, -16)
AddBorders(22, 30, 24, 16)

Obviously this should be adjusted according to you own source.


Then I got fancy and made a black-and-white "mask" of a slide frame, with the same borders but with rounded corners, for the "oldies" effect.
It is combined with each frame:

slideframe = ImageSource("S:\Super8Salvage\slideframe.bmp")
Overlay(slideframe, mask=slideframe.Invert())

This is completely optional, and commented out by default in the script.

Colour correction is the usual last step of film improvement, as decades-old movies often have too much blue, or red, etc.
Serious players all seem to use a plugin called "autolevels", but in my case I was unable to get meaningful results from it (maybe because of the source, or another system glitch).
So I just used the built-in Avisynth function:

ColorYUV(autogain=true, autowhite=true)

Of course, for best results, you are supposed to use manual settings, and tweak them for each specific scene.
As usual, I was lazy and in a hurry, so I left it "automatic", and found it did a competent job.

The last line is simple but super-important:

AssumeFPS(25)

Remember, after the SRestore magic spell, we were left with a clip running at its true speed, 19.75 fps.
But no standards-compliant MPEG encoder will accept that, so we cheat by declaring the clip is actually 25 fps.
Of course that is much too fast, but hey, "there's an app for that", too :-)

At this point we declare the Avisynth part officially "good enough", and the script "frozen".
The hard part is knowing where to stop, as you could go on tweaking literally forever, but you have to draw the line somewhere, for there is a lot left to do...

[fpp]

Well, now we need a competent, DVD-compliant MPEG encoder that will act as an Avisynth client, use the result of the script and recompress it into MPEG2 files with which we can later author our new DVD.

That would be HCenc : http://www.bitburners.com/hc-encoder/

HC has an interactive mode (HCgui), so you can just run that, give it a name for the results ("Title_1.m2v"), and after some happy crunching-time you will have your first "salvaged" equivalent of the original clip... at last !

IMPORTANT WARNING :
HC also has a command-line mode, which we will use later. One nasty pitfall is that both modes use the same HC.ini settings file. When you use HCgui for testing, it will record the last settings in it; when you use the command line later on, it will use those to fill in for any missing parameter in that command, which can lead to unpleasant suprises. If the result is not what you expect, think of that first!

[fpp]

Well, of course, because we cheated by declaring it was 25 fps (so that HC would accept the job), but it's still really 19.75...

Again there are several ways of correcting this.
In his script, VideoFred offers two, both of which involve creating new frames from the original ones (through "interpolation" or "blending", your choice), and mixing them in so that the new clip will run smoothly and at the right speed.
This is the "right way" when you are starting from a high-definition AVI source and want to be free to do anything with the results.
However, it does add computing time (as always), and also makes the files bigger.

In our case, as we are starting from a DVD and will only make another one, we can cheat once again, by taking advantage of a mysterious property of MPEG files called "pulldown".

So let me introduce our very last magic tool, DGPulldown, by the famous Avisynth wizard Donald A. Graft : http://neuron2.net/dgpulldown/dgpulldown.html

DGPulldown has a very bare interface : you specify the source clip, the target name, select "Custom", type "19.75" (our clip rate) in the first field, "25"(PAL DVD) in the second, then click "Convert".
It is blindingly fast, because the target is the exact same file and size, only with the "pulldown" property changed.
But when you run it in a media player, or from DVD, the "Charlie Chaplin effect" is gone!

Well... after all, if you believed all you've read so far, you can take that tiny much more on faith :-)

Note: one side-effect of DGPulldown is that most software gets the clip duration wrong. This is not a real problem, and the final DVD works just fine.


#17 - Congratulations !

If you made it this far, you are a brave person, but you also now have everything you need to get started, if you're on the same boat I was.

If you read the rest, however, you will find things to make your journey a lot easier still!

[fpp]

My PC is neither very new nor fast - it's a Core2Duo E8400 from 2008 running at 3 GHz.
Processing a DVD's worth of video, frame by frame, is a heavy load.
This was an important consideration all along while choosing which methods and filters to use, to achieve a good compromise between output quality and computing time.
The law of diminishing returns applies here, too, especially with such a bad source.

Here are some test results to illustrate, with different versions of the script applied to the same sample clip, running exactly 60 seconds.
Durations are for HCenc output, taking into account both the Avisynth processing and MPEG encoding.
The latter is supposed to be roughly constant, so differences are attributed to the Avisynth script.

Base reference : the above script, WITHOUT stabilizing nor sharpening (so, remove flicker and dirt, mostly)
Duration : 09:32 -- we're running almost 10-to-1, at a minimum...

With stabilization :
MVTools method 1 (found on Net) : 12:32
MVTools method 2 (found on Net) : 12:45
DePan method (VideoFred) with deflicker call at end: 20:21
DePan method (VideoFred) without deflicker call at end: 10:23

With stabilization and sharpening :
As shown above : 10:23

Mongrel : mine up to "remove flicker", then chained with VideoFred's (interpolated)
Duration : 18:54

So, stabilization can be costly and deserves some investigation. Sharpen/blur are negligible. Full "VideoFred" is overkill for this material.
At best, the computer will be doing nothing else than crunching video for over ten hours straight. Better get things right, first :-)

Speaking of which...

[fpp]

Hopefully not :-)

If we recap, once you've extracted all your titles (Title_XX.mpg) from your DVD, and cut each of them into rushes (Title_XX_0YY.mpg), then for each one of these you need to :

• use DGIndex on it to create the Title_XX_0YY.d2v index
• create or edit an Avisynth script that opens Title_XX_0YY.d2v and contains the processing code
• open that script in HCenc to process the clip and re-encode the result to Title_XX_0YY.m2v
• open that new clip in DGPulldown to produce its clone with the corrected speed

My first DVD had 11 titles, which produced over 50 rushes.
That's a lot of clicking around, with over ten hours of encoding mixed in, and a lot of similar-looking files to keep track of.
Repetition is boring, so there is a strong probablility of messing up somewhere...

Fortunately, all these tools have a command-line mode, and can be used to automate the process.
DGIndex even goes the extra mile : you can pass it the name of a "template" .avs script, with a placeholder instead of the source name inside, and it will create a new one using the name of the source it is indexing. How considerate!

Being a Python programmer myself, I have written a python script that will scan the current folder for files like Title_XX_0YY.mpg, and create three Windows batch files : one for indexing, one for encoding, one for pulldown.
To keep the source folder tidy, the various files are written to three separate subfolders, one of them being the final result folder.
The batch files can be run separately, but there is a fourth one that will run them in order.
Optionally, it will also shut down your computer when done, if left unattended.

Clips are only processed if one of the result files is missing, so you can use it incrementally as you add rushes.

Obviously you will need the Python (2.x) run-time installed to use this, but no knowledge of Python itself : the few things you might need to change are set as variables in the first few lines of the script.
It is well commented for those who would tinker with it, however.

To make things even easier, you can download a single archive with almost everything inside it (except Avisynth and plugins) :

• the "04_Salvage.avs" script, with the code turned into a function, and its supporting script "03_RemoveDirtMC.avs"
• the "05_template.avs" script which imports the above function (this way, you can change the code and still use all the already generated .avs files)
• the Python script to generate batch files for a source folder (_create_jobs.py)
• AvsPmod, Mpg2Cut2, DGMPGDec, DGIndex, HCenc and DGPulldown
• a directory tree to hold all this : just unzip at the root of any drive and it should work.

IMPORTANT: If you use the Python script, you will need to change the first variable to reflect your source's actual speed, as in the 04_Salvage.avs script.


Note: there is also a minimal package with only the directory tree and Avisynth/Python scripts. You will find them both in the first post above.

[fpp]

The archive also contains a second Python script, which may or may not be of use to you.
This one does not create batch files.
It acts directly upon .bmp image files in the current folder, to generate short clips out of them.

If a file's name ends with ".i.bmp" ("i" is for "in"), a .m2v clip with the same name will be created with fade-in, display 5 seconds, fade-out.
This is to make title screens for clips : the bitmap will often be a capture of the first frame of the clip, with some text added.

If a file's name ends with ".o.bmp" ("o" is for "out"), a .m2v clip with the same name will be created with fade-out in 2 seconds.
This is to make end screens for clips : the bitmap will often be a capture of the last frame of the clip, for smooth transitions.

Clips are only processed if one of the result files is missing, so you can use it incrementally as you add bitmaps.

Most authoring software offers this kind of functionality, but often at the cost of re-encoding the video (for the third time!).
This replacement is rather basic, but it lets you treat opening and ending scenes like ordinary rushes.

In my case, I created my new titles by simply assembling the relevant rushes in the right order, using yet another magic tool, Avidemux (in "Copy" mode, very fast).

The DVD authoring software was only used, at the very end, for its most basic purpose : ordering titles, adding sound tracks, designing the menu, and creating the ISO image.

IMPORTANT: If you use this script, you will need to change the first variable to reflect your source's actual speed, as in the 04_Salvage.avs script.

Note: this script is also part of the packages above (_create_ios.py).
It should be run in a specific subfolder of the source, like shown, as it can get cluttered pretty fast.