The power of Avisynth: salvaging "botched" transfers of old 8mm films to DVD.

[GlemGem]

Thanks again.

I've downloaded the Microsoft Visual C++ 2005 SP1 Redistributable Package (x86), installed it and re-booted my system. However, I got still the same error "there is no function named...". :(

[fpp]

Well I'm out of ideas, although I'm sure it must be something similarly silly (I know I had some hurdles like this at the star, unfortunately I was lazy and didn't keep track).

I suggest taking the specific question to a more general Avisynth topic (here on Doom9 or elsewhere), where it could get better exposure by more knowledgeable people...

[GlemGem]

Ok, I just started a new thread.

[GlemGem]

Merry Christmas all! :)

The error "There is no function named MPEG2Source" is fixed now, see this thread: http://forum.doom9.org/showthread.php?t=166756

Anyway, I have another error now "MDegrain: wrong source or super frame size", see here: http://oi49.tinypic.com/15i71bs.jpg

Do you know what this is and how I can fix it?

[cretindesalpes]

Your input frame is probably not 720x576. Try to replace the bicubicresize(720,576,1,0) expressions with bicubicresize(rest.Width,rest.Height,1,0).

[GlemGem]

Your input frame is probably not 720x576. Try to replace the bicubicresize(720,576,1,0) expressions with bicubicresize(rest.Width,rest.Height,1,0).
Cheers mate! That worked. :)

I can run the my_DVD_sample.avs in AvsPmod without any errors and the preview looks good. However, if I run the automation scripts by using the 0_run_all.bat it doesn't work.

I got 2 errors by HCenc_025.exe:
Error 1: there is no function named "SetMTMode", see: http://oi50.tinypic.com/f3zx9v.jpg
Error 2: incompatible input, use ConvertToYV12() or ConvertToYUY2(), see: http://oi50.tinypic.com/121322v.jpg

I am not using the multi-threaded version (MT) of AS. I comment out with "#" every line containing SetMTMode() in 04_Salvage.avs and 05_template.avs like user fpp suggest here: http://forum.doom9.org/showthread.php?p=1505200#post1505200

Also, I am using ConvertToYV12() already after the "Bob()", suggested by user StainlessS here: http://forum.doom9.org/showthread.php?p=1607132#post1607132

I am wondering why it works in AvsPMode and when testing with the automation script not?! Any ideas?

My current script:

LoadPlugin("C:\Super8Salvage\dgmpgdec\DGDecode.dll")

Import("C:\Super8Salvage\avs_scripts\03_RemoveDirtMC.avs")
my_frame_rate = 19.75 # in my case, could be anything around 18, above or below...

MPEG2Source("C:\Super8Salvage\source\my_DVD_sample.d2v")
#DGDecode_mpeg2source("C:\Program Files (x86)\AviSynth 2.5\Super8Salvage\source\my_DVD_sample.d2v", cpu=0,info=3)

#Return Info()

bob(-.2,.6) # deinterlace MPEG source
ConvertToYV12()
SRestore(frate=my_frame_rate)
requestlinear()
o=last

maxstabH=20 maxstabV=20 # maximum values for the stabiliser (in pixels) - 20 is a good start value
est_left=40 est_top=40 est_right=40 est_bottom=40 est_cont=1.4 #crop and contast values for special Estimate clip
stab_reference= o.crop(est_left,est_top,-est_right,-est_bottom).tweak(cont=est_cont).MT_binarize(threshold=80).greyscale().invert()
mdata=DePanEstimate(stab_reference,trust=1.0,dxmax=maxstabH,dymax=maxstabV)
DePanStabilize(o,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=0,mirror=15)#.deflicker() <-- twice the encoding time on my machine !

rest=last
calm = rest.temporalsoften(1,255,255,32,2).merge(rest,0.25)#.repair(rest,1).removegrain(11)
calm = calm.temporalsoften(1,255,255,16,2).merge(calm,0.25)
LOP = calm.bicubicresize(180,144).bicubicresize(rest.Width,rest.Height,1,0)
HIP = rest.bicubicresize(180,144).bicubicresize(rest.Width,rest.Height,1,0)
HIP = mt_makediff(rest,HIP)
mix = LOP.mt_adddiff(HIP,U=2,V=2)
sup1=rest.msuper(levels=1)
sup2=mix.removegrain(11).msuper()
bv2=sup2.manalyse(isb=true, delta=2,blksize=16,overlap=8,DCT=5)
bv1=sup2.manalyse(isb=true, delta=1,blksize=16,overlap=8,DCT=5)
fv1=sup2.manalyse(isb=false,delta=1,blksize=16,overlap=8,DCT=5)
fv2=sup2.manalyse(isb=false,delta=2,blksize=16,overlap=8,DCT=5)
rest.mdegrain2(sup1,bv1,fv1,bv2,fv2,thSAD=640)

RemoveDirtMC(40,false)
sharpen(0.4).sharpen(0.4).blur(0.2)

Crop(22, 30, -24, -16)
AddBorders(22, 30, 24, 16)
ColorYUV(autogain=true, autowhite=true)
AssumeFPS(25)

[GlemGem]

Another thing I just noticed, when I run the script in AvsPmode and choose "run script with external program" (Windows Media Player Classic in my case), the file plays very slow, first 3 seconds needs almost 10 seconds to show, then pause, plays another few seconds very slow, and then again, it plays the whole clip in super slow motion. My sample mpeg-file has a length of just 24 seconds. The problem mentioned above still exists and until now I can't find the reason what's wrong.

Would be great if someone can help.

[fpp]

Are the 04_Salvage.avs and 05_template.avs scripts, that you have modified, those in \Super8Salvage\avs_scripts ? These are what 0_run_all.bat & Co should be calling...

Please also note that the "my_DVD_sample" kit is just a quick way to check that your setup is working (not so quick in your case, but useful :-)
The _create_jobs.py and batch files require that your own media files follow the naming conventions as described in post #19.

As for the "external program" issue, I would think that is normal. This kind of processing is far from real-time, although exactly how much depends on how powerful your computer is (or not).
On mine, the various versions of the script run between 10x and 20x (that is, ten to twenty seconds of processing for each second of video - see post #18).
So your media player is just waiting for Avisynth to send the frames as it computes them one by one, with probably some buffering added in to make things even choppier :-)

[fpp]

Glemglem,

It seems your setup has several differences relative to mine : apart from SetMTMode not working, it also does not autoload plugins.

Also, the mpg file extracted from your DVD is in a peculiar 720x480 YUV format.
My script assumes YV12, and had two bicubicresize calls hard-coded for 720x576.

Here is a modified 04_Salvage.avs with these corrected (much as you did) and the relevant plugins/imports explicitly loaded (you may have to correct the paths for your system).
It goes into "\Super8Salvage\avs_scripts" on your work drive.
It works on my system, please try it on yours.

Note: the batch files generated by the python script are correct, but they only work if the avs scripts work.
If something fails, as you've seen, the error messages displayed by HCenc are generally not helpful.
The only effective way of debugging a problem is to load the avs script with the same name as the mpg source (generated together with the d2v index file, in the 1_index folder) into AvsPmod and run it from there. This gives you meaningful error messages with the file and place where where they happen.

HTH,
fp

[GlemGem]

Happy New Year everyone!

fpp, thanks so much for your help, it's appreciated!

I've downloaded your script and corrected the paths etc.. Unfortunately I have still the same problem with HCenc, it crashes. :(

As suggested I checked the script file generated in 1_index folder and it works in AvsPmod, I get no errors. When HCenc open I got this http://oi45.tinypic.com/2vmva1f.jpg window and then it crashes, see: http://oi50.tinypic.com/6hnpcg.jpg

I wish I could give you a meaningful error message but that's all I get, and as I said, when I run the script from 1_index folder in AvsPmod there are no errors, it runs fine wihout any error.

Any idea guys?

[fpp]

Hmm, nasty one that... nice error message in German :-)
Hope it's a not a problem between HCenc and your system...

Could you post your HC.ini from the HCenc folder ?
Also, did you try running HCgui_xx.exe manually ? Tell it to load your .d2v index and generate a .m2v mpg file.
This will be exactly the same as your source, but at least it would confirm HCenc runs all by itself. If so, the problem would be between AVIsynth and HCenc...

[GlemGem]

Hi, the german error message basically says that the mpeg 2 encoder does not work anymore.

Yes, I've just tried running HCgui manually but same thing.

HC.ini file:

*INFILE c:\super8salvage\source\1_index\my_dvd_sample.avs
*OUTFILE C:\Super8Salvage\source\my_DVD_sample.m2v
*LOGFILE C:\Super8Salvage\source\0_titles\1-Durban.log
*BITRATE 5500
*MAXBITRATE 9500
*FRAMES 0 598
*PROFILE best
*ASPECT 4:3
*AUTOGOP 15
*PROGRESSIVE
*CLOSEDGOPS
*INTRAVLC 2
*MATRIX mpeg
*CPU ssse3
*PRIORITY high

[fpp]

Are you VERY sure you tried HCgui with source\1_index\my_dvd_sample.d2v as the input file, NOT source\1_index\my_dvd_sample.avs as in the INI file above ?

If so, the problem is definitely with HCenc itself, not with AviSynth or the scripts or anything else.

You need to stay in HCgui and try changing the numerous parameters in the three Settings tabs, one by one, to see if one of them is causing the problem. I would start with "CPU auto" in Settings 3.
Or look for help in a HCenc - specific topic ?

[GlemGem]

fpp, you are my Hero!!!

It works finally. I've changed to "CPU auto" in Settings 3 and everythig is fine now.

:thanks:

[fpp]

Aha, good for you !

If I had remembered my own "IMPORTANT WARNING" in post #16 earlier, we could have saved a lot of time :-)

[johnmeyer]

I finally had a reason to use this script. The SRestore does a nice job recovering the original film, and the deflicker absolutely nails the flicker ...

... but ...

the deflicker also introduces terrible ghosting in high-motion areas. Here is a still shot of the output of SRestore:

http://i177.photobucket.com/albums/w208/johnmeyer/Original-4_zpsba4fbc09.jpg

And here is the same frame after it has been processed by the deflicker averaging and masking:

http://i177.photobucket.com/albums/w208/johnmeyer/Deflickered_zpsae4c3de3.jpg


Just to make sure I didn't screw something up, here's the version of the script I'm using:

my_frame_rate = 24
width = 720
heigh = 480

AVISource("e:\fs.avi").killaudio().ConvertToYV12(interlaced=true)
assumeBFF().assumefps(29.97)

# remove combed frames by Didée
bob(-.2,.6)
SRestore(frate=my_frame_rate)
requestlinear()

rest = last

calm = rest.temporalsoften(1,255,255,32,2).merge(rest,0.25)
calm = calm.temporalsoften(1,255,255,16,2).merge(calm,0.25)
LOP = calm.bicubicresize(width/4,height/4).bicubicresize(width,height,1,0)
HIP = rest.bicubicresize(width/4,height/4).bicubicresize(width,height,1,0)
HIP = mt_makediff(rest,HIP)
mix = LOP.mt_adddiff(HIP,U=2,V=2)
sup1=rest.msuper(levels=1)
sup2=mix.removegrain(11).msuper()
bv2=sup2.manalyse(isb=true, delta=2,blksize=16,overlap=8,DCT=5)
bv1=sup2.manalyse(isb=true, delta=1,blksize=16,overlap=8,DCT=5)
fv1=sup2.manalyse(isb=false,delta=1,blksize=16,overlap=8,DCT=5)
fv2=sup2.manalyse(isb=false,delta=2,blksize=16,overlap=8,DCT=5)
rest.mdegrain2(sup1,bv1,fv1,bv2,fv2,thSAD=640)

output =ColorYUV(autogain=true, autowhite=true).deflicker()

return output



Any ideas of what I should change?

edit: looks like thSAD value is at least part of the issue ...

[johnmeyer]

thSAD was the entire problem. I played around with it and found 200 was a good value for this particular film.

[fpp]

Thanks for the Q&A John, now we know where to look first when that happens :-)

[szabi]

Hi

I tried to use this restoring process, but problem came up.
My script:
SetMemoryMax(512)
SetMTMode(5, 4)
DGDecode_mpeg2source("D:\Karcsiba_mix\VTS_01_1.d2v", info=3)
ConvertToYV12(interlaced=true)
SetMTMode(2)
ColorMatrix(hints=true, interlaced=true, threads=0)
QTGMC( Preset="Slow", EdiThreads=2, Sharpness=1.2, SLMode=2 )
SelectEven()

SRestore(frate=25)
requestlinear()

o=last
maxstabH=20 maxstabV=20 # maximum values for the stabiliser (in pixels) - 20 is a good start value
est_left=40 est_top=40 est_right=40 est_bottom=40 est_cont=1.4 #crop and contast values for special Estimate clip
stab_reference= o.crop(est_left,est_top,-est_right,-est_bottom).tweak(cont=est_cont).MT_binarize(threshold=80).greyscale().invert()
mdata=DePanEstimate(stab_reference,trust=1.0,dxmax=maxstabH,dymax=maxstabV)
DePanStabilize(o,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=0,mirror=15)#.deflicker() <-- twice the encoding time on my machine !

rest=last
calm = rest.temporalsoften(1,255,255,32,2).merge(rest,0.25)#.repair(rest,1).removegrain(11)
calm = calm.temporalsoften(1,255,255,16,2).merge(calm,0.25)
LOP = calm.bicubicresize(180,144).bicubicresize(rest.width, rest.height,1,0)
HIP = rest.bicubicresize(180,144).bicubicresize(rest.width, rest.height,1,0)
HIP = mt_makediff(rest,HIP)
mix = LOP.mt_adddiff(HIP,U=2,V=2)
sup1=rest.msuper(levels=1)
sup2=mix.removegrain(11).msuper()
bv2=sup2.manalyse(isb=true, delta=2,blksize=16,overlap=8,DCT=5)
bv1=sup2.manalyse(isb=true, delta=1,blksize=16,overlap=8,DCT=5)
fv1=sup2.manalyse(isb=false,delta=1,blksize=16,overlap=8,DCT=5)
fv2=sup2.manalyse(isb=false,delta=2,blksize=16,overlap=8,DCT=5)
rest.mdegrain2(sup1,bv1,fv1,bv2,fv2,thSAD=640)

output =ColorYUV(autogain=true, autowhite=true).deflicker()
return output

RemoveDirtMC(40,false)
sharpen(0.4).sharpen(0.4).blur(0.2)
AssumeFPS(25)
crop(6, 8, -6, -8)
LanczosResize(336,256)

Failure message is the following:
Avisynth open failure:
Script error: there is no function named requestlinear

I copied all files to avisynth plugin directory from The power of Avisynth: restoring old 8mm films (http://forum.doom9.org/showthread.php?p=1237054#post1237054) and The power of Avisynth: salvaging "botched" transfers of old 8mm films to DVD (http://forum.doom9.org/showthread.php?p=1505175#post1505175) topics.

What is wrong?

bye
szabi

[szabi]

I disabled the function "requestlinear()".
Now the script can be loaded, but after return output nothing works.
It means these do not work:
RemoveDirtMC(40,false)
sharpen(0.4).sharpen(0.4).blur(0.2)
AssumeFPS(25)
crop(6, 8, -6, -8)
LanczosResize(336,256)

I do not unerstand why?

EDIT: If they are copied before "return output" they are active, only RemoveDirtMC can not be used, because no function named failure message, even "03_RemoveDirtMC.avs" file is copied to plugin directory.

[szabi]

Ok, something is wrong here, with calling some functions.
If RemoveDirtMC function is inserted to script, it is working.
Sorry for wasting the time. :cool:

[szabi]

Anyway this is the outcome salvage+qtgmc, 2nd picture is the result of qtgmc only, 3rd one is the source.
http://abload.de/img/106759_qtgmc_salvu6u0r.png http://abload.de/img/106759_qtgmcskut6.png http://abload.de/img/106759_origiyfys1.png

I have a feeling something is missed, due to others got better outcome.

[StainlessS]

even "03_RemoveDirtMC.avs" file is copied to plugin directory.

Only *.avsi (and [EDIT: avisynth] dll's) files are auto loaded in plugins directory.

[poisondeathray]

I disabled the function "requestlinear()".
Now the script can be loaded, but after return output nothing works.


Not sure if your sorted this out yet, or if this is part of the reason for your results, but RequestLinear() is part of TIVTC.dll
http://web.missouri.edu/~kes25c/

Put it in the plugins folder or load it manually with LoadPlugin()

[szabi]

Thanks a lot, both of you were right.
Renamed to 03_RemoveDirtMC.avsi and copied of TIVTC.dll to plugin direrctory helped.

[szabi]

Hi

I am testing it again on a sort clip.
The original and the new one has different lenght.

Original clip has 25fps / 414frames / 16.56sec lenght.
Bob(-.2,.6) 50fps / 828frames / 16.56sec lenght.
SRestore(frate=18) 18fps / 298frames / 16.55sec lenght.
AssumeFPS(25) 25fps / 298frames / 11.92sec lenght.

Is it ok?

And the stabilization makes picture error:
http://abload.de/thumb/stabilization_issuepikim.jpg (http://abload.de/img/stabilization_issuepikim.jpg)
The effect can be seen top of picute and left side.

bye
szabi

[JClement]

I have a botched transfer and have tried to extract low flicker segments with telecide and fdecimate. While it works for some sections, other have serious flicker on only parts of the picture. Since the frame rate of NTSC is nominally 30, but the projector had been adjusted to around 14fps. So there are a series of 4 almost identical frames followed by 2 pairs of interlaced, noninterlaced frames and then an interlaced followed by 2 identical frames. The big problem is that the almost identical frames have grossly different intensity with one of them oversaturated. The interlaced frames also seem to be oversaturated.

What I think would work is to discard all interlaced segments and the oversaturated frames of a pair. This would reduce it to the original frame rate, whatever that was, and also greatly reduce the flicker. So does anyone have a suggestion how this might be accomplished???

[johnmeyer]

You have to post 5-15 seconds of the source footage. You need to use a tool which simply cuts, but does not re-encode. If you re-encode during the cutting process, the resulting clip will not be useful.

Without a sample, it is impossible to know what is going on, and therefore what settings to suggest. Also, the script that Didée created and which was the basis for this thread, made certain assumptions about how the film was transferred. Those assumptions are going to be different for each and every transfer and, at the very least, will require significantly modifying the settings in the script, and possibly changing some of the script operations entirely.

[poisondeathray]

@szabi - RE: stabilization "error"

Those are mirrored edges. When you stabilize, you will be left with "black bars" on the peripheral borders. The options are to leave them black, or zoom in , or fill them with something (e.g. mirror edge). Other programs have some other options to attempt to fill (e.g deshaker has fill from past & future frames). None of these approaches are "perfect", all have problems

[johnmeyer]

poisondeathray: this thread was "necro'd" by JClement, and you're answering a question from an earlier post ("szabi") that is from almost four months ago ...

[poisondeathray]

crap, I always get suckered in ... should have checked the date

[johnmeyer]

crap, I always get suckered in ... should have checked the dateDon't feel bad; I've done it many times myself. I only posted because I didn't want you to waste any time doing something to research the problem further.

[gdemonta]

First, thank you so much fpp for the work you did: very quickly I was able to restore some video I had transfered in 2008 but had never reworked since. The stabilization and scratch removing filters are amazing!
To explain the background: I transfered 8mm film on a video camera in interlaced SD 25fps (HD was to expensive at the time). The projector ran at 16.67fps to avoid flickering, and the camera recorded at 25 fps.

The image was inverted so I had to flip it.
I don't know if it comes from windows 10, but I couldn't make dgpulldown nor dgenc to work. So I just ran your sample bat to create the d2v, and then launched the sample script in virtualdub, flipped and cropped my video and compressed the lot in avi (I don't need to burn it on dvd). I just changed the framerate in your script from 19.75 to 16.67, and also changed the assumeframerate at 16.67 at the end.

Small glitch: on panning scenes, the deinterlacing is a little strange, I'll post a video soon.

Again, thank you so much for your work, I was so happy I didn't have to edit the script so much!

[johnmeyer]

If your original transfer is not "frame accurate" (only one frame of film on each frame of video, without blending or repeats), then none of these scripts will work correctly.

[edit]Woops, I didn't realize this was the thread about recovering individual frames from non-frame accurate transfers. My mistake.

[fpp]

Glad that this old thread of mine has been of help ! Although your use case is different enough that it might need some tweaking...

[sventamyra]

Am I the only one having trouble downloading the "full package" in post#21?

Seem to be a dead link.

I was looking fwd to using this for my dads old super8 films of my youth

[StainlessS]

Post #21 was from 2011, some time ago, everybody involved is probably now well dead, you can throw out invites to the dead if you like but, dont know what kind of answer you may/may not get. Good luck on that.
Dead links are rife, here on the D9. Currrent links [EDIT: attachments] dont even get authorised, (cant remember last one that did).

[johnmeyer]

I assume the OP won't mind. Here is the scripts-only download:

Super8 Projector Film Recover From Bad Transfer Salvage_scripts_only_3.zip (http://www.mediafire.com/file/xgdu1p5jnnuagaa/Super8%20%20Projector%20Film%20Recover%20From%20Bad%20Transfer%20Salvage_scripts_only_3.zip)

Sorry, I don't have the download that includes all the filters.

[fpp]

[ Cue "Monthy Python Intro" ]... "I'm not dead yet ! Feeling much better, actually !" :-)

Just stopped using Dropbox somewhere along the way... Thanks for filling in, John !

This is bordering on archaeology, and I'm not sure I'd understand what I was doing then if I tried :-)

But I do still have the files on disk, so here are new links for the two archives :

Super8Salvage_full_pack_3 (https://monbo.net/upndown/f.php?h=05Zlnxzf&d=1)

my_DVD_sample (https://monbo.net/upndown/f.php?h=0OHTmU56&d=1)

The host has 1-month retention time, so these will be available until Sat Apr 14 2018 11:49:12 GMT+0200.
(feel free to ping from the grave if renewals are needed :-)

Good luck sventamyra!

[StainlessS]

Hi fpp, glad that you are now amonst the undead :)

Here hosted on my MediaFire account (easy and free to set up with my current limit of 54GB, only about a couple of hundred MB used so far):

Super8Salvage_full_pack_3(~9.8MB):- http://www.mediafire.com/file/efei57aa8ifabu7/Super8Salvage_full_pack_3.zip

my_DVD_sample (~ 2KB):- http://www.mediafire.com/file/nz0t4hdv8wn83eb/my_DVD_sample.zip

No expiry date.

[fpp]

Thanks much ! Now I can serenely wait out the next 7 years, until someone stumbles across these ancient scrolls :-)

[sventamyra]

Good luck sventamyra!
Thank you for the recovery!
Despite the comments, I assum that scrips and Avisynth vill work just fine still, eventhough its been seven years. Or am I missing something?

[johnmeyer]

If you use the plugins he provided and the version of AVISynth he used, it will work the same. The software has no idea that seven years have gone by.

Heck, I still use dBase III every day, and it was written in 1984 for an IBM XT with a 4.77 MHz processor. It still serves me well as a basic database program.