The power of Avisynth: restoring old 8mm films.

[2Bdecided]

Originally Posted by videoFred

The artifacts created by MFlowFPS() are way less when using RemoveDirtMC() before. MFlowFPS() is great for low motion scenes. Nice smooth panning scenes etc. On high motion scenes it fails sometimes.

That's my experience too.

The artefacts are less noticeable if you (for example) double the frame rate using MFlowFPS (so you still keep all the original frames), and then changefps or pulldown from that to whatever you really want. Of course it's not as smooth.

There was a script (can't remember its name - I think it was in the Alchemist thread) that adaptively switched between motion compensation and blending (block-based), depending on the complexity of motion (I think). Unfortunately that added block-based artefacts.

Cheers,
David.

[videoFred]

Originally Posted by 2Bdecided

There was a script (can't remember its name - I think it was in the Alchemist thread) that adaptively switched between motion compensation and blending (block-based), depending on the complexity of motion (I think).

This is something I was thinking of myself. But would it not be better to switch between motion compensation and simple frame duplicating?

Fred.

[johnmeyer]

I decided to try some of the things manono has been suggesting. Here is what I found.

ChangeFPS(59.94)
AssumeTFF() # or BFF, as required
SeparateFields()
SelectEvery(4, 0,3)
Weave()
AssumeFPS(29.97, true)

This is from one of the links in manono's posts. This produces correct pulldown because, while ChangeFPS does indeed duplicate entire frames, by using it to go to twice the final framerate and then decimating with SelectEvery, we end up with the desired 4 3 3 pulldown pattern. As he described in the linked thread, this technique can be used for any starting framerate (15, 16, 18, etc.). So, this is good stuff. Thank you!

I am doing some tests with DGPulldown, and am less convinced as to whether this is going to produce a useful result. I'll create a test DVD and see what happens.

P.S. I did try encoding using the "ChangeFPS(19.98)" script, but this (as I expected) changes from 18 fps to 19.98 fps by duplicating entire frames. Given this, I don't see how DGPulldown is going to give me a smooth looking result.

[fabioseixal]

I've got a silly question. In your video hosted at vimeo, on the 6 second is that a bird that have been removed? xD
And on the 26 second that mosquito appeared from nothing.
At the end of this idiocy of mine, congratulations, very, very good work that you've acomplished.
Sorry for my bad english...

[videoFred]

Originally Posted by fabioseixal

I've got a silly question. In your video hosted at vimeo, on the 6 second is that a bird that have been removed? xD
And on the 26 second that mosquito appeared from nothing.
At the end of this my congratulations, very, very good work.
Sorry for my bad english...

You have a good eye

Yes, the bird is gone and the mosquito appears from nothing. RemovedirtMC has done this. Perhaps I should have set the RemoveDirt strength a bit lower on these scenes..

Fred.

[manono]

Originally Posted by johnmeyer

I decided to try some of the things manono has been suggesting.

You mean Gavino, not me. His is a more elegant way of hard telecining it.

P.S. I did try encoding using the "ChangeFPS(19.98)" script, but this (as I expected) changes from 18 fps to 19.98 fps by duplicating entire frames.

Of course it does, roughly one dupe frame in every 10. Now, play your output in a hardware or software DVD player and figure the percentage of duplicate frames.

And you haven't ever addressed my main points. I contend that yours will be at least as jerky playing, but having purposely interlaced it, which requires a lot more bits for the same quality as progressive encoding, and with 50% more encoded frames, the output for the same filesize will also be of considerable poorer quality, probably something like 40% poorer quality as measured by the average quant.

[johnmeyer]

And you haven't ever addressed my main points.

Well if the basic point you are making is that duplicating frames will produce better "quality" measured by average quant, I agree that is true. It is also, however, not the only measure of quality and in fact is a measurement that will lead you into doing the wrong thing.

The problem with any sort of pulldown is that it introduces stutter (which is different than judder, the artifact you get on film when the camera pans horizontally). Stutter is where you can actually perceive the motion starting and stopping as frames are duplicated. The problem is that when you duplicate an entire frame (rather than duplicating only one field, a process which results in half the temporal disturbance), you introduce a HUGE discontinuity in the temporal cadence of the resulting video. This is a really bad thing and is visually very disturbing. I see it frequently on commercial broadcasts of theatrical films that have been (badly) decimated to fit a two hour time slot. I also have seen it in many NTSC->PAL and PAL->NTSC conversions that people have given to me to rescue. As people in the TFM threads have discovered, there is some amazingly bad stuff that gets done to video.

The amount of visual disturbance any of these bad conversions create is a function of how often these discontinuities occur. The less frequent they are, the worse, because they are not part of the flow and when they happen during fast motion, they stick out like a sore thumb (for example, the 19.98 video I created in my failed attempt to use DGPulldown looked absolutely horrible because it repeats one entire frame, but only every ten frames). By contrast, the difference in encoding quality between encoding purely progressive video (with entire frames duplicated), and encoding interlaced video with pulldown fields is going to be noticed as a slight increase in encoding artifacts (mosquito noise, blockiness at transitions, etc.). At reasonably high bitrates, with a professional encoder using multi-pass encoding, the difference in these encoding artifacts can be so minor as to be unnoticeable.

In addition, I think that underlying your comments is the belief that interlaced is somehow "worse" than progressive. It is true that interlaced presents additional challenges when editing, and it is also unfortunately true that many of the modern television sets cannot display it natively (something that does not have to be that way and which I hope will change), but interlaced video looks fantastic, unless you try to freeze an individual frame.

Oh, and as I mentioned in passing above, I tried to follow your suggestion to encode a 19.98 AVI file (resulting from ChangeFPS(19.98) to a 19.98 MPEG-2 file and then apply DGPulldown to the result. However, I could not get Vegas, the standalone MainConcept MPEG-2 encoder, or TMPGEnc to encode a 19.98 MPEG-2 file.

[Gavino]

Originally Posted by johnmeyer

I did try encoding using the "ChangeFPS(19.98)" script, but this (as I expected) changes from 18 fps to 19.98 fps by duplicating entire frames. Given this, I don't see how DGPulldown is going to give me a smooth looking result.

The DGPulldown method is well proven and should produce identical results to hard pulldown when played back on a conforming player. However, for DGPulldown the input framerate must be no lower than 2/3 of the playback rate (I think this is because the MPEG flags can only produce a maximum of 3 fields per frame, ie up to 3:3 pulldown). So for playback at 29.97 fps, the minimum input rate is 19.98.

Manono's solution is to use ChangeFPS to get from 18 to 19.98 and then use DGPulldown to go the rest of the way. It seems to me this will produce, instead of the optimal 4:3:3 pattern, something like 6:6:3:3:...:3, because of the repeated frames resulting from the initial ChangeFPS.

Are you sure this works, Manono?

Edit: Actually, what might work is to combine 'my' hard pulldown (instead of a simple ChangeFPS) to get to 19.98 before using DGPulldown:
ChangeFPS(39.96)
SeparateFields()
SelectEvery(4, 0,3)
Weave() # is now 19.98

This would allow to encode the smallest number of frames while still getting as smooth playback as possible.

[johnmeyer]

Originally Posted by Gavino

It seems to me this will produce, instead of the optimal 4:3:3 pattern, something like 6:6:3:3:...:3, because of the repeated frames resulting from the initial ChangeFPS.

That's what I thought, and is what I was seeing when I actually did it, and then used "separatefields()" to analyzer the resulting pattern. And, since 6:6:3:3 ....:3 is repeating frames instead of fields, this is what I was trying to avoid in order to minimize the stuttering artifact.

The method that seems to hold the best promise (at least for my work) for 16 and 18 fps material is to use MFlowFPS to go from either of these two framerates to 24 fps progressive. This reduces the number of interpolations that MVTools must do (compared to having it interpolate all the way to 59.94 interlaced, i.e., 60i). Then, encode this 24 fps as 23.976 progressive MPEG-2 with the pulldown flags set, something that any MPEG-2 encoder can do. I've test this, and for most material it works well. You get the clean encoding that manono talked about; and you produce a DVD that can either be viewed as 29.97 with traditional 3:2 pulldown (which will be supplied by the DVD player) or as 24 fps progressive, if the DVD player and TV set can ignore the pulldown flag and display the material as it was encoded.

For this application, I have found that using blksize=16 (for the MAnalyze function that precedes MFlowFPS) reduces the "broken leg" artifacts that MFlowFPS often produces. I call it this because you see the problem when people walk in front of the camera. Depending on the distance from the camera, their legs will appear to warp ("break") in mid stride. With the larger block size, this is reduced. Since 8mm and Super8 (and even most 16mm film) doesn't have the detail of HD video, this larger block size seems to work well for this application without much downside compared to blksize=8.

P.S. I saw that Gavino was posting at the same time I posted. That code looks more promising, but I still don't see how I can encode a 19.98 MPEG-2 file, which I think is what is required in order to have the proper input to DGPulldown. And, I'm still not sure whether the output of DGPulldown can be used as the input to a DVD authoring tool. But, I'm willing to try to see what it look like. Any advice on how to encode 19.98 MPEG-2 would be appreciated (or, if I am not understanding correctly, a gentle nudge in the right direction ...)

[Boulder]

Originally Posted by johnmeyer

For this application, I have found that using blksize=16 (for the MAnalyze function that precedes MFlowFPS) reduces the "broken leg" artifacts that MFlowFPS often produces. I call it this because you see the problem when people walk in front of the camera. Depending on the distance from the camera, their legs will appear to warp ("break") in mid stride. With the larger block size, this is reduced. Since 8mm and Super8 (and even most 16mm film) doesn't have the detail of HD video, this larger block size seems to work well for this application without much downside compared to blksize=8.

You might also want to try using overlap of blocksize/2. It should help the motion compensation process.

[johnmeyer]

Originally Posted by Boulder

You might also want to try using overlap of blocksize/2. It should help the motion compensation process.

I tried various overlaps, including blocksize/2, but as far as artifacts, using more overlap didn't help.

[manono]

Originally Posted by johnmeyer

Well if the basic point you are making is that duplicating frames will produce better "quality" measured by average quant, I agree that is true.

Your method at output also produces dupe frames, and many more of them.

The problem with any sort of pulldown is that it introduces stutter

3:3 pulldown (19.98->29.97) introduces no stutter at all. The only stutter is from the original dupe frames.

The problem is that when you duplicate an entire frame (rather than duplicating only one field, a process which results in half the temporal disturbance),

If played to a progressive display (computer monitor, HDTV), you're playing frames. And your frames will all have been deinterlaced where with progressive encoding you're playing the original progressive frames. You're at the mercy of the deinterlacer in the player or display. It might be good; it might not. Nor do I see the advantage of 4 duplicate frames in every 10 at playback over having just 1 dupe in 10. The difference is only noticeable during certain kinds of movement and, in my opinion, the quality gain from progressive encoding far outweighs any slight stutter. After all, we're NTSC and are so used to 3:2 pulldown jerkiness that we don't even notice it.

However, I could not get Vegas, the standalone MainConcept MPEG-2 encoder, or TMPGEnc to encode a 19.98 MPEG-2 file.

You can only encode at valid MPEG-2 framerates. Add an AssumeFPS(23.976) to the bottom of the script and when done run it through DGPulldown set for 19.98->29.97. CCE will accept the script as-is, but still requires 23.976fps encoding.

I ran some tests on both methods and have to modify something I said earlier because your script doesn't have as high a percentage of interlaced frames as I thought. Rather than your script producing 40% poorer quality, it's more like 30-35%. First I encoded 1000 frames using both scripts at constant quality (quant) in CCE. The progressive script produced a file of 35.7 MB and the interlaced script of 49.1 MB, a size difference of 35.7%. Then I did encodes for the same file size and the average quant as measured by BitRate Viewer was 6.08 for the progressive script and 7.63 for the interlaced script. I say these are significant differences.

Originally Posted by Gavino

Are you sure this works, Manono?

If I've done this once for sub-23.976 framerates, I've done it a hundred times.

Originally Posted by Gavino

Edit: Actually, what might work is to combine 'my' hard pulldown (instead of a simple ChangeFPS) to get to 19.98 before using DGPulldown:

You can't apply pulldown to interlaced output.

[manono]

More notes:

Originally Posted by Gavino

It seems to me this will produce, instead of the optimal 4:3:3 pattern, something like 6:6:3:3:...:3, because of the repeated frames resulting from the initial ChangeFPS

Yes, with this 3:3 pulldown each frame is repeated 3 times, and the dupes get repeated 6 times (lasting roughly one-tenth of a second in total). Your method, when bobbed for a 60 Hz progressive display, has 4 times as many dupe frames, and they're repeated twice. Four times as many dupe frames, each repeated fewer times. More but less noticeable stutter, either kind only noticeable at all during certain kinds of movement. Look, almost all DVDs with a 'base' framerate of less than 23.976 (silent films, home or cheaply made movies of different sorts) have used something similar to your script to get it to 29.97fps. I'm saying that's no longer necessary when there's a way to keep it progressive all the way through. When encoding for a DVD5 (as I do all the time as DVD9s are quite expensive where I am), and where bitrate is a consideration, and given the max bitrate of 9800 for DVD, often not high enough for interlaced encoding, interlaced encoding is sub-optimal, in my opinion.

About encoding at noncompliant framerates: Because you'll be encoding at 23.976fps rather than at 19.98fps, you have to make adjustments in the bitrates to get the file sizes you want. You multiply each by a factor of encoded framerate/'real' framerate. In this example, 23.976/19.98=1.2. You multiply each bitrate by a factor of 1.2. If, for example, you use min-ave-max bitrates of 500-4000-9000, then adjust to 600-4800-10800. Everything gets straightened out after the pulldown is applied

And if johnmeyer wants to try MFlowFPS on something like this, he might consider converting it to 19.98fps, rather than 23.976fps or 29.97fps, or whatever he's been trying. Fewer frames get interpolated and thus fewer artifacts. And by applying 3:3 pulldown after the encoding, there'll be no pulldown judder at all.

[Gavino]

Originally Posted by manono

If I've done this once for sub-23.976 framerates, I've done it a hundred times.

I wasn't doubting the method itself. When I asked 'does it work?', I was referring to whether it could produce a 4:3:3 pattern. Your answer seems to be "no it can't, but it doesn't really matter". I won't argue with that - I know you have a lot of experience with this sort of thing, while my knowledge is entirely theoretical.

You can't apply pulldown to interlaced output.

Right. Good point!

[manono]

Originally Posted by Gavino

I wasn't doubting the method itself. When I asked 'does it work?', I was referring to whether it could produce a 4:3:3 pattern. Your answer seems to be "no it can't, but it doesn't really matter".

Oh, I misunderstood. Sorry. Well, it could with a ChangeFPS(29.97). But that would defeat the purpose of keeping the encoded frames (and the stuttering) to a minimum.

I saw a novel way to do this which the Flicker Alley folks who restore and release silent films did. I'm a little foggy on the details now because it was a while back. If I remember correctly the 'base' framerate was 22 fps and they, instead of hard telecining it to 29.97fps, and rather than adding in duplicate frames to bring it up to 23.976fps so they could encode at that framerate progressively and add pulldown, added in blended frames to bring it up to 23.976fps, and then encoded it progressively with pulldown. It wasn't quite like what a ConvertFPS(23.976) does I don't think, because that seems to create more blended frames than they did. They created 2 new blended frames every second, with the original frames being left alone. If, for example, you were going from 18fps to 19.98fps, they would have had only 2 blended frames spread 10 frames apart created from the progressive frames on either side, with the rest the original progressive frames. You could probably figure a script for that. Although you might get a kind of strobing and blurring effect, the stutter wouldn't be anywhere near as jarring during certain kinds of movement.

[Didée]

No need to script that. For that purpose you can use BlendFPS from mg262's motion.dll. It has an "aperture" parameter to control the amount of blending ... aperture is related to the relative weight that a frame would get by ConvertFPS - if frame's weight is smaller than "aperture", no blending is used. Else, do normal blending.

[Gerald1937]

John

Quote:
I'm not sure what you mean by this. Aren't Blu-Ray blanks pretty cheap now?

I have no Blueray player, writer or disks. So yes the cost to me would be significant and probably unjustifiable. At 73 my friends and relations tend to have VHS tape or DVD players but few have Blueray players. So DVD is my currently preferred method of distribution used for SD transfers.

For distribution of the HD transfers I investigated: -
- Burn to Blueray disk. Currently too costly and the recipient must have a Blueray player.
- Burn a DVD disk compatible with the format and playable in a Blueray player. This works but needs a lower bit rate than that used by the HV20 tape transfer file. Cost effective but some degradation and again the recipient must have a Blueray player.
- Use Avisynth to frame serve MultiAVCHD to generate a SD/SDHC memory card MPEG4 file compatible with the Panasonic Viera SD requirements. Not tested yet. Is SD card a viable distribution media? Can all TV's read the same SD card video file?
-Burn a DVD data disk with the HV20 tape transfer file. The recipient or I copy the file to HDD or memory stick and they can play the file via USB2 to a Western Digital media box into their TV using the HDMI or other interface. The media box can apparently handle a wide range of input video file formats and types successfully. If the conversion quality is good/excellent then this could be the way to go.

The Cineform software package you quoted costs far more than my telecine system did to build (excluding my HV20 video camera). Does your IVTC method of capture work OK with the Cineform package? The BMP route I chose is simple and inexpensive, my current software program works for SD or HD and emulsion side or base layer capture. The original film transport system layout was mirrored when I changed to emulsion side capture.

If your film transport system repeatable positional accuracy of the film frame to frame is greater than zero pixels it produces a vertical displacement error of the same lines between adjacent frames. So any change of fps using fields from adjacent original frames to create an intermediate frame seems to me flawed because lines combined would be displaced from their desired position by the error. This is the beauty of MFlowFPS as it inherently computes the pixel positions for the generated frame from adjacent original frames, which includes the displacement error.

I agree MFlowFPS does introduce artifacts; your "broken leg" naming is appropriate. When you look at the source frames that cause such artifacts these are often blurred due to the slow 1/25 s shutter speed of the cine camera used and/or contra-movement of different objects in a frame and the adjacent ones. The other situation is at scene changes. My criteria for what to do when editing the transfer are as follows. If I cut the affected frames out will it spoil the flow of the scene? If I need to keep the flow of the scene can I substitute other frames for the affected ones, with the original frames only and accept the transient fps speed change or with frames generated from the originals normally using PPro speed change facility. You also need to consider the number of adjacent frames showing the artefacts and if there is a soundtrack involved with the transfer.

VideoFred

I like your restoration script very much and thank you for sharing it with us. Please can you explain the strategy you use for the stunning colour correction you achieve when using an Avisynth script? Have you an Avisynth script for generating the MPEG 4 uploads you make to the Vimeo site that you are willing to share with us?

Regards

Gerald

[johnmeyer]

Originally Posted by Gerald1937

I have no Blueray player, writer or disks. ... Can all TV's read the same SD card video file?

Given this, and given that your goal is to have something that all your friends can play, you have only two viable choices: burn to DVD, or get a Blu-Ray burner and burn to that. Any of the other options (like encoding to MPEG4 and saving on a memory device) is not guaranteed to play on all TV sets. Also, you need to decide whether your HD captures really will look much worse when encoded to DVD. Unless you are starting with 16mm film, I am pretty sure the loss will be minimal, if indeed it can be noticed at all. So, I recommend doing the best possible encode to DVD, using nothing but pulldown using the script code provided by Gavino. It will give you the sharpest, smoothest result, and it will play on equipment that everyone owns.

Originally Posted by Gerald1937

Burn a DVD data disk with the HV20 tape transfer file. The recipient or I copy the file to HDD or memory stick and they can play the file via USB2 to a Western Digital media box into their TV using the HDMI or other interface.

I can't believe that anyone who would have this equipment, and also have the technical skill to install and use it, wouldn't also have a Blu-Ray player. After all, Blu-Ray players are getting pretty cheap.

Originally Posted by Gerald1937

Does your IVTC method of capture work OK with the Cineform package?

I never tried it out because my initial tests, using HDV, convinced me that the gain in quality by capturing at 1440x1080 or 1920x1080 wouldn't be worth not only the financial cost, but the considerable cost in increased editing and rendering times.

Originally Posted by Gerald1937

If your film transport system repeatable positional accuracy of the film frame to frame is greater than zero pixels it produces a vertical displacement error of the same lines between adjacent frames. So any change of fps using fields from adjacent original frames to create an intermediate frame seems to me flawed because lines combined would be displaced from their desired position by the error.

I understand what you are saying, but I'm not sure the logic is quite correct. Remember that each frame has movement compared to the adjacent frames. That's the whole point of film and video: everything is moving! So, since things are moving anyway, the additional slight movement caused by gate weave won't matter. In addition, you have to remember that even if you can get each frame of film to register perfectly in the gate compared to its adjacent neighbors, you still will have gate weave. Why? Because the camera that took the film has the same problem! So, I would argue that you are solving a problem that cannot really be solved or fixed because half of it is already "baked into" the film.

Also, the errors that become visible when using motion estimation seem, in my experience, to come more from motion of small objects within the frame, or from things which move into the frame, close to the camera. If the whole frame moves, motion estimation does an amazing job of correcting for this. VideoFred's script shows what can be done, and indeed the Depan settings he has created pretty much completely eliminate the motion caused by gate weave.

As to any scan line errors, I guess you may be referring to the artifacts such as you get when bobbing interlaced material, problems on diagonal lines, and other things that happen when details jump back and forth across scan line or pixel boundaries. Again, this happens all the time not only for the reasons I mentioned, but also because most amateur 8mm, Super8, and 16mm film was shot hand-held, without the benefit of motion stabilization, and so is already bouncing all over the place.

[manono]

Originally Posted by Didée

For that purpose you can use BlendFPS from mg262's motion.dll.

Thank you for the suggestion, Didée. Ordinarily I shun blending like the plague, but for framerates below 19.98fps (the DGPulldown threshold for NTSC) where dupe frames have to be added, this works well. Although ChangeFPS(19.98) is fine for me, I guess because I'm not very sensitive to the added stutter, creating an 18fps source and then using:

BlendFPS(19.98,Aperture=0.2)

made it play silky smooth with the blending not very noticeable. The next time I have an appropriate source I'll give BlendFPS a try. Thanks again, and to mg262, because I wasn't aware of this filter.

[videoFred]

Originally Posted by Gerald1937

VideoFred

I like your restoration script very much and thank you for sharing it with us. Please can you explain the strategy you use for the stunning colour correction you achieve when using an Avisynth script?

It's the lack of color correction. The denoising helps a lot for better color reproduction and I increase saturation. I have added color correction in the script, but I seldom use it. However, the Avisynth auto-white "ColorYUV(autowhite=true)" can help sometimes on very 'bluish' colored films for example.

Let me explain further: As you might know, I'm using a machine vision camera for capturing. With a machine vision camera one can set the correct color balance according to the live histogram. The (RGB) histogram is included in the camera software. But I'm using a custom app. made by Frank Vine. He has custom made it for capturing film with machine vision cameras. It has a live histogram and lots of other options.

http://www.cine2digits.co.uk/

Please check 'Control and Capture' to see some screenshots from the software.

Resumed: I have barely to do any color corrections because the source has a correct (RGB)color balance to begin with.

Have you an Avisynth script for generating the MPEG 4 uploads you make to the Vimeo site that you are willing to share with us?

No special Mpeg4 script. It's the output of my filmscript. In VDub, I use Microsofts Mpeg4 codec, V3. One keyframe every second, highest bitrate. That's it.


Fred.