inverse telecine and de-interlace of 8mm film scan

[CaptainTivo]

Hi,
In 2012 I had a couple of 7" reels of standard 8mm scan by these guys. They scanned them into digital files and also a DVD.
The digital files are MJPEG interlaced in an avi container. I now want to try to clean them up using VideoFred's scripts. To do this I need to apply RemoveDirt and DeScratch . To use these filters I need to remove any duplicate frames and de-interlace. The telecine seems to be aabbccddeeffggh to convert from 16 FPS to 29.97 FPS. I don't know how the interlace works, but it does seem to be interlaced.
Here is a sample of the file: sample.avi

Maybe someone could take a look and give me some advice on how to do inverse telecine and deinterlace into 16 FPS progressive?
Thanks.

[johnmeyer]

I developed IVTC scripts for both 18 fps (Super 8) and 16 fps (common speed for 16mm and 8mm amateur silent). Yours has the standard pulldown for 16 fps, requiring removal of 7 frames out of every 15 frames. I walked through the clip, just to make sure, and then applied this simple two-line IVTC:

tfm(display=false)  
tdecimate(cycleR=7,cycle=15) #16 fps

And, just so you know for the future, inverse telecine and interlacing are two completely different operations. It is confusing, to be sure, because the act of adding telecine (duplicate fields or frames) does often result in the two fields in some frames being from two different points in time, just like you find in interlaced video. However, some frames have both fields from the same point in time.

Your sample simply repeats entire frames to achieve the goal of having 16 fps material play at the correct speed when shown on a device only capable of playing 30 fps (which is actually 29.97 fps, but that subtlety doesn't matter at all for non-sound silent film)

You can get slightly smoother playback on CRT devices when you repeat fields across frames rather than simply repeating frames, but since you're removing all the repeats in order to do your restoration with Fred's script, that doesn't matter.

Bottom line: you cannot use deinterlacing tools on your sample and hope to achieve any useful result. You must use IVTC, and TIVTC (where you'll find TFM/TDecimate) is the tool you should use.

[CaptainTivo]

Originally Posted by johnmeyer

I developed IVTC scripts for both 18 fps (Super 8) and 16 fps (common speed for 16mm and 8mm amateur silent). Yours has the standard pulldown for 16 fps, requiring removal of 7 frames out of every 15 frames. I walked through the clip, just to make sure, and then applied this simple two-line IVTC:

tfm(display=false)  
tdecimate(cycleR=7,cycle=15) #16 fps

And, just so you know for the future, inverse telecine and interlacing are two completely different operations. It is confusing, to be sure, because the act of adding telecine (duplicate fields or frames) does often result in the two fields in some frames being from two different points in time, just like you find in interlaced video. However, some frames have both fields from the same point in time.

Your sample simply repeats entire frames to achieve the goal of having 16 fps material play at the correct speed when shown on a device only capable of playing 30 fps (which is actually 29.97 fps, but that subtlety doesn't matter at all for non-sound silent film)

You can get slightly smoother playback on CRT devices when you repeat fields across frames rather than simply repeating frames, but since you're removing all the repeats in order to do your restoration with Fred's script, that doesn't matter.

Bottom line: you cannot use deinterlacing tools on your sample and hope to achieve any useful result. You must use IVTC, and TIVTC (where you'll find TFM/TDecimate) is the tool you should use.

Thanks for your input.
Now that I have studied your solution, I realize that the tfm filter does the deinterlace operation before the tdecimate removes the duplicate frames, resulting in a true progressive 16 FPS file.

Thanks for your time.

[johnmeyer]

TFM doesn't deinterlace. Instead it matches fields and then provides "hints" to TDecimate to tell it which fields are duplicates, from the same moment in time.

As I said in my earlier post, deinterlacing is related, but it is a different animal and you can't use deinterlacing tools to achieve inverse telecine.

[hello_hello]

Originally Posted by johnmeyer

TFM doesn't deinterlace. Instead it matches fields and then provides "hints" to TDecimate to tell it which fields are duplicates, from the same moment in time.

TFM is both a field matcher and a de-interlacer, but for the OP's sample de-interlacing is all it's doing. If you check the help file I think you'll find TFM has something like six different de-interlacing methods to choose from, or it'll take pixels to repair combing from another clip if you want to de-interlace with something else. For telecined film it matches fields to put the progressive frames back together, and de-interlaces when after the matching process it still detects combing. For interlaced video it de-interlaces to make it progressive. I can't remember exactly what's included in the "hints", but I think it's info such as the type of match used for each output frame and whether combing was detected etc to help TDecimate decide which frame(s) to drop in a cycle, or if the frames in a cycle are video rather than film. TDeint can also read TFM's hints.

CaptainTivo,
As best as I can tell looking at your sample, IVTC isn't required as it's not telecined. There's definitely what appears to be combing, but TFM is doing nothing but de-interlacing, so in that respect you could de-interlace with your de-interlacer of choice and follow it with TDecimate. However....

While I know nothing about digitising old films and how it's done, it looks to me like one of the fields is shifted by something like half a scanline, causing the combing. Maybe that's something to do with the MJPG format and the way fields are encoded. I know nothing about that either.

Try the following (edited to fix pasting PointResize onto the wrong line)

SeparateFields()
A = SelectEven()
B = SelectOdd().PointResize(1920,540, 0,-0.5,1920,540)
Interleave(A,B)
Weave()

Follow it with TFM this way, and for your sample at least, you'll see it isn't needing to do much.

TFM(Display=true, PP=1)

The trouble is though, I either didn't get the shift 100% correct or it's not consistent (the fade in at the beginning is mostly the culprit), so for some frames there's very fine "combing" TFM doesn't detect unless you make it more sensitive. If you use TFM on it's own it'll be de-interlacing virtually every frame when, in theory at least, it shouldn't need to, but after shifting a field if you make the combing detection too sensitive it'll still do the same, so I played around a little and the following seemed a reasonable compromise. TFM should leave most frames untouched while still de-interlacing where any residual combing might be noticeable.

SeparateFields()
A = SelectEven()
B = SelectOdd().PointResize(1920,540, 0,-0.5,1920,540)
Interleave(A,B)
Weave()
TFM(Metric=1, MI=50)
TDecimate(CycleR=7, Cycle=15)

[johnmeyer]

I do film transfer for a living, and have used every single TFM mode. The previous post, while well-meaning, is wrong.

Deinterlacing is NOT inverse telecine.

The statement "de-interlacing is all it's doing" is wrong. The sample is 100% progressive. There is not one frame which has its two fields from different moments in time, which is the definition of interlacing. Instead, as I stated in my long post, the company that did his transfer performed the telecine (a.k.a. "pulldown") by simply repeating entire frames.

Therefore, there is not one frame which is interlaced, and de-interlacing will fail to do anything.

It is important to be clear on this because so many people already have incorrect views about interlacing, and this just adds to it.

As for field shifting, I didn't notice that, but I suppose it is possible. I'd have to go back to the original clip, deconstruct it into individual fields, and then look to see if things line up.

[poisondeathray]

In this specific sample , all fields are already matched (all frames already have their matched field pair) - technically you don't need to field match. But on a longer section , some fields might be out of sequence, so it does not "hurt" to include TFM in case you need to field match

By default, TFM potentially deinterlaces as a post processing function when detection thresholds are met. If you use TFM(PP=0), you can check with deinterlacing 100% disabled

TFM is deinterlacing this sample, when using FFMS2 as the source filter , otherwise there would be no differences detected by this script

FFVideoSource("sample1.avi")
TFM()

a=last

FFVideoSource("sample1.avi")
TFM(pp=0)

b=last

Overlay(a,b , mode="Difference", pc_range=true)
Levels(127, 1, 129, 0, 255, false)

It might be that some of you have different observations based on different decoders.

Some MJPEG en/decoders often have fields swapped, some automatically adjust for this. You can use swapfields() in avisynth if your mjpeg decoder does not have that toggle option. If you check with swapfields, it's clearly better, and this is the likely the explanation. But there is still residual combing. This can occur from interlaced encoding compression artifacts, and also from a lighting mismatch from the sample times . 16 is not evenly divisible into 59.94 fields/s and the fades are worse as expected (one field has a different brightness than it's partner field)

I would finish with vinverse2 for the residual combing. You don't lose fine detail because this is a very soft upscale to begin with. 8mm does not need 1080 lines; it's really overkill

FFVideoSource()
SwapFields()
TFM(pp=0) #just in case
TDecimate(CycleR=7, Cycle=15) 
Vinverse2()

Note, that FFVideoSource returns the original full range 0-255 YUV 422 mjpeg data . This is the original storage format. If you use AVISource, some decoders may range compress the output automatically, others might return RGB

[StainlessS]

PDR Masterclass

[hello_hello]

Originally Posted by johnmeyer

I do film transfer for a living, and have used every single TFM mode. The previous post, while well-meaning, is wrong.

Deinterlacing is NOT inverse telecine.

I never said it was, but TFM can and does de-interlace. If you have a hybrid DVD source (combinations of telecine and interlaced video), it's TFM de-interlacing the video sections. It's referred to as post processing in the help file (pp) but repairing combing is the definition of de-interlacing. Under PP mode the help file says:

"You can also use the "clip2" parameter to specify an externally deinterlaced clip from which TFM will take frames instead of doing the deinterlacing itself."

For Clip2 it says:
"Don't specify anything for clip2 to use TFM's internal deinterlacing."

Originally Posted by johnmeyer

The statement "de-interlacing is all it's doing" is wrong. The sample is 100% progressive. There is not one frame which has its two fields from different moments in time, which is the definition of interlacing. Instead, as I stated in my long post, the company that did his transfer performed the telecine (a.k.a. "pulldown") by simply repeating entire frames.

Therefore, there is not one frame which is interlaced, and de-interlacing will fail to do anything.

If you try TFM(Display=true) you'll see the word "deinterlaced" used for almost every frame in the sample.
TFM(Display=true, pp=1) shows the match used is almost always "c" which means:
"the field from the current frame and of the opposite parity to that of the field being matched from is used"
So it's not shuffling fields around to create progressive frames because there's no better match. It's failing to find a better match, detecting the combing and de-interlacing to repair it. I can't imagine what else you think is happening to produce uncombed frames, given you're sure only whole frames are duplicated and the video is progressive. Unless the fields are "out of phase", why would field matching normally be required when there's no unmatched fields created by duplicating whole progressive frames?

[johnmeyer]

Agree on most points. TFM does field matching, and part of that means that it can sense when fields are from the same moment in time, or not. However, deinterlacing involves making a decision as to how to get two fields in each frame which are from the same moment in time, either by synthesizing those fields, or by discarding fields. I guess you could to this with TFM, but I'm not sure why.

As for all the matching parameters, I've used all the "micmatching" parameters when I invented a film transfer system which uses a standard projector from which the shutter has been removed. I used all the advanced settings of TFM in order to take the 1/1000 second shutter speed capture and determine which fields included the pulldown (blurred, and doesn't match anything, so must be deleted), and which fields need to be combined together to create a full, progressive frame. When capturing 24 fps (sound speed) film with a 60 field/second NTSC interlaced camera, some fields are duplicates of fields in adjacent frames and must be removed. Other fields need to be re-combined with fields in adjacent frames. It is very complicated, and to make it work I exported all the TFM metrics to a spreadsheet where I developed formulas which look forward and backwards through all the matching metrics in order to determine which fields need to be moved to other frames, and which have to be decimated.

I use MultiDecimate for the actual decimation.

It is a real real Rube Goldberg, but it works.

The point is, I know TFM about as well as I think anyone can know it. However, if I want to deinterlace, I'll use Yadif, QTGMC, or something that is dedicated to the task.

[hello_hello]

Originally Posted by poisondeathray

Note, that FFVideoSource returns the original full range 0-255 YUV 422 mjpeg data . This is the original storage format. If you use AVISource, some decoders may range compress the output automatically, others might return RGB

It does look more natural with ColorYUV(Levels="PC->TV") at the end of the script.

As you said, SwapFields still leaves some mild combing, mainly during the fade in, and a quick look that seems to indicate the result is the same as shifting one field, at least in respect to fixing the combing, only SwapFields is obviously the better way to do it. I'm not sure I've got my head wrapped around why that is though, as I thought I was only shifting one field by half a scanline. After the fields are interleaved and re-weaved again though, does that mean the field ends up shifted by more than just half a pixel? Or maybe the result only appears similar to SwapFields()? I'll continue to ponder that one...

[poisondeathray]

Originally Posted by hello_hello

It does look more natural with ColorYUV(Levels="PC->TV") at the end of the script.

It's expected, because mjpeg specifies full range YUV. FFmpeg based decoders return the actual storage data . Many decoders convert to RGB with full range 601 matrix

If you open up a regular image jpeg with FFMS2, it returns full range YUV too (for YUV jpegs, which are 99.9999% of jpegs) . Imagesource will return converted RGB. mjpeg is just an extension of jpeg

As you said, SwapFields still leaves some mild combing, mainly during the fade in, and a quick look that seems to indicate the result is the same as shifting one field, at least in respect to fixing the combing, only SwapFields is obviously the better way to do it. I'm not sure I've got my head wrapped around why that is though, as I thought I was only shifting one field by half a scanline. After the fields are interleaved and re-weaved again though, does that mean the field ends up shifted by more than just half a pixel? Or maybe the result only appears similar to SwapFields()? I'll continue to ponder that one...

It's mainly because of the brightness difference in fields. If you step through with separatefields, field pairs (from the same frame), have a very noticable different brightness during that fade sequence at the beginning. A single progressive frame has 2 different brightness levels for it's even and odd scanlines, thus it manifests as combing. There is no object or camera motion, it's just a difference in brightness

For the other residual combing, it's mainly from interlaced encoding compression artifacts . Another way to treat those is amdct with some tweaked settings . But this source is so soft at 1080 that just vinverse/vinverse2 is probably more than good enough

[hello_hello]

Originally Posted by johnmeyer

When capturing 24 fps (sound speed) film with a 60 field/second NTSC interlaced camera, some fields are duplicates of fields in adjacent frames and must be removed. Other fields need to be re-combined with fields in adjacent frames. It is very complicated, and to make it work I exported all the TFM metrics to a spreadsheet where I developed formulas which look forward and backwards through all the matching metrics in order to determine which fields need to be moved to other frames, and which have to be decimated.

It seems like you'd end up with standard 2:3 pulldown to me, although I no doubt don't understand your process at all, however unless you're looking for a matching field in something other than adjacent frames, that sounds exactly like what TFM does. I think the default mode only tries to find a match for the dominant field, and if the best match produces a combed frame it de-interlaces, but it can switch to matching from the non-dominant field when the metrics say it produces a cleaner frame. I think by default it only does so on scene changes as switching fields is more likely to create an additional duplicate frame that can't be decimated for standard IVTC. Maybe your formulas work better than TFM's in your situation. I have no idea.

Originally Posted by johnmeyer

The point is, I know TFM about as well as I think anyone can know it. However, if I want to deinterlace, I'll use Yadif, QTGMC, or something that is dedicated to the task.

That wasn't really the point.

[johnmeyer]

Originally Posted by hello_hello

It seems like you'd end up with standard 2:3 pulldown to me, although I no doubt don't understand your process at all, however unless you're looking for a matching field in something other than adjacent frames, that sounds exactly like what TFM does.

It is much more complicated than that. I described some of it here:

https://forum.doom9.org/showthread.php?t=162784

I was actually going to apply for a patent (I've always wanted one), and I'm sure I would have gotten one, but I found out the price of my vanity when I spent an hour with a patent lawyer who told me what I already pretty much knew: $10,000 just for a provisional, and $50,000 to see it all the way through.

I ain't that vain.

The reason I'm sure it could be patented, however, is that the idea is extremely difficult to grasp, and therefore not "obvious," a key requirement for a patent.

Since I have posted the idea here, I obviously don't intend to defend it in any way.