Correctly IVTCing ~20fps silent films

[Blue_MiSfit]

Hey folks,

So I've been working with a lot of old silent films recently. My sources are relatively good MPEG-2 encodes at 60i. There are lots of "old film" artifacts, but I'm leaving those in. I will be encoding with x264 at ~1.3 mbit, which usually looks okay.

Most of these silent films appear to have a true frame rate of 19.98fps progressive.

When I bob-deinterlace and inspect the fields, I get a 3:3 (AAA:BBB) pattern, which seems simple enough. Now, I've been performing IVTC by just decimating the result like this:


Tdeint(mode=1)
TDecimate(cycle=6, cycleR=4)


However, I realize that I'm potentially throwing away a lot of resolution by doing this.

The "duplicate" frames aren't really duplicates - they're all slightly different but temporally the same. The weird thing is a lot of these "duplicates" have what look like blending in them.

At any rate, I'm perfectly happy with throwing away 4/6 fields, but I'm just wondering if there's a proper way to accomplish my end result by using TFM or Telecide instead of TDeint(mode=1) or some other bobber.

Thanks,
~MiSfit

[Guest]

Your post has inconsistences and ambiguities. E.g., you talk about throwing away 4/6 of the fields, but you also say that the base frame rate is 19.97. That doesn't compute. Have you tried:

telecide(post=0)
decimate(3)

You should post a link to an unprocessed source sample with good motion for a definitive analysis.

[Blue_MiSfit]

Neuron2,

Sorry to confuse!

I merely mean that when I bob-deinterlace, I end up throwing away 4 of the 6 interpolated frames to recover the original frame rate.

I'm not totally familiar with IVTC. That much is apparent!

From what I understand, a field matcher like telecide or TFM attempts to re-create the original frames by matching fields, as opposed to interpolation. Compare this to bob-deinterlacing, which interpolates a full frame from each field.

From my understanding, since a bobber like tdeint(mode=1) interpolates each field to re-create frames, there would be less spatial resolution in each of these frames than if I had successfully field matched them. In that case, no interpolation would have been performed.

This situation would have been considerably simpler for me to understand / explain if bobbing the input would have literally produced a clean pattern of 3:3. This wasn't the case, as some of these frames were blended and weren't spatially identical.

I guess I'll stop trying to explain what I don't fully understand!

Here's an unprocessed sample from a Charlie Chaplin film.

http://www.megaupload.com/?d=SHWNU94Z

The artifacts I mentioned can be clearly seen with a simple tdeint(mode=1).

Your suggested telecide/decimate works well, but leaves a few combs. Vinverse terminates them.

Thank you very much for your time and input. I may be the expert where I work, but I'm definitely still learning - as you can tell! I appreciate your teaching.

~MiSfit

[Blue_MiSfit]

Here's another sample - this one has a very different pattern. I couldn't get a lock on it...

http://www.megaupload.com/?d=5E41A8HB

[IanB]

@Blue_MiSfit,

Well first up you should not be using a bobber. Bobbing is for completing frames out of fields for genuinly interlaced content, e.g. sports broadcasts, etc. You have chosen the right term IVTC, InVerse TeleCine, this involves matching top and bottom fields together to recover original progressive frames. No interpolation is required you have complete top and bottom fields for each frame, you just need to match them up.

If the original film was captured appropriatly then each field should be from 1 original film frame, i.e. there should be no blending or film gate transition junk. Use this script to inspect the nature of fields in your stream.Mpeg2Source(...)
SeparateFields()If all is well you should see, as you have indicated above, a clear 3:3 pattern. Check this very carefully sampling many points in the entire film. You are looking for anomollies like an occasional 2:4 pattern or worse a 3:2:3:3:3:3:4:3 pattern.

If the pattern is clean and clear then either Telecide() with Decimate() or TFM() with TDecimate() are 2 possible ways to go. Anyway the idea is Field Match then remove duplicates.

If the pattern is contorted and unstable then you may need some special treatment, so report back if you have such problems. Post a short clip demonstrating any such anomolly for further ideas.

If 2 same parity fields from same film frame have noise differences then you could use the Dup() filter to replace them with common averaged fields. i.e....
SeparateFields()
E=SelectEven().Dup(Maxcopies=2, blend=true)
O=SelectOdd().Dup(Maxcopies=2, blend=true)
Interleave(E, O)
Weave()

Telecide(... or TFM(...

[Blue_MiSfit]

@IanB
Interesting!

I had been using a bobber merely for inspection, and didn't think to switch over to field matching to actually do the work!

Thank you for the clarification!

Indeed, manual inspection of each of these films is a daunting task (I've got 70 or so on my plate!), but most of them have a clean 3:3 pattern. Some do have anomalies like you mention, a long string of 3:3:3:3 etc... with a 2 thrown in somewhere for good measure :P

I'm not totally concerned with getting the cleanest possible version of each film - I just want something that works well on most of them. To be honest, the bob followed by decimation doesn't look half bad. I've actually encoded a lot of these already with just that method :devil:.

I'm pretty familiar with the overall concept of IVTC, and in common forms like 3:2 pulldown on 24p film to 60i NTSC I can handle it no problem. It's the unusual stuff like this that throws me a curve ball! This is actually the first time I've dealt with content like this.

Most of the Indian stuff we have works well enough with RePal. That's probably why I chose to start with a bob, since RePal assumes that - and I was kind of in the "unusual telecined source" brain mode :D!

If you wouldn't mind taking a look at the second clip I posted, it's definitely not an example of 3:3, but some other pattern.

Finally, the code snippet you include at the end is a bit confusing - mostly because I'm not familiar with Dup()! So, you're separating the fields, and running Dup() on the evens and odds separately to look for duplicate film frames and average together a maximum of two to remove noise.

Does this decrease the number of frames? I'm assuming the point of all that would be to make the field matcher's job easier?

Thanks much for helping me learn!

~MiSfit

[Guest]

You shouldn't accept bobbing when you don't have to. It throws away quality unnecessarily.

For the first clip, you can enable Telecide's postprocessing to catch the combing that slips through. It's due to imperfections in the fields.

telecide()
decimate(3)

The second clip appears to be a frame pattern of 221221221... after field matching, so you have to decimate to 3/5. You can use FDecimate as below, for example:

telecide()
fdecimate(17.982)


Also try decimating 2 out of 5 using TDecimate. It will probably give better results. Although, if there are breaks in the pattern either method can pass through a duplicate frame. That's not so bad because silents were jerky anyways. :)

Your samples are too short to assess the consistency of the patterns.

[Blue_MiSfit]

You shouldn't accept bobbing when you don't have to. It throws away quality unnecessarily.

I agree wholeheartedly! :) That's why I posted.

Again, I appreciate your input.

~MiSfit

[Guest]

Ah, but you were vacillating:

To be honest, the bob followed by decimation doesn't look half bad.

[IanB]

Finally, the code snippet you include at the end is a bit confusing - mostly because I'm not familiar with Dup()! So, you're separating the fields, and running Dup() on the evens and odds separately to look for duplicate film frames and average together a maximum of two to remove noise.Yes Dup() replaces frames that are identical apart from some noise with bit identical versions. If they are already identical then this serves no purpose.
Does this decrease the number of frames? I'm assuming the point of all that would be to make the field matcher's job easier?It does not change the number of frames. And yes it can make the field matchers job easier because the field duplicates are now bit identical.

And you have the possible option for a bit of noise reduction here, where as the field matcher/decimater will simply discard one of the (near) duplicate fields.

[Blue_MiSfit]

Oh indeed! It doesn't look half bad - especially considering most of these films are so old and dilapidated that a softer image is the least of their problems.

But - they could certainly be better with proper IVTC and I will be doing this in the future! Thanks again.

~MiSfit