Decimating while removing duplicates

[ABurns]

I'm working on a restoration project with an unusual problem. I have a source that's 29.97 progressive and needs to be decimated to 23.976. However, almost all of the scene cuts are dirty and I've had to duplicate the frames before and after most of the cuts like ABB|CCD, which naturally creates a bit of stutter in the motion. I'm trying to figure out the decimation method that will give me the best chance of making sure those duplicate frames are detected and discarded as part of the process, as opposed to my good frames next to them.

Anyone have any suggestions for how to approach this?

[johnmeyer]

I've done a lot of this sort of thing. The usual thing is that you have a splice at the scene boundary (if it is film), or if it is from a camcorder, you might have some video breakup at the scene due to lack of a flying erase head in the camcorder.

The simplest thing is to just chop off the frames in your NLE. If you cut the bad frame, this will also cut the audio, so the audio after that point will stay in sync.

I'm not sure why you chose instead to duplicate the frames.

You did not mention if the original source was film. If that is the case and if you want to use TFM/Tdecimate to do the inverse telecine, you have indeed created a problem for yourself because you've added additional duplicates and, since they are perfect duplicates, that ensures that the IVTC process will eliminate those first, which means the nearby duplicates that were inserted when the film was telecined will not be removed.

One solution would be to replace the perfect duplicates you inserted with a motion estimated frame. The script below will do that. However, it may screw up with the "CC" duplicate at the beginning of the scene.

The other solution is to simply delete the bad frames rather than replace them with a duplicate. That is what I'd do.

#This script finds exact duplicate frames and then interpolates a new frame in place of the duplicate.
#It works on interlaced video or progressive video (use Filldrops instead of FilldropsI for progressive)

loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\mvtools2.dll")

source=AVISource("e:\fs.avi").ConvertToYV12.killaudio()

output=filldropsI(source,0.01)
return source

#-------------------
function filldropsI (clip c, float "thresh")
{
  thresh = Default(thresh, 0.1)
  even = c.SeparateFields().SelectEven()
  super_even=MSuper(even,pel=2)
  vfe=manalyse(super_even,truemotion=true,isb=false,delta=1)
  vbe=manalyse(super_even,truemotion=true,isb=true,delta=1)
  filldrops_e = mflowinter(even,super_even,vbe,vfe,time=50)

  odd  = c.SeparateFields().SelectOdd()
  super_odd=MSuper(odd,pel=2)
  vfo=manalyse(super_odd,truemotion=true,isb=false,delta=1)
  vbo=manalyse(super_odd,truemotion=true,isb=true,delta=1)
  filldrops_o = mflowinter(odd,super_odd,vbo,vfo,time=50)

  evenfixed = ConditionalFilter(even, filldrops_e, even, "YDifferenceFromPrevious()", "lessthan",  String(thresh))
  oddfixed  = ConditionalFilter(odd,  filldrops_o, odd,  "YDifferenceFromPrevious()", "lessthan",  String(thresh))

  Interleave(evenfixed,oddfixed)
  Weave()
}

function filldrops (clip c)
{
  super=MSuper(c,pel=2)
  vfe=manalyse(super,truemotion=true,isb=false,delta=1)
  vbe=manalyse(super,truemotion=true,isb=true,delta=1)
  filldrops = mflowinter(c,super,vbe,vfe,time=50)
  fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")
  return fixed
}

[ABurns]

Originally Posted by johnmeyer

The simplest thing is to just chop off the frames in your NLE. If you cut the bad frame, this will also cut the audio, so the audio after that point will stay in sync.

I considered this, but that's going to be a lot of frames and I'm trying to stay as time-accurate as possible. There are also some points where even dropping 2 frames together might create a noticeable stutter in the audio. This would be a last resort for me.

Originally Posted by johnmeyer

You did not mention if the original source was film.

I didn't mention it because it's complicated. It's 16mm film from the 1970s but no prints are known to exist anymore. My distributor was only able to get me a source made from a broadcast master somewhere along the way in which the interlaced fields are already blended. It landed in my lap because the distributor wants a better version to sell for streaming.

However, the blending is not horrible and it actually looks surprisingly OK for what it is. There's not a whole lot of ghosting. That's why I said it's just 29.97 progressive, which is essentially what they sent me.

Originally Posted by johnmeyer

The other solution is to simply delete the bad frames rather than replace them with a duplicate. That is what I'd do.

I may end up doing that on the high-motion scenes where the stutter is noticeable. I can get away with a duplication on low-motion scenes if I have to, but again I consider that a last resort.

I tried the script but it just created a blended frame at the cut, so I don't think that's going to work for my use case.

[poisondeathray]

To interpolate at a scene cut you would need 1 directional vector and forward prediction; not both directions like that filldrops version - otherwise you will contaminate with the next or previous scene

An interesting machine learning project is DMVFN, which does forward prediction . There is no vapoursynth or avisynth port yet, but the demo works
https://github.com/megvii-research/CVPR2023-DMVFN

Some examples, discussion in this thread
https://forum.doom9.org/showthread.php?t=184387

[johnmeyer]

The other thing to do would be to do the IVTC first and then add the duplicates to mask the bad frames. You will never notice duplicates just before or after a scene change. I know because I've done it many, many times when editing old films that I've transferred for clients.

Originally Posted by ABurns

I tried the script but it just created a blended frame at the cut, so I don't think that's going to work for my use case.

Yes, I mentioned in my earlier post, and poisondeathray said the same thing: the motion interpolation only works in one direction, so if you have dups on the frame beyond the scene change, that script won't work. It should work, however, on duplicates just prior to the scene change.

[ABurns]

Originally Posted by johnmeyer

The other thing to do would be to do the IVTC first and then add the duplicates to mask the bad frames.

Interesting. Why do you say that? My thought was that if 1 in 5 frames is going to be discarded anyway, there's a likelihood that at least some of the dupes will get dropped as par for the course. If I do it after, there's a zero chance that any will get dropped. Do I have that wrong?

[johnmeyer]

Originally Posted by ABurns

Interesting. Why do you say that? My thought was that if 1 in 5 frames is going to be discarded anyway, there's a likelihood that at least some of the dupes will get dropped as par for the course. If I do it after, there's a zero chance that any will get dropped. Do I have that wrong?

Perhaps you didn't understand what I was saying.

First, I assume (perhaps incorrectly) that this is 24 fps sound film that to which pulldown fields or frames were added in order to get it to play on NTSC 30 fps TVs (the actual NTSC frame rates are 23.976 and 29.97, but that isn't important for this discussion).

If this is what you have, and since it has somehow been kept progressive (which is actually pretty unusual) then your original video should have a duplicate frame every four frames. If that is the case, then using the TFM()/Tdecimate() combination will remove those duplicate frames and you will end up with the original 24 fps film, exactly as it existed on the original film. That version of your movie will still have the bad splices that you want to fix. At that point you can simply overwrite the bad frame (or frames) with a duplicate of the previous (good) frame or the next (good frame). If you have two bad frames in a row at the spice point, you simply overwrite the first bad frame with a duplicate of the previous frame, and then overwrite the second bad frame with a duplicate of the following frame.

You will end up with the same number of frames; the movie is now 24 fps instead of 30 fps; and the audio should match perfectly.

Even if you end up with two dups at the scene transition, you'll never notice it.

[ABurns]

Originally Posted by johnmeyer

First, I assume (perhaps incorrectly) that this is 24 fps sound film that to which pulldown fields or frames were added in order to get it to play on NTSC 30 fps TVs (the actual NTSC frame rates are 23.976 and 29.97, but that isn't important for this discussion).

If this is what you have, and since it has somehow been kept progressive (which is actually pretty unusual) then your original video should have a duplicate frame every four frames.

This is where we are missing each other. It's 24p with a 3:2 pulldown, which I'm assuming is what was on the broadcast master tape. I wish I had it. What I have is a copy of that tape (unknown gen), upon which somebody along the way has run FieldDeinterlace() or something similar. The pattern is 3 original frames followed by two blended frames. The saddest part is this is the best known source for this movie, so whatever I can do with this is probably as good as it's ever going to get unless a print or a master turns up.

Once those fields are blended, they can't be unblended - at least not by any means I can comprehend. When I said it looks surprisingly good for what it is, I meant that the blended frames are good enough that they could almost pass for original frames to somebody who wasn't paying attention. Why the ghosting is so minor is the real mystery here.

I'm treating it as if it's true 30p because it could almost pass. Smell what I'm stepping in now?

[johnmeyer]

OK, that is totally different than what you posted in #1 above. In that post you said the problem was a scene changes. I'd like to help, but if the problem description changes, it makes it really tough to respoind.

[poisondeathray]

Is the pattern consistent ? Or were there additional problems like post telecine edits, cadence breaks on top of that?

For 3 frames, 2 blended - I would expect it to be quite jerky if you decimate 1 in 5. Another approach might be to replace the 2 blend frame with the proper frame using RIFE or similar interpolation


If you've already replaced the "bad scene change" frames with duplicates, any adaptive decimation algorithm (e.g. tdecimate) will preferentially choose to drop duplicates by metrics - duplicates have no difference (especially true duplicates - they do not even compression differences) .

You can use a longer window by adjusting the cycleR and cycle parameters - e.g. 3 in 15, 10 in 50, etc... instead of 1 in 5 - because there is a chance that if you used "regular 1 in 5" that the wrong frame gets decimated with "ABB|CCD" depending on where the previous cycle was , and the previous drop. For example - "ABB|CCD" - if you started with that , there are 2 pairs of duplicates in the 1st 5 frames.

[ABurns]

Originally Posted by johnmeyer

OK, that is totally different than what you posted in #1 above. In that post you said the problem was a scene changes. I'd like to help, but if the problem description changes, it makes it really tough to respoind.

The scene changes are my problem. On approx 30% of the cuts either the frame immediately before or immediately after is blended and showing both scenes. On another 40% more, both frames are blended and show two scenes. So a total of around 70% of the cuts need fixing. A frame with a little bit of ghosting is OK. A frame that contains two different scenes is unacceptable, hence the duplication at the cuts.

[StainlessS]

ABurns,
Maybe this is useful (or not),
https://forum.doom9.org/showthread.php?t=183790
Only for detecting single scene change blend frame.

EDIT: Creates a frames file.
Also additional script to show "PreviousFrame, BlendFrame, NextFrame", for inspection of detect 'goodness'.
Before : Bad : After (clickme twice)


EDIT:

On another 40% more, both frames are blended and show two scenes.

Is the 2nd such blend a dupe of the first blend, if so may need get rid of those dupes before single SC blend detect (maybe).

[johnmeyer]

StainlessS,

I follow your work, but I missed DetBlendFrameSC.avs. It sounds like it is exactly what the OP needs.

[ABurns]

Thanks for the suggestions guys. DetBlendFrameSC may be what I need, I don't know. It might be a few days before I can work on it again, but I'll report results.

@poisondeathray I'll try the longer cycle. I already tried tdecimate, but I just used the standard 1 in 5 so I'll give that a go too. That also might do the trick.

[ABurns]

Originally Posted by poisondeathray

You can use a longer window by adjusting the cycleR and cycle parameters - e.g. 3 in 15, 10 in 50, etc... instead of 1 in 5 - because there is a chance that if you used "regular 1 in 5" that the wrong frame gets decimated with "ABB|CCD" depending on where the previous cycle was , and the previous drop. For example - "ABB|CCD" - if you started with that , there are 2 pairs of duplicates in the 1st 5 frames.

Ding ding ding!!! We have a winner. I plugged it in for a quick test and 5 in 25 does exactly what I want. It misses the odd dup here and there, but it gets most of them. What I have now could almost pass for a proper IVTC and only somebody really looking would ever notice anything amiss.

Thanks for the assist, everyone.