Request: Frame rate determination

[Xesdeeni]

As the proud new owner of a Sigma Designs based HD (no hyphen) DVD player (Zensonic Z500), I'm converting my collection of OTA HD TS movies to HD DivX so they will fit on a single DVD for simple (i.e. family friendly) playback.

But ABC's movies are a bear. It seems that they don't use the standard telecine. So far I've examined one movie that appears to be something on the order of 23.2603 fps and another that is something faster than 30 fps (I haven't completed the analysis, but on the first examination it appears to have ...2,2,2,1,2... patterns).

To inverse the telecine, I need to know the correct frame rate to decimate. To date, it seems that determining this frame rate can only be done by hand.

(As an additional burden, my local ABC affiliate converts the 720p to 1080i, so I have to inverse the telecine from an interlaced source.)

So I'm wondering if there is a way to use DGIndex to analyze the video, detect repeats, and then calculate the actual source frame rate. I realize there will be sequences of frames that are repeats, but should not be dropped. But I wondered if you had any ideas about this.

In the interest of full disclosure, I must confess that because FDecimate has serious problems choosing the correct frames to discard, in spite of my adjusting the threshold liberally (the source has a fairly significant amount of pre-processing and MPEG-2 artifacts), I tried tivtc and found it did a much better job (there was a problem with tivtc and the resulting video speed, but that's being pursued in a different thread). I found the problem whether I used Telecide or tivtc's tfm.

But no matter what tool I use to inverse the telecine, I need to determine the correct source frame rate. Is there any way to do this with better accuracy than stepping through the video and writing down the repeat pattern for analysis?

Xesdeeni

[Guest]

It depends on whether the telecine is done with field/frame repeat flags or with hard pulldown. If it is the former, then you can derive it from the field/frame repeat counts reported by the latest DGIndex beta. If it is the latter, then I have used MultiDecimate to do it (first field match and then compare the frame counts before and after global duplicate decimation, or just do the first phase and analyze the text files it creates).

I'm sure scharfi will have his own suggestions.

FDecimate needs an overhaul. It should allow some limited running deviation from the desired frame rate, while maintaining the global rate, in order to better choose the duplicates. Right now it just reaches a point where a frame should be ditched and just chooses from a pair of possibilities. If the duplicate was one frame further it ought to ditch that. It would have to be controlled, however, so as not to throw off local audio sync.

If you'd like to give me a test file, I would be willing to try to implement that strategy for FDecimate().

[Xesdeeni]

It depends on whether the telecine is done with field/frame repeat flags or with hard pulldown. If it is the former, then you can derive it from the field/frame repeat counts reported by the latest DGIndex beta.I'm no expert, but I see duplicate frames that have different noise, so it looks to me like they are not using the MPEG flags. If it is the latter, then I have used MultiDecimate to do it (first field match and then compare the frame counts before and after global duplicate decimation, or just do the first phase and analyze the text files it creates).Sometimes the answer is under your nose. MultiDecimate looks like it would at least eliminate the human error in counting frame repeats.

I'll have to experiment, but do I feed interlaced video through Telecide (or tfm) before using MultiDecimate, or would something like Bob be better?

You mention silent movies with odd source frame rates, which is certainly similar to this case. But say you had a 20 fps film that was telecined, but you didn't know the frame rate. However, your player would handle the telecine on playback, so you wanted to recreate the original frame rate, no matter what it was.

Is there a less manual way to determine the original frame rate?If you'd like to give me a test file, I would be willing to try to implement that strategy for FDecimate().While I was trying to determine why the inverse telecine wasn't working, I uploaded this sample (http://rapidshare.com/files/17656314/Toy_Story__Excerpt_.ts.html) that you can try. I'm not entirely sure that this particular clip can't be handled by setting FDecimate's threshold to 4.5 or above. But maybe it will help with the algorithm.

Xesdeeni

[Guest]

I'll have to experiment, but do I feed interlaced video through Telecide (or tfm) before using MultiDecimate, or would something like Bob be better? You're confusing me now by talking about interlaced video. You won't expect duplicate frames for interlaced video. The whole concept applies only to telecined progressive video (either hard or soft). As I said, you need to field match the stream before looking for duplicates.

Is there a less manual way to determine the original frame rate? Not that I know of.

I uploaded this sample that you can try Thanks, I'll have a look at it.

[Xesdeeni]

You're confusing me now by talking about interlaced video. You won't expect duplicate frames for interlaced video. The whole concept applies only to telecined progressive video (either hard or soft). As I said, you need to field match the stream before looking for duplicates.The local ABC affiliate converts the telecined 720p to 1080i. If I got only the 720p, I'd only have to decimate. But I also have to reclaim progressive fields from which I can decimate (Decomb's Telecide). So I was just asking whether MultipleDecimate would be happier with the output of Telecide, (tivtc's) tfm, or Bob.Not that I know of.How difficult would it be? How tough would it be to process the text log file and figure it out?

Xesdeeni

[Xesdeeni]

I don't want to get into trouble if this should be moved...

I've tried a few things by way of analyzing the mfile.txt generated by MultiDecimate(). If you don't know the frame rate, determining it seems pretty complicated. There are places where there are duplicates that are supposed to be there. But without the cycle pattern, how many do you leave?

???

Xesdeeni

[Guest]

Well, it's a tricky thing. You can use static scene protection to avoid counting the static scenes as duplicates to be removed.

With anime like this, why is it so important to get the EXACT base frame rate?

[Guest]

OK, I processed your Toy Story clip using Multidecimate with:

Remove duplicates: Global: Protect static scenes
threshold: 0.3
run length: 2

The result is pretty good and the base frame rate is:

632 / 781 * 29.97 = 24.25

EDIT: Yes, I know that protected static scenes should be decimated at the prevailing rate (which we don't know!), but Multidecimate doesn't support that. So you can have extra dups in the static scenes. You won't notice them because the scene is static. Anyway, to keep AV sync you'll probably need to do cycle-based decimation. I hope to have at least shown you why what you ask for is not so easy.

[Xesdeeni]

OK, I processed your Toy Story clip using Multidecimate with:

Remove duplicates: Global: Protect static scenes
threshold: 0.3
run length: 2

The result is pretty good and the base frame rate is:

632 / 781 * 29.97 = 24.25Hmm. That's on the other side of 24 fps from what I (and another kind soul) have calculated. He calculated the 23.2603, and I calculated something less than that.EDIT: Yes, I know that protected static scenes should be decimated at the prevailing rate (which we don't know!), but Multidecimate doesn't support that. So you can have extra dups in the static scenes. You won't notice them because the scene is static. Anyway, to keep AV sync you'll probably need to do cycle-based decimation. I hope to have at least shown you why what you ask for is not so easy.There's no doubt this is difficult. Take a look at this: http://forum.doom9.org/attachment.php?attachmentid=6907&stc=1&d=1173156796. This is a chart of an excerpt of the mfile.txt created by MultiDecimate(pass=1) of the movie (not the uploaded piece) after it was passed through AVISynth's Bob(), to reclaim some semblance of the 720p ABC actually started with. The pattern is obvious at first, but about frame 585, things get pretty rough. At this point in the movie, this is a "static scene" in the classic sense.

But more subtly, before that, there is a problem. The pattern appears to be ...3, 2, 3, 2... for a while. But about frame 528, you get a sequence of 4 (...3, 2, 4, 2, 3...). Why not replicate one of the 2's!? Then, about frame 560, there is a more logical extra 3 (...3, 2, 3, 3, 2...).

I assume that the cycle-based analysis would handle this, right?

Then, toward the right, it seems all hell breaks loose on this sequence of frames. I don't see how a single threshold can handle this. If I chose 1.0, for example, a number of frames starting about 658 would be included in the candidates to be dropped, right? If the cycle was logical, this might work. But since there is that ...2, 4, 2... cycle at about frame 663, the chosen frames could be 661, 663, 666, 669, instead of what I believe is correct: 661, 663, 667, 669. This not only replicates a frame (663-666), but drops one as well (667-668).

And of course, other thresholds have their problems. The lowest frame that I would want to choose is about frame 734, at 0.53. But choosing that as a threshold would cause confusion for areas like frames 532-534 or 630-633.

I'm not sure whether a dynamic threshold would help here, mostly because I'm not sure how you could calculate one. But maybe?With anime like this, why is it so important to get the EXACT base frame rate?I happened to attack Toy Story first (which I would not call anime, something I associate with frame doubled sources in the 12 fps range), but ABC screws with all their films. The next one I'll probably look at is Anchor Man, and my first pass at analyzing that one seemed to show it was faster than 24 fps.

As for why it's important to reclaim the exact original frames. That wouldn't be an issue if I were to encode the 29.97 interlaced frames they gave me. Obviously I'd get exactly what they put in. But of course I'd need a significantly higher bitrate to get a decent quality. Obviously, it's best to reclaim the progressive frames.

But replication at 24 fps is significantly more evident than replication at 60 fps. When you replicate a field (or for 720p, a 60 Hz frame), you only extend the time a frame shows by 1/60 sec. But when you replicate a source frame at ~24 fps, you extend the time a frame shows by 1/24 sec, or 250% that amount of time! This is MUCH more noticeable on playback.

If instead, I encode at an odd frame rate (like 23.2603), and allow the player to handle the replication at the field level. I minimize the impact.

And of course, as I mentioned above, it's much more significant if I actually lose frames when one is replicated.

Thanks for continuing the discussion.

Xesdeeni

[Guest]

Automatic duplicate detection is still imperfect. It just can't match a human. :(

If you have an idea to improve it over thresholded windowed SAD, I am all ears.

[Xesdeeni]

If you have an idea to improve it over thresholded windowed SAD, I am all ears.I'm sorry, I missed that acronym (SAD).

So I guess I'm trying to figure out how to dynamically determine the threshold, and to algorithmically determine the source frame rate.

I'm not good at fuzzy logic, so I'll have to stick to the old fashioned kind. It seems something is amiss with the chart I posted above (were you able to view it?). And I'm at work right now, so I can't repost it until later. But if you could see the chart, you'd see a relatively easy set of frames to decimate on the left. The mfile.txt values are significantly different, so detection of the pattern would be pretty straightforward.

So I'm thinking that when you get to the ambiguous area in the middle, you should be able to detect that the values are not helpful. So you could switch to re-using the pattern until things get back to obvious.

I realize your current implementation does something like this, but the user must input the pattern. Instead, I'm proposing determining the pattern based on the content. To start with, let's assume that the beginning of a clip has the pattern in an easily recognizable form (later, we'll have to deal with clips where the start is ambiguous and the pattern must be determined from later in the clip). When the pattern disappears, you could keep using it until things became more sane.

I'm just thinking out loud here. I'm not sure how reasonable this may be.

Again, thanks for continuing the discussion.

Xesdeeni

[Guest]

SAD = Sum of Absolute Differences

The chart link isn't working as you say.

I need a stream with the relevant parts to comment.