HDTV TS IVTC Problem

[Xesdeeni]

I have a 1080i OTA HD ATSC TS that is definitely from film. But after running it through DGIndex/DGDecode, there are some discontinuities in the frame repeat pattern, even in the middle of scenes/sequences. DGIndex says the clip is about 60% Film. When I use "Honor the flags," I can see film frames that are only in one field. Within a few video frames, I then see film frames that are in more than three fields. I tried "Forced Film," but as you can expect, this didn't help.

Perhaps I need to be asking this in Decomb thread instead, but I can't seem to get DGIndex/DGDecode or Decomb to recognize these unique fields and just interpolate, and then recognize the over-used frames and drop them. Instead, I lose the unique frame and get a repeated one later.

This doesn't seem to be isolated, so I don't really want to have to manually adjust by single-stepping through an hour and a half film.

Any advice?

Xesdeeni

[scharfis_brain]

provide a sample (as long as possible) of this video. just cut it out. don't recode.

maybe there is a regular pattern, that can be automatically detected and removed.

broadcasters very often seem to harm sources in very different ways to squeeze or stretch running times of their movies.

[Xesdeeni]

provide a sample (as long as possible) of this video. just cut it out. don't recode.

maybe there is a regular pattern, that can be automatically detected and removed.

broadcasters very often seem to harm sources in very different ways to squeeze or stretch running times of their movies.I'll have to see if I can find a place to put something so big. I'll get back to you.

But in the mean time, I've done some more analysis. I was wrong about frames appearing in a single field. But there are definitely 3:3:3 and 2:4:2 patterns in there.

[This part may shift the topic to Decomb or FDecimate(), so if this needs to be moved, please let me know...]
I believe I've been able to get Telecide() to create the individual frames correctly (it's sometimes difficult to judge whether a frame is missing or not if the motion isn't very constant). But no matter what I do, I can't get Decimate() or FDecimate() to remove the duplicates correctly. They remove unique frames but leave in duplicates.

One thing I'm wondering is how long the (perceived) error in FDecimate() can hold on before it must be corrected. I mean, if FDecimate() thinks it has too many unique frames, or too many duplicates, how long will it wait before it duplicates or throws out a frame to get back in sync?

On that same note, is there a way to just have FDecimate() (or Decimate()) throw out all duplicates, without paying attention to the pattern, just so I can see whether there's any real time dilation being done? This might also tell me whether it is throwing out/leaving in duplicates because of the above-mentioned perceived error, or if it genuinely thinks it's doing the right thing.

I also found some very strange artifacts in the encoding.

In addition to overzealous noise reduction that results in multiple halos of objects' previous frame positions, there are also some frames with serious garbage noise. The ones like this I've seen have been the frame before a scene change, so I'm pretty sure this is in one of the encoders (either the one from the network, or the one from the station).

I've also seen duplicate frames (the frontmost objects that are in motion don't change position), but where some background objects appear to shear a bit. I'm guessing this might be a noise reduction artifact too, but it seriously confuses FDecimate(). I've raised the FDecimate() threshold to 3.5, but it still keeps duplicates and throws out unique frames.

Also, I seem to have had better luck getting FDecimate() to throw out the right frames (I didn't go through more than a couple of trouble regions, so I'm not positive) if I use Bob(). I wonder if there's any way to output the decisions made on this stream and use it as an input on a Telecide()'d stream?

Xesdeeni

[scharfis_brain]

you should try tritical's tivtc.dll. it offers functions such as tfm() and tdecimate() which both are very good working on such weird streams.

you could upload the samples to www.rapidshare.com

the max. file limit is 100 MB (or 300MB dunno anymore). split archives are possible.

[Xesdeeni]

you should try tritical's tivtc.dll. it offers functions such as tfm() and tdecimate() which both are very good working on such weird streams.

you could upload the samples to www.rapidshare.com

the max. file limit is 100 MB (or 300MB dunno anymore). split archives are possible.I downloaded that and tried it, but I saw similar problems and went back to Decomb. Now that I've exhausted more combos on Decomb, I'll try it again.

I'll see if I can extract some samples this evening and upload them (128 Kbps is agonizingly slow). 100 MB will be about 45 seconds, which should be enough.

Thanks.

Xesdeeni

[Xesdeeni]

I downloaded that and tried it, but I saw similar problems and went back to Decomb. Now that I've exhausted more combos on Decomb, I'll try it again.Same type of problems with tivtc.I'll see if I can extract some samples this evening and upload them (128 Kbps is agonizingly slow). 100 MB will be about 45 seconds, which should be enough.I uploaded 48 MB to rapidshare (http://rapidshare.com/files/17656314/Toy_Story__Excerpt_.ts.html). IVTC it and then single-step. I particularly look at the shot where Woody appears to be riding a horse in the desert, and the shot from on the chair as it spins.

Thanks again!

Xesdeeni

[scharfis_brain]

I got this thing decimated without dups or drops using tfm & tdecimate.
Telecide & decimate probably won't help here,
because your video has an actual framerate of:

fps = 59.94 fps * 2/5 * (67 - 2) / 67 = 23.2603 fps

The broadcaster seemed to have slowed down that movie in order to fit it into the schedule.

This slowdown leads into this field repetition pattern after separating the fields:

3 2 3 2 3 2 3 2 4 2 3 2 3 2 3 2 3 2 3 2 3 3 3 2 3 2
3 2 3 2 3 2 3 2 4 2 3 2 3 2 3 2 3 2 3 2 3 3 3 2 3 2 3 ...

this means that every 67 fields 2 fields are inserted.

so if you use DGIndex and honoring pulldown flags and this script:

mpeg2source("C:\forum\Xesdeeni\Toy_Story__Excerpt_.d2v")
tfm(pp=0)
tdecimate(mode=2, rate=23.2603)

you'll get a nice decimated 23.2603 fps progressive movie out of it.

you should speed it up to 23.976, 24 or 25 fps. This of course needs you to accelerate the audio as well.

And btw.: scale that movie down to 720p (or less). This transfer isn't worth 1080i resolution.

[Xesdeeni]

I got this thing decimated without dups or drops using tfm & tdecimate.
Telecide & decimate probably won't help here,
because your video has an actual framerate of:

fps = 59.94 fps * 2/5 * (67 - 2) / 67 = 23.2603 fps

The broadcaster seemed to have slowed down that movie in order to fit it into the schedule.

This slowdown leads into this field repetition pattern after separating the fields:

3 2 3 2 3 2 3 2 4 2 3 2 3 2 3 2 3 2 3 2 3 3 3 2 3 2
3 2 3 2 3 2 3 2 4 2 3 2 3 2 3 2 3 2 3 2 3 3 3 2 3 2 3 ...

this means that every 67 fields 2 fields are inserted.I mentioned above that I wondered if they had done this. Can you elaborate on how you determined this? Surely you didn't manually step through several hundred frames, writing down the pattern, right?so if you use DGIndex and honoring pulldown flags and this script:

mpeg2source("C:\forum\Xesdeeni\Toy_Story__Excerpt_.d2v")
tfm(pp=0)
tdecimate(mode=2, rate=23.2603)

you'll get a nice decimated 23.2603 fps progressive movie out of it.

you should speed it up to 23.976, 24 or 25 fps. This of course needs you to accelerate the audio as well.Interesting. That gives me a couple of choices.

If I want to stick to the AC3 as transmitted, I'd need to leave the frame rate alone. I'll be playing this back on a Zensonice Z500, so it makes me wonder whether that can handle the odd frame rate correctly. It does handle PAL rates played back on an NTSC TV, so maybe it would. If it did things right, instead of repeating a full film frame every 67 film frames, the player could repeat a field as needed (I play back at 1080i for my HDTV). That should make the repeats much less noticable.

If I want to speed the video up to 23.976, then I'll need to speed up the audio and re-encode it. I've had poor luck with speed changing audio with pitch shifting, so I'd have to live with the slight pitch change (I'm not sure whether I'd notice a 3% change). I'm also not sure how to keep the 5.1 encoding.

Or, I could speed the video up and rip the audio from my Toy Story DVD. I'd have to edit it to match, but the only way I can think of to do this without decoding it would be to edit in VirtualDubMod alongside the DVD video. Since VDM doesn't have two timelines, I'd have to do some work to cut it to match. And I wouldn't be able to fade where they go to/from commercials.

It seems the first choice might be worth a try.And btw.: scale that movie down to 720p (or less). This transfer isn't worth 1080i resolution.Yes, that's already in my script.

One of the major pain in the backsides we face here in the DFW area is that the local ABC affiliate is owned by Belo. Some moron in corporate management decided a while back that all Belo owned stations could save money on HD equipment if they all used 1080i. So WFAA takes the 720p ABC feed and converts it to 1080i. The fools at WFAA claim that 1080i looks better than 720p. The problem is that it doesn't matter which side of that religious format argument you fall on, no-one with two or more brain cells agrees that CONVERTING from one to the gives you better quality. We get to live with the worst of both worlds: only 720 lines of resolution (well, actually less due to misalignment of the resampling requiring filtering) PLUS interlace flicker!

Ugh! (Sorry, rant mode off...)

Xesdeeni

[scharfis_brain]

I mentioned above that I wondered if they had done this. Can you elaborate on how you determined this? Surely you didn't manually step through several hundred frames, writing down the pattern, right?Actually I did excatly this: wirting down the pattern of the first frames until a pattern was visible. I counted 160 fields. Then I estimated the other pattern breaks with the simple math (every 67 fields a break) to be sure that the pattern stays constant.
That took about 5 minutes concentrated work. Not more.

after this was ensured, I calculated the original framerate and fed it into tdecimate: et violà a smooth result came out.

There is no other way to examine this properly.
You have to count ther frames if the decimation leaves drops or dupes.
Interesting. That gives me a couple of choices.

If I want to stick to the AC3 as transmitted, I'd need to leave the frame rate alone. I'll be playing this back on a Zensonice Z500, so it makes me wonder whether that can handle the odd frame rate correctly. It does handle PAL rates played back on an NTSC TV, so maybe it would. If it did things right, instead of repeating a full film frame every 67 film frames, the player could repeat a field as needed (I play back at 1080i for my HDTV). That should make the repeats much less noticable. you may try that.

If you want to go for DVD you also can make usage of DGPulldown which should be perfectly suited to press 23.26 fps into MPEG without the need to hardcode the dupes.

If I want to speed the video up to 23.976, then I'll need to speed up the audio and re-encode it. I've had poor luck with speed changing audio with pitch shifting, so I'd have to live with the slight pitch change (I'm not sure whether I'd notice a 3% change). I'm also not sure how to keep the 5.1 encoding.
Actually 24 fps (23.976fps) is the original speed of the movie. so the broadcast should have the wrong pitch! The speedup should bring it back to the normal level.
I cannot help you wiith 5.1 audio transcoding´. But I am sure that there are some Guides on doom9.org.
At least you may try besweet.

Or, I could speed the video up and rip the audio from my Toy Story DVD. I'd have to edit it to match, but the only way I can think of to do this without decoding it would be to edit in VirtualDubMod alongside the DVD video. Since VDM doesn't have two timelines, I'd have to do some work to cut it to match. And I wouldn't be able to fade where they go to/from commercials. this is the preferrable way to do so!
I've done this several times before (HD broadcast synced with DVD audio).
I ripped the movie from the DVD and stacked it in vertical with the HD-movie in an AVS-script. Then I edited the HD-movie with the trim() function so it fitted the dvd-movie framewise. missing frames in the hd-movie have to been take out of the dvd-movie.

One of the major pain in the backsides we face here in the DFW area is that the local ABC affiliate is owned by Belo. Some moron in corporate management decided a while back that all Belo owned stations could save money on HD equipment if they all used 1080i. So WFAA takes the 720p ABC feed and converts it to 1080i. The fools at WFAA claim that 1080i looks better than 720p. The problem is that it doesn't matter which side of that religious format argument you fall on, no-one with two or more brain cells agrees that CONVERTING from one to the gives you better quality. We get to live with the worst of both worlds: only 720 lines of resolution (well, actually less due to misalignment of the resampling requiring filtering) PLUS interlace flicker!

LOL. couldn't you've stated earlier that this is an ABC broadcast ;)
Cause ABC seems to alter films very often in this way!
(As I experienced from many problematic samples that were sent to me)

ABC does both: speedups as well as slowdowns of movies.

[laserfan]

One of the major pain in the backsides we face here in the DFW area is that the local ABC affiliate is owned by Belo. Some moron in corporate management decided a while back that all Belo owned stations could save money on HD equipment if they all used 1080i. So WFAA takes the 720p ABC feed and converts it to 1080i.Xesdeeni I'm in Austin where Belo does the same thing, and while Boston Legal always seems to IVTC just fine, Grey's Anatomy does not--I gave-up trying to fix it and just use:

tfm().tdecimate(hybrid=1)

...and this does a fine job for me--I don't even think about it any more.

Just another data point--I'm no expert and for all I know the tfm script could be "better". I just tried it and it worked as well as I needed...

[scharfis_brain]

tfm().tdecimate(hybrid=1)

...and this does a fine job for me--I don't even think about it any more.
this will create blends of the neighboring frames, when dupes or drops are found.

It is always better to count the frames to estimated the real framerate of the content so that you can decimate it properly without dupes, drops or blends.

[Xesdeeni]

Actually I did excatly this: wirting down the pattern of the first frames until a pattern was visible. I counted 160 fields. Then I estimated the other pattern breaks with the simple math (every 67 fields a break) to be sure that the pattern stays constant.
That took about 5 minutes concentrated work. Not more.Many thanks for that masochistic approach!If you want to go for DVD you also can make usage of DGPulldown which should be perfectly suited to press 23.26 fps into MPEG without the need to hardcode the dupes.Yeah, I'm kind of familiar with that one (see the first post in the DGPulldown thread) ;-) I realized that this didn't apply to DivX/XviD/MP4.2, but it was that thought process that made me wonder whether the Z500 will play it as-is.Actually 24 fps (23.976fps) is the original speed of the movie. so the broadcast should have the wrong pitch!Yeah, but 0.1% is considerably less pitch shift than 3%.I cannot help you wiith 5.1 audio transcoding´. But I am sure that there are some Guides on doom9.org.
At least you may try besweet.I've tried it before with not much luck, at least for AC3. And I only tried 5.1 one time, and it didn't work right....this is the preferrable way to do so!
I've done this several times before (HD broadcast synced with DVD audio).
I ripped the movie from the DVD and stacked it in vertical with the HD-movie in an AVS-script. Then I edited the HD-movie with the trim() function so it fitted the dvd-movie framewise. missing frames in the hd-movie have to been take out of the dvd-movie.Oh good grief! I'm going to have to call you a masochist AGAIN ;-)LOL. couldn't you've stated earlier that this is an ABC broadcast ;)
Cause ABC seems to alter films very often in this way!
(As I experienced from many problematic samples that were sent to me)

ABC does both: speedups as well as slowdowns of movies.Good to know. Many of the movies I'm planning on converting to single HD (no hyphen) DVDs are from ABC. In some cases I have the DVD, in others I don't. So if I could find a smooth process that would make things easier.

Thanks for all your help. If the Z500 will play this as-is. Then the only real issue I have to deal with is figuring out the effective frame rate for each movie by hand. After that, things should be smooth....that's my theory anyway ;-)

Xesdeeni

[laserfan]

...the only real issue I have to deal with is figuring out the effective frame rate for each movie by hand.
I wonder Xesdeeni if you have learned how to do this. I must suffer from some unique form of dyslexia that I can't seem to count fields reliably nor can I determine the "effective frame rate" for a clip--heck I can't even follow scharfis_brain's technique above.

I encountered again an ABC program where it was speeded-up slightly i.e. 2 minutes over 2 hours. No doubt they squeezed a bunch of commercials into those two minutes!

I'm surprised no one has invented a more automated way to do this?

[Xesdeeni]

I wonder Xesdeeni if you have learned how to do this. I must suffer from some unique form of dyslexia that I can't seem to count fields reliably nor can I determine the "effective frame rate" for a clip--heck I can't even follow scharfis_brain's technique above.

I encountered again an ABC program where it was speeded-up slightly i.e. 2 minutes over 2 hours. No doubt they squeezed a bunch of commercials into those two minutes!

I'm surprised no one has invented a more automated way to do this?Unfortunately no, I haven't found a reasonable way to do this. My best attempt was to punt completely and do my encode at 60 fps. I rely on the encoder recognizing the weird telecine pattern and optimizing the encode appropriately.

I'm not particularly fond of this, but it's the best I could come up with. Even when I counted myself, neither of the inverse telecine tools I tried (Decomb and tfm) would end up giving me good results. Both of them dropped frames that were obviously not duplicates and kept duplicates, usually both within a few frames of one-another.

If you (or anyone) has any better ideas, I'm all ears.

Xesdeeni

[laserfan]

I appreciate your candor, and must admit I am somewhat gratified to have found someone else, and I suspect Xesdeeni you are more knowledgeable/experienced than I, who is having the same trouble unravelling these funky HDTV broadcasts. I worry that nothing short of a Vulcan mind-meld w/sharfis_brain or ChiDragon or neuron2 or manano is gonna help me at this point. Has that been invented yet? :p

I will keep plunking away, and if I trip upon something that works better I will let you know about it. In the meantime we can hope some genius comes-across this thread and takes pity on us to make a tool of some sort....

[Xesdeeni]

I spent way to many brain calories on this one, and I just can't see a way to do to completely automated. If we had a tool that just removed duplicate frames, that would be wonderful for scenes with continuous motion. But any time things settled down, you'd throw out way too many frames.

The above-mentioned techniques should have theoretically worked. But for some reason, the tools themselves fell short. Oddly enough, they both work fine for genuine 3:2 pulldown content. But for the speed-adjusted videos, no dice.

Xesdeeni

[laserfan]

The above-mentioned techniques should have theoretically worked.Xesdeeni, you DID try sharfis_brain's script on your entire original video, yes??? Was it smooth and in-sync thruout?

I just tried it on mine, and IT DID NOT WORK i.e. it didn't remove all the dupes nor was the end result smooth or in-sync either. But at least I DID learn to count!!! ;) Here's the pattern I found in my speeded-up ABC program:

3132323232323232322232323232323232
3132323232323232322232323232323232

So for every 83 fields, 2 fields are removed:

59.94 * 2/5 * (83 + 2) / 83 = 24.553734939

My script:

tfm(pp=0).tdecimate(mode=2,rate=24.553734939759)

As I look at this again now, maybe I did the multiplier wrong i.e. maybe it should be 83/81 instead for an fps of 24.568 instead of 24.554. Guess I will try that for grins (will take hours with my 2 hour original of course). Oh, the pain. OK I will try once more... :(

[laserfan]

I tried again with the 2nd calculation and got the same result--herky-jerky w/dupes, and drifts out of audio sync. Not sure if there is some sort of tfm/tdecimate problem but at 11 hours for re-encoding (tdecimate must be working hard to maintain the framerate?) I'm disinclined to try again.

Brain's too skinny already to waste any more calories I guess! ;) :eek:

[scharfis_brain]

do NOT use a fieldmatcher before decimation!

as you are counting FIELDS you also need too decimate on the field-framerate.

this means you need to bob-deinterlace you video before decimating it. (in order to preserve the orphaned fields!)

you can use
tdeint(mode=1, tryweave=true, full=false).tdecimate(mode=2,rate=24.553734939759)

this will result in a smooth result. with fieldmatched double/triple fields and properly deinterlaced orphan fields.

audio however will be out of sync if you force the final framerate back to 23.976.
To avoid A/V async either slow down the audio if you insist on 23.976 fps or leave the video running at 24.55 fps.

[plugh]

Just passing by...

I've been running into weird frame-rate / decimation problems with ABC-HD and TNT-HD captures, too. The TIVTC package has some nice tools for dealing with this, once you do the counts and figure out what the actual M-in-N and/or frame-rate is. Haven't gone so far as to speed correct one after decimation yet (ie re-do the audio) as my target seems to handle the odd-ball frame rate video ok.

Anyway, you're not alone...

[laserfan]

...you need to bob-deinterlace you video before decimating it. (in order to preserve the orphaned fields!)

you can use
tdeint(mode=1, tryweave=true, full=false).tdecimate(mode=2,rate=24.553734939759)

this will result in a smooth result...Thanks, but... this doesn't work. Lots of extra dupes.

I'd gotten some field-order errors from DGIndex 1.4.8 which it corrected, but then I notice that my test clip appears to be BFF where the entire program is TFF (again according to DGIndex's parse of the d2v). Could these be affecting me? I try AssumeBFF() but it seems whether BFF or TFF I still meet with problems somewhere along the timeline. I'd upload a sample but my ISP is too slow. :(

[Xesdeeni]

you can use
tdeint(mode=1, tryweave=true, full=false).tdecimate(mode=2,rate=24.553734939759)

this will result in a smooth result. with fieldmatched double/triple fields and properly deinterlaced orphan fields.Do you mean there should be interpolated orphan fields?

Because there should not be any interpolated frames. Both sets of lines (odds and evens) are there for every film frame in the original source (at least in my captures).

Xesdeeni

[laserfan]

I can't believe I'm back here saying this, but IT'S FIXED! I have a perfectly sharp, smooth, and IN-SYNC 24.554 Xvid avi from my hosed-up ABC time-compressed 720p-to-1080i HDTV program! In the end, the keys I think were as follows:

1. I had to ignore the field order problems, and use DGIndex's uncorrected d2v
2. scharfis_brain's tdeint idea seemed to need "order=1" added
3. tdecimate was not working (left dupes and sync problems)so I switched to FDecimate instead. Here's what worked:

tdeint(mode=1,order=1,tryweave=true,full=false).FDecimate(rate=24.553735,threshold=7.5)

I'm still unclear as to what the "field order" errors (corrections) were doing, and why "order=1" was needed, nor do I understand "bobbing & weaving" so I have to study that some more. But I hope neuron2 sees this because FDecimate was very easy to understand and use; his instructions are wonderful and I even used his method to check my target fps math, not to mention that the metrics output showed me easily how to adjust the threshold for my video.

Processing was brutally slow, though 8 hours was an improvement over earlier attempts (10-12 hours each, maybe five previous tries???).

This one's a wrap. Thanks neuron2, and especially scharfis_brain, and Xesdeeni for your help & encouragement. I know now I don't have to settle for herky-jerky even if ABC messes with their broadcasts!!! :cool:

[Guest]

I think tritical might like to know about this as it may point to a possible problem with TDecimate() (or possible user error, which could be ameliorated with a doc change). Perhaps you could give him a courtesy ping. Regarding the field order correction, as I said in the user manual, sometimes it's just a minor change in a black scene and correction is therefore not required, and could ruin the rest of the following video. That's why the correction is no longer automatic.

[plugh]

Perhaps a small helper
sep=" "
dupthresh=2.5
dupcounter=0

mpeg2source("cap.d2v",cpu=0).trim(0,14984) # 720p clip

frameevaluate("dif = CFrameDiff(chroma=false,nt=1)", after_frame=true)
scriptclip("""( dif > dupthresh ) ? WriteFile("dupcnt.log", "current_frame","sep","dupcounter","sep","dif") : last""", after_frame=true)
frameevaluate("dupcounter = ( dif <= dupthresh ) ? dupcounter+1 : 0", after_frame=true)
You'll need TIVTC.dll for CFrameDiff, and you'll probably need a couple iterations to pick a dupthresh.
Then examine a long sequence with values safely above that threshold to find the cycle / pattern.

(Add a SeperateFields() if 1080i source to do field counting)

[Xesdeeni]

I can't believe I'm back here saying this, but IT'S FIXED! I have a perfectly sharp, smooth, and IN-SYNC 24.554 Xvid avi from my hosed-up ABC time-compressed 720p-to-1080i HDTV program! In the end, the keys I think were as follows:

1. I had to ignore the field order problems, and use DGIndex's uncorrected d2v
2. scharfis_brain's tdeint idea seemed to need "order=1" added
3. tdecimate was not working (left dupes and sync problems)so I switched to FDecimate instead. Here's what worked:

tdeint(mode=1,order=1,tryweave=true,full=false).FDecimate(rate=24.553735,threshold=7.5)

I'm still unclear as to what the "field order" errors (corrections) were doing, and why "order=1" was needed, nor do I understand "bobbing & weaving" so I have to study that some more. But I hope neuron2 sees this because FDecimate was very easy to understand and use; his instructions are wonderful and I even used his method to check my target fps math, not to mention that the metrics output showed me easily how to adjust the threshold for my video.

Processing was brutally slow, though 8 hours was an improvement over earlier attempts (10-12 hours each, maybe five previous tries???).

This one's a wrap. Thanks neuron2, and especially scharfis_brain, and Xesdeeni for your help & encouragement. I know now I don't have to settle for herky-jerky even if ABC messes with their broadcasts!!! :cool:As I mentioned above, my local affiliate cross-converts ABC's feed to 1080i. I have to inverse the telecine, and this seems to be too much for me. The problem is that the frame rate calculation just won't be exact, unless you count the entire film. I used DoubleWeave() to check two sets of 100/2 frames on a film, finding a pattern I calculated to 24.59076923 fps. I also had the exact pattern, so I used SelectEvery() to check it. Obviously it worked for a while, but a few tens of frames later, it failed. It eventually gets back into sync for a while. Unfortunately, an incorrectly inverse telecined frame is much more obvious than an occasional dropped or replicated one. So while I could probably accept a hand-calculated frame rate that was a bit off when decimating a 720p source, for a 1080i source, I'm screwed.

Xesdeeni

[laserfan]

As I mentioned above, my local affiliate cross-converts ABC's feed to 1080i.But Xesdeeni as *I* mentioned too, I also have Belo which cross-converts (Austin). Mine was a very ugly 720-to-1080-then-time-compressed mess. But the method worked...

...of course maybe I was just lucky in this case that the program kept the same (odd but consistent) pattern throughout; this is the only time-compressed program I've tried the method on.

[Xesdeeni]

But Xesdeeni as *I* mentioned too, I also have Belo which cross-converts (Austin). Mine was a very ugly 720-to-1080-then-time-compressed mess. But the method worked...

...of course maybe I was just lucky in this case that the program kept the same (odd but consistent) pattern throughout; this is the only time-compressed program I've tried the method on.Well, I've got several butchered broadcasts to choose from. Maybe I'd have better luck with a different one.

Can you post your full AVISynth script?

Also, what specific method did you use to determine the frame rate? I.e. what AVISynth script did you use for that part?

What did you use for inversing the telecine? I tried Decomb before taking a look through DoubleWeave(), but it made some incorrect field matches, so all of the progressive frames didn't in fact come out.

(It'd be great if someone could do an analysis as the movies are broadcast and post the frame rate.)

Xesdeeni

[laserfan]

Can you post your full AVISynth script? Also, what specific method...Sorry somehow Xesdeeni I missed your post, but here now is the method I used. First is the simple script to open in VDubMod in order to count the fields:

MPEG2Source("Grey's Anatomy 322-3 - The Other Side of This Life.d2v")
AssumeTFF()
SeparateFields()

It takes some patience of course to find a 10-30sec scene with movement thruout, and which clearly exposes the problems (BTW when I found the scene I wanted I made a clip of it to test with, using VideoReDo Plus), and then write-down the pattern you find (and double-check that it's right), but here again is the result I posted earlier. I was lucky as this show has a scene where the characters are walking a dog and so there was constant movement--there was just no way to miss if the dog's tail didn't wag smoothly!!!:
3132323232323232322232323232323232
3132323232323232322232323232323232

So for every 83 fields, 2 fields are removed:

59.94 * 2/5 * (83 + 2) / 83 = 24.553734939 (I used 24.553735)

It was scharfis_brain that led me to the above calc, but if you look at the docs for FDecimate neuron2 gave me another method to cross-check it. Next I had to determine the correct threshold parameter for FDecimate. It was (surprisingly) pretty high for this very clean HD program. IIRC I just stepped again thru the program adding this time FDecimate(metrics=true):

MPEG2Source("Grey's Anatomy 322-3 - The Other Side of This Life.d2v")
AssumeTFF()
SeparateFields()
FDecimate(metrics=true)

This is easier to do than to describe, as I stepped thru the critical segment of program, I found my threshold; so here is the final script that resulted:

MPEG2Source("D:\Grey's Anatomy 322-3 - The Other Side of This Life.d2v")
tdeint(mode=1,order=1,tryweave=true,full=false).FDecimate(rate=24.553735,threshold=7.5)
Crop(0,0,-0,-4)
BilinearResize(656,368)

Worked perfectly. I have not had another opportunity to try this but I'm glad you asked me to document it so I can refer to it again next time. Hope this helps...

[rebkell]

I realize this is an old thread, but it's the best one I've found on the subject. I finally got a PC capable of encoding x264 that doesn't take days, and I've got quite a backlog of hdtv shows in ts files that I've recorded, and I'm running into these strange cadences, House has them, Bones has them, Men In Trees has them, etc...

I just stepped through over a thousand frames from a House episode and I have yet to discern a repeating pattern, it is dotted with 4:3,3:3,4:2, but I haven't seen any 3:1 or 2:2's in it. Just wondering if anyone has found any kind of a compromise for these shows short of frame by frame editing.

[laserfan]

I just stepped through over a thousand frames from a House episode and I have yet to discern a repeating patternI have never seen this (and never recorded "House") but hopefully if we keep this thread alive someone will come-in and offer some advice.

My favorite show this year has been "Cane" on CBS which I re-encode to 720p XVID and thankfully it only requires a simple IVTC. Makes life SO much easier! :whew:

[rebkell]

Recorded LOST last night and the strange cadence is back, naturally. It looks like FDecimate(rate=24.553735 is the correct rate.

I have a question about the threshold in FDecimate. I normally use ColorMatrix(Hints=true) in my avs script, does this have any effect on the threshold value for FDecimate? I've probably read the answer, but I can't remember the answer. Basically would it be a better idea to put FDecimate in front of ColorMatrix or behind it or does it even have any effect?

[Adub]

I am pretty sure Colormatrix should not affect your FDecimate. They work on totally separate things.

However, you should place your ColorMatrix(hints=true) line directly after your source line, otherwise the hints will be lost. Just to be sure, make sure you are using the correct "info" parameter for DGDecode.

[plugh]

Recorded LOST last night and the strange cadence is back, naturally. It looks like FDecimate(rate=24.553735 is the correct rate.
Thanks for the confirmation - thought something was wrong with my cap. Mostly 212121 pattern, but then these breaks with 210121 or 212021 in no discernable pattern, and the cadence seems to vary in differant segments (wrote a script to record instances of three consecutive 'differant' frames). What a PITA!

[NorthPole]

sep=" "
dupthresh=2.5
dupcounter=0

mpeg2source("cap.d2v",cpu=0).trim(0,14984) # 720p clip

frameevaluate("dif = CFrameDiff(chroma=false,nt=1)", after_frame=true)
scriptclip("""( dif > dupthresh ) ? WriteFile("dupcnt.log", "current_frame","sep","dupcounter","sep","dif") : last""", after_frame=true)
frameevaluate("dupcounter = ( dif <= dupthresh ) ? dupcounter+1 : 0", after_frame=true)

@plugh

I tried your script to count dup frames to get a threshold value and I had a couple of questions.


ie...

2538 2 10.996362
2544 5 10.310538
2547 2 13.506885

I set the dupthresh = 10.0 on the example above.

The script produces a log file that lists the first frame number in a series of duplicates follow by the number of duplicates of that frame followed by dupthresh value that decimate would calculate in determining whether the frames are duplicates.

Is that correct?

so in the example... frame 2538 is the first of 2 duplicates (2538 and 2539) or are there 2 other duplicates of 2538 (2539 and 2540)?

So you can run and rerun the script with different dupthresh values to see how decimate would respond?

A lower threshold value would allow for more matches and a higher value would reduce the number of matches?

thanks.

Was thinking of using tdecimate(mode=1,hybrid=1,concycle=1,nt=1,vidDetect=0,dupthresh=?,vidthresh=?) (real monster)...to prevent jerkiness
but still eliminate as much telecine as possible.

Would use the dupthresh and vidthresh obtained from your script.

[plugh]

2538 2 10.996362
2544 5 10.310538
2547 2 13.506885

so in the example... frame 2538 is the first of 2 duplicates (2538 and 2539) or are there 2 other duplicates of 2538 (2539 and 2540)?No.
2538 was 'differant' from 2537 (by the indicated amount). It was followed by 5 'duplicate' frames - 2539 was similar to 2538, 2540 was similar to 2539, 2541 was similar to 2540, ... but 2544 was differant from 2543 (by the indicated amount) and we had detected 5 'dups' of 2538 leading up to that point
So you can run and rerun the script with different dupthresh values to see how decimate would respond?

A lower threshold value would allow for more matches and a higher value would reduce the number of matches?

thanks.

Was thinking of using tdecimate(mode=1,hybrid=1,concycle=1,nt=1,vidDetect=0,dupthresh=?,vidthresh=?) (real monster)...to prevent jerkiness
but still eliminate as much telecine as possible.

Would use the dupthresh and vidthresh obtained from your script.Not sure, I'd have to think about that application.

My intended use for that script was simply to automate the process people go through when manually stepping through and trying to identify a cadence. You can run the script across a whole movie, and quickly scan the output file looking for a nice *long* sequence (ie with sufficient motion) to exactly determine frame rate / decimation ratios -- instead of stepping through in virtualdub, making notes on a piece of paper as you go along, and hoping you picked a segment that is long enough to reveal things like 1000/1001 type ratios.

I mean, if you've got some basic cadence, say like 212121 but you notice a break in it like 210121, you can then scan the file looking for a reoccurance of the break, determine how many frames between them, and then check that number of frames further down (or up) to see of it reoccurs.

You know, I should probably add a second counter to that, incr'd at each 'differant' frame, and also output it. Then you've got both values - the 'telecined' frame number and the 'source' frame number. Not that you would use them as is, but the _difference_ between them (for example, at two occurrances of the 210121 cadence break) is what you need to determine cycle lengths / decimation ratios.

[plugh]

thought something was wrong with my cap. Mostly 212121 pattern, but then these breaks with 210121 or 212021 in no discernable pattern, and the cadence seems to vary in differant segments (wrote a script to record instances of three consecutive 'differant' frames). What a PITA!Curiouser and curiouser...

Further study has revealed something rather puzzling.

The places where the 2121 cadence breaks occur are *HIGHLY* correlated with scene changes :confused: :eek:

For example, it is going along 212121, about to do a '2' - ie 'F d d'

instead we get 'F C c'

The new frame 'F' arrives, but rather than it's two dups the scene change 'C' arrives out of step. Not only that but it 'resets' the cadence

Following 'F d d' should be a 'F d', instead we get 'F C c F d d F d' ...
ie the scene change yielded
210121
__FC__

Another example: Going along 21212 about to do a '1' - ie 'F d'

Instead we get 'C c c' and the cadence is reset, yeilding 'F d d C c c' - ie
212212
__FC__


Just to make it clear; I'm not saying that every scene change results in a cadence break, but that nearly every cadence break I've checked so far is associated with a scene change.

Has anyone seen anything like this before?

[rebkell]

Are we still referring to that 2-hour LOST repeat of last season's finale that aired a couple of weeks ago?

I'm encoding it right now, but I'm going to try it at a rate of 29.97. And if it doesn't look good, I'll probably try one at a rate of 25. I'll let you know how it looks, it will be a while before it completes. :)

[rebkell]

How does FDecimate handle the higher rate? How many frames will it examine and how does it determine which duplicates it keeps?

[rebkell]

Are we still referring to that 2-hour LOST repeat of last season's finale that aired a couple of weeks ago?

I'm encoding it right now, but I'm going to try it at a rate of 29.97. And if it doesn't look good, I'll probably try one at a rate of 25. I'll let you know how it looks, it will be a while before it completes. :)

Well, 29.97 looks pretty good, a little stiff in places, but overall a lot better than anything I've tried so far. I'm going to try 25 now, 29.97 didn't drop any frames that I could see. I'll be interested to see how 25 does. 25 should be much closer to the actual FPS and it should have a little margin for error, whereas if you figure out the exact FPS and the threshold isn't right, you'll have two glitches as it maintains. We'll see. I'll report back tomorrow on the results.

I have a question, I guess it really belongs in the avisynth forum, but most of this thread probably belongs in Avisynth. Is there anyway, to change the threshold value passed to FDecimate in a certain range of the clip.

Something like:
if(Frame > 1000) ThresholdVal=2.9
if(Frame > 3000) ThresholdVal=4.0
if(Frame > 10000) ThresholdVal=2.1
FDecimate(rate=25,threshold=ThresholdVal)

[plugh]

without putting much thought into it, a scriptclip?

scriptclip("fdecimate(rate=25,threshold=(current_frame > 10000 ? 2.5 : 4.0))")

don't know how fdecimate itself will react

[NorthPole]

@plugh

Thanks for your response... Very useful script.

[hkazemi]

I am pretty sure Colormatrix should not affect your FDecimate. They work on totally separate things.

However, you should place your ColorMatrix(hints=true) line directly after your source line, otherwise the hints will be lost. Just to be sure, make sure you are using the correct "info" parameter for DGDecode.

I put together a (long) color coefficients and Colormatrix usage summary that you may find useful:
http://forum.doom9.org/showthread.php?t=133982

ColorMatrix has a parameter to specify whether you are handling interlaced or progressive content, and this parameter must be manually specified.

ColorMatrix also should only be used when converting from HD to SD (< 720p) resolutions (or vice versa), or when the source material is in the wrong colorspace. I would not recommend using it blindly.

[rebkell]

Ran into another weird cadence on Jericho the other night:

21112111211121112111211121112111211111
21112111211121112111211121112111211111

Looks like it's keeping 38(dropping 9) out of every 47 frames, which would work out to 24.231 FPS. I only examined one section of it, but that section seemed to be consistent .... I wanted to cry, CBS has been so great, I think it's the first strange pattern I've ever run into on CBS broadcasts.