Strange FILM/VIDEO Percentage in Captured Cable/Broadcasts Shows

[CarlEdman]

I have been transcoding digital (mostly HD) MPEG-2 broadcasts for archiving in a more efficient format (i.e., x264 -> h264) and the FILM/VIDEO percentage (as reported by dgindex) somewhat puzzles me.

1. Many broadcasts have percentages like you'd expect (i.e., close to 100% video or 100% film). Fine.

2. Some broadcasts have 90%-100% film--I can understand that as film material with perhaps a like interlaced/progressive format intermixed for CGI/etc.

3. However, many broadcasts have very weird reported mixtures of 40-60% FILM/VIDEO.

A few possibilities can be excluded:

1. I am not talking about intermixed other material (like commercials). All of these materials are pure movies/tv shows with the extraneous material at the beginning or end cut off.

2. I'd be very surprised if there actually was a substantial intermixture of FILM/VIDEO in the source material. They are typically theatrically released movies with little or no noticeable CGI, or materials which have been fully generated in the same manner (e.g., fully CGI movies or hand-drawn animation).

3. I'm not talking about beginning or end material (e.g., opening or closing credits). Lots of material will hoover around 60% VIDEO or FILM during its entire run.

So how am I getting these weird percentages? Do broadcasters (or cable movie channels) partially hard-telecine 24p movies?

And what is the proper way of treating such material? I have been tfm.tdecimating them (after honoring pull-down flags in dgindex) on the assumption that the material was just partially hard-telecined, but I am not sure.

[Blue_MiSfit]

It's quite common for all material to have hard 3:2 pulldown. I've never seen a scenario as you describe. Broken cadence, sure...

Is this all 1080i60 stuff?

[CarlEdman]

Yes, it is all 1080i60 material. Good to know that hard 3:2 pulldown is not uncommon (which presumably will be fixed by tfm.tdecimate).

[MrVideo]

Yes, it is all 1080i60 material. Good to know that hard 3:2 pulldown is not uncommon (which presumably will be fixed by tfm.tdecimate).

All 23.976/24 fps source material is transferred to 29.97i/59.94p video via 2:3 pulldown.

And yes, if you want to get it back to 23.976, you will have to inverse telecine it.

You might want to look into the author's newer dgindexnv program: http://www.neuron2.net/dgdecnv/dgdecnv.html

I don't pay any attention to video/film content values. You have to know you source material.

2. Some broadcasts have 90%-100% film--I can understand that as film material with perhaps a like interlaced/progressive format intermixed for CGI/etc.

No such thing. All movies are produced at 23.976 fps, or true 24 fps. As mentioned above, when transferred for broadcast, 2:3 pulldown is used.

Keep in mind that pretty much all scripted TV shows are produced at 23.976 fps, or PsF. So, the 1080p23.976 masters are converted to 1080i29.97 using 2:3 pulldown. Same thing if converted to 720p59.94.

[Ghitulescu]

Animes, I presume? Then it's normal, go hunt for deinterlacing techniques specifically used for animes.

[jmac698]

I think you're missing something here, it's normal for shows to be time-compressed so random fields may be dropped. You can get some extremely odd patterns and the only way to IVTC them is to count through a hundred frames by hand and look for it.

[CarlEdman]

All 23.976/24 fps source material is transferred to 29.97i/59.94p video via 2:3 pulldown. And yes, if you want to get it back to 23.976, you will have to inverse telecine it.
I understand that this is necessary. However 2:3 pulldown can be done soft- (i.e., by marking certain fields as identical to previous ones and not repeating them) or hard (i.e., by repeating the extra fields in identical or nearly identical form). The former can easily be detected by dgindex and is, if I understand correctly, what is reflected in the dgindex reported FILM/VIDEO percentage. The latter can only be detected by visual inspection compensated for by cleverer teleciders (like tfm.tdecimate).

You might want to look into the author's newer dgindexnv program: http://www.neuron2.net/dgdecnv/dgdecnv.html

I have and would switch, except:
(1) I have a Radeon GPU, rather than an NVidia one, so dgdecnv would not be accelerated.
(2) Dgindex (or dgdecode) barely manages to move my CPU usage from idle, even with 1080p material being served off a very fast (400-500 MByte/second) SSD. Any performance improvement in dgdecnv would be negligible, compared to the time consumed by other filters and x264 encoding.
(3) Dgdecnv costs money. While I have used dgindex so much over the years that I'd gladly pay several times for having a few minor bugs fixed or features add (and in fact have offered bounties to do so), I don't really want a buy a program that I can't use and that wouldn't help me at all, even if I could.

I don't pay any attention to video/film content values. You have to know you source material.

I have used VIDEO/FILM content percentages for years to choose the processing of DVD-content with great success. And it works just fine on a majority of broadcast content, just not all of it.

[CarlEdman]

Animes, I presume? Then it's normal, go hunt for deinterlacing techniques specifically used for animes.

A small number of animes, but far from all or even most cases, in which I've observed this.

[CarlEdman]

I think you're missing something here, it's normal for shows to be time-compressed so random fields may be dropped. You can get some extremely odd patterns and the only way to IVTC them is to count through a hundred frames by hand and look for it.

Really? When paid, premium movie channels broadcast commerical-free high-def versions of recent movies, they do stuff this hackish just to make them fit neatly into time-slots? That is very disappointing and in such a caase, I have no idea how to properly IVTC, much less keep the audio/subtitle/etc. information.

[jmac698]

http://forum.doom9.org/showthread.php?t=122472&page=3

Jericho ... it's keeping 38(dropping 9) out of every 47 frames, which would work out to 24.231 FPS ... on CBS

Also problems with Grey's Anatomy on NBC, etc.
And I agree, sometimes the flags aren't accurate, it's hard telecined.

[Ghitulescu]

Really? When paid, premium movie channels broadcast commerical-free high-def versions of recent movies, they do stuff this hackish just to make them fit neatly into time-slots? That is very disappointing and in such a caase, I have no idea how to properly IVTC, much less keep the audio/subtitle/etc. information.

The broadcasters don't care about this stuff. They rely on HW gear, they are supposed to do the job, for the rest - who cares?

Secondly, at least in Germany, there's not enough bandwidth to carry out HD audio (there are several HD channels on the same transponder), and the video is reencoded to a much lower bitrate than on a Blu-ray. They simply feed the stream/s into a multi-stream encoder (an HW encoder that encodes several programmes at once to be directly aired on a single transponder, or on the allocated bandwidth, for subrented transponders) then air it.

Who cares about the quality of a premium channel? You? You are even not suppose to record them in the first place. :)

And if a restoration has been performed (ie someone took a lil'bit more care about the settings), well, this is a marketing tool, a bonus. "I do my job lousily, I'll get my paycheck. I'll do my job well, hmm, I'll ask you twice the money". Why would they do this for almost nothing?

[MrVideo]

I understand that this is necessary. However 2:3 pulldown can be done soft- (i.e., by marking certain fields as identical to previous ones and not repeating them) or hard (i.e., by repeating the extra fields in identical or nearly identical form). The former can easily be detected by dgindex and is, if I understand correctly, what is reflected in the dgindex reported FILM/VIDEO percentage. The latter can only be detected by visual inspection compensated for by cleverer teleciders (like tfm.tdecimate).

For material that comes from a DVD, you are correct, soft marking can be done. But, for broadcast material, either OTA, cable, or DBS, that can't be done. Each and every field (for 1080i) and each and every frame (for 720p), must be there, so only hard 2:3 pulldown can be used.

(1) I have a Radeon GPU, rather than an NVidia one, so dgdecnv would not be accelerated.

The cards are cheap.

(3) Dgdecnv costs money. While I have used dgindex so much over the years that I'd gladly pay several times for having a few minor bugs fixed or features add (and in fact have offered bounties to do so), I don't really want a buy a program that I can't use and that wouldn't help me at all, even if I could.

$15 isn't going to break the bank and it is a one-time cost for all updates to the program.

When you start dealing with H.264 sources, DGdecNV will be a must. DGindex will not handle H.264 video.

I have used VIDEO/FILM content percentages for years to choose the processing of DVD-content with great success. And it works just fine on a majority of broadcast content, just not all of it.

DVD content != Broadcast content, as stated above.

As mentioned previous, all scripted dramas are 2:3 pulldown converted for air. You can count on that being the case. I've yet to find one that wasn't.

[MrVideo]

I think you're missing something here, it's normal for shows to be time-compressed so random fields may be dropped. You can get some extremely odd patterns and the only way to IVTC them is to count through a hundred frames by hand and look for it.

The only network that time compresses a lot of their scripted shows is ABC. They definitely screw up the 2:3 cadence with the crap they pull.

I don't watch CBS, so I do not know about them. NBC, FOX and The CW do not time compress. The 2:3 cadence is intact for each segment.

I've not heard any complaints about the major cable sources.

[MrVideo]

Really? When paid, premium movie channels broadcast commerical-free high-def versions of recent movies, they do stuff this hackish just to make them fit neatly into time-slots?

I've not heard of the Pay channel times compressing. I don't know about movies, but original programming has not had any issues with IVTC.

[MrVideo]

Also problems with Grey's Anatomy on NBC, etc.

I've not tried to IVTC the few Jericho episodes that I got off air, yet. Most are from another source (better).

Sorry, but Grey's Anatomy is on ABC, the only offender that I know of the time compresses.

[MrVideo]

Secondly, at least in Germany, there's not enough bandwidth to carry out HD audio (there are several HD channels on the same transponder), and the video is reencoded to a much lower bitrate than on a Blu-ray. They simply feed the stream/s into a multi-stream encoder (an HW encoder that encodes several programmes at once to be directly aired on a single transponder, or on the allocated bandwidth, for subrented transponders) then air it.

What are you considering HD audio? The only audio allowed in the states is AC3, DD2.0 or DD5.1.

What you are talking about is a stat-mux encoder/multiplexer.

You are even not suppose to record them in the first place. :)

Not true in the states. Look up the famous Sony-Betamax case.

[Vurbal]

When you start dealing with H.264 sources, DGdecNV will be a must. DGindex will not handle H.264 video.
DgDecNV is not the only option for H.264. You can purchase DiAVC and DGAVCDecDI or use FFMS2, which is free but less reliable. Since I don't have an nVidia card right now, I've gone with DGAVCDecDI. The fact that I'll automatically have access to DGDecNV when I do eventually get a supported card made it an easy decision.

[jmac698]

I don't mean to sidetrack too much, but I don't see the point of dgindex anymore. Even a simple directshowsource works for me, and certainly ffms2 does. If it's a straight n-pass from start to finish, that is. Why bother with the extra step?

[jmac698]

Some mentions in that thread,
I'm running into these strange cadences, House has them, Bones has them, Men In Trees has them, etc... and Lost. There's talk of ABC affiliates owned by Belo who convert from ABC's 720p to 1080i and messing things up.
Mentions of ABC doing slowdowns and speedups. TNT-HD. More shows: Grey's Anatomy, House, Lost, Jericho.
Findings that cadence breaks at scene changes.
The odd show on CBS messed up.

I might mention also that I found a technique to improve the quality from HD captures. Record it twice, then take the best frames from each. You need ffms2 to do this, to see if it's I,B, or P. I found a complicated pattern that depends on the movie and compression cadence. Usually the B after an I looks good.

[CarlEdman]

For material that comes from a DVD, you are correct, soft marking can be done. But, for broadcast material, either OTA, cable, or DBS, that can't be done. Each and every field (for 1080i) and each and every frame (for 720p), must be there, so only hard 2:3 pulldown can be used.
Really? So what does it mean if dgindex reports that a broadcast captured show is, e.g., 90% or 100% FILM, as it sometimes does. How does dgindex detect that if there are no pulldown flags in broadcast content?
When you start dealing with H.264 sources, DGdecNV will be a must. DGindex will not handle H.264 video.
Maybe. But as my principal reason for bothering with any of this is to convert bulky mpeg-2 video to slender, high-quality h.264 video through x264, that'll be a while.

[wonkey_monkey]

No such thing. All movies are produced at 23.976 fps, or true 24 fps. As mentioned above, when transferred for broadcast, 2:3 pulldown is used.

That may be true for movies, and most likely all modern broadcasts, but I think Star Trek: The Next Generation was a mix of telecined film and minatures shot on video and shown at 29.97fps.

David

[jmac698]

while the show was shot on film, all the effects and post production was done on SD analog video

http://www.thedigitalbits.com/mytwocentsa183.html#081310
It's been remastered in HD, btw.

[wonkey_monkey]

It's been remastered in HD, btw.

The page you linked to only says they're "considering" it, but still, woo-hoo! I'm actually not sure whether I'm more excited about it being in HD or all at a single framerate...

David

[jmac698]

Definitely the news here is single framerate :)
You didn't read the update

may be the first place you'll finally see it - in HD - starting sometime in the fall.

Referring to Netflix, a North American video streaming service. They already have all Trek tv series available now (except DS9), and TOS and Enterprise in HD now.

[Blue_MiSfit]

A few words in defense of broadcasters - we're typically trying to meet the requirements of a client at the lowest possible cost. Also, remember that the true purpose of a linear TV channel is to sell advertising. Sometimes this means the client pre-processes their media through a time compressor (in order to meet scheduling requirements), and it always means we push their channel through a live hardware encoder. It would be quite impossible to broadcast a channel without re-encoding all your content.

Even if we start with high quality 50mbps 1080i MPEG-2, it's not feasible to concatenate all the elements that go into making a live linear channel. Consider that you have the actual program segments, commercials, promos, interstitials, etc, and more often than not these are sourced from different places, with different frame sizes, bitrates, and encoding settings. We also often have to apply real-time graphics (or overlay pre-rendered graphics), and it's the job of the playout server (see Omneon, Miranda, Snell etc for manufacturers of these uber fancy, enterprise grade, uber redundant media players) to string all this together, and output a final, master HD-SDI feed.

This then hits a (hideously expensive) broadcast encoder, which compresses and encrypts the feed, live, in hardware to either MPEG-2 or H.264, depending on the compression/transmission platform. Bitrates here are necessarily low, because of limited transponder capacity. Sure, it's technically possible to broadcast a sexy 18mbps H.264 1080i feed, but nobody wants to pay for that when the average Joe is hard pressed (the majority of the time) to tell a difference between this and an 8 mbps 1080i channel. You or I certainly can, but we're the annoying minority... Finally, this transport stream is muxed with other transport streams to fill a satellite transponder.

Don't even get me started on audio or closed captions. I'll have an aneurysm. Even worse, after the master feed is assembled, it's usually stepped on yet again by the distributor (DirecTV, Dish, Comcast etc) to fit their broadcast requirements. So, in the end, we've re-encoded media 3, 4, or even more times.

Broadcasting is naturally a series of compromises. That same pristine 50mbps XDCAM HD422 stream will be utterly brutalized by the time the air-path performs all its tasks, and it ends up on your set-top box. More often than not, client and budgetary demands override most technical quality considerations. This is slowly changing with the rising popularity of VOD, since offline viewing gives you the ability to do a nice job of preparing your source content.

So, yes, there's lots of shittiness out there, no doubt about it. It's a necessary evil. Bit by bit, things are getting better... Hey, at least there's (somewhat) watchable HD out there, eh? :devil:

Derek