trouble encoding hybrid DVD

[dejected_is_me]

i have a hybrid DVD, Colorful from ADV films, that refuses to encode right. when i say hybrid i mean this:

If in DGIndex a source is switching between interlaced and progressive but is also being reported as completely FILM then don't worry - it's progressively encoded and Force FILM will restore the original frames. However, if it is switching and it's being reported as mostly NTSC then you have a hybrid dvd (just like the colorful disc I'm using below) and this will need to be handled in a different way, as I will describe later. Hybrid sources have two forms, one is where you have mostly FILM and only some parts interlaced and the other where only some parts will be FILM. These will be discussed in more detail later. Remember that some sources that only have a very small about of interlaced footage (such as FILM 98%) can often have Force FILM used.


I have the later. this is my script:

# PLUGINS
LoadPlugin("C:\PROGRA~1\GORDIA~1\DGMPGDec\DGDecode.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\decomb.dll")

# SOURCE
mpeg2source("C:\DVD encodes\Raws\Colorfull\d2v\color.d2v", idct=7)

# IVTC
Telecide(order=1, post=4).Decimate(mode=1, threshold=1.0)

# CROPPING
#crop(0,0,720,480)

# RESIZING
LanczosResize(720,480)


I can encode a small part of the film, about 300 frames or so, and it comes out ok. 2 pass with latest xvid build and MEgui. But when i encode the whole movie and play the resulting file all i get is a black screen. The size looks right and the time is accurate.

Someone please help me. Im not sure if this my script error or a megui error.

-dejected_is_me

[Mug Funky]

not sure what this has to do with avisynth, but...

sounds like a playback issue. does it open in virtualdub (is it an avi?)? what player are you using? do you have ffdshow installed?

what i'm trying to figure out is whether the frames actually ARE black, or whether the decoder is just messing up perfectly good frames.

considering filesize is right, i doubt the encoder is getting black frames (otherwise the file would be grossly under-sized), so i'll guess avisynth is working properly.

just out of curiosity, what percent film was the d2v given? AFAIK ADV do all their stuff as hard-telecine (that is, without enabling "inverse telecine" on their encoders) even if the content is pure film.

[manono]

AFAIK ADV do all their stuff as hard-telecine (that is, without enabling "inverse telecine" on their encoders) even if the content is pure film.

Which is why his statement that, "...if it is switching and it's being reported as mostly NTSC then you have a hybrid dvd..." ain't necessarily so.

[Mug Funky]

well, actually, switching from NTSC to FILM can also mean somebody cut corners and did an analog transfer, or the master has been dubbed a few times (almost all formats are lossy).

the pulldown detection in hardware encoders (at least the one i've used...) is incredibly conservative - i think it might want fields to be identical up to the 8th or 9th bits on a 10-bit source. which is why a lot of pure film DVDs show "98% FILM" instead of "FILM" - the encoder has to find pattern, but must not take the risk of IVTC'ing stuff that is not IVTC'able. if a master had an analog transfer, then the added noise is often enough to prevent IVTC in an encoder. Inuyasha is a good example of this - it usually reads about 70% film when it's closer to 99% (i don't think there's any 30p or 60i in it, but there may be some on the english title sequences). then there's stuff like Burst Angel which is 100% film but encoded as NTSC by ADV.

the only way to know is by watching it, unfortunately.

however, you can be pretty safe if you treat "mostly NTSC" as hybrid - that is, be aware that there may not be a telecine pattern all the way through, and so have some kind of postprocessing in place to catch any combs that get through.

i dream of an IVTC function that catches all kinds of video (telecine, 30p, 60i) and treats each type in the best way possible for the output medium. when i get my head around it i may write one, but it's a difficult problem.

[manono]

Are we defining hybrid differently? A hybrid has 2 or more different base framerates. For example, a lot of anime these days might have an opening song, and maybe some other bits scattered around the episodes, at 30fps progressive, and the rest is 24fps (ignoring the fact that they may have been drawn at 8, 12, or 16fps), either encoded as progressive (FILM) 23.976fps, or hard telecined interlaced (NTSC) 29.97fps, or a mix of the two. Those are hybrids, as I understand the term. You seem to be defining a mix of progressive film and hard telecined interlaced film as hybrid, which can be IVTC'd with greater or lesser difficulty. And yes, it's often a good idea to have the post-processor on.

So, when I said that a mix of NTSC and FILM when running DGIndex isn't proof of it being a hybrid, I meant that in many, if not most, such cases it can be IVTC'd, as the original framerate for the whole thing was 24fps. If it can be IVTC'd successfully, then it's not a hybrid, as I understand the term. Just because there might be some bad edits that break the pattern from time to time doesn't change the fact.

You'll notice, Mug Funky, that he's using Decimate with Mode=1, based entirely on DGIndex telling him that the majority of the video is NTSC. That'll keep it at 29.97fps. Now, if it's just hard telecined, without any true 30fps video in there, then that's a very poor way to handle it. As you mentioned, the only way to be sure is to examine it carefully. DGIndex sure as hell won't tell him what the source is like, but only how it was encoded.

[Wilbert]

So, when I said that a mix of NTSC and FILM when running DGIndex isn't proof of it being a hybrid, I meant that in many, if not most, such cases it can be IVTC'd, as the original framerate for the whole thing was 24fps. If it can be IVTC'd successfully, then it's not a hybrid, as I understand the term. Just because there might be some bad edits that break the pattern from time to time doesn't change the fact.
Hmm. I guess you remember our conversation about the definition of hybrid and vfr stuff (and things related to it). I took it to mean that hybrid and vfr is the same, but i guess you don't agree :)

Thus according to your definition:

1) native 29.97 fps stuff + hard-telecined stuff (29.97) = hybrid (although it is cfr) ?

2) native 23.976 stuff + '8, 12 or 16 telecined to 23.976' (although it is cfr) ?

If everyone agrees with this, i will change the docs a bit :)

[Guest]

To me, a hybrid is a mix of pulled-down material and non-pulled down material (where the pulldown can be of fields, as in standard 3:2 pulldown, or frames, as in frame duplication). It's not relevant whether the pulldown is hard or soft. My statement of it is equivalent to manono's.

What makes hybrids challenging is the need to decide what final frame rate to use, because the source has portions with different base frame rates. The base frame rate is the rate before any pulldown.

[manono]

Hi Wilbert-

Hehe, when I was writing earlier, I was wondering to myself what you'd think when you saw this. That's why I purposely wrote that I was ignoring for the sake of this discussion the 8, 12, and 16fps at which anime is often drawn. I don't think dejected_is_me is making a VFR encode. You saw the script, and he's making an XviD for a fixed framerate. Anime drawn at framerates below 24fps have duplicate frames inserted to bring them up to 24fps, and aren't telecined to reach 24fps.
1) Yes, that's hybrid.
2) That's also hybrid technically, but for the purpose of this discussion it can be treated as 24fps CFR.

I don't know this Colorful anime series. Maybe there is some native 30fps material in there. But you won't learn if there is just from what DGIndex tells you. You have to examine the source. In any event, if there is some true progressive or interlaced 30fps material, it's almost always a small percentage of the entire episode and I, for one, wouldn't make the whole thing 29.97fps, thereby ruining the vast majority that's 23.976fps and wasting valuable bits. But maybe that's just me.

[dejected_is_me]

wow this spawned a whole lot...

DGindex reports it, for the most part at 70 to 80 percent NTSC. The field order also changes from top to bottom pretty randomly and also switches between interlaced and progresive. It it my understanding that i couldn't give decimate an order variable or the other matieral in the interlcaced portion would come out wrong. Because of no ordering the framerate comes out 29.97.

The intersting part of all this is i can take i segment, from any part of the dvd and put it through the 2 pass with the script and it comes out ok. But when i do the whole file, I get the black screen.

Has anyone ever dealt with hybrid sources before, or have any idea of a proper script i can be using.

-dejected_is_me

[Guest]

DGindex reports it, for the most part at 70 to 80 percent NTSC. The latest DGIndex now reports Video versus Film. 70-80 percent video means that most of the movie is video and doesn't have 3:2 pulldown.

The field order also changes from top to bottom pretty randomly No. The field order does not switch. When you start a play or preview, it is determined and written into the dialog box based on the order of the first frame. It is not ever written again during a play or preview. If, however, you start previews from different spots, the order may vary, because you are jumping into pulled down sections, and if you start at certain points you cause a one-field phase shift.

and also switches between interlaced and progressive. That's not unusual.

It it my understanding that i couldn't give decimate an order variable or the other material in the interlcaced portion would come out wrong. There is no order parameter for Decimate(). If you meant Telecide(), yes you can give a proper order variable based on the order of the clip as saved to the D2V. You can use Parse D2V to get the right order from the D2V file.

Because of no ordering the framerate comes out 29.97. That is just totally incoherent. The output framerate depends on which Decimate mode you select.

But if you have 75% video, you're in a domain where applying Telecide() is dubious to start with. With some help from YATTA, though, it can be useful.

The interesting part of all this is i can take i segment, from any part of the dvd and put it through the 2 pass with the script and it comes out ok. But when i do the whole file, I get the black screen. This is difficult for us to address without having the entire stream. Try to strip it down. Does it fail with one pass. Does it fail if you trim a bit from the end with DGIndex? Etc.

Has anyone ever dealt with hybrid sources before, or have any idea of a proper script i can be using. I believe YATTA may be useful for this application.

[Wilbert]

Hehe, when I was writing earlier, I was wondering to myself what you'd think when you saw this. That's why I purposely wrote that I was ignoring for the sake of this discussion the 8, 12, and 16fps at which anime is often drawn. I don't think dejected_is_me is making a VFR encode.
Yes, i agree.

Anime drawn at framerates below 24fps have duplicate frames inserted to bring them up to 24fps, and aren't telecined to reach 24fps.
So, suppose you have 12 fps stuff for example, and you duplicate each second frame to get 24 fps. Isn't that telecining (ie 4:4 pulldown)? This is basically what neuron2 says i think ...

[manono]

Well, I could be wrong. It's happened before, and it'll happen again, I'm sure. When I think of the telecine process, I think of frames being added, but in the form of spread out fields, in order to lessen the jerkiness that comes from adding duplicate frames. With animations of all kinds you get both, duplicate frames as well as a telecine on top of that.

[Guest]

The MPEG2 specification encourages us to think of pulldown as applying to either fields or frames, because there is a mode in which the repeat flags can repeat frames. I've never actually seen soft frame pulldown but just as we don't hesitate to refer to hard field pulldown as "telecine", we ought not to hesitate to call hard frame pulldown as "telecine". Both are methods of bringing a lower base rate up to a higher rate.

[manono]

Cool, I stand corrected. Thanks, neuron2.

[Guest]

Cool, I stand corrected. Thanks, neuron2. It's a minor correction, or more accurately, addition. Your major insight stands.