Words About IVTC and frame schema

==[ DivXviD ENCODiNG OF SiLENTS AND/OR FUCKED UP DVDs WiTH AVS AND DECOMB ]==


- A Tale by Herr MACHiAVELLi -


+ Version 0.1 +

I decided to write down some notes on the encoding of silents and fucked up PAL or NTSC region
DVDs, since there have been some problems lately and people couldn't find out how to fix them.
What kind of fucked up PAL/NTSC DVDs do I have in mind? Only one category: bad mastered PAL ->
NTSC or NTSC -> PAL conversions. An extension of the theory applied on "restoring" this kind
of fuck-ups lead me to developing a method to encode silents.

Some notes before we start with the real job:
- PAL standard is 25 fps
- NTSC standard is 29.97 fps, but I round it up to 30 fps (easier calculations)
- FILM is 23.976 fps, and I rounded this up to 24 fps (for easier calculations)
- I assume that people know the basics behind telecining 24 -> 30 fps
- I further assume that you are aware of encoding utils as avisynth (avs) and Decomb. If not,
I advise to read things over at doom9's utopia (www.doom9.org). It should be explicitly
noted that the methods only work for Decomb (at least, that's the only thing I found
capable of doing the job right). Consequently, all presented methods are described for
avs in combination with Decomb.
- I use certain examples from the "DivX ;-) SCENE HiSTORY" to illustrate certain aspects of
the theory and its application in practice. These groups are not targeted whatsoever: I
don't have the intention to praise them, nor to bash them. They are just examples... :P
- It seems that up till now, few encoders are aware of what will be described in the following
text. I can only hope that those who are really interested in this technique, take the
effort and TIME to experiment, and that their experience will lead to an enhancement of
the DivX ;-) scene quality.

Table of contents:

1. Introduction
2. Bad NTSC -> PAL conversion
3. Bad PAL -> NTSC conversion
4. The art of encoding silents
5. Conclusion & final notes


1. iNTRODUCTiON
---------------

The latest TDX rules (TDX 2002) contain this small line: "MUST be as close to original
source framerate as possible." A simple soul will immediately think of the inverse telecining
(IVTC) that should be applied when dealing with NTSC DVDs (at least, if the DVD source is
not 100% NTSC or hybrid). But there are other possible scenarios, more exotic and pretty
rare, but they exist ergo encoded and kicked on somebody's ftp. These scenarios are worked
out in the following chapters...



2. BAD NTSC -> PAL CONVERSiON
-----------------------------

The normal case scenario for converting NTSC to PAL is as follows: The 30 fps NTSC source is
IVTCed to 24 fps FILM. The FILM source is speeded up to 25 fps, which complies to the PAL
standards. This works well if the source is telecined FILM. If the source is native NTSC, id
est shot at 30 fps (most notably concerts and TV shows), then IVTCing of the source is off
course impossible (this would give very choppy results).

Good conversion methods:
- NTSC -> FILM -> PAL => IVTCing of the NTSC source, followed by speeding up the FILM source
by 1 fps. This speeding up process causes a difference in runtime
between movies shown on TV in NTSC and PAL regions. This difference
amounts a factor 25/23.97; check the runtime difference for The
Matrix for instance: 136 minutes in the USA, 131 minutes in Germany.
Also note that the majority of companies have the FILM source available,
so they won't have to IVTC some obscure NTSC discs.
- NTSC(native) -> PAL => Since native NTSC cannot be IVTCed to obtain PAL, frames will have to
be merged together. Decimation of 5 fps is impossible, since it would
result in choppy movies. Merging of some frames can result in less
frames, but you'll have ghosting effects. Some good examples are the
Star Trek Enterprise DVDs. Those DVDs are hybrids in NTSC regions
(mixed FILM and native NTSC parts), but are 25 fps in PAL regions.
The parts where native NTSC parts occured on NTSC discs show clearly
ghosting artefacts on PAL discs. Anyway, there isn't a better method
to convert native NTSC to PAL.

So far, the good examples. Now, how can things go wrong? Simple. You take an amateurish DVD
company (the algouni kind) and a lazy ignorant engineer. Combine them, and tadaaaaa! Fucked up
DVDs. There are two sorts of NTSC -> PAL fuckups. In one case, you can do something about it
using Decomb. In the other case, it's just hopeless.

Bad conversion methods:
- NTSC -> PAL => as written above, NTSC sources can be IVTCed to FILM, unless the source is
native NTSC. Now, it can occur that some dumbass doesn't IVTC, and just merges
some frames to obtain 25 fps. In other words: he treats a perfectly IVTCable
NTSC source as native NTSC. It happens when a PAL DVD company uses an NTSC
region VHS as source. Don't believe me? Time to check out some ViPCO DVDs.
They are located in the UK, and they know absolutely nothing about good
quality DVDs. They are specialized in horror. Ban them!
- FILM -> PAL => as written above, FILM is converted to PAL by speeding up. This speeding up
is only noticed by a minority of the human population. Actually, you shouldn't
notice unless your name is FiFi or WooF. Instead of speeding up, some engineers
add 1 frame every second to the FILM source to obtain PAL. This frame is a duped
frame and can cause some choppiness. It seems to prevail in anime DVD companies,
but I've seen only very few cases.

In the latter case, the dupe frame can be removed using Decomb. The added frame is often a duped
frame (identical to the proceeding or following frame) or a merged frame (an interpolation of the
proceding and the following frame). Let's consider the following 25 frame sequence (0 = unique
frame, 1 = dupe frame):

[PAL SOURCE] - 00000 00000 00100 00000 00000 [25 frames/second]

Using Decomb(cycle=X), X can be a maximum span of 25 frames. But as you see, that works out
perfectly in our case. A typical avs script could look like this:

LoadPlugin("C:\RipTools\Decomb.dll")
LoadPlugin("C:\RipTools\MPEG2DEC.dll")
MPEG2Source("C:\DVD\SomeMovieILegallyObtained\vts_02.d2v")
Telecide()
Decimate(cycle=25)
BilinearResize(512,384)

The resulting DivX ;-) or XviD should look like this (0 = unique frame, 1 = dupe frame):

[AVi SOURCE] - 00000 00000 0000 00000 00000 [24 frames/second]

The runtime is the same and the added frame has been decimated. This means that you can restore
the error made by the lazy engineer of the DVD company. There should be a small increase in
quality. The first pass size probably (since I didn't test this myself yet) decreases with
abit less then 1/25th, or 4%.

How do I recognise these DVDs? Frame by frame checking of the source. That is all you can do.
DVD2AVi will NOT give you this information.



3. BAD PAL -> NTSC CONVERSiON
-----------------------------

Bad PAL -> NTSC conversions are a real pain in the ass. This type of errors is much easier to
find then the types described in bad NTSC -> PAL conversion. To make a good conversion, PAL is
slowed down to FILM, which then is telecined to NTSC.

Good conversion:
- PAL -> FILM -> NTSC => The PAL source is slowed down by 1 fps. There will be a change in
running time by factor 25/23.97; the obtained FILM source (24 fps)
is then telecined to NTSC (30 fps). Tadaaaaa! Doesn't this sound just
soooooo easy? It's just the inverse of NTSC -> PAL! Indeed. This is
a perfect demonstration of why some DVD labels are utter shit and
some or not. If you buy DVD by labels as MGM, Fox or Warner, you are
safe. If you buy a DVD by NeWCuLTLABEL, well... read further

Now, how can this go wrong? Well, let's imagine we are lazy engineers with brains that fit into
the fruits of the Myristica fragrans. Our boss just gave us this überleet copy of The Invasion
Of The Rapist Martians, but it runs at 25 fps. My boss just bought it on the black market in
Napoli... Now how do I have to make this compatible to NTSC devices? Hey! I know! Let's add
5 frames every second, so I obtain NTSC compatible source. Indeed, you obtain a movie that is
running at 30 fps, but I wouldn't call it an achievement. I'd call it murder. By adding 5 fps,
the source will become ugly as fuck and display ghosting effects all over.

Bad conversion:
- PAL -> NTSC => Instead of slowing down the PAL source to 24 fps and then telecining to 30
fps, the PAL source is converted to NTSC by adding 5 fps. These 5 fps can
be duped frames or the interpolation of its adjacent frames.

And how are we going to solve this problem? Yet again, Decomb comes to our aid! Let's consider
we have the following NTSC DVD (0 = unique frame, 1 = dupe frame):

[NTSC SOURCE] - 001000 010000 100000 100000 001000 [30 frames/second]

The above mentioned NTSC source has been obtained by adding 1 frame in every 5 frame PAL sequence.
5 of those sequences will yield NTSC (since 5*(5+1)=30). Now, let's remove them again. 1 frame in
a 6 frame cyclus has to be removed. A typical avs with Decomb could be as follows:
(...continues)