Words About IVTC and frame schema

==[ DivXviD ENCODiNG OF SiLENTS AND/OR FUCKED UP DVDs WiTH AVS AND DECOMB ]==


- A Tale by Herr MACHiAVELLi -


+ Version 0.1 +

I decided to write down some notes on the encoding of silents and fucked up PAL or NTSC region
DVDs, since there have been some problems lately and people couldn't find out how to fix them.
What kind of fucked up PAL/NTSC DVDs do I have in mind? Only one category: bad mastered PAL ->
NTSC or NTSC -> PAL conversions. An extension of the theory applied on "restoring" this kind
of fuck-ups lead me to developing a method to encode silents.

Some notes before we start with the real job:
- PAL standard is 25 fps
- NTSC standard is 29.97 fps, but I round it up to 30 fps (easier calculations)
- FILM is 23.976 fps, and I rounded this up to 24 fps (for easier calculations)
- I assume that people know the basics behind telecining 24 -> 30 fps
- I further assume that you are aware of encoding utils as avisynth (avs) and Decomb. If not,
I advise to read things over at doom9's utopia (www.doom9.org). It should be explicitly
noted that the methods only work for Decomb (at least, that's the only thing I found
capable of doing the job right). Consequently, all presented methods are described for
avs in combination with Decomb.
- I use certain examples from the "DivX ;-) SCENE HiSTORY" to illustrate certain aspects of
the theory and its application in practice. These groups are not targeted whatsoever: I
don't have the intention to praise them, nor to bash them. They are just examples... :P
- It seems that up till now, few encoders are aware of what will be described in the following
text. I can only hope that those who are really interested in this technique, take the
effort and TIME to experiment, and that their experience will lead to an enhancement of
the DivX ;-) scene quality.

Table of contents:

1. Introduction
2. Bad NTSC -> PAL conversion
3. Bad PAL -> NTSC conversion
4. The art of encoding silents
5. Conclusion & final notes


1. iNTRODUCTiON
---------------

The latest TDX rules (TDX 2002) contain this small line: "MUST be as close to original
source framerate as possible." A simple soul will immediately think of the inverse telecining
(IVTC) that should be applied when dealing with NTSC DVDs (at least, if the DVD source is
not 100% NTSC or hybrid). But there are other possible scenarios, more exotic and pretty
rare, but they exist ergo encoded and kicked on somebody's ftp. These scenarios are worked
out in the following chapters...



2. BAD NTSC -> PAL CONVERSiON
-----------------------------

The normal case scenario for converting NTSC to PAL is as follows: The 30 fps NTSC source is
IVTCed to 24 fps FILM. The FILM source is speeded up to 25 fps, which complies to the PAL
standards. This works well if the source is telecined FILM. If the source is native NTSC, id
est shot at 30 fps (most notably concerts and TV shows), then IVTCing of the source is off
course impossible (this would give very choppy results).

Good conversion methods:
- NTSC -> FILM -> PAL => IVTCing of the NTSC source, followed by speeding up the FILM source
by 1 fps. This speeding up process causes a difference in runtime
between movies shown on TV in NTSC and PAL regions. This difference
amounts a factor 25/23.97; check the runtime difference for The
Matrix for instance: 136 minutes in the USA, 131 minutes in Germany.
Also note that the majority of companies have the FILM source available,
so they won't have to IVTC some obscure NTSC discs.
- NTSC(native) -> PAL => Since native NTSC cannot be IVTCed to obtain PAL, frames will have to
be merged together. Decimation of 5 fps is impossible, since it would
result in choppy movies. Merging of some frames can result in less
frames, but you'll have ghosting effects. Some good examples are the
Star Trek Enterprise DVDs. Those DVDs are hybrids in NTSC regions
(mixed FILM and native NTSC parts), but are 25 fps in PAL regions.
The parts where native NTSC parts occured on NTSC discs show clearly
ghosting artefacts on PAL discs. Anyway, there isn't a better method
to convert native NTSC to PAL.

So far, the good examples. Now, how can things go wrong? Simple. You take an amateurish DVD
company (the algouni kind) and a lazy ignorant engineer. Combine them, and tadaaaaa! Fucked up
DVDs. There are two sorts of NTSC -> PAL fuckups. In one case, you can do something about it
using Decomb. In the other case, it's just hopeless.

Bad conversion methods:
- NTSC -> PAL => as written above, NTSC sources can be IVTCed to FILM, unless the source is
native NTSC. Now, it can occur that some dumbass doesn't IVTC, and just merges
some frames to obtain 25 fps. In other words: he treats a perfectly IVTCable
NTSC source as native NTSC. It happens when a PAL DVD company uses an NTSC
region VHS as source. Don't believe me? Time to check out some ViPCO DVDs.
They are located in the UK, and they know absolutely nothing about good
quality DVDs. They are specialized in horror. Ban them!
- FILM -> PAL => as written above, FILM is converted to PAL by speeding up. This speeding up
is only noticed by a minority of the human population. Actually, you shouldn't
notice unless your name is FiFi or WooF. Instead of speeding up, some engineers
add 1 frame every second to the FILM source to obtain PAL. This frame is a duped
frame and can cause some choppiness. It seems to prevail in anime DVD companies,
but I've seen only very few cases.

In the latter case, the dupe frame can be removed using Decomb. The added frame is often a duped
frame (identical to the proceeding or following frame) or a merged frame (an interpolation of the
proceding and the following frame). Let's consider the following 25 frame sequence (0 = unique
frame, 1 = dupe frame):

[PAL SOURCE] - 00000 00000 00100 00000 00000 [25 frames/second]

Using Decomb(cycle=X), X can be a maximum span of 25 frames. But as you see, that works out
perfectly in our case. A typical avs script could look like this:

LoadPlugin("C:\RipTools\Decomb.dll")
LoadPlugin("C:\RipTools\MPEG2DEC.dll")
MPEG2Source("C:\DVD\SomeMovieILegallyObtained\vts_02.d2v")
Telecide()
Decimate(cycle=25)
BilinearResize(512,384)

The resulting DivX ;-) or XviD should look like this (0 = unique frame, 1 = dupe frame):

[AVi SOURCE] - 00000 00000 0000 00000 00000 [24 frames/second]

The runtime is the same and the added frame has been decimated. This means that you can restore
the error made by the lazy engineer of the DVD company. There should be a small increase in
quality. The first pass size probably (since I didn't test this myself yet) decreases with
abit less then 1/25th, or 4%.

How do I recognise these DVDs? Frame by frame checking of the source. That is all you can do.
DVD2AVi will NOT give you this information.



3. BAD PAL -> NTSC CONVERSiON
-----------------------------

Bad PAL -> NTSC conversions are a real pain in the ass. This type of errors is much easier to
find then the types described in bad NTSC -> PAL conversion. To make a good conversion, PAL is
slowed down to FILM, which then is telecined to NTSC.

Good conversion:
- PAL -> FILM -> NTSC => The PAL source is slowed down by 1 fps. There will be a change in
running time by factor 25/23.97; the obtained FILM source (24 fps)
is then telecined to NTSC (30 fps). Tadaaaaa! Doesn't this sound just
soooooo easy? It's just the inverse of NTSC -> PAL! Indeed. This is
a perfect demonstration of why some DVD labels are utter shit and
some or not. If you buy DVD by labels as MGM, Fox or Warner, you are
safe. If you buy a DVD by NeWCuLTLABEL, well... read further

Now, how can this go wrong? Well, let's imagine we are lazy engineers with brains that fit into
the fruits of the Myristica fragrans. Our boss just gave us this überleet copy of The Invasion
Of The Rapist Martians, but it runs at 25 fps. My boss just bought it on the black market in
Napoli... Now how do I have to make this compatible to NTSC devices? Hey! I know! Let's add
5 frames every second, so I obtain NTSC compatible source. Indeed, you obtain a movie that is
running at 30 fps, but I wouldn't call it an achievement. I'd call it murder. By adding 5 fps,
the source will become ugly as fuck and display ghosting effects all over.

Bad conversion:
- PAL -> NTSC => Instead of slowing down the PAL source to 24 fps and then telecining to 30
fps, the PAL source is converted to NTSC by adding 5 fps. These 5 fps can
be duped frames or the interpolation of its adjacent frames.

And how are we going to solve this problem? Yet again, Decomb comes to our aid! Let's consider
we have the following NTSC DVD (0 = unique frame, 1 = dupe frame):

[NTSC SOURCE] - 001000 010000 100000 100000 001000 [30 frames/second]

The above mentioned NTSC source has been obtained by adding 1 frame in every 5 frame PAL sequence.
5 of those sequences will yield NTSC (since 5*(5+1)=30). Now, let's remove them again. 1 frame in
a 6 frame cyclus has to be removed. A typical avs with Decomb could be as follows:
(...continues)

LoadPlugin("C:\RipTools\Decomb.dll")
LoadPlugin("C:\RipTools\MPEG2DEC.dll")
MPEG2Source("C:\DVD\SomeMovieILegallyObtained\vts_02.d2v")
Telecide()
Decimate(cycle=6)
BilinearResize(512,384)

The resulting DivX ;-) or XviD should look like this (0 = unique frame, 1 = dupe frame):

[AVi SOURCE] - 00000 00000 00000 00000 00000 [25 frames/second]

An example for this kind of fuckup is the R1 DVD of Kandahar. A movie that has been recently
ripped and should have been treated this way is Z. The runtime of the movie stays the same by
decimating the unnecessary frames, and you can increase the quality of your encode significantly.
The ghosting originating from the added frames disappears, and there is a decrease of abit less
than 15% in first pass size. Less frames per second to encode generally mean more available bits
per frame.

How do I recognise these DVDs? Yet again by frame per frame checking. DVD2AVi will NOT give you
this information, though in all cases I've encountered in the past, DVD2AVi told me the source
was hybrid or 100% NTSC.



4. THE ART OF ENCODiNG SiLENTS
------------------------------

So, what is special about silents? Silents have been made in times there were no or few standards
in the filming industry. Films were shot at speeds varying between 12 and 28 fps, and it even
occured that different reels from the same movie had different running speeds (e.g. Birth of a
Nation). High fps had the advantage that movements looked much more natural (silents often look
jerky on TV or on the big screen), but it could happen that the cinema's projecter got overheated
and caught fire. More then one pioneering movie theatre has been destroyed by overheated projecters.
Now, one problem of these silents is that we do have standards today. This means that a silent
with a non-standard fps has to be adapted to be shown on TV, or to be made available on DVD. The
classic trick was speeding up the movie. Sometimes wondered why Charlie Chaplin makes such funny
movements sometimes? Indeed, his comedy show has been shot at 18 fps and is now broadcasted at
30 fps. Another - and IMO a much better - method is by adding new frames. Adding frames can be
done in two ways for NTSC standards:
(1) Add dupe or interpolated frames till your source is 24 fps (FILM), and then telecine it to
obtain NTSC compatible source.
(2) Add dupe or interpolated frames till your source becomes 30 fps (NTSC).
For PAL, there are several possibilities:
- Dupe or interpolated frames are added till the source is 25 fps.
- The FILM source (with added or interpolated frames) is bought from a US based movie studio
and then speeded up by 1 fps.
- The NTSC source is bought and frames are blended to obtain 25 fps (this sucks, but there are
examples available... fuck you, Eureka).

Before I continue, a short note on DVD2AVi: don't trust it, NEVER! It can say your source is
FILM, 100% NTSC or hybrid, but in each of these cases, the original running speed could have been
18 fps. It just depends on how the movie source has been telecined. Actually, for silents, not the
running speed but the telecining speed is important. Sometimes, the telecining speed is mentioned
on the DVD box: Nanook of the North (Criterion) is 21.5 fps, Häxan (Criterion) is 20 fps and
La Passion De Jeanne D'Arc (Criterion) is 24 fps. Häxan has been released in the scene at 24 fps,
and thus based on technical aspects, is not TDX 2002 compliant.

Now, let's start with encoding some silents... The first problem is... how do I find out the
correct running speed? Easy answer: frame per frame checking. the easiest way is to write a small
avs that contains the Telecide() filter. Then load the avs and count the number of dupe frames
for a 30 frame sequence. For the silent The Cheat, this is the found sequence (0 = original frame,
1 = dupe frame):

[NTSC DVD SOURCE] 00101 00101 00101 00101 00101 00101 00101 00101 00101 00101 00101 00101

I count 24 dupe frames in a sequence of 60 frames, or 12 dupe frames in a sequence of 30 frames.
Since the DVD is NTSC, this means that 12 out of 30 frames are "useless" and can easily be
dropped. This means the runtime will be preserved and that the average quality can be increased,
since the same number of bits can be used for encoding less frames. The running speed of The
Cheat has been determined to be 18 fps. Now, how can I drop those fromes? Decomb! The "secret"
is to use the Decomb() filter more then once. In this case, all dupe frames can be achieved by
applying two Decomb() filters:

[Original source] - 00101 00101 00101 00101 00101 00101 (30 fps)
[Decomb(cycle=5)] - 0001 0001 0001 0001 0001 0001 (24 fps)
[Decomb(cycle=4)] - 000 000 000 000 000 000 (18 fps)

The avs for this encode could have been

LoadPlugin("C:\RipTools\Decomb.dll")
LoadPlugin("C:\RipTools\MPEG2DEC.dll")
MPEG2Source("C:\DVD\SomeMovieILegallyObtained\vts_02.d2v")
Telecide()
Decimate(cycle=5)
Decimate(cycle=4)
BilinearResize(512,384)

Another example: Parisian Love. This silent has been determined to have a running speed of 22 fps
based on the DVD source (0 = original frame, 1 = dupe frame):

[NTSC DVD SOURCE] 00010 00100 10001 00010 00100 10001

Hmmmm... How can I obtain 22 fps in this case? Easy, by applying two times the Decomb() filter:

[Original Source ] - 00010 00100 10001 00010 00100 10001 (30 fps)
[Decomb(cycle=5) ] - 0000 0000 0001 0000 0000 0001 (24 fps)
[Decomb(cycle=12)] - 0000 0000 000 0000 0000 000 (22 fps)

The avs for this encode could have been

LoadPlugin("C:\RipTools\Decomb.dll")
LoadPlugin("C:\RipTools\MPEG2DEC.dll")
MPEG2Source("C:\DVD\SomeMovieILegallyObtained\vts_02.d2v")
Telecide()
Decimate(cycle=5)
Decimate(cycle=12)
Crop(40,26,639,432)
BiLinearResize(512,384)

Wonderful, isn't it My personal experience is that adding up to three Decomb() filters is about
the maximum. More then 3 will slow down the encoding process a bit too much, even for the most
patient among us. An example of using 3 Decomb filters to obtain 17 fps from a 30 fps NTSC source:

Telecide()
Decimate(cycle=18)
Decimate(cycle=4)
Decimate(cycle=5)

Euuhm, nice, but how can I calculate how to apply these decimate() commands? I found a general
formula, that works both for NTSC as PAL sources:

FPS(silent) = FPS(source) * [(a(1)-1)/a(1)] * [(a(2)-1)/a(2)] * ... * [(a(n)-1)/a(n)]

for n cycles of Decimate(cycle=a(x)).

In case of the 17 fps example, this becomes:

FPS(silent) = 30 * [(18-1)/18] * [(4-1)/4] * [(5-1)/5]
FPS(silent) = 17

Voila, you can all start encoding silents now. But you should keep a couple of things in mind:
some fps can be obtained in different ways (1, 2 or 3 decimation steps, other orders etc), but
that doesn't mean it will get a correct result. Elucidation of the correct pattern is very
important. It is advised to look for the pattern first, then empirically write an avs/decomb
script and then use the formula above to check your result. The formula is also useful to check
all possible alternatives. I made an excel file that contains all posibilities up to 3 decimation
steps. I advise you to do the same if you are serious about encoding silents. Another note: some
running speed cannot be obtained by simple decimation steps. You can make an approximation, but it
will be hard or impossible to remove all dupe frames. Also, it appeared to me that NTSC silents are
much easier to encode then PAL silents.



5. CONCLUSiON & FiNAL NOTES
---------------------------

The conclusion is simple: if you build up some experience, there will be no problems with encoding
fucked up DVDs (at least the types mentioned here). All you need is patience This "article"
also proves that not only the DivX ;-) scene is a bunch of morons, but the DVD company "scene" has
plenty of idiots as well. I will not add my e-mail in this text, since I don't want to be bothered
by zillions of people which will always have the same questions. Those who need me will eventually
find me. I guess some of you already know me
Furthermore, this text is not copyrighted. It would be stupid and it wouldn't give me any extra
pennies anyway, bloody pirates! So, feel free to distribute the text. Just don't change the text
(or change it as you KNOW you can improve it).

Greetings:
- developers and encoding sources:: DA Graft (decomb.dll), A Lee (vdub), doom9, nando
and SBC developers, XviD Team, etc.
- encoding discussions: crypt0r, ccandela "the useless SI unit", fungus.
- entertainment: Deanna Brooks, Allbert Hofmann, Sassafras albidum, Suicide Commando and IC434.
[/nfo]


if somebod could explain me how to identificate video type of frame schemas i would be greatfull!
00001-10000-11000-01100-00110-00011 = ??? (pure ntsc???)

hope somebody drope me few words

cheers

[Acaila]

That's quite a story, and a very well done overview. I'm sure this will be useful to a lot of people.

This deserves a sticky