25fps Interlaced PAL - The madman creation.

[Aesop]

Howdy,

Firstly, Merry Christmas and happy new year to all...

Secondly, has anyone have any stories of encodes of Interlaced PAL DVD's... The horrible sort with the nasty 25fps conversion contining blended fields and a lot of garbage that turns into a jerky progressive stream no matter what you do to it?

having a bash at an anime DVD encoded with this sort of thing at the moment. My current protocol involves something like;


LoadPlugin("E:\c\GORDIA~1\mpeg2dec3.dll")
LoadPlugin("E:\c\GORDIA~1\avisynthplugin\decomb yv12.dll")
LoadPlugin("E:\c\gordia~1\avisynthplugin\convolution3d yv12")

mpeg2source("E:\Temp\xtest150seconds.d2v")

FieldDeinterlace(blend=false)
Convolution3d(preset="animeLQ")
crop(17,6,687,563)
BicubicResize(512,384,0.333,0.333)

With VirtualDubs 2dcleaner(10,1,1) putting its head in the door too. Output quality is well... as good as I can get it... and speed is abysmal.


What Have others tried with these things?

[hakko504]

Well, this is an awful situation. Try Smoothdeinterlacer and SpatialSmootherMMX. (both YUY2 filters) That combination usually gives quite good results.
Something like this:
LoadPlugin("E:\c\GORDIA~1\mpeg2dec.dll")
LoadPlugin("E:\c\GORDIA~1\avisynthplugin\Smoothdeinterlace.dll")
LoadPlugin("E:\c\gordia~1\avisynthplugin\convolution3d.dll")
LoadPlugin("e:\c\GORDIA~1\Avisynth_Spatial.dll")

mpeg2source("E:\Temp\xtest150seconds.d2v")

SmoothDeinterlace()
Convolution3d(preset="animeLQ")
crop(17,6,687,563)
BicubicResize(512,384,0.333,0.333)
SpatialSoftenMMX(2,4,6,false,false,4,4,6,8)

[Aesop]

have not tried Smoothdeinterlacer nor SpatialSmootherMMX before... will look into it...

The main problem though is the conversion from NTSC to PAL that Madman perform. When you look at the stream, at a sequence of even or odd fields, what you see is every fith field is a blend of the two surrounding it, and every 25th field is a duplicate of the 24th. This duplication is also the sign of a phase shift in the field order (top to bottom and vice versa).

Dunno if that makes sense ;-)

When played back in its native interlace state it looks fine. But as a progressive stream it takes on a certain fuzzed jerky kind of movement. Have been driving myself insane trying to rebuild the fields into something more amicable and in the end I gave up. Figured I could tolerate fuzzy edges if I smoothed out the rest of the picture too... And It is anime so a bit of jerky movement should not be too offensive to the eye. But now I am beginning to wonder if even this is too much of an ask.



P.S. Madman are an Australian company responsible for bringing much of the anime to the market around here. Whilst we love them for this we hate them for what they do to it.

[hakko504]

Interesting: Do you think you could make a short clip available? Just a few seconds with a lot of motion.

Maybe you should try Telecide(guide=2,post=false) #or blend=false instead of post=false
Decimate(cycle=25) as deinterlacing. That should take care of the duplicated fields every 25 fields and the fieldshifting. Oh, and I forgot: You should always use even numbers for the crop, especially in YV12! i.e. crop(18,6,686,562)

[Dreassica]

dont think that ull ever be able to get a true progressive stream out of that without blends etc. Tried a lot and deemed impossible. Not that i have probs with you having a go at it.

[hakko504]

I think that the latest version of Decomb with manual overrides actually should be able to be forced into removing most of the blending, and all duplicated frames. But like I said, I'd like a short clip to play with, at least 4 seconds and preferrably 15-30.

[Dreassica]

Ive got a sample VOB of aprox. 10 seconds of high motion. Its DBZ and a typical Madman PAL DVD release.



part 1 (http://www.geocities.com/majinbubbles2001/test.part1.zip)

part 2 (http://www.geocities.com/majinbubbles2001/test.part2.zip)

btw rename files to .rar and extract with winrar only, cause my site doesnt allow ra file types, just zip

[hakko504]

Thank you.

I just downloaded them and I will take a look as soon as I can. Expect a reply here within 24h. (It's midnight here and I'm about to go to sleep)

[DJ Bobo]

No matter how you shake it, this problem has been discussed 1000 times before, and the answer was: no way around normal deinterlacing using blend.

So use

BlendFields()

And that's it.
Add a sharpening filter if you want, like UNFILTER, to sharpen the image, as blending reduces the sharpness of the picture.
I recommend here blendfields and not fielddeinterlace, to avoid continuous sharpness variation within the video.

You want better picture quality? get the NTSC DVD.

[hakko504]

@DJB

I don't quite agree. I think it should be possible to get rather clean frames from using something like this on the clip dreassica posted:Mpeg2source("DBZ.d2v")
Smoothdeinterlace(doublerate=true)
Decimate(cycle=2,ovr="DBZ.dec")
SpatialSoftenMMX(2,4,6,false,false,4,4,6,8)I have not tested it properly yet, and in order to use it on larger clips you will need to get a version that supports patterns in the override file.

Blendfields will in my opinion make a bad situation worse, but that is a matter of taste.

[Dreassica]

Nice script, a pity that it wont work in avisynth 2.5 though :(

[DJ Bobo]

@ hakko
Anime PAL DVDs aren't suited for ripping anyway, I avoid them as much as I can, almost my whole collection is NTSC.
There is no way to get decent results using PAL DVDs (as said, no matter how you shake it), if they are not progressive or field shifted (which is rare when it comes to anime, unfortunately!)
People go ahead and other go back! almost all old anime shows are progressive, all new anime shows are interlaced (I mean not speeded up). This is weird! (at least in France!)

[EDIT]
I wanted to notice, that even professionals blend the fields when they deal with interlaced PAL video. I got original VCDs from the UK, and as you might know, VCD can only be progressive. Nevertheless, the anime wasn't speeded up, but blended all the way, go figure!

[^^-+I4004+-^^]

only solution would be to throw away blended field/frame and make
intermediatory field/frame (whichever is easier,probablay it's easier
to make frame...)
just in a manner that deinterlacers interpolate
in space only,this one would interpolate in time.......
but i doubt anyone can build that filter
(i guess by now they would if they could...)
although this stuff can be done and is done in top-grade
hardware converters
(yes,i've looked perfect ntsc->pal conversion for auto-races...
no blending was visible with separatefields,motion is smooooooth etc.)

[DJ Bobo]

@ i4004
What you're talking about is PAL Speed Up. They accelerate the video from 24 to 25fps (and audio by about 4%). This is the right method and gives perfect video quality (audio must be accelerated properly and pitch corrected, which is perfectly possible with high-end software like Prosoniq TimeFactory or hardware like Roland VP9000). Unfortunately, anime distributors don't care about quality like their big brothers, the movie distributors, so they tend to use the cheapest & fastest method, which is to use blends and stuff :(
What we get is a PAL DVD that looks good only on regular TV sets :o

[hakko504]

@DJBobo
I agree, PAL anime is the worst thing around. The creators of these DVD's should be shot.

@all
Ok, now I've had a little time to study the clip Dreassica posted, and it's full of unblended fields. In order to get something useful out of these fields, I've bobbed the video with Smoothdeinterlace, but it is possible to use bob as well. Then I simply sat down to look at every frame in order to find out which ones to keep and which ones to decimate. Then I created a decimation override file (attached along with the .avs) that could be used to recreate the 25 fps. It seems like it is working quite OK actually. I did however only do this for the first two seconds as this process is very slow, something that clearly limits the practical use of the method. I was thinking I should attach a clip here, but I couldn't make the filesize small enough, so those who wants to see it just have to dl dreassicas clip and reencode themselves.

@dreassica
Just change the smoothdeinterlace to bob, and change smoothing filter
and it will work in 2.5 also.

One note about Smoothdeinterlace: there seems to be a problem with acessing the first frame when doublerate=true, therefor the trim has been added. Do not remove it as it changes the numbering for Decimate.

[hakko504]

I just turned my brain off: I forgot the attachment

[^^-+I4004+-^^]

>What you're talking about is PAL Speed Up

not quite.....if you would read my post carefully you could see that i know exactly what happens with poor ntsc->pal transfer
(this is manga's worst nightmare....also i have seen quite of those that look bad even on tv.......i mean anime and ntsc tv movies too...they simply jerk ie. have blended fields.......sometimes even they have inproper conversion where last 5 frames (30->25fps) are discarded:i have seen this with my own eyes on local tv-station...it looks like 25fps stuff captured at less than 25fps....)
what's more i know how those lousy hardware converters work...(but i won't go into detail here..anyhow it's pretty obvious what happens...)

i've refered to ntsc sports broadcasts that were converted "on-line" to pal (for "eurosport" transmission..)
they looked like million $ when compared to the stuff with blended fields.....
( separatefields()&co. shows no blended fields )

so:if those manga companies could afford that equipment it could be done in proper way
(i strongly believe that those devices produce intermediate frames
to compensate for the lack of motion from throwing some frames out.......obviously for 30->25fps conversion something has to go!
these devices know what they can throw without thazard(lo-motion) ,and know when they have to introduce intermeditate frame for motion to look smooth (hi-motion).......there are some pdf's on that stuff available.....but it's highly technical mixed analog/digital broadcast stuff so i won't bother you........)
i'm just saying how this thing should be adressed....
( it's not like there are 10's of ways to combat blended fields...)

hakko may come with some solution,but that's not good enough.......
deinterlacing that stuff just sux..........( handpicking good fields/frames too.....you would have to DRAW new frames for it to look fluent...if it was fluent in the first place(original ntsc manga)..LOL! )

(but who knows.....perhaps he found some better way?huh?)

manga is not my pair of trousers,but any interlacing anomaly is interesting to me.......


[i'm quite happy with my tv-rips of pal-speed-ups...they have no problems what so ever!]

[hakko504]

@ ^^-+I4004+-^^
Well, I thank you for the confidence :p but unless someone (DonG? :rolleyes: ) can modify Decimate to automatically remove blended frames from a sequence I'm quite certain that well just have to live with these kind of horrible NTSC->PAL conversions. My personal preference is to do some form of deinterlacing followed by heavy spatial filtering. That usually produces quite acceptable results, and usually removes the worst blending.
Remind me to check eurosports conversion technique when I have the time ;)

[DJ Bobo]

@ i4004
I've read your post carefully, and was aware of the "auto-races" word, but was assuming you mean pal speed up, because until today, I never knew of a method to convert real 30fps to 25fps without blends ;)
Anyway, that eurosport method can't/shouldn't be used with anime.
AFAIK, the anime companies store the series on 30fps telecined tapes, to save costs (I guess, 24fps tapes are quite expensive). So the proper way would be to first IVTC to 24fps and then accelerate to 25fps (I'm ignoring real 30fps anime here, because they are rare)
One of two: the machines that can do that are too expensive, or anime distributors are just too lazy to do proper conversion ¬.¬
Again, why are most old PAL anime shows progressive, and most new interlaced, I thought machines get cheaper when time goes on :confused:

[^^-+I4004+-^^]

>I never knew of a method to convert real 30fps to 25fps without blends

probably because no one wrote avs filter for it(?)
hehehe
there are some very sophisticated machines that can do
it....
(i have seen pal transmission of live ntsc event that looks
"smooth as a sunday ride" (i belive that's the expression..hehe)

i'm pretty sure it was ntsc as it was nascar race (hehe)
and i think it wasn't progressive camera that was used (dtv is underway
in usa...) because it looked the usual lo-res-ish ntsc way....

(btw. i have looked some stuff on recent olympics .and i wondered.....hmmm have these americans improved their resolution or what.....i would bet it was hdtv transmission and even down-converting it to pal it looked smashing!)

>Anyway, that eurosport method can't/shouldn't be used with anime.

hmm,yes i know what you anime fans are speaking:first IVTC-it to 24,and then speed it up to pal.....obviously it's too much work for those companies..hehe
(huh i'm reading as i'm writing...obviously you know right way it should be done.hehe)

btw. do you know how many fps anime's really have?
(i mean how many different drawings are there in one second of original (film) manga)
that would be interesting to know....
i doubt it's 25 different images.....with cartoons that's not being done.....

>AFAIK, the anime companies store the series on 30fps telecined tapes, to save costs (I guess, 24fps tapes are quite expensive).

hmm...why?it's the same tape but only on different speed etc.
proper way to do it is to HAVE film source (it's done on film,right?anime's originate on film?)and speed it up for PAL,and pulldown to NTSC.......

>Again, why are most old PAL anime shows progressive, and most new interlaced, I thought machines get cheaper when time goes on


that stuff smells like piracy to me: companies taht produce dvd's get ntsc videotapes instead of getting film negatives.....
and they are thinking:this is cartoon,no one will know if we convert ntsc to pal witht some lousy vintage (tv stations discarded?) equipment.........
hehehe
(i would kill someone if my "futurama" was killed in that way!...hehe)

also i think telecine equipment is some of the priciest stuff on market.....except maybe for tv-transmiters themselves.......
that stuff is MEAN i tell you.......
(kodak is making ccd reader,philips is making electronics for "spirit data cine"....you know it makes sense!..hehe)

[DJ Bobo]

99% Anime is 24fps. 24fps is vital for panning scenes. Normal scenes are most of the time drawn @ 12 or 8fps though.

I don't think 24fps and 30fps are the same. I think there is different standards: a 35mm FILM tape is a whole other thing than a regular 30fps NTSC tape.

What DVD companies become or not, is another big mystery. I've got some genuine japanese DVDs from the same show (I mean the real things, not Hong Kong bootlegs). One DVD contains some episodes, the other contains one of the movies of the anime. The episodes were encoded 30fps interlaced (telecined from 24fps of course), the movie DVD was encoded 24fps progressive, although both from the same company!! go figure!

[^^-+I4004+-^^]

Originally posted by DJ Bobo
99% Anime is 24fps. 24fps is vital for panning scenes. Normal scenes are most of the time drawn @ 12 or 8fps though.

I don't think 24fps and 30fps are the same. I think there is different standards: a 35mm FILM tape is a whole other thing than a regular 30fps NTSC tape.



ok thanks for the fps info....

also you can't say "film tape".....
we can speak in terms of film (it's 24fps .period.)like celuloid

video tape (ahh that's the term:video tape!hehe)
can be for any standard and is esentially the same (same composition)
only duration (length)is adjusted according to the VTR speed/tv standard etc.

anyhow.......i can't see the logic behind this:
>AFAIK, the anime companies store the series on 30fps telecined tapes, to save costs (I guess, 24fps tapes are quite expensive).

why should pal video tapes be more expensive than ntsc video tapes?
what's the 24 or 30fps tape anyhow?
it can be either pal or ntsc........
i didn't quite catch your drift there........
do you mean "master tapes" from which other things are being made?

[Guest]

Originally posted by hakko504
...unless someone (DonG? :rolleyes: ) can modify Decimate to automatically remove blended frames from a sequence I'm quite certain that well just have to live with these kind of horrible NTSC->PAL conversions. There is one improvement that we should explore. That is to allow a mode of field matching whereby if the three matches do not produce a valid result, the matches to the other field are checked as well. So we'd have a 6-way search instead of a 3-way one. The idea is that since Telecide normally matches on the bottom field, if that bottom field is a blended field, there will not be a match. But if you fall back to look for a match to the top field you might find a valid match for that frame. It seems that it could at least reduce the number of blends coming through to the output. The VirtualDub version of Telecide does this.

[hakko504]

@neuron2

Interesting, and in this case it could be useful as a lot of frames are unblended. I haven't checked the VD telecide, but do you allow same match to appear twice? Like frame 1 being matched as bottom->next and the in frame 2 you match it as top->previous.

[Guest]

Originally posted by hakko504
@neuron2

Interesting, and in this case it could be useful as a lot of frames are unblended. I haven't checked the VD telecide, but do you allow same match to appear twice? Like frame 1 being matched as bottom->next and the in frame 2 you match it as top->previous. That's a good point. The VirtualDub Telecide would reuse matches. A better implementation would avoid that, as you suggest.

[DJ Bobo]

@ i4004
By 24fps tapes, I mean tapes like those used for hollywood movies, call them master tapes or whatever you want, I think you guessed what I mean.
Anyway, the episodes were encoded telecined, the movie was encoded progressive. I also noticed that the episodes have much less noise than the movie, so the master tapes are subsequently different, that's for sure.

[^^-+I4004+-^^]

>Originally posted by hakko504
...unless someone (DonG? ) can modify Decimate to automatically remove blended frames from a sequence I'm quite certain that well just have to live with these kind of horrible NTSC->PAL conversions.

i've seen manga's with lots of blends:
in that case it's not enough to remove the blends only as
fluent motion is lost with them or without them(even when watched on tv it's lousy motion-having in mind that dj bobo said that pan scenes get pure 24fps! i hope that's true for manga... )..that's what i'm saying.....
there's timeline missing on that ntsc->pal conversions and only removing blends alone won't help....
that's why i've talked about doing "frames interpolation" on the timeline itself...but that seems like asking too much out of developers or systems...(every line (of interpolated frame) would have to be calculated from the line of previous and following frame so it has mean value->mean
position in the timeline...that filter would take forever to do 1h of movie)
for example_(1 line of subsequent frames="-"):

frame1:*-
frame2:**- (interpolated)
frame3:***-
frame4:****- (interpolated)
frame5:*****-
(in this example frame2's line and frame4's line are calculated from lines in previous and folowing frames (ie. from line in frame1&3 and
from line in frame3&5 respectively....originally frame2 and frame 4 were blends with no motion on thier own and are substituted with interpolated lines from adjacent frames!)

[note: * is instead of <SPACE> because it was formatted wrong in first attempt..therefore edit]
i hope now it's clear?

> so the master tapes are subsequently different, that's for sure.

i think you're mixing standards for tv and standards for production here:
1 there's film (celuloid) that's original
2 there's master(magnetic) tape that's been telecined from film itself (that's first copy
of the film on another medium..)
3 there are subsequent copies (also magnetic tape) for different standards:there's pal version,there's ntsc version etc.
these originate from the "2"...ie. from the master magnetic tape....

and for sure it's better to have "2" than "3" as a source for dvd!
i think "2" is universal format for interchanging of telecined video (one for all world ....off course another manufacturer of telecide machine may have another standard etc. "rank cintel" makes such machines) and "3" isn't and it's tv-standard particular...

so if your manga production company starts with ntsc magnetic tape ("3") sure it's worse than doing it in proper way.......

(i've seen that some telecine equipment has cca.240,000£ price tag..etc.)

cheers

Ivo

[Aesop]

For what it is worth

Been trying a whack of things with this interlaced PAL anime stuff... The conclusion; there are three options. Firstly a blended deinterlace which will yield smooth panning but god awful hard edges. Secondly a non blended deinterlace which will yield jerky pans but fairly good results elsewhere. Thirdly, and this was the prefered option, to remove de-interlace entirely and shrink resolution down to half that of source (i.e 360x288)...


This stuff is impossible to work with :angry: .