Problem with a PAL Interlaced Source

[Suiryc]

Hi

I recently obtained an anime DVD (Zone 2, PAL, 25fps). DVD2AVI says it's Interlaced (which is true according to the preview window).
Using Gordian Knot (last version : 0.26) I checked the FieldDeinterlace option while saving the AVISynth script.

Of course this worked well (no remaining interlaced artifacts), but on moving parts of the picture objects seems kinda doubled (I'm not sure but I think it isn't really what people call "ghost effect"). Watching the resulting avs file in VirtualDub it appeared that a lot of frames have this problem (it seems one frame of the avs script is the result of the addition of two frames of the original clip).

I tested pure deinterlace filters and they all gave me similar results.
So I read various posts in this forum about DeInterlace and IVTC, and had a look at tools like Telecide (Decomb 3.8), GreedyHMA or ivtc.

In particular I used Telecide() (or Telecide(guide=2,...)), which gave me good results for my problem (less "doubled" pictures, and so more details in the frames), except it regularly introduced duplicated frames.

I tried Decimate(mode=0/1,threshold=50) but it doesn't solved my new problem.
I searched a lot (in this forum and over the web), and tried the Decimate(cycle=25) trick (for PAL of course).
The problem is that it is not only one frame that is duplicated every second (as it should be in an utopic world), but two or three consecutives frames that are duplicated, and not allways every 25 frames :(.

Does anybody else experienced this problem and found a solution ?
Or maybe I'm doing something wrong here.

In any case please help me, I would like to keep the best quality for encoding this anime :).

[Acaila]

I have experienced the exact same problem, and as far as I know there is no solution. FieldDeinterlace will get rid of the interlacing nicely, I've learned to live with the leftover doubled-frames output.

[hakko504]

Ghosting can be eliminated by setting blend=false in fielddeinterlace, or use GreedyHMA for de-interlacing.

[Acaila]

It's not ghosting (at least not with me), the original fields are sometimes two fields blended together. And it doesn't happen at a regular interval, otherwise decimating might be a solution.

[hakko504]

I wasn't very clear when I wrote the other post.
Setting blend=false will not introduce any new ghosting. If the source itself has ghosting the situation is much worse. From the original description of this thread it was not possible to decide if the ghosting was introduced by fielddeinterlace/telecide or if it was present on the DVD.

[manono]

Hi-

If I read Suiryc's original post correctly, he's claiming that Decomb is introducing new progressive frames which are duplicates of the previous frames. If I'm correct in that interpretation, I'd say that's impossible. That would imply (unless it's also dropping frames along the way to compensate), that the movie would wind up with more frames than it started with, would be longer, and would have progressively asynch audio.

As we all know, anime often depicts motion as move, pause, move, pause. Or even move, pause, pause, move, pause, pause. That might explain the duplicate frames he's seeing. I think he should check the source to be sure. My deduction was based on this phrase:

except it regularly introduced duplicated frames.

That seems different from the blended fields Acaila's talking about. But I could be wrong in the interpretation. And I've had no experience with PAL material (although I've seen some nasty PAL to NTSC conversions).

[Suiryc]

Originally posted by manono
Hi-

If I read Suiryc's original post correctly, he's claiming that Decomb is introducing new progressive frames which are duplicates of the previous frames ...

As we all know, anime often depicts motion as move, pause, move, pause. Or even move, pause, pause, move, pause, pause. That might explain the duplicate frames he's seeing. I think he should check the source to be sure. My deduction was based on this phrase:

except it regularly introduced duplicated frames.


OK I have to apologize for the misunderstanding but it's sometimes difficult to explain things in a foreign language. I'm not saying that Decomb introduced (i.e. added on purpose) new frames (that were duplicated from the original clip), but that (almost) regularly there are 2 or three frames duplicated in the final video.

However the motion (from the avs opened in VirtualDub) is not like move, pause, move, pause ... or move, pause, pause, move, pause, pause ... (which is quite regular), but something like :

move (20 times), pause, move, pause, move, pause

This pattern appears two or three times, and then it is

move (19 times), pause, move, pause, move, move, move

and then

move (18 times), pause, move, pause, move, pause, move, move

and so on.
And this is not quite regular (as you can see) and clearly visible when things are moving in the video. :(

[Suiryc]

By the way I tested those two scripts :

mpeg2source(c:\... .d2v)
SeparateFields()
SelectOdd()

and

mpeg2source(c:\... .d2v)
SeparateFields()
SelectEven()

Both showed "doubled" frames, but not at the same time : when the image with the first script is good, the image (at the same time) in the second clip is "doubled" (two frames are mixed in one), and when the image is "doubled" in the first clip, the image in the second clip is good.

I guess this means the problem comes from the original clip?

Well after all I think I will have to cope with this effect and continue using FieldDeinterlace as Acaila does ...
(aaaah if only there were a tool able to show the clip like my Hollywood+ does ...)

[manono]

Hi-

My apologies Suiryc. I didn't give you enough credit. So if you are talking about the blended fields, same as Acaila, then welcome to the cruel world of anime encoding, where nothing's easy, and everything can go wrong.

You'd think there'd be a way to get Telecide to ignore the blended fields and match on the good fields, but if there is, I haven't found it. Using Telecide(Reverse=true) just can't work when the blended fields are mixed up between the even and odd fields. My guess is you have a bad NTSC to PAL conversion, and nothing can be done.

[trbarry]

To the extent the blended fields are from a rate conversion you would think they would occur regularly. If so then some combination of SeparateFields and SelectEvery() dropping pairs of fields should get rid of them.

But I don't know what you would do after that and if they don't even seem to occur at regular intervals then I'm pretty well stuck anyway.

Donald Graft once mentioned he could recognize blended fields just because they cause more weave artifacts but I've never come up with a really good way if they aren't in a pattern.

- Tom

[avih]

tom!, i'm already with the 'reply' window open, and i see u just posted a reply. with nerly the same words as i intended to ;)

Suiryc, that sounds like an interesting problem, and basically, i agree with tom, and understands donald's idea (although i thought of different implementation). can u upload a short clip (about 30 seconds) compressed with xvid, with 'interlaced' checked of your ORIGINAL sequence? try to select a scene with much action. you can crop to a smaller resolution (i.e. 320x240 in the center is ok), just make sure the effect is visible with your resulting clip. u can use low bitrate such as 300-400kbps so it's easier to download.

if i'll find some time, i'll have a look at it since it sounds interesting.

cheers
avi.

[trbarry]

avih -

It would be great if you could find a solution for this. It seems everybody and his brother is encoding with some sort of blended frames these days, especially when there is any sort of rate conversion and it makes for some ugly results.

In cases where frames have been added to speed up the video it would be nice to know if there is any usual pattern to the blended fields/frames so the extra ones may be dropped somehow.

But once any given field is blended I'm not aware of anything useful that can be done with it besides deleting it.

Anyone have any ideas?

- Tom

[Suiryc]

Originally posted by avih
can u upload a short clip (about 30 seconds) compressed with xvid, with 'interlaced' checked of your ORIGINAL sequence? try to select a scene with much action. you can crop to a smaller resolution (i.e. 320x240 in the center is ok), just make sure the effect is visible with your resulting clip. u can use low bitrate such as 300-400kbps so it's easier to download.

if i'll find some time, i'll have a look at it since it sounds interesting.

cheers
avi.

with 'interlaced' checked ?
Do you want me to use this script :

mpeg2source(... .d2v)
crop(218,168,320,240)

and then use the "Interlacing enabled" option in XviD (I get the latest version from Koepi's website)?

Because I tested with and without this option and the resulting clip is interlaced in both cases.

By the way what is the effect you want to look at? The fact that frames seem doubled when I deinterlace (with FieldDeinterlace for instance), or the "strange" stuttering of video (when I use Telecide)?

[avih]

Suiryc: yes, that's the way i ment. interlaced should be checked since that way the interlaced nature of the clip will be preserved better, regardless of the low bitrate (i.e. it won't blend the fields because of the low bitrate, but will encode each of them separately).

i would like to have a look on the blended ORIGINAL fields, try to find a pattern, try to recognize the original framerate and possible framerate conversions, and eventually, try to find a method to restore the original clip.

tom: the method i thought about was to use the usually low framerate of the clip (2-3 frames/fields have the same image many times in anime), and try to produce a 'delta' of a blended field from the previous and next field, and that way to recognize this blend.... but it's just a general idea.. i'll have to look at the clip to see if i'll really have a practical idea.

thx
avi

ps, i asked for the crop and the reduced bitrate since i'm using 56k modem, and downloading files bigger than 3-5M is starting to be a hussle for me :(

[Suiryc]

OK here is 24s of one of my clips (for instance the opening part).
The avs script used :

LoadPlugin("C:\PROGRA~1\GORDIA~1\mpeg2dec.dll")
mpeg2source("... .d2v")
crop(218,168,320,240)

(I have GordianKnot version 0.26)
I used VirtualDub (Fast Recompress) and XviD with those settings :
1-Pass CBR : 320 kbps
"Enable Interlacing" option checked (in the advanced settings pane)
All other options are the defult ones.

The file is about 1Mb large.
Hope the attachement will be OK.

In the meantime I had a look at this page (http://www.100fps.com/) about deinterlacing. I noticed it one week ago (while I was searching over the web), and I saw doom9 mentioned it in a recent news. During my first tries AVISynth gave me errors :(

However now this script :

mpeg2source("... .d2v")
SeparateFields()

combined with the "Deinterlace - Smooth" filter in VirtualDub seems to give good results ... (even if the final framerate is now 50fps)
Maybe it could be a solution for Acaila too.

[avih]

thx. downloading now. i'll let u know if i find something.

avi.

[morsafr]

For me it is impossible to reconstruct the original information.

I'm French and I know those PAL Anime DVDs (this one is from Declic Image, right ?).

As it has already been said, the problem relies on the NTSC to PAL conversion.

Instead of doing a proper conversion in two steps :
-1) NTSC interlaced 29.97fps -> FILM progressive 23.976fps by inverse telecinig (restoring the original frames)
-2) FILM progressive 23.976fps -> PAL progressive 25fps by simply speeding up the video and audio

They do a one step conversion :
-NTSC interlaced 29.97fps -> PAL interlaced 25fps by simply discarding fields or frames (thus deleting precious information)

Doing so they erase the information necessary to restore the original frames. That's why I think it's impossible. If avih find a proper solution to this problem I shall pray him :o)

[Suiryc]

I'm French and I know those PAL Anime DVDs (this one is from Declic Image, right ?).
"Bien deviné" :)

[B]If avih find a proper solution to this problem I shall pray him :o)
Me too :devil:

PS : as I said the solution given on the www.100fps.com seems to work (I'm presently encoding ... but with the tools I have to use and the fact the final framerate is 50fps, it takes 5h - for my Athlon 900 - to encode this 20min clip in 2-pass mode :( )

[Acaila]

I get 12fps with that method, which is freakin' slow I agree :D

However, I don't think that will do the trick. I think you'll still end up with those blending artifacts, because the fields themselves are blended.

I think the best way to deal with this is to speed it up to 50 frames/sec and decimate the frames that are still blended. But this could result in a choppy-playing movie. Maybe convert that back to 25fps, but I think that would take us back to square one :)

[vinetu]

Hi Suiryc !
I have take a closer look at your TESTclip and my decision is :
the movie was composed with the "help" of MotionPerfect or Realviz Retimer or similar
application (which can produce 30 fps from 12 fps source for example).
More ! - I can clearly see that there are at least 2 different layers morphed separately ( in this case - morphed = time stretched by
producing extra non existing in source frames ) and then mixed - Foreground and Background ...
So I think - this is non PAL<->NTSC issue ...
But may be I'm just a stupid cretenium ...

vinetu

[Suiryc]

@Acaila :

At last the encoding is finished :). Maybe it's my eyes but the playback seems really smoother.
I opened the final clip in VirtualDub to see, and yes there are still blended frames but less than with using FieldDeinterlace : with FieldDeinterlace almost every other frame was blended, and now it is almost 1 frame over 4 (due to the 50 fps framerate ?).
So as you see the same number of blended frames per second, but 2 times more frames (25fps -> 50fps), you don't notice those blended frames too much (it's my opinion).
The counterpart of this : as the framerate is 50fps, to keep the same quality one must raise the bitrate for encoding (unless you use constant quality of course).


@vinetu :
Wow! How do you ended to that conclusion? Are you using special tools?
And do you know a solution to reverse the process? (well I think the part ... and then mixed - Foreground and Background ... means no)

[avih]

after 1st analysis, i think it was originally true 24 fps clip. then converted to ntsc (prob telecine to 30fps) then to pal using what i call 'relative blend'. (a conversion method i thought of theoretically, which i think actually exists).

the good thing, it seems that i can reproduce all the original distinct images without blend (but with half resolution). the bad thing is that i couldn't find a method to do that for the whole clip.

after reviewing the pattern over 25 fields (=1/2 sec) i could extract the 12 distinct images like this:
---------------------------------
avifilesource("C:\Documents and Settings\avih\Desktop\PAL Interlace - XviD Test.avi")
complementparity
bob
#showframenumber
selectevery (25,0,2,4,6,8,11,12,15,16,19,21,23)
------------------------

my test sequence was the moving landscape (originally starting after about 4 seconds). play the clip with that script, and you'll see that for the moving landscape you get 24 distinct images every second, and it scrolls smooth. the problem is that it doesn't seems to work well for the 'dancing' scene (originally starting at about 1 second).

which leads me to a possible conclusion that a framerate of 23.97 or 29.97 might have been involved in the process. which makes it harder to find a consistant pattern...

that's it for now :(

if i'll think of something better, i'll let u know.

cheers
avi.

[vinetu]

Hi

@Suiryc:
Well,for analisis usualy only VirtualDub
and 2 eyes (3 eyes get better results).
Unfortunatelly there is no way to reconstruct...or may
be you should run After Efects ,make some masks on every frame and rerender
the movie scene by scene ...


Avih,
Your Avisynth script is helped me to make the
same conlusion - some of the layers - for example the _heros_layer_ in dancing
scene ( 15->100 frame in TESTClip ) is nearly 12 fps ,while the _sky_ and other
background or foreground components are up to 25(24) fps ( from 12 to 25(24) for example ).

The _moving landscape_ ( 101 - 197 frame in testClip ) is may be 24 (25) originaly ...
... and I still think that there ( usualy on the _heros_layers_ ) was used some timestretching

vinetu

[avih]

definately, some of the layers are 12fps, but it doesn't matter, since generallt, this film is probably 24fps (as any 'normal' film), and it IS possible to extract 24 distinct images from every second, as seen in the 'landscape' scene. for the 'dancers', well, it should be possible to extract 24 unblended images/sec as well... but i think that maybe 23.97 or 29.97 framerate was involved, it makes it much harder to do that...

[vinetu]

definately.
And I think on anime it's not absolutelly necessery to gain 24 distinct images - if the result looks good (in play mode) - then fine.

[Suiryc]

Originally posted by avih
but i think that maybe 23.97 or 29.97 framerate was involved, it makes it much harder to do that...

There is maybe something else.
I'm presently trying a solution like the one you gave me. For me it's a little bit easier since I have the original clip.
And I found a pattern that gave me good results on a large number of frames :

SelectEvery(25,0,2,5,7,9,11,13,15,17,19,21,23)

Except after 1000 frames (in fact 1000 fields since we use Bob), where everything seems delayed by one frame ...
So I added SelectRangeEvery(1000,999) before the SelectEvery, and this seems to work for the rest of the clip (some blended frames remains here and there).

Frame number 0 of the clip I posted for you is frame number 508 of the original clip, so field number 0 with your script is field number 1016 of the original clip.
Since I use the SelectRangeEvery(1000,999), the pattern I use should be (for you) :

SelectEvery(25,0,2,4,6,8,10,12,15,17,19,21,23)

And after frame (field) number 984 of your script there should be more frames with the blend problem (when there is movement).

[avih]

so... does it work for u??? ;)

[Suiryc]

Originally posted by avih
so... does it work for u??? ;)

Not as well as I would, but it removed a lot of blended frames :)

However the problem remains later in the original clip (this thing is making me crazy!! :D)

[vinetu]

However the problem remains later in the original clip (this thing is making me crazy!! )

... watching on TV or computer ?

[Suiryc]

On a computer (Otherwise I think that keeping the clip interlaced should be OK when viewing it on a TV ?).

[Guerrand]

I am also finding similar problems in attempts to compress my buffy series when Divx5.02 was released. The movies were interlaced (it is obviouse when watching) and when i applied the deinterlacing filters through virtualdub it produced a "ghost" effect. However this is huge. Any movement in the foreground is very high quality, there are no signs of remaining interlaced lines or any serious artifact generations. But when they move quickly or something in the background moves quickly across the screen it produces a ghost trail as shown in this picture (normally it isnt this bad ... but you get the idea :P)

http://www.wn.com.au/mbratych/problem.jpg

Normally i would have associated this with the bad quality of the source (assuming ntsc converted to pal). But recently i tried compressing the movie "Atlantis" and found the same problems in that. However unlike the buffy, atlantis only showed these ghosts on the characters movement (the cartoon animations) while the rest of the action and CGI graphics compressed perfectly ... regardless of motion (even the fire was incredible). I am assuming this is a problem with the delays between interlaced frames described in www.100fps.com but when i created a 50fps file the problem was only 'slightly' reduced ... but still there??? have i stuffed something up or is there something else i should have done?

Any help would be appreciated because i really want to fix this problem. Thanx

[hakko504]

Use Don Graft's Smart deinterlacer (http://sauron.mordor.net/dgraft/smart.html) and don't check the blend option.

[trbarry]

I've compressed a lot of US ATSC digital Buffy captures and all of them respond to normat IVTC film type processing, using either Force Film or GreedyHMA. But they will have a single blended frame at scene changes for some reason, no matter what I do.

- Tom

[Guerrand]

Originally posted by hakko504
Use Don Graft's Smart deinterlacer (http://sauron.mordor.net/dgraft/smart.html) and don't check the blend option.

Thanx .. this is one of the codecs i havent used successfuly yet ... but will read up on it. :p

[morsafr]

Here is what I've done.
I have 2 DVDs of the same anime movie (Kenshin OVA). One is NTSC interlaced. The other is its PAL counterpart.

After examining the PAL one. I found that the fields were interlaced resulting in the ghosting effect after deinterlacing (even without blending during deinterlacing).

So I took a look at the NTSC one. And I found that it has been telecined (badly). So I tried two methods:
- IVTC (29.97 -> 23.976 fps) + speed up to 25fps. The quality was great, far better than my PAL DVD
- Changing the frame rate from 29.97 to 25fps with the ChangeFPS() function of Avisynth (using the blending method). And guess what ? The output was the same that my PAL DVD.

So I think that the problem is indeed a bad conversion of the NTSC master to PAL. This is acceptable when the source is really interlaced material but not when it is film material that has been telecined.

The only thing to do is to ask PAL DVD editors to change their method of conversion. But I don't know if they would listen.

[Guerrand]

Thanx for the help on that buffy encode i was having problems with. Trying both the Smart Deinterlacer by Don Graft and GreedyHMA in differnt attempts i got a much better result than the initial attempt. However the ghost image is still there. I reloaded the vobs into dvd2avi and used forcefilm (which made things worse) .. when i noticed that this is a PAL source with a 20fps framerate. As far as i understood PAL was 25fps?? this may be what is causing the odd results (i am only guessing atm), but i dont know enough about it to say for sure. Anyone have any idea what is going on???

I am greatful for any advice or comments.

PS: Morsafr i am an unfortunate owner of that Kenshin PAL DVD ... however i managed to get a decent encode from the original, do you have screenshots so i can view the difference?

(oOo and Don Graft made some excellent filters so if you havent seen them GO LOOK :P)

[trbarry]

If you are getting 20 fps then you may still have Force Film turned on in DVD2AVI for PAL source, which isn't a good idea.

This can also happen if you IVTC/Decimate twice with NTSC stuff.

- Tom

[Dreassica]

or decimated a pal source with a cycle of 5 does the same!!

[morsafr]

@Guerrand

I'm not sure you would appreciate the difference on still images but when you play both of them, the PAL and the NTSC correctly converted to PAL, you clearly see the smoothness of a correct conversion.

What I can do if people are interested is to make a XviD version of both of them and put them on an FTP.