DVD from NTSC to PAL + addAudioTrack + sub

[franketto]

I live in Europe, but I bought an old, bit noisy NTSC DVD at 720x480@29.97 4:3 with audio+sub in English. I have recorded from TV those episodes in my language in PAL.
In one dvd there are 6 episodes.
I want to split the episodes, keep the subtitle and add dubbed audio in my language, all at 25fps.
So the final result should be an mkv with dual audio + English Sub, compressed h.264 at 25fps (at 720x480?) to play with pc.

It's my first transcoding... I would use Avisynth + Megui.

I've read some on inverse telecide and deinterlacing, but I'm not so ready yet.
The original was filmed in 35 mm in 1966 (imdb), so it should(could) be IVTC.

I uploaded a clip extracted from vob to mkv, without reencoding it (I cut 3 scenes with Avidemux from two different episodes on the dvd):
http://www.mediafire.com/download/kzyjy6h5zczhhd7/exampleOriginalVobCut_Short.mkv

EDIT:
First thing, I tried to get 23,976fps, so I tried:

AssumeTFF()
telecide(guide=1).Decimate(cycle=5,mode=2,quality=3)
but guide=1 doesn't work ok, so it shouldn't be 3:2 pulldown, instead:

telecide(guide=0, post=4).Decimate(cycle=5)
seems better but only in some scenes... (before 16'')

I tried also what suggested Megui:

edeintted = AssumeTFF().SeparateFields().SelectEven().EEDI2(field=-1)
tdeintted = TDeint(edeint=edeintted,order=1)
tfm(order=1,clip2=tdeintted).tdecimate(hybrid=3)

but no success in the scenes after 16''

also:

tfm(order=1).tdecimate(hybrid=1)

nothing good.

Could you give me some hints?
TIA

[Guest]

There are some rule 6 red flags here:

You say you extracted VOB to MKV, but the MKV stream has no audio.

You say it wasn't re-encoded, but the MKV tags indicate otherwise.

The video is in MPEG-1 format, which is rare for a DVD.

Please post a sample of the original VOB. You can use DGSplit to cut it.

[franketto]

I used avidemux in copy mode.
I deselected audio track, as we are dealing with video problems and I wanted to reduce filesize.
In copy mode it can't recompress and in fact the processed video lasted 1 second: without reencode. Weird the mpeg1 flag, maybe it's an avidemux bug...
The original vob is in mpeg2.

Anyway here's the vob cuts corresponding at the precedent mkv scenes:
Vob1 (http://www.mediafire.com/watch/b98z6s1ifx44m7w/Vob1Cut.vob)
vob7 (http://www.mediafire.com/watch/9a1kfy831pkqkfr/Vob7Cut.VOB)

[Guest]

Weird the mpeg1 flag, maybe it's an avidemux bug...
The original vob is in mpeg2. Interesting. I think you are right about that as when I extract it to ES it then opens as MPEG-2.

Anyway here's the vob cuts corresponding at the precedent mkv scenes:
Vob1 (http://www.mediafire.com/watch/b98z6s1ifx44m7w/Vob1Cut.vob)
vob7 (http://www.mediafire.com/watch/9a1kfy831pkqkfr/Vob7Cut.VOB) Thanks. I'll have a look.

[Guest]

You have field-blended video, so simple IVTC is not going to be effective. Look into deblending solutions, e.g., SRestore.

[franketto]

Thank you for SRestore.
I read a bit and tried it and it works quite well with these parameters:


bobby = source.AssumeTFF().bob(-0.2,0.6) #(maybe I could try tdeint instead of bob)
calmy = bobby.Clense(reduceflicker=false).merge(bobby,0.5)
V1 = bobby.Srestore(dclip=calmy, frate=25, omode=6,mode=-4)


It's a bit blurrer than the IVTC, but the IVTC is ugly in the scenes with the blending fields, so...
Should I set Srestore directly to 25fps, or should I go first to 23.976 and then convert to 25fps (as I did with IVTC)?

After SRestore I get also some halos where before there was the blending, so I put noiseremover and dehalo:

For the denoise I tried in order from the worse to the best:
RemoveGrain, DegrainMedian, MVDenoise, MVDegrain, fft3d.
But the best seems this combination:


V1 = V1.Masked_DHA( rx=5 , ry=5 )
V1a = V1.DeGrainMedian(limitY=2 , mode=1)
V1noise = V1a.fft3dfilter(sigma=2, bt=5, bw=32, bh=32, ow=16, oh=16, sharpen=1)

#last sharpening:
V1fin = contrasharpening(V1noise,V1)
#or
sharpy = V1noise.LSFmod()
V1fin = V1.mt_makediff(mt_makediff(V1noise,sharpy),U=2,V=2)


I found it quite good, but sometimes a little trasparent halo inside faces i.e. mouth's lips...

So, I'm open at your suggestions, even to change completely my filters.
Of course I would like no blend, no halo, good denoise and good sharpen :cool:
Thank you

[Guest]

Glad to hear that you are making progress with it. I'm not very experienced with deblending so I invite others to weigh in to answer your questions.

[Asmodian]

Should I set Srestore directly to 25fps, or should I go first to 23.976 and then convert to 25fps (as I did with IVTC)?

Set Srestore to the original source material's frame rate, like IVTC. This looks like 23.976 to me.

I like MVDegrain1 or MVDegrain2 from mvtools (http://avisynth.org.ru/mvtools/mvtools.html) for denoising or strong denoising but I usually do not use other filtering. The examples on the mvtools wiki page are a good place to start.

[franketto]

Thank you Asmodian.
I set it @23.976 and I will then use at the end of script: AssumeFPS(25,1) (or is it better I do a ConvertFps(25)?).

I will try again MVDegrain2 to check vs fft3d.

Anyway I found why I have those trasparent halo in the lips:
it's the dehalo Masked_dha:

V3halo = V3.Masked_DHA( rx=5 , ry=5 )

but I need it to eliminate the very visible white contour halo created by SRestore in some ex-blended frames.
The ideal should be use the dehalo *only* in the blended frames and not in the entire film when there's no use.

[Asmodian]

AssumeFPS will play the video faster and ConvertFPS will blend frames to create new ones. Most people seem to like AssumeFPS more (a 4% speedup is a very common way to convert 23.976 to 25) but you will need to convert the audio as well (unless your PAL audio is already 4% shorter?).

I would try to fix SRestore's output instead of dehaloing but if that isn't possible I don't have experience with dehalo, sorry.

[franketto]

Yes, if Srestore could do the job without halos it should be the best option, but it seems I can't do it...

As denoiser I tried MVDegrain2

backward_vec2 = V2.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
backward_vec1 = V2.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec1 = V2.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec2 = V2.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
V2noise = V2.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=1)
V2 = V2noise.FFT3DFilter(bt=-1, sharpen=1.5)


and fft3d:

V2 = V2.DeGrainMedian(limitY=2 , mode=1)
V2noise = V2.fft3dfilter(sigma=2, bt=5, bw=32, bh=32, ow=16, oh=16, sharpen=1.5)

previewing in VD, fft3d has less noise, but in some background textures it washes out a bit also some details.


I have other problems:
I'm going to become crazy with the combinations of:
DirectShowSource/DSS2/Mpeg2source
bob/tdeint
MVDegrain2/fft3d
in which Srestore deblends or not!


With DirectShowSource or DSS2:
MVDegrain doesn't work well: it has some frames blended, particularly with tdeint, but also with bob.
FFt3d works well (only with tdeint 1 rare frame still blended)

Now I tried also the FileIndexer of Megui and loaded the d2v with DGDecode_mpeg2source:
SRestore doesn't work well anymore: it mantains some frames blended, both with bob or tdeint, MVDegrain2 or fft3d!

So, up to now, the only script that's working is with DSS/DSS2+bob+fft3d

What's going on with d2v? And MVDegrain2?

Just to visualize:

original field blended:
http://i.imgur.com/mygSrzR.png

correct deblended but halo:
http://i.imgur.com/eUECYLz.png
http://i.imgur.com/j9PGsKl.png (man's hat)

final deblended dehalo:
http://i.imgur.com/M7xAsKw.png
The dehalo makes also toon effect on faces...

[franketto]

In my previous post I talked about the d2v file doesn't work with SRestore.
I did that file with FileIndexer in Megui of the entire DVD.
Instead now I tried to make a d2v file based on the 2 vob I posted above: that works well.

I found out what happens, I have:


source = DGDecode_mpeg2source("Z:\3NipotiDVD\01-01\vobedit\Vob1Cut_1.d2v") #NOn work with SRESTORE: blend present!
bobby = source.AssumeTFF().bob(-0.2,0.6)

With the DirectShowSource of the entire DVD it needs AssumeTFF.
Instead using mpeg2source, the d2v of the entire dvd wants the AssumeBFF; instead if I load a d2v of the 2 little vob I uploaded above, then it works well with AssumeTFF...
So the point is, using DS I need to put AssumeTFF otherwise it doesn't blend correctly, whereas with mpeg2source I need to change to AssumeBFF, or eliminate the Assume, as it set automatically to BFF.
Quite weird...
Anyway solved it.
Now I can try also the MVDegrain2 correctly to compare with fft3d.

[manono]

It's from a PAL source so:

Yadif(Mode=1)#or your favorite bobber
Srestore()

will give you back 25fps. You can use the subs from the DVD unchanged.
...and add dubbed audio in my language, all at 25fps.
The audio may or may not fit with only a delay added, but if you have true PAL audio it shouldn't need to be stretched. Fitting audio from a different source often winds up being more trouble than it's worth. Never know till you try.
In my previous post I talked about the d2v file doesn't work with SRestore.
A D2V has nothing to do with SRestore. If you're getting blends it's for some other reason. An especially poor source, for example, that can't be fully unblended.

[franketto]

Hi manono.
I think it's impossible it's from a Pal source: it was filmed in 1966 in USA and broadcasted there; in Europe it was broadcasted after some years. Also, up to now in Europe we haven't any official DVD, that is the only one. We have only the broadcasted tv recorded.

Yes I have a true Pal audio (dubbed) and I can add it as is, but I will need to stretch the English original audio to fit the correct 25fps. But that's not a problem, I already did it with Timestretch(tempo) and it fit ok.

A D2V has nothing to do with SRestore. If you're getting blends it's for some other reason.
Yes, of course, what I wanted to say is that loading video with d2v I continued to have the blending, that is Srestore doesn't do its job correctly.
But the reason was that problem of Field order I described, not of course in d2v itself.


Anyway I found out a great improvement using QTGMC as deinterlacer before Srestore, thanks to NNEDI3: more details in contours.

Now I'm tryng to denoise/sharpen it best. The denoise is good, it's the sharpen it's not so great. Obviously there's not so much detail to sharpen, but I want to try the best possible.
The last problem is sort of halo in some frames to remove (the halo is there also without sharpen...)

I'm trying to set the denoiser inside QTGMC instead of doing later after Srestore. I tried QTGMC(Preset="Super Fast", NoisePreset="Medium")#NoisePreset="Fast",

with fft3d and also dfttest in QTGMC, but it is not better than a fft3d/dfttest *after* Srestore!

[manono]

I think it's impossible it's from a Pal source
You may think it's impossible, but it is definitely PAL2NTSC. Here, do this:

Yadif(Mode=1)#or your favorite bobber
Srestore(Frate=29.97)

Then count the cycle between either duplicate frames or blended frames. If 5, it's 23.976fps, if 6 it's 24.975 or 25fps. Or do the same thing with the unfiltered video. Just advance a frame at a time and count the progressive frame cycle (the rest are interlaced). 5 or 6? The two people coming out of the elevator is a good place to check.
Also, up to now in Europe we haven't any official DVD
Means nothing.

[Guest]

up to now in Europe we haven't any official DVD OK, I'll bite. Please specify the source of the DVD you obtained, including a link if possible, and explain why you are discussing a "non-official" DVD. Thank you.

[franketto]

@Manono, I'm not expert in technical telecine, so if you say that I believe you.
I was only thinking why a film series made in USA had to be transcoded in PAL and then used that version to make a NTSC DVD. Seems weird to me they haven't used the NTSC version they must had to broadcast it in the '60.
Maybe did they lost it and they had to use the PAL one?
What should be the correct procedure? Using the master @24p and make it @29,97? Or if they had to use the PAL version, which procedure should they have used to not blend the frames so bad?
In short, what they made wrong and how should they have done it?

@Neuron2, I don't understand what are you saying. In Europe there is no DVD of that series in none language, no PAL DVD at all, neither Italian, neither German, neither France.
The only DVD series to buy is in USA in NTSC format.
That's so simple, I buyed DVD from USA because here I could only record from TV (in my dubbed language, not the original audio that is American).
I already explained this in my first post, I don't understand what you want to know more.

EDIT:
the DVDBox is Family Affair Complete seasons, 24 discs. And it's an official DVD. But it's only in NTSC. Only you can't find one in PAL, only buy that NTSC version. You can find in Amazon, ebay, but in every case it's not in Europe, they must import from USA.

EDIT2:
Those DVDs are better than they looked at first. See here:
http://forum.doom9.org/showthread.php?p=1683547#post1683547

[Guest]

Just doing my due diligence as a moderator. Thank you for your edit supplying the information I requested.

[manono]

What should be the correct procedure? Using the master @24p and make it @29,97? Or if they had to use the PAL version, which procedure should they have used to not blend the frames so bad?
If all you have is a PAL source and you'll be using it to make an NTSC DVD, the 'right' way is to slow it back to film speed and encode it as progressive with 3:2 pulldown. But that takes knowledge, time, and money. So cheap companies take the cheap way out. They run it through a standards converter box that ruins it by blending the hell out of it. They could even have applied a hard 3:2:3:2:2 pulldown on it and the result would have been better than your field-blended garbage. I have yet to see a retail PAL2NTSC DVD that is encoded as 720x480 progressive 25fps with soft 3:2:3:2:2 pulldown although, thanks to neuron2 and his DGPulldown, it's been possible for several years now.

[franketto]

Thank you manono.
It's incredible that people doing this as a job, can't really take and use *free* tools to make a good product to sell. Bah.
Thank you all for the info and tools to make it right. There's no excuse in doing it wrong, nowadays. What's a pity of lazyness.

I will use Srestore @25fps, thank you.

[franketto]

Maybe I want to make it in 2 steps:
QTGMC+Srestore--> huff (intermediate lossless file) with VD
Denoise+sharpen+(dehalo)+etc --> x264 final file with Megui

Should I change something with colors? Such as ConvertTo or colormatrix, as I will use DVD->huff->x264?
The final file x264 will play on pc players.
I set ffDshow-Huff in YV12 planes, but in the resulting file, when played in potplayer or MPC-HC in the properties I get RGB...

If there's some quality color loss with huff and some convert, I could do also an intermediate x264 CRF0, but it's slower, so I'd prefer huff.

[franketto]

Manono, I tried to use Srestore @25fps as you suggested, but I think it's not correct.

I compared three clips:
1) a Pal broadcasted version @25f (I suppose that's the same version used as a master to make the NTSC DVD as you suggested)
2) a clip with Srestore @23.976 + AssumeFps(25)
3) a clip with Srestore @25fps as you suggested

I upload about 1' to see how the 3° clip differs significantly from the pal version (55 frames at the end in the changing scenes) while the 2° is correct at every change scene.

So I think I'd go with the 2° method as I need it also to sync the pal audio.

Anyway I think it's also logical, as it was originally filmed @24, then they trasposed it @25, then they used this pal master to make the NTSC DVD @29,97.
So, using Srestore @23,976 we have restored the very original frames and then playing @25 we got the same frames of the pal master.

EDIT 10MB:
https://www.mediafire.com/?bs78x18g2z1m1bq

[manono]

Yes, with your 'CompareSrestore.avi it's all 23.976fps and there are duplicate frames with the Srestore at 25fps version, the one on the right. And it's my fault for not checking both samples as I only downloaded and checked the Vob7Cut. It's all 25fps. At 25fps there are no duplicate frames and at 23.976fps there is, on average, one missing frame every second. I believe if you check your reencoded version of that part from Vob7 you should be able to notice the missing frames and stuttery playback. I don't know if 25fps extends beyond where the sample ends or how much of the film is 25fps.The Vob1 (and your comparison) is 23.976fps.

I made a D2V using DGIndex and then opened it in a script using DGDecode with MPEG2Source. The script used was:

LoadPlugin("F:\AVISynth\dlls\DGDecode.dll")
MPEG2Source("test.d2v")
Yadif(Mode=1)#for testing
Srestore()

Thank you for taking the time to prepare that comparison.

[franketto]

Ok, Manono.
Things get more difficult...

I tried the vob7 compare and the result is, like you said, the correct conversion is with Srestore@25, even if I got 3 frames of difference respect the pal original. The Srestore@23+Assume25 got -17 frames this time.

So one DVD could have episodes made from the master@24p and others converted from a PAL master @25.
Mmm
Is there a way to autodetect this in avisynth, so I can choose the correct Srestore func?
Or should I check every episode manually :eek:

I originally thought making one Srestore of all the dvd, now I could make some different func Srestore with trim commands, but it should be better if it could be an automated analysis (there are a lot of episodes...).

Also, is there an autodetect if there are some fields blended in an episode?

[manono]

Oh, it's episodic? That explains it then.

I guess you have to check each of the episodes individually, now that you know they're from different sources.

Autodetect? No. You have to figure them all out on your own.

Also, is there an autodetect if there are some fields blended in an episode?

Nothing autodetects field-blending. That's why, when you have such sources the analysis parts of such programs as AutoGK or MeGUI can't help. They'll either just deinterlace it or decide they're hybrids and mess it up even more.

[franketto]

Ok, I tried 3 episodes, 2@23 and 1@25.
After the Srestore, the result is quite good, but I get from -4f to -6f of difference at the end on about 35000 frames.
That is a scene change appears 6 frames before than the pal broadcasted. Probably it accumulates dropped frames in the time:
i.e: -1f @ 1059, -3f@33430, -4f@35038

Is it normal? Can/should it be corrected?
I don't know, as a thought: it could be compared with the pal broadcasted and when it sees 2 very different frames (scene change), it could duplicate a frame before it.

EDIT:
I tried your detecting rule of the repeating pattern of 5 or 6 and it's good, but I found also a clip that has sometimes repeating pattern as 6-5-7. Surely it's not a 23, but 25, but it's not consistent in 6 repeating. Is it possible?

[manono]

Is it normal? Can/should it be corrected?
Dropped frames as compared to what? A TV broadcast of the same video? As long as the audio synchs I wouldn't worry about it.
but I found also a clip that has sometimes repeating pattern as 6-5-7. Surely it's not a 23, but 25...
Yes, you're probably correct If, on average, the cycle is 6 frames then it should be 25fps. Another way to check (now that you're getting good at this) is to set SRestore for 29.97fps and see how often you get dupe frames or blends. Again, is the average a 5 or 6 frame cycle?
...but it's not consistent in 6 repeating. Is it possible?
Yes, very possible. I see that all the time. Sometimes SRestore won't do the dupe removal properly. In such cases you might have to set it for 29.97fps and have TDecimate get rid of the dupes with something like this:

TDecimate(Mode=0,Cycle=18,CycleR=3)

[franketto]

I looked all the dvds: there was only that episode from a pal master... So those DVDs are better than what they seemed at first! Maybe they lost that original and they needed to convert from the PAL master.
Anyway, I made some conversions and the audio gets synched correctly after making an increment equivalent from 23,976 to 25fps without pitch changing.

But the audio from that episode with pal master doesn't work if stretched. It synchs only leaving it at default extraction rate. It's another confirm that the original master is from PAL @25fps.