Telecine/combing/blending removal: have there been any recent advances?

[Chainmax]

A few days ago, while taking some DVDs to a friends house, among them the whole of the 1992's X-Men cartoon (Marvel Comic book edition), and remembered just how awful (http://forum.doom9.org/showpost.php?p=1452869&postcount=9) they were, and how I was never able to get a clean rip out of them.

As far as I recall, the most widely recommended tools for these issues were TIVTC, QTGMC, QTGMC+SRestore, and YATTA or freezeframing. Have these been superseded, or are they still the high watermark?

[Revgen]

Not much. Analog video is practically a dead format, relegated to the anals of history. All the problems and issues related to analog video or NTSC/PAL (rainbows, hard-telecine, field-blending, etc) are becoming less important as productions have moved to utilizing progressive high-def formats. Hence, further solutions for solving these problems have not been so forthcoming.

[ChiDragon]

I think that's a uniquely American view of the current situation. There is tons of field-blending still happening in standards conversions between the 1080i flavours.

I've been meaning to compare software-based IVTC to hardware-based, since HDMI capture is possible (but limited to 4:2:2).

[Revgen]

I've yet to come across an 1080i international Blu-Ray anywhere, although I'm sure they do exist. In my experience, most blu-rays are 1080p. Including the international blu-rays. Hence, very little if any field conversions take place.

[ChiDragon]

If "Blu-ray" means "just movies" to you, sure. You must not be interested in BBC content.

[Revgen]

Correct. I don't buy BBC shows on blu-ray.

[Sharc]

I've yet to come across an 1080i international Blu-Ray anywhere, although I'm sure they do exist. In my experience, most blu-rays are 1080p. Including the international blu-rays. Hence, very little if any field conversions take place.
All full HD 25fps blu-ray productions must be encoded as interlaced 1080i - even though the video might be progressive. There are quite some of these in Europe, and there is nothing wrong about it except perhaps that it gets a bit more complicated when one wants to downsize these to 720p ;)

[Chainmax]

Aren't most issue related to telecining and blending caused by framerate conversion?

[Sharc]

Often so, yes, in particular when not properly done, e.g. producing blended fields which cannot be undone.
Fortunately, blue-ray does no longer differentiate between PAL and NTSC standards which makes framrate conversion largely obsolete.

[Chainmax]

I was not aware of that, it's good to know.

Would you guys be willing to take a crack at a short sample (1-2mins) and see what you come up with, based on the info in the post I linked to?

[Sharc]

I suggest that you upload a short sample. Someone of the real avisynth cracks may be able to give you good advice.... ;)

[Chainmax]

Ok, here's an ~80s sample:

https://copy.com/vgXHhaROiRT7

It's just a little bit over 66MB. Curiously enough, this time DGIndex did not detect any field order transitions unlike last time. Anyway, have at it. :)

[Guest]

Blended fields. Try SRestore().

[Chainmax]

I was afraid of that. :(

I'll try different omode settings and replace Tdeint and Bob with QTGMC, and see what happens. Thanks.


[edit] I just ripped a whole chapter, and this time DGIndex did report a field order transition. neuron2, do you know what might be causing this detection discrepancy?

[Chainmax]

This is the portion of the script up to the IVTCing part:

MPEG2Source("x:\wherever\somed2v.d2v")

SetMTMode(2)

TComb(fthreshL=6,fthreshC=7,othreshL=7,othreshC=8)

d = last.bob(-0.2,0.6).reduceflicker(strength=1)
QTGMC(Preset="Placebo", EdiThreads=1)
srestore(frate=23.976, speed=-25, mode=-1, dclip=d)

vinverse()

I stepped through the script until I found a frame with blending, then tried every omode switch, adding TDecimate(mode=7) right after vinverse when external decimation was needed. None of the switches eliminated the blending on that frame. If past experiences are any indication, loading the "bad" d2v will not make a difference in this regard. Can I add some kind of pre/post processing next to the SRestore line, or tweak it in more ways than choosing omode? Maybe using another filter altogether?

[wonkey_monkey]

Fortunately, blue-ray does no longer differentiate between PAL and NTSC standards which makes framrate conversion largely obsolete.

I'm not sure what you mean by "no longer differentiate." The framerate issue is still largely the same as it was in the DVD days, except that people in PAL-land no longer get PAL speed-up. The majority of people in NTSC-land still can't play PAL* discs.

Oh, and they also seem more willing to lump us PAL-landers with the NTSC* conversions of our own material:angry:

*it wasn't even strictly correct to refer to DVDs as PAL/NTSC, but it's still a useful shorthand for 25p/50i/60i.

[kolak]

All these "blended" DVDs/BDs should be banned.
It's all down to poor QC and fact that many companies (post-houses/authoring houses) have no clue what they are doing.
DVD and BD does support all needed fps to provide content at native frame rate- there is no need for all this blended crap at all!
If you really have to do fps conversion use proper equipment/software, not crap blending converters.

[kolak]

All full HD 25fps blu-ray productions must be encoded as interlaced 1080i - even though the video might be progressive. There are quite some of these in Europe, and there is nothing wrong about it except perhaps that it gets a bit more complicated when one wants to downsize these to 720p ;)

Many of these are perfect 25p videos (either shot at 25p or speed ups from 23.976p/film masters) and scaling them can't be easier. It's the same as you would have 23.976p BD, just fps is 25.
Watch them without deinterlacing applied if you see progressive image treat them the same as "normal" progressive (eg 23.976p) sources.

[wonkey_monkey]

DVD and BD does support all needed fps to provide content at native frame rate

But not all TVs do.

if you see progressive image treat them the same as "normal" progressive (eg 23.976p) sources.

Not quite - chroma placement may (should be) different. It won't look terrible if you do it wrong, but it will still be wrong.

[kolak]

TVs also support all required fps (+ sometimes more) in specific country- if you do need fps conversion than this can be done with good results, it just need to be done properly.
Chroma placement may be different- depends on the encoder- I had no issues.

I worked 7 years in authorig house and produced 0 blended crap (and this was worldwide content with sources from all over the world)- it's all about knowledge and educating post houses about this crap. In 95% cases there is always original fps master and if you complain enough you will get it.
I also worked as a freelancer and this what I have seen recently (in big authoring houses) is really depressing, so I'm not surprised that some many discs are "wrong". It does not change fact that it can be done properly.

[Sharc]

Many of these are perfect 25p videos (either shot at 25p or speed ups from 23.976p/film masters) and scaling them can't be easier. It's the same as you would have 23.976p BD, just fps is 25.
Watch them without deinterlacing applied if you see progressive image treat them the same as "normal" progressive (eg 23.976p) sources.
Widely agree, just keeping in mind that 720p@25fps is not blu-ray standards compliant (I guess that many players won't complain though) . One needs to either change the framerate to 50fps (i.e. frame duplication which is straightforward and reasonably efficient with x264) or try with --pulldown double which however has it's issues with tsMuxer - at least in my experience.

[ChiDragon]

I worked 7 years in authorig house and produced 0 blended crap (and this was worldwide content with sources from all over the world)

But not worldwide output, right? Or did you convert everything to 23.976 and 29.97?

[kolak]

Most things were ending up as 23.976 or NTSC (59.94i HD) for worldwide compatibility.
Many masters were eg. NTSC tapes, but actually these were 23.976p+pulldown. In this case it was properly put back to 23.976p and encoded with soft pulldown.

[Chainmax]

So, given that stepping through the filterchain revealed that the problem was not eliminated or even significantly reduced, do you guys have more suggestions for me to try?

[osgZach]

To be honest, as far as blends go, this doesn't look half bad. It is certainly watchable quality. You almost don't notice it except for a few spots, unless you examine it frame by frame.

Otherwise, if you want a good source you'd probably have to pony-up and find a way to import NTSC/R1 Discs if you can afford it. Of course those may come with their own issues (no telling what other problems it might have aside from Telecined content or bad edits, etc)

[Sharc]

I'm not sure what you mean by "no longer differentiate." The framerate issue is still largely the same as it was in the DVD days, except that people in PAL-land no longer get PAL speed-up......
What I meant was that blu-ray defines a common (global) set of framerates which - as you mention - have their roots in PAL and NTSC TV history. However, blu-ray players have to comply with the full set, means they are no longer locked to either PAL or NTSC formats. From this point of view frame rate conversion of any blu-ray compliant source is no longer required just to make it playable. Of course this does not prevent that something has already been messed up when producing the "original".

[Chainmax]

To be honest, as far as blends go, this doesn't look half bad. It is certainly watchable quality. You almost don't notice it except for a few spots, unless you examine it frame by frame.

Otherwise, if you want a good source you'd probably have to pony-up and find a way to import NTSC/R1 Discs if you can afford it. Of course those may come with their own issues (no telling what other problems it might have aside from Telecined content or bad edits, etc)


The problem is that this is an NTSC source: it's from the Marvel DVD Comic Book Collection DVDs. And yes, from far away it's not too noticeable. I would still like to try and get a better rip than this from them, though.

[ChiDragon]

However, blu-ray players have to comply with the full set, means they are no longer locked to either PAL or NTSC formats. From this point of view frame rate conversion of any blu-ray compliant source is no longer required just to make it playable.

Not true. The situation is identical to DVD. That's why the first few waves of BBC Blu-ray releases were converted to 29.97fps -- and the same video encode was used even in the UK.

[osgZach]

The problem is that this is an NTSC source: it's from the Marvel DVD Comic Book Collection DVDs. And yes, from far away it's not too noticeable. I would still like to try and get a better rip than this from them, though.

Wow, that is total crap on their part. Sorry to hear this :(

I would mail them back to whoever produced these with a pile of dog crap just out of principle :mad:

[Chainmax]

Not doing double-blend removal and toying with the mode parameter solved many issues but caused just as many new ones. I'll try some more variations and see what comes out of it.

[kolak]

What I meant was that blu-ray defines a common (global) set of framerates which - as you mention - have their roots in PAL and NTSC TV history. However, blu-ray players have to comply with the full set, means they are no longer locked to either PAL or NTSC formats. From this point of view frame rate conversion of any blu-ray compliant source is no longer required just to make it playable. Of course this does not prevent that something has already been messed up when producing the "original".

When it comes to BD players than most of them support all fps, but problems is with TVs. Still many of them (even new ones) don't support PAL based fps in USA. In Europe 99% TVs/players support all fps- this has been discussed many times already.

[kolak]

Not true. The situation is identical to DVD. That's why the first few waves of BBC Blu-ray releases were converted to 29.97fps -- and the same video encode was used even in the UK.

It may not be identical, but very similar (problem lies mainly in TVs not BD players).

Yes- in some cases (if you have worldwide rights) some companies release one NTSC fps based version to cut costs.
This is not the biggest issues as long as fps conversion is done properly- not some crappy blending etc.

[udobroemme]

I just noticed this thread and coincidentally I have to deal with a fieldblended source too. It's a japanese DVD where a PAL source was converted to NTSC with fieldblending. I tried several tools like repal and srestore. The best result so far I could achieve with restore24 using the following:
userbob=yadif(mode=1)
#userbob=qtgmc(preset="placebo",noiseprocess=0,lossless=2,sourcematch=3,Sharpness=0, TR2=0)
Restore24(r24fps=25.000,numr=2500,deno=5994,useLL=true,r24deint=0,renderbob=userbob)
userbob defines the bobber used be the restore24 function.
For testing purposes I use yadif as a bobber and it maybe good enough for your anime clip. As I have a relatively fast cpu, I use QTGMC for the end result with the above settings. The resulting clip looks better than with any other restore function I've seen so far. Maybe this does help you with your problem too.

[Chainmax]

I was thinking of trying Restore24 after I exhausted my options with SRestore, after asking here if the former might work where the latter couldn't. Thanks for the suggestion. :)

While I exhaust the possibilities in SRestore, would you be willing to download the sample and give your alternative a try by stepping through a few consecutive frames? This is a film source, so destination fps will be 23.976.

[udobroemme]

I just tried my approach but it also didn't work with your clip. The problem is that there're too many blended fields. You can see that when you bob your source. So every restore function will fail as far as I know. But the original frame rate seems to be much lower than 23.976 (11 fps maybe?). So maybe there's another way, but I don't know anything about the treatment of animes so I can't help you here.

[Chainmax]

It just ocurred to me to make two intermediate files: one with the double-blend removal and another one with different parameters. I would then framestep through a portion of the first clip, try to sort out a more or less numerical rule indicating the troubled frames and use the second clip in those.

The problem is that after stepping through many frames in both versions, I found out that jumping to a specific frame number does not yield the same frame every time. Why could this be happenning? Would setting cache=whatever so that RequestLinear is used solve this problem?

[johnmeyer]

The problem is that after stepping through many frames in both versions, I found out that jumping to a specific frame number does not yield the same frame every time. Why could this be happenning? Would setting cache=whatever so that RequestLinear is used solve this problem?Are you using SetMTMode? If you are, it can cause this to happen. Don't use any multi-threading and see if that helps.

[Chainmax]

Yes, I am using it. Oh boy, this will make framestepping even more fun. :(

[Chainmax]

I finally got around to try it. Things seemed to go normally, but when I was stepping through one of the alternate filterchains I jumped from one frame to a previous one and the frame shown was not the one it had been the first time around. I'm going to finish this first batch of tests and then try setting the cache parameter to something and see if it solves the issue.

It's bad enough that I have to compare 24 different settings combinations, but if stepping through those des not even yield consistent results...:mad:

[johnmeyer]

The other thing I've found can cause frames to not feed into an AVISynth script in the same way when you step forward, and then step backwards, is if you are reading the file directly into AVISynth by using the MPEG-2 reader. I've had enough problems with this over the years that I always use Vegas to read the original MPG or M2V file and then frameserve out of that. So, another thing to try is to create an intermediate AVI file from a portion of the source, and then try stepping through that.

[Guest]

Or just use a reliable frame accurate source filter and stop worrying and jumping through hoops by transcoding, remuxing, etc.

[Chainmax]

johnmeyer: doing an intermediate file of a portion of the source might be a good option, thanks for the suggestion.

neuron2: so MPEG2Source is not frame-accurate? What other source filter that's frame-accurate can I use to load an .d2v?

[Guest]

Forgive me, I may have misunderstood. Of course, MPEG2Source() is frame accurate, but if you are using some MT stuff, all bets are off. I'd wonder why you want to suffer like that, when the MT stuff is known to have issues and requires very exact placement of things in your script, choice of modes, etc.? If you need reliable frame accuracy, I would suggest going single-threaded with a source filter known to be reliably frame accurate.

[Chainmax]

That's the thing though, as per johnmeyer's suggestion in post #37, I tried doing just that. Here's the filterchain:

MPEG2Source("D:\wherever\somefile.d2v")

TComb(fthreshL=6,fthreshC=7,othreshL=7,othreshC=8)

d = last.bob(-0.2,0.6).reduceflicker(strength=1)
QTGMC(Preset="Placebo", EdiThreads=1)
srestore(omode=3, speed=-25, mode=4, dclip=d)
vinverse()
TDecimate(mode=7)

Crop(4,0,716,480,align=true)

Spline36Resize(720,480)

Since the test is about different combinations of omode and mode, everything else is commented out, including the SetMTMode lines.

[Guest]

Start with just the source filter. See if it is frame accurate and doesn't change frames when jumping around. Then add the filters one by one to see which one is screwing up random access.

[Chainmax]

And here is when I get embarrassed for not remembering the basics of filterchain diagnostics. :o

[Guest]

Really, what is hard to understand about what I suggested:

"Start with just the source filter. See if it is frame accurate and doesn't change frames when jumping around. Then add the filters one by one to see which one is screwing up random access."

[Chainmax]

Oh, I understood that perfectly. I was just expressing (granted, in an unnecessarily roundabout way) my shame at not having thought of doing a filter-by-filter test.

[Guest]

Ah, I see.

So, what are the results of your diagnostics?

[Chainmax]

So far, nothing. I was sure it might have been TDecimate's mode=7, but I have not been able to reproduce the issue yet. No idea what's going on.