Yet another Interlacing/telecine Issue

[TotalChaos]

I'm so sick and tired of the difficulties with getting these combing lines out of source video. This is my number one gripe when working with videos.

I've read many articles including some at Doom9. Everytime I think I understand things I cannot seem to apply the concepts. I used to use the tools here at doom9 in the gordianknot suite, but I've since moved on to TMPGEnc5. Which application to use isn't exactly my question here.

My question is how can I remove these damn interlace lines. I'm trapped with one filter handling some lines and another handling others. This has been the case with much anime/cartoon video source material in the past. Using the GordianKnot suite I was only able to use either the 29.9fps to 23.9fps ?detelecine? filter or one of the deinterlace filters. With TMPGEnc I am able to use both simultaneously. This still isn't enough in many cases where the source material is a DVD with TV shows.

My question, for the moment, is how in the hell can one map the frames on a source video and apply the proper deinterlacing techniques to it? I've been using the visual inspection for a long time but I'm utterly sick of the trial and error. Also, what about video that switch interlacing techniques? ..Say during an intro theme scene. I'm not at all interestested making clip art out of every video I come across just to reducing interlace lines. I mean our players seem to do a great job of on the fly deinterlacing. Why can't frame server software match and beat these techniques???

(I'm sure it's just me being a noob, so feel free to throw me a link to some noob tool that rids video of lines while preserving as much of the source content as possible.)

[TotalChaos]

Would posting pictures be of help? I reviewed the rules and didn't read anythings specific to pictures of frames. Hell, I can even make them partial pictures of the frames if that helps.

[manono]

Pictures are pretty useless. If you want help with a specific video, upload an untouched source file, one that shows the problems you're having. 10 seconds or so with steady movement. And it had better not be something you downloaded from somewhere.

Ordinarily you wouldn't both IVTC and deinterlace at the same time. You do one or you do the other.

Most software players and many hardware/standalone players do not do a great job of deinterlacing. Especially if you're considering an IVTC to be a type of deinterlacing. Many are incapable of doing an on-the-fly IVTC.

And if all you work with is anime, it can be especially difficult and there are sometimes no easy answers. It sometimes comes down to picking the lesser of two evils. But one size doesn't fit all. You can't apply a technique that works on one source and then figure you can apply it to all. You have to study each source to learn what you have so you can then figure out how to treat it.

[TotalChaos]

Thank you for your response. It sounds like you understand exactly where I'm coming from. I've been at this for years (note my join date), but even to this day taking an interlaced/telecined source to a progressive format for encoding has proven to be a huge headache. I agree that it would not seem that much of these source videos would have some interlaced material that was then telecined/IVTC (if I'm reading this correctly those are the same?). I don't always backup anime/cartoon DVDs, but in the past I've worked with several. In this case, we're talking about the cult classic series Tremors. I have no problem what so ever uploading source material. I just got flagged years ago for even mentioning copywritten material, so I figured upping source material would be even worse. This srouce is completely legit. As in I watched an online version, liked it, and decided to purchase the entire series.

What format would be best? Should I take the raw VOB and extract the video MPEG or just up a portion of vob? or?

[manono]

It's OK under Fair Use laws to provide a small sample. Open a VOB in DGIndex, use the [ and ] buttons to isolate a small section and then File->Save Project and Demux Video. If the resulting M2V fits the requirements, upload it to Sendspace or MediaFire and give us the link.

Telecine is the opposite of InVerse TeleCine. The former converts progressive film to interlaced video and the latter converts video back to film.

[TotalChaos]

Right right, IVTC 'removes' telecine deinterlace 'removes' interlace. I totally said it wrong. I'll get on this video parsing and upload it. Thanx for the assist to this point.

[TotalChaos]

Hmmmm There is no option "Save Project and Demux Video" only Save Project. This only results in a D2V file. We're not talking about DVD2AVI are we? Is there some sort of setup proceedure I've overlooked? I'm using version 1.2.1. Does this one not have that option?

[Guest]

Use the latest version. That one is very old.

[TotalChaos]

It's the one I have in the GorianKnot 0.35 pack?! Will do. I was on the site looking for a the DL briefly already. When I have some sit down time I'm on it. Thanks again for the support.

[TotalChaos]

Got v1.5.8 and it works as described. I've wondered that one for a long time. Never was able to extract the video stream. Anyways I've taken clips from the first episode only.

One deinterlace filter (Inverse pulldown) displays the lines in the window seal near his arm towards the end of clip 1 properly and also displays the lines in the roof on clip 2 properly
Clip1 (http://www.sendspace.com/file/t3vaq7)
Clip2 (http://www.sendspace.com/file/9qzzql)
What this filter does not do is remove all of the larger deinterlacing lines through the entire video. I've used what TMPGEnc calls 24FPS which as far as I can tell is just IVTC and used other deinterlace filters with no aditional gains. I've had the best luck using the filter Interpolation - Adjusted. It removed nearly all of the interlace line, but it does not display certain lines properly, namely those mentioned above. There is one filter that often takes out all combing lines and I've been forced to use it often, but it tends to blur things and I can noticeably see quality loss. This one is Replication - Simple. I cannot say how these filter names translate into filter names in avisynth (isn't that the app that applies filters and serves frames used in GordianKnot?)

[manono]

It's a tough one, alright, but not because of any left-over interlacing. That's easy. The problem is that it's a true hybrid with several 'base' framerates in it, and some field-blended parts also. Only the parts that have been hard telecined can be successfully IVTC'd. If for DVD I think I'd leave it alone. Otherwise, if it has to be made progressive for a single framerate I might use an unblender to get it to 29.97fps, something like:

Yadif(Mode=1)#or your favorite bobber
SRestore(Frate=29.97)

http://avisynth.org/mediawiki/Srestore

You could also do a VFR encode and keep all the framerates, but I'm not very familiar with those kinds of things. Others might do things differently or have other suggestions.

[TotalChaos]

Hmmmm What I'd like to learn is exactly how to determine such things for myself. Not only that, but how to correct these issues. I also have the Star Trek TNG series, and it's proven every bit as difficult. My goal has been to take my entire DVD collection and store them on my fileserver. I find the freedom of a hardrive based format superior to the disc format. As said I've been at this for years and at this rate probably years to come before I'm finished. I'm interested in standards of a sort. In other words, I'd like all DVD content to end up in a sort of similar format. I've already decided on the codecs I prefer (DivX historically and H.264 currently). I have also set target bitrates. I prefer to maintain the exact aspect ratio or as close as possible and I'd like things in a progressive format so there are no lines, motions skips, artificial blurring, or large filter induced quality loss.

This is a tall order. Without taking this thread in a new direction, what can be done about identifying frame rates and interlace methods on the fly?

[manono]

Well, to help figure out what you have, separating the fields or bobbing the video are good starts. If you do that you'll find that some (I only looked at the first clip) is pure interlace (59.94 unique fields per second), some is hard telecine (IVTCable back to 23.976 progressive frames per second) and some third progressive framerate I didn't bother to figure out exactly. In addition, some of it is field-blended. So, progressive 29.97fps isn't ideal, but sort of a compromise. The only way you'll ever get it to play entirely smoothly is to teach yourself to do VFR (variable framerate) encodes. You could also just bob the whole thing and then keep it bobbed and encode for 59.94 progressive frames per second. That'll play fairly smoothly and is the way the 720p Hi-Def TV channels do it. You'll have to throw more bits at it, though, for the same quality, than if you encode it as progressive 29.97fps.

TNG is similar - lots of film-based stuff but with the 30fps computer graphics. It's a nightmare for home video enthusiasts to reencode. But I would never do any of this via TMPGEnc.

[TotalChaos]

From the looks of it... I may have to agreed. Personally, the effort one has to exert just to use the tools found here, must less do complicated things, is a bit much. With TMPGEnc the editing and such are decent enough and the interface handles all of the settings. Is there something more powerful here with a more friendly GUI?

[manono]

Maybe the closest thing to a GUI in the AviSynth world is AvsP:

http://forum.doom9.org/showthread.php?t=153248

Outside of AviSynth I don't know and don't care.

[TotalChaos]

Thanx for your assistance Manono. I appreciate the time you took explaining things (I'm sure you've had to do it many times before). If TMPGEnc isn't the answer and AviSynth + encoder is.... I'm most likely screwed. I doubt I'll ever become proficient with console based or script file based applications. I'll just have to stick with doing the movie DVDs.

[TotalChaos]

I thought I would throw an update on this project. I looked into VFR encoding. I can see the power of AviSynth. You essentially chain filters in nearly unlimited ways for tweaking the output. I'm not ready to learn a scripting language just to setup a frame server. I'd love to see a GUI front end for AviSyth (complete with huge help files).

I'm still stuck with TMPGEnc 5. Using Manono's suggestions I went back to 29.975 vs the 23.976 I had originally set and used a filter that appears to do two things. TMPGEnc has one deinterlace filter that does this:
"Creates a progressive picture by using the Inverse pulldown method, then applies a duplication method where interlace stripes are detected." The Inverse pulldown method is: "Creates a progressive picture by selecting which of the preceding or following pictures does not have interlace stripes." This appears to have removed much of the interlacing. I can still see at least one of the interlace artifacts, but only using a magnified view.

This appears to have worked out well. Any comments to the methods used?

[manono]

Any comments to the methods used?
Not by me. I stopped using TMPGEnc for IVTC/Deinterlace sometime around ten years ago when the first reasonably robust AviSynth IVTC (GreedyHMA) became available.

[TotalChaos]

There has to be a better way than to create text files to run an app. I'm never going to adjust or 'learn' it. It's the same reason I hate admininstration on my Linux Server. GUI is where it's at. I'm going to read the beginer's guide and see what I can come up with on AviSynth.

[TotalChaos]

BTW would you mind criticising a snippet of the output?

[manono]

Not really interested. Sorry.

[Stereodude]

There has to be a better way than to create text files to run an app. I'm never going to adjust or 'learn' it. It's the same reason I hate admininstration on my Linux Server. GUI is where it's at. I'm going to read the beginer's guide and see what I can come up with on AviSynth.There is no way a GUI would ever really work with AVIsynth because the plugins aren't all written by one person. They have too many options and features to integrate into a GUI. If someone tried it, you'd end up with a warmed over GUI that didn't unlock 1/10th of it's potential and would only support whatever limited set of plugins the GUI author chose to implement.

It's not that hard to learn. Put the effort in and reap the rewards.

[TotalChaos]

Well when you put it like that.... I'm not afraid of the learning aspect. Let me ask you this. How efficient is it to create a project and queue these projects to load a dedicated encoding machine. (where AviSnyth + encoder are concerned) I've never been able to even get the GordianKnot suite to queue projects well.

[Stereodude]

Well, what encoder are you using?

I find it pretty trivial using x264 + AVIsynth. I set up each movie the way I want in AVIsynth, then I use x264 command line style from a batch file for the 2 passes. When those are done that batch file calls the next batch file and the process repeats.

Then, later I mux the encoded video together with the audio into whatever container I want...

[TotalChaos]

hahaha wow. Just like that huh? I guess I'll just have to deal with what I have. While reading that post the only thing I keep hearing is tedious script editing and/or creation. Come on, batch files that are edited for every job? Now it appears I'm creating a flaming thread. This isn't my goal here. In fact, the goal for this thread was already reached. Nanono was able to point me in the right direction.

I completely see where you are going with the GUI front end. Chaining/scripting modules via GUI only may not be the way. I can see how it would essentially just limit things. Surely, there is a case for creating a work flow GUI though. I'm not necessarily promoting TMPGEnc here, but the project/workflow management is pretty organized and deffinitely productive. When you're faced with hundreds of episodes of a series to process it really fits the bill.

The idea here is simple.
1. Open source media.
2. Edit source
---A. Examine video
---B. Decide on desired clips/duration
---C. Script some module chains (perhaps with a built in AvsP type interface)
3. Send output to Encoder (some encoder parameters can be set ahead of time and saved while others can be sent from the previous step like output pixels etc)
4. Process encoder output into container format
Rinse and repeat with a few clicks or by sending this entire process (via a click of course) as a script to some batch job processor. I see this as more of a unified framework than a pure GUI only interface like TMPGEnc.

I assume we all know the process above (and probably better than me). Is it really ludacris to desire such a thing? Am I trying to water down something already powerful and user friendly? Does anyone know of a way to accomplish this?

[mswaino2]

I find Megui to make that issue alot easier to solve. It basically does alot of the work for you. All you have to do is analyze the file with the program and it does the rest. if It cant be analyzed for some reason then just read the DVD case or use DGindex to get the info it needs.