Need help decombing/deinterlacing some DVDs

[An IP BreAKDoWN]

I have been encoding for years, however I always use BDs, so its always progressive and minimal artifacting. I'm now starting to work on my huge collection of DVDs, and this is my first series I started. Below I will have an encrypted (I included an unencrypted version because, yeah....) file that will contain the video clip that highlights the worst artifacts.

The worst frames are 359-550 and 3410-3550, you should see combing, blocking, chroma subshifting(?). If anyone can look at the clip, it would be very appricated.

Here is the unencrypted version...
Depreciated... LINK to the clip. Key: ZSj2JPp84t$*hx$eaXQD#REt^fPKy6G8A69nm^eD5ss*&uDKNX&$^UN*brQM*9*#H@!@KAKTsvpW%MkU$6kXXtHkrnB#W4p2qsT6

Here is the script:

vMPEG2Source("C:\Users\Jared\Desktop\Spongebob Project NEW\DEMUXED\raw video.d2v")
AssumeTFF()
Telecide(post=4,vthresh=30,dthresh=6,gthresh=10,blend=false,show=true) #2773 0:01:55.657 #410
#DeBlock_QED(quant1=64, quant2=66, aOff1=4, aOff2=4, bOff1=8, bOff2=8, uv=3)
#SeparateFields() 
#crop(2, 2, -2, -2)
Lanczos4Resize(656,480) # might be better to leave the sizing and have MKV take the aspect ratio.
GrainFactory3 (g1str=2, g2str=2, g3str=2, g1shrp=130, g2shrp=130, g3shrp=130, g1size=1, g2size=1, g3size=1, g1tex=20, g2tex=10)
#Use below for banding
#dfttest (sigma=1, lsb=true)
#SmoothGrad ()
#DitherPost ()

[colours]

Please don't encrypt the files you upload; it just makes it less convenient for people to help you.

I see some chroma field blending (ugh) along with pointresize chroma in the second section you mentioned. I haven't had time to look too carefully at your source, but fixing the pointresize chroma is as simple as mergechroma(nnedi3(field=1)), though this should only be applied to the affected frames.

[An IP BreAKDoWN]

Thanks for the quick reply, I'll add that to my script and will adjust accordingly. I have a hard time identifying some of these artifacts.

I applied the syntax you suggested, however it didn't decomb (was it supposed to?). I feel like I'm not doing this right; the way telecide handles this, it looks bad to me. Should I ditch telecide for a different filter? Is nnedi3 a better solution?

Here are some images that are compressed for fast page loading, so click on the images to see the full png. (I really wish there was some spoiler tags...)

Here is with: Telecide(post=4,vthresh=30,dthresh=6,gthresh=10,blend=false,show=true)

Here is with: mergechroma(nnedi3(field=1,nns=4,qual=2))




Here is another frame with: Telecide(post=4,vthresh=30,dthresh=6,gthresh=10,blend=false,show=true)

Here is the last one with: mergechroma(nnedi3(field=1,nns=4,qual=2))


I encrypt the files so it wouldn't get pulled down. I didn't think it was all that hard to decrypt them, if you have 7zip installed. It's a very handy tool to have, while we still can rely on it!!!! I know that everyone here is hard at work on something, so I'll reupload it to make everyone's life easier. When dealing with material that can have a bad repercussion because of intellectual property laws, I like to play it safe than sorry. But again, I do thank you for the time you spent to look at it.

[colours]

Mega already encrypts the content (allegedly!) so performing an extra encryption step is meaningless. Trying to copy the password directly also includes some invisible whitespace or something.

The nnedi3 line is only really useful for frames 3460-3560, and no, it doesn't decomb; all it does is properly interpolate a missing chroma field. It's kind of useless for dealing with the myriad of other problems with the DVD, which I have no experience fixing. You can use rfs to apply a filter to only specified frames thus:

rfs(last,mergechroma(nnedi3(field=1)),"[3460 3560]")

The most egregious problem is the chroma field blending, really. You can worry about minor things like residual combing and deblocking after that's fixed.

[An IP BreAKDoWN]

The last thing I'm going to say about encryption, the #1 rule is trust no one. That's it, I'm not going to talk about this since it's not the topic I wanted to talk about...

I was confused about the nnedi3 thing becasue this is the first thing you see in the info.

nnedi3 is an intra-field only deinterlacer. It takes in a frame, throws away one field, and then interpolates the missing pixels using only information from the kept field.

But I will try to work on that chroma! Now if anyone else can help with the combing, that would be so above and beyond! I really am about to lose some of my hair over this issue!

Again, thanks colours for helping me with the chroma, because that would have been my next target.

[8-BaLL]

The thing is- I looked into your source really quick and using Telecide() only doesnt leave any interlaced frames. But there is still some small interlacing on some small areas- on the letters for example, only some few frames though.

Or for example using qtgmc()- doesnt leave any interlaced frames, however qtgmc creates some ghosting in certain areas as Ive noticed. The source seems to be rather difficult, it has some inconsistent 3:2 pulldown as far as I can tell, I might be wrong though. Not an anime expert.. Mb someone else can look closer into the source.

However you could try to apply qtgmc() to the problematic ranges by using ".trim", as far as I can tell QTGMC didnt leave any interlacing left. You would need to match the framerate when you use qtgmc for some areas with something like qtgmc().selecteven() or qtgmc().srestore(frate=23.976) or something else, depending on your script.

[An IP BreAKDoWN]

Oh nice 8-BaLL! This is very very close to what I was looking for. I didn't know qtgmc was so powerful. I saw what you meant about the ghosting though. But I'm happy with the progress, its 98% better already. I'm going to see how far I can push qtgmc in the mean time. I have work, work, work so I can't go too far into researching, however this is coming along so well!

I think I got this fixed, will update later!
The only complaint I have with qtgmc is that it still is deleting frames thus is alerting the framerate. I did use srestore(frate=23.976), however the problem is still there. Am I doing something wrong?

This is with:QTGMC(Preset="Slower", NoiseProcess=2, RepChroma=true, GrainRestore=0.8, NoiseRestore=0.5, Sigma=2, NoiseDeint="Generate", StabilizeNoise=true, tuning="DV-SD").srestore(frate=23.976)


Notice the pearl's tongue, how the outline still has chroma-bleeding(?) and under the sponge's nose.

[ChiDragon]

You don't want to use QTGMC for the entire thing, since it isn't a truly interlaced source.

You want to alter the frame rate. 29.97 is not correct. Most of this is normal, proper 23.976 that was telecined to 59.94Hz. Normally you would follow Telecide with Decimate.

As mentioned, there are random breaks in the telecine pattern as well as the other issues. It sort of looks like they started with a 100% pure telecine and then broke it during editing at NTSC rate, to shorten up certain sections. But the opening portion is actually slower than 23.976.

As for your most recent screenshot, that's one of the parts Colours was talking about where the chroma is "point-resized" in the source. I think the problem is actually that these parts use interlaced 4:2:0 instead of progressive, but Colours knows more than me.

Source frame 3560 [UtoY]


Source frame 3560 + nnedi3 [UtoY]


Source frame 3561 (good) [UtoY]


All you can do is mitigate the damage by making it soft instead of blocky.

[An IP BreAKDoWN]

How would I tackle the combing then? Telecide() produces undesired results. How would I use Decimate(), this isn't a normal 3:2 pulldown? As far as I can tell the duplicate frames was done on purpose, not because of improper teleciding. Although I am FARRR from an expert with film (I stick to BD source for these types of reasons.) I see you're using UtoY, I'll look into that to find out what I can.

What's wrong with using QTGMC though? It looks pretty good, except the slight ghosting. This is the closest I have ever gotten with any attempt.

Thanks ChiDragon, I know that this is all starting to come down to my limited knowledge of professional video editing, and my lack of Avisynth knowledge. So any in-sight is a huge help!

[osgZach]

I think the idea here is you're going to have to write your scripts to account for these problem sections as you come across them. i.e if traditional IVTC is working good for the majority of your source, you just split off the problem sections with trim commands, process those sections with QTGMC and whatever else to keep it at the proper framerate, then merge those sections back in.

You are going to need multiple tools to address multiple aspects of the entire source material. There is likely not going to be a fire and forget solution that will not leave some other part of your clips wrecked.

[An IP BreAKDoWN]

But the problem is that traditional IVTC creates artifacts in most of the frames, so it's not a good fix. But I do understand what you're saying though. The other issue is that there are ~320 11 minute episodes that I would need to look at frame by frame if I wanted to get this perfect. Man I wish the was an analytical script that can hint towards all of the artifacts for me. But I should get off my cloud9.

[ChiDragon]

Originally Posted by An IP BreAKDoWN

How would I use Decimate(), this isn't a normal 3:2 pulldown?

Yes, it will leave some sections jerky. If you leave it at 29.97, like your script, it's more jerky.

What's wrong with using QTGMC though? It looks pretty good, except the slight ghosting. This is the closest I have ever gotten with any attempt.

Using a deinterlacer when you can use an field matcher just isn't optimal. The former has to invent information to fill the extra lines while the latter gets to use the original data.

If you're willing to go to 59.94fps, though, a combination of deinterlacing and field-matching can be thrown at every episode and you don't have to worry about the varying underlying frame rates. The only remaining concerns then are where to set your thresholds (missing small combs vs deinterlacing frames that could be field-matched) and frames where the field matcher guesses wrong.

I see you're using UtoY, I'll look into that to find out what I can.

It's just an easy way to visualize one of the chroma channels in isolation.

Originally Posted by An IP BreAKDoWN

But the problem is that traditional IVTC creates artifacts in most of the frames

Actually 90+% of your sample is regular soft 3:2 pulldown, according to DGIndex. It's just that the parts where IVTC fails aren't subtle.

[osgZach]

A tool like YATTA might help make things a little easier in some respects. You can do all kinds of stuff with it, but it is a little rough to learn at first.
I suspect it might be helpful in aiding you to quickly identify, and deal with changing telecine patterns.

http://ivtc.org/yatta


Other useful things like the required tools/plugins for using it, as well as some sparse (but still helpful) documentation can be found if you back out to the parent directory.

[An IP BreAKDoWN]

These are all very good suggestions. I really, really, really want to read more of the documentation for yatta and some other field matcher plugins. I can't really test these yet since I'm going to be very busy the rest of this week.

Is there anymore documentation that anyone knows about that has more information on field matching, 2:3 pulldown, and chroma information? I googled stuff that helps, but I don't know if they are missing any information, because I feel like I'm too much of a n00b to get the results I'm looking for. I already know through my experiences that it usually comes down to, trial-and-error, trying new things, reading over and over and OVER again, and a pinch of luck. However I want to see what the professionals use, and any critical writings would be a huge blessing.

Again, don't close this or anything, but I'm going to give this a rest for like 5 days so I can focus on reading in my spare time. A little off topic, but does anyone do this for a living?

[osgZach]

I've got a couple text files sitting around, one from an old forum post from someone, and another much longer document, written for a probably much older version of Yatta but still may have relevant info for you. Though this info may be more geared towards VFR MKV production so you have to take that in mind when deciding what you may or may not need to pay attention to.

I have attached them, so they should show up when approved or w/e

The larger of the 2 is probably the more interesting read. Although it may look funny in Notepad on Windows. I use Notepad++ tho and its fine there.

[colours]

The YATTA Manifesto is also worth a read. (Idk if this is one of the files osgZach has uploaded, what with attachments pending approval.)

[osgZach]

yeah that's one of them