Seeking Advice To Improve An AviSynth Script For NTSC Simpsons DVDs

[LouieChuckyMerry]

Hello and thanks in advance for any help :) . I'm in the process of encoding my NTSC DVD's of the first ten seasons of the Simpsons and, after seemingly endless trial-and-error, dumb luck, and kind help from various people in various places, I've come to the following script:

# Set DAR in encoder to 6480 : 4739. The following line is for automatic signalling
global MeGUI_darx = 6480
global MeGUI_dary = 4739
SetMemoryMax(256)
SetMTMode(3,3)
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\ffms\ffms2.dll")
FFVideoSource("SourcePath", fpsnum=30000, fpsden=1001, threads=1)
### Deinterlace ###
SetMTMode(5)
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\avisynth_plugin\TIVTC.dll")
TFM(Order=-1,Slow=2,PP=0).TDecimate(Mode=1)
SetMTMode(2)
Vinverse()
### Deshaker ###
Stab(Mirror=15)
### Crop ###
Crop(8,0,-8,0)
### Resize ###
RatioResize(10/11.0,"PAR")
### Gibbs Noise Block ###
Edge=MT_Edge("prewitt",thY1=20,thY2=40).RemoveGrain(17)
Mask=MT_Logic(Edge.MT_Expand().MT_Expand().MT_Expand().MT_Expand(),Edge.MT_Inflate().MT_Inpand(),"xor")
MT_Merge(dfttest(),Mask,Luma=True)
### Overall Temporal Denoise ###
SMDegrain(tr=2,thSAD=600,ContraSharp=True,RefineMotion=True,Plane=0,Lsb=True,Lsb_Out=True,PreFilter=2)
### Debanding ###
GradFun3(thR=0.55,Radius=12,Mask=2,SMode=1,Lsb=True,Lsb_In=True, StaticNoise=True,Y=3,U=3,V=3)
DitherPost(Stacked=True,Prot=False,Mode=0)
### Line Darkener And Thinner ###
FastLineDarkenMod(Strength=20,Prot=6,Thinning=0)
aWarpSharp2(Blur=4,Type=1,Depth=3,Chroma=2)

The video from Season 1 through most of Season 7 is like this: S2.E1-TestClip (https://www.mediafire.com/?effpbb705a10b34), and the video from the end of Season 7 through Season 10 is like this: S8.E1-TestClip (https://www.mediafire.com/?effpbb705a10b34). I'm encoding with 10-bit x264 with --preset veryslow --tune animation --crf 13.0 --deblock 2:1 --ref 12 --rc-lookahead 120 --merange 32 --no-fast-pskip --stitchable, looking to the future when I hopefully have a proper large screen TV instead of my current 14" laptop monitor ;) . With my i5-3320M, 8GB RAM setup I'm achieving ~2.4 Fps. I've some questions before I attack:

1) Is there any reason quality-wise to use DGIndex-M2v instead of FFMSIndex-Mkv? I'm using FFMSIndex because it allows me to utilize AviSynth 2.6 MT; however, I'd switch to DGIndex (and run simultaneous encodes for more production) if it would output higher quality video.

2) Speed not being the main issue is there any way to streamline or amend the above script, any redundancies to rectify, any fat to trim? I'm able to run the script and understand the gist of it, but the finer details are still beyond me (someday ;) ).

3) Looking to that large screen TV, are the x264 settings reasonable or are they overkill? Can they be improved?

I can't think of anything else, but I'm sure I will. Thank you for your time, all well-intentioned feedback is much appreciated.

[kuchikirukia]

What's RatioResize?

Good chance all that's going to be using more than 256MB.

And you should probably put the high color depth stuff last and Dither_out() to x264 10 bit.

[videoh]

1) Is there any reason quality-wise to use DGIndex-M2v instead of FFMSIndex-Mkv? I'm using FFMSIndex because it allows me to utilize AviSynth 2.6 MT; however, I'd switch to DGIndex (and run simultaneous encodes for more production) if it would output higher quality video. No. The only possible difference would be due to different IDCT algorithms but that would be insignificant, or a difference in random access handling, but your script appears to be linear.

[LouieChuckyMerry]

What's RatioResize?

RatioResize (http://forum.videohelp.com/threads/369143-ResizersPack-MasksPack-PlaygroundPack-SmoothContrast-Logo-mod-functions?p=2364034&viewfull=1#post2364034)

Good chance all that's going to be using more than 256MB.

I spent a week testing with AVSMeter (thanks Groucho2004!) and those MT settings are batch-stable and give me the highest fps.

And you should probably put the high color depth stuff last and Dither_out() to x264 10 bit.

Please, would you explain that in more detail? I'm a primary school student when it comes to AviSynth ;) . Also, I just decided to go with the x264 10-bit, after receiving all the previous help on the script, so this is something new to me.

No. The only possible difference would be due to different IDCT algorithms but that would be insignificant, or a difference in random access handling, but your script appears to be linear.

Thank you :) .

[Groucho2004]

When you're doing this:
RatioResize(10/11.0,"PAR")

you probably also want to flag the video stream with "--sar 10:11".


You could probably remove a lot of stuff after TDecimate by just using SRestore() (http://avisynth.nl/index.php/Srestore) (unless you tried this already).

Since you are going to encode a lot of episodes I would avoid the AVS MT hassle and potential frustration and encode 2 - 4 (depending on the number of cores of your CPU) episodes simultaneously. Any decent "gui" should offer that feature. If not, use batch files.

[LouieChuckyMerry]

When you're doing this:

you probably also want to flag the video stream with "--sar 10:11".

Thanks, does it matter where in line I add "--sar 10:11"?

You could probably remove a lot of stuff after TDecimate by just using SRestore() (http://avisynth.nl/index.php/Srestore) (unless you tried this already).

I've not tried SRestore, but have it available. Would it replace Vinverse and Stab? Please, could you suggest a usage given the situation?

Since you are going to encode a lot of episodes I would avoid the AVS MT hassle and potential frustration and encode 2 - 4 (depending on the number of cores of your CPU) episodes simultaneously. Any decent "gui" should offer that feature. If not, use batch files.

Actually, figuring out the settings for AviSynth 2.6 MT has been the easiest part of this project thus far, thanks to AVSMeter :) . Really, I can queue episodes for days in MeGUI without any issues and at a higher rate of production than simultaneously running episodes. Now if you'd only come up with an app to help me test and improve scripts...

[Groucho2004]

Thanks, does it matter where in line I add "--sar 10:11"?
No.

I've not tried SRestore, but have it available. Would it replace Vinverse and Stab? Please, could you suggest a usage given the situation?
I have almost zero experience with these, I just know that SRestore has been used successfully by others to cure nasty field blending. Search the forum.


Actually, figuring out the settings for AviSynth 2.6 MT has been the easiest part of this project thus far
Having such a complex process run and not crash is one thing. I stopped using AVSMT for "serious" work when I discovered random artefacts at various places in a BD backup. The script only had MDegrain + LSFMod, I used AVSMT to speed it up. I don't know if it was a problem with AVSMT or one of the filters, point is that I only discovered this months later when I watched the backup. Nowadays MT may be more reliable but it's something you have to consider.

[kuchikirukia]

Please, would you explain that in more detail? I'm a primary school student when it comes to AviSynth ;) . Also, I just decided to go with the x264 10-bit, after receiving all the previous help on the script, so this is something new to me.

The problem with 8 bit is it doesn't have enough colors to avoid banding. So to avoid obvious banding in 8 bit requires some magic -- dithering. So if you deband and then want to encode 8 bit you want to dither.
10 bit doesn't need dithering.
You are converting it to 16 bit color, debanding, using LSB_out to tell gradfun3 not to dither and keep it 16 bit, but then using DitherPost which dithers and converts it to 8 bit... before sending it off to x264 10 bit. Just do your high bit depth color processing last and send it out in 16 bit color with dither_out() and tell x264 what's going on.
Add these to your x264 command line with appropriate resolution and FPS:
--demuxer raw --input-depth 16 --input-res 1280x720 --fps 59.97

[creaothceann]

But... is there going to be banding in the Simpsons?

[LouieChuckyMerry]

No.

Thanks, I added it.

I have almost zero experience with these, I just know that SRestore has been used successfully by others to cure nasty field blending. Search the forum.

OK, I'll do some research on SRestore.

Having such a complex process run and not crash is one thing. I stopped using AVSMT for "serious" work when I discovered random artefacts at various places in a BD backup. The script only had MDegrain + LSFMod, I used AVSMT to speed it up. I don't know if it was a problem with AVSMT or one of the filters, point is that I only discovered this months later when I watched the backup. Nowadays MT may be more reliable but it's something you have to consider.

Ahhh, thanks for that. I never would've considered this.

The problem with 8 bit is it doesn't have enough colors to avoid banding. So to avoid obvious banding in 8 bit requires some magic -- dithering. So if you deband and then want to encode 8 bit you want to dither.
10 bit doesn't need dithering.
You are converting it to 16 bit color, debanding, using LSB_out to tell gradfun3 not to dither and keep it 16 bit, but then using DitherPost which dithers and converts it to 8 bit... before sending it off to x264 10 bit. Just do your high bit depth color processing last and send it out in 16 bit color with dither_out() and tell x264 what's going on.
Add these to your x264 command line with appropriate resolution and FPS:
--demuxer raw --input-depth 16 --input-res 1280x720 --fps 59.97

If 10-bit x264 doesn't need dithering, then should I then eliminate the entire "### Debanding ###" section (the GradFun3 and DitherPost lines) in the script? And which lines are the "high bit depth color processing"? Sorry for my ignorance. The source resolution is 720x480 and original frame rate is 29.970, so would I add:

--demuxer raw --input-depth 16 --input-res 720x480 --fps 29.970

to the x264 command line? Thanks.

[kuchikirukia]

If 10-bit x264 doesn't need dithering, then should I then eliminate the entire "### Debanding ###" section (the GradFun3 and DitherPost lines) in the script? And which lines are the "high bit depth color porcessing"? Sorry for my ignorance. The source resolution is 720x480 and original frame rate is 29.970, so would I add:

--demuxer raw --input-depth 16 --input-res 720x480 --fps 29.970

to the x264 command line? Thanks.

If it has banding you should deband. Gradfun3 is a debander, the "adding dither" part is only needed for 8 bit.
I don't exactly know where you're converting to 16 bit, though a quick look says it's probably SMDegrain. So you'd end with SMDegrain --> Gradfun3 --> Dither_out()

I wouldn't resize back to 720x480 after cropping. There's really no need. Just set the appropriate DAR. Though I don't know how to do that with 16 bit out. I don't know if the MeGUI lines get passed or not. Try it and see.
Oh, and it should be 23.97fps after IVTC, no?

[LouieChuckyMerry]

I wouldn't resize back to 720x480 after cropping. There's really no need. Just set the appropriate DAR. Though I don't know how to do that with 16 bit out. I don't know if the MeGUI lines get passed or not. Try it and see.
Oh, and it should be 23.97fps after IVTC, no?

Ahhh, my mistake, I took "--input-res" to mean the source's resolution (and fps). So, I've added:

--demuxer raw --input-depth 16 --input-res 640x480 --fps 23.976

to the command line. Is this correct?

If it has banding you should deband. Gradfun3 is a debander, the "adding dither" part is only needed for 8 bit.
I don't exactly know where you're converting to 16 bit, though a quick look says it's probably SMDegrain. So you'd end with SMDegrain --> Gradfun3 --> Dither_out()

It indeed has banding and you're right, it's SMDegrain that's converting to 16-bit (Lsb=True) then sending this (Lsb_Out=True) to GradFun3 (Lsb_In=True). So, with your dither fix would the end of the script change to:

### Overall Temporal Denoise ###
SMDegrain(tr=2,thSAD=600,ContraSharp=True,RefineMotion=True,Plane=0,Lsb=True,Lsb_Out=True,PreFilter=2)
### Debanding ###
GradFun3(thR=0.55,Radius=12,Mask=2,SMode=1,Lsb=True,Lsb_In=True, StaticNoise=True,Y=3,U=3,V=3)
Dither_Out()
### Line Darkener And Thinner ###
FastLineDarkenMod(Strength=20,Prot=6,Thinning=0)
aWarpSharp2(Blur=4,Type=1,Depth=3,Chroma=2)

from the original "DitherPost(Stacked=True,Prot=False,Mode=0)"?

Edit: the above change caused serious flashbacks with MeGUI's preview window, with stacked frames and odd color spaces. Removing "Lsb=True" from GradFun3 eliminated the stacked frames but the color space was still trippy; removing the "Lsb=True" line from GradFun3 and deleting the entire "Dither_Out() resulted in normal looking output, however.

EditEdit: seems I typed too soon. The preview window in MeGUI shows normal video with the proper aspect ratio, but the output of a short test clip produces a file with the proper overall aspect ratio but four copies of the video, with the wrong color space, in a grid. Something to do with the command line additions, maybe?

[kuchikirukia]

Put them last.
(edit: fixed to address the 4-in-one problem below)

# Set DAR in encoder
global MeGUI_darx = 628
global MeGUI_dary = 480
SetMemoryMax(256)
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\ffms\ffms2.dll")
FFVideoSource("SourcePath", fpsnum=30000, fpsden=1001, threads=1)
### Deinterlace ###
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\avisynth_plugin\TIVTC.dll")
TFM(Order=-1,Slow=2,PP=0).TDecimate(Mode=1)
Vinverse()
### Deshaker ###
Stab(Mirror=15)
### Crop ###
Crop(8,0,-8,0)
### Resize ###
### Gibbs Noise Block ###
Edge=MT_Edge("prewitt",thY1=20,thY2=40).RemoveGrain(17)
Mask=MT_Logic(Edge.MT_Expand().MT_Expand().MT_Expand().MT_Expand(),Edge.MT_Inflate().MT_Inpand(),"xor")
MT_Merge(dfttest(),Mask,Luma=True)
### Line Darkener And Thinner ###
FastLineDarkenMod(Strength=20,Prot=6,Thinning=0)
aWarpSharp2(Blur=4,Type=1,Depth=3,Chroma=2)
### Overall Temporal Denoise ###
SMDegrain(tr=2,thSAD=600,ContraSharp=True,RefineMotion=True,Plane=0,Lsb=True,Lsb_Out=True,PreFilter=2)
### Debanding ###
GradFun3(thR=0.55,Radius=12,Mask=2,SMode=1,Lsb=True,Lsb_In=True, StaticNoise=True,Y=3,U=3,V=3)
Dither_out()

--demuxer raw --input-depth 16 --input-res 704x480 --fps 23.976

MeGUI preview will be borked since it doesn't know what to do with 16 bit. Use DitherPost to convert to 8 bit if you want to use the preview.

[LouieChuckyMerry]

Thanks for your reply, kuchikirukia. Running a test clip with the suggested script and the suggested command line addition resulted in an output video suffering from four-pictures-in-a-grid-with-trippy-color-space. However, running a test clip with the suggested scrpt but no command line additions resulted in a proper video output. So, it seems that something's wrong with the command line additions. Ahhh, Happy Friday :beer: .

[Scythe42]

I'm in the process of encoding my NTSC DVD's of the first ten seasons of the SimpsonsI am doing exactly the same as you right now. These DVDs are really horrible in quality.

That's what I use currently (still experimenting a bit over various disc and seasons before a long batch job):

First I load stuff. Loading from an remuxed MKV works nicely. No need at all for DGIndex, provides the same result for me. So I went with my selected and configured DirectShow filters here.

DirectShowSource("E:\Encode\S01E01.mkv",audio=false).ConvertToYV12()


There's a ton of dot crawl all over the scenes, hard to remove. Checkmate() did the best for me. Still some left but it's in an acceptable range now. This is clearly an analog tape transfer, and a bad one. Need to check if I can do better here with other filters or stack filters. Especially visible on the the credits.

Checkmate(thr=10, max=20, tthr2=0)


Next deinterlacing. These DVD have strange pulldown patterns, similar to Futurama. You can do various TFMs and TDecimate cycles, where you have to remove like every x frame in addition to a normal IVTC. You can also use AnimeIVTC, but didn't bother to try. The pattern changes now and then and I am not going through every frame to find some blended frames or adjust stuff per episode. So I went for an QTGMC/Strestore combo here (Yadif doesn't work very, worked better on Futurama then QTGMC). That takes care of nearly everything for me so far.


QTGMC(Preset="medium") #fast should work as well, going slower will produce blended artifacts.
SRestore(frate=23.976)


Next 4:3 resizing from the anamorph DVD encode to square pixels. Optional and could be done over x264.

Crop(12,0,-4,-0) # not ITU compliant, more on the left then on the right.
Spline36Resize(640,480)


Certain scenes have Gibbs artifacts, I remove them the same way you do (we both probably found the same post when googling)

Edge=MT_Edge("prewitt", thY1=20, thY2=40).RemoveGrain(17)
Mask=MT_Logic(Edge.MT_Expand().MT_Expand().MT_Expand().MT_Expand(),Edge.MT_Inflate().MT_Inpand(),"xor")
MT_Merge(dfttest(), Mask, Luma=True)


There were still some stuff left around the edges and some halos now and then. Still a lot of noise and no clear edges. With that noise they cannot be really enhanced later on. So let's add some not so agressive stuff. Can always remove later if not not needed.

Edgecleaner(strength=50)
Dehalo_alpha(darkstr=0.5, brightstr=0.5, rx=1, ry=1)


Next, because there is so much noise and everything is very blocky. I went with deen() here as you really need to be aggressive to remove all the artifacts, blocking and other crap. And after the spatial smoothing I applied a temporal smoothing as well as the one in Deen doesn't do anything for me. Yes, I am very aggressive here.

The problem is that even after these filters there is noise in the corners left near the black lines and in small areas, that are smudgy on the DVD to begin with. So I stacked the smoothers and added an FFT3D on top. That finally removed all the crap.

Now the picture is of course too smooth. So I need to adjust the parameters probably a bit and see if I can get some details out around the edges, areas in general are fine now. Though some "smuge" is left. FFT3D cannot cleanup the picture alone, and neither can Deen. The nice thing is that these stacked combo also took care of the leftover of the Dot Crawl.

FFT3DFilter(sigma=3, bw=32, bh=32, ow=16, oh=16, plane=4)
Deen(mode="a2d", rad=4, thrY=7, thrUV=9, min=0.5)
FluxSmoothST()


Next up is line darkening. I tried various plugins and Toon gave me the best overall result, given that the lines were never sharp and are interrupted on the DVD to begin with.

Toon(strength=0.75)


Then as usual sharpening follows. I didn't overdo it or stuff I smoothed out will be visible again. Just trying to get the edges crisper again. But if you want more you could add a LimitedSharpenFaster() in addition to the warp sharpening as well or just for with it.

aWarpSharp2(blur=4, type=1, depth=11, chroma=2)


And because the smoothing could not remove all blocking artifacts and created some color banding and the sharpening makes them way more visible now, I use a deband filter to smooth these areas out . In some scenes it is not possible anymore as they are so bad in source that nothing can fix them.

GradFun3(smode=2)


Now the problem is that certain scenes have a wrong white balance, colors are general oversaturated or understaturated. Not really something you can solve automatically here. So, I live with it.

Yes, the edges could be a bit more defined and crisper and the picture is very smoothed. But even if you scale it up on a TV to 1080p it is still better then the cropped HD upscales that air for old episodes since some time. If not scaled up the result is more or less like the DVDs should have been in the first place.

Will work a bit on the smoothing to lower it. Maybe I'll find a sweet spot. Right now it's a little too much of course, since I stacked filters to get all the noise out.

If someone has a suggestion or sees a mistake pls. let me know.

And here are some screenshots on a frame with not too much noise on the DVD scaled up to 1080p in the player (so don't mind the jaggies), so you can easily see all the noise. As you see I need still some adjustment with the lines on which I have not yet focussed.
http://screenshotcomparison.com/comparison/131783

Edit: Update post

[kuchikirukia]

Running a test clip with the suggested script and the suggested command line addition resulted in an output video suffering from four-pictures-in-a-grid-with-trippy-color-space. However, running a test clip with the suggested scrpt but no command line additions resulted in a proper video output. So, it seems that something's wrong with the command line additions. Ahhh, Happy Friday :beer: .

It was something wrong with my script. When I cut and pasted yours to change the order I left in the DitherPost instead of changing it to Dither_Out. So it was sending 8 bit to x264 which was now expecting 16 bit.

It's fixed now.

[Scythe42]

So, as the tests for first thin dark lines to get a sharper impression and later thicken them didn't really work out in general as it was too far away from the source. I made some changes:


switching from Toon to ToonLite as the thin lines really didn't work out and introduce too much aliasing if scale on a TV to 1080p.
Removal of additional filters I throw in that are not needed anymore. Little bit of unwanted noise remains in a few scenes, but not worth the effort.
Re-Ordered some filters
Using UnFilter for sharpening now. aWarpSharp created to much halos with the changes. PSharpen or LimitedSharpenFaster should work as well. Settled on this one for now.


Here is what is running now over the first seasons and then I check my results and see if something goes really wrong somewhere.

Picture is now a bit more smooth compared to before but closer to the original thicker blurry lines.


#Video Source
DirectShowSource("E:\Encode\S01E01.mkv",audio=false).ConvertToYV12()

#Dot Crawl removal
Checkmate(video, thr=10, max=20, tthr2=0)

#Deinterlace and Restore
QTGMC(Preset="Medium")
SRestore(frate=23.976)

#Crop and Resize
Crop(12,0,-4,-0)
Spline36Resize(640,480)

#Gibbs Removal
Edge=MT_Edge("prewitt", thY1=20, thY2=40).RemoveGrain(17)
Mask=MT_Logic(Edge.MT_Expand().MT_Expand().MT_Expand().MT_Expand(),Edge.MT_Inflate().MT_Inpand(),"xor")
MT_Merge(dfttest(), Mask, Luma=True)

#General Noise Removal
FFT3DFilter(sigma=3, bw=32, bh=32, ow=16, oh=16, plane=4)

#Edge Cleaning as there is additional noise and halo leftovers
Edgecleaner(strength=20)

#Temporal Smoothing
FluxSmoothST()

#Main Smoothing
Deen(mode="a2d", rad=4, thrY=7, thrUV=9, min=0.5)

#Darken Lines without sharpening to avoid thinning
ToonLite(strength=0.75)

#Sharpen
UnFilter(50, 50)

#Color Debanding to get better gradients on larged deblocked areas.
GradFun3(video, smode=2)

[LouieChuckyMerry]

It was something wrong with my script. When I cut and pasted yours to change the order I left in the DitherPost instead of changing it to Dither_Out. So it was sending 8 bit to x264 which was now expecting 16 bit.

It's fixed now.

Thanks for that. Running the script with the command line additions output a file with proper color space but a DAR of 0.683. Very thin, tall everything! Returning the RatioResize line and editing the command line output a file with proper color space and a proper DAR (although MediaInfo shows the aspect ratio as 1.212). Also, I cleaned up the MT settings in case someone finds this useful some day. The script is now:

# Set DAR in encoder to 6480 : 4739. The following line is for automatic signalling
global MeGUI_darx = 6480
global MeGUI_dary = 4739
SetMemoryMax(256)
SetMTMode(3,3)
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\ffms\ffms2.dll")
FFVideoSource("D:\Temp\zSimpTemp\S1.E1-[I-4028].mkv", fpsnum=30000, fpsden=1001, threads=1)
### Deinterlace ###
SetMTMode(5)
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\avisynth_plugin\TIVTC.dll")
TFM(Order=-1,Slow=2,PP=0).TDecimate(Mode=1)
SetMTMode(2)
Vinverse()
### Deshaker ###
Stab(Mirror=15)
### Crop ###
Crop(8,0,-8,0)
### Resize ###
RatioResize(10/11.0,"PAR")
### Gibbs Noise Block ###
Edge=MT_Edge("prewitt",thY1=20,thY2=40).RemoveGrain(17)
Mask=MT_Logic(Edge.MT_Expand().MT_Expand().MT_Expand().MT_Expand(),Edge.MT_Inflate().MT_Inpand(),"xor")
MT_Merge(dfttest(),Mask,Luma=True)
### Line Darkener And Thinner ###
FastLineDarkenMod(Strength=20,Prot=6,Thinning=0)
aWarpSharp2(Blur=4,Type=1,Depth=3,Chroma=2)
### Overall Temporal Denoise ###
SMDegrain(tr=2,thSAD=600,ContraSharp=True,RefineMotion=True,Plane=0,Lsb=True,Lsb_Out=True,PreFilter=2)
### Debanding ###
GradFun3(thR=0.55,Radius=12,Mask=2,SMode=1,Lsb=True,Lsb_In=True, StaticNoise=True,Y=3,U=3,V=3)
Dither_Out()

with the command line additions:

--demuxer raw --input-depth 16 --input-res 640x480 --fps 23.976 --sar 10:11

I think that I'll try to improve the deinterlacing section of the script, so it's due to change again.

Thanks for your kind help, kuchikirukia, and a couple more questions if you've time ;) . I remember that the line darkening and thinning was last because that's the usual way (to my understanding). Do you think that this line is now pointless, given that it comes before the denoising-debanding? Or should it be edited or moved?

Also, any idea why the aspect ratio according to MediaInfo is 1.212? As I typed above, it displays properly so I reckon that it doesn't rally matter; I just find myself curious. Ahhh, and did I "fix" the 0.683 aspect ratio in a "correct" fashion, or is there a better way? Thanks.

Edit: it seems that these lines from MeGUI's "Clever (TM) anamorphic encoding:" setting:

global MeGUI_darx = 6480
global MeGUI_dary = 4739

were buggering the works. With them removed all is well.

Scythe42: you may want to reconsider using QTGMC for deinterlacing. When I began this project (seemingly forever ago, ha ha), I asked some questions on the QTGMC thread at Doom9 and was informed that QTGMC is not recommended for animation. Read starting here (http://forum.doom9.org/showthread.php?p=1713189#post1713189) for details.

Also, I take little credit for my script but the results are really good (to my eyes, at least). Start reading here (http://forum.videohelp.com/threads/369143-ResizersPack-MasksPack-PlaygroundPack-SmoothContrast-Logo-mod-functions?p=2382326&viewfull=1#post2382326) for more information (my Gibb's noise line was born there).

Ahhh, thanks for the idea to change my cropping from 8,0,-8,0 to 12,0,-4,0; you're right, the left side is consistently much worse than the right side :) .

[Scythe42]

Scythe42: you may want to reconsider using QTGMC for deinterlacing. When I began this project (seemingly forever ago, ha ha), I asked some questions on the QTGMC thread at Doom9 and was informed that QTGMC is not recommended for animation. Will read the thread, thanks for pointing that one out. I used it because the pattern of blended frames is so highly irregular and QTGMC/Srestore was the only way of getting rid of them quick and dirty and the result is actually quite good. But proper field matching as far as possible should be nicer for this kind of source.

Will dive into the fields again and see if I can find an pattern beside a standard IVTC that deals with these frames without making pans choppy by replacing too much or having too much duplicates.

[LouieChuckyMerry]

Will read the thread, thanks for pointing that one out. I used it because the pattern of blended frames is so highly irregular and QTGMC/Srestore was the only way of getting rid of them so far - or the only one I knew of to deal with it.

Will dive into the fields again and see if I can find an overall pattern that beside a standard IVTC deals with these frames without making pans choppy by replacing too much. Well I'll give it a shot. Not using QTGMC would speed up my encodes a lot.

I know what you mean about the poor quality of the source video. I was actually into Season 9 when I decided to upgrade to 10-bit x264, so I figured I might as well try to improve my script before starting over. The first four seasons went well using TIVTC with the settings in the above posted script, but starting with Season 5 the source material began suffering from occasional bits of field blending and other irregularities. Eventually I posted here (http://forum.videohelp.com/threads/370888-Suggestions-For-Optimizing-SD-Animation-Encoding) and received some good suggestions, but there are some source episodes in seasons 5,6,7,and 8 so bad that they require a frame-by-frame inspection then the use of ReplaceFrameNext or ReplaceFramePrev which, unfortunately, isn't automated (you have to find the frame numbers yourself). If you ever figure out the pattern, please let me know ;) .

[Scythe42]

The first four seasons went well using TIVTC with the settings in the above posted script, but starting with Season 5 the source material began suffering from occasional bits of field blending and other irregularities.With your script I stumbled directly over problems in S01E01 within the frist 1000 frames a couple of times.

TFM cannot properly match the fields and not detecting combed frames (this is where the problem comes from) as what they are and instead reporting them as clean. You try to fix this with vinverse() that doesn't really work well for that case.

As a consequence TDecimate might not drop these frames and instead going for other duplicate frames to decimate, leaving the combed frames from the hard telecine process in there that should be removed in the first place.

Also there are other issues in general due to the bad chroma on the DVDs on course over the seaoson. Some fields do not have the chroma of the last, current or next fields as it should be. In addition I found orphaned fields, combed progressive frames, or frames that consist of line doubled frames in editing a field was missing. And of course actually blended frames.

Here is something I just used on S01E01 for IVTCing for testing:

The TFM comes down to this:

Let avisynth decide the order as I am lazy
Don't use the chroma for field matching
Use previous/current/next + next/previous with opposite parity for matching
Always allow next/previous with opposite parity matches and not only on scene changes
Increased the combing threshold a bit from the default of 9 to 10 because of all the noise (may need to adjust even higher)
chroma is included to decide if a frame is combed.
render debug output into the stream for testing purposes
using pp=1 here so combed frames stay combed for processing outside TFM later, pp=5 should work fine though for a simple blend if wanted


You could also weave two fields together with TFM and then ran another TFM over it like normal as suggested in the other thread you mentioned. But that could include duplicate frames where there should be none and more judder then needed.


#Load Video
#yada yada yada

#Match Fields
tfm(order=-1, mode=5, pp=1, mchroma=false, ubsco=false, cthresh=10, chroma=true, display=true)

#Decimate Frames
TDecimate(mode=1) #haven't fine tuned yet or tested other modes yet if they make a difference.


The result was a smooth video with no combed frames except during the opening titles of course. Titles are their own can of worms...

Will continue to work on this and see if I can get it better...

[LouieChuckyMerry]

Thanks for the information, Scythe42, I really appreciate it. Honestly I didn't notice any major issues (beyond the low source quality, ha ha) in the first four seasons, but to be fair I haven't watched any of the episodes in their entirety, either. I'll find the bright side about starting over: the encodes should definitely be improved. Are you using 10-bit x264? The upgrade on my 14" 1600x900 laptop screen is obvious, so I had no choice but to begin anew looking to a (hopefully) future large screen TV. From what I've read about 10-bit x264 the only real issue is playback compatibility and the encoding time increase, but I use my laptop with an HDMI cable the infrequent occasions I'm visiting friends or family with proper TV's.

What are you using for indexing-source?

[Scythe42]

Are you using 10-bit x264? The upgrade on my 14" 1600x900 laptop screen is obvious, so I had no choice but to begin anew looking to a (hopefully) future large screen TV.Without having a 10bit display, and I mean the panel is actually 10bit and does not just accept a higher bit signal for down-sampling by the video processor, it makes no sense to me. And in addition to it there are no 10bit sources. And on top of that we talk about a different color space.

DVD and Bluray uses 8 bits per channel for 4:2:0 YCbCr, which would need actual 12 bits.

A BluRay player outputs 4:2:2 at 16 bits or 20 bits or even 4:4:4 at 24 bits. Already up-sampling is taking place. Throw Deep Color on it and you go up to 48 bits.

And then your display's actual hardware is 8bit again. Bummer.

Yes, you have all your fancy Deep Color stuff and awesome encodes just so that the display's cheap video processor dithers it down to actually display it. And even worse a ton of consumer displays only have 6bit hardware and cannot even display 8bit depths properly.

In your encodes you probably see a benefit as you didn't filter the source much, so the additional upsampling/downsampling helped you, which you probably do better with a filter to have consistent results across hardware.

Only real benefit of 10bit x264 would be for compression size for use with animation because areas are largely filled with uniform colors compared to live action. For SD source I do not care about it at all. Big HDDs are cheap anywhere.

And regarding MPEG errors compared to the uncompressed source? They are already introduced the encoding of the MPEG2 for DVD. You can't get them out again and fully restore the source material. Shit In > Shit out.

But back to deep color and stuff:

This is all consumer crap. You get a way better pictures if you actually calibrate your display as close as possible and just pass the signal through with as less post-processing as possible (beside a decent up-scaling of course).

Yes, all these fancy features might enhance stuff in one area but destroy stuff in another. All these stuff needs to be turned off on a TV. These are cheap workaround features in a displays video processor to compensate for the extreme crappy factory settings or what ppl. adjusted with their eyes.

I am watching stuff on a nearly perfectly calibrated 55" panel. Color calibration is one the most important things for good playback quality. I can't repeat this often enough. It make a 1K USD TV look like 3K USD one, and do it on a top tier device and you will be amazed. Nothing else will give you such a increase in picture quality that to actually see colors as intended.

Doing 10bit encodes of 8bit sources by up-sampling and then letting these down-sampled for playback is just a bad idea. Two unnecessary conversions. For really low quality it is like you are using additional filters during an encode, which is probably a better choice as the quality is consistent across hardware.

Playback compatibility and encoding times? Not of concern for me, my hardware can handle the format. If encoding takes longer so be it, I will only do it once for a source medium and then never again.

10bit will be thing for 4K displays with native 10bit color and UHD BluRay sources. And we will use HEVC for encoding and not AVC.

Interlaced MPEG2 DVDs look bad and will always do so. Nothing you can do about it, beside trying to remove what was introduced in mastering and clearing stuff up a bit. Nice thing is for animation you can be very aggressive here. For live action I don't use any filters only some mild deblocking and smoothing if it was a really really bad DVD and there will probably never be a remastered version. Have such exotic stuff in my collection.

What are you using for indexing-source?
Nothing. No need for me.

The MKVs I created with MakeMKV work fine. My DirectShow filters work fine here. There is a lot of voodoo still around here from days where stuff was problematic.

But sure it still works fine, but nothing I need in my workflow...

And before the usual complainig comes in: I checked with DGIndex, the result is exactly the same. No difference in the problematic areas whatsoever. Results between the DGIndex and the DirectShow approach are frame exact for me. Only difference would be by using different mpeg2 decoder. But given these low quality DVDs it doesn't matter at all.

DGIndex was as a big thing back then when using AVIs and unbearable low quality xvids encodes. And ppl. using Windows XP with whatever shitty installed MPEG2 codec they got from what ever player with a fancy UI. Similar to seeing still seeing deblock=1:-3:-3 in x264 encodes settings today.

At that time I stored remuxed MPEG2 streams on my RAID as there was no other acceptable option. And as the older Simpsons stuff will not appear on Bluray and no remasters are in sight (horrible cropped upscales started to air on TV), I decided to encode them now with way more computing power then back then.

Once done I will never re-encode them again.

But back the the encoding. Will work tonight a bit on the filtering side and see what I can make out of the DVDs with the TFM/TDecimate.

Also need to work on the processing time. It is way too high right now, so I need to try some different filters for removing all that noise and see if I can get it up to resonable speeds as doing 12 seasons would take forever.

[videoh]

DGIndex was as a big thing back then when using AVIs and unbearable low quality xvids encodes. Nonsense. The advantage of DGIndex is the robust frame accuracy. If you encode with purely linear access, then sure, DGIndex has no particular advantage over DirectShowSource(). But if you have a non-linear script that requires frame accuracy, then DGIndex (and other indexed source filters) comes to the fore.

[Scythe42]

Nonsense. The advantage of DGIndex is the robust frame accuracy. If you encode with purely linear access, then sure, DGIndex has no particular advantage over DirectShowSource()..I never talked about anything else this thread.

And also back in the days, as I mentioned, DGIndex was a big thing, because even pure linear access to video streams had a lot of issues.

I never said that there is no reason for indexing.

[LouieChuckyMerry]

Without having a 10bit display, and I mean the panel is actually 10bit and does not just accept a higher bit signal for down-sampling by the video processor, it makes no sense to me. And in addition to it there are no 10bit sources. And on top of that we talk about a different color space.

DVD and Bluray uses 8 bits per channel for 4:2:0 YCbCr, which would need actual 12 bits.

A BluRay player outputs 4:2:2 at 16 bits or 20 bits or even 4:4:4 at 24 bits. Already up-sampling is taking place. Throw Deep Color on it and you go up to 48 bits.

And then your display's actual hardware is 8bit again. Bummer.

Yes, you have all your fancy Deep Color stuff and awesome encodes just so that the display's cheap video processor dithers it down to actually display it. And even worse a ton of consumer displays only have 6bit hardware and cannot even display 8bit depths properly.

In your encodes you probably see a benefit as you didn't filter the source much, so the additional upsampling/downsampling helped you, which you probably do better with a filter to have consistent results across hardware.

Only real benefit of 10bit x264 would be for compression size for use with animation because areas are largely filled with uniform colors compared to live action. For SD source I do not care about it at all. Big HDDs are cheap anywhere.

I agree with what you typed, but from what I've read the other main advantage of 10-bit over 8-bit (besides better compression) is that all those scaling stages would benefit greatly from the 20% (8-->10) more information with which to work (thus improving overall picture quality).

But back to deep color and stuff:

This is all consumer crap. You get a way better pictures if you actually calibrate your display as close as possible and just pass the signal through with as less post-processing as possible (beside a decent up-scaling of course).

Yes, all these fancy features might enhance stuff in one area but destroy stuff in another. All these stuff needs to be turned off on a TV. These are cheap workaround features in a displays video processor to compensate for the extreme crappy factory settings or what ppl. adjusted with their eyes.

I am watching stuff on a nearly perfectly calibrated 55" panel. Color calibration is one the most important things for good playback quality. I can't repeat this often enough. It make a 1K USD TV look like 3K USD one, and do it on a top tier device and you will be amazed. Nothing else will give you such a increase in picture quality that to actually see colors as intended.[Emphasis LouieChuckyMerry]

Thanks, I'll surely remember this when I (hopefully) have a large screen TV.

Nothing. No need for me.

The MKVs I created with MakeMKV work fine. My DirectShow filters work fine here. There is a lot of voodoo still around here from days where stuff was problematic.

But sure it still works fine, but nothing I need in my workflow...

And before the usual complainig comes in: I checked with DGIndex, the result is exactly the same. No difference in the problematic areas whatsoever. Results between the DGIndex and the DirectShow approach are frame exact for me. Only difference would be by using different mpeg2 decoder. But given these low quality DVDs it doesn't matter at all.

DGIndex was as a big thing back then when using AVIs and unbearable low quality xvids encodes. And ppl. using Windows XP with whatever shitty installed MPEG2 codec they got from what ever player with a fancy UI. Similar to seeing still seeing deblock=1:-3:-3 in x264 encodes settings today.

At that time I stored remuxed MPEG2 streams on my RAID as there was no other acceptable option. And as the older Simpsons stuff will not appear on Bluray and no remasters are in sight (horrible cropped upscales started to air on TV), I decided to encode them now with way more computing power then back then.

Once done I will never re-encode them again.

But back the the encoding. Will work tonight a bit on the filtering side and see what I can make out of the DVDs with the TFM/TDecimate.

Also need to work on the processing time. It is way too high right now, so I need to try some different filters for removing all that noise and see if I can get it up to resonable speeds as doing 12 seasons would take forever.


Sorry, I'm confused: are you using DirectShow filters to watch them from the MakeMKV .mkv's or are you encoding them with an indexer, etc.?

And as the older Simpsons stuff will not appear on Bluray and no remasters are in sight (horrible cropped upscales started to air on TV)...

Don't get me started on those cropped abominations...

[Scythe42]

Sorry, I'm confused: are you using DirectShow filters to watch them from the MakeMKV .mkv's or are you encoding them with an indexer, etc.?I am not using any Indexer, because I am not randomly accessing frames or jumping around, where I need a significant speed boot to access specific frames faster or face the danger of getting a wrong frame by from a directshow filter.

The fields come one after another from the MKV container anyway, so additional indexing doesn't do much for me and just blows up my workflow as I have to prepare more stuff instead of just remuxing the DVD tracks to MKVs and then batch process them later.

I know that using an indexer is in theory more reliable, but honestly never had a need for one for encoding at all. As already mentioned indexing was a must back in the SD xvid days when you even had problems getting frames realiable in a linear fashion from a stream.

And when it comes to actual editing. dubbing or subbing foreign releases I use other tools and then just process the video with avisynth for encoding.

I ripped about 1000 DVDs from my collection over the years with MakeMKV. Found it to be the most reliable remuxing tool for DVDs. It came far over the years. Rarely I find a disc that requires other approaches. But yes, these exist. For BluRays I have a different workflow.

I kept all the tracks in MPEG2 of my collection for later encoding. Because back when I started putting my media on a RAID system the CPU power was too low for the kind of encoding quality I had in mind.

I load everything in AviSynth just with DirectShowSource. Nothing more nothing less.

Yes I know that ppl. will scream at me here, but it's working really nice for me with zero issues. Yes, certain discs gave me problems and required manual work. But these were few.

And here is what I recent revision for the fist few season for the Simpsons NTSC. Needs probably another round of adjustments.

I played around with more filtering, frame matching and so on to find a good middle ground between quality and processing speed.

1st Pass

#Load Video
DirectShowSource("E:\ENCODE\S01E01.mkv",audio=false).ConvertToYV12())

#Match Fields - could be optimized in regards to combing detection
tfm(order=-1, mode=5, pp=1, slow=2, mchroma=false, ubsco=false, cthresh=14, chroma=true, output="matches.txt")


2nd Pass

#Load Video
DirectShowSource("E:\ENCODE\S01E01.mkv",audio=false).ConvertToYV12())

#Remove Dot Crawl - only use in 2nd pass or it will screw up frame matching.
Checkmate(thr=10, max=20, tthr2=0)

#Match Fields - could be optimized in regards to combing detection
tfm(order=-1, mode=5, pp=2, clip2=Deint(mode=2, type=3), input="matches.txt")

#Decimate
TDecimate(mode=1)

#Temporal Smoothing
FluxSmoothST()

#Spatial Smoothing
Deen(mode="a2d", rad=4, thrY=7, thrUV=9, min=0.5)

#Clean Edges - remove left overnoise near edges
Edgecleaner(strength=20, smode=1)

#Line Darken without thinning
ToonLite(strength=0.75)

#Color Debanding in case smoothing created bands from noise/blocking
GradFun3(smode=2)

#Anti-Aliasing for aliasing present on source or introduced by filtering
maa2()

#Sharpen
PSharpen(strength=25, threshold=50, ss_x=1.5, ss_y=1.5)

#Crop - could crop more though (e.g. 2 lines top and bottom and just edges for that).
Crop(12,0,-6,0) #not ITU compliant more on the left than right

#Resize
#Spline36Resize(640,480) # handled by x264 in batch processing

#using crf16, veryslow preset with increased rc-lookahead=250 and merange=48. Probably total overkill for this source.

Problems left:
- Credits and background are not following the same telecine pattern. Broken on the DVD, not possible to restore.
- Frames with just line-doubled single fields exist in the source and are strongly visible
- Picture is not fully stable, but the early Simpsons never were

A QTGMC/SRestore approach could solve these issues, but might introduce noise somewhere else. You could run the post-processing stuff of QTGMC on a progressive frame though (e.g. QTGMC(InputType=1, Preset="slow", EZDenoise=2.0). But it takes too long, so I decided to not work on these issues. The only really annoying one are line doubled fields now and then.

For just testing the quality of individual frames move the Checkmate() after the TDecimate. Dot Crawl removal on the interleaved frame will screw up field matching on this source, so a two pass approach is required. And if using on progressive frames creates problems of courses.

I also found out that some Anti-Aliasing helps after the smoothing, cleaning and enhancing the edges, followed by very subtle sharpening. That gave me the best edges I could get out of the source without boosting the processing time.

After all this cleanup work I finally crop and resize, gives better result doing it after denoising with this source.

The result is actually watchable in 1080p. And an episode only needs about 35-40 minutes to encode for me, which is very nice give the large number of seasons. Trade offs needs to be made.

Yes, the picture is very smoothed to get all the noise and blocking out and some details are lost along the way. But looks scaled up on a big ass TV better this way. Yeah, some real HD Simpsons would be so nice.

If you try out my suggestion let me know what you think about the result. I might work on the TFM and fine tune to combing detection parameters a bit though.

Onwards to check how this works on later seasons, while re-encoding the first season again for spot checking how it turns out.

[LouieChuckyMerry]

I'd be happy to try your script, but a question first. When you typed:

#Match Fields - could be optimized in regards to combing detection
tfm(order=-1, mode=5, pp=2, slow=2, clip2=Deint(mode=2, type=3), mchroma=false, ubsco=false, cthresh=14, chroma=true)

did you mean:

#Match Fields - could be optimized in regards to combing detection
tfm(order=-1, mode=5, pp=2, slow=2, clip2=TDeint(mode=2, type=3), mchroma=false, ubsco=false, cthresh=14, chroma=true)

Also, what's the "ubsco" call; I couldn't find an answer searching.

[Scythe42]

I'd be happy to try your script, but a question first. When you typed:

Oh that was just a typo here as I copy and pasted stuff in, missed the "T" when copying pasting. Yes, of course it is TDeint. You can use any deinterlacer you want or stuff from TFM itself. I find TDeint working nicely for me here when there were obviously combed part of the picture.

Also, what's the "ubsco" call; I couldn't find an answer searching.

From the readme of version 1.5:


ubsco -

Allow u/b matches to be used only around scenechanges. The 'scthresh' parameter sets the
scenechange threshold as a percentage of maximum change on the luma plane. To disable
this (allow u/b matches everywhere) set ubsco to false. The ubsco setting only effects
automatic matching, i.e. u/b matches via an overrides or input file are not changed.
The only matching modes that ubsco is used with are 2, 3, 5, and 6 (the only ones that
use u/b matches).

Default: true (bool)


With the source we have the problem that the chroma of a frame does often not belong to the previous, current or next field. Instead a match is found in the opposite polarity in the previous or next frame. Setting chroma=true was not enough for me here to give TDecimate a proper steady sequence for frame removal.

So the combo "mchroma=false, ubsco=true, chroma=true" worked for me so far. Will see if this hold up across more episodes, seasons.

Also on this source you could get away with using Dot Crawl removal after field matching, even though running Checkmate() on a progressive frame is just totally wrong it seems to work here nicely.

Only thing that bothers me are the line doubled fields that show up now and then and the unstable picture. Well maybe I'll find something to make it less visible by playing with some parameters. A QTGMC/Srestore approach would smooth all of this out though, but takes very long.

Off running some more experiments for these specific cases...

Edit:
Meh, found some frames that still show combing and are not filtered out way later in my test episodes then I checked before. Even though a better frame exists.I tackled most of then with the above approach. But a few minutes later some show up again. TDEint doesn't really do a good job on all these frames compared to the other ones. *sigh*. OK, one more evening of testing then I am back to QTGMC/srestore for good as the source is too messed up to reliable automatically ITVC it with whatever approach.

[Scythe42]

And now the extreme "I have no other ideas" attempt.

TFM as before, but if a cropped frame is detected for which no matching field can be found because it is missing or has been used to reconstruct another frame, I now take a field from another frame anyway that matches or if non just deinterlace the frame. Yeah, very brutal attempt on these blended frames.

Lowered the combed threshold a bit, a few frames with combing still might be there. Probably best to adjust cthresh, blockx, blocky and MI here for that source a bit more, but it is probably impossible to get it 100% right.

Dot Crawl removal after field matching works nicely on this source, even though it is totally wrong.

And then to deal with the occasional line doubled frames I put a needi3 and merge the result again in post-processing. I have no clue if this really makes sense or introduces crap somewhere else. Just the only thing I could think of to get these frames to look acceptable. And it seems to work nicely. You can just remove the two lines and leave the frames as they are.

And because every now and then, sometimes consistently once per second over longer time, we have some shimmering probably because of phase shifting during the episodes that is sometimes even variable. Man, did they butcher the episodes during mastering with whatever crappy filter they used. So letting QTGMC process the now progressive frame.

I wish I could get it of the shaking between frames now and then, but Stab() is not really helping here. Even though it works nice with the small constant shaking it makes pans and zooms juddery, probably there isn't enough smooth movement to begin with. Don't know if some good animation deshaker exists.


#VideoSource
DirectShowSource("E:\ENCODE\S01E01.mkv",audio=false).ConvertToYV12()

# based on http://forum.doom9.org/showpost.php?p=846781&postcount=10
function FieldMatch(clip c) {
global pp = c.DuplicateFrame(0)
global cc = c
global nn = c.DeleteFrame(0)
p2 = pp.SeparateFields()
c2 = cc.SeparateFields()
n2 = nn.SeparateFields()
global pc = Interleave(p2.SelectEven(), c2.SelectOdd()).Weave()
global cp = Interleave(c2.SelectEven(), p2.SelectOdd()).Weave()
global cn = Interleave(c2.SelectEven(), n2.SelectOdd()).Weave()
global nc = Interleave(n2.SelectEven(), c2.SelectOdd()).Weave()
global deint = QTGMC(cc).SelectEven()
return ScriptClip(cc, \
"!cc.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? cc : " + \
"!nn.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? nn : " + \
"!cn.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? cn : " + \
"!nc.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? nc : " + \
"!pp.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? pp : " + \
"!cp.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? cp : " + \
"!pc.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? pc : deint")
}

#Match Fields
tfm(order=-1, mode=5, pp=2, clip2=FieldMatch(), slow=2, mchroma=false, ubsco=false, cthresh=12, chroma=true)

#Decimate
TDecimate(mode=1)

#Remove Dot Crawl
Checkmate(thr=10, max=20, tthr2=0)

#Fix line aliasing of line doubled frames in source
nnedi3(field=-2)
Merge(SelectEven(), SelectOdd())

#Fix horizontal shimmering caused by phase shifts in source
QTGMC(InputType=2)

#Temporal Smoothing
FluxSmoothST()

#Spatial Smoothing
Deen(mode="a2d", rad=4, thrY=7, thrUV=9, min=0.5)

#Clean Edges - remove leftover noise near edges
Edgecleaner(strength=20, smode=1)

#Line Darken without thinning
ToonLite(strength=0.75)

#Color Debanding in case smoothing created bands from noise
GradFun3(smode=2)

#Anti-Aliasing for aliasing introduced by filtering
maa2()

#Sharpen
PSharpen(strength=10,ss_x=1.5, ss_y=1.5)

#Crop
Crop(12, 0, -4 ,0)

#Resize
Spline36Resize(640, 480)


Running this over some episodes and check the results and see how it turns out. The first few finished looking promising...

[LouieChuckyMerry]

Thanks for the Usbco information; perhaps next time I'll skip the online search and check my plugins folder instead ;) . I ran your script from Post #27. Impressive! The only real difference I can see is that the colors look less "bright" and there's slightly less detail than with my script (probably because of the masking). And your script is ~2.5x faster, ha ha, and your deinterlacing is light years ahead of mine. Seriously, your TIVTC lines were able to deal automatically with field blending I had to sort manually in Season 5 (S5.E4-TestClip[FieldBlending] (http://www.mediafire.com/download/57vu6hscezxft5z/S5.E4-TestClip[FieldBlending].7z)). I'm really curious for the results of your latest tests (and please don't flee to the evil clutches of QTCMC). I wish I had more time...

Ahhh, have you tried Stab(Mirror=15)? This gives me much better results than Stab().

[Scythe42]

I smoothed the picture out a lot at the cost of some detail and colors to remove all the noise. I am probably way to aggressive here.

Maybe some adjustment to the Deen() can be made here or I'll replace it with something less aggressive. Need to try out a couple of smoothers. FFT3DFilter seems to work better but it takes way too long.

Your masking preserved more detail, but took too much time for my taste, which I decided to spend on other issues (line doubled frames and horizontal shimmering) with post #30 (this stuff is optional of course and not related to the filed patterns - takes a lot of time as well).

I was mainly focused on getting the TFM right in the last few days after you pointed out the QTGMC issues which I started to see in my test encodes.

Glad to hear my TFM approach solved you test clip cases. The approach still cannot match everything that's why I added a field matching function in post #30 to take care of the rest. As frames are repeated there is a good chance to re-use a field again for a proper match.

Regarding Stab: yes, tried it with Mirror(15). It works very nicely to solve the minimal shaking of few 1-2 pixels in still scenes and fill up the borders. But as soon as a camera pan or a zoom comes, it creates judder in the motion because the pans/zoom look to it like judder, which I can fully understand. Not enough movement information I guess, so it fails there and no parameters to tune the behavior a bit more. I need to check for some other stabilizers that could help here. Understanding what the Stab() script does and adjust it myself is beyond my pay grade though.

But I'll work a bit on the smoothing next and see if I can come up with something I like more and doesn't take too much additional time. I'll keep you posted with my efforts.

[LouieChuckyMerry]

How did your tests with the frame matching from Post #30 turn out? I hope to have some time this weekend to use your TCM-TDeint lines from Post #27 on some sections in Seasons 5,6,7, and 8 with field blending so bad that I was forced to use the manual method involving ReplaceNext or ReplacePrev, so I'm curious about your newest (Post #30) attempts. I'll also check for juddering on pans in my earlier encodes; I really don't remember anything bad enough to catch my eye but, again, I've not watched any of the episodes in their entirety. Ahhh, as you understand these things better than I, check here (http://forum.videohelp.com/threads/371336-Stabilization-Tools-Pack-v1-7) for a possible solution to your shaking (and as an added bonus the author is usually quite willing to help).

[Scythe42]

I haven't made much more changes, just moving the temporal smoothing after the spatial smoothing as it should be, which of course does nothing real visible beside helping compression.

Experimented with DePanEstimate and DePan a bit. Can stabilize the picture a bit but don't really like the overall result as it has impact on pans and zooms. Tried many variations, didn't really work out for me.

What really works awesome is: http://avisynth.nl/index.php/CelStabilize - but could not find a recent version and not much documentation. And it is slooooow. Too bad, this would probably be perfect. I tried to mimic what is does with masks and then moving these so they fit and apply the result on the actual picture. That works nicely here, but is again terrible slow so not really usable.

I kept the Deen() as it was the only filter that got the noise out as much as I wanted. Tried other filters and they left too much of the dirty noise for my taste. There isn't much detail in the first seasons anyway, so no real harm. Your mask approach is still probably better, but takes too long for my taste.

Basically everything starting with the Deen can be changed to your liking. You can basically use any filter you like and probably some less intrusive. As this might change a bit in later seasons, some adjustments are probably in order on a per season base.

Let me know what combo you finally used and I give it a shot.

The nnedi3() is kinda optional as well, depending if an epsiode has line double fields for a frame or not (or if you do not care about them). But should cause no harm and removes all the had aliasing that an maa2() or similar cannot really fix and is already in the source.

The QTGMC is very effective to remove the horizontal shimmering because of all the luma shifting as mentioned before. But not all epsiodes show this. So again best to check the MPEG2 source if it's needed, but should cause no real harm.

Time for encoding some seasons and check the results...


#VideoSource
DirectShowSource("E:\ENCODE\S01E01.mkv", audio=false).ConvertToYV12()

# based on http://forum.doom9.org/showpost.php?p=846781&postcount=10
function FieldMatch(clip c) {
global pp = c.DuplicateFrame(0)
global cc = c
global nn = c.DeleteFrame(0)
p2 = pp.SeparateFields()
c2 = cc.SeparateFields()
n2 = nn.SeparateFields()
global pc = Interleave(p2.SelectEven(), c2.SelectOdd()).Weave()
global cp = Interleave(c2.SelectEven(), p2.SelectOdd()).Weave()
global cn = Interleave(c2.SelectEven(), n2.SelectOdd()).Weave()
global nc = Interleave(n2.SelectEven(), c2.SelectOdd()).Weave()
global deint = QTGMC(cc).SelectEven()
return ScriptClip(cc, \
"!cc.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? cc : " + \
"!nn.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? nn : " + \
"!cn.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? cn : " + \
"!nc.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? nc : " + \
"!pp.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? pp : " + \
"!cp.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? cp : " + \
"!pc.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? pc : deint")
}

#Match Fields
tfm(order=-1, mode=5, pp=2, clip2=FieldMatch(), slow=2, mchroma=false, ubsco=false, cthresh=12, chroma=true)

#Decimate
TDecimate(mode=1)

#Remove Dot Crawl
Checkmate(thr=10, max=20, tthr2=0)

#Fix line double fields
nnedi3(field=-2)
Merge(SelectEven(), SelectOdd())

#Fix horizontal shimmering
QTGMC(InputType=2)

#Spatial Smooter
Deen(mode="a2d", rad=4, thrY=7, thrUV=9, min=0.5)

#Temporal Smoothing
FluxSmoothST()

#Clean Edges - remove leftover noise near edges
Edgecleaner(strength=20, smode=1)

#Line Darken without thinning
ToonLite(strength=0.75)

#Color Debanding in case smoothing created bands from noise
GradFun3(smode=2)

#Anti-Aliasing for aliasing introduced by filtering
maa2()

#Slightly Sharpen to remove filtering artifacts at lines
PSharpen(strength=10, ss_x=1.5, ss_y=1.5)

#Crop
Crop(12, 0, -4, 0)

#Resize
Spline36Resize(640,480)

[ChiDragon]

Just curious, is your guys' goal here to make it look like the show was virtually-painted (Reliance MediaWorks' "restoration" of the series (https://vimeo.com/97466924)), like the physically-painted cells were captured digitally (Disney Blu-ray style, I guess), or like the film frames? I'm guessing not the latter from the screenshot comparison (which I found very impressive, though too smooth for my own personal taste).

[Scythe42]

Just curious, is your guys' goal here to make it look like the show was virtually-painted (Reliance MediaWorks' "restoration" of the series (https://vimeo.com/97466924)), like the physically-painted cells were captured digitally (Disney Blu-ray style, I guess), or like the film frames? I'm guessing not the latter from the screenshot comparison (which I found very impressive, though too smooth for my own personal taste).Yes, I want to make it look like painted cells exactly like in the MediaWorks' video you linked or any given Disney BluRay of an old movie.

Why do I want to create the cell animation look? Because that's what the Simpsons ultimately are. Up to season 13 the Simpsons were traditional hand drawn cells.

Only after season 13 they started to use computers. And starting with season 13 the Simpsons are available on BluRay or iTunes in HD already (with the exception of season 18, for which I only have my own crappy SD TV captures from some re-run as not even a DVD version exist).

Getting it exactly like in the MediaWorks' sample would be perfect. Still not there yet, but I wasn't able to find a better filter combination.

Currently working with filters on an upscaled versions trying to get better overall edges and interpolate details in. That should counter the smoothing effect after cleaning up. Compared to professional software my options are of course limited.

Yes, the result currently is too smooth with all the noise out at the moment.

But the source is so crappy, especially in the early seasons. With later seasons I can tone down the filters and even remove some of them.

Some stuff is just in for dealing with certain episodes. I use S01E01 as a reference, which is a horrible tape transfer. If I can get this to an acceptable level for me, the rest should be easy as this episode has all the problems. For other episodes and seasons I just remove filters or turn their settings down.

In the end it is just the lines that are too blurry. But using any of the sharpening filters just create aliasing or halos not making it better. Have not yet found a good approach yet.

If you have some ideas, I am eager to hear them.

[Scythe42]

Here are some comparisons with the latest version of my script as posted.

Note I upscaled them to 960x720 in avisynth before taking a screenshot, so you an better see the differences. Naturally upscaled everything looks a bit more blurry..

Frame 1: Line Doubling on the NTSC DVD
https://c1.staticflickr.com/1/264/19273813842_00e791865b_o.jpg
https://c1.staticflickr.com/1/397/18659251063_c5b5ce8afb_o.jpg

Frame 2: Mosquito noise on the NTSC DVD
https://c1.staticflickr.com/1/444/18659257923_6f3c4a1a32_o.jpghttps://c1.staticflickr.com/1/488/19273820792_6da89963fa_o.jpg

Frame 3: Smeared edges on the NTSC DVD source
https://c1.staticflickr.com/1/285/18659266603_852eeb8878_o.jpghttps://c4.staticflickr.com/4/3853/19279807795_a9d1aafbf2_o.jpg

Frame 4: Horizontal shimmering between frames during playback on the NTSC DVD
https://c1.staticflickr.com/1/385/19273834132_4e5605f3e2_o.jpghttps://c4.staticflickr.com/4/3774/19273833342_922724cce2_o.jpg

Frame 4: Orphaned Field on the NTSC DVD
https://c1.staticflickr.com/1/284/19093703619_52af11f3ca_o.jpghttps://c1.staticflickr.com/1/310/19253750206_0b3d179ef6_o.jpg

As you can see, there is not much detail in the source to begin with. Natural some details are lost as they cannot really be distinguished from noise.

Also the Deen()/GradFun3() brightens the overall colors a bit. You can compensate for this possible with a simple tweak(). But the colors are off on DVD already and vary between scenes a a lot.

I probably adjust the script so the skin tone of the characters is not too far off from what it should be. Pantone 116C, #FECB00, should be the right color for Caucasian Simpson's characters. I'll include a line I can use for overall color adjustment before re-sizing. Need to check how it works out on the course of a season and if it's worth the trouble.

Open to suggestions for improvements!

[LouieChuckyMerry]

Yes, I want to make it look like painted cells exactly like in the MediaWorks' video you linked or any given Disney BluRay of an old movie.

Why do I want to create the cell animation look? Because that's what the Simpsons ultimately are. Up to season 13 the Simpsons were traditional hand drawn cells.

Only after season 13 they started to use computers. And starting with season 13 the Simpsons are available on BluRay or iTunes in HD already (with the exception of season 18, for which I only have my own crappy SD TV captures from some re-run as not even a DVD version exist).

Getting it exactly like in the MediaWorks' sample would be perfect. Still not there yet, but I wasn't able to find a better filter combination.

Currently working with filters on an upscaled versions trying to get better overall edges and interpolate details in. That should counter the smoothing effect after cleaning up. Compared to professional software my options are of course limited.

Yes, the result currently is too smooth with all the noise out at the moment.

But the source is so crappy, especially in the early seasons. With later seasons I can tone down the filters and even remove some of them.

Some stuff is just in for dealing with certain episodes. I use S01E01 as a reference, which is a horrible tape transfer. If I can get this to an acceptable level for me, the rest should be easy as this episode has all the problems. For other episodes and seasons I just remove filters or turn their settings down.

In the end it is just the lines that are too blurry. But using any of the sharpening filters just create aliasing or halos not making it better. Have not yet found a good approach yet.

If you have some ideas, I am eager to hear them.

Interesting about the history, thanks (now I understand why the Blu-ray releases begin with Season 13). It's a real shame the recent "restorations" are cropped (seems to me that eliminating ~16% of each video frame is the antithesis of restoration ;) ). Oh well, now I have a project.

I finally found time to run some tests on field blended sections that I'd previously had to fix manually. I used three different methods. One:

TFM(Order=-1,Mode=5,PP=2,Slow=2,Clip2=TDeint(Mode=2,Type=3),MChroma=False,Ubsco=False,CThresh=14,Chroma=True)
TDecimate(Mode=1)

two:

Function FieldMatch(Clip C) {
global pp = c.DuplicateFrame(0)
global cc = c
global nn = c.DeleteFrame(0)
p2 = pp.SeparateFields()
c2 = cc.SeparateFields()
n2 = nn.SeparateFields()
global pc = Interleave(p2.SelectEven(), c2.SelectOdd()).Weave()
global cp = Interleave(c2.SelectEven(), p2.SelectOdd()).Weave()
global cn = Interleave(c2.SelectEven(), n2.SelectOdd()).Weave()
global nc = Interleave(n2.SelectEven(), c2.SelectOdd()).Weave()
global deint = QTGMC(cc).SelectEven()
return ScriptClip(cc, \
"!cc.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? cc : " + \
"!nn.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? nn : " + \
"!cn.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? cn : " + \
"!nc.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? nc : " + \
"!pp.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? pp : " + \
"!cp.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? cp : " + \
"!pc.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? pc : deint")
}
TFM(Order=-1,Mode=5,PP=2,Clip2=FieldMatch(),Slow=2,MChroma=False,Ubsco=False,CThresh=12,Chroma=True)
TDecimate(Mode=1)
NNEDI3(Field=-2)
Merge(SelectEven(), SelectOdd())

and three:

QTGMC("Preset",TR0=1,TR1=1,TR2=0)
SRestore(FRate=23.976,Speed=-1,Thresh=12,Cache=50)

The first two are yours and the third I bumped into here (http://forum.videohelp.com/threads/359308-How-to-improve-chances-of-SRestore-blend-detection).

All seemed to work equally well with regards to field matching-decimating-decombing, at least on the clips I tested, but the QTGMC-SRestore method varies as to picture quality depending on which QTGMC preset is used, I think due to the denoise settings. For example, the difference in the output video using the "Slow" preset and using the "Ultra Fast" preset with "ProcessNoise=0" is very apparent. I've asked on the Doom9 QTGMC thread what QTGMC settings to use so that QTGMC does the least intensive noise processing when combined with SRestore; if I could use QTGMC as a bobber (?) only, with zero noise processing, then I'd be stoked. In your above second method, does QTGMC do any noise processing? As soon as I can sort the field matching-decimating-decombing lines I'll focus on the rest.

Edit: the speeds for the three methods were similar, with the first method fastest, the second method about 5% slower, and the third method anywhere from 5-25% slower depending on the preset used.


Just curious, is your guys' goal here to make it look like the show was virtually-painted (Reliance MediaWorks' "restoration" of the series (https://vimeo.com/97466924)), like the physically-painted cells were captured digitally (Disney Blu-ray style, I guess), or like the film frames? I'm guessing not the latter from the screenshot comparison (which I found very impressive, though too smooth for my own personal taste).

My goal is to make the episodes look better than the sources :D .

[Scythe42]

In your above second method, does QTGMC do any noise processing? As soon as I can sort the field matching-decimating-decombing lines I'll focus on the rest.The QTGMC in the FieldMatch function is only used in case it is an orphaned field where no matching 2nd field can be found at all. I have to deinterlace them or to be specific in this case interpolate or stick with a combed or blended frame, which I do not want.

You can completely remove the QTGMC and use the variable "cc" directly to keep the combed/blended frame if you like instead of using the value of deint as the default later.

Like this:

#Match Fields
#http://forum.doom9.org/showpost.php?p=846781&postcount=10
function FieldMatch(clip c) {
global pp = c.DuplicateFrame(0)
global cc = c
global nn = c.DeleteFrame(0)
p2 = pp.SeparateFields()
c2 = cc.SeparateFields()
n2 = nn.SeparateFields()
global pc = Interleave(p2.SelectEven(), c2.SelectOdd()).Weave()
global cp = Interleave(c2.SelectEven(), p2.SelectOdd()).Weave()
global cn = Interleave(c2.SelectEven(), n2.SelectOdd()).Weave()
global nc = Interleave(n2.SelectEven(), c2.SelectOdd()).Weave()
return ScriptClip(cc, \
"!cc.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? cc : " + \
"!nn.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? nn : " + \
"!cn.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? cn : " + \
"!nc.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? nc : " + \
"!pp.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? pp : " + \
"!cp.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? cp : " + \
"!pc.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? pc : cc")
}


I did not tune any QTGMC settings beside selecting a preset as I do not care much what it does to these frames. They are already crap as one half is missing, can only get better. There should do no noise processing as per readme NoiseProcess has a default of 0 (same for EZDenoise). So no additional noise removal takes place beside what is part of the deinterlacing approach anyway.

Your third one:

QTGMC("Preset",TR0=1,TR1=1,TR2=0)
SRestore(FRate=23.976,Speed=-1,Thresh=12,Cache=50)


Is a standard QTGMC/SRestore, nothing special about it.

The TRx settings are the same for any Faster to Ultra Fast preset. You can totally omit them and just go with the preset of "Faster" or higher and keep it simple. In case you care for the preset settings check like 152 and onward in QTGMC.avsi.

For the SRestore(), you used with Speed=-1, the most precise detection. This is for very detailed HD material according to the readme.

On the Simpsons it would detect some frames as different frames just because of the noise instead of seeing that they are the same. It is a speed-vs-detail setting, the lower the number the slower it gets as it is looking for more details.

The default of 9 should work fine here. If you want some tuning then its probably better to increase the value. But anyway, as so many frames are repeated in cell drawn animation it doesn't really matter which one is removed. Especially not as QTGMC double the rate in addition.

Same for Thresh. You set it to the lowest possible value, meaning your source jas very little temporal artifacts. Well the Simpsons have a lot, just the DotCrawl alone. Again you went into the opposite direction you probably intended to.

And finally Cache=50. This setting is first invalid as the setting according to the readme only goes from -1 to 10. Second, you only specify a cache if you are going non-linear, usually in a multipass approach. But your script and the encode are purely linear one pass approach. Frame by frame gets encoded. So enabling it makes no real sense beside memory hogging and slower processing.

Your QTGMC/SRestre lines are the same as using this:


QTGMC(preset="Faster") # or Very Fast, Super Fast, Ultra Fast or Draft.
SRestore(frate=23.976)


Beside choosing a faster preset then I did initially, this will give you exact the QTGMC artifacts you pointed me out to and which I was able to see and changed my mind to figure this out with TFM. Why the change of mind now? Yes the result is still perfectly OK for the Simpsons. QTGMC/SRestore is a valid approach.

Yes, the frames between all approaches look similar until you come to specific combed/blended fields where problems arise or you can see meaningful differences.

My approach at the moment (see last version of my script) is the following:


TFM matching as before dealing with the bad chroma, plus a FieldMatch function that looks for a near matching field in case TFM cannot because a field has already been used. If this function cannot find a match and a frame is still decided to be combed (you could tune the setting of TFM for combing detection if you want), then QTGMC just deinterlaces the frame as there is no other option and as we have a double rate I just take the first frame and be done with it.

TDecimate with no special options, as we have a 3:2 pulldown now basically - or should have. I still used mode = 1 instead of mode = 0, cause of all the double frames in animation to give TDecimate some leeway. You could increase the cycle if you want to, but should not make any differences on this source.

Dot Crawl removal. You can use any other method here. But I found that Checkmate() works nicely even the frames are now progressive. You can use what ever you want.

needi3() upscale with frame doubling which are then merged. This removes the nasty line doubled frames that occur now and then as strong aliasing. Also deal already with some of the noise. See 1st example of my last post. Again if you don't mind these few frames or deal with them in other ways, no need to use that lines.

QTGMC in progressive mode. This cleans up all the horizontal shimmering. The phasing you see in black lines usually once a second on in various episodes in a lot of scenes, because the luma is shifting. This also happens on a lot of DVDs where a shitty filter was used during production. A very prominent example would be the "Crusade" DVDs (short lived spin off of Babylon 5). Again if an episode does not has it no need to use it. Use what ever works for you here, not related to deinterlacing at all.

Then all the other post processing and noise removal stuff to my liking as the source has now somewhat decent progressive frames.

As said, the nnedi3() and the QTGMC in progressive mode are optional if an episode does not show these problems for faster processing. On clear frames they do not do any real harm beside removing some of the noise. Again you can use whatever you want to deal with these problems, has nothing to do with IVTCing here.

As said with the following my ITVCing for the Simpsons is done.

Will only adjust once I come across seasons that are more properly encoded or show completely new patterns.. We'll see if that happens.

#Match Fields
# based on http://forum.doom9.org/showpost.php?p=846781&postcount=10
function FieldMatch(clip c) {
global pp = c.DuplicateFrame(0)
global cc = c
global nn = c.DeleteFrame(0)
p2 = pp.SeparateFields()
c2 = cc.SeparateFields()
n2 = nn.SeparateFields()
global pc = Interleave(p2.SelectEven(), c2.SelectOdd()).Weave()
global cp = Interleave(c2.SelectEven(), p2.SelectOdd()).Weave()
global cn = Interleave(c2.SelectEven(), n2.SelectOdd()).Weave()
global nc = Interleave(n2.SelectEven(), c2.SelectOdd()).Weave()
global deint = QTGMC(cc).SelectEven()
return ScriptClip(cc, \
"!cc.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? cc : " + \
"!nn.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? nn : " + \
"!cn.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? cn : " + \
"!nc.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? nc : " + \
"!pp.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? pp : " + \
"!cp.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? cp : " + \
"!pc.IsCombedTIVTC(cthresh=12, chroma=true, blockx=16, blocky=32) ? pc : deint")
}

video=tfm(video, order=-1, mode=5, pp=2, clip2=FieldMatch(video), slow=2, mchroma=false, ubsco=false, cthresh=12, chroma=true)

#Decimate
video=TDecimate(video, mode=1)


And I don't think I can get it any better for an automatic approach. Only other option would be QTGMC/SRestore which brings on its own problems to which you pointed me.

[LouieChuckyMerry]

The QTGMC in the FieldMatch function is only used in case it is an orphaned field where no matching 2nd field can be found at all. I have to deinterlace them or to be specific in this case interpolate or stick with a combed or blended frame, which I do not want.

You can completely remove the QTGMC and use the variable "cc" directly to keep the combed/blended frame if you like instead of using the value of deint as the default later.

Like this...

...I did not tune any QTGMC settings beside selecting a preset as I do not care much what it does to these frames. They are already crap as one half is missing, can only get better. There should do no noise processing as per readme NoiseProcess has a default of 0 (same for EZDenoise). So no additional noise removal takes place beside what is part of the deinterlacing approach anyway.

Your third one:

QTGMC("Preset",TR0=1,TR1=1,TR2=0)
SRestore(FRate=23.976,Speed=-1,Thresh=12,Cache=50)


Is a standard QTGMC/SRestore, nothing special about it.

The TRx settings are the same for any Faster to Ultra Fast preset. You can totally omit them and just go with the preset of "Faster" or higher and keep it simple. In case you care for the preset settings check like 152 and onward in QTGMC.avsi.

For the SRestore(), you used with Speed=-1, the most precise detection. This is for very detailed HD material according to the readme.

On the Simpsons it would detect some frames as different frames just because of the noise instead of seeing that they are the same. It is a speed-vs-detail setting, the lower the number the slower it gets as it is looking for more details.

The default of 9 should work fine here. If you want some tuning then its probably better to increase the value. But anyway, as so many frames are repeated in cell drawn animation it doesn't really matter which one is removed. Especially not as QTGMC double the rate in addition.

Same for Thresh. You set it to the lowest possible value, meaning your source jas very little temporal artifacts. Well the Simpsons have a lot, just the DotCrawl alone. Again you went into the opposite direction you probably intended to.

And finally Cache=50. This setting is first invalid as the setting according to the readme only goes from -1 to 10. Second, you only specify a cache if you are going non-linear, usually in a multipass approach. But your script and the encode are purely linear one pass approach. Frame by frame gets encoded. So enabling it makes no real sense beside memory hogging and slower processing.

Your QTGMC/SRestre lines are the same as using this:


QTGMC(preset="Faster") # or Very Fast, Super Fast, Ultra Fast or Draft.
SRestore(frate=23.976)


Beside choosing a faster preset then I did initially, this will give you exact the QTGMC artifacts you pointed me out to and which I was able to see and changed my mind to figure this out with TFM. Why the change of mind now? Yes the result is still perfectly OK for the Simpsons. QTGMC/SRestore is a valid approach.

To be honest I didn't change my mind, I simply became confused as I ran around full circle, back to my post on the QTGMC thread a couple months ago. I'm learning on the fly and am sometimes overwhelmed by the sheer amount of information involved. Anyway, thanks for the above explanation (and reminder), I appreciate it, and really really really thanks for your effort to develop a deinterlacing-field matching-decimating script that's as good as possible for something automatic :thanks: . With that I'm now able to get on with fine-tuning the rest of my script.


My approach at the moment (see last version of my script) is the following:


TFM matching as before dealing with the bad chroma, plus a FieldMatch function that looks for a near matching field in case TFM cannot because a field has already been used. If this function cannot find a match and a frame is still decided to be combed (you could tune the setting of TFM for combing detection if you want), then QTGMC just deinterlaces the frame as there is no other option and as we have a double rate I just take the first frame and be done with it.

TDecimate with no special options, as we have a 3:2 pulldown now basically - or should have. I still used mode = 1 instead of mode = 0, cause of all the double frames in animation to give TDecimate some leeway. You could increase the cycle if you want to, but should not make any differences on this source.

Dot Crawl removal. You can use any other method here. But I found that Checkmate() works nicely even the frames are now progressive. You can use what ever you want.

needi3() upscale with frame doubling which are then merged. This removes the nasty line doubled frames that occur now and then as strong aliasing. Also deal already with some of the noise. See 1st example of my last post. Again if you don't mind these few frames or deal with them in other ways, no need to use that lines.

QTGMC in progressive mode. This cleans up all the horizontal shimmering. The phasing you see in black lines usually once a second on in various episodes in a lot of scenes, because the luma is shifting. This also happens on a lot of DVDs where a shitty filter was used during production. A very prominent example would be the "Crusade" DVDs (short lived spin off of Babylon 5). Again if an episode does not has it no need to use it. Use what ever works for you here, not related to deinterlacing at all.

Then all the other post processing and noise removal stuff to my liking as the source has now somewhat decent progressive frames.

As said, the nnedi3() and the QTGMC in progressive mode are optional if an episode does not show these problems for faster processing. On clear frames they do not do any real harm beside removing some of the noise. Again you can use whatever you want to deal with these problems, has nothing to do with IVTCing here.

As said with the following my ITVCing for the Simpsons is done.

Will only adjust once I come across seasons that are more properly encoded or show completely new patterns.. We'll see if that happens.

I'll run some tests ASAP, but I've a question: in you final deinterlacing-field matching-decimating lines above the word "video" is in several places, but when I try to run this AviSynth doesn't know what "video" is. Would you be willing to post your entire final script so I can try to sort it out? Thanks. Ahhh, Happy Friday.

[LouieChuckyMerry]

Thanks to all your fine work I've now a basic script:

LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\DGIndexNV\DGDecodeNV.dll")
DGSource("SourcePath")
### Deinterlace-Match Fields-Decimate-Fix Line Doubled Fields ###
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\avisynth_plugin\TIVTC.dll")
Function FieldMatch(Clip C) {
Global PP = C.DuplicateFrame(0)
Global CC = C
Global NN = C.DeleteFrame(0)
P2 = PP.SeparateFields()
C2 = CC.SeparateFields()
N2 = NN.SeparateFields()
Global PC = Interleave(P2.SelectEven(),C2.SelectOdd()).Weave()
Global CP = Interleave(C2.SelectEven(),P2.SelectOdd()).Weave()
Global CN = Interleave(C2.SelectEven(),N2.SelectOdd()).Weave()
Global NC = Interleave(N2.SelectEven(),C2.SelectOdd()).Weave()
Global Deint = QTGMC(CC).SelectEven()
Return ScriptClip(CC, \
"!CC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CC : " + \
"!NN.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? NN : " + \
"!CN.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CN : " + \
"!NC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? NC : " + \
"!PP.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? PP : " + \
"!CP.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CP : " + \
"!PC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? PC : Deint")
}
TFM(Order=-1,Mode=5,PP=2,Clip2=FieldMatch(),Slow=2,MChroma=False,Ubsco=False,CThresh=12,Chroma=True)
TDecimate(Mode=1)
NNEDI3(Field=-2)
Merge(SelectEven(),SelectOdd())
### Stabilize ###
Stab(Mirror=15)
### Crop ###
Crop(8,0,-8,0)
### Gibbs Noise Block ###
Edge=MT_Edge("prewitt",ThY1=20,ThY2=40).RemoveGrain(17)
Mask=MT_Logic(Edge.MT_Expand().MT_Expand().MT_Expand().MT_Expand(),Edge.MT_Inflate().MT_Inpand(),"xor")
MT_Merge(DFTTest(),Mask,Luma=True)
### Overall Temporal Denoise ###
SMDegrain(TR=2,ThSAD=400,ContraSharp=True,RefineMotion=True,Plane=0,Lsb=True,Lsb_Out=True,PreFilter=2,Chroma=False)
### Resize ###
LinearResize(640,480,Lsb_In=True,Lsb_Out=True)
### Darken-Thin Lines ###
F=DitherPost(Mode=-1)
S=F.FastLineDarkenMod(Strength=20,Prot=6).aWarpSharp2(Blur=4,Type=1,Depth=3,Chroma=2)
D=MT_MakeDiff(S,F).Dither_Convert_8_To_16()
Dither_Add16(Last,D,Dif=True,U=2,V=2)
### Deband ###
GradFun3(Thr=0.55,SMode=2,Lsb=True,Lsb_In=True,StaticNoise=True)
### Preview Source OR Send 16-bit Output To x264 10-bit ###
# DitherPost()
Dither_Out()

that looks pretty good to my eyes on my 14" 1600x900 laptop screen, but I'd love to improve the line (edge?) quality with regards to aliasing (shimmering?). After testing this script with the addition of Checkmate and-or EdgeCleaner and-or MAA2 I didn't see much difference; however, removing the NNEDI3 line makes the aliasing much worse so I think I'll use it for every episode. Any ideas for improvement? I'm a bit worried that this script's results won't look quite as good on a (hopeful) future large screen TV. If you've some spare time, I'd really value your opinion on the quality of the above script upscaled and viewed on a big TV. This S5.E21-TestClip (http://www.mediafire.com/download/bex5825c2wtwd1r/S5.E21-TestClip.zip) is a good source example, I think, for the aliasing after running the above script (the dining room table's front edge and the birthday cake, especially; and your Death Script does wonders for the field blending). Man, I can't wait to attack the first four seasons of Futurama after this...

[Scythe42]

I'll run some tests ASAP, but I've a question: in you final deinterlacing-field matching-decimating lines above the word "video" is in several places, but when I try to run this AviSynth doesn't know what "video" is. Would you be willing to post your entire final script so I can try to sort it out?That's just the variable holding my clip. Forgot to remove it when posting. It's there because all that stuff is in a function that gets a clip passed in a variable called video.

I'm a bit worried that this script's results won't look quite as good on a (hopeful) future large screen TV. If you've some spare time, I'd really value your opinion on the quality of the above script upscaled and viewed on a big TV.

Ran your script over the sample. It's fine, major difference is that I smoothed out more noise compared to you at the cost of some detail and it runs a bit longer. But IMHO you cannot do much more with this source. And as said before regarding detail, your Mosquito Noise approach is probably better then mine.

For the horizontal shimmering I added the QTGMC in progressive mode for right after the nneedi3() as it takes care of the visible phasing (you can try to remove the needi3 if you want). You might want to give it a shot, though it as its side effects. But these are not as annoying as the phasing itself. You need to try how the Stab() fits in there then (remove? move around?).

QTGMC(InputType=2)


Only difference left it basically that I smoothed the video out more and you thin the lines a bit, which I decided to leave alone.

And in both approaches some annoying blocking is left now and then (e.g. end of your sample clip on March's hair at 20 seconds), you can only counter this defect in the DVD by way more aggressive smoothing and debanding I guess.

It's annoying as it changes from frame to frame of the same painted cell. But well it doesn't happen that often and only in a few scenes. March's hair seems to have it quite often. Whatever filtering they used it had an issue with the blue...

[LouieChuckyMerry]

That's just the variable holding my clip. Forgot to remove it when posting. It's there because all that stuff is in a function that gets a clip passed in a variable called video.

After rerereading it I figured it was something to that effect and ignored it. Thanks for the confirmation.


Will try your script later...

That would be awesome. Just so you know, the S5.E21-TestClip I uploaded is from the source, not processed in any way, chosen specifically for it's miserable quality (love that field blending ;) ). I'd be very happy for your opinion of that clip processed with my script then viewed on a large TV. As it is now it's set for 10-bit processing, but drop it to 8-bit if you prefer. Thanks for that. How's your SuperBatch coming along?

[LouieChuckyMerry]

Ran your script over the sample. It's fine, major difference is that I smoothed out more noise compared to you at the cost of some detail and it runs a bit longer. But IMHO you cannot do much more with this source. And as said before regarding detail, your Mosquito Noise approach is probably better then mine.

Thanks a lot for doing that, I really really appreciate it :) . It's nice to get a valued opinion that months of encoding won't be for naught when (if) I get an actual TV (and I get about 7% credit for the script, with you and Dogway sharing the rest). Curiously, did you encode to 8-bit or 10-bit (I ask because 10-bit seems to really help with the blocking-banding).


For the horizontal shimmering I added the QTGMC in progressive mode for right after the nneedi3() as it takes care of the visible phasing (you can try to remove the needi3 if you want). You might want to give it a shot, though it as its side effects. But these are not as annoying as the phasing itself. You need to try how the Stab() fits in there then (remove? move around?).

QTGMC(InputType=2)



I've tried my script with and without the NNDEI3 bit and the difference was remarkable. Without looked as if every line was made of mirrors and there was a bright light shining on the screen as it was being shaken, every line was alive and super twitchy. With looked waaay better, with the shimmering reduced by about 80% but very little loss of detail that I could see, and is so much better I'll keep it in the script for every episode. I'll quick-like try some clips with your QTGMC addition, then some Stab tweaks if necessary. What do you refer to when you typed about QTGMC "...give it a shot, though it (h)as its side effects."?


Edit: I ran my script with the addition of QTGMC, trying QTGMC(InputType=1) and QTGMC(InputType=2). They both look very good, with 1 being slightly faster than 2 (and less detail-crushing, according to the docs), but they cost ~25% of my encoding speed. I need to think about this...


Only difference left it basically that I smoothed the video out more and you thin the lines a bit, which I decided to leave alone.

And in both approaches some annoying blocking is left now and then (e.g. end of your sample clip on March's hair at 20 seconds), you can only counter this defect in the DVD by way more aggressive smoothing and debanding I guess.

It's annoying as it changes from frame to frame of the same painted cell. But well it doesn't happen that often and only in a few scenes. March's hair seems to have it quite often. Whatever filtering they used it had an issue with the blue...

I have to agree that Marge's hair is a force unto itself, ha ha. Do I remember you mentioning somewhere that you've already encoded the classic first four seasons of Futurama? That's next for me, so any tricks you learned would be happily accepted ;) .

[Scythe42]

Curiously, did you encode to 8-bit or 10-bit (I ask because 10-bit seems to really help with the blocking-banding).I did an 8-bit encode as the 10bit version did not work for whatever reasons with my setup (lot of random dithered colors in the lower 3rd especially). Did not investigate it, but sure it's my setup that doesn't support it. 10bit during Noise Remove should be in general better on this source.

But as said, didn't work on my end at all. Probably some DLL or script version conflict in my 2.6 MT setup - I dumped so many filters in there over the years...

I've tried my script with and without the NNDEI3 bit and the difference was remarkable. Without looked as if every line was made of mirrors and there was a bright light shining on the screen as it was being shaken, every line was alive and super twitchy.Yeah, doubling and merging seems to do a good job at interpolating what is missing here, without the loss of too much detail. Glad that this idea is working for you. Tried it out of desperation.

What do you refer to when you typed about QTGMC "...give it a shot, though it (h)as its side effects."?Well, when QTGCM is removing the horizontal shimmering you might loose some subtle detail. But with a hand drawn cell animation it should not really be that noticeable. Especially not with your better Noise Removal.

I ran my script with the addition of QTGMC, trying QTGMC(InputType=1) and QTGMC(InputType=2).I settled for InputType 2 as - according the readme - tries so preserve a bit more detail. But InputType 1 should be fine as well. Doubt there will be any real difference between the InputTypes on this source.


I have to agree that Marge's hair is a force unto itself, ha ha. Do I remember you mentioning somewhere that you've already encoded the classic first four seasons of Futurama? That's next for me, so any tricks you learned would be happily accepted ;) .Futurama isn't as bad as the Simpsons as it was digitally produced right from the start and is not an analog tape transfer that was badly encoded on top of it. Not so much noise and you can clean them up nicely with just a tad of smoothing and line darkening.

Regarding IVTC: the NSTC DVDs are manageable, the PAL DVDs are beyond repair and require QTGMC/SRestore to minimize the blending. You cannot fully get rid of it.

For PAL instead of speeding 23,976fps up to 25fps they added an extra frame every 24 frames and speed these 24.976fps up to 25fps. So every 25th frame is mix between frame 24 and 25. On top of this this pattern becomes irregular with scene changes, so you cannot simply decimate every 25th frame.

I have the PAL set, which I bought cheap and shortly afterwards I knew why it was so cheap. Wish I had an NTSC set to do a proper version of the first four volumes.

If you go for a TFM approach for your NTSC set then just do it normally, if you see blended frames play with "chroma=true", "mchroma=false" and "ubsco=false" on TFM and see where it goes (usually in that order). Slowly work your way forward and try out what works up to the same approach as you now using for the Simpsons.

[LouieChuckyMerry]

I settled for InputType 2 as - according the readme - tries so preserve a bit more detail. But InputType 1 should be fine as well. Doubt there will be any real difference between the InputTypes on this source.

I was thinking InputType=1 because the .html file I have for QTGMC states:

Generally mode 1 will retain more detail, but repair less artefacts[sic] than modes 2,3. You may consider setting TR2 to a higher value (e.g. 2 or 3) when repairing progressive material.

Of course, my overall understanding is lacking so maybe I'm misinterpreting something.


Futurama isn't as bad as the Simpsons as it was digitally produced right from the start and is not an analog tape transfer that was badly encoded on top of it. Not so much noise and you can clean them up nicely with just a tad of smoothing and line darkening.

Regarding IVTC: the NSTC DVDs are manageable, the PAL DVDs are beyond repair and require QTGMC/SRestore to minimize the blending. You cannot fully get rid of it.

For PAL instead of speeding 23,976fps up to 25fps they added an extra frame every 24 frames and speed these 24.976fps up to 25fps. So every 25th frame is mix between frame 24 and 25. On top of this this pattern becomes irregular with scene changes, so you cannot simply decimate every 25th frame.

I have the PAL set, which I bought cheap and shortly afterwards I knew why it was so cheap. Wish I had an NTSC set to do a proper version of the first four volumes.

If you go for a TFM approach for your NTSC set then just do it normally, if you see blended frames play with "chroma=true", "mchroma=false" and "ubsco=false" on TFM and see where it goes (usually in that order). Slowly work your way forward and try out what works up to the same approach as you now using for the Simpsons.

Thanks for the information. I've the NTSC DVD's so a bit of good luck there as the PAL DVD's seem like a nightmare (perhaps you could find the NTSC DVD's online for a good price, used even, if the source was trustworthy). Plus, I'll have the vast knowledge I've gained from The Seemingly Neverending Re-Re-Started Simpsons Project©® ;) . Seriously, it's probably beneficial that I started with the lower quality source because with the kind help of people such as you I really have learned quite a lot :) .

Ahhh, that reminds me: as you've been encoding have you noticed any changes in the source drastic enough to call for a change in your settings? Or are you actually previewing each episode and tweaking its settings accordingly?

[Scythe42]

I was thinking InputType=1 because the .html file I have for QTGMC states:Probably I was mixing it up then, my bad...

Ahhh, that reminds me: as you've been encoding have you noticed any changes in the source drastic enough to call for a change in your settings? Or are you actually previewing each episode and tweaking its settings accordingly?I did some spot checking what problems are there.

It gets a bit better around season 7. But all seasons up to 9 are shitty analog tape transfers with a ton of Dot Crawl and all issues now and then. 10 seems to be a good analog tape transfer, and 11-12 are digital transfers (they basically look like Futurama). Fox was going digital at the time, so it really shows.

Season 1-8 are showing all the issues. They seem to have changed the way they transferred the tapes during season 8, less episodes showing strong horizontal shimmering, gets ever better with season 9 where it's mostly the Dot Crawl plus Mosquito Noise. After that it gets a notch better, but with season 12 they went a step back somehow or I just picked a bad episode for checking.

I won't change my script and let 'em run over everything. Did some tests and it didn't do any harm with later seasons. Too much work to actually check which issues an episode has. Some problems surface again in 11 and 12 some scenes.

But I am only in season 4 with my encoding at the moment as I was working on encoding other stuff...

Will spot check the result after each season and then decide if I need to make changes to the IVTC approach and the noise removal based on watching 2-3 episodes in full. But up and including to season 8 it should not be necessary. We'll see.

Yes, working on such bad source helps a lot of dealing with other animation. Still have to encode a ton of animes as well...

[LouieChuckyMerry]

I did some spot checking what problems are there.

It gets a bit better around season 7. But all seasons up to 9 are shitty analog tape transfers with a ton of Dot Crawl and all issues now and then. 10 seems to be a good analog tape transfer, and 11-12 are digital transfers (they basically look like Futurama). Fox was going digital at the time, so it really shows.

Season 1-8 are showing all the issues. They seem to have changed the way they transferred the tapes during season 8, less episodes showing strong horizontal shimmering, gets ever better with season 9 where it's mostly the Dot Crawl plus Mosquito Noise. After that it gets a notch better, but with season 12 they went a step back somehow or I just picked a bad episode for checking.

I won't change my script and let 'em run over everything. Did some tests and it didn't do any harm with later seasons. Too much work to actually check which issues an episode has. Some problems surface again in 11 and 12 some scenes.

But I am only in season 4 with my encoding at the moment as I was working on encoding other stuff...

Will spot check the result after each season and then decide if I need to make changes to the IVTC approach and the noise removal based on watching 2-3 episodes in full. But up and including to season 8 it should not be necessary. We'll see.

Yes, working on such bad source helps a lot of dealing with other animation. Still have to encode a ton of animes as well...

Thanks for the information, I'll bookmark this thread so I don't forget it. Last night, as a test, I encoded one of the worst episodes source-wise from Season 1 that I'd encountered my first go, S1.E9-LifeOnTheFastLane. I used my script from Post #41 with the addition of your QTGMC(InputType=1) line after the NNDEI3 lines. Wow, the results are very very impressive, at least to my eyes. The only reasonable complaint I have, given the shite source, is the juddering on pans. But at least it's uniform juddering, ha ha. If I remember it's the result of the deinterlacing-field matching-decimating attack which, to be fair, is utterly necessary. So I'll live with it. Unfortunately, that single episode took 9 hours to encode on my dual-core machine (my quad-core gets the 1080p sources). Ouch. But damn, it looks amazing. That's single-thread (I can't multi-thread QTGMC), so I figure I can run 2 episodes simultaneously every night and be finished in 4 months ;) . Before I rerestart, however, I want to make sure all my ducks are in a row, especially the order of the lines in my script (I'm waiting on confirmation).

Do I remember correctly that you tried pretty much every possibility for the shimmering and settled on QTGMC because it was obviously the best (if a bit slow)?

[Scythe42]

The only reasonable complaint I have, given the shite source, is the juddering on pans. But at least it's uniform juddering, ha ha. If I remember it's the result of the deinterlacing-field matching-decimating attack which, to be fair, is utterly necessary. So I'll live with it. Agree, some pans and zooms are very juddery, but not much you can do about it given the quality of this source. But it is uniform and not the NTSC back-and-forth judder, so I can live with it. And way better then blended frames in my opinion.


Unfortunately, that single episode took 9 hours to encode on my dual-core machine (my quad-core gets the 1080p sources). Ouch. But damn, it looks amazing. That's single-thread (I can't multi-thread QTGMC), so I figure I can run 2 episodes simultaneously every night and be finished in 4 months ;) . Before I rerestart, however, I want to make sure all my ducks are in a row, especially the order of the lines in my script (I'm waiting on confirmation). Yeah, on a dual core MTing QTGMC will not really work out. Just try to find very bad episodes and see how they turn out, then try some better episodes and see if the same script still works nice or destroys stuff. Better to wait and experiment before running the huge batch, so you don't have to do it again the near future.


Do I remember correctly that you tried pretty much every possibility for the shimmering and settled on QTGMC because it was obviously the best (if a bit slow)?I tried every filter I came across to reduce this phasing/horizontal shimmering. In the end QTGMC in progressive mode in combination with a needi3() before it to attack some aliasing worked the best. In the end that is what QTGMC is: a horizontal shimmering reduction filter. Bad thing is it is very slow. But using it in progressive mode works out way better then using it for deinterlacing in this case.

If I make any change to my approach I'll let you know in this thread, which I have subscribed to...

Right now I am trying to restore "The Twilight Zone (1985)" and let the Simpsons encode over night. The TZ epsiodes on the NTSC DVDs are bad tape transfers paired with bad encoding. Really, they are among the worst live action TV shows I own on DVD.

But at least no IVTCing needed as the show seems to be shot entirely on video. At least the DVDs were not telecined. Right now reading threads how ppl. restore VHS tapes to get some idea what "exotic" filters might be a good idea for such source and see how it turns out. Guess there will be not much I can do...

[LouieChuckyMerry]

Agree, some pans and zooms are very juddery, but not much you can do about it given the quality of this source. But it is uniform and not the NTSC back-and-forth judder, so I can live with it. And way better then blended frames in my opinion.

Absolutely, the occasional bit of juddering is less distracting than the patches of blended frames. Also, I checked and even using a standard TIVTC-TDecimate script results in the same juddering but with everything else not nearly as nice.


Yeah, on a dual core MTing QTGMC will not really work out. Just try to find very bad episodes and see how they turn out, then try some better episodes and see if the same script still works nice or destroys stuff. Better to wait and experiment before running the huge batch, so you don't have to do it again the near future.

I tried every filter I came across to reduce this phasing/horizontal shimmering. In the end QTGMC in progressive mode in combination with a needi3() before it to attack some aliasing worked the best. In the end that is what QTGMC is: a horizontal shimmering reduction filter. Bad thing is it is very slow. But using it in progressive mode works out way better then using it for deinterlacing in this case.

Just to be paranoid, er, safe, I ran some tests: my basic script with neither NNEDI3 nor QTGMC, my basic script with both, and my basic script with one of each. The NNEDI3-QTGMC combo was easily the best option. I also tried the QTGMC with its default preset ("Slower"), with "Medium", and with "Fast". The difference in encoding speed was only about 5%, but the difference in visual quality was very noticeable on my laptop, so default it is. Thus, in case anyone finds this useful someday, the final script is:

LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\DGIndexNV\DGDecodeNV.dll")
DGSource("SourcePath")
### Deinterlace-Match Fields-Decimate ###
LoadPlugin("F:\[0]StandAloneApps\MeGUI-2500(core)2443(data)0.3.5(libs)[Portable]\tools\avisynth_plugin\TIVTC.dll")
Function FieldMatch(Clip C) {
Global PP = C.DuplicateFrame(0)
Global CC = C
Global NN = C.DeleteFrame(0)
P2 = PP.SeparateFields()
C2 = CC.SeparateFields()
N2 = NN.SeparateFields()
Global PC = Interleave(P2.SelectEven(),C2.SelectOdd()).Weave()
Global CP = Interleave(C2.SelectEven(),P2.SelectOdd()).Weave()
Global CN = Interleave(C2.SelectEven(),N2.SelectOdd()).Weave()
Global NC = Interleave(N2.SelectEven(),C2.SelectOdd()).Weave()
Global Deint = QTGMC(CC).SelectEven()
Return ScriptClip(CC, \
"!CC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CC : " + \
"!NN.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? NN : " + \
"!CN.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CN : " + \
"!NC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? NC : " + \
"!PP.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? PP : " + \
"!CP.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? CP : " + \
"!PC.IsCombedTIVTC(CThresh=12,Chroma=True,BlockX=16,BlockY=32) ? PC : Deint")
}
TFM(Order=-1,Mode=5,PP=2,Clip2=FieldMatch(),Slow=2,MChroma=False,Ubsco=False,CThresh=12,Chroma=True)
TDecimate(Mode=1)
### Fix Line-Doubled Fields ###
NNEDI3(Field=-2)
Merge(SelectEven(),SelectOdd())
### Reduce Shimmering ###
QTGMC(InputType=1)
### Stabilize ###
Stab(Mirror=15)
### Crop ###
Crop(8,0,-8,0)
### Gibbs Noise Block ###
Edge=MT_Edge("prewitt",ThY1=20,ThY2=40).RemoveGrain(17)
Mask=MT_Logic(Edge.MT_Expand().MT_Expand().MT_Expand().MT_Expand(),Edge.MT_Inflate().MT_Inpand(),"xor")
MT_Merge(DFTTest(),Mask,Luma=True)
### Overall Temporal Denoise ###
SMDegrain(TR=2,ThSAD=400,ContraSharp=True,RefineMotion=True,Plane=0,Lsb=True,Lsb_Out=True,PreFilter=2,Chroma=False)
### Resize ###
LinearResize(640,480,Lsb_In=True,Lsb_Out=True)
### Darken-Thin Lines ###
F=DitherPost(Mode=-1)
S=F.FastLineDarkenMod(Strength=20,Prot=6).aWarpSharp2(Blur=4,Type=1,Depth=3,Chroma=2)
D=MT_MakeDiff(S,F).Dither_Convert_8_To_16()
Dither_Add16(Last,D,Dif=True,U=2,V=2)
### Deband ###
GradFun3(thR=0.55,SMode=2,Lsb_In=True,Lsb=True,StaticNoise=True)
### Preview Source OR Send 16-bit Output To x264 10-bit ###
# DitherPost()
Dither_Out()

### !!!Minus The Quotation Marks, "--demuxer raw --input-depth 16 --input-res 640x480 --fps 23.976 --sar 1:1" Must Be Added To The x264 Cmd Line For The Output Video To Be Correct!!! ###

The addition of QTGMC means no AviSynth MT (someday I'll figure out how to multi-thread QTGMC...), which is OK because from what I've read and experienced MPEG2 in Mkv can be inconsistent.

Running the above script on my dual-core (i5 3320M, 4GB RAM) laptop takes 7-20 hours per episode, and running two episodes simultaneously maxes out the CPU while doubling production (two episodes overnight, usually). Finally, using 10-bit x264 at CRF 13.0 results in a video bit rate of ~1850 Kbps for Season 1. Starting with Season 2 the source quality improves enough that CRF 12.0 is required for an ~1850 Kbps video bit rate. I'm targeting ~1850 Kbps for the video bit rate so that the Bits/(Pixel*Frame) is ~0.250. To my eyeballs, given the source material, the output is excellent.

Edit: after 8 episodes of Season 2, the output video bit rate with CFR 12.0 varies from ~1500 Kbps to ~2050 Kbps. The variation in Season 1 (with CRF 13.0) was less, from ~1650 Kbps to ~2050 Kbps. To my eyes on my 14" 1600x900 laptop screen I see no difference, so I'll trust the CFR and ignore the fluctuations.


If I make any change to my approach I'll let you know in this thread, which I have subscribed to...

Right now I am trying to restore "The Twilight Zone (1985)" and let the Simpsons encode over night. The TZ epsiodes on the NTSC DVDs are bad tape transfers paired with bad encoding. Really, they are among the worst live action TV shows I own on DVD.

But at least no IVTCing needed as the show seems to be shot entirely on video. At least the DVDs were not telecined. Right now reading threads how ppl. restore VHS tapes to get some idea what "exotic" filters might be a good idea for such source and see how it turns out. Guess there will be not much I can do...

Thanks for all your kind help, Scythe42, you've taught me a lot (so much that I honestly hope I can remember it all ;) ). Good luck with The Twilight Zone (nee nee noo noo, nee nee noo noo), and Happy Friday!