Dot crawl filters are all failing me

[Katie Boundary]

TaleSpin DVDs have some pretty nasty checkerboarding along sharp edges, and for the first time, I'm experimenting with ways to get rid of it other than by crushing the video down to VCD resolution. So far, I've tried Checkmate, DeDot, DotKill, and Tcomb. All loaded without any errors, but seemingly didn't touch the video. I think I need a bigger gun.

[Selur]

Can you share a short sample?

[Katie Boundary]

Even brutal, indiscriminate flattening along the horizontal axis...

GeneralConvolution(0, "
0 0 0 0 0
0 0 0 0 0
1 1 1 1 1
0 0 0 0 0
0 0 0 0 0 ")

...doesn't fully kill what I'm seeing:

https://i.imgur.com/LCFome7.jpeg

Can you share a short sample?

If you can remind me how.

[DTL]

"Even brutal, indiscriminate flattening along the horizontal axis..."

As I remember the phase switching in analog systems like PAL/NTSC is designed to make residual dot crawl distortions less visible by temporal averaging (in slow human visual system). So to emulate this you can try any temporal denoise filter with averaging of 2 or several sequential frames (motion compensated maybe better). The most simple test for static areas maybe some temporal smooth plugin (like vsTTempSmooth ?) with all thresholds disabled. Though it averages samples from current and +-tr frames so 3 frames minimum with tr=1. And here it looks we need 2 frames only (or pairs of frames).

Or some manual scripting like separate frame sequence to 2 (clips with 1 frame trimmed ?) and apply Average() filter with 0.5 weights or even Layer/Overlay from AVS core. So you need to do 'temporal convolution' not spatial.

But it will only work as expected if PAL/NTSC encoder was correct and provide required phase switching over sequence of frames. Some cheap encoders may miss this.

I think there exist some old plugins for this effect fixing ? It is close to emulating 3D-comb filter at composite hardware decoders. Where 3D mean temporal averaging as described above.

First you can check several frames in frame by frame stepping - do the phase switching temporal pattern present over time ? Maybe it not present (bad encoder ?) or somehow broken so standard dotcrawl fixing tools can not lock on it and apply required correction. So maybe required some manual scripting.

Some other solution is applying blind H-notch filter for PAL/NTSC subcarrier (it will damage more H-sharpness if it present at all). It is something like DFreq plugins or any other typically FFT-based with some user interface to input required frequency transfer.
H-notch filter also possible somehow with GeneralConvolution AVS core filter if yon can create required convolution coefficients and apply long enough H-convolution (maybe 7x7 or 9x9 will somehow work already).

Other poor solution is simple H-LPF with very sharp cut-off using SincResize (so many taps):

SincResize(width/2, height, taps=30)
SincResize(width*2, height, taps=30)

It is too simple example - maybe /2 can be replaced with close to /1.5 (or any required for current pixel format if not 4:4:4 used) to keep better H-resolution while fixing that PAL/NTSC residual sub-carrier at the required ratio. The FFT or convolution-based H-notch filter can keep more resolution it source was real full-resolution PAL/NTSC and not poor VHS.

[Selur]

If you can remind me how.
open source in DGIndex, set cut marks (a few seconds are enough), export video, upload video to a file hoster of your choice (i.e. GoogleDrive), share link to file here.

[Zarxrax]

For strong dot crawl, the existing avisynth filters can fall short, and the best solution is using neural networks that are designed to deal with it. https://openmodeldb.info/models/1x-Dotzilla-Compact
See the link at the very top of the page for how to use.

[Selur]

See the link at the very top of the page for how to use.
https://openmodeldb.info/docs/faq doesn't list Avisynth or Vapoursynth, but both can use at least a part of the models. (through vsgan and mlrt)

[kedautinh12]

For strong dot crawl, the existing avisynth filters can fall short, and the best solution is using neural networks that are designed to deal with it. https://openmodeldb.info/models/1x-Dotzilla-Compact
See the link at the very top of the page for how to use.

Avs-mlrt can use his model to work on avisynth

[johnmeyer]

A couple of simple ideas in this thread (about three posts down):

Remove cross hatch pattern from Kinescope (https://forum.doom9.org/showthread.php?t=173030)

I got the best results from the convolution approach, using quite a different kernel than you posted (see post #5 in the link above for the convolution kernel).

It turns out that I gave you this link before when you posted, seven years ago, about dot crawl in your South Park DVDs. I found then that many other people were irritated about the lousy South Park DVD transfers. That led me to all sorts of posts eventually culminating in a link to a poisondeathray short script which seemed to do a good job on one specific sample. Here is a link to his post which I linked to in post #12 from your South Park thread:

poisondeathray's dot crawl script (https://forum.videohelp.com/threads/342027-Hideous-dot-crawl#post2130213)

[SaurusX]

I've dealt with that TaleSpin dotcrawl. You need a combination of approaches to get rid of it all in the conventional AVISynth approach, i.e. non-AI. It's a combination of 32-bit and 64-bit AVISynth since there were some filter incompatibilities.

32-bit step:
MPEG2SOURCE("E:\...\Talespin01.d2v")
j=QTGMC(preset="slow", matchpreset="slow", matchpreset2="slow", FPSDivisor=2, sourcematch=3, tr1=2, tr2=1, NoiseTR=0, sharpness=0.1)
TFM(slow=2,pp=4,mode=4,micmatching=0,clip2=j)
TDecimate(mode = 1)

a=last
edgmask=a.hysteria(showmask=false,luma_cap=150,lowthresh=8,highthresh=11,strength=3).tedgemask(threshY=7).mt_inflate().mt_inflate().mt_inflate()
g=ddcomb()
overlay(a,g,mask=edgmask)

From that output I fed it into 64-bit AVISynth with this:
lwlibavvideosource("E:\...\TaleSpin01_DD_mask.mkv")
a=last
edgmask=a.hysteria(showmask=false,luma_cap=150,lowthresh=8,highthresh=11,strength=3).TEMmod(threshY=12).mt_inflate()
g=a.TComb(mode=0, fthreshL=5, othreshL=7).QTGMC(preset="slow",inputtype=1,tr0=0,tr1=1,tr2=1,smode=0,sharpness=.1) #,ezdenoise=1)
overlay(a,g,mask=edgmask)

a=last
edgmask=a.hysteria(showmask=false,luma_cap=150,lowthresh=8,highthresh=11,strength=3).TEMmod(threshY=14).mt_inflate()
g=knlmeanscl(D=3, A=1, h=2.5, device_type="GPU")
overlay(a,g,mask=edgmask)

maa2(show=0,mask=-21,aa=40)

And that should zap the dotcrawl. There are other problems with the video that I am not addressing here, but I'm sure you're aware.

[Katie Boundary]

set cut marks

Yeah, that's the part that I forgot how to do.

[Katie Boundary]

SincResize(width/2, height, taps=30)
SincResize(width*2, height, taps=30)

Yeah, a variation of that with Blur().sharpen() between the resize steps was my Plan C.

A couple of simple ideas in this thread (about three posts down):

Remove cross hatch pattern from Kinescope (https://forum.doom9.org/showthread.php?t=173030)

I got the best results from the convolution approach, using quite a different kernel than you posted (see post #5 in the link above for the convolution kernel).

Oh, yeah, I tried that one already. Super ineffective.

It turns out that I gave you this link before when you posted, seven years ago, about dot crawl in your South Park DVDs. I found then that many other people were irritated about the lousy South Park DVD transfers. That led me to all sorts of posts eventually culminating in a link to a poisondeathray short script which seemed to do a good job on one specific sample. Here is a link to his post which I linked to in post #12 from your South Park thread:

poisondeathray's dot crawl script (https://forum.videohelp.com/threads/342027-Hideous-dot-crawl#post2130213)

It looks like the relevant part of that script is just TComb().Checkmate()

Worth a shot but I'm not hopeful.

a combination of 32-bit and 64-bit AVISynth

I'm definitely passing on that.

I think that for now, I'll just do downscale().blur().sharpen().upscale()

[poisondeathray]

I've dealt with that TaleSpin dotcrawl. You need a combination of approaches to get rid of it all in the conventional AVISynth approach, i.e. non-AI. It's a combination of 32-bit and 64-bit AVISynth since there were some filter incompatibilities.

Even now ? What was the incompatibility? DDComb?

There is an updated version
https://github.com/realfinder/AVS-Stuff/blob/master/avs%202.5%20and%20up/DDComb.avsi

And prerequisites checkmate, frfun7 have x64 versions now

[SaurusX]

i'm definitely passing on that.

I think that for now, i'll just do downscale().blur().sharpen().upscale()
lol. Ok. It's there if you want it done nicely and without destroying the image. It's a hard problem with a complex solution.

Even now ? What was the incompatibility? DDComb?

There is an updated version
https://github.com/realfinder/AVS-Stuff/blob/master/avs%202.5%20and%20up/DDComb.avsi

And prerequisites checkmate, frfun7 have x64 versions now

Those scripts are from over two years ago at this point. It probably was frfun7, so thanks for letting me know.

[Selur]

Yeah, that's the part that I forgot how to do.
Then how about sharing a snipped of the source? (or did I overlook a link somewhere?)

Cu Selur

[Reel.Deel]

Then how about sharing a snipped of the source?

Second that request.

[Mounir]

https://imgsli.com/MjM4MjI5

that's all you need to remove it:

avisource()
Spline36Resize(width/2, height)
Sharpen(0.2, 0.0)
TurnLeft().nnedi3(dh=true).TurnRight()

[DTL]

If it is SD PAL capture and 720x576 - the sharp cut-off LPF for about 4.43 MHz PAL colour subcarrier may be like
SincResize(500, 576, taps=15) (or slightly better and slower LanczosResize(500,576, taps=30))
SincLin2Resize(720, 576) (or old LanczosResize(720,576, taps=30) )

Where 500 may be adjusted for balance between details loss and dots removing in about 450..550 range.

[kedautinh12]

https://i.imgur.com/BaXRZiJ.png

My script:
LoadPlugin("C:\Encode Tools\MeGUI-2944-64\tools\ffms\ffms2.dll")
FFVideoSource("C:\Users\ADMIN\Downloads\LCFome7.jpeg")
z_ConvertFormat(pixel_type="rgbps", colorspace_op="709:709:709:l=>rgb:709:709:f")
mlrt_ort(network_path="C:/Program Files (x86)/AviSynth+/plugins64+/models/RealESRGAN/RealESRGAN_x4plus_anime_6B.onnx", builtin=false, fp16=true, provider="cuda")
propSet("_FieldBased",0)
deep_resize(512,384,grain=0)
z_ConvertFormat(pixel_type="yuv420p8", colorspace_op="rgb:709:709:f=>709:709:709:l")

[Katie Boundary]

Then how about sharing a snipped of the source? (or did I overlook a link somewhere?)

Cu Selur

You don't seem to be understanding the necessary order of events here. I can't share a snippet of anything until someone reminds me how to set the beginning and end markers for the snip.

If it is SD PAL capture

It's NTSC.

https://imgsli.com/MjM4MjI5

that's all you need to remove it:

I wasn't aware of the turnleft/turnright filters. I'll give that approach a shot, though with a different resampler. The Spline filters are known to be cursed :)

My script:

Did you... try your script on an image that was magnified 4x? I don't think it'll deliver the same results on a non-magnified image.

[kedautinh12]

This is upscale models. And almost models auto x2 or x4 of the base image, video after apply. You can turn it to back original with resizer plugins

[Reel.Deel]

You don't seem to be understanding the necessary order of events here. I can't share a snippet of anything until someone reminds me how to set the beginning and end markers for the snip.

With DGindex, set start/stop markers [ ] on the bottom right and then save project and demux video. Easy peasy.

https://i.ibb.co/zQBm0Jp/dgindex.jpg

[Katie Boundary]

avisource()
Spline36Resize(width/2, height)
Sharpen(0.2, 0.0)
TurnLeft().nnedi3(dh=true).TurnRight()

I'm very happy to report that a variation of this method has worked BEAUTIFULLY! My variation is as follows:

- I used an external resizer for downscaling because NNEDI3 doesn't work in RGB colorspace and AVIsynth's internal resizers have a bug when used in YUV colorspace
- The final result was shifted a half-pixel to the right so I added resize(640,480,0.5,0,640,480) at the end
- I got rid of the unnecessary sharpen()

Thanks!

[ ]

That was all you had to say, though it doesn't matter anymore.

[Reel.Deel]

That was all you had to say, though it doesn't matter anymore.

Lol, I'm sure there would of been follow up questions if I just said "[]" and nothing else.

A sample would be appreciated though. I'm sure better can be done without nuking the image...

[SaurusX]

If anyone wants to try, here's a clip from the first episode. I also included my attempt using essentially the same script I provided before with some additional sharpening at the end.

https://mega.nz/folder/tP9jUSwT#7RlwxCyfYZtYxBKCWR-KiQ

The dot crawl is absolutely brutal. Especially surrounding the plane near the end of the clip.

[Reel.Deel]

If anyone wants to try, here's a clip from the first episode. I also included my attempt using essentially the same script I provided before with some additional sharpening at the end.

https://mega.nz/folder/tP9jUSwT#7RlwxCyfYZtYxBKCWR-KiQ

The dot crawl is absolutely brutal. Especially surrounding the plane near the end of the clip.

Thanks for the sample. Your results look pretty good. Out of curiosity, I downloaded the first episode from Disney+ and it does not have dotcrawl but the overall image is softer and colors are less vivid. https://mega.nz/file/A71D3ZxT#CFUOM_4sU7P_zuFw_s0YRjoMmbSmGbkZsxDr5ErnaoY

[kedautinh12]

If anyone wants to try, here's a clip from the first episode. I also included my attempt using essentially the same script I provided before with some additional sharpening at the end.

https://mega.nz/folder/tP9jUSwT#7RlwxCyfYZtYxBKCWR-KiQ

The dot crawl is absolutely brutal. Especially surrounding the plane near the end of the clip.

can you share your script?

[poisondeathray]

can you share your script?

post #10 ?
https://forum.doom9.org/showpost.php?p=1996999&postcount=10

[kedautinh12]

Oh, sr i'm blind :D

[DTL]

"It's NTSC."

NTSC colour subcarrier in North America was 3.58 MHz instead of 4.43 for PAL so it possibly require lower size in samples to cut-off it with LPF. So may be try about SincResize(400, height, taps=30). Though I not sure if 720 samples line of NTSC by rec.601 can be recalculated in such way. So user can simply try to lower width after SincResize with big enough taps number using some steps (like 20..50) to lower size until it will cut-off the colour subcarrier regular dots.

[SaurusX]

Thanks for the sample. Your results look pretty good. Out of curiosity, I downloaded the first episode from Disney+ and it does not have dotcrawl but the overall image is softer and colors are less vivid. https://mega.nz/file/A71D3ZxT#CFUOM_4sU7P_zuFw_s0YRjoMmbSmGbkZsxDr5ErnaoY

Disney made an edit where Louie has the sword at his throat and gets backed up. Is nothing sacred anymore?

[Reel.Deel]

Disney made an edit where Louie has the sword at his throat and gets backed up. Is nothing sacred anymore?

Disney is known for censoring things... shame

[Julek]

DDComb (https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/DDComb.avsi) is a good filter for that, no?

[kedautinh12]

I've dealt with that TaleSpin dotcrawl. You need a combination of approaches to get rid of it all in the conventional AVISynth approach, i.e. non-AI. It's a combination of 32-bit and 64-bit AVISynth since there were some filter incompatibilities.

32-bit step:
MPEG2SOURCE("E:\...\Talespin01.d2v")
j=QTGMC(preset="slow", matchpreset="slow", matchpreset2="slow", FPSDivisor=2, sourcematch=3, tr1=2, tr2=1, NoiseTR=0, sharpness=0.1)
TFM(slow=2,pp=4,mode=4,micmatching=0,clip2=j)
TDecimate(mode = 1)

a=last
edgmask=a.hysteria(showmask=false,luma_cap=150,lowthresh=8,highthresh=11,strength=3).tedgemask(threshY=7).mt_inflate().mt_inflate().mt_inflate()
g=ddcomb()
overlay(a,g,mask=edgmask)

From that output I fed it into 64-bit AVISynth with this:
lwlibavvideosource("E:\...\TaleSpin01_DD_mask.mkv")
a=last
edgmask=a.hysteria(showmask=false,luma_cap=150,lowthresh=8,highthresh=11,strength=3).TEMmod(threshY=12).mt_inflate()
g=a.TComb(mode=0, fthreshL=5, othreshL=7).QTGMC(preset="slow",inputtype=1,tr0=0,tr1=1,tr2=1,smode=0,sharpness=.1) #,ezdenoise=1)
overlay(a,g,mask=edgmask)

a=last
edgmask=a.hysteria(showmask=false,luma_cap=150,lowthresh=8,highthresh=11,strength=3).TEMmod(threshY=14).mt_inflate()
g=knlmeanscl(D=3, A=1, h=2.5, device_type="GPU")
overlay(a,g,mask=edgmask)

maa2(show=0,mask=-21,aa=40)

And that should zap the dotcrawl. There are other problems with the video that I am not addressing here, but I'm sure you're aware.
you can try change tedgemask to vsTEdgeMask (http://avisynth.nl/index.php/VsTEdgeMask)to make compatible with x64 whole script

[Katie Boundary]

Lol, I'm sure there would of been follow up questions if I just said "[]" and nothing else.

Nope. I did the .m2v thing many years ago. I just forgot which keys set the beginning and end.

Disney made an edit where Louie has the sword at his throat and gets backed up. Is nothing sacred anymore?

Disney is known for censoring things... shame

I'm still looking for the original cut of that one extremely racist episode of Rescue Rangers...

DDComb (https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/DDComb.avsi) is a good filter for that, no?

(looks at script)

(sees Bicubic being used at default Mitchell-Netravali values)

What kind of madman would do such a thing?

[Katie Boundary]

I just re-ripped the first few episodes of Talespin so I could make a proper apples-to-apples comparison. Unfortunately, the turnleft().nnedi3(dh=true).turnright() approach does NOT completely kill dotcrawl in the first episode. I modified the approach as follows:

resize(320,480).blur(1,0).sharpen(1,0).turnleft().nnedi3(dh=true).turnright().resize(640,480,0.5,0,640,480)

Unfortunately, this did not produce better results than my original resize(360,480).blur(1,0).sharpen(1,0).resize(640,480) method.

Mounir has, however, found a way to partially reduce dotcrawl without noticeably blurring the image or mixing different versions of AVIsynth, so I'll take that small victory and go home.

EDIT: it appears that the RGB issue was because converttorgb() converts to RGB32, and NNEDI3 works in RGB24 but not RGB32. Therefore, we can perform the subpixel shift during the downscale:

mpeg2source("01.d2v").yadifmod2().converttorgb24() #or whatever bobber you want

catromresize(320,478,0.55,0,704,478) #16 pixels of overscan were already cropped in the d2v; another 2 need to be cropped off the bottom

turnleft().nnedi3(dh=true).turnright()

bicubicresize(640,480,0,1) #because of the 2 pixels from earlier

[Mounir]

Mounir has, however, found a way to partially reduce dotcrawl without noticeably blurring the image or mixing different versions of AVIsynth, so I'll take that small victory and go home.
More than likely Jagabo did found by himself that sweet piece of code on videohelp forum, but thanks for the credit.

[anton_foy]

Replace the resizers with JPSDR's mt resizers for speed and NNEDI3CL?

[DTL]

"resize(320,480)"

To keep more residual details under colour subcarrier and to cut-off colour subcarrier better you need to use some sinc-based resizer with large enough taps-number. Like SincResize/LanczosResize/BlackmanResize and some others. The shape of sinc-weighting filter is not critical with big enough taps number for this task. And taps-number as large as possibe (at the desired performance - like 10..20..30 and more).

And width-downsize param is adjusted from 640 to 320 to finally touch and remove the dot-crawl H-frequency. With steps like 20..30. With too big width-downsize param filter will not touch and remove dot-crawl frequency and with too low will too more blur result.

[screamingtrees]

So does this half the horizontal detail in a tradeoff to remove the dotcrawl?

[DTL]

If you have full resolution PAL/NTSC natural source - yes. It will about half resolution. But if it is VHS source - it have about nothing above half of bandwidth. If it is simple anime/cartoon drawing - you can good upsize it with new NN plugins and about restore resolution.

[flossy_cake]

I'm not sure the dot crawl in for example S01E01 is legit dotcrawl as it doesnt always seem to align perfectly with 1px rows and it's absent from some scenes and present in others, and if you look at the end credits where they've paused on that sunset image the dotcrawl down the right side is paused too.

If there is only 1 generation of legit orthodox dotcrawl I would get rid of it on stationary parts of the image using Dedot and then for moving parts of the image a pass of QTGMC/nnedi3, the latter seems to work better at double rate otherwise decimation stutters the dotcrawl animation resulting in worse temporal blending by QTGMC/nnedi3, can try increasing the TR2 value to 3 as well.

Just started watching this series and personally I just prefer the dotcrawl over other artefacts, but image quality is very soft and bad in the first 4 pilot episodes, seems to get better by e05.