Dotcrawl (?) from hell - help please

[snarfies]

I've captured the following from a laserdisc: https://www.otakubell.com/temp/OMG-DOTZ.avi - its a five second clip saved in Lagarith.

I don't deal with dotcrawl very much in laserdisc anime sources, its almost never an issue, but this particular source seems to have it bad, and I'm having trouble with it. The few times I've encountered it usually involve bad 80s CG, and DeDot usually does the trick nicely.

So, with regards to this clip: Am I even correct in identifying the issue as "dotcrawl," and whatever it is, how to fix?

The source also has unusually stubborn rainbowing, which I assume is a related issue. I ususally used DFMDerainbow, but it simply wasn't doing the trick and I had to break out Bifrost, which did.

[real.finder]

Analog side ringing then it's become Dotcrawl

yes it too bad, checkmate(thr=25, max=15, tthr2=5) fix it but... tthr2 will made some artifacts in moving scenes, even if tthr2=0 IIRC

don't know if MCcheckmate will be a good idea to avoid that

[Motenai Yoda]

eeeeee!
It looks like more rainbow to me than dot_crawl

[LemMotlow]

It's telecined LD foolishly capped to DV. What a shame. Horrible color and levels, and some kind of weird awful processing.

[snarfies]

Hey, I'm back, thanks for the replies.

Analog side ringing then it's become Dotcrawl

yes it too bad, checkmate(thr=25, max=15, tthr2=5) fix it but... tthr2 will made some artifacts in moving scenes, even if tthr2=0 IIRC

don't know if MCcheckmate will be a good idea to avoid that

I'll try that, thanks. But what is "MCcheckmate?" The only reference Google has to that is this post.

eeeeee!
It looks like more rainbow to me than dot_crawl

I've successfully derainbowed the thing, but the crawl remains...

It's telecined LD foolishly capped to DV. What a shame. Horrible color and levels, and some kind of weird awful processing.

Yeah. well, "foolish" is what I got. I have not processed it in any way, except to convert it to Lagarith, and I don't have this problems on any of the dozens of other disks I've capped and filtered, successfully. Got anything actually helpful to add?

[LemMotlow]

It's telecined LD foolishly capped to DV. What a shame. Horrible color and levels, and some kind of weird awful processing.

Yeah. well, "foolish" is what I got. I have not processed it in any way, except to convert it to Lagarith, and I don't have this problems on any of the dozens of other disks I've capped and filtered, successfully. Got anything actually helpful to add?
Well, for starters, you could IVTC the thing. That cleans up a lot of the line shimmer:
AssumeBFF()
TFM(pp=0).TDecimate()
That gets you progressive video at 23.976 fps and less disturbance on the telecined frames. You still have DV compression noise to contend with. I've never been able to do much with DV's noise, which is why I don't waste time with DV but maybe someone else has a solution that doesn't destroy everything. Then there's the edge ringing, which can be calmed a bit with HQdering (maybe, but not much). Lowering v-channel contrast a lot would help the color a bit.

I still don't see any dot crawl.

[Mounir]

Try this script i made (assuming you want to keep it interlaced)


AVISource("OMG-DOTZ.avi")
assumetff()
ylevels(0,0.800,255,12,255)

converttoyuy2(interlaced=true)
Bob(0.0, 1.0) # YUY2 is best for Bob

## Remove some chroma noise with CCD
ConverttoRGB32(matrix="rec601",interlaced=false)
LoadVirtualDubPlugin("C:\Program Files (x86)\virtualdubmod1.5\plugins\Camcorder_Color_Denoise_sse2.vdf", "CCD", 0)
CCD(10,1) # 0-100, default 30
converttoyv12(matrix="Rec601",interlaced=false)

## HQ Dering+ santiag on edges ONLY
main=last
main
HQDering(strength=255).HQDering(strength=255) # 2pass seems enough for source: OMG-DOTZ.avi
Santiag(strv=1,strh=0)
filtered=last
main
mt_edge(mode="min/max",chroma="copy").mt_expand # mode="prewitt" can be good sometimes
mymask=last
mt_merge(main, filtered, mymask)
####
ConvertToYV16(matrix="Rec601",interlaced=false)
U = UToY().ttempsmooth(maxr=1,lthresh=150, strength=1)
V=VToY().ttempsmooth(maxr=1,lthresh=150, strength=1)
YToUV(U,V,last)
converttoyv12(matrix="Rec601",interlaced=false)
#Reinterlacing:
separatefields().selectevery(4,0,3).weave()

## Remove some grain
fields=AssumeTFF().SeparateFields() # or AssumeBFF
super = MSuper(fields)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=2)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=2)
MDegrain1(fields, super, backward_vec2,forward_vec2,thSAD=400)
Weave()

see attached images

[LemMotlow]

I tried that script. Makes a blended, fuzzy mess of the telecined frames and does little else. Maybe you could tweak it. Isn't santiag for progressive video only?

[EDIT] Oops, methinks I over-reacted. An awful lot of poop coming down in the original sample,so I guess it ain't that easy. I gave it a try, but whoever owned this video previously really pulled a number on it. Below is one of what must be a thousand possible scripts. Would be interesting to see what others come up with. Note the color correction is for separate segments.

results: OMG_DePoop_Effort_3.mpg (4MB), w/pulldown for 29.97fps. http://www.mediafire.com/file/40n779y9rmr7uj2/OMG_DePoop_Effort_3.mpg
## -- Change the path statements to
## -- match with locations in your setup

vidpath="E:\forum\doom9\snafies\"
ppath="D:\Avisynth 2.5\plugins\"
vdpath="D:\virtualdub\plugins\"
Import(ppath+"Hysteria.avs")
Import(ppath+"Santiag_v16.avs")
Import(ppath+"RemoveDirtMC.avs")
LoadVirtualDubPlugin(vdpath+"ccd_sse2.vdf", "CCD", 0) ### <-CamcorderColorDenoise
LoadVirtualDubPlugin(vdpath+"ColorMill.vdf","ColorMill", 0)

AviSource(vidpath+"OMG-DOTZ.avi")
AssumeBFF()
HQDering().HQDering()
TFM(pp=0).TDecimate()
SMoothLevels(16,0.85,255,16,255,protect=6)
QTGMC(InputType=1,preset="slow",EZDenoise=4,denoiser="dfttest",ChromaMotion=true,\
border=true,ChromaNoise=true,DenoiseMC=true,GrainRestore=0.4)
Santiag(4,4)

## -- MDegrain2 --
source=last
super = source.MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=8, overlap=4, dct=0)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=8, overlap=4, dct=0)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=8, overlap=4, dct=0)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=8, overlap=4, dct=0)
MDegrain2(source,super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)
Hysteria()
GradFun2DBmod(thr=1.8,mask=false)
AddGrainC(1.1,1.1)

#-- color correction for seprate segments --

a=last
b=a.Trim(0,77)
c=a.Trim(78,0)

b
ColorYUV(cont_v=-75,off_v=-8)
ConvertToRGB32(interlaced=false)
CCD(40,1)
ColorMill(24442,23134,27236,26454,22086,25700,28270,25710,25700,25700,26468,51365,25700,1124,2053)
b1 = last

c
ColorYUV(cont_v=-15,off_v=-3)
SmoothTweak(hue2=-10)
ConvertToRGB32(interlaced=false)
CCD(40,1)
ColorMill(28766,25705, 25700,20077,22611,25701,27756,24684,24672,25700,24932,49734,25700,1124,6149)
c1=last

bc=b1+c1
bc=bc.ConvertToYV12(interlaced=false)
return bc

[VS_Fan]

... An awful lot of poop coming down in the original sample ...The chroma planes in the sample clip provided are completely ruined. It seems as if the clip has been incorrectly resized, and blend-deinterlaced, specially for the chroma planes.
... I have not processed it in any way, except to convert it to Lagarith ...Why the need to convert to lagarith at more than 50Mb/s?. You can save disk space by processing the original DV format with Smaller file size: most probably it was at 25 Mb/s, the standard consumer level DV bitrate (less than half the size)

Also, converting from 4:1:1 (the standard consumer level DV chroma subsampling) to 4:2:0 chroma subsampling is problematic because of different chroma placements and resizing of chroma planes. This conversion could be adding problems. I suggest starting your processing chain by converting to 4:2:2 chroma subsampling, taking care of chroma placement, so you have interlaced chroma more accurately represented.

To discard you aren’t adding artifacts in this conversion, you could review and provide a sample clip in the original DV format of your capture

[VS_Fan]

Well, for starters, you could IVTC the thing. That cleans up a lot of the line shimmer:
AssumeBFF()
TFM(pp=0).TDecimate() ...Because of the botched chroma planes, you should discard chroma information for the IVTC process:
TFM(mchroma=False, chroma=False) #, display=True)
TDecimate(chroma=False)#, display=True)

[lansing]

Where are the admins? OP's images has been posted for 2 weeks and still no approval?

[althor1138]

Here's my attempt. Use mvtools 2.6.0.5 or later. This does not address the ringing/ghosting but should remove a lot of noise and dot crawl.

AVISource("OMG-DOTZ.avi")
tfm(pp=0,mode=6)
tdecimate()
tsmc(.7,10,300,255,4,true)
eedi2()
spline64resize(720,480)

function TSMC(clip input, float "preblur", int "tradius", int "mthresh", int "lumathresh", int "blocksize", bool "MT")
{
preblur=default(preblur,.3)

tradius=default(tradius,5)
# temporal radius-number of frames analyzed before/after current frame.

mthresh=default(mthresh,70)
# motion threshold-higher numbers denoise areas with higher motion.
#Anything above this number does not get denoised.

lumathresh=default(lumathresh,255)
# luma threshold- Denoise pixels that match in surrounding frames.
#255 is the maximum and default. 0-255 are valid numbers.
#Also adjusts chroma threshold.

blocksize=default(blocksize,4)
#larger numbers = faster processing times. Must be 4,8,16, or 32.

MT=default(MT,true)
#turn multi-threading on or off. If you are using single threaded avisynth this should be true.
#If you are using multi-threaded avisynth this should be false.

prefilt=input.blur(preblur).blur(preblur).blur(preblur).blur(preblur).blur(preblur).blur(preblur).blur(preblur).blur(preblur).blur(preblur).blur(preblur)
super=MSuper(input, pel=2,mt=MT)
superfilt=MSuper(prefilt,pel=2,mt=MT)

multivectors=Manalyse(superfilt,multi=true,delta=tradius,mt=MT)
multivectors2=Mrecalculate(superfilt,multivectors,thsad=mthresh,tr=tradius,mt=MT,blksize=blocksize,overlap=2)
mc=Mcompensate(input,super,multivectors2,thsad=mthresh,mt=MT,tr=tradius,center=true)
dnmc=mc.temporalsoften(tradius,lumathresh,lumathresh,15,2)
decimate=selectevery(dnmc,tradius * 2 + 1,tradius)
return(decimate)
}

[Logan9778]

Thanks for the scripts, guys. Very nice! Something good to study.