Script for removal horizontal stripes (alpha release)

[yup]

mammo1789!
I try Your short clip with thicknessline=3,2,1 and see at some frames removing lines (not all).
Script can remove no more than 2 sequential lines at time line. If have 4 sequential lines at time line need 2 or 3 pass.
Also try new version

Global thicknessline=2# thickness horizontal spike in pixels at field scale (not frame)
Global radvertmed=2*thicknessline-1
Global distser=25# distance between short lines
Global halflength=3# detect line longer 2*halflength+1 pixels
#SetMemoryMax(768)
#SetMTMode(3,4)
source=AVISource("selnew.avi")
#source file
#SetMTMode(2,4)
source=source.AssumeTFF().ConvertToYV12(interlaced=true)
bobnn=source.nnedi3(field=-2,threads=1)
# bobbing for better motion estimation
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,radvertmed),U=3,V=3)
# vertical median filter
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
#dft filter remove segregation after median vertical filter
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnn, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnn, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnn, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnn, super, fw2, thSAD=16000, thSCD1=16000)
# motion estimation on filtered and compensating on source
bwabs1=mt_lutxy(bobnn,bc1,"x y - abs")
bwabs2=mt_lutxy(bobnn,bc2,"x y - abs")
fwabs1=mt_lutxy(bobnn,fc1,"x y - abs")
fwabs2=mt_lutxy(bobnn,fc2,"x y - abs")
# calculation absolute difference between source and compensated 
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
# calculation spike detection index for center, forwad and backward
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
# calculating SAD for filtered for estimation better choice for motion compensation
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
# maximum SAD from forward and backward measure for quality center compensation
bwsad=mt_logic(mb1sad,mb2sad,"max")
#for backward compensation
fwsad=mt_logic(mf1sad,mf2sad,"max")
#for forward compensation
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
# inverting mask for using mt_merge
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
# motion compensated median filtering for center, backward and forward compensation
#
# sort SAD for finding better motion compensated median filtering, this approach can use without SDI for pixels and 1 line thickness stripes
mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?")
# center filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?")
# backward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?")
# forward filtered
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
#best value based on 3 motion compensated median time filtered value and spatial filtered
# where motion compensation bad, work only for one sequintial frame with spike
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
# backward and forward compensated full for SDI approach and spatial filtered where motion compensation bad
#end sort SAD
#

#
# sort SDI for choose center, forward or forward compensation work with 2 sequential frames, for 3 need 2 pass filtering, for 4 3 pass
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# center SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# backward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
# forward SDI
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


# this place for repair now I am thinking

fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
StackVertical(Separatefields(source),fieldmaskedsdi,mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#Weave(fieldmaskedsdi)
# some kind comparing and weave for interlaced source, may be at double rate and for second pass not need bobbing

# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
	threshsp=10
	# threshold for pixels value absolute difference from 2 sequitial frames for spike detection
	THP=string(threshsp)
	threshavg=2
	# threshold how much times actual SDI greater avearage and decimated SDI for remove false alarm for bad motion compensation 
	THAVG=string(threshavg)
	SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
	#  SDI calculation
	SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
	# SDI averaging
	SDIAvgblk=SDIavg.PointResize(90,72)# for PAL (90,72), FOR NTSC (90,60)
	# decimation
	SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
	# median point at X position
	SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
	#median point at Cross position
	SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
	# multi median filtering for remove false alarm SDI
	SDIavg=SDIAvgblk.PointResize(720,576)# for PAL (720,576), FOR NTSC (720,480)
	# backward resize to original frame size
	SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
	# comparing with actual SDI and if greater it is real spike, value 128 can increase to 192
	# be carefull I can not find this at original paper, big value can loose spike, small false alarm
	SDIad = (radvertmed>1) ? SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,radvertmed-1)) : SDIad
	# rectangle size could one less than for bobnnmed, for thinner lines need additional pass with thiknessline=thicknessline-1 until thicknessline=1
	SDIad = SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(thicknessline,0))
	#remove line shotter than thicknessline+1
	SDIadleft=SDIad.mt_luts(SDIad,mode="max",pixels=mt_freerectangle(-distser,0,0,0))
	# fill distser pixels from spike left side 
	SDIadright=SDIad.mt_luts(SDIad,mode="max",pixels=mt_freerectangle(0,0,distser,0))
	# fill distser pixels from spike right side
	SDIadser=mt_logic(SDIadleft,SDIadright,"min")
	# if pixels exist at  SDIadleft and SDIadright fill pixels between 2 short lines
	SDIadser = (radvertmed>1) ? SDIadser.mt_luts(SDIadser,mode="median",pixels=mt_rectangle(0,radvertmed-1)) : SDIadser
	# ADDITIONAL VERTICAL MEDIAN FILTER FOR REMOVE THIN JOIN PART
	SDImask=SDIadser.mt_luts(SDIadser,mode="min",pixels=mt_rectangle(halflength,0))                            
	# remove spikes shorter than 2*halflength+1 pixels tune for source                  
	SDImask=SDImask.mt_luts(SDImask,mode="max",pixels=mt_rectangle(halflength,0))                            
	# expand after shrinking                  
	SDIad=mt_logic(SDIad,SDImask,"min")              
	# combination source and adapting SDI for better robustness  
#	SDIad = SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(thicknessline,0))
	# additional remove short spikes which can be artefacts
	return(SDIad)

}

Now I am not ready, but result better than before.
yup.

[yup]

Some examples

Two last sample show how work different thicknessline.
yup.

[GMJCZP]

I suggest you try your script something like:

mergechroma (dctfun4b (5.5)) # set to your like

or

mergechroma (ReduceFlicker) or other function within the plugin with mergechroma

[yup]

GMJCZP!
Now script find place for luma spike and repairing including chroma. I see that at place where luma spike exist also is chroma dig wider than luma spike.
I am planing also find solution for this.
yup.

[GMJCZP]

I made ​​the suggestions because you saw colored lines.
There are other alternatives to use: DeScratch, DeSpot, or the wonderful idea (the best) of ​​John Meyer, RemoveDirtMC.
Good luck, yup!

[yup]

Hi all!
Upload avsi version of script with small changes (work little better with thicker lines 3 and 4 due to use gaussianblur instead of average filter before decimation SDI mask). See first post.
yup.

[Mounir]

I've tried your new script now, i find it very slow (!) and not necessarily better than Devcr which i like. Maybe you could borrow some ideas from devcr.

[yup]

Mounir!
Yes slow!
for testing.
Script based on motion compensated median filtering and analyzing not only current frame but 2 before and 2 ahead.
It is spent CPU time.
Short test with devcr, not remove all horizontal lines especially series (not solid with series short lines), introduce artifacts.
I am sure that script can be optimized. For this we need help from guru script writers.
yup.

[greymouse]

hi Yup, was wondering if I could get some help with one of your scripts, it works as it should as it creates a mask with the lines I want to remove from each frame, but how do I apply the mask to actually remove the black lines?

Setmemorymax(512)

AviSource("F:\Video-Capture\Home Improvement.avi") #ok

AssumeBFF()
ConvertToYV12(interlaced=true)
fields=SeparateFields()
backward_vec = fields.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1,dct=1)
forward_vec = fields.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1,dct=1)
fc = fields.MVFlow(forward_vec, idx=1, thSCD1=10000)
bc = fields.MVFlow(backward_vec, idx=1, thSCD1=10000)
masksdi=SDIPixelMask(bc, fields, fc, 50, 0.8) # define your thresholds here
StackVertical(fields,masksdi)
#Weave()


# this is for Abs(p - f) or Abs(p - b)
Function absdiff(clip c1, clip c2) { return mt_lutxy(c1, c2, "x y - abs") }

# this build the expression for the SDI function
Function SDI_RPN(string expr_d1, string expr_d2, string expr_t1, string expr_t2) {
_d1 = expr_d1 + " "
_d2 = expr_d2 + " "
_t1 = expr_t1 + " "
_t2 = expr_t2 + " "
return _d1 + _t1 + "> " + _d2 + _t1 + "> | 1.0 " \
+ _d1 + _d2 + "- " + _d1 + _d2 + "+ / abs - 0.0 ? " \
+ _t2 + "> 1.0 " + _d1 + _t1 + "> " + _d2 + _t1 + "> | 1.0 " \
+ _d1 + _d2 + "- " + _d1 + _d2 + "+ / abs - 0.0 ? "
}

# this creates the mask
Function SDIPixelMask(clip prev, clip curr, clip next, float t1, float t2) {

c_d1 = absdiff(curr, next)
c_d2 = absdiff(curr, prev)

# since SDI returns 0..1 we multiply with 255

mask = mt_lutxy(c_d1, c_d2, SDI_RPN("x", "y", String(t1), String(t2)) + " 255 *")
return mask
}

[greymouse]

the script is working perfectly at creating a mask which has the black lines I want to remove in while, but how do I use that mask to remove the lines themselves? thank you

[greymouse]

I know that overlay () will use the mask but is there another way? Is that the only function which you pass a mask through?

[poisondeathray]

Originally Posted by greymouse

I know that overlay () will use the mask but is there another way? Is that the only function which you pass a mask through?

mt_merge can use a mask as well

[greymouse]

Originally Posted by poisondeathray

mt_merge can use a mask as well

thanks, still unsure as to how I can use the mask created in the above script to remove the lines

[greymouse]

my thinking at the moment is to create a clip with only the mask, also create a 'fixed' clip which has all the black lines removed, then merge the two together, i guess they'll work? using overlay() and one of its parameters to "blend" etc the two together

[greymouse]

Mt_merge should do the job thanks!

[yup]

greymouse & poisondeathray!

Now I am out my working horse (after weekend). This spring I am little rewriting script for last version avisynth and plugin.
For some source work relative good.
yup.

[greymouse]

sounds good yup! your script works nicely

[yup]

greymouse!

See link for updated version:
https://www.dropbox.com/s/3b4p4zr6cj...med3n.avs?dl=0
For script need Avisynth+ and all updated plugin. 32 bit version work like before with old variableblur plugin.

SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
#SDIavg=SDI.gaussianblur(varY=40)
#SDIavg=SDI.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false,threads=1)
# SDI averaging

If using for averaging gaussianblur (when I writing script, give me better result), only 32 bit version.
If two other mt_luts or dffttest You can use 64 bit (some speed up and stability).
Now I try Mohan plugin GBlur.
One more improvement, support YV16 colorspace (lossless for YUY2 captured source), before only YV12.
Any suggestion welcome.
yup.