Script for removal horizontal stripes (alpha release)

[fonzzie31]

Hello Yup,

Thanks for your message...
I tried your script and the black lines/stripes were not removed.
Here is a screen capture of what I got :

http://revival03.free.fr/temp/blines4.jpg

Why don't I have a 720x576 single frame in input/output ?

As You see the black lines are still here... Is there something wrong with this script ? Perhaps am I wrong too :)

I've installed all the plugins in my avisynth plugin directory.
Then I copy & paste your script in a text file names "test.avs".
I replace one line on your script (in red) :

SetMemoryMax(768)
SetMTMode(3,4)
source=AVISource("test3.avi",pixel_type="YUY2")#.Trim(750,1250)
SetMTMode(2,4)
source=source.AssumeTFF().ConvertToYV12(interlaced=true)#.ColorYUV(autogain=true, autowhite=true)
bobnn=source.nnedi3(field=-2)
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,5),U=3,V=3)# rectangle 5 for 3

pixels thikness line, 3 for 2, 1 for 1
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
bobnnf=bobnn#.dfttest(tbsize=1,ftype=1,sbsize=16,sosize=12,sigma=1000,U=false,V=false)
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
superff=MSuper(bobnnf)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnnf, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnnf, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnnf, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnnf, super, fw2, thSAD=16000, thSCD1=16000)
bwabs1=mt_lutxy(bobnnf,bc1,"x y - abs")#,U=-128,V=-128)
bwabs2=mt_lutxy(bobnnf,bc2,"x y - abs")#,U=-128,V=-128)
fwabs1=mt_lutxy(bobnnf,fc1,"x y - abs")#,U=-128,V=-128)
fwabs2=mt_lutxy(bobnnf,fc2,"x y - abs")#,U=-128,V=-128)
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
#
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
bwsad=mt_logic(mb1sad,mb2sad,"max")#,U=2,V=2)
fwsad=mt_logic(mf1sad,mf2sad,"max")#,U=2,V=2)
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
#
# sort SAD
# center filtered
mt_logic(mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y

<="),"and"),mt_lut(centersad,"x 255 <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?",U=3,V=3)
# backward filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?",U=3,V=3)
# forward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?",U=3,V=3)
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
#end sort SAD
#

#
# sort SDI
# center SDI
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y

>="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# backward SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255

0 ?")
# forward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0

?")
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
StackVertical(Separatefields(source),fieldmaskedsdi)
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source)))
#Weave(fieldmaskedsdi)


# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
threshsp=10
THP=string(threshsp)
threshavg=2
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
SDIAvgblk=SDIavg.PointResize(90,72)
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
SDIavg=SDIAvgblk.PointResize(720,576)
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4))# rectangle 4 for 3

pixels thikness line, 2 for 2, not need for 1
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(4,0))# remove line shorter 5

pixels
SDImaskleft=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 -1 0 -2 0 -3 0 -4 0 -5 0 -6 0 -7 0

-8 0 -9 0 -10 0 -11 0")# try
SDImaskright=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9

0 10 0 11 0") # join
SDIMask=mt_logic(SDImaskleft,SDImaskright,"min")

# sequental horizontal line
SDImask=SDImask.mt_luts(SDImask,mode="min",pixels=mt_rectangle(18,0))

# remove line shorter 37 pixels
SDImask=SDImask.HorlLinExp()

# fill line with noise
SDIad=mt_logic(SDIad,SDImask,"min")

# refine source SDI
SDIad=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4))

# rectangle 4 for 3 pixels thikness line, 2 for 2, not need for 1
return(SDIad)

}

# fill full horizontal line if exist one pixel per line

function HorlLinExp(clip mask1) {

mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}

YUP, do you want me to send you a source video sample, in order to make some testings ?

[yup]

fonzzie31!
Your source interlaced? Or film?
Try increase to 7
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,7),U=3,V=3)#
Script can remove line from 2 sequential frames, if greater You can use second, three or greater pass.
Can upload Your video and I am try this.
yup.

[fonzzie31]

Theorically, as the source is taken from Laserdisc, the video is interlaced...

I can prepare you a video sample, if you want.
I can open my ftp for you. I hope you've a great internet connection.

The file will be a bit heavy, because it's AVI RGB 24Bits.

I tried with the new value.... no change.... still dropout on screen...

[yup]

fonzzie31!
During weekend I am out from broadband Internet, but from Monday no problem.
yup.

[Mounir]

Have you tried the DvDropout script (http://forum.doom9.org/showthread.php?t=128649&highlight=dropout) Fonzzie31 works well on such lines depending if there is lot of motion or not.I'm from Tlse by the way :)

[fonzzie31]

@Yup : I prepare some samples for you... I'll send you a private message on Monday, in order to give my ftp details.

@Mounir : DvDropout does not work, with my videos. Nice to see you're in Toulouse too :)

[jmac698]

@Yup
I am making installer, do you have any improvements?

#Remove Black Horizontal Stripes
#Author: Yup, documented and packaged by jmac698
#Version: 2011-10-11, mod1.0
#Homepage: http://forum.doom9.org/showthread.php?t=121197
#Requirements:
#Masktools
#http://forum.doom9.org/showthread.php?t=98985
#MVTools2
#http://www.avisynth.org.ru/mvtools/mvtools2.html
#Removegrain RemovegrainT
#http://www.removegrain.de.tf/
#only AVSRecursion (need install to Windows\system32 directory)
#http://videoprocessing.11.forumer.com/
#Nnedi3
#http://web.missouri.edu/~kes25c/
#Optional:
#Avisynth MT
#http://avisynth.org/mediawiki/MT_support_page
#Usage:
#Provide a clip for source
#Set linethick to the desired amount. Note that in this version, the linethick parameter is NOT fully implemented, just use 3!
#Set mt to true if you are using Avisynth MT

SetMemoryMax(768)
mt=false
mt?SetMTMode(3,4):nop
#source=DirectShowSource("lecture1.avi",pixel_type="YUY2")#.Trim(750,1250)
source=colorbars(pixel_type="YV12")
mt?SetMTMode(2,4):nop
source=source.AssumeTFF().ConvertToYV12(interlaced=true)#.ColorYUV(autogain=true, autowhite=true)
fieldmaskedsdi=removestripes(source, 3)
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source)))
#Weave(fieldmaskedsdi)
StackVertical(Separatefields(source),fieldmaskedsdi)

function removestripes(clip source, int linethick){
#rect1=5 for 3 pixels thickness line, rect1=3 for 2 pixels thickness line, rect1=1 for 1 pixel thickness line
rect1=linethick*2-1
bobnn=source.nnedi3(field=-2)
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,rect1),U=3,V=3)# rectangle 5 for 3 pixels thickness line, 3 for 2, 1 for 1
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
bobnnf=bobnn#.dfttest(tbsize=1,ftype=1,sbsize=16,sosize=12,sigma=1000,U=false,V=false)
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
superff=MSuper(bobnnf)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnnf, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnnf, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnnf, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnnf, super, fw2, thSAD=16000, thSCD1=16000)
bwabs1=mt_lutxy(bobnnf,bc1,"x y - abs")#,U=-128,V=-128)
bwabs2=mt_lutxy(bobnnf,bc2,"x y - abs")#,U=-128,V=-128)
fwabs1=mt_lutxy(bobnnf,fc1,"x y - abs")#,U=-128,V=-128)
fwabs2=mt_lutxy(bobnnf,fc2,"x y - abs")#,U=-128,V=-128)
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
#
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
bwsad=mt_logic(mb1sad,mb2sad,"max")#,U=2,V=2)
fwsad=mt_logic(mf1sad,mf2sad,"max")#,U=2,V=2)
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
#
# sort SAD
# center filtered
mt_logic(mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and"),mt_lut(centersad,"x 255 <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?",U=3,V=3)
# backward filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?",U=3,V=3)
# forward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?",U=3,V=3)
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
#end sort SAD
#

#
# sort SDI
# center SDI
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# backward SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# forward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#

fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
fieldmaskedsdi
}

# Spike Detection Function
#
function SDIAdapt(clip b1,clip b2)
{
threshsp=10
THP=string(threshsp)
threshavg=2
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
SDIAvgblk=SDIavg.PointResize(90,72)
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
SDIavg=SDIAvgblk.PointResize(720,576)
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4))# rectangle 4 for 3 pixels thickness line, 2 for 2, not need for 1
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(4,0))# remove line shorter 5 pixels
SDImaskleft=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 -1 0 -2 0 -3 0 -4 0 -5 0 -6 0 -7 0 -8 0 -9 0 -10 0 -11 0")# try
SDImaskright=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 11 0") # join
SDIMask=mt_logic(SDImaskleft,SDImaskright,"min") # sequental horizontal line
SDImask=SDImask.mt_luts(SDImask,mode="min",pixels=mt_rectangle(18,0)) # remove line shorter 37 pixels
SDImask=SDImask.HorlLinExp() # fill line with noise
SDIad=mt_logic(SDIad,SDImask,"min") # refine source SDI
SDIad=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4)) # rectangle 4 for 3 pixels thikness line, 2 for 2, not need for 1
return(SDIad)
}

# fill full horizontal line if exist one pixel per line
function HorlLinExp(clip mask1) {
mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}

[jmac698]

Installer
http://hotfile.com/dl/133672065/58c5414/Avisynth-Stripes.exe.html

[jmac698]

@yup,
We need some help. I don't know how to use your script.
http://forum.videohelp.com/threads/340068-sort-of-black-comets-trails-in-hi8-video/page2

[yup]

jmac698!
I try find time and write comment inside body script. Now I am not ready for script with parameters set.
Script work for example remove 2 horizontal lines (1 horizontal line untouched and will be removed later at additional pass).
After bobbing using nnedi3 we get 3 lines, for median filtering need 7 vertical pixels mt_rectangle(0,3).
After filtering with dfttest remove segregation for better motion estimation.
After motion estimation i search better choice for median filtering from 3 motion compensating cases (1 p-1 ,p, p+1; 2 p, p, p+1, p+2; 3 p, p-1, p-2) with minimum SAD.
Also I search spike detection index for the same 3 cases. SDI have priority under motion search because median time filtering introduce artifacts.
At SDI i am using post processing for find only current thickness horizontal stripe mt_rectangle(0,2) for thickness 2. Additional calculation remove artifacts (small horizontal stripe false alarm from SDI) and join small stripes in one long.
SDI algo from
http://www.mee.tcd.ie/~ack/papers/a4ackphd.ps.gz
in my case for 720x576 PAL.
You need different approach for remove long lines and short post-processing SDI.
Also can add repair at end using filtering and source clip
diff=source-filtered
df=medianverticalfiltered(with thickness less 1)
filterd=source-df
Main idea touch only bad part of frame.
yup.

[yup]

Script with comments:
SetMemoryMax(768)
SetMTMode(3,4)
source=DirectShowSource("lecture1.avi",pixel_type="YUY2")
#source file
SetMTMode(2,4)
source=source.AssumeTFF().ConvertToYV12(interlaced=true)#.ColorYUV(autogain=true, autowhite=true)
bobnn=source.nnedi3(field=-2)
# bobbing for better motion estimation, for 3 pixels thickness line became 5 pixels thickness, for 2 3, for 1 1
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,5),U=3,V=3)
# vertical median filter rectangle 5 for 3 pixels source (not bobbing) thikness line, 3 for 2, 1 for 1
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
#dft filter remove segregation after median vertical filter
bobnnf=bobnn#.dfttest(tbsize=1,ftype=1,sbsize=16,sosize=12,sigma=1000,U=false,V=false)
# motion estimation on filtered and compensating on source
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
superff=MSuper(bobnnf)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnnf, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnnf, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnnf, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnnf, super, fw2, thSAD=16000, thSCD1=16000)
# calculation absolute difference between source and compensated
bwabs1=mt_lutxy(bobnnf,bc1,"x y - abs")#,U=-128,V=-128)
bwabs2=mt_lutxy(bobnnf,bc2,"x y - abs")#,U=-128,V=-128)
fwabs1=mt_lutxy(bobnnf,fc1,"x y - abs")#,U=-128,V=-128)
fwabs2=mt_lutxy(bobnnf,fc2,"x y - abs")#,U=-128,V=-128)
# calculation spike detection index for center, forwad and backward
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
# calculating SAD for filtered for estimation better choice for motion compensation
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
# maximum SAD from forward and bacward measure for quality center compensation
bwsad=mt_logic(mb1sad,mb2sad,"max")#,U=2,V=2)
#for bacward compensation
fwsad=mt_logic(mf1sad,mf2sad,"max")#,U=2,V=2)
#for forward compensation
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
# inverting mask for using mt_merge
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
# motion compensated median filtering for center, backward and forward compensation
#
# sort SAD for finding better motion compensated median filtering, this approach can use without SDI for pixels and 1 line thickness stripes
# center filtered
mt_logic(mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and"),mt_lut(centersad,"x 255 <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?",U=3,V=3)
# backward filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?",U=3,V=3)
# forward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?",U=3,V=3)
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
#best value based on 3 motion compensated median time filterd value and spatial filtered
# where motion compensation bad, work only for one frame with stripe
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
# backward and forward compensated full for SDI approach
#end sort SAD
#

#
# sort SDI for choose center, forward or forward compensation work with 2 sequential frames, for 3 need 2 pass filtering, for 4 3
# center SDI
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# backward SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# forward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


# this place for repair now I am thinking

fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
StackVertical(Separatefields(source),fieldmaskedsdi)
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source)))
#Weave(fieldmaskedsdi)
# some kind comparing and weave for interlaced source, may be at double rate and for second pass not need bobbing

# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
threshsp=10
# threshold for pixels value absolute difference
THP=string(threshsp)
threshavg=2
# threshold how much times actual SDI greater avearage and decimated SDI for remove false alarm for bad motion compensation
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
# SDI calculation
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
# SDI averaging
SDIAvgblk=SDIavg.PointResize(90,72)
# decimation
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
# median point at X position
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
#median point at Cross position
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
# multi median filtering for remove false alarm SDI
SDIavg=SDIAvgblk.PointResize(720,576)
# backward resize to original frame size
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
# comparing with actual SDI and if greater it is real spike, value 128 can increase to 192
# be carefull I can not find this at original paper, big value can loose spike, small false alarm
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4))
# rectangle size could one less than for bobnnmed
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(4,0))
# remove line shorter depend Your source
SDImaskleft=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 -1 0 -2 0 -3 0 -4 0 -5 0 -6 0 -7 0 -8 0 -9 0 -10 0 -11 0")
# try join single short spike to long 11 distance between spikes from left side
SDImaskright=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 11 0")
# from right
SDIMask=mt_logic(SDImaskleft,SDImaskright,"min")
# if pixels exist from left and right simultanessly this spike for detections series short spikes
SDImask=SDImask.mt_luts(SDImask,mode="min",pixels=mt_rectangle(18,0))
# remove spikes shorter than 37 pixels tune for source
SDImask=SDImask.HorlLinExp()
# if line have only one pixel line will be filled
SDIad=mt_logic(SDIad,SDImask,"min")
# combination source and adapting SDI for better robustness
SDIad=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4))
# refinement rectangle size could one less than for bobnnmed
return(SDIad)

}

# fill full horizontal line if exist one pixel per line
# basaed on Didee idea se masktools thread
function HorlLinExp(clip mask1) {

mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}
Now i do not have any top secrets :).
One more welcome to suggesting and developing.
yup.

[mammo1789]

Thanks yup for your script i tried it and i get some removal of the black lines (especially the thin ones ), but the rest is still here can you suggest me some tweaking of your script or what threshold should i use for these lines.
I'm posting pictures to see the difference

http://forum.videohelp.com/attachments/9449-1320203871/za%20crni%20linii_0.00.01.15.jpg

http://forum.videohelp.com/attachments/9449-1320203871/za%20crni%20linii_0.00.01.15.jpg

http://forum.videohelp.com/attachments/9450-1320203875/ddd555_0.00.01.15.jpg

[yup]

mammo1789!
Welcome
try for preview
source=DirectShowSource("Your.avi")
source=source.AssumeTFF().ConvertToYV12(interlaced=true)
bobnn=source.nnedi3(field=-2)
# bobbing for better motion estimation, for 3 pixels thickness line became 5 pixels thickness, for 2 3, for 1 1
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,X),U=3,V=3)
#X need tuned
field=bobnnmed.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
stackvertical(SeparateFields(source),field)
Increase value second parameter for mt_rectangle(0,X) until remove your black lines at preview.
Inside SDIAdapt function You need mt_rectangle(0,X-1). Read comments. All threshold work relatively good. For Your source need no one pass. Once I use 4 pass for my source 2 with X=5, and 2 with X=3 and get good result (see my screenshot at #45 post).
Upload 100 frames Your video.
yup.

[Jenyok]

yup
.
Whether correctly I have understood you, that the following (next) Code is for progressive video ?
See below.
May be need some corrections in Code ?
.

LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\MVTOOLS-V2_5_10\mvtools2.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\MASKTOOLS-V2_0A48\mt_masktools-25.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\DFTTESTV_1_6\dfttest.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAINT-1_0\removegraint.dll")

# Spike Detection Function
#
function SDIAdapt(clip b1, clip b2)
{
threshsp=10
# threshold for pixels value absolute difference
THP=string(threshsp)
threshavg=2
# threshold how much times actual SDI greater avearage and decimated SDI for remove false alarm for bad motion compensation
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
# SDI calculation
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
# SDI averaging
SDIAvgblk=SDIavg.PointResize(90,72)
# decimation
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
# median point at X position
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
#median point at Cross position
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
# multi median filtering for remove false alarm SDI
SDIavg=SDIAvgblk.PointResize(720,576)
# backward resize to original frame size
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
# comparing with actual SDI and if greater it is real spike, value 128 can increase to 192
# be carefull I can not find this at original paper, big value can loose spike, small false alarm
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4))
# rectangle size could one less than for bobnnmed
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(4,0))
# remove line shorter depend Your source
SDImaskleft=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 -1 0 -2 0 -3 0 -4 0 -5 0 -6 0 -7 0 -8 0 -9 0 -10 0 -11 0")
# try join single short spike to long 11 distance between spikes from left side
SDImaskright=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 11 0")
# from right
SDIMask=mt_logic(SDImaskleft,SDImaskright,"min")
# if pixels exist from left and right simultanessly this spike for detections series short spikes
SDImask=SDImask.mt_luts(SDImask,mode="min",pixels=mt_rectangle(18,0))
# remove spikes shorter than 37 pixels tune for source
SDImask=SDImask.HorlLinExp()
# if line have only one pixel line will be filled
SDIad=mt_logic(SDIad,SDImask,"min")
# combination source and adapting SDI for better robustness
SDIad=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4))
# refinement rectangle size could one less than for bobnnmed
return(SDIad)
}

# fill full horizontal line if exist one pixel per line
# basaed on Didee idea se masktools thread
#
function HorlLinExp(clip mask1)
{
mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

# v1=interleave(mask1, black).assumefieldbased.assumetff.weave
# v2=interleave(mask2, black).assumefieldbased.assumetff.weave
# mt_hysteresis(v1, v2, u=-128, v=-128)
# return separatefields.selecteven.assumeframebased

v1=interleave(mask1, black)
v2=interleave(mask2, black)
return mt_hysteresis(v1, v2, u=-128, v=-128)
}

AVISource("1991 Programma A.avi")

ConvertToYUY2()

# Для отладки фильтров часть 1
#
SourceView=last
SourceView=AddBorders(SourceView, 0, 0, 8, 0, Color=$FFFFFF)
SourceView=ConvertToYUY2(SourceView)

ConvertToYV12()

bobnn = last

# bobbing for better motion estimation, for 3 pixels thickness line became 5 pixels thickness, for 2 3, for 1 1
bobnnmed = bobnn.mt_luts(bobnn, mode = "median", pixels = mt_rectangle(0, 5), U = 3, V = 3)

# vertical median filter rectangle 5 for 3 pixels source (not bobbing) thikness line, 3 for 2, 1 for 1
bobnnmedf = bobnnmed.dfttest(tbsize = 1, ftype = 1, sbsize = 8, sosize = 6, sigma = 1000, U = false, V = false)

#dft filter remove segregation after median vertical filter
bobnnf = bobnn#.dfttest(tbsize = 1, ftype = 1, sbsize = 16, sosize = 12, sigma = 1000, U = false, V = false)

# motion estimation on filtered and compensating on source
super = MSuper(bobnn)
superf = MSuper(bobnnmedf, chroma = false)
superff = MSuper(bobnnf)
bw1 = MAnalyse(superf, blksize = 8, isb = true, delta = 2, overlap = 4, dct = 5, chroma = false)
bw2 = MAnalyse(superf, blksize = 8, isb = true, delta = 4, overlap = 4, dct = 5, chroma = false)
fw1 = MAnalyse(superf, blksize = 8, isb = false, delta = 2, overlap = 4, dct = 5, chroma = false)
fw2 = MAnalyse(superf, blksize = 8, isb = false, delta = 4, overlap = 4, dct = 5, chroma = false)
bc1 = MCompensate(bobnnf, super, bw1, thSAD = 16000, thSCD1 = 16000)
bc2 = MCompensate(bobnnf, super, bw2, thSAD = 16000, thSCD1 = 16000)
fc1 = MCompensate(bobnnf, super, fw1, thSAD = 16000, thSCD1 = 16000)
fc2 = MCompensate(bobnnf, super, fw2, thSAD = 16000, thSCD1 = 16000)

# calculation absolute difference between source and compensated
bwabs1 = mt_lutxy(bobnnf, bc1, "x y - abs")#, U = -128, V = -128)
bwabs2 = mt_lutxy(bobnnf, bc2, "x y - abs")#, U = -128, V = -128)
fwabs1 = mt_lutxy(bobnnf, fc1, "x y - abs")#, U = -128, V = -128)
fwabs2 = mt_lutxy(bobnnf, fc2, "x y - abs")#, U = -128, V = -128)

# calculation spike detection index for center, forwad and backward
SDIc = SDIAdapt(bwabs1, fwabs1)
SDIb = SDIAdapt(bwabs1, bwabs2)
SDIf = SDIAdapt(fwabs1, fwabs2)

# calculating SAD for filtered for estimation better choice for motion compensation
mb1sad = bobnnmedf.MMask(bw1, kind = 1, ml = 100, Ysc = 255, thSCD1 = 16000)
mb2sad = bobnnmedf.MMask(bw2, kind = 1, ml = 100, Ysc = 255, thSCD1 = 16000)
mf1sad = bobnnmedf.MMask(fw1, kind = 1, ml = 100, Ysc = 255, thSCD1 = 16000)
mf2sad = bobnnmedf.MMask(fw2, kind = 1, ml = 100, Ysc = 255, thSCD1 = 16000)
centersad = mt_logic(mf1sad, mb1sad, "max")#, U = 2, V = 2)

# maximum SAD from forward and bacward measure for quality center compensation
bwsad = mt_logic(mb1sad, mb2sad, "max")#, U = 2, V = 2)

# for bacward compensation
fwsad = mt_logic(mf1sad, mf2sad, "max")#, U = 2, V = 2)

# for forward compensation
centermask = mt_invert(centersad)
bwmask = mt_invert(bwsad)
fwmask = mt_invert(fwsad)

# inverting mask for using mt_merge
mcfcenter = clense(bc1, bobnn, fc1, increment = 0, grey = false)
mcfbw = clense(bc1, bobnn, bc2, increment = 0, grey = false)
mcffw = clense(fc1, bobnn, fc2, increment = 0, grey = false)

# motion compensated median filtering for center, backward and forward compensation
# sort SAD for finding better motion compensated median filtering, this approach can use without SDI for pixels and 1 line thickness stripes
# center filtered
mt_logic(mt_logic(mt_lutxy(centersad, bwsad, "x y <= "), mt_lutxy(centersad, fwsad, "x y <="), "and"), mt_lut(centersad, "x 255 <="), "and")
maskcentersad = mt_lutxy(last, centersad, "x 255 y - 0 ?", U = 3, V = 3)

# backward filtered
mt_logic(mt_lutxy(bwsad, centersad, "x y < "), mt_lutxy(bwsad, fwsad, "x y <="), "and")
maskbwsad = mt_lutxy(last, bwsad, "x 255 y - 0 ?", U = 3, V = 3)

# forward filtered
mt_logic(mt_lutxy(fwsad, centersad, "x y < "), mt_lutxy(fwsad, bwsad, "x y < "), "and")
maskfwsad = mt_lutxy(last, fwsad, "x 255 y - 0 ?", U = 3, V = 3)

#
mcfmaskedsad = mt_merge(bobnnmed, mcfbw, maskbwsad, luma = true)
mcfmaskedsad = mt_merge(mcfmaskedsad, mcffw, maskfwsad, luma = true)
mcfcentersad = mt_merge(mcfmaskedsad, mcfcenter, maskcentersad, luma = true)

# best value based on 3 motion compensated median time filterd value and spatial filtered
# where motion compensation bad, work only for one frame with stripe
mcfbwsad = mt_merge(bobnnmed, mcfbw, maskbwsad, luma = true)
mcffwsad = mt_merge(bobnnmed, mcffw, maskfwsad, luma = true)

# backward and forward compensated full for SDI approach
# end sort SAD

# sort SDI for choose center, forward or forward compensation work with 2 sequential frames, for 3 need 2 pass filtering, for 4 3
# center SDI
maskcentersdi = mt_logic(mt_logic(mt_lutxy(SDIc, SDIb, "x y >= "), mt_lutxy(SDIc, SDIf, "x y >="), "and"), SDIc, "and").mt_lut("x 255 0 ?")

# backward SDI
maskbwsdi = mt_logic(mt_lutxy(SDIb, SDIc, "x y > "), mt_lutxy(SDIb, SDIf, "x y >="), "and").mt_lut("x 255 0 ?")

# forward SDI
maskfwsdi = mt_logic(mt_lutxy(SDIf, SDIc, "x y > "), mt_lutxy(SDIf, SDIb, "x y >"), "and").mt_lut("x 255 0 ?")

#
maskedsdi = mt_merge(bobnn, mcfcentersad, maskcentersdi, luma = true)
maskedsdi = mt_merge(maskedsdi, mcffwsad, maskfwsdi, luma = true)
maskedsdi = mt_merge(maskedsdi, mcfbwsad, maskbwsdi, luma = true)
# end sort SDI

last = maskedsdi

ConvertToYUY2()

# Для отладки фильтров часть 2
#
StackHorizontal(SourceView, last)

[yup]

Jenyok!
SDIAdapt for progressive frame. I get progressive from interlaced using nnedi3. Also this code for PAL 720x576.
Spike detection index algorithm see post
http://forum.doom9.org/showthread.php?p=1534999#post1534999
Rmaining code try remove artifacts. After weekend I make small update.
If your source progressive you not need use nnedi3 and delta at MAnalyse divide by 2.
yup.

[Jenyok]

yup
Are you in IXBT.com forum of AVISynth ?
If yes, please contact with me there.
Thanks...

[yup]

Hi All!
Please
thicknessline=2# thickness horizontal spike in pixels
Global radvertmed=2*thicknessline-1
Global halflength=18# detect line longer 2*halflength+1 pixels
#SetMemoryMax(768)
#SetMTMode(3,4)
source=DirectShowSource("flower.avi",pixel_type="YUY2")
#source file
#SetMTMode(2,4)
source=source.AssumeTFF().ConvertToYV12(interlaced=true)
bobnn=source.nnedi3(field=-2,threads=1)
# bobbing for better motion estimation, for 3 pixels thickness line became 5 pixels thickness, for 2 3, for 1 1
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,radvertmed),U=3,V=3)
# vertical median filter radvertmed 5 for 3 pixels source (not bobbing) thikness line, 3 for 2, 1 for 1
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
#dft filter remove segregation after median vertical filter
# motion estimation on filtered and compensating on source
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnn, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnn, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnn, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnn, super, fw2, thSAD=16000, thSCD1=16000)
# calculation absolute difference between source and compensated
bwabs1=mt_lutxy(bobnn,bc1,"x y - abs")
bwabs2=mt_lutxy(bobnn,bc2,"x y - abs")
fwabs1=mt_lutxy(bobnn,fc1,"x y - abs")
fwabs2=mt_lutxy(bobnn,fc2,"x y - abs")
# calculation spike detection index for center, forwad and backward
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
# calculating SAD for filtered for estimation better choice for motion compensation
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
# maximum SAD from forward and bacward measure for quality center compensation
bwsad=mt_logic(mb1sad,mb2sad,"max")
#for bacward compensation
fwsad=mt_logic(mf1sad,mf2sad,"max")
#for forward compensation
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
# inverting mask for using mt_merge
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
# motion compensated median filtering for center, backward and forward compensation
#
# sort SAD for finding better motion compensated median filtering, this approach can use without SDI for pixels and 1 line thickness stripes
# center filtered
mt_logic(mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and"),mt_lut(centersad,"x 255 <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?")
# backward filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?")
# forward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?")
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
#best value based on 3 motion compensated median time filterd value and spatial filtered
# where motion compensation bad, work only for one frame with stripe
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
# backward and forward compensated full for SDI approach
#end sort SAD
#

#
# sort SDI for choose center, forward or forward compensation work with 2 sequential frames, for 3 need 2 pass filtering, for 4 3
# center SDI
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# backward SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# forward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


# this place for repair now I am thinking

fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
StackVertical(Separatefields(source),fieldmaskedsdi)
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source)))
#Weave(fieldmaskedsdi)
# some kind comparing and weave for interlaced source, may be at double rate and for second pass not need bobbing

# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
threshsp=10
# threshold for pixels value absolute difference
THP=string(threshsp)
threshavg=2
# threshold how much times actual SDI greater avearage and decimated SDI for remove false alarm for bad motion compensation
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
# SDI calculation
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
# SDI averaging
SDIAvgblk=SDIavg.PointResize(90,72)
# decimation
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
# median point at X position
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
#median point at Cross position
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
# multi median filtering for remove false alarm SDI
SDIavg=SDIAvgblk.PointResize(720,576)
# backward resize to original frame size
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
# comparing with actual SDI and if greater it is real spike, value 128 can increase to 192
# be carefull I can not find this at original paper, big value can loose spike, small false alarm
SDIad = (radvertmed>1) ? SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,radvertmed-1)) : SDIad
# rectangle size could one less than for bobnnmed
SDIad = (radvertmed>1) ? SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(radvertmed-1,0)) : SDIad
SDImask=SDIad.mt_luts(SDIad,mode="min",pixels=mt_rectangle(halflength,0))
# remove spikes shorter than 2*halflength+1 pixels tune for source
SDImask=SDImask.HorlLinExp()
# if line have only one pixel line will be filled
SDIad=mt_logic(SDIad,SDImask,"min")
# combination source and adapting SDI for better robustness
return(SDIad)

}

# fill full horizontal line if exist one pixel per line
# basaed on Didee idea see masktools thread
function HorlLinExp(clip mask1) {

mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}
It is very good work for relatively long stripe. If You want remove series short need replace to

SDIadleft=SDIad.mt_luts(SDIad,mode="max",pixels=mt_freerectangle(-11,0,0,0))
# try join single short spike to long 11 distance between spikes from left side
SDImadright=SDIad.mt_luts(SDIad,mode="max",pixels=mt_freerectangle(0,0,11,0))
# from right
SDIMask=mt_logic(SDIadleft,SDIadright,"min")
# if pixels exist from left and right simultanessly this spike for detections series short spikes
SDImask=SDImask.mt_luts(SDImask,mode="min",pixels=mt_rectangle(halflength,0))
SDIMask=........
yup.

[Jenyok]

yup
Maybe write two (2) functions, for example
Long_Stripes(...) and Short_Stripes(...)
with different parameters (...)
and do *.avsi module ???
Think about this ?!

[Ghitulescu]

Thanks yup for your script i tried it and i get some removal of the black lines (especially the thin ones ), but the rest is still here can you suggest me some tweaking of your script or what threshold should i use for these lines.
Try a new player? Preferably in a better (newer) condition?

[yup]

yup
Maybe write two (2) functions, for example
Long_Stripes(...) and Short_Stripes(...)
with different parameters (...)
and do *.avsi module ???
Think about this ?!
It is depend from source.
I have source with only long spike, short spike comet style and combination. Script not automatic need tuning on source.
Also do not forget first need remove thick spike and after thin.
About avsi i am not ready, but think about this.
yup.

[Jenyok]

yup
Oh...
I need the same filter (script) to WHITE horizontal stripes !!!
I reaaly have such video content with WHITE horizontal stripes...
What could I change in your script ?

[yup]

Jenyok!
No problem. It is will be work. Need absolute difference between normal value and stripe. Name historical. May be need change thread name to horizontal stripe wiothout black for clarity?
yup.

[Jenyok]

Tested this script.
Script does not works.
White stripes.

[yup]

Jenyok!

Need choose right parameter.
First try remove from thick to thin lines thicknessline parameter. If You choose thicknessline=3 for source with lines only 2 and 1 pixels script make nothing.
For total remove need use sequential from thicker to thiner lines, starting with thicknessline=max to thicknessline=1.
If line gone in 3 frames successively need 2 pass with the same thicknessline parameter
The same situation with halflength parameter.
If You have lines with length shorter than 2*halflength+1 script untouch source.
You can upload small sample and I try find near opt set .
For white lines see example from this thread
http://forum.doom9.org/showthread.php?p=1531897#post1531897
even at old version script, now script work little better.
yup.

[fonzzie31]

Hello Yup !!!

Happy new year for you !!!!

Could you give me the last version of your script, please ?
I think I'll contact you because I encounter some problem with the script version I have. But before that, I would like to verify if the same problem occurs with latest version :)

Best regards,

[yup]

fonzzie31!
Hello!
Also Happy New Year!
See
http://forum.doom9.org/showthread.php?p=1541580#post1541580
But Your source not pure interlaced, need some correction when I made I post it.
For telecined source better approach give sequintial median filter with radius 1 at frame basis not field like prefilter not radius 3 one time.
yup.

[fonzzie31]

Hello Yup :)

Let me explain my problem, but I'll need to send you some screen capture in order to illustrate.

I had a true interlaced video to process with your script (one pass), and I noticed some visible artefacts on the result video which are not present in the source.

I tried with avs or with virtualdub : same problem occurs
I deinterlaced the video with avisynth : same problem occurs
Whatever codec I use for the video, the problem occurs.

I try other thing :
I put also the source video on Adobe After Effects, and I Use the professionnal Revision-FX Deinterlace plugin, in order to create 2 differents video files containing the top and bottom fields of the source video.
The 2 video files were processed by your script. The 2 video result files were loaded into After Effects in order to reinterlace the field, using professional Revision-FX Reinterlace plugin.
The same problem occurs.

So, the conclusion is that your script seems to create artefacts.

The better thing is to send you the complete source file. It's a pure interlaced Bananarama music video, grabbed from NTSC laserdisc, and converted into PAL, with a professionnal post-processing equipment (SNELL & WILCOX CVR450)

Tomorrow, I'll try to send you there some screen captures showing the problem.

[yup]

Hi fonzzie31!

I am do not speak that script not introduce artefacts:(
Artefacts place where do not work Spike Detection Index, this algorithm can give false alarm and with combination temporal median filter introduce artefacts.
Script tested on VHS capture material not very sharp and without small details.
I am see artefacts at source with artificial objects with sharp straight borders, because script using median filtering.
Right determine trhicknesline and length for line critical.
For interlaced source I am using my script with thicknesline parameter from maximum to 1.
If use script 1 pass with bigger thicknessline and 2 pass with less, at 2 pass can remove some artefacts.
After remove line I use QTGMC like deinterlacer which also can remove some artefacts.
yup.

[fonzzie31]

Hi Yup,

Let me explain and illustrate my problem.
The horizontal resolution for a NTSC Laserdisc is 425 analog lines. VHS is only 240 analog lines.

So, I took a interlaced PAL video file. This file was created by grabbing a laserdisc, with my Blackmagic Decklink Studio 2. This card use v210 codec for video.
As The video is interlaced, I loaded it in Adobe After Effects, with no field detection.
I used a professional Deinterlacer AE plugin, RevisionFX Fieldskit, in order to separate the field of the video.
I configured this plugin in order to make an output frame from the field selected (top or botton) in each source frame. The other field was discarded, and new information was created from the discarded field scanlines using the field selected.
The output files (one with top fields, one with bottom fields) are saved as RGB24 progressive videos.
I hope you understand what I did with After effects.

Then I created a batch file for running your script. 2 files to process. one pass each (the maximum number of passes I used for a video files is 6)
The result video file was in YV12.

I compared the source and the result video files, and I noticed some artefacts in the result video, in some frames, in particular during transitions or quick movements.
These frames are supposed not to contain any drop outs to remove.

Let me show you some transitions samples from a video:

Example 1 :

The source frame :
http://revival03.free.fr/temp/bananarama1s.jpg

The result frame :
http://revival03.free.fr/temp/bananarama1r.jpg

----------------
Example 2 :

The source frame :
http://revival03.free.fr/temp/bananarama2s.jpg

The result frame :
http://revival03.free.fr/temp/bananarama2r.jpg


----------------
Example 3 :

The source frame :
http://revival03.free.fr/temp/bananarama3s.jpg

The result frame :
http://revival03.free.fr/temp/bananarama3r.jpg

-----------------

I thought it was the after effect plugins which did a bad job.
So I decided to put a field separation function in Avisynth. But the same problem occurs on output. Same artefacts.
I tried to convert the source file in YV12 before processing it with your script. Same artefacts.
If I use AVS or Virtualdub, I've the same problem.

What do you think about that ?

[yup]

fonzzie31!
IMO it not related to my script.
I am try make some advice.
Try capture laserdisc like NTSC without any hardware conversion in YUY2 colorspace for AVisynth compatibility using lossless codec huff very good for start. After use QTGMC deinterlacer . You get 720x480 at 59.94 fps video. If result good, You can transform to PAL but this will be second step.
At source I see blended frame related with NTSC to PAL conversion.
Also color bar can be related with colorspace conversion.
Complex task coulde separated to simple one.
You can get answer at capture part this forum also.
yup.

[fonzzie31]

Thanks Yup

As a start, I will try to create an YV12 sample from the PAL converted video file.

I'll also try to create an YV12 sample from the original NTSC video.

I"ll send you the results in the next few days.

[fonzzie31]

Hello Yup,

Today, I made some tests...

I captured the video from a laserdisc. The video was kept into NTSC.
The video was grabbed into 10 Bits UYV Uncompressed (v210). I don't have any other possibilities.
So, I convert the video into YV12 (YUV 4:2:0) format, after separating the fields in After Effects (see my last post, for the procedure).

I processed the bottom fields with your script. One pass processing through AVS.
Input video : YV12 .Output video : YV12

I still see some artefacts in the result video.
I really think that one of the masks in your script creates these artefacts.
The artefacts is not the result of a color space conversion.
Input and output video format are the same.

Do you want to get the video file in order for you to make some tests ?


Example 1 :
------------

Source :

http://revival03.free.fr/temp/bananarama4s.jpg

Result :

http://revival03.free.fr/temp/bananarama4r.jpg


Example 2 :
-----------

Source :

http://revival03.free.fr/temp/bananarama5s.jpg

Result :

http://revival03.free.fr/temp/bananarama5r.jpg


Example 3 :
-----------

Source :

http://revival03.free.fr/temp/bananarama6s.jpg

Result :

http://revival03.free.fr/temp/bananarama6r.jpg


I used the following script :

-------------------------------

thicknessline=1# thickness horizontal spike in pixels
Global radvertmed=2*thicknessline-1
Global halflength=18# detect line longer 2*halflength+1 pixels

SetMemoryMax(768)
SetMTMode(3,4)
source=AVISource("../Sources/1984 - BANANARAMA - Rough Justice - NTSC - BF.avi", pixel_type="YV12")
#source file
#SetMTMode(2,4)
#source=source.ConvertToYV12(matrix="rec601")
#source=source.assumeframebased().Separatefields()
SetMTMode(4,4)
bobnn=source
#bobnn=Telecide(source,guide=2,post=0)
SetMTMode(2,4)
# bobbing for better motion estimation, for 3 pixels thickness line became 5 pixels thickness, for 2 3, for 1 1
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,radvertmed),U=3,V=3)
# vertical median filter rectangle radvertmed 5 for 3 pixels source (not bobbing) thikness line, 3 for 2, 1 for 1
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
#dft filter remove segregation after median vertical filter
#.dfttest(tbsize=1,ftype=1,sbsize=16,sosize=12,sigma=1000,U=false,V=false)
# motion estimation on filtered and compensating on source
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnn, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnn, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnn, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnn, super, fw2, thSAD=16000, thSCD1=16000)
# calculation absolute difference between source and compensated
bwabs1=mt_lutxy(bobnn,bc1,"x y - abs")#,U=-128,V=-128)
bwabs2=mt_lutxy(bobnn,bc2,"x y - abs")#,U=-128,V=-128)
fwabs1=mt_lutxy(bobnn,fc1,"x y - abs")#,U=-128,V=-128)
fwabs2=mt_lutxy(bobnn,fc2,"x y - abs")#,U=-128,V=-128)
# calculation spike detection index for center, forwad and backward
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
# calculating SAD for filtered for estimation better choice for motion compensation
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
# maximum SAD from forward and bacward measure for quality center compensation
bwsad=mt_logic(mb1sad,mb2sad,"max")#,U=2,V=2)
#for bacward compensation
fwsad=mt_logic(mf1sad,mf2sad,"max")#,U=2,V=2)
#for forward compensation
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
# inverting mask for using mt_merge
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
# motion compensated median filtering for center, backward and forward compensation
#
# sort SAD for finding better motion compensated median filtering, this approach can use without SDI for pixels and 1 line thickness stripes
# center filtered
mt_logic(mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and"),mt_lut(centersad,"x 255 <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?",U=3,V=3)
# backward filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?",U=3,V=3)
# forward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?",U=3,V=3)
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
#best value based on 3 motion compensated median time filterd value and spatial filtered
# where motion compensation bad, work only for one frame with stripe
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
# backward and forward compensated full for SDI approach
#end sort SAD
#

#
# sort SDI for choose center, forward or forward compensation work with 2 sequential frames, for 3 need 2 pass filtering, for 4 3
# center SDI
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# backward SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# forward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


# this place for repair now I am thinking

#fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#StackVertical(Crop(bobnn,12,64,-22,-54),Crop(maskedsdi,12,64,-22,-54))
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source)))
maskedsdi



# some kind comparing and weave for interlaced source, may be at double rate and for second pass not need bobbing

# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
threshsp=10
# threshold for pixels value absolute difference
THP=string(threshsp)
threshavg=2
# threshold how much times actual SDI greater avearage and decimated SDI for remove false alarm for bad motion compensation
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
# SDI calculation
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
# SDI averaging
SDIAvgblk=SDIavg.PointResize(90,72)
# decimation
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
# median point at X position
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
#median point at Cross position
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
# multi median filtering for remove false alarm SDI
SDIavg=SDIAvgblk.PointResize(720,480)
# backward resize to original frame size
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
# comparing with actual SDI and if greater it is real spike, value 128 can increase to 192
# be carefull I can not find this at original paper, big value can loose spike, small false alarm
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,1))
# rectangle size could one less than for bobnnmed
SDImask=SDIad#.mt_luts(SDIad,mode="median",pixels=mt_rectangle(4,0))
# remove line shorter depend Your source
# SDImaskleft=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 -1 0 -2 0 -3 0 -4 0 -5 0 -6 0 -7 0 -8 0 -9 0 -10 0 -11 0")
# try join single short spike to long 11 distance between spikes from left side
# SDImaskright=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 11 0")
# from right
# SDIMask=mt_logic(SDImaskleft,SDImaskright,"min")
# if pixels exist from left and right simultanessly this spike for detections series short spikes
SDImask=SDImask.mt_luts(SDImask,mode="min",pixels=mt_rectangle(3,0))
# remove spikes shorter than 11 pixels tune for source
SDImask=SDImask.HorlLinExp()
# if line have only one pixel line will be filled
SDIad=mt_logic(SDIad,SDImask,"min")
# combination source and adapting SDI for better robustness
# SDIad=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,1))
# refinement rectangle size could one less than for bobnnmed
return(SDIad)

}

# fill full horizontal line if exist one pixel per line
# basaed on Didee idea se masktools thread
function HorlLinExp(clip mask1) {

mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}

[yup]

fonzzie31!
Do you want to get the video file in order for you to make some tests ?
Yes.
I am agree about artefacts.
Will be think.
yup.

[fonzzie31]

I send you details in a private message :)

[yup]

Hi All!

Small update code and comments.
Script for remove long lines
Global thicknessline=2# thickness horizontal spike in pixels at field scale (not frame)
Global radvertmed=2*thicknessline-1
Global halflength=18# detect line longer 2*halflength+1 pixels
#SetMemoryMax(768)
#SetMTMode(3,4)
source=AVISource("sel97.avi")
#source file
#SetMTMode(2,4)
source=source.AssumeTFF().ConvertToYV12(interlaced=true)
bobnn=source.nnedi3(field=-2,threads=1)
# bobbing for better motion estimation
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,radvertmed),U=3,V=3)
# vertical median filter
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
#dft filter remove segregation after median vertical filter
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnn, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnn, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnn, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnn, super, fw2, thSAD=16000, thSCD1=16000)
# motion estimation on filtered and compensating on source
bwabs1=mt_lutxy(bobnn,bc1,"x y - abs")
bwabs2=mt_lutxy(bobnn,bc2,"x y - abs")
fwabs1=mt_lutxy(bobnn,fc1,"x y - abs")
fwabs2=mt_lutxy(bobnn,fc2,"x y - abs")
# calculation absolute difference between source and compensated
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
# calculation spike detection index for center, forwad and backward
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
# calculating SAD for filtered for estimation better choice for motion compensation
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
# maximum SAD from forward and backward measure for quality center compensation
bwsad=mt_logic(mb1sad,mb2sad,"max")
#for backward compensation
fwsad=mt_logic(mf1sad,mf2sad,"max")
#for forward compensation
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
# inverting mask for using mt_merge
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
# motion compensated median filtering for center, backward and forward compensation
#
# sort SAD for finding better motion compensated median filtering, this approach can use without SDI for pixels and 1 line thickness stripes
mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?")
# center filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?")
# backward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?")
# forward filtered
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
#best value based on 3 motion compensated median time filtered value and spatial filtered
# where motion compensation bad, work only for one sequintial frame with spike
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
# backward and forward compensated full for SDI approach and spatial filtered where motion compensation bad
#end sort SAD
#

#
# sort SDI for choose center, forward or forward compensation work with 2 sequential frames, for 3 need 2 pass filtering, for 4 3 pass
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# center SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# backward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
# forward SDI
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


# this place for repair now I am thinking

fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
StackVertical(Separatefields(source),fieldmaskedsdi,mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#Weave(fieldmaskedsdi)
# some kind comparing and weave for interlaced source, may be at double rate and for second pass not need bobbing

# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
threshsp=10
# threshold for pixels value absolute difference from 2 sequitial frames for spike detection
THP=string(threshsp)
threshavg=2
# threshold how much times actual SDI greater avearage and decimated SDI for remove false alarm for bad motion compensation
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
# SDI calculation
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
# SDI averaging
SDIAvgblk=SDIavg.PointResize(90,72)# for PAL (90,72), FOR NTSC (90,60)
# decimation
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
# median point at X position
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
#median point at Cross position
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
# multi median filtering for remove false alarm SDI
SDIavg=SDIAvgblk.PointResize(720,576)# for PAL (720,576), FOR NTSC (720,480)
# backward resize to original frame size
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
# comparing with actual SDI and if greater it is real spike, value 128 can increase to 192
# be carefull I can not find this at original paper, big value can loose spike, small false alarm
SDIad = (radvertmed>1) ? SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,radvertmed-1)) : SDIad
# rectangle size could one less than for bobnnmed, for thinner lines need additional pass with thiknessline=thicknessline-1 until thicknessline=1
SDImask=SDIad.mt_luts(SDIad,mode="min",pixels=mt_rectangle(halflength,0))
# remove spikes shorter than 2*halflength+1 pixels tune for source
SDImask=SDImask.HorlLinExp()
# if line have only one pixel line will be filled
SDIad=mt_logic(SDIad,SDImask,"min")
# combination source and adapting SDI for better robustness
# SDIad = (radvertmed>1) ? SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(radvertmed-1,0)) : SDIad
SDIad = SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(thicknessline,0))
# additional remove short spikes which can be artefacts
return(SDIad)

}

# fill full horizontal line if exist one pixel per line
# basaed on Didee idea see masktools thread
function HorlLinExp(clip mask1) {

mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}
Any suggestion welcome.
yup.

[yup]

and series short lines
Global thicknessline=2# thickness horizontal spike in pixels at field scale (not frame)
Global radvertmed=2*thicknessline-1
Global distser=21# distance between short lines
Global halflength=65# detect line longer 2*halflength+1 pixels
#SetMemoryMax(768)
#SetMTMode(3,4)
source=AVISource("sel97.avi")
#source file
#SetMTMode(2,4)
source=source.AssumeTFF().ConvertToYV12(interlaced=true)
bobnn=source.nnedi3(field=-2,threads=1)
# bobbing for better motion estimation
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,radvertmed),U=3,V=3)
# vertical median filter
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
#dft filter remove segregation after median vertical filter
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnn, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnn, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnn, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnn, super, fw2, thSAD=16000, thSCD1=16000)
# motion estimation on filtered and compensating on source
bwabs1=mt_lutxy(bobnn,bc1,"x y - abs")
bwabs2=mt_lutxy(bobnn,bc2,"x y - abs")
fwabs1=mt_lutxy(bobnn,fc1,"x y - abs")
fwabs2=mt_lutxy(bobnn,fc2,"x y - abs")
# calculation absolute difference between source and compensated
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
# calculation spike detection index for center, forwad and backward
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
# calculating SAD for filtered for estimation better choice for motion compensation
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
# maximum SAD from forward and backward measure for quality center compensation
bwsad=mt_logic(mb1sad,mb2sad,"max")
#for backward compensation
fwsad=mt_logic(mf1sad,mf2sad,"max")
#for forward compensation
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
# inverting mask for using mt_merge
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
# motion compensated median filtering for center, backward and forward compensation
#
# sort SAD for finding better motion compensated median filtering, this approach can use without SDI for pixels and 1 line thickness stripes
mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?")
# center filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?")
# backward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?")
# forward filtered
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
#best value based on 3 motion compensated median time filtered value and spatial filtered
# where motion compensation bad, work only for one sequintial frame with spike
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
# backward and forward compensated full for SDI approach and spatial filtered where motion compensation bad
#end sort SAD
#

#
# sort SDI for choose center, forward or forward compensation work with 2 sequential frames, for 3 need 2 pass filtering, for 4 3 pass
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# center SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# backward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
# forward SDI
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


# this place for repair now I am thinking

fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
StackVertical(Separatefields(source),fieldmaskedsdi,mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#Weave(fieldmaskedsdi)
# some kind comparing and weave for interlaced source, may be at double rate and for second pass not need bobbing

# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
threshsp=10
# threshold for pixels value absolute difference from 2 sequitial frames for spike detection
THP=string(threshsp)
threshavg=2
# threshold how much times actual SDI greater avearage and decimated SDI for remove false alarm for bad motion compensation
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
# SDI calculation
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
# SDI averaging
SDIAvgblk=SDIavg.PointResize(90,72)# for PAL (90,72), FOR NTSC (90,60)
# decimation
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
# median point at X position
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
#median point at Cross position
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
# multi median filtering for remove false alarm SDI
SDIavg=SDIAvgblk.PointResize(720,576)# for PAL (720,576), FOR NTSC (720,480)
# backward resize to original frame size
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
# comparing with actual SDI and if greater it is real spike, value 128 can increase to 192
# be carefull I can not find this at original paper, big value can loose spike, small false alarm
SDIad = (radvertmed>1) ? SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,radvertmed-1)) : SDIad
# rectangle size could one less than for bobnnmed, for thinner lines need additional pass with thiknessline=thicknessline-1 until thicknessline=1
SDIad = SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(thicknessline,0))
#remove line shotter than thicknessline+1
SDIadleft=SDIad.mt_luts(SDIad,mode="max",pixels=mt_freerectangle(-distser,0,0,0))
# fill distser pixels from spike left side
SDIadright=SDIad.mt_luts(SDIad,mode="max",pixels=mt_freerectangle(0,0,distser,0))
# fill distser pixels from spike right side
SDIad=mt_logic(SDIadleft,SDIadright,"min")
# if pixels exist at SDIadleft and SDIadright fill pixels between 2 short lines
SDIad = (radvertmed>1) ? SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,radvertmed-1)) : SDIad
# need additional vertical median filtering after joining short lines for remove thiner lines
SDImask=SDIad.mt_luts(SDIad,mode="min",pixels=mt_rectangle(halflength,0))
# remove spikes shorter than 2*halflength+1 pixels tune for source
SDImask=SDImask.HorlLinExp()
# if line have only one pixel line will be filled
SDIad=mt_logic(SDIad,SDImask,"min")
# combination source and adapting SDI for better robustness
SDIad = SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(thicknessline,0))
# additional remove short spikes which can be artefacts
return(SDIad)

}

# fill full horizontal line if exist one pixel per line
# basaed on Didee idea see masktools thread
function HorlLinExp(clip mask1) {

mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}
yup.

[mammo1789]

I know is dumb question but how to make only output one video ( don't need comparison) instead of stack horizontal 4

[yup]

mammo1789!
#StackVertical(Separatefields(source),fieldmaskedsdi,mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
Weave(fieldmaskedsdi)
Comment before stackvertical and uncomment before weave.
Comparing need for tuning parameters, output will be frame.
yup.

[mammo1789]

Thanks yup I figure it out after looking at the script ( for an hour :)). I'm still not very familiar with long avisynth scripts but I'm getting there.
Concerning result I listen to your advice of using multiple passes with lower threshold but I get blurry results( at least for my taste).
And on default its not doing anything to the lines i see you made long and short streaks script separate and i tried both but no avail
http://img838.imageshack.us/img838/5828/92187688.png

[mammo1789]

I uploaded original short part of the clip with problems 3mb mjpeg http://www.mediafire.com/?uz148p9dyky999z for you if you have the time to look at it, and what will you suggest me to do with the script what parameter( or all the script that you will be using to get rid of them) I will very appreciate
Thanks Yup

[yup]

mammo1789!
Parameters could tune for source.
Try decrease parameter halflength (remove lines less than 2*halflength+1). If halflength high frame will be untouched.
Also try original script with StackVertical, third low part show difference between original and filtered (if You see only gray script untouched frame) and tune halflength and thicknessline parameter (for last more than 3 introduce a lot of artifacts) at low part you could see removed lines (white or black).
For Your sample need use version for long lines.
try updated script
Global thicknessline=2# thickness horizontal spike in pixels at field scale (not frame)
Global radvertmed=2*thicknessline-1
Global halflength=18# detect line longer 2*halflength+1 pixels
#SetMemoryMax(768)
#SetMTMode(3,4)
source=AVISource("sel97.avi")
#source file
#SetMTMode(2,4)
source=source.AssumeTFF().ConvertToYV12(interlaced=true)
bobnn=source.nnedi3(field=-2,threads=1)
# bobbing for better motion estimation
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,radvertmed),U=3,V=3)
# vertical median filter
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
#dft filter remove segregation after median vertical filter
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnn, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnn, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnn, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnn, super, fw2, thSAD=16000, thSCD1=16000)
# motion estimation on filtered and compensating on source
bwabs1=mt_lutxy(bobnn,bc1,"x y - abs")
bwabs2=mt_lutxy(bobnn,bc2,"x y - abs")
fwabs1=mt_lutxy(bobnn,fc1,"x y - abs")
fwabs2=mt_lutxy(bobnn,fc2,"x y - abs")
# calculation absolute difference between source and compensated
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
# calculation spike detection index for center, forwad and backward
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
# calculating SAD for filtered for estimation better choice for motion compensation
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
# maximum SAD from forward and backward measure for quality center compensation
bwsad=mt_logic(mb1sad,mb2sad,"max")
#for backward compensation
fwsad=mt_logic(mf1sad,mf2sad,"max")
#for forward compensation
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
# inverting mask for using mt_merge
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
# motion compensated median filtering for center, backward and forward compensation
#
# sort SAD for finding better motion compensated median filtering, this approach can use without SDI for pixels and 1 line thickness stripes
mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?")
# center filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?")
# backward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?")
# forward filtered
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
#best value based on 3 motion compensated median time filtered value and spatial filtered
# where motion compensation bad, work only for one sequintial frame with spike
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
# backward and forward compensated full for SDI approach and spatial filtered where motion compensation bad
#end sort SAD
#

#
# sort SDI for choose center, forward or forward compensation work with 2 sequential frames, for 3 need 2 pass filtering, for 4 3 pass
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# center SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# backward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
# forward SDI
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


# this place for repair now I am thinking

fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
StackVertical(Separatefields(source),fieldmaskedsdi,mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#Weave(fieldmaskedsdi)
# some kind comparing and weave for interlaced source, may be at double rate and for second pass not need bobbing

# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
threshsp=10
# threshold for pixels value absolute difference from 2 sequitial frames for spike detection
THP=string(threshsp)
threshavg=2
# threshold how much times actual SDI greater avearage and decimated SDI for remove false alarm for bad motion compensation
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
# SDI calculation
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
# SDI averaging
SDIAvgblk=SDIavg.PointResize(90,72)# for PAL (90,72), FOR NTSC (90,60)
# decimation
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
# median point at X position
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
#median point at Cross position
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
# multi median filtering for remove false alarm SDI
SDIavg=SDIAvgblk.PointResize(720,576)# for PAL (720,576), FOR NTSC (720,480)
# backward resize to original frame size
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
# comparing with actual SDI and if greater it is real spike, value 128 can increase to 192
# be carefull I can not find this at original paper, big value can loose spike, small false alarm
SDIad = (radvertmed>1) ? SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,radvertmed-1)) : SDIad
# rectangle size could one less than for bobnnmed, for thinner lines need additional pass with thiknessline=thicknessline-1 until thicknessline=1
SDIad=SDIad.mt_luts(SDIad,mode="min",pixels=mt_rectangle(halflength,0))
# remove spikes shorter than 2*halflength+1 pixels and shrink longer
SDIad=SDIad.mt_luts(SDIad,mode="max",pixels=mt_rectangle(halflength,0))
# extend spikes longer than 2*halflength+1 pixels
return(SDIad)

}

# fill full horizontal line if exist one pixel per line
# basaed on Didee idea see masktools thread
function HorlLinExp(clip mask1) {

mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}

yup.

[Mounir]

Error: Invalid arguments to Clense that's what i get. Pointing to line 52 in the script -> SDIc=SDIAdapt(bwabs1,fwabs1)

Which filters (and versions) do i need Yup for your last script , thanks

[yup]

Error: Invalid arguments to Clense that's what i get. Pointing to line 52 in the script -> SDIc=SDIAdapt(bwabs1,fwabs1)

Which filters (and versions) do i need Yup for your last script , thanks
Mounir!
You could use RemoveGrainT version dated 24:01:2008

http://home.arcor.de/kassandro/prerelease/RemoveGrainT-1.0.rar
yup.

[Mounir]

i get an error with removegrainT again "Unable to load removegrainT.dll error 0x7e

[yup]

Mounir!
All Kassandro plugin required avsrecursion.dll
http://home.arcor.de/kassandro/AvsRecursion/AvsRecursion.htm
Place dll at windows/system32 directory.
yup.

[Mounir]

I have it installed on sys32 already, im running win7 x64

[Didée]

On 64bit Windows, the "system32" folder (for 32bit applications) now is "SysWOW64". Has been mentioned a few dozen dozen times already.

[Mounir]

No luck on syswow64 folder --> "invalid args to function "clense"

Ok nevermind now the RemoveGrainT.dll is loaded w/o problem, script working

Well i can't say it's working great for long white stripes,i'll stick with manual work for now

[yup]

Mounir!
Can try RemovegrainTSSE2 instead of RemovegrainT, little faster.
mammo1789!
I try Your source starting point thicknessline=3 and halflength=8 remove some lines at 1 pass.
For one my source I made 2 pass with thicknessline=3 and after 2 pass with thicknessline=2.
You can try more pass with thicknessline=2.
See difference field for estimation removing lines.
One more, need tuning thicknessline and halflength parameter, if choose line thicker and longer for removing script make nothing.
yup.

[mammo1789]

I used what you suggested but i don't see any change the first is source the second is cleaned right? the look the same.
http://img21.imageshack.us/img21/7256/45528592.jpg

It's interesting that the script that you post it to me post 63 before so I can test it it did some removing but it has to be kept low bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,X -around 3 max ),U=3,V=3) instead it blurs very much.

The whole scripts ( long and short ones seems are less effective than that small script)
I'm I doing anything wrong.
If I understand you correctly i should load the script ( with 3 or 2 and 8 half and thik ) render it in vdub and load it again with the same parameters or 2 instead of 3 and so one right ?