Script for removal horizontal stripes (alpha release)

[yup]

Hi folk!
Alpha release :scared:!
For script need the same plugin like for QTGMC
http://forum.doom9.org/showthread.php?t=156028
and AVSRecursion, RemoveGrainT from kassandro
http://home.arcor.de/kassandro/AvsRecursion/AvsRecursion.htm
http://home.arcor.de/kassandro/prerelease/RemoveGrainT-1.0.rar
Using
AVISource("selnew.avi")
removeline(thicknessline=2, distser=1, halflength=3,comparing=false)
Script tested with Set last build Avisynth 2.6.
short parameters description
work with interlaced source (for film need some small adaptation)
thicknessline thickness horizontal spike in pixels at field scale (not frame) 2 good start for many VHS capture (if choose thicker than at source scrit do not work)
distser distance between short lines, if source have single spike (not series) no sense, but for speed use distser=1
halflength detect line longer 2*halflength+1 pixels, if source have spike 7 pixels (not longer) could be equal 3, if set higher than at source script do not work
comparing if true ouput will be stacked fields source, filtered and diff for tuning parameters
For some captures need sequintial using script, always need starting with biger thicknessline (first pass with thicknessline=2 second thicknessline=1, for very bad may be need three pass and starting from thicknessline=3).
Script can remove two sequintial spikes at the same place frame, if we have three sequintial spikes need use script 2 times with same thicknessline value.
Work very slowly 1.5-2 fps on i7 Core.
Any suggestion for optimization script welcome.

yup.

[Guest]

Post a screenshot showing the problem. Sounds like motor noise interference or something like that.

(With 165 posts I suppose you know about interlacing.)

[yup]

top field from source
http://img443.imageshack.us/img443/4222/stripes9vt.png
yup.

[Guest]

Yup, looks like analog shot noise is affecting your capture. Check all your cabling and grounds. Make sure you have no big motors, generators, refrigerators, etc., running nearby or on the same circuit. No car engines running nearby. That sort of thing. This is really hard to remove with filters.

[yup]

neuron2!
Thank you for reply. For power supply I use online UPS with isolated transformer (for PC and VCR). Early I made one capture without this stripes. I think this coupled with tape. I try capture 2 times and get stripes at same places.
yup.

[DeathWolf]

yes, it sounds like tape reading heads arent able to read these parts, i had the exact same problem when digitizing tapes. Unfortunatelly i couldnt find any decent way of removing them without hurting the rest of the video. And apparently the only solution is to get a better tape reader which can come very expensive for some formats(even though they are very old...)

[yup]

DeathWolf!
I use Sony 4-head VCR. VHS cassete made during edit from one VCR to other.
yup.

[yup]

Hi folk!
After many try and error i find script which remove stripes
AviSource("sel.avi",pixel_type="YUY2")
AssumeTFF()
Crop(16,16,-16,-16)
SeparateFields()
e=SelectEven()
o=selectOdd()

bvo = o.MVAnalyse(isb = true, truemotion=true, delta = 1, idx = 1, overlap=4, dct=0)
fvo = o.MVAnalyse(isb = false, truemotion=true, delta = 1, idx = 1, overlap=4, dct=0)
fco = o.MVCompensate(fvo, idx=1, thSCD1=500)
bco = o.MVFlow(bvo, idx=1, thSCD1=500) MVCompensate interleave(fco, o, bco)
DeGrainMedian(limitY=255,limitUV=255,mode=1, norow=true)
od=selectevery(3,1)

bve = e.MVAnalyse(isb = true, truemotion=true, delta = 1, idx = 1, overlap=4, dct=0)
fve = e.MVAnalyse(isb = false, truemotion=true, delta = 1, idx = 1, overlap=4, dct=0)
fce = e.MVFlow(fve, idx=1, thSCD1=500)
bce = e.MVFlow(bve, idx=1, thSCD1=500)
interleave(fce, e, bce)
DeGrainMedian(limitY=255,limitUV=255,mode=1, norow=true)
ed=selectevery(3,1)

interleave(ed,od)
Weave()
AddBorders(16,16,16,16)
Degrainmedian need use in pure median mode. Now I try find other median filter for time plane and postfilter.
yup.

[Didée]

Hi folk!
After many try and error i find script which remove stripes
AviSource("sel.avi",pixel_type="YUY2")
AssumeTFF()
Crop(16,16,-16,-16)
SeparateFields()
e=SelectEven()
o=selectOdd()

bvo = o.MVAnalyse(isb = true, truemotion=true, delta = 1, idx = 1, overlap=4, dct=0)
fvo = o.MVAnalyse(isb = false, truemotion=true, delta = 1, idx = 1, overlap=4, dct=0)
fco = o.MVCompensate(fvo, idx=1, thSCD1=500)
bco = o.MVFlow(bvo, idx=1, thSCD1=500) MVCompensate interleave(fco, o, bco)
DeGrainMedian(limitY=255,limitUV=255,mode=1, norow=true)
od=selectevery(3,1)

bve = e.MVAnalyse(isb = true, truemotion=true, delta = 1, idx = 1, overlap=4, dct=0)
fve = e.MVAnalyse(isb = false, truemotion=true, delta = 1, idx = 1, overlap=4, dct=0)
fce = e.MVFlow(fve, idx=1, thSCD1=500)
bce = e.MVFlow(bve, idx=1, thSCD1=500)
interleave(fce, e, bce)
DeGrainMedian(limitY=255,limitUV=255,mode=1, norow=true)
ed=selectevery(3,1)

interleave(ed,od)
Weave()
AddBorders(16,16,16,16)

From the idea, that should work. But the script as you posted is buggy.

Blue seems fishy: one time flow, one time compensate?

Green will cause some syntax error.

Red is a big no-no: you must not use the same idx value for even and odd fields. These have to be different. (At least with pel=2, which is default.)

Apart from that, the script is fine. ;)

[Didée]

A small demo of wrong idx usage:

Correct idx usage:
http://img255.imageshack.us/img255/9890/twoclipsvu9.th.png (http://img255.imageshack.us/my.php?image=twoclipsvu9.png)

Wrong idx usage:
http://img259.imageshack.us/img259/6923/twoclipswrongidxsk5.th.png (http://img259.imageshack.us/my.php?image=twoclipswrongidxsk5.png)


v = BlankClip(width=304,height=224, pixel_type="yuy2")
v1 = v.Subtitle("this one (v1)",y=32).AddBorders(8,8,8,8,color=$FF0000)
v2 = v.Subtitle("that one (v2)",y=64).AddBorders(8,8,8,8,color=$00FF00)

bw1 = v1.MVAnalyse(isb=true, idx=1)
fw1 = v1.MVAnalyse(isb=false, idx=1)
comp1bw = v1.MVCompensate(bw1,idx=1)
comp1fw = v1.MVCompensate(fw1,idx=1)
x1 = Interleave(comp1fw,v1,comp1bw).TemporalSoften(1,255,255,255,2).SelectEvery(3,1)

bw2 = v2.MVAnalyse(isb=true, idx=1) # idx = 1 is WRONG! (just for demonstration!)
fw2 = v2.MVAnalyse(isb=false, idx=1) # --""--
comp2bw = v2.MVCompensate(bw2,idx=1) # --""--
comp2fw = v2.MVCompensate(fw2,idx=1) # --""--
x2 = Interleave(comp2fw,v2,comp2bw).TemporalSoften(1,255,255,255,2).SelectEvery(3,1)

#StackHorizontal(v1,v2)
StackHorizontal(x1,x2)

[yup]

Didée!
Thanks for reply.
I make cut and paste from script with Russian comments.
Also please advise postfilter more suitable for pure median prefiltering.
yup.

[Didée]

What kind of postfilter do you mean now?

Your script is basically fine, you just have to use idx=2 in the 2nd part of your script.

[yup]

Didée!
Now i do not use postfilter I only search him:)
yup.

[yup]

Hi folk!
I remove error from script.
AviSource("sel.avi",pixel_type="YUY2")
AssumeTFF()
Crop(16,16,-16,-16)
SeparateFields()
e=SelectEven()
o=selectOdd()

bvo = o.MVAnalyse(isb = true, truemotion=true, delta = 1, idx = 1, overlap=4, dct=1)
fvo = o.MVAnalyse(isb = false, truemotion=true, delta = 1, idx = 1, overlap=4, dct=1)
fco = o.MVFlow(fvo, idx=1, thSCD1=500)
bco = o.MVFlow(bvo, idx=1, thSCD1=500)
interleave(fco, o, bco)
DeGrainMedian(limitY=255,limitUV=255,mode=1, norow=true)
#DeGrainMedian(limitY=255,limitUV=255,mode=1)
od=selectevery(3,1)

bve = e.MVAnalyse(isb = true, truemotion=true, delta = 1, idx = 2, overlap=4, dct=1)
fve = e.MVAnalyse(isb = false, truemotion=true, delta = 1, idx = 2, overlap=4, dct=1)
fce = e.MVFlow(fve, idx=2, thSCD1=500)
bce = e.MVFlow(bve, idx=2, thSCD1=500)
interleave(fce, e, bce)
DeGrainMedian(limitY=255,limitUV=255,mode=1, norow=true)
#DeGrainMedian(limitY=255,limitUV=255,mode=1)
ed=selectevery(3,1)

interleave(ed,od)
Weave()
AddBorders(16,16,16,16)

I totaly remove all black stripe only after 3 pass through this script my clip and clip little blured. I try use compensation from -2 frame and +2 frame positoion (DegrainMedian work on 5 frames), but result get worst. If somebode have any idea please advise.
With kind regards yup.

[yup]

Hi folk!
After more try and error. :thanks: tsp, Fizick, Manao, Mug Funky and other i find better way for remove black stripes.
First I use MVTools for prefiltering:
source=AVISource("clip.avi")
fields=source.AssumeBFF().SeparateFields()
backward_vec = fields.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1,dct=1)
forward_vec = fields.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1,dct=1)
fields.MVDegrain1(backward_vec,forward_vec,thSAD=400,idx=1)
Weave()
I use dct=1 for better accuracy motion estimation as recomended Fizick, and save this file in VirtualDub as degr1.avi. After I use ml3dex function from MedianBlur plugin, for estimation motion vector I use prefiltered source and for filtering use source.
AviSource("clip.avi")
AssumeBFF()
ConvertToYV12(interlaced=true)
SeparateFields()
source=last
AviSource("degr1.avi")
AssumeBFF()
ConvertToYV12(interlaced=true)
SeparateFields()
filtered=last
bv1 = filtered.MVAnalyse(blksize=8, isb = true, truemotion=true, delta=2, overlap=4,idx = 1,dct=1)
fv1 = filtered.MVAnalyse(blksize=8, isb = false, truemotion=true, delta=2, overlap=4,idx = 1,dct=1)
bc1 =source.MVFlow(bv1, idx=2, thSCD1=1000)
fc1 = source.MVFlow(fv1, idx=2, thSCD1=1000)
interleave(bc1, source, fc1)
ml3dex(mc=false,Y=3, U=3,V=3)
SelectEvery(3,1)
Weave()
Source clip DV YUY2, but ml3dex work only with YV12. For many parts my source enough one pass, but for worst parts I use 2 pass (for second pass I prefiltered pass1 use MVTools). Before encoding, after ml3dex need use MVDegrain1 for remove grain which bring ml3dex. Also I find that use 2 pass 1 step motion estimation (prefiltering MVDegrain1 and ml3dex) work better than use MVDEgrain2 for prefiltering and 2 call ml3dex in script.
See one field for comparing (top filtered, bottom source):
http://www.sendspace.com/file/ig8pgo
If any have idea for improvement please advice.
With kind regards yup.

[Fizick]

you get blurring on dress.
Do you try Despot plugin in seg=1 mode or median mode?
with width=big and height=1?

[yup]

Fizick! Thank You for reply.
Please precise parameter set.
I try for spot removal mode i use seg=1, pheight=1, pwidth=30 (maximum length black stripe)
for median mode
median=true, mheight=1, mwidth=30 instead ml3dex in my script. Picture less blured than ml3dex but many stripe do not removed. Please advise way for tuning other parameters.

With kind regards yup.

[Fizick]

sign = 2

[yup]

Hi folk!
Fizick I try find magic parameter set for Despot but without success.
I little modified supposed by gzarkadas script for creation spike detection index using motion compensated frames, see below
LoadPlugin("mt_masktools.dll") # v2.0a30
SetMemoryMax(256)
AVISource("seldv2.avi")
AssumeBFF()
ConvertToYV12(interlaced=true)
fields=SeparateFields()
backward_vec = fields.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1,dct=1)
forward_vec = fields.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1,dct=1)
fc = fields.MVFlow(forward_vec, idx=1, thSCD1=10000)
bc = fields.MVFlow(backward_vec, idx=1, thSCD1=10000)
masksdi=SDIPixelMask(bc, fields, fc, 16, 0.8) # define your thresholds here
StackVertical(fields,masksdi)
#Weave()


# this is for Abs(p - f) or Abs(p - b)
Function absdiff(clip c1, clip c2) { return mt_lutxy(c1, c2, "x y - abs") }

# this build the expression for the SDI function
Function SDI_RPN(string expr_d1, string expr_d2, string expr_t1, string expr_t2) {
_d1 = expr_d1 + " "
_d2 = expr_d2 + " "
_t1 = expr_t1 + " "
_t2 = expr_t2 + " "
return _d1 + _t1 + "> " + _d2 + _t1 + "> | 1.0 " \
+ _d1 + _d2 + "- " + _d1 + _d2 + "+ / abs - 0.0 ? " \
+ _t2 + "> 1.0 " + _d1 + _t1 + "> " + _d2 + _t1 + "> | 1.0 " \
+ _d1 + _d2 + "- " + _d1 + _d2 + "+ / abs - 0.0 ? "
}

# this creates the mask
Function SDIPixelMask(clip prev, clip curr, clip next, float t1, float t2) {

c_d1 = absdiff(curr, next)
c_d2 = absdiff(curr, prev)

# since SDI returns 0..1 we multiply with 255

mask = mt_lutxy(c_d1, c_d2, SDI_RPN("x", "y", String(t1), String(t2)) + " 255 *")
return mask
}
see image
http://imgplace.com/directory/dir3096/1179845621_4088.png (http://www.imgplace.com/getcode.php?img=directory/dir3096/1179845621_4088.png)
This script successfully detecting black and white lines, but also find place where exist occlusion during motion estimation. My idea very simple find occlusion mask use MVMask with kind=2 (advice how use ml parameter), for both direction, forward and backward, binarize both mask to 0 and 255 level, use logical AND for find mask occlusion in both directions, because spike detection index do not sensitive for occlusion in one direction and last use logical XOR for remove from SDI occlusion mask. Unfortunately I not familiarly with mt_masktools. Please advice.
With kind regards yup.

[yup]

Hi!
After small break I back to this problem. I upgrade hardware for capture (Panasonic DMR-ES35V DVD recorder with VCR, time base corrector Cypress CTB-100, Ixos S-Video cables), rewrite script. Script based on ideas which i find on this forum.
LoadPlugin("mt_masktools.dll")
AVISource("leb1.avi")
AssumeTFF()
ConvertToYV12(interlaced=true)
fields=SeparateFields()
snf=bob(fields,0,0.5)
f=fields.mt_luts(fields,mode="median",pixels="0 -1 0 0 0 1",U=1,V=1)
source=bob(f,0,0.5)
bv1 = source.MVAnalyse(blksize=8, isb = true, truemotion=true, delta = 1, idx = 1, overlap=4, dct=1,chroma=false)
bv2 = source.MVAnalyse(blksize=8, isb = true, truemotion=true, delta = 2, idx = 1, overlap=4, dct=1,chroma=false)
fv1 = source.MVAnalyse(blksize=8, isb = false, truemotion=true, delta = 1, idx = 1, overlap=4, dct=1,chroma=false)
fv2 = source.MVAnalyse(blksize=8, isb = false, truemotion=true, delta = 2, idx = 1, overlap=4, dct=1,chroma=false)
#bc1 =snf.MVFlow(bv1, idx=2, thSCD1=1000)
#fc1 = snf.MVFlow(fv1, idx=2, thSCD1=1000)
bc1 =snf.MVCompensate(bv1, idx=2, mode=1)
bc2 =snf.MVCompensate(bv2, idx=2, mode=1)
fc1 = snf.MVCompensate(fv1, idx=2, mode=1)
fc2 = snf.MVCompensate(fv2, idx=2, mode=1)
interleave(bc2, bc1, source, fc1, fc2)
medianblurt(radiusy=0,radiusu=0,radiusv=0,temporalradius=2)
SelectEvery(5,2)
SeparateFields()
SelectEvery(4,0,3)
StackVertical(last,fields)

source and processing image
http://img405.imageshack.us/img405/1136/blacklineremovesp3.th.png (http://img405.imageshack.us/my.php?image=blacklineremovesp3.png)
Quantity black line decrease, horizontal stability increase, that help motion compensated filtering. Small problems I decrease color space from YUY2 to YV12 and after need increase sharpness. May be I need postprocessing after median filtering before sharpening? Please advice.
With kind regards yup.

[R3Z]

Hi!
After small break I back to this problem. I upgrade hardware for capture (Panasonic DMR-ES35V DVD recorder with VCR, time base corrector Cypress CTB-100, Ixos S-Video cables), rewrite script. Script based on ideas which i find on this forum.
LoadPlugin("mt_masktools.dll")
AVISource("leb1.avi")
AssumeTFF()
ConvertToYV12(interlaced=true)
fields=SeparateFields()
snf=bob(fields,0,0.5)
f=fields.mt_luts(fields,mode="median",pixels="0 -1 0 0 0 1",U=1,V=1)
source=bob(f,0,0.5)
bv1 = source.MVAnalyse(blksize=8, isb = true, truemotion=true, delta = 1, idx = 1, overlap=4, dct=1,chroma=false)
bv2 = source.MVAnalyse(blksize=8, isb = true, truemotion=true, delta = 2, idx = 1, overlap=4, dct=1,chroma=false)
fv1 = source.MVAnalyse(blksize=8, isb = false, truemotion=true, delta = 1, idx = 1, overlap=4, dct=1,chroma=false)
fv2 = source.MVAnalyse(blksize=8, isb = false, truemotion=true, delta = 2, idx = 1, overlap=4, dct=1,chroma=false)
#bc1 =snf.MVFlow(bv1, idx=2, thSCD1=1000)
#fc1 = snf.MVFlow(fv1, idx=2, thSCD1=1000)
bc1 =snf.MVCompensate(bv1, idx=2, mode=1)
bc2 =snf.MVCompensate(bv2, idx=2, mode=1)
fc1 = snf.MVCompensate(fv1, idx=2, mode=1)
fc2 = snf.MVCompensate(fv2, idx=2, mode=1)
interleave(bc2, bc1, source, fc1, fc2)
medianblurt(radiusy=0,radiusu=0,radiusv=0,temporalradius=2)
SelectEvery(5,2)
SeparateFields()
SelectEvery(4,0,3)
StackVertical(last,fields)

source and processing image
http://img405.imageshack.us/img405/1136/blacklineremovesp3.th.png (http://img405.imageshack.us/my.php?image=blacklineremovesp3.png)
Quantity black line decrease, horizontal stability increase, that help motion compensated filtering. Small problems I decrease color space from YUY2 to YV12 and after need increase sharpness. May be I need postprocessing after median filtering before sharpening? Please advice.
With kind regards yup.


Great script yup :) works really well. I dont completely understand why though :(

Could you (or someone else) explain to me what the mt_luts function does ? I dont understand why its there !

Cheers,

R3Z

[yup]

Hi R3Z!
This Didée idea http://forum.doom9.org/showthread.php?p=1017230#post1017230
Pure median only vertical filter. All remain as usual, use prefiltering for motion vectors estimation and use nonfiltering clip for motion compensation, and last use pure median temporal filter. If you have idea for improve let me know.
yup.

[yup]

Hi all!
Change code and added masks. Thanks Manao for hacking planar Colorspace for mt_masktools ans thanks to Kassandro for plugin VerticalCleaner http://videoprocessing.11.forumer.com/viewtopic.php?t=107
LoadPlugin("mt_masktools-25.dll")
LoadPlugin("VerticalCleanerSSE2.dll")
LoadPlugin("SSE2Tools.dll")
LoadPlugin("RemoveGrainTSSE2.dll")
DirectShowSource("seldv.avi")
AssumeBFF()
fields=SeparateFields()
Interleaved2Planar(fields)
fieldsfpl=VerticalCleaner(mode=2,planar=true)
fieldsf=Planar2Interleaved()
bv1 = fieldsf.MVAnalyse(blksize=8, isb = true, truemotion=true, search=2, delta = 2, idx = 1, overlap=4, dct=1,chroma=false)
fv1 = fieldsf.MVAnalyse(blksize=8, isb = false, truemotion=true, search=2, delta = 2, idx = 1, overlap=4, dct=1,chroma=false)
bc1 =fields.MVCompensate(bv1, idx=2, mode=1, thSAD=16000)
fc1 =fields.MVCompensate(fv1, idx=2, mode=1, thSAD=16000)
mf1=Greyscale(fieldsf).MVMask(fv1,kind=1,Ysc=255)
mf1pl=Interleaved2Planar(mf1).mt_binarize(threshold=254,upper=true)
mb1=Greyscale(fieldsf).MVMask(bv1,kind=1,Ysc=255)
mb1pl=Interleaved2Planar(mb1).mt_binarize(threshold=254,upper=true)
mmaskpl=mt_logic(mf1pl,mb1pl,"max")
#clense(Interleaved2Planar(bc1), Interleaved2Planar(fields), Interleaved2Planar(fc1),increment=0)
mcf=clense(bc1, fields,fc1,increment=0)
mcfpl=Interleaved2Planar(mcf)
mcfmaskpl=mt_merge(mcfpl,fieldsfpl,mmaskpl,u=3,v=3)
#Planar2Interleaved()
#Weave()
mcfmasked=Planar2Interleaved(mcfmaskpl)
StackVertical(fields,mcfmasked)
See picture
http://img338.imageshack.us/img338/7571/girlbz7.th.png (http://img338.imageshack.us/img338/7571/girlbz7.th.png)
Any advice for impovement. Now script work in YUY2 and will be usefull for analog capture (huffyuv or DV) without colorspace conversion.
yup.

[2Bdecided]

yup,

I think your script is very clever, and tries to solve a near-impossible problem. Sadly, the script also seems to change the "good" image content quite a lot.

I have the same problem on some of my tapes, so I have been looking for a solution. Reading the videohelp.com forums, the most likely cause is worn heads on the playback machine, the second most likely cause is worn heads on the original record machine, and the third most likely cause is dirty heads on either machine.

So, first thing you can try is a head cleaner. The second thing is playing back the tape with the original record VCR (if available). The third thing is to try any other VCR - it doesn't have to be a good one - some times really bad (cheap or worn) VCRs are good at solving the problem (though make sure they don't break your precious tapes).


If the solution must be within AVIsynth, I have two suggestions (based on what I see on my tapes - yours may be different).

1. The black lines appear in slightly different positions each time I capture the same tape on the same VCR. Maybe you can capture the tape two or three times, and then keep the maximum value for each pixel across the three captures?

2. The black lines appear in slightly predictable places - often near a dark/light or light/dark luma border - maybe only apply any filtering in such areas? (This sounds very difficult to me).

Hope this is some help. If I find a reasonable solution from any of this, I'll post back.

I never dreamt capturing a few old tapes would cause me so many problems. I guess the good news is that, at least for now, they still play back surprisingly well. If we wait another ten years, the tapes may decay, and there will be very few decks left to play them back on!

Cheers,
David.

[yup]

@2Bdecided!
1. The black lines appear in slightly different positions each time I capture the same tape on the same VCR. Maybe you can capture the tape two or three times, and then keep the maximum value for each pixel across the three captures?
Yes, I try it.
http://videoprocessing.11.forumer.com/viewtopic.php?t=109
This only first stage.
Problem only with dubbed VHS, frequently the different capture not sync.
yup.

[2Bdecided]

By "in sync" do you mean frame loss, or movement within/of frame?

that should be an easier thing to correct than all the other problems you face, though I don't know how to do it in AVIsynth.

In general, assuming vertical (i.e. line numbers) are correct, I would simply convolve (maybe I mean cross correlate?) the signals horizontally, and the peak in the convolution result shows the offset between the two signals. This allows you to match them both.

Cheers,
David.

[yup]

Hi all!
Now mask for bad motion estimation could work
LoadPlugin("mt_masktools-25.dll")
LoadPlugin("VerticalCleanerSSE2.dll")
LoadPlugin("SSE2Tools.dll")
LoadPlugin("RemoveGrainTSSE2.dll")
DirectShowSource("seldv.avi")
#AVISource("flower.avi")
AssumeBFF()
ConvertToYV12(interlaced=true)
fields=SeparateFields()
fieldsf=fields.VerticalCleaner(mode=2)
bv1 = fieldsf.MVAnalyse(blksize=8, isb = true, truemotion=true, search=2, delta = 2, idx = 1, overlap=4, dct=0,chroma=false)
fv1 = fieldsf.MVAnalyse(blksize=8, isb = false, truemotion=true, search=2, delta = 2, idx = 1, overlap=4, dct=0,chroma=false)
bc1 =fields.MVCompensate(bv1, idx=2, mode=1, thSAD=16000)
fc1 =fields.MVCompensate(fv1, idx=2, mode=1, thSAD=16000)
mf1=fieldsf.MVMask(fv1,kind=2,ml=30,Ysc=255)
mb1=fieldsf.MVMask(bv1,kind=2,ml=30,Ysc=255)
mmask=mt_logic(mf1,mb1,"max",u=-128,v=-128)
mcf=clense(bc1, fields,fc1,increment=0)
mcfmask=mt_merge(mcf,fieldsf,mmask,luma=true)
#Weave()
#StackVertical(fields,mcfmask)
Weave(mcfmask)
ConvertToYUY2(interlaced=true)
LetterBox(16,16,16,16)

But now only in YV12 colorspace. This work better than for kind=1 MVMask. Next stage preserve detail in source video.
yup.

[Didée]

This work better than for kind=1 MVMask.

Really? If that's your experience, fine. My experience is that MVMask(kind=2) is WAY less reliable to detect bad motion.

Random examples:

http://img371.imageshack.us/img371/6803/mvmaskstuffath7.th.jpg (http://img371.imageshack.us/my.php?image=mvmaskstuffath7.jpg) - http://img370.imageshack.us/img370/9853/mvmaskstuffbny8.th.jpg (http://img370.imageshack.us/my.php?image=mvmaskstuffbny8.jpg)

Left-top: source
Left-bottom: compensation (delta=2)
Middle-top: your mask, exactly the code from above (max-merging forward+backward kind=2 mask)
middle-bottom: the usual MVMask(kind=1)
right-top: compensation corrected by your mask
right-bottom: compensation corrected by MVMask(kind=1)

That's the typical kind of "performance" I'm used to get from kind=2. For me it marks too much areas "bad" that infact would be usable, still it often does not detect areas that are really unusable.

[yup]

Didée!:thanks:
For my source with pulse noise, if using kind=1 I get only Verticalcleaner output i. e. motion compensated filter practicaly do not work. May be main reason for this problem output median prefilter, which segregate picture and move border. kind=2 give less restriction. May be use prefilter for source now I try Deblock:
LoadPlugin("mt_masktools-25.dll")
LoadPlugin("VerticalCleanerS.dll")
LoadPlugin("RemoveGrainT.dll")
AVISource("selnew.avi")
AssumeTFF()
ConvertToYV12(interlaced=true)
fields=SeparateFields()
fieldsf=fields.VerticalCleaner(mode=2)
fieldsf=fieldsf.Deblock(quant=25)
bv1f = fieldsf.MVAnalyse(blksize=8, isb = true, truemotion=true, search=2, delta = 2, idx = 1, overlap=4, dct=0,chroma=false)
fv1f = fieldsf.MVAnalyse(blksize=8, isb = false, truemotion=true, search=2, delta = 2, idx = 1, overlap=4, dct=0,chroma=false)
bc1 =fields.MVCompensate(bv1f, idx=2, mode=1, thSAD=16000)
fc1 =fields.MVCompensate(fv1f, idx=2, mode=1, thSAD=16000)
mf1f=fieldsf.MVMask(fv1f,kind=2,ml=100,Ysc=255)
mb1f=fieldsf.MVMask(bv1f,kind=2,ml=100,Ysc=255)
mmaskf=mt_logic(mf1f,mb1f,"max")
mmmaskf=mmaskf.Deblock(quant=25)
mcf=clense(bc1, fields,fc1,increment=0)
mcfmask=mt_merge(mcf,fieldsf,mmaskf,luma=true)
#
#try preserve good motion compensated part of frame
fieldspre=fields.Deblock(quant=25)
bv1 = fieldspre.MVAnalyse(blksize=8, isb = true, truemotion=true, search=2, delta = 2, idx = 3, overlap=4, dct=0,chroma=false)
fv1 = fieldspre.MVAnalyse(blksize=8, isb = false, truemotion=true, search=2, delta = 2, idx = 3, overlap=4, dct=0,chroma=false)
mf1=fieldspre.MVMask(fv1,kind=1,ml=15,Ysc=255)
mb1=fieldspre.MVMask(bv1,kind=1,ml=15,Ysc=255)
mmask=mt_logic(mf1,mb1,"max")
mmask=mt_invert(mmask,U=-128,V=-128)
mmask=mmask.Deblock(quant=25)
mcfmasktot=mt_merge(mcfmask,fields,mmask,luma=true)
StackVertical(fields,mcfmasktot)

yup.

[yup]

Hi all!
Folowing script:
LoadPlugin("mt_masktools-25.dll")
LoadPlugin("VerticalCleanerSSE2.dll")
LoadPlugin("SSE2Tools.dll")
LoadPlugin("RemoveGrainTSSE2.dll")
LoadPlugin("RemoveGrainHDS.dll")
AVISource("flower.avi")
#DirectShowSource("seldv.avi")
AssumeTFF()
fields=SeparateFields()
fieldspl=Interleaved2Planar(fields)
fieldsfpl=fieldspl.VerticalCleaner(mode=2,planar=true)
fieldsf2pl=Quantile(fieldspl,planar=true)
fieldsf=Planar2Interleaved(fieldsfpl)
bv1 = fieldsf.MVAnalyse(blksize=8, isb = true, truemotion=true, search=3, delta = 2, idx = 1, overlap=4, dct=0,chroma=false)
bv2 = fieldsf.MVAnalyse(blksize=8, isb = true, truemotion=true, search=3, delta = 4, idx = 1, overlap=4, dct=0,chroma=false)
fv1 = fieldsf.MVAnalyse(blksize=8, isb = false, truemotion=true, search=3, delta = 2, idx = 1, overlap=4, dct=0,chroma=false)
fv2 = fieldsf.MVAnalyse(blksize=8, isb = false, truemotion=true, search=3, delta = 4, idx = 1, overlap=4, dct=0,chroma=false)
bc1 =fields.MVCompensate(bv1, idx=2, mode=1, thSAD=16000)
bc2 =fields.MVCompensate(bv2, idx=2, mode=1, thSAD=16000)
fc1 =fields.MVCompensate(fv1, idx=2, mode=1, thSAD=16000)
fc2 =fields.MVCompensate(fv2, idx=2, mode=1, thSAD=16000)
mlth=100
mf1=fieldsf.MVMask(fv1,kind=1,ml=mlth,Ysc=255)
mf1pl=Interleaved2Planar(mf1)
mf2=fieldsf.MVMask(fv2,kind=1,ml=mlth,Ysc=255)
mf2pl=Interleaved2Planar(mf2)
mb1=fieldsf.MVMask(bv1,kind=1,ml=mlth,Ysc=255)
mb1pl=Interleaved2Planar(mb1)
mb2=fieldsf.MVMask(bv2,kind=1,ml=mlth,Ysc=255)
mb2pl=Interleaved2Planar(mb2)
maskcenter=mt_invert(mt_logic(mf1pl,mb1pl,"max"))
maskbw=mt_invert(mt_logic(mb1pl,mb2pl,"max"))
maskfw=mt_invert(mt_logic(mf1pl,mf2pl,"max"))
mcfcenter=clense(bc1, fields,fc1,increment=0)
mcfcenterpl=Interleaved2Planar(mcfcenter)
mcfbw=clense(bc1, fields,bc2,increment=0)
mcfbwpl=Interleaved2Planar(mcfbw)
mcffw=clense(fc1, fields,fc2,increment=0)
mcffwpl=Interleaved2Planar(mcffw)
mcfmaskedpl=mt_merge(fieldsfpl,mcfbwpl,maskbw)
mcfmaskedpl=mt_merge(mcfmaskedpl,mcffwpl,maskfw)
mcfmaskedpl=mt_merge(mcfmaskedpl,mcfcenterpl,maskcenter)
#Weave()
mcfmasked=Planar2Interleaved(mcfmaskedpl)
StackVertical(Greyscale(fields),Greyscale(mcfmasked))

give better result than all my previous try. Former I try remove luma and chroma noise based on only luma value, not very good approach. Also I could be increase search parameter MVTools to 3 for better motion estimation. Change to search=3 more effective than decrease thSAD. Small step done.
:thanks: to all for advice and support.
yup.

[yup]

Hi all!
After some try and error i find script which practicaly remove 1 pixel thickness black and white lines
LoadPlugin("mt_masktools-25.dll")
LoadPlugin("VerticalCleanerSSE3.dll")
LoadPlugin("SSE3Tools.dll")
LoadPlugin("RemoveGrainTSSE3.dll")
LoadPlugin("RemoveGrainSSE3.dll")
AVISource("melted3.avi")
source=AssumeTFF()
ConvertToYV12(source,interlaced=true)
fields=SeparateFields()
fieldsf2=fields.VerticalCleaner(mode=2)
fieldsf=fieldsf2.RemoveGrain(11)
bv1 = fieldsf.MVAnalyse(blksize=8, isb = true, truemotion=true, search=2, delta = 2, idx = 1, overlap=4, dct=0,chroma=false)
bv2 = fieldsf.MVAnalyse(blksize=8, isb = true, truemotion=true, search=2, delta = 4, idx = 1, overlap=4, dct=0,chroma=false)
fv1 = fieldsf.MVAnalyse(blksize=8, isb = false, truemotion=true, search=2, delta = 2, idx = 1, overlap=4, dct=0,chroma=false)
fv2 = fieldsf.MVAnalyse(blksize=8, isb = false, truemotion=true, search=2, delta = 4, idx = 1, overlap=4, dct=0,chroma=false)
bc1 =fields.MVCompensate(bv1, idx=2, mode=1, thSAD=16000)
bc2 =fields.MVCompensate(bv2, idx=2, mode=1, thSAD=16000)
fc1 =fields.MVCompensate(fv1, idx=2, mode=1, thSAD=16000)
fc2 =fields.MVCompensate(fv2, idx=2, mode=1, thSAD=16000)
mf1=fieldsf.MVMask(fv1,kind=1,ml=100,Ysc=255)
mf2=fieldsf.MVMask(fv2,kind=1,ml=100,Ysc=255)
mb1=fieldsf.MVMask(bv1,kind=1,ml=100,Ysc=255)
mb2=fieldsf.MVMask(bv2,kind=1,ml=100,Ysc=255)
maskcenter=mt_invert(mt_logic(mf1,mb1,"max"))
maskbw=mt_invert(mt_logic(mb1,mb2,"max"))
maskfw=mt_invert(mt_logic(mf1,mf2,"max"))
mcfcenter=clense(bc1, fields,fc1,increment=0)
mcfbw=clense(bc1, fields,bc2,increment=0)
mcffw=clense(fc1, fields,fc2,increment=0)
mcfmasked=mt_merge(fieldsf2,mcfbw,maskbw)
mcfmasked=mt_merge(mcfmasked,mcffw,maskfw)
mcfmasked=mt_merge(mcfmasked,mcfcenter,maskcenter)
Weave(mcfmasked)
convertToYUY2()
MergeChroma(Source)

Very interesting thing MVAnalyse better work with dct=0 than with dct=1. Also this script can use for analog capture with large grain noise and use instead VerticalCleaner RemoveGrain(1) only as first stage denoising. Second stage MVDegrainX.
yup.

[yup]

Hi all!
Time by time I try back to solving this problem. :thanks: to plugin writer which constantly expand feature especially tritical.
New version black line remover
SetMemoryMax(768)
LoadPlugin("mt_masktools-25.dll")
LoadPlugin("RemoveGrainTSSE3.dll")
SetMTMode(5)
source=DirectShowSource("cap1.avi")
SetMTMode(2)
source=source.Trim(1019,0).AssumeTFF().ConvertToYV12(interlaced=true)
bobnn=source.eedi3(sclip=nnedi3(source,field=-2,nsize=3,qual=2,U=false, V=false),field=-2,vcheck=3,U=false, V=false)
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels="0 -3 0 -2 0 -1 0 0 0 1 0 2 0 3",U=2,V=2)
#bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels="0 -2 0 -1 0 0 0 1 0 2",U=2,V=2)
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false, V=false)
().SelectEvery(4,0,3)
super=MSuper(bobnn)
superf=MSuper(bobnnmedf)
bv1 = MAnalyse(superf, blksize=16, isb = true, delta = 2, overlap=8, dct=5,chroma=false)
bv2 = MAnalyse(superf, blksize=16, isb = true, delta = 4, overlap=8, dct=5,chroma=false)
fv1 = MAnalyse(superf, blksize=16, isb = false,delta = 2, overlap=8, dct=5,chroma=false)
fv2 = MAnalyse(superf, blksize=16, isb = false,delta = 4, overlap=8, dct=5,chroma=false)
bc1 = MCompensate(bobnn, super, bv1, thSAD=16000)
bc2 = MCompensate(bobnn, super, bv2, thSAD=16000)
fc1 = MCompensate(bobnn, super, fv1, thSAD=16000)
fc2 = MCompensate(bobnn, super, fv2, thSAD=16000)
mf1=bobnnmedf.MMask(fv1,kind=1,ml=100,Ysc=255)
mf2=bobnnmedf.MMask(fv2,kind=1,ml=100,Ysc=255)
mb1=bobnnmedf.MMask(bv1,kind=1,ml=100,Ysc=255)
mb2=bobnnmedf.MMask(bv2,kind=1,ml=100,Ysc=255)
maskcenter=mt_invert(mt_logic(mf1,mb1,"max",U=2,V=2))
maskbw=mt_invert(mt_logic(mb1,mb2,"max",U=2,V=2))
maskfw=mt_invert(mt_logic(mf1,mf2,"max",U=2,V=2))
mcfcenter=clense(bc1, bobnn,fc1,increment=0)
mcfbw=clense(bc1, bobnn,bc2,increment=0)
mcffw=clense(fc1, bobnn,fc2,increment=0)
mcfmasked=mt_merge(bobnnmed,mcfbw,maskbw,U=2,V=2)
mcfmasked=mt_merge(mcfmasked,mcffw,maskfw,U=2,V=2)
mcfmasked=mt_merge(mcfmasked,mcfcenter,maskcenter,U=2,V=2)

fieldf=mcfmasked.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#StackVertical(Separatefields(source),fieldf.MergeChroma(Separatefields(source)))
Weave(fieldf.MergeChroma(Separatefields(source)))

eedi3 work more better especially at artificial objects with straight borders (windows, wall and fence). Also post-filtering using dfttest give better motion search, because after using median vertical filter picture very segregated. I am almost satisfied result, but want change priority for use different temporal median filtered frames (mcfbf,
mcff and mcfcenter) and need help
http://forum.doom9.org/showthread.php?p=1445155#post1445155
yup.

[yup]

I try write code using priority based on minimum SAD
SetMemoryMax(768)
LoadPlugin("RemoveGrainTSSE2.dll")
source=AVISource("sel97.avi")
source=source.AssumeTFF().ConvertToYV12(interlaced=true)
bobnn=source.nnedi3(field=-2)
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels="0 -3 0 -2 0 -1 0 0 0 1 0 2 0 3",U=2,V=2)
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=16,sosize=12,sigma=1000,U=false, V=false)
super=MSuper(bobnn)
superf=MSuper(bobnnmedf)
bv1 = MAnalyse(superf, blksize=16, isb = true, delta = 2, overlap=8, dct=5,chroma=false)
bv2 = MAnalyse(superf, blksize=16, isb = true, delta = 4, overlap=8, dct=5,chroma=false)
fv1 = MAnalyse(superf, blksize=16, isb = false,delta = 2, overlap=8, dct=5,chroma=false)
fv2 = MAnalyse(superf, blksize=16, isb = false,delta = 4, overlap=8, dct=5,chroma=false)
bc1 = MCompensate(bobnn, super, bv1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnn, super, bv2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnn, super, fv1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnn, super, fv2, thSAD=16000, thSCD1=16000)
mf1=bobnnmedf.MMask(fv1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2=bobnnmedf.MMask(fv2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb1=bobnnmedf.MMask(bv1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2=bobnnmedf.MMask(bv2,kind=1,ml=100,Ysc=255, thSCD1=16000)
maskcenter=mt_logic(mf1,mb1,"max",U=2,V=2)
maskbw=mt_logic(mb1,mb2,"max",U=2,V=2)
maskfw=mt_logic(mf1,mf2,"max",U=2,V=2)
mcfcenter=clense(bc1, bobnn,fc1,increment=0)
mcfbw=clense(bc1, bobnn,bc2,increment=0)
mcffw=clense(fc1, bobnn,fc2,increment=0)
#
output=bobnnmed
# center filtered
mt_lutxyz(maskcenter,maskbw,maskfw,"x y <= x z <= & x 255 < &")
output=mt_lutxyz(last,mcfcenter,output,"x y z ?")
# backward filtered
mt_lutxyz(maskbw,maskcenter,maskfw,"x y < x z < &")
output=mt_lutxyz(last,mcfbw,output,"x y z ?")
# forward filtered
mt_lutxyz(maskfw,maskcenter,maskbw,"x y < x z < &")
output=mt_lutxyz(last,mcffw,output,"x y z ?")

fieldf=output.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
StackVertical(Separatefields(source),fieldf.MergeChroma(Separatefields(source)))

frame example
http://img69.imageshack.us/img69/9223/removeline.th.png (http://img69.imageshack.us/i/removeline.png/) Uploaded with ImageShack.us (http://imageshack.us)
Any suggestion welcome.
yup.

[yup]

Hi all!
I little change condition for SAD and change hard threshold to soft.

source=AVISource("selnew.avi")
source=source.AssumeTFF().ConvertToYV12(interlaced=true)
bobnn=source.nnedi3(field=-2,U=false, V=false)
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels="0 -3 0 -2 0 -1 0 0 0 1 0 2 0 3",U=2,V=2)
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=16,sosize=12,sigma=1000,U=false, V=false)
super=MSuper(bobnn,chroma=false)
superf=MSuper(bobnnmedf,chroma=false)
bv1 = MAnalyse(superf, blksize=16, isb = true, delta = 2, overlap=8, dct=5,chroma=false)
bv2 = MAnalyse(superf, blksize=16, isb = true, delta = 4, overlap=8, dct=5,chroma=false)
fv1 = MAnalyse(superf, blksize=16, isb = false,delta = 2, overlap=8, dct=5,chroma=false)
fv2 = MAnalyse(superf, blksize=16, isb = false,delta = 4, overlap=8, dct=5,chroma=false)
bc1 = MCompensate(bobnn, super, bv1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnn, super, bv2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnn, super, fv1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnn, super, fv2, thSAD=16000, thSCD1=16000)
mb1sad=bobnnmedf.MMask(bv1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bv2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fv1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fv2,kind=1,ml=100,Ysc=255, thSCD1=16000)
centersad=mt_logic(mf1sad,mb1sad,"max",U=2,V=2)
bwsad=mt_logic(mb1sad,mb2sad,"max",U=2,V=2)
fwsad=mt_logic(mf1sad,mf2sad,"max",U=2,V=2)
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=true)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=true)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=true)
# center filtered
mt_lutxyz(centersad,bwsad,fwsad,"x y <= x z <= & x 255 < &",U=2,V=2)
maskcenter=mt_lutxy(last,centersad,"x 255 y - 0 ?",U=2,V=2)
# backward filtered
mt_lutxyz(bwsad,centersad,fwsad,"x y < x z <= &",U=2,V=2)
maskbw=mt_lutxy(last,bwsad,"x 255 y - 0 ?",U=2,V=2)
# forward filtered
mt_lutxyz(fwsad,centersad,bwsad,"x y < x z < &",U=2,V=2)
maskfw=mt_lutxy(last,fwsad,"x 255 y - 0 ?",U=2,V=2)
#
mcfmasked=mt_merge(bobnnmed,mcfbw,maskbw,U=2,V=2)
mcfmasked=mt_merge(mcfmasked,mcffw,maskfw,U=2,V=2)
mcfmasked=mt_merge(mcfmasked,mcfcenter,maskcenter,U=2,V=2)


fieldf=mcfmasked.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#StackVertical(Separatefields(source),fieldf.MergeChroma(Separatefields(source)))
Weave(fieldf.MergeChroma(Separatefields(source)))
Any advice for improvement.
yup.

[Hagbard23]

Sorry for interfering here, but i would simply use the

DESCRATCH() (http://avisynth.org.ru/descratch/descratch.html)

Plugin and set it up for black lines which are horizontal (90 degrees). This seems like a perfect scenario for that plugin. I definitely would not use Despot nor mask operations (a little too much, for my taste)

Sorry - just my 2 cents...

[yup]

Hi Hagbard23!
I try descratch before without success.
See comparing
http://img814.imageshack.us/img814/9349/compare.th.png (http://img814.imageshack.us/i/compare.png/)

Uploaded with ImageShack.us (http://imageshack.us)
At top source, medium median motion compensated filtered, at bottom descratch output with suggested set for VHS.
yup.

[Hagbard23]

i still believe, a little tweaking should get you nicer results. Maybe flipping the image 90Degrees before feeding to descratch or something like that.

I KNOW that it works quite well, because i used it some couple of weeks ago with strongly scratched black/white material. The Results were very good.

sorry - i do not want to disturb here - do it like you want to do it... ;)

[StainlessS]

Have not really looked at your prob, but you might want to take a look here:-

http://forum.doom9.org/showthread.php?t=151426&highlight=depulse

About horizontal lines on analog captures & Depulse plugin to fix such.


EDIT:- Did not look at your prob as Depulse temporal in nature and only stills
supplied (I think). Also, In "New Plugins Sticky", "KillPulse" is mentioned and
seem to do the same thing, however could not locate a link that works for
this. There is also a plugin called "DepulseC" which I think I saw somewhere
does similar with coloured streaks. Link to that, below (which does work):

http://www5e.biglobe.ne.jp/~hoe/dv-and-movie/tipsxxx/index.html

[jmac698]

I have the same problem. Your script didn't work well for me, however clense with DePulse was great. Median of 5 captures was even better. By sync he means dropped frames in the captures. Virtualdub will drop frames by default without any indication. I developed Glitch Analyzer (see new plugins) to investigate this and find settings which work. There is also each line offset horizontally. I'm writing a plugin like 2Bedecided mentioned, except I think a direct minimum SAD is easier.
The problem isn't dirty heads. Tapes play fine in the same machine and also the tape itself only developed the problem after 15 years. The black lines are a different problem, and I can recreate that with test signals to see what causes them. Black to white transitions, for example. I've made a dramatic improvement already.

[yup]

jmac698
This script work fine for me. Need small tuning. Vertical median filter could remove horizontal lines. If line thicker need use more point for median filter (will introduce artifacts) if thiner less point. Now I think about building mask for bad part frame.
Also can use hierarchical approach with different vertical median filter.
I try like You median5 for my capture without success.
I have relatively new VCR/DVD writer 2006 year model.
It is very good that this problem interesting not only for me.
yup.

[jmac698]

yup,
Would you like to try your script on another example? Here is mine:
http://www.sendspace.com/file/l1500b

[yup]

jmac698
http://img513.imageshack.us/img513/5971/filt.th.png (http://img513.imageshack.us/i/filt.png/)
LoadPlugin("mt_masktools-25.dll")
LoadPlugin("RemoveGrainTSSE3.dll")
source=AVISource("stripes demo.avi")
source=source.AssumeTFF().ConvertToYV12(interlaced=true)
#med=source.mt_luts(source,mode="median",pixels="0 -4 0 -3 0 -2 0 -1 0 0 0 1 0 2 0 3 0 4",U=2,V=2)
med=source.mt_luts(source,mode="median",pixels="0 -3 0 -2 0 -1 0 0 0 1 0 2 0 3",U=2,V=2)
#med=source.mt_luts(source,mode="median",pixels="0 -2 0 -1 0 0 0 1 0 2",U=2,V=2)
medf=med.dfttest(tbsize=1,ftype=1,sbsize=16,sosize=12,sigma=1000,U=false, V=false)
super=MSuper(source,chroma=false)
superf=MSuper(medf,chroma=false)
bv1 = MAnalyse(superf, blksize=16, isb = true, delta = 1, overlap=8, dct=5,chroma=false)
bv2 = MAnalyse(superf, blksize=16, isb = true, delta = 2, overlap=8, dct=5,chroma=false)
fv1 = MAnalyse(superf, blksize=16, isb = false,delta = 1, overlap=8, dct=5,chroma=false)
fv2 = MAnalyse(superf, blksize=16, isb = false,delta = 2, overlap=8, dct=5,chroma=false)
bc1 = MCompensate(source, super, bv1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(source, super, bv2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(source, super, fv1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(source, super, fv2, thSAD=16000, thSCD1=16000)
mb1sad=medf.MMask(bv1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=medf.MMask(bv2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=medf.MMask(fv1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=medf.MMask(fv2,kind=1,ml=100,Ysc=255, thSCD1=16000)
centersad=mt_logic(mf1sad,mb1sad,"max",U=2,V=2)
bwsad=mt_logic(mb1sad,mb2sad,"max",U=2,V=2)
fwsad=mt_logic(mf1sad,mf2sad,"max",U=2,V=2)
mcfcenter=clense(bc1, source,fc1,increment=0, grey=true)
mcfbw=clense(bc1, source,bc2,increment=0, grey=true)
mcffw=clense(fc1, source,fc2,increment=0, grey=true)
# center filtered
#mt_lutxyz(centersad,bwsad,fwsad,"x y <= x z <= & x 255 < &",U=2,V=2) replace to lutxy
mt_logic(mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and"),mt_lut(centersad,"x 255 <="),"and")
maskcenter=mt_lutxy(last,centersad,"x 255 y - 0 ?",U=2,V=2)
# backward filtered
#mt_lutxyz(bwsad,centersad,fwsad,"x y < x z <= &",U=2,V=2) replace to lutxy
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbw=mt_lutxy(last,bwsad,"x 255 y - 0 ?",U=2,V=2)
# forward filtered
#mt_lutxyz(fwsad,centersad,bwsad,"x y < x z < &",U=2,V=2) replace to lutxy
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfw=mt_lutxy(last,fwsad,"x 255 y - 0 ?",U=2,V=2)
#
mcfmasked=mt_merge(med,mcfbw,maskbw,U=2,V=2)
mcfmasked=mt_merge(mcfmasked,mcffw,maskfw,U=2,V=2)
mcfmasked=mt_merge(mcfmasked,mcfcenter,maskcenter,U=2,V=2)

StackVertical(source,mcfmasked.MergeChroma(source))
I applied script for median5 filtered source.
Your source very bad quality (my also bad but better:))
yup.

[jmac698]

Ok, I try it now. It does remove a lot of noise. If you also enable my depulse depulse(80,79,0)
source=last it's even a little better. Even some black streaks are gone. However it is more blurry. I will have to try some more video. Does it help with rainbows that seem to appear at the start of a new recording?
thanks.

[yup]

@jmac698!
Last version script. Little comments.
SetMemoryMax(768)
SetMTMode(3,4)
source=DirectShowSource("lecture1.avi",pixel_type="YUY2")#.Trim(750,1250)
SetMTMode(2,4)
source=source.AssumeTFF().ConvertToYV12(interlaced=true)#.ColorYUV(autogain=true, autowhite=true)
bobnn=source.nnedi3(field=-2)
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,5),U=3,V=3)# rectangle 5 for 3 pixels thikness line, 3 for 2, 1 for 1
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
bobnnf=bobnn#.dfttest(tbsize=1,ftype=1,sbsize=16,sosize=12,sigma=1000,U=false,V=false)
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
superff=MSuper(bobnnf)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnnf, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnnf, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnnf, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnnf, super, fw2, thSAD=16000, thSCD1=16000)
bwabs1=mt_lutxy(bobnnf,bc1,"x y - abs")#,U=-128,V=-128)
bwabs2=mt_lutxy(bobnnf,bc2,"x y - abs")#,U=-128,V=-128)
fwabs1=mt_lutxy(bobnnf,fc1,"x y - abs")#,U=-128,V=-128)
fwabs2=mt_lutxy(bobnnf,fc2,"x y - abs")#,U=-128,V=-128)
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
#
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
bwsad=mt_logic(mb1sad,mb2sad,"max")#,U=2,V=2)
fwsad=mt_logic(mf1sad,mf2sad,"max")#,U=2,V=2)
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
#
# sort SAD
# center filtered
mt_logic(mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and"),mt_lut(centersad,"x 255 <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?",U=3,V=3)
# backward filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?",U=3,V=3)
# forward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?",U=3,V=3)
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
#end sort SAD
#

#
# sort SDI
# center SDI
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# backward SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# forward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
StackVertical(Separatefields(source),fieldmaskedsdi)
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source)))
#Weave(fieldmaskedsdi)


# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
threshsp=10
THP=string(threshsp)
threshavg=2
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
SDIAvgblk=SDIavg.PointResize(90,72)
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
SDIavg=SDIAvgblk.PointResize(720,576)
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4))# rectangle 4 for 3 pixels thikness line, 2 for 2, not need for 1
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(4,0))# remove line shorter 5 pixels
SDImaskleft=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 -1 0 -2 0 -3 0 -4 0 -5 0 -6 0 -7 0 -8 0 -9 0 -10 0 -11 0")# try
SDImaskright=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 11 0") # join
SDIMask=mt_logic(SDImaskleft,SDImaskright,"min") # sequental horizontal line
SDImask=SDImask.mt_luts(SDImask,mode="min",pixels=mt_rectangle(18,0)) # remove line shorter 37 pixels
SDImask=SDImask.HorlLinExp() # fill line with noise
SDIad=mt_logic(SDIad,SDImask,"min") # refine source SDI
SDIad=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4)) # rectangle 4 for 3 pixels thikness line, 2 for 2, not need for 1
return(SDIad)

}

# fill full horizontal line if exist one pixel per line

function HorlLinExp(clip mask1) {

mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}

Now my main computer busy filtering multipass this script wit avs2avi. 2 pass wit 5 median vertical filter and 2 pass with 3., total time near 160 hour for 68 min clip. Screenshot will be at friday. Also I think about repairing using filtered and source clip.
Any advice welcome.
yup.

[yup]

Please!
Screenshot.
http://thumbnails36.imagebam.com/15398/af6df3153976503.jpg (http://www.imagebam.com/image/af6df3153976503) http://thumbnails34.imagebam.com/15398/1c91b0153976540.jpg (http://www.imagebam.com/image/1c91b0153976540)
Pictures from 4 and 5 sequential fields with horizontal lines.
yup.

[hansima99]

hi yup!

I tried tu use your script but I always get the error

"Script error: Invalid arguments to function "clense""

line 97 and line 23

I tried several filters but nothing worked. which versions of plugins are you useing

thats my project, try to get rid of the black stripes on the left

http://i52.tinypic.com/wkpser.jpg

[yup]

hansima99!
welcome to forum!
Need Kassandro plugin
http://www.removegrain.de.tf/
yup.

[Mounir]

Try THIS (http://avisynth.org/mediawiki/DeVCR) hansima99

[fonzzie31]

Hello,

I got some black horizontal stripes/lines on Laserdiscs digitizations I made.
The video format is : 720x576 PAL 25fps or 720x480 NTSC 29.97 fps - Enterlaced - YUV 10 Bits 4:2:2 Uncompressed.
The laserdisc players I used : PIONEER CLD-D925 PAL/NTSC and PIONEER Elite CLD-97 NTSC.

Here are some captures of the black stripes/lines I would like to remove :

http://revival03.free.fr/temp/blines1.jpg

http://revival03.free.fr/temp/blines2.jpg

http://revival03.free.fr/temp/blines3.jpg

These black strips/lines can be visible between half a frame to one complete frame.

I tried to use the Yup script, but I was unable to run it. I got an error message on very small red letters and It was impossible to read it. I'm not an Avisynth expert.

So I would like to know if the Yup script can help to remove the black stripes/lines, and how to use it without any problem.

I would like to obtain an enterlaced video, because the video source is enterlaced too.

Thanks beforehand for your help.

[yup]

fonzzie31!
Welcome to forum.
You need some plugin
masktools
http://forum.doom9.org/showthread.php?t=98985
MVTools2
http://www.avisynth.org.ru/mvtools/mvtools2.html
Kassandro plugin Removegrain RemovegrainT
only AVSRecursion (need install to Windows\system32 directory)
http://videoprocessing.11.forumer.com/
Tritical nnedi3
http://web.missouri.edu/~kes25c/
All plugin place to plugin directory Avisynth (for example C:\Program Files\Avisynth\plugin)
Try this. Do nor worry speak and we try answer.
Load avs script to VirtualDub
http://www.virtualdub.org/
and see error message.
yup.

[fonzzie31]

Hello Yup,

Thanks for your message...
I tried your script and the black lines/stripes were not removed.
Here is a screen capture of what I got :

http://revival03.free.fr/temp/blines4.jpg

Why don't I have a 720x576 single frame in input/output ?

As You see the black lines are still here... Is there something wrong with this script ? Perhaps am I wrong too :)

I've installed all the plugins in my avisynth plugin directory.
Then I copy & paste your script in a text file names "test.avs".
I replace one line on your script (in red) :

SetMemoryMax(768)
SetMTMode(3,4)
source=AVISource("test3.avi",pixel_type="YUY2")#.Trim(750,1250)
SetMTMode(2,4)
source=source.AssumeTFF().ConvertToYV12(interlaced=true)#.ColorYUV(autogain=true, autowhite=true)
bobnn=source.nnedi3(field=-2)
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,5),U=3,V=3)# rectangle 5 for 3

pixels thikness line, 3 for 2, 1 for 1
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
bobnnf=bobnn#.dfttest(tbsize=1,ftype=1,sbsize=16,sosize=12,sigma=1000,U=false,V=false)
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
superff=MSuper(bobnnf)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnnf, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnnf, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnnf, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnnf, super, fw2, thSAD=16000, thSCD1=16000)
bwabs1=mt_lutxy(bobnnf,bc1,"x y - abs")#,U=-128,V=-128)
bwabs2=mt_lutxy(bobnnf,bc2,"x y - abs")#,U=-128,V=-128)
fwabs1=mt_lutxy(bobnnf,fc1,"x y - abs")#,U=-128,V=-128)
fwabs2=mt_lutxy(bobnnf,fc2,"x y - abs")#,U=-128,V=-128)
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
#
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
bwsad=mt_logic(mb1sad,mb2sad,"max")#,U=2,V=2)
fwsad=mt_logic(mf1sad,mf2sad,"max")#,U=2,V=2)
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
#
# sort SAD
# center filtered
mt_logic(mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y

<="),"and"),mt_lut(centersad,"x 255 <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?",U=3,V=3)
# backward filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?",U=3,V=3)
# forward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?",U=3,V=3)
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
#end sort SAD
#

#
# sort SDI
# center SDI
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y

>="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# backward SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255

0 ?")
# forward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0

?")
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
StackVertical(Separatefields(source),fieldmaskedsdi)
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source)))
#Weave(fieldmaskedsdi)


# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
threshsp=10
THP=string(threshsp)
threshavg=2
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
SDIAvgblk=SDIavg.PointResize(90,72)
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
SDIavg=SDIAvgblk.PointResize(720,576)
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4))# rectangle 4 for 3

pixels thikness line, 2 for 2, not need for 1
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(4,0))# remove line shorter 5

pixels
SDImaskleft=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 -1 0 -2 0 -3 0 -4 0 -5 0 -6 0 -7 0

-8 0 -9 0 -10 0 -11 0")# try
SDImaskright=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9

0 10 0 11 0") # join
SDIMask=mt_logic(SDImaskleft,SDImaskright,"min")

# sequental horizontal line
SDImask=SDImask.mt_luts(SDImask,mode="min",pixels=mt_rectangle(18,0))

# remove line shorter 37 pixels
SDImask=SDImask.HorlLinExp()

# fill line with noise
SDIad=mt_logic(SDIad,SDImask,"min")

# refine source SDI
SDIad=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4))

# rectangle 4 for 3 pixels thikness line, 2 for 2, not need for 1
return(SDIad)

}

# fill full horizontal line if exist one pixel per line

function HorlLinExp(clip mask1) {

mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}

YUP, do you want me to send you a source video sample, in order to make some testings ?

[yup]

fonzzie31!
Your source interlaced? Or film?
Try increase to 7
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,7),U=3,V=3)#
Script can remove line from 2 sequential frames, if greater You can use second, three or greater pass.
Can upload Your video and I am try this.
yup.

[fonzzie31]

Theorically, as the source is taken from Laserdisc, the video is interlaced...

I can prepare you a video sample, if you want.
I can open my ftp for you. I hope you've a great internet connection.

The file will be a bit heavy, because it's AVI RGB 24Bits.

I tried with the new value.... no change.... still dropout on screen...

[yup]

fonzzie31!
During weekend I am out from broadband Internet, but from Monday no problem.
yup.

[Mounir]

Have you tried the DvDropout script (http://forum.doom9.org/showthread.php?t=128649&highlight=dropout) Fonzzie31 works well on such lines depending if there is lot of motion or not.I'm from Tlse by the way :)

[fonzzie31]

@Yup : I prepare some samples for you... I'll send you a private message on Monday, in order to give my ftp details.

@Mounir : DvDropout does not work, with my videos. Nice to see you're in Toulouse too :)

[jmac698]

@Yup
I am making installer, do you have any improvements?

#Remove Black Horizontal Stripes
#Author: Yup, documented and packaged by jmac698
#Version: 2011-10-11, mod1.0
#Homepage: http://forum.doom9.org/showthread.php?t=121197
#Requirements:
#Masktools
#http://forum.doom9.org/showthread.php?t=98985
#MVTools2
#http://www.avisynth.org.ru/mvtools/mvtools2.html
#Removegrain RemovegrainT
#http://www.removegrain.de.tf/
#only AVSRecursion (need install to Windows\system32 directory)
#http://videoprocessing.11.forumer.com/
#Nnedi3
#http://web.missouri.edu/~kes25c/
#Optional:
#Avisynth MT
#http://avisynth.org/mediawiki/MT_support_page
#Usage:
#Provide a clip for source
#Set linethick to the desired amount. Note that in this version, the linethick parameter is NOT fully implemented, just use 3!
#Set mt to true if you are using Avisynth MT

SetMemoryMax(768)
mt=false
mt?SetMTMode(3,4):nop
#source=DirectShowSource("lecture1.avi",pixel_type="YUY2")#.Trim(750,1250)
source=colorbars(pixel_type="YV12")
mt?SetMTMode(2,4):nop
source=source.AssumeTFF().ConvertToYV12(interlaced=true)#.ColorYUV(autogain=true, autowhite=true)
fieldmaskedsdi=removestripes(source, 3)
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source)))
#Weave(fieldmaskedsdi)
StackVertical(Separatefields(source),fieldmaskedsdi)

function removestripes(clip source, int linethick){
#rect1=5 for 3 pixels thickness line, rect1=3 for 2 pixels thickness line, rect1=1 for 1 pixel thickness line
rect1=linethick*2-1
bobnn=source.nnedi3(field=-2)
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,rect1),U=3,V=3)# rectangle 5 for 3 pixels thickness line, 3 for 2, 1 for 1
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
bobnnf=bobnn#.dfttest(tbsize=1,ftype=1,sbsize=16,sosize=12,sigma=1000,U=false,V=false)
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
superff=MSuper(bobnnf)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnnf, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnnf, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnnf, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnnf, super, fw2, thSAD=16000, thSCD1=16000)
bwabs1=mt_lutxy(bobnnf,bc1,"x y - abs")#,U=-128,V=-128)
bwabs2=mt_lutxy(bobnnf,bc2,"x y - abs")#,U=-128,V=-128)
fwabs1=mt_lutxy(bobnnf,fc1,"x y - abs")#,U=-128,V=-128)
fwabs2=mt_lutxy(bobnnf,fc2,"x y - abs")#,U=-128,V=-128)
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
#
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
bwsad=mt_logic(mb1sad,mb2sad,"max")#,U=2,V=2)
fwsad=mt_logic(mf1sad,mf2sad,"max")#,U=2,V=2)
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
#
# sort SAD
# center filtered
mt_logic(mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and"),mt_lut(centersad,"x 255 <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?",U=3,V=3)
# backward filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?",U=3,V=3)
# forward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?",U=3,V=3)
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
#end sort SAD
#

#
# sort SDI
# center SDI
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# backward SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# forward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#

fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
fieldmaskedsdi
}

# Spike Detection Function
#
function SDIAdapt(clip b1,clip b2)
{
threshsp=10
THP=string(threshsp)
threshavg=2
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
SDIAvgblk=SDIavg.PointResize(90,72)
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
SDIavg=SDIAvgblk.PointResize(720,576)
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4))# rectangle 4 for 3 pixels thickness line, 2 for 2, not need for 1
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(4,0))# remove line shorter 5 pixels
SDImaskleft=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 -1 0 -2 0 -3 0 -4 0 -5 0 -6 0 -7 0 -8 0 -9 0 -10 0 -11 0")# try
SDImaskright=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 11 0") # join
SDIMask=mt_logic(SDImaskleft,SDImaskright,"min") # sequental horizontal line
SDImask=SDImask.mt_luts(SDImask,mode="min",pixels=mt_rectangle(18,0)) # remove line shorter 37 pixels
SDImask=SDImask.HorlLinExp() # fill line with noise
SDIad=mt_logic(SDIad,SDImask,"min") # refine source SDI
SDIad=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4)) # rectangle 4 for 3 pixels thikness line, 2 for 2, not need for 1
return(SDIad)
}

# fill full horizontal line if exist one pixel per line
function HorlLinExp(clip mask1) {
mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}

[jmac698]

Installer
http://hotfile.com/dl/133672065/58c5414/Avisynth-Stripes.exe.html

[jmac698]

@yup,
We need some help. I don't know how to use your script.
http://forum.videohelp.com/threads/340068-sort-of-black-comets-trails-in-hi8-video/page2

[yup]

jmac698!
I try find time and write comment inside body script. Now I am not ready for script with parameters set.
Script work for example remove 2 horizontal lines (1 horizontal line untouched and will be removed later at additional pass).
After bobbing using nnedi3 we get 3 lines, for median filtering need 7 vertical pixels mt_rectangle(0,3).
After filtering with dfttest remove segregation for better motion estimation.
After motion estimation i search better choice for median filtering from 3 motion compensating cases (1 p-1 ,p, p+1; 2 p, p, p+1, p+2; 3 p, p-1, p-2) with minimum SAD.
Also I search spike detection index for the same 3 cases. SDI have priority under motion search because median time filtering introduce artifacts.
At SDI i am using post processing for find only current thickness horizontal stripe mt_rectangle(0,2) for thickness 2. Additional calculation remove artifacts (small horizontal stripe false alarm from SDI) and join small stripes in one long.
SDI algo from
http://www.mee.tcd.ie/~ack/papers/a4ackphd.ps.gz
in my case for 720x576 PAL.
You need different approach for remove long lines and short post-processing SDI.
Also can add repair at end using filtering and source clip
diff=source-filtered
df=medianverticalfiltered(with thickness less 1)
filterd=source-df
Main idea touch only bad part of frame.
yup.

[yup]

Script with comments:
SetMemoryMax(768)
SetMTMode(3,4)
source=DirectShowSource("lecture1.avi",pixel_type="YUY2")
#source file
SetMTMode(2,4)
source=source.AssumeTFF().ConvertToYV12(interlaced=true)#.ColorYUV(autogain=true, autowhite=true)
bobnn=source.nnedi3(field=-2)
# bobbing for better motion estimation, for 3 pixels thickness line became 5 pixels thickness, for 2 3, for 1 1
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,5),U=3,V=3)
# vertical median filter rectangle 5 for 3 pixels source (not bobbing) thikness line, 3 for 2, 1 for 1
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
#dft filter remove segregation after median vertical filter
bobnnf=bobnn#.dfttest(tbsize=1,ftype=1,sbsize=16,sosize=12,sigma=1000,U=false,V=false)
# motion estimation on filtered and compensating on source
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
superff=MSuper(bobnnf)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnnf, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnnf, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnnf, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnnf, super, fw2, thSAD=16000, thSCD1=16000)
# calculation absolute difference between source and compensated
bwabs1=mt_lutxy(bobnnf,bc1,"x y - abs")#,U=-128,V=-128)
bwabs2=mt_lutxy(bobnnf,bc2,"x y - abs")#,U=-128,V=-128)
fwabs1=mt_lutxy(bobnnf,fc1,"x y - abs")#,U=-128,V=-128)
fwabs2=mt_lutxy(bobnnf,fc2,"x y - abs")#,U=-128,V=-128)
# calculation spike detection index for center, forwad and backward
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
# calculating SAD for filtered for estimation better choice for motion compensation
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
# maximum SAD from forward and bacward measure for quality center compensation
bwsad=mt_logic(mb1sad,mb2sad,"max")#,U=2,V=2)
#for bacward compensation
fwsad=mt_logic(mf1sad,mf2sad,"max")#,U=2,V=2)
#for forward compensation
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
# inverting mask for using mt_merge
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
# motion compensated median filtering for center, backward and forward compensation
#
# sort SAD for finding better motion compensated median filtering, this approach can use without SDI for pixels and 1 line thickness stripes
# center filtered
mt_logic(mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and"),mt_lut(centersad,"x 255 <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?",U=3,V=3)
# backward filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?",U=3,V=3)
# forward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?",U=3,V=3)
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
#best value based on 3 motion compensated median time filterd value and spatial filtered
# where motion compensation bad, work only for one frame with stripe
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
# backward and forward compensated full for SDI approach
#end sort SAD
#

#
# sort SDI for choose center, forward or forward compensation work with 2 sequential frames, for 3 need 2 pass filtering, for 4 3
# center SDI
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# backward SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# forward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


# this place for repair now I am thinking

fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
StackVertical(Separatefields(source),fieldmaskedsdi)
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source)))
#Weave(fieldmaskedsdi)
# some kind comparing and weave for interlaced source, may be at double rate and for second pass not need bobbing

# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
threshsp=10
# threshold for pixels value absolute difference
THP=string(threshsp)
threshavg=2
# threshold how much times actual SDI greater avearage and decimated SDI for remove false alarm for bad motion compensation
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
# SDI calculation
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
# SDI averaging
SDIAvgblk=SDIavg.PointResize(90,72)
# decimation
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
# median point at X position
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
#median point at Cross position
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
# multi median filtering for remove false alarm SDI
SDIavg=SDIAvgblk.PointResize(720,576)
# backward resize to original frame size
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
# comparing with actual SDI and if greater it is real spike, value 128 can increase to 192
# be carefull I can not find this at original paper, big value can loose spike, small false alarm
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4))
# rectangle size could one less than for bobnnmed
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(4,0))
# remove line shorter depend Your source
SDImaskleft=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 -1 0 -2 0 -3 0 -4 0 -5 0 -6 0 -7 0 -8 0 -9 0 -10 0 -11 0")
# try join single short spike to long 11 distance between spikes from left side
SDImaskright=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 11 0")
# from right
SDIMask=mt_logic(SDImaskleft,SDImaskright,"min")
# if pixels exist from left and right simultanessly this spike for detections series short spikes
SDImask=SDImask.mt_luts(SDImask,mode="min",pixels=mt_rectangle(18,0))
# remove spikes shorter than 37 pixels tune for source
SDImask=SDImask.HorlLinExp()
# if line have only one pixel line will be filled
SDIad=mt_logic(SDIad,SDImask,"min")
# combination source and adapting SDI for better robustness
SDIad=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4))
# refinement rectangle size could one less than for bobnnmed
return(SDIad)

}

# fill full horizontal line if exist one pixel per line
# basaed on Didee idea se masktools thread
function HorlLinExp(clip mask1) {

mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}
Now i do not have any top secrets :).
One more welcome to suggesting and developing.
yup.

[mammo1789]

Thanks yup for your script i tried it and i get some removal of the black lines (especially the thin ones ), but the rest is still here can you suggest me some tweaking of your script or what threshold should i use for these lines.
I'm posting pictures to see the difference

http://forum.videohelp.com/attachments/9449-1320203871/za%20crni%20linii_0.00.01.15.jpg

http://forum.videohelp.com/attachments/9449-1320203871/za%20crni%20linii_0.00.01.15.jpg

http://forum.videohelp.com/attachments/9450-1320203875/ddd555_0.00.01.15.jpg

[yup]

mammo1789!
Welcome
try for preview
source=DirectShowSource("Your.avi")
source=source.AssumeTFF().ConvertToYV12(interlaced=true)
bobnn=source.nnedi3(field=-2)
# bobbing for better motion estimation, for 3 pixels thickness line became 5 pixels thickness, for 2 3, for 1 1
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,X),U=3,V=3)
#X need tuned
field=bobnnmed.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
stackvertical(SeparateFields(source),field)
Increase value second parameter for mt_rectangle(0,X) until remove your black lines at preview.
Inside SDIAdapt function You need mt_rectangle(0,X-1). Read comments. All threshold work relatively good. For Your source need no one pass. Once I use 4 pass for my source 2 with X=5, and 2 with X=3 and get good result (see my screenshot at #45 post).
Upload 100 frames Your video.
yup.

[Jenyok]

yup
.
Whether correctly I have understood you, that the following (next) Code is for progressive video ?
See below.
May be need some corrections in Code ?
.

LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\MVTOOLS-V2_5_10\mvtools2.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\MASKTOOLS-V2_0A48\mt_masktools-25.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\DFTTESTV_1_6\dfttest.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAINT-1_0\removegraint.dll")

# Spike Detection Function
#
function SDIAdapt(clip b1, clip b2)
{
threshsp=10
# threshold for pixels value absolute difference
THP=string(threshsp)
threshavg=2
# threshold how much times actual SDI greater avearage and decimated SDI for remove false alarm for bad motion compensation
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
# SDI calculation
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
# SDI averaging
SDIAvgblk=SDIavg.PointResize(90,72)
# decimation
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
# median point at X position
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
#median point at Cross position
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
# multi median filtering for remove false alarm SDI
SDIavg=SDIAvgblk.PointResize(720,576)
# backward resize to original frame size
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
# comparing with actual SDI and if greater it is real spike, value 128 can increase to 192
# be carefull I can not find this at original paper, big value can loose spike, small false alarm
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4))
# rectangle size could one less than for bobnnmed
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(4,0))
# remove line shorter depend Your source
SDImaskleft=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 -1 0 -2 0 -3 0 -4 0 -5 0 -6 0 -7 0 -8 0 -9 0 -10 0 -11 0")
# try join single short spike to long 11 distance between spikes from left side
SDImaskright=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 11 0")
# from right
SDIMask=mt_logic(SDImaskleft,SDImaskright,"min")
# if pixels exist from left and right simultanessly this spike for detections series short spikes
SDImask=SDImask.mt_luts(SDImask,mode="min",pixels=mt_rectangle(18,0))
# remove spikes shorter than 37 pixels tune for source
SDImask=SDImask.HorlLinExp()
# if line have only one pixel line will be filled
SDIad=mt_logic(SDIad,SDImask,"min")
# combination source and adapting SDI for better robustness
SDIad=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,4))
# refinement rectangle size could one less than for bobnnmed
return(SDIad)
}

# fill full horizontal line if exist one pixel per line
# basaed on Didee idea se masktools thread
#
function HorlLinExp(clip mask1)
{
mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

# v1=interleave(mask1, black).assumefieldbased.assumetff.weave
# v2=interleave(mask2, black).assumefieldbased.assumetff.weave
# mt_hysteresis(v1, v2, u=-128, v=-128)
# return separatefields.selecteven.assumeframebased

v1=interleave(mask1, black)
v2=interleave(mask2, black)
return mt_hysteresis(v1, v2, u=-128, v=-128)
}

AVISource("1991 Programma A.avi")

ConvertToYUY2()

# Для отладки фильтров часть 1
#
SourceView=last
SourceView=AddBorders(SourceView, 0, 0, 8, 0, Color=$FFFFFF)
SourceView=ConvertToYUY2(SourceView)

ConvertToYV12()

bobnn = last

# bobbing for better motion estimation, for 3 pixels thickness line became 5 pixels thickness, for 2 3, for 1 1
bobnnmed = bobnn.mt_luts(bobnn, mode = "median", pixels = mt_rectangle(0, 5), U = 3, V = 3)

# vertical median filter rectangle 5 for 3 pixels source (not bobbing) thikness line, 3 for 2, 1 for 1
bobnnmedf = bobnnmed.dfttest(tbsize = 1, ftype = 1, sbsize = 8, sosize = 6, sigma = 1000, U = false, V = false)

#dft filter remove segregation after median vertical filter
bobnnf = bobnn#.dfttest(tbsize = 1, ftype = 1, sbsize = 16, sosize = 12, sigma = 1000, U = false, V = false)

# motion estimation on filtered and compensating on source
super = MSuper(bobnn)
superf = MSuper(bobnnmedf, chroma = false)
superff = MSuper(bobnnf)
bw1 = MAnalyse(superf, blksize = 8, isb = true, delta = 2, overlap = 4, dct = 5, chroma = false)
bw2 = MAnalyse(superf, blksize = 8, isb = true, delta = 4, overlap = 4, dct = 5, chroma = false)
fw1 = MAnalyse(superf, blksize = 8, isb = false, delta = 2, overlap = 4, dct = 5, chroma = false)
fw2 = MAnalyse(superf, blksize = 8, isb = false, delta = 4, overlap = 4, dct = 5, chroma = false)
bc1 = MCompensate(bobnnf, super, bw1, thSAD = 16000, thSCD1 = 16000)
bc2 = MCompensate(bobnnf, super, bw2, thSAD = 16000, thSCD1 = 16000)
fc1 = MCompensate(bobnnf, super, fw1, thSAD = 16000, thSCD1 = 16000)
fc2 = MCompensate(bobnnf, super, fw2, thSAD = 16000, thSCD1 = 16000)

# calculation absolute difference between source and compensated
bwabs1 = mt_lutxy(bobnnf, bc1, "x y - abs")#, U = -128, V = -128)
bwabs2 = mt_lutxy(bobnnf, bc2, "x y - abs")#, U = -128, V = -128)
fwabs1 = mt_lutxy(bobnnf, fc1, "x y - abs")#, U = -128, V = -128)
fwabs2 = mt_lutxy(bobnnf, fc2, "x y - abs")#, U = -128, V = -128)

# calculation spike detection index for center, forwad and backward
SDIc = SDIAdapt(bwabs1, fwabs1)
SDIb = SDIAdapt(bwabs1, bwabs2)
SDIf = SDIAdapt(fwabs1, fwabs2)

# calculating SAD for filtered for estimation better choice for motion compensation
mb1sad = bobnnmedf.MMask(bw1, kind = 1, ml = 100, Ysc = 255, thSCD1 = 16000)
mb2sad = bobnnmedf.MMask(bw2, kind = 1, ml = 100, Ysc = 255, thSCD1 = 16000)
mf1sad = bobnnmedf.MMask(fw1, kind = 1, ml = 100, Ysc = 255, thSCD1 = 16000)
mf2sad = bobnnmedf.MMask(fw2, kind = 1, ml = 100, Ysc = 255, thSCD1 = 16000)
centersad = mt_logic(mf1sad, mb1sad, "max")#, U = 2, V = 2)

# maximum SAD from forward and bacward measure for quality center compensation
bwsad = mt_logic(mb1sad, mb2sad, "max")#, U = 2, V = 2)

# for bacward compensation
fwsad = mt_logic(mf1sad, mf2sad, "max")#, U = 2, V = 2)

# for forward compensation
centermask = mt_invert(centersad)
bwmask = mt_invert(bwsad)
fwmask = mt_invert(fwsad)

# inverting mask for using mt_merge
mcfcenter = clense(bc1, bobnn, fc1, increment = 0, grey = false)
mcfbw = clense(bc1, bobnn, bc2, increment = 0, grey = false)
mcffw = clense(fc1, bobnn, fc2, increment = 0, grey = false)

# motion compensated median filtering for center, backward and forward compensation
# sort SAD for finding better motion compensated median filtering, this approach can use without SDI for pixels and 1 line thickness stripes
# center filtered
mt_logic(mt_logic(mt_lutxy(centersad, bwsad, "x y <= "), mt_lutxy(centersad, fwsad, "x y <="), "and"), mt_lut(centersad, "x 255 <="), "and")
maskcentersad = mt_lutxy(last, centersad, "x 255 y - 0 ?", U = 3, V = 3)

# backward filtered
mt_logic(mt_lutxy(bwsad, centersad, "x y < "), mt_lutxy(bwsad, fwsad, "x y <="), "and")
maskbwsad = mt_lutxy(last, bwsad, "x 255 y - 0 ?", U = 3, V = 3)

# forward filtered
mt_logic(mt_lutxy(fwsad, centersad, "x y < "), mt_lutxy(fwsad, bwsad, "x y < "), "and")
maskfwsad = mt_lutxy(last, fwsad, "x 255 y - 0 ?", U = 3, V = 3)

#
mcfmaskedsad = mt_merge(bobnnmed, mcfbw, maskbwsad, luma = true)
mcfmaskedsad = mt_merge(mcfmaskedsad, mcffw, maskfwsad, luma = true)
mcfcentersad = mt_merge(mcfmaskedsad, mcfcenter, maskcentersad, luma = true)

# best value based on 3 motion compensated median time filterd value and spatial filtered
# where motion compensation bad, work only for one frame with stripe
mcfbwsad = mt_merge(bobnnmed, mcfbw, maskbwsad, luma = true)
mcffwsad = mt_merge(bobnnmed, mcffw, maskfwsad, luma = true)

# backward and forward compensated full for SDI approach
# end sort SAD

# sort SDI for choose center, forward or forward compensation work with 2 sequential frames, for 3 need 2 pass filtering, for 4 3
# center SDI
maskcentersdi = mt_logic(mt_logic(mt_lutxy(SDIc, SDIb, "x y >= "), mt_lutxy(SDIc, SDIf, "x y >="), "and"), SDIc, "and").mt_lut("x 255 0 ?")

# backward SDI
maskbwsdi = mt_logic(mt_lutxy(SDIb, SDIc, "x y > "), mt_lutxy(SDIb, SDIf, "x y >="), "and").mt_lut("x 255 0 ?")

# forward SDI
maskfwsdi = mt_logic(mt_lutxy(SDIf, SDIc, "x y > "), mt_lutxy(SDIf, SDIb, "x y >"), "and").mt_lut("x 255 0 ?")

#
maskedsdi = mt_merge(bobnn, mcfcentersad, maskcentersdi, luma = true)
maskedsdi = mt_merge(maskedsdi, mcffwsad, maskfwsdi, luma = true)
maskedsdi = mt_merge(maskedsdi, mcfbwsad, maskbwsdi, luma = true)
# end sort SDI

last = maskedsdi

ConvertToYUY2()

# Для отладки фильтров часть 2
#
StackHorizontal(SourceView, last)

[yup]

Jenyok!
SDIAdapt for progressive frame. I get progressive from interlaced using nnedi3. Also this code for PAL 720x576.
Spike detection index algorithm see post
http://forum.doom9.org/showthread.php?p=1534999#post1534999
Rmaining code try remove artifacts. After weekend I make small update.
If your source progressive you not need use nnedi3 and delta at MAnalyse divide by 2.
yup.

[Jenyok]

yup
Are you in IXBT.com forum of AVISynth ?
If yes, please contact with me there.
Thanks...

[yup]

Hi All!
Please
thicknessline=2# thickness horizontal spike in pixels
Global radvertmed=2*thicknessline-1
Global halflength=18# detect line longer 2*halflength+1 pixels
#SetMemoryMax(768)
#SetMTMode(3,4)
source=DirectShowSource("flower.avi",pixel_type="YUY2")
#source file
#SetMTMode(2,4)
source=source.AssumeTFF().ConvertToYV12(interlaced=true)
bobnn=source.nnedi3(field=-2,threads=1)
# bobbing for better motion estimation, for 3 pixels thickness line became 5 pixels thickness, for 2 3, for 1 1
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,radvertmed),U=3,V=3)
# vertical median filter radvertmed 5 for 3 pixels source (not bobbing) thikness line, 3 for 2, 1 for 1
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
#dft filter remove segregation after median vertical filter
# motion estimation on filtered and compensating on source
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnn, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnn, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnn, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnn, super, fw2, thSAD=16000, thSCD1=16000)
# calculation absolute difference between source and compensated
bwabs1=mt_lutxy(bobnn,bc1,"x y - abs")
bwabs2=mt_lutxy(bobnn,bc2,"x y - abs")
fwabs1=mt_lutxy(bobnn,fc1,"x y - abs")
fwabs2=mt_lutxy(bobnn,fc2,"x y - abs")
# calculation spike detection index for center, forwad and backward
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
# calculating SAD for filtered for estimation better choice for motion compensation
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
# maximum SAD from forward and bacward measure for quality center compensation
bwsad=mt_logic(mb1sad,mb2sad,"max")
#for bacward compensation
fwsad=mt_logic(mf1sad,mf2sad,"max")
#for forward compensation
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
# inverting mask for using mt_merge
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
# motion compensated median filtering for center, backward and forward compensation
#
# sort SAD for finding better motion compensated median filtering, this approach can use without SDI for pixels and 1 line thickness stripes
# center filtered
mt_logic(mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and"),mt_lut(centersad,"x 255 <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?")
# backward filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?")
# forward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?")
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
#best value based on 3 motion compensated median time filterd value and spatial filtered
# where motion compensation bad, work only for one frame with stripe
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
# backward and forward compensated full for SDI approach
#end sort SAD
#

#
# sort SDI for choose center, forward or forward compensation work with 2 sequential frames, for 3 need 2 pass filtering, for 4 3
# center SDI
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# backward SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# forward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


# this place for repair now I am thinking

fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
StackVertical(Separatefields(source),fieldmaskedsdi)
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source)))
#Weave(fieldmaskedsdi)
# some kind comparing and weave for interlaced source, may be at double rate and for second pass not need bobbing

# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
threshsp=10
# threshold for pixels value absolute difference
THP=string(threshsp)
threshavg=2
# threshold how much times actual SDI greater avearage and decimated SDI for remove false alarm for bad motion compensation
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
# SDI calculation
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
# SDI averaging
SDIAvgblk=SDIavg.PointResize(90,72)
# decimation
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
# median point at X position
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
#median point at Cross position
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
# multi median filtering for remove false alarm SDI
SDIavg=SDIAvgblk.PointResize(720,576)
# backward resize to original frame size
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
# comparing with actual SDI and if greater it is real spike, value 128 can increase to 192
# be carefull I can not find this at original paper, big value can loose spike, small false alarm
SDIad = (radvertmed>1) ? SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,radvertmed-1)) : SDIad
# rectangle size could one less than for bobnnmed
SDIad = (radvertmed>1) ? SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(radvertmed-1,0)) : SDIad
SDImask=SDIad.mt_luts(SDIad,mode="min",pixels=mt_rectangle(halflength,0))
# remove spikes shorter than 2*halflength+1 pixels tune for source
SDImask=SDImask.HorlLinExp()
# if line have only one pixel line will be filled
SDIad=mt_logic(SDIad,SDImask,"min")
# combination source and adapting SDI for better robustness
return(SDIad)

}

# fill full horizontal line if exist one pixel per line
# basaed on Didee idea see masktools thread
function HorlLinExp(clip mask1) {

mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}
It is very good work for relatively long stripe. If You want remove series short need replace to

SDIadleft=SDIad.mt_luts(SDIad,mode="max",pixels=mt_freerectangle(-11,0,0,0))
# try join single short spike to long 11 distance between spikes from left side
SDImadright=SDIad.mt_luts(SDIad,mode="max",pixels=mt_freerectangle(0,0,11,0))
# from right
SDIMask=mt_logic(SDIadleft,SDIadright,"min")
# if pixels exist from left and right simultanessly this spike for detections series short spikes
SDImask=SDImask.mt_luts(SDImask,mode="min",pixels=mt_rectangle(halflength,0))
SDIMask=........
yup.

[Jenyok]

yup
Maybe write two (2) functions, for example
Long_Stripes(...) and Short_Stripes(...)
with different parameters (...)
and do *.avsi module ???
Think about this ?!

[Ghitulescu]

Thanks yup for your script i tried it and i get some removal of the black lines (especially the thin ones ), but the rest is still here can you suggest me some tweaking of your script or what threshold should i use for these lines.
Try a new player? Preferably in a better (newer) condition?

[yup]

yup
Maybe write two (2) functions, for example
Long_Stripes(...) and Short_Stripes(...)
with different parameters (...)
and do *.avsi module ???
Think about this ?!
It is depend from source.
I have source with only long spike, short spike comet style and combination. Script not automatic need tuning on source.
Also do not forget first need remove thick spike and after thin.
About avsi i am not ready, but think about this.
yup.

[Jenyok]

yup
Oh...
I need the same filter (script) to WHITE horizontal stripes !!!
I reaaly have such video content with WHITE horizontal stripes...
What could I change in your script ?

[yup]

Jenyok!
No problem. It is will be work. Need absolute difference between normal value and stripe. Name historical. May be need change thread name to horizontal stripe wiothout black for clarity?
yup.

[Jenyok]

Tested this script.
Script does not works.
White stripes.

[yup]

Jenyok!

Need choose right parameter.
First try remove from thick to thin lines thicknessline parameter. If You choose thicknessline=3 for source with lines only 2 and 1 pixels script make nothing.
For total remove need use sequential from thicker to thiner lines, starting with thicknessline=max to thicknessline=1.
If line gone in 3 frames successively need 2 pass with the same thicknessline parameter
The same situation with halflength parameter.
If You have lines with length shorter than 2*halflength+1 script untouch source.
You can upload small sample and I try find near opt set .
For white lines see example from this thread
http://forum.doom9.org/showthread.php?p=1531897#post1531897
even at old version script, now script work little better.
yup.

[fonzzie31]

Hello Yup !!!

Happy new year for you !!!!

Could you give me the last version of your script, please ?
I think I'll contact you because I encounter some problem with the script version I have. But before that, I would like to verify if the same problem occurs with latest version :)

Best regards,

[yup]

fonzzie31!
Hello!
Also Happy New Year!
See
http://forum.doom9.org/showthread.php?p=1541580#post1541580
But Your source not pure interlaced, need some correction when I made I post it.
For telecined source better approach give sequintial median filter with radius 1 at frame basis not field like prefilter not radius 3 one time.
yup.

[fonzzie31]

Hello Yup :)

Let me explain my problem, but I'll need to send you some screen capture in order to illustrate.

I had a true interlaced video to process with your script (one pass), and I noticed some visible artefacts on the result video which are not present in the source.

I tried with avs or with virtualdub : same problem occurs
I deinterlaced the video with avisynth : same problem occurs
Whatever codec I use for the video, the problem occurs.

I try other thing :
I put also the source video on Adobe After Effects, and I Use the professionnal Revision-FX Deinterlace plugin, in order to create 2 differents video files containing the top and bottom fields of the source video.
The 2 video files were processed by your script. The 2 video result files were loaded into After Effects in order to reinterlace the field, using professional Revision-FX Reinterlace plugin.
The same problem occurs.

So, the conclusion is that your script seems to create artefacts.

The better thing is to send you the complete source file. It's a pure interlaced Bananarama music video, grabbed from NTSC laserdisc, and converted into PAL, with a professionnal post-processing equipment (SNELL & WILCOX CVR450)

Tomorrow, I'll try to send you there some screen captures showing the problem.

[yup]

Hi fonzzie31!

I am do not speak that script not introduce artefacts:(
Artefacts place where do not work Spike Detection Index, this algorithm can give false alarm and with combination temporal median filter introduce artefacts.
Script tested on VHS capture material not very sharp and without small details.
I am see artefacts at source with artificial objects with sharp straight borders, because script using median filtering.
Right determine trhicknesline and length for line critical.
For interlaced source I am using my script with thicknesline parameter from maximum to 1.
If use script 1 pass with bigger thicknessline and 2 pass with less, at 2 pass can remove some artefacts.
After remove line I use QTGMC like deinterlacer which also can remove some artefacts.
yup.

[fonzzie31]

Hi Yup,

Let me explain and illustrate my problem.
The horizontal resolution for a NTSC Laserdisc is 425 analog lines. VHS is only 240 analog lines.

So, I took a interlaced PAL video file. This file was created by grabbing a laserdisc, with my Blackmagic Decklink Studio 2. This card use v210 codec for video.
As The video is interlaced, I loaded it in Adobe After Effects, with no field detection.
I used a professional Deinterlacer AE plugin, RevisionFX Fieldskit, in order to separate the field of the video.
I configured this plugin in order to make an output frame from the field selected (top or botton) in each source frame. The other field was discarded, and new information was created from the discarded field scanlines using the field selected.
The output files (one with top fields, one with bottom fields) are saved as RGB24 progressive videos.
I hope you understand what I did with After effects.

Then I created a batch file for running your script. 2 files to process. one pass each (the maximum number of passes I used for a video files is 6)
The result video file was in YV12.

I compared the source and the result video files, and I noticed some artefacts in the result video, in some frames, in particular during transitions or quick movements.
These frames are supposed not to contain any drop outs to remove.

Let me show you some transitions samples from a video:

Example 1 :

The source frame :
http://revival03.free.fr/temp/bananarama1s.jpg

The result frame :
http://revival03.free.fr/temp/bananarama1r.jpg

----------------
Example 2 :

The source frame :
http://revival03.free.fr/temp/bananarama2s.jpg

The result frame :
http://revival03.free.fr/temp/bananarama2r.jpg


----------------
Example 3 :

The source frame :
http://revival03.free.fr/temp/bananarama3s.jpg

The result frame :
http://revival03.free.fr/temp/bananarama3r.jpg

-----------------

I thought it was the after effect plugins which did a bad job.
So I decided to put a field separation function in Avisynth. But the same problem occurs on output. Same artefacts.
I tried to convert the source file in YV12 before processing it with your script. Same artefacts.
If I use AVS or Virtualdub, I've the same problem.

What do you think about that ?

[yup]

fonzzie31!
IMO it not related to my script.
I am try make some advice.
Try capture laserdisc like NTSC without any hardware conversion in YUY2 colorspace for AVisynth compatibility using lossless codec huff very good for start. After use QTGMC deinterlacer . You get 720x480 at 59.94 fps video. If result good, You can transform to PAL but this will be second step.
At source I see blended frame related with NTSC to PAL conversion.
Also color bar can be related with colorspace conversion.
Complex task coulde separated to simple one.
You can get answer at capture part this forum also.
yup.

[fonzzie31]

Thanks Yup

As a start, I will try to create an YV12 sample from the PAL converted video file.

I'll also try to create an YV12 sample from the original NTSC video.

I"ll send you the results in the next few days.

[fonzzie31]

Hello Yup,

Today, I made some tests...

I captured the video from a laserdisc. The video was kept into NTSC.
The video was grabbed into 10 Bits UYV Uncompressed (v210). I don't have any other possibilities.
So, I convert the video into YV12 (YUV 4:2:0) format, after separating the fields in After Effects (see my last post, for the procedure).

I processed the bottom fields with your script. One pass processing through AVS.
Input video : YV12 .Output video : YV12

I still see some artefacts in the result video.
I really think that one of the masks in your script creates these artefacts.
The artefacts is not the result of a color space conversion.
Input and output video format are the same.

Do you want to get the video file in order for you to make some tests ?


Example 1 :
------------

Source :

http://revival03.free.fr/temp/bananarama4s.jpg

Result :

http://revival03.free.fr/temp/bananarama4r.jpg


Example 2 :
-----------

Source :

http://revival03.free.fr/temp/bananarama5s.jpg

Result :

http://revival03.free.fr/temp/bananarama5r.jpg


Example 3 :
-----------

Source :

http://revival03.free.fr/temp/bananarama6s.jpg

Result :

http://revival03.free.fr/temp/bananarama6r.jpg


I used the following script :

-------------------------------

thicknessline=1# thickness horizontal spike in pixels
Global radvertmed=2*thicknessline-1
Global halflength=18# detect line longer 2*halflength+1 pixels

SetMemoryMax(768)
SetMTMode(3,4)
source=AVISource("../Sources/1984 - BANANARAMA - Rough Justice - NTSC - BF.avi", pixel_type="YV12")
#source file
#SetMTMode(2,4)
#source=source.ConvertToYV12(matrix="rec601")
#source=source.assumeframebased().Separatefields()
SetMTMode(4,4)
bobnn=source
#bobnn=Telecide(source,guide=2,post=0)
SetMTMode(2,4)
# bobbing for better motion estimation, for 3 pixels thickness line became 5 pixels thickness, for 2 3, for 1 1
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,radvertmed),U=3,V=3)
# vertical median filter rectangle radvertmed 5 for 3 pixels source (not bobbing) thikness line, 3 for 2, 1 for 1
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
#dft filter remove segregation after median vertical filter
#.dfttest(tbsize=1,ftype=1,sbsize=16,sosize=12,sigma=1000,U=false,V=false)
# motion estimation on filtered and compensating on source
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnn, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnn, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnn, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnn, super, fw2, thSAD=16000, thSCD1=16000)
# calculation absolute difference between source and compensated
bwabs1=mt_lutxy(bobnn,bc1,"x y - abs")#,U=-128,V=-128)
bwabs2=mt_lutxy(bobnn,bc2,"x y - abs")#,U=-128,V=-128)
fwabs1=mt_lutxy(bobnn,fc1,"x y - abs")#,U=-128,V=-128)
fwabs2=mt_lutxy(bobnn,fc2,"x y - abs")#,U=-128,V=-128)
# calculation spike detection index for center, forwad and backward
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
# calculating SAD for filtered for estimation better choice for motion compensation
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
# maximum SAD from forward and bacward measure for quality center compensation
bwsad=mt_logic(mb1sad,mb2sad,"max")#,U=2,V=2)
#for bacward compensation
fwsad=mt_logic(mf1sad,mf2sad,"max")#,U=2,V=2)
#for forward compensation
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
# inverting mask for using mt_merge
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
# motion compensated median filtering for center, backward and forward compensation
#
# sort SAD for finding better motion compensated median filtering, this approach can use without SDI for pixels and 1 line thickness stripes
# center filtered
mt_logic(mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and"),mt_lut(centersad,"x 255 <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?",U=3,V=3)
# backward filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?",U=3,V=3)
# forward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?",U=3,V=3)
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
#best value based on 3 motion compensated median time filterd value and spatial filtered
# where motion compensation bad, work only for one frame with stripe
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
# backward and forward compensated full for SDI approach
#end sort SAD
#

#
# sort SDI for choose center, forward or forward compensation work with 2 sequential frames, for 3 need 2 pass filtering, for 4 3
# center SDI
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# backward SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# forward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


# this place for repair now I am thinking

#fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#StackVertical(Crop(bobnn,12,64,-22,-54),Crop(maskedsdi,12,64,-22,-54))
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source)))
maskedsdi



# some kind comparing and weave for interlaced source, may be at double rate and for second pass not need bobbing

# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
threshsp=10
# threshold for pixels value absolute difference
THP=string(threshsp)
threshavg=2
# threshold how much times actual SDI greater avearage and decimated SDI for remove false alarm for bad motion compensation
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
# SDI calculation
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
# SDI averaging
SDIAvgblk=SDIavg.PointResize(90,72)
# decimation
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
# median point at X position
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
#median point at Cross position
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
# multi median filtering for remove false alarm SDI
SDIavg=SDIAvgblk.PointResize(720,480)
# backward resize to original frame size
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
# comparing with actual SDI and if greater it is real spike, value 128 can increase to 192
# be carefull I can not find this at original paper, big value can loose spike, small false alarm
SDImask=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,1))
# rectangle size could one less than for bobnnmed
SDImask=SDIad#.mt_luts(SDIad,mode="median",pixels=mt_rectangle(4,0))
# remove line shorter depend Your source
# SDImaskleft=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 -1 0 -2 0 -3 0 -4 0 -5 0 -6 0 -7 0 -8 0 -9 0 -10 0 -11 0")
# try join single short spike to long 11 distance between spikes from left side
# SDImaskright=SDImask.mt_luts(SDIad,mode="max",pixels="0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 11 0")
# from right
# SDIMask=mt_logic(SDImaskleft,SDImaskright,"min")
# if pixels exist from left and right simultanessly this spike for detections series short spikes
SDImask=SDImask.mt_luts(SDImask,mode="min",pixels=mt_rectangle(3,0))
# remove spikes shorter than 11 pixels tune for source
SDImask=SDImask.HorlLinExp()
# if line have only one pixel line will be filled
SDIad=mt_logic(SDIad,SDImask,"min")
# combination source and adapting SDI for better robustness
# SDIad=SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,1))
# refinement rectangle size could one less than for bobnnmed
return(SDIad)

}

# fill full horizontal line if exist one pixel per line
# basaed on Didee idea se masktools thread
function HorlLinExp(clip mask1) {

mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}

[yup]

fonzzie31!
Do you want to get the video file in order for you to make some tests ?
Yes.
I am agree about artefacts.
Will be think.
yup.

[fonzzie31]

I send you details in a private message :)

[yup]

Hi All!

Small update code and comments.
Script for remove long lines
Global thicknessline=2# thickness horizontal spike in pixels at field scale (not frame)
Global radvertmed=2*thicknessline-1
Global halflength=18# detect line longer 2*halflength+1 pixels
#SetMemoryMax(768)
#SetMTMode(3,4)
source=AVISource("sel97.avi")
#source file
#SetMTMode(2,4)
source=source.AssumeTFF().ConvertToYV12(interlaced=true)
bobnn=source.nnedi3(field=-2,threads=1)
# bobbing for better motion estimation
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,radvertmed),U=3,V=3)
# vertical median filter
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
#dft filter remove segregation after median vertical filter
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnn, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnn, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnn, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnn, super, fw2, thSAD=16000, thSCD1=16000)
# motion estimation on filtered and compensating on source
bwabs1=mt_lutxy(bobnn,bc1,"x y - abs")
bwabs2=mt_lutxy(bobnn,bc2,"x y - abs")
fwabs1=mt_lutxy(bobnn,fc1,"x y - abs")
fwabs2=mt_lutxy(bobnn,fc2,"x y - abs")
# calculation absolute difference between source and compensated
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
# calculation spike detection index for center, forwad and backward
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
# calculating SAD for filtered for estimation better choice for motion compensation
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
# maximum SAD from forward and backward measure for quality center compensation
bwsad=mt_logic(mb1sad,mb2sad,"max")
#for backward compensation
fwsad=mt_logic(mf1sad,mf2sad,"max")
#for forward compensation
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
# inverting mask for using mt_merge
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
# motion compensated median filtering for center, backward and forward compensation
#
# sort SAD for finding better motion compensated median filtering, this approach can use without SDI for pixels and 1 line thickness stripes
mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?")
# center filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?")
# backward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?")
# forward filtered
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
#best value based on 3 motion compensated median time filtered value and spatial filtered
# where motion compensation bad, work only for one sequintial frame with spike
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
# backward and forward compensated full for SDI approach and spatial filtered where motion compensation bad
#end sort SAD
#

#
# sort SDI for choose center, forward or forward compensation work with 2 sequential frames, for 3 need 2 pass filtering, for 4 3 pass
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# center SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# backward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
# forward SDI
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


# this place for repair now I am thinking

fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
StackVertical(Separatefields(source),fieldmaskedsdi,mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#Weave(fieldmaskedsdi)
# some kind comparing and weave for interlaced source, may be at double rate and for second pass not need bobbing

# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
threshsp=10
# threshold for pixels value absolute difference from 2 sequitial frames for spike detection
THP=string(threshsp)
threshavg=2
# threshold how much times actual SDI greater avearage and decimated SDI for remove false alarm for bad motion compensation
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
# SDI calculation
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
# SDI averaging
SDIAvgblk=SDIavg.PointResize(90,72)# for PAL (90,72), FOR NTSC (90,60)
# decimation
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
# median point at X position
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
#median point at Cross position
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
# multi median filtering for remove false alarm SDI
SDIavg=SDIAvgblk.PointResize(720,576)# for PAL (720,576), FOR NTSC (720,480)
# backward resize to original frame size
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
# comparing with actual SDI and if greater it is real spike, value 128 can increase to 192
# be carefull I can not find this at original paper, big value can loose spike, small false alarm
SDIad = (radvertmed>1) ? SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,radvertmed-1)) : SDIad
# rectangle size could one less than for bobnnmed, for thinner lines need additional pass with thiknessline=thicknessline-1 until thicknessline=1
SDImask=SDIad.mt_luts(SDIad,mode="min",pixels=mt_rectangle(halflength,0))
# remove spikes shorter than 2*halflength+1 pixels tune for source
SDImask=SDImask.HorlLinExp()
# if line have only one pixel line will be filled
SDIad=mt_logic(SDIad,SDImask,"min")
# combination source and adapting SDI for better robustness
# SDIad = (radvertmed>1) ? SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(radvertmed-1,0)) : SDIad
SDIad = SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(thicknessline,0))
# additional remove short spikes which can be artefacts
return(SDIad)

}

# fill full horizontal line if exist one pixel per line
# basaed on Didee idea see masktools thread
function HorlLinExp(clip mask1) {

mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}
Any suggestion welcome.
yup.

[yup]

and series short lines
Global thicknessline=2# thickness horizontal spike in pixels at field scale (not frame)
Global radvertmed=2*thicknessline-1
Global distser=21# distance between short lines
Global halflength=65# detect line longer 2*halflength+1 pixels
#SetMemoryMax(768)
#SetMTMode(3,4)
source=AVISource("sel97.avi")
#source file
#SetMTMode(2,4)
source=source.AssumeTFF().ConvertToYV12(interlaced=true)
bobnn=source.nnedi3(field=-2,threads=1)
# bobbing for better motion estimation
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,radvertmed),U=3,V=3)
# vertical median filter
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
#dft filter remove segregation after median vertical filter
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnn, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnn, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnn, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnn, super, fw2, thSAD=16000, thSCD1=16000)
# motion estimation on filtered and compensating on source
bwabs1=mt_lutxy(bobnn,bc1,"x y - abs")
bwabs2=mt_lutxy(bobnn,bc2,"x y - abs")
fwabs1=mt_lutxy(bobnn,fc1,"x y - abs")
fwabs2=mt_lutxy(bobnn,fc2,"x y - abs")
# calculation absolute difference between source and compensated
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
# calculation spike detection index for center, forwad and backward
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
# calculating SAD for filtered for estimation better choice for motion compensation
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
# maximum SAD from forward and backward measure for quality center compensation
bwsad=mt_logic(mb1sad,mb2sad,"max")
#for backward compensation
fwsad=mt_logic(mf1sad,mf2sad,"max")
#for forward compensation
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
# inverting mask for using mt_merge
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
# motion compensated median filtering for center, backward and forward compensation
#
# sort SAD for finding better motion compensated median filtering, this approach can use without SDI for pixels and 1 line thickness stripes
mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?")
# center filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?")
# backward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?")
# forward filtered
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
#best value based on 3 motion compensated median time filtered value and spatial filtered
# where motion compensation bad, work only for one sequintial frame with spike
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
# backward and forward compensated full for SDI approach and spatial filtered where motion compensation bad
#end sort SAD
#

#
# sort SDI for choose center, forward or forward compensation work with 2 sequential frames, for 3 need 2 pass filtering, for 4 3 pass
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# center SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# backward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
# forward SDI
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


# this place for repair now I am thinking

fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
StackVertical(Separatefields(source),fieldmaskedsdi,mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#Weave(fieldmaskedsdi)
# some kind comparing and weave for interlaced source, may be at double rate and for second pass not need bobbing

# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
threshsp=10
# threshold for pixels value absolute difference from 2 sequitial frames for spike detection
THP=string(threshsp)
threshavg=2
# threshold how much times actual SDI greater avearage and decimated SDI for remove false alarm for bad motion compensation
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
# SDI calculation
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
# SDI averaging
SDIAvgblk=SDIavg.PointResize(90,72)# for PAL (90,72), FOR NTSC (90,60)
# decimation
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
# median point at X position
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
#median point at Cross position
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
# multi median filtering for remove false alarm SDI
SDIavg=SDIAvgblk.PointResize(720,576)# for PAL (720,576), FOR NTSC (720,480)
# backward resize to original frame size
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
# comparing with actual SDI and if greater it is real spike, value 128 can increase to 192
# be carefull I can not find this at original paper, big value can loose spike, small false alarm
SDIad = (radvertmed>1) ? SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,radvertmed-1)) : SDIad
# rectangle size could one less than for bobnnmed, for thinner lines need additional pass with thiknessline=thicknessline-1 until thicknessline=1
SDIad = SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(thicknessline,0))
#remove line shotter than thicknessline+1
SDIadleft=SDIad.mt_luts(SDIad,mode="max",pixels=mt_freerectangle(-distser,0,0,0))
# fill distser pixels from spike left side
SDIadright=SDIad.mt_luts(SDIad,mode="max",pixels=mt_freerectangle(0,0,distser,0))
# fill distser pixels from spike right side
SDIad=mt_logic(SDIadleft,SDIadright,"min")
# if pixels exist at SDIadleft and SDIadright fill pixels between 2 short lines
SDIad = (radvertmed>1) ? SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,radvertmed-1)) : SDIad
# need additional vertical median filtering after joining short lines for remove thiner lines
SDImask=SDIad.mt_luts(SDIad,mode="min",pixels=mt_rectangle(halflength,0))
# remove spikes shorter than 2*halflength+1 pixels tune for source
SDImask=SDImask.HorlLinExp()
# if line have only one pixel line will be filled
SDIad=mt_logic(SDIad,SDImask,"min")
# combination source and adapting SDI for better robustness
SDIad = SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(thicknessline,0))
# additional remove short spikes which can be artefacts
return(SDIad)

}

# fill full horizontal line if exist one pixel per line
# basaed on Didee idea see masktools thread
function HorlLinExp(clip mask1) {

mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}
yup.

[mammo1789]

I know is dumb question but how to make only output one video ( don't need comparison) instead of stack horizontal 4

[yup]

mammo1789!
#StackVertical(Separatefields(source),fieldmaskedsdi,mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
Weave(fieldmaskedsdi)
Comment before stackvertical and uncomment before weave.
Comparing need for tuning parameters, output will be frame.
yup.

[mammo1789]

Thanks yup I figure it out after looking at the script ( for an hour :)). I'm still not very familiar with long avisynth scripts but I'm getting there.
Concerning result I listen to your advice of using multiple passes with lower threshold but I get blurry results( at least for my taste).
And on default its not doing anything to the lines i see you made long and short streaks script separate and i tried both but no avail
http://img838.imageshack.us/img838/5828/92187688.png

[mammo1789]

I uploaded original short part of the clip with problems 3mb mjpeg http://www.mediafire.com/?uz148p9dyky999z for you if you have the time to look at it, and what will you suggest me to do with the script what parameter( or all the script that you will be using to get rid of them) I will very appreciate
Thanks Yup

[yup]

mammo1789!
Parameters could tune for source.
Try decrease parameter halflength (remove lines less than 2*halflength+1). If halflength high frame will be untouched.
Also try original script with StackVertical, third low part show difference between original and filtered (if You see only gray script untouched frame) and tune halflength and thicknessline parameter (for last more than 3 introduce a lot of artifacts) at low part you could see removed lines (white or black).
For Your sample need use version for long lines.
try updated script
Global thicknessline=2# thickness horizontal spike in pixels at field scale (not frame)
Global radvertmed=2*thicknessline-1
Global halflength=18# detect line longer 2*halflength+1 pixels
#SetMemoryMax(768)
#SetMTMode(3,4)
source=AVISource("sel97.avi")
#source file
#SetMTMode(2,4)
source=source.AssumeTFF().ConvertToYV12(interlaced=true)
bobnn=source.nnedi3(field=-2,threads=1)
# bobbing for better motion estimation
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,radvertmed),U=3,V=3)
# vertical median filter
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
#dft filter remove segregation after median vertical filter
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnn, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnn, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnn, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnn, super, fw2, thSAD=16000, thSCD1=16000)
# motion estimation on filtered and compensating on source
bwabs1=mt_lutxy(bobnn,bc1,"x y - abs")
bwabs2=mt_lutxy(bobnn,bc2,"x y - abs")
fwabs1=mt_lutxy(bobnn,fc1,"x y - abs")
fwabs2=mt_lutxy(bobnn,fc2,"x y - abs")
# calculation absolute difference between source and compensated
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
# calculation spike detection index for center, forwad and backward
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
# calculating SAD for filtered for estimation better choice for motion compensation
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
# maximum SAD from forward and backward measure for quality center compensation
bwsad=mt_logic(mb1sad,mb2sad,"max")
#for backward compensation
fwsad=mt_logic(mf1sad,mf2sad,"max")
#for forward compensation
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
# inverting mask for using mt_merge
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
# motion compensated median filtering for center, backward and forward compensation
#
# sort SAD for finding better motion compensated median filtering, this approach can use without SDI for pixels and 1 line thickness stripes
mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?")
# center filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?")
# backward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?")
# forward filtered
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
#best value based on 3 motion compensated median time filtered value and spatial filtered
# where motion compensation bad, work only for one sequintial frame with spike
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
# backward and forward compensated full for SDI approach and spatial filtered where motion compensation bad
#end sort SAD
#

#
# sort SDI for choose center, forward or forward compensation work with 2 sequential frames, for 3 need 2 pass filtering, for 4 3 pass
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# center SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# backward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
# forward SDI
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


# this place for repair now I am thinking

fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
StackVertical(Separatefields(source),fieldmaskedsdi,mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#Weave(fieldmaskedsdi)
# some kind comparing and weave for interlaced source, may be at double rate and for second pass not need bobbing

# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
threshsp=10
# threshold for pixels value absolute difference from 2 sequitial frames for spike detection
THP=string(threshsp)
threshavg=2
# threshold how much times actual SDI greater avearage and decimated SDI for remove false alarm for bad motion compensation
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
# SDI calculation
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
# SDI averaging
SDIAvgblk=SDIavg.PointResize(90,72)# for PAL (90,72), FOR NTSC (90,60)
# decimation
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
# median point at X position
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
#median point at Cross position
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
# multi median filtering for remove false alarm SDI
SDIavg=SDIAvgblk.PointResize(720,576)# for PAL (720,576), FOR NTSC (720,480)
# backward resize to original frame size
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
# comparing with actual SDI and if greater it is real spike, value 128 can increase to 192
# be carefull I can not find this at original paper, big value can loose spike, small false alarm
SDIad = (radvertmed>1) ? SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,radvertmed-1)) : SDIad
# rectangle size could one less than for bobnnmed, for thinner lines need additional pass with thiknessline=thicknessline-1 until thicknessline=1
SDIad=SDIad.mt_luts(SDIad,mode="min",pixels=mt_rectangle(halflength,0))
# remove spikes shorter than 2*halflength+1 pixels and shrink longer
SDIad=SDIad.mt_luts(SDIad,mode="max",pixels=mt_rectangle(halflength,0))
# extend spikes longer than 2*halflength+1 pixels
return(SDIad)

}

# fill full horizontal line if exist one pixel per line
# basaed on Didee idea see masktools thread
function HorlLinExp(clip mask1) {

mask2 = mask1.mt_lut(Y=-255, u=-128, v=-128)
black = mask1.mt_lut(Y=0, u=-128, v=-128)

v1=interleave(mask1,black).assumefieldbased.assumetff.weave
v2=interleave(mask2,black).assumefieldbased.assumetff.weave

mt_hysteresis(v1,v2, u=-128, v=-128)
separatefields.selecteven.assumeframebased
}

yup.

[Mounir]

Error: Invalid arguments to Clense that's what i get. Pointing to line 52 in the script -> SDIc=SDIAdapt(bwabs1,fwabs1)

Which filters (and versions) do i need Yup for your last script , thanks

[yup]

Error: Invalid arguments to Clense that's what i get. Pointing to line 52 in the script -> SDIc=SDIAdapt(bwabs1,fwabs1)

Which filters (and versions) do i need Yup for your last script , thanks
Mounir!
You could use RemoveGrainT version dated 24:01:2008

http://home.arcor.de/kassandro/prerelease/RemoveGrainT-1.0.rar
yup.

[Mounir]

i get an error with removegrainT again "Unable to load removegrainT.dll error 0x7e

[yup]

Mounir!
All Kassandro plugin required avsrecursion.dll
http://home.arcor.de/kassandro/AvsRecursion/AvsRecursion.htm
Place dll at windows/system32 directory.
yup.

[Mounir]

I have it installed on sys32 already, im running win7 x64

[Didée]

On 64bit Windows, the "system32" folder (for 32bit applications) now is "SysWOW64". Has been mentioned a few dozen dozen times already.

[Mounir]

No luck on syswow64 folder --> "invalid args to function "clense"

Ok nevermind now the RemoveGrainT.dll is loaded w/o problem, script working

Well i can't say it's working great for long white stripes,i'll stick with manual work for now

[yup]

Mounir!
Can try RemovegrainTSSE2 instead of RemovegrainT, little faster.
mammo1789!
I try Your source starting point thicknessline=3 and halflength=8 remove some lines at 1 pass.
For one my source I made 2 pass with thicknessline=3 and after 2 pass with thicknessline=2.
You can try more pass with thicknessline=2.
See difference field for estimation removing lines.
One more, need tuning thicknessline and halflength parameter, if choose line thicker and longer for removing script make nothing.
yup.

[mammo1789]

I used what you suggested but i don't see any change the first is source the second is cleaned right? the look the same.
http://img21.imageshack.us/img21/7256/45528592.jpg

It's interesting that the script that you post it to me post 63 before so I can test it it did some removing but it has to be kept low bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,X -around 3 max ),U=3,V=3) instead it blurs very much.

The whole scripts ( long and short ones seems are less effective than that small script)
I'm I doing anything wrong.
If I understand you correctly i should load the script ( with 3 or 2 and 8 half and thik ) render it in vdub and load it again with the same parameters or 2 instead of 3 and so one right ?

[yup]

mammo1789!
I try Your short clip with thicknessline=3,2,1 and see at some frames removing lines (not all).
Script can remove no more than 2 sequential lines at time line. If have 4 sequential lines at time line need 2 or 3 pass.
Also try new version
Global thicknessline=2# thickness horizontal spike in pixels at field scale (not frame)
Global radvertmed=2*thicknessline-1
Global distser=25# distance between short lines
Global halflength=3# detect line longer 2*halflength+1 pixels
#SetMemoryMax(768)
#SetMTMode(3,4)
source=AVISource("selnew.avi")
#source file
#SetMTMode(2,4)
source=source.AssumeTFF().ConvertToYV12(interlaced=true)
bobnn=source.nnedi3(field=-2,threads=1)
# bobbing for better motion estimation
bobnnmed=bobnn.mt_luts(bobnn,mode="median",pixels=mt_rectangle(0,radvertmed),U=3,V=3)
# vertical median filter
bobnnmedf=bobnnmed.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false)
#dft filter remove segregation after median vertical filter
super=MSuper(bobnn)
superf=MSuper(bobnnmedf,chroma=false)
bw1 = MAnalyse(superf, blksize=8, isb = true, delta = 2, overlap=4, dct=5,chroma=false)
bw2 = MAnalyse(superf, blksize=8, isb = true, delta = 4, overlap=4, dct=5,chroma=false)
fw1 = MAnalyse(superf, blksize=8, isb = false,delta = 2, overlap=4, dct=5,chroma=false)
fw2 = MAnalyse(superf, blksize=8, isb = false,delta = 4, overlap=4, dct=5,chroma=false)
bc1 = MCompensate(bobnn, super, bw1, thSAD=16000, thSCD1=16000)
bc2 = MCompensate(bobnn, super, bw2, thSAD=16000, thSCD1=16000)
fc1 = MCompensate(bobnn, super, fw1, thSAD=16000, thSCD1=16000)
fc2 = MCompensate(bobnn, super, fw2, thSAD=16000, thSCD1=16000)
# motion estimation on filtered and compensating on source
bwabs1=mt_lutxy(bobnn,bc1,"x y - abs")
bwabs2=mt_lutxy(bobnn,bc2,"x y - abs")
fwabs1=mt_lutxy(bobnn,fc1,"x y - abs")
fwabs2=mt_lutxy(bobnn,fc2,"x y - abs")
# calculation absolute difference between source and compensated
SDIc=SDIAdapt(bwabs1,fwabs1)
SDIb=SDIAdapt(bwabs1,bwabs2)
SDIf=SDIAdapt(fwabs1,fwabs2)
# calculation spike detection index for center, forwad and backward
mb1sad=bobnnmedf.MMask(bw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mb2sad=bobnnmedf.MMask(bw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf1sad=bobnnmedf.MMask(fw1,kind=1,ml=100,Ysc=255, thSCD1=16000)
mf2sad=bobnnmedf.MMask(fw2,kind=1,ml=100,Ysc=255, thSCD1=16000)
# calculating SAD for filtered for estimation better choice for motion compensation
centersad=mt_logic(mf1sad,mb1sad,"max")#,U=2,V=2)
# maximum SAD from forward and backward measure for quality center compensation
bwsad=mt_logic(mb1sad,mb2sad,"max")
#for backward compensation
fwsad=mt_logic(mf1sad,mf2sad,"max")
#for forward compensation
centermask=mt_invert(centersad)
bwmask=mt_invert(bwsad)
fwmask=mt_invert(fwsad)
# inverting mask for using mt_merge
mcfcenter=clense(bc1, bobnn,fc1,increment=0, grey=false)
mcfbw=clense(bc1, bobnn,bc2,increment=0, grey=false)
mcffw=clense(fc1, bobnn,fc2,increment=0, grey=false)
# motion compensated median filtering for center, backward and forward compensation
#
# sort SAD for finding better motion compensated median filtering, this approach can use without SDI for pixels and 1 line thickness stripes
mt_logic(mt_lutxy(centersad,bwsad,"x y <= "), mt_lutxy(centersad,fwsad,"x y <="),"and")
maskcentersad=mt_lutxy(last,centersad,"x 255 y - 0 ?")
# center filtered
mt_logic(mt_lutxy(bwsad,centersad,"x y < "), mt_lutxy(bwsad,fwsad,"x y <="),"and")
maskbwsad=mt_lutxy(last,bwsad,"x 255 y - 0 ?")
# backward filtered
mt_logic(mt_lutxy(fwsad,centersad,"x y < "), mt_lutxy(fwsad,bwsad,"x y < "),"and")
maskfwsad=mt_lutxy(last,fwsad,"x 255 y - 0 ?")
# forward filtered
#
mcfmaskedsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcfmaskedsad=mt_merge(mcfmaskedsad,mcffw,maskfwsad,luma=true)
mcfcentersad=mt_merge(mcfmaskedsad,mcfcenter,maskcentersad,luma=true)
#best value based on 3 motion compensated median time filtered value and spatial filtered
# where motion compensation bad, work only for one sequintial frame with spike
mcfbwsad=mt_merge(bobnnmed,mcfbw,maskbwsad,luma=true)
mcffwsad=mt_merge(bobnnmed,mcffw,maskfwsad,luma=true)
# backward and forward compensated full for SDI approach and spatial filtered where motion compensation bad
#end sort SAD
#

#
# sort SDI for choose center, forward or forward compensation work with 2 sequential frames, for 3 need 2 pass filtering, for 4 3 pass
maskcentersdi=mt_logic(mt_logic(mt_lutxy(SDIc,SDIb,"x y >= "), mt_lutxy(SDIc,SDIf,"x y >="),"and"),SDIc,"and").mt_lut("x 255 0 ?")
# center SDI
maskbwsdi=mt_logic(mt_lutxy(SDIb,SDIc,"x y > "), mt_lutxy(SDIb,SDIf,"x y >="),"and").mt_lut("x 255 0 ?")
# backward SDI
maskfwsdi=mt_logic(mt_lutxy(SDIf,SDIc,"x y > "), mt_lutxy(SDIf,SDIb,"x y >"),"and").mt_lut("x 255 0 ?")
# forward SDI
#
maskedsdi=mt_merge(bobnn,mcfcentersad,maskcentersdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcffwsad,maskfwsdi,luma=true)
maskedsdi=mt_merge(maskedsdi,mcfbwsad,maskbwsdi,luma=true)
#end sort SDI
#


# this place for repair now I am thinking

fieldmaskedsdi=maskedsdi.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
#fieldmaskedsdi=SDIc.AssumeTFF().SeparateFields().SelectEvery(4,0,3)
StackVertical(Separatefields(source),fieldmaskedsdi,mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#StackVertical(Separatefields(source),mt_makediff(fieldmaskedsdi,Separatefields(source),u=3,v=3))
#Weave(fieldmaskedsdi)
# some kind comparing and weave for interlaced source, may be at double rate and for second pass not need bobbing

# Spike Detection Function
#

function SDIAdapt(clip b1,clip b2)
{
threshsp=10
# threshold for pixels value absolute difference from 2 sequitial frames for spike detection
THP=string(threshsp)
threshavg=2
# threshold how much times actual SDI greater avearage and decimated SDI for remove false alarm for bad motion compensation
THAVG=string(threshavg)
SDI=mt_lutxy(b1,b2,"x "+THP+" > y "+THP+" > | 1 x y - x y + / abs - 255 * 0 ? ")
# SDI calculation
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
# SDI averaging
SDIAvgblk=SDIavg.PointResize(90,72)# for PAL (90,72), FOR NTSC (90,60)
# decimation
SDIavgblkX=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 -1 0 0 -1 1 1 1 -1 -1")
# median point at X position
SDIavgblkCross=SDIAvgblk.mt_luts(SDIAvgblk,mode="median",pixels="1 0 0 0 -1 0 0 1 0 -1")
#median point at Cross position
SDIAvgblk=clense(SDIAvgblk,SDIavgblkCross,SDIavgblkX,increment=0, grey=true).mt_lut("x")
# multi median filtering for remove false alarm SDI
SDIavg=SDIAvgblk.PointResize(720,576)# for PAL (720,576), FOR NTSC (720,480)
# backward resize to original frame size
SDIad=mt_lutxy(SDI,SDIavg,"x "+THAVG+" y * > x 128 > & 255 0 ?")
# comparing with actual SDI and if greater it is real spike, value 128 can increase to 192
# be carefull I can not find this at original paper, big value can loose spike, small false alarm
SDIad = (radvertmed>1) ? SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(0,radvertmed-1)) : SDIad
# rectangle size could one less than for bobnnmed, for thinner lines need additional pass with thiknessline=thicknessline-1 until thicknessline=1
SDIad = SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(thicknessline,0))
#remove line shotter than thicknessline+1
SDIadleft=SDIad.mt_luts(SDIad,mode="max",pixels=mt_freerectangle(-distser,0,0,0))
# fill distser pixels from spike left side
SDIadright=SDIad.mt_luts(SDIad,mode="max",pixels=mt_freerectangle(0,0,distser,0))
# fill distser pixels from spike right side
SDIadser=mt_logic(SDIadleft,SDIadright,"min")
# if pixels exist at SDIadleft and SDIadright fill pixels between 2 short lines
SDIadser = (radvertmed>1) ? SDIadser.mt_luts(SDIadser,mode="median",pixels=mt_rectangle(0,radvertmed-1)) : SDIadser
# ADDITIONAL VERTICAL MEDIAN FILTER FOR REMOVE THIN JOIN PART
SDImask=SDIadser.mt_luts(SDIadser,mode="min",pixels=mt_rectangle(halflength,0))
# remove spikes shorter than 2*halflength+1 pixels tune for source
SDImask=SDImask.mt_luts(SDImask,mode="max",pixels=mt_rectangle(halflength,0))
# expand after shrinking
SDIad=mt_logic(SDIad,SDImask,"min")
# combination source and adapting SDI for better robustness
# SDIad = SDIad.mt_luts(SDIad,mode="median",pixels=mt_rectangle(thicknessline,0))
# additional remove short spikes which can be artefacts
return(SDIad)

}


Now I am not ready, but result better than before.
yup.

[yup]

Some examples
http://thumbnails32.imagebam.com/17940/babd32179392926.jpg (http://www.imagebam.com/image/babd32179392926) http://thumbnails67.imagebam.com/17940/2b5abc179392957.jpg (http://www.imagebam.com/image/2b5abc179392957) http://thumbnails62.imagebam.com/17940/57a3d3179392979.jpg (http://www.imagebam.com/image/57a3d3179392979) http://thumbnails58.imagebam.com/17940/da5cd2179392993.jpg (http://www.imagebam.com/image/da5cd2179392993) http://thumbnails64.imagebam.com/17940/f6c02a179393011.jpg (http://www.imagebam.com/image/f6c02a179393011) http://thumbnails43.imagebam.com/17940/0cbefa179393032.jpg (http://www.imagebam.com/image/0cbefa179393032)
Two last sample show how work different thicknessline.
yup.

[GMJCZP]

I suggest you try your script something like:

mergechroma (dctfun4b (5.5)) # set to your like

or

mergechroma (ReduceFlicker) or other function within the plugin with mergechroma

[yup]

GMJCZP!
Now script find place for luma spike and repairing including chroma. I see that at place where luma spike exist also is chroma dig wider than luma spike.
I am planing also find solution for this.
yup.

[GMJCZP]

I made ​​the suggestions because you saw colored lines.
There are other alternatives to use: DeScratch, DeSpot, or the wonderful idea (the best) of ​​John Meyer, RemoveDirtMC.
Good luck, yup!

[yup]

Hi all!
Upload avsi version of script with small changes (work little better with thicker lines 3 and 4 due to use gaussianblur instead of average filter before decimation SDI mask). See first post.
yup.

[Mounir]

I've tried your new script now, i find it very slow (!) and not necessarily better than Devcr which i like. Maybe you could borrow some ideas from devcr.

[yup]

Mounir!
Yes slow!
:thanks: for testing.
Script based on motion compensated median filtering and analyzing not only current frame but 2 before and 2 ahead.
It is spent CPU time.
Short test with devcr, not remove all horizontal lines especially series (not solid with series short lines), introduce artifacts.
I am sure that script can be optimized. For this we need help from guru script writers.
yup.

[greymouse]

hi Yup, was wondering if I could get some help with one of your scripts, it works as it should as it creates a mask with the lines I want to remove from each frame, but how do I apply the mask to actually remove the black lines?

Setmemorymax(512)

AviSource("F:\Video-Capture\Home Improvement.avi") #ok

AssumeBFF()
ConvertToYV12(interlaced=true)
fields=SeparateFields()
backward_vec = fields.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1,dct=1)
forward_vec = fields.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1,dct=1)
fc = fields.MVFlow(forward_vec, idx=1, thSCD1=10000)
bc = fields.MVFlow(backward_vec, idx=1, thSCD1=10000)
masksdi=SDIPixelMask(bc, fields, fc, 50, 0.8) # define your thresholds here
StackVertical(fields,masksdi)
#Weave()


# this is for Abs(p - f) or Abs(p - b)
Function absdiff(clip c1, clip c2) { return mt_lutxy(c1, c2, "x y - abs") }

# this build the expression for the SDI function
Function SDI_RPN(string expr_d1, string expr_d2, string expr_t1, string expr_t2) {
_d1 = expr_d1 + " "
_d2 = expr_d2 + " "
_t1 = expr_t1 + " "
_t2 = expr_t2 + " "
return _d1 + _t1 + "> " + _d2 + _t1 + "> | 1.0 " \
+ _d1 + _d2 + "- " + _d1 + _d2 + "+ / abs - 0.0 ? " \
+ _t2 + "> 1.0 " + _d1 + _t1 + "> " + _d2 + _t1 + "> | 1.0 " \
+ _d1 + _d2 + "- " + _d1 + _d2 + "+ / abs - 0.0 ? "
}

# this creates the mask
Function SDIPixelMask(clip prev, clip curr, clip next, float t1, float t2) {

c_d1 = absdiff(curr, next)
c_d2 = absdiff(curr, prev)

# since SDI returns 0..1 we multiply with 255

mask = mt_lutxy(c_d1, c_d2, SDI_RPN("x", "y", String(t1), String(t2)) + " 255 *")
return mask
}

:thanks:

[greymouse]

the script is working perfectly at creating a mask which has the black lines I want to remove in while, but how do I use that mask to remove the lines themselves? thank you

[greymouse]

I know that overlay () will use the mask but is there another way? Is that the only function which you pass a mask through?

[poisondeathray]

I know that overlay () will use the mask but is there another way? Is that the only function which you pass a mask through?

mt_merge can use a mask as well

[greymouse]

mt_merge can use a mask as well

thanks, still unsure as to how I can use the mask created in the above script to remove the lines

[greymouse]

my thinking at the moment is to create a clip with only the mask, also create a 'fixed' clip which has all the black lines removed, then merge the two together, i guess they'll work? using overlay() and one of its parameters to "blend" etc the two together

[greymouse]

Mt_merge should do the job thanks!

[yup]

greymouse & poisondeathray!

Now I am out my working horse (after weekend). This spring I am little rewriting script for last version avisynth and plugin.
For some source work relative good.
yup.

[greymouse]

sounds good yup! your script works nicely :cool:

[yup]

greymouse!

See link for updated version:
https://www.dropbox.com/s/3b4p4zr6cjaqfum/pass1med3n.avs?dl=0
For script need Avisynth+ and all updated plugin. 32 bit version work like before with old variableblur plugin.
SDIavg=SDI.mt_luts(SDI,mode="avg",pixels=mt_square(5))
#SDIavg=SDI.gaussianblur(varY=40)
#SDIavg=SDI.dfttest(tbsize=1,ftype=1,sbsize=8,sosize=6,sigma=1000,U=false,V=false,threads=1)
# SDI averaging

If using for averaging gaussianblur (when I writing script, give me better result), only 32 bit version.
If two other mt_luts or dffttest You can use 64 bit (some speed up and stability).
Now I try Mohan plugin GBlur.
One more improvement, support YV16 colorspace (lossless for YUY2 captured source), before only YV12.
Any suggestion welcome.
yup.

[Mounir]

You can try Yup script for an interlaced source:
http://forum.doom9.org/showthread.php?p=943287#post943287