Moving dots I cant seem to get rid of

[DarkDream]

Ive seen and fixed plenty of dot crawl in sources but this really takes the cake at being a pain in the ass.

Perhaps its not dot crawl or something Ive not yet come across?

These lines during playback seem to have plenty of moving dots running along the lines. It looks pretty much like dot crawl to me but seems a little different.

I used the dot crawl filter checkmate() and an antialiaser and they still wont go away. They seem to be around a bit less afterwards though.

Its several times more noticeable in the DVD video source.

any suggestion?

Heres a sample vob clip unaltered video. I used DGIndex and saved project + demuxed video. Its 1 minute 28 seconds long.

They show up alot once it gets to around the 1 minute point and you see the guy wearing the red jacket. They show up like that in various places all through the entire video and DVD.

http://glumbouploads.com/0qoh16vn3j9p.html

Here are a few screenshots of the source after I have filtered it with checkmate() and an antialiaser.

http://img59.imageshack.us/img59/4408/dot1.png

http://img11.imageshack.us/img11/6936/dot2p.png

http://img839.imageshack.us/img839/1805/dot3.png

http://img14.imageshack.us/img14/4271/dot4.png

[cretindesalpes]

Screenshots of a filtered video are almost useless to make a diagnostic. Please post an untouched sample of the DVD VOB.

[DarkDream]

Screenshots of a filtered video are almost useless to make a diagnostic. Please post an untouched sample of the DVD VOB.

Its been so long since I needed to post an unaltered sample that I forget what I used to use.

was it dgindex?

how can I get you a vob file from the dvd without it being over 1GB?

[StainlessS]

Yes, just mark with the sliders in DGIndex.

EDIT: Then "Save Project".

[DarkDream]

Yes, just mark with the sliders in DGIndex.

EDIT: Then "Save Project".

Alrighty, Ill work on getting the sample up asap if I can figure it out again. In the meantime heres a couple screenshots of the video unaltered coming from the original DVD. I hope these can help some. The moving dots are way worse before I filtered it.

http://img528.imageshack.us/img528/2756/vts011vobsnapshot021520.png

http://img803.imageshack.us/img803/2756/vts011vobsnapshot021520.png

http://img17.imageshack.us/img17/3065/vts011vobsnapshot062020.png

[DarkDream]

Ok heres the sample clip requested. I used DGIndex and saved project + demuxed video. Its 1 minute 28 seconds long.

They show up alot once it gets to around the 1 minute point and you see the guy wearing the red jacket. They show up like that in various places all through the entire video and DVD.

http://glumbouploads.com/0qoh16vn3j9p.html

[DarkDream]

A shorter 30 second clip can be found here

http://glumbouploads.com/jq5qwkb0oplb.html

[librarian]

Try this script:
Mpeg2Source("C:\Script Avisynth\Sample\Sample\VTS_01_1.demuxed.d2v")

telecide()
decimate()
o=last
spline36resize(640,480)
removegrain(19,22)
bicubicresize(720,480)
awarpsharp2(128,chroma=3)
contrasharpening(last,o)
return last



function ContraSharpening (clip denoised, clip original)
{
s = denoised.minblur(1,1) # Damp down remaining spots of the denoised clip.
allD = mt_makediff(original,denoised) # The difference achieved by the denoising.
ssD = mt_makediff(s,s.removegrain(11,-1)) # The difference of a simple kernel blur.
ssDD = ssD.repair(allD,1) # Limit the difference to the max of what the denoising removed locally.
ssDD = SSDD.mt_lutxy(ssD,"x 128 - abs y 128 - abs < x y ?") # abs(diff) after limiting may not be bigger than before.

denoised.mt_adddiff(ssDD,U=2,V=2) # Apply the limited difference. (Sharpening is just inverse blurring.)

return( last )
}
# Nifty Gauss/Median combination
function MinBlur(clip clp, int r, int "uv")
{
uv = default(uv,3)
uv2 = (uv==2) ? 1 : uv
rg4 = (uv==3) ? 4 : -1
rg11 = (uv==3) ? 11 : -1
rg20 = (uv==3) ? 20 : -1
medf = (uv==3) ? 1 : -200


RG11D = (r==1) ? mt_makediff(clp,clp.removegrain(11,rg11),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20).removegrain(20,rg20),U=uv2,V=uv2)
RG4D = (r==1) ? mt_makediff(clp,clp.removegrain(4,rg4),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.medianblur(2,2*medf,2*medf),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.medianblur(3,3*medf,3*medf),U=uv2,V=uv2)
DD = mt_lutxy(RG11D,RG4D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",U=uv2,V=uv2)
clp.mt_makediff(DD,U=uv,V=uv)
return(last)
}

[DarkDream]

Thanks for the script, Ill give it a try later today and post about how it worked.

Judging by the script you've given, it has something to do with grain and removing grain to get rid of those dots?

Is there a name or term for what that is in the video?

[librarian]

Thanks for the script, Ill give it a try later today and post about how it worked.

Judging by the script you've given, it has something to do with grain and removing grain to get rid of those dots?

Is there a name or term for what that is in the video?

Removegrain doesn't remove grain only (with mode from 13 to 16 it is a deinterlacer!). I used RG mode 19 and 22 (blur and median blur), then contrasharpening to limit the blurring.
Sorry I dont know the name of this particular artifact.

[DarkDream]

Try this script:
Mpeg2Source("C:\Script Avisynth\Sample\Sample\VTS_01_1.demuxed.d2v")

telecide()
decimate()
o=last
spline36resize(640,480)
removegrain(19,22)
bicubicresize(720,480)
awarpsharp2(128,chroma=3)
contrasharpening(last,o)
return last



function ContraSharpening (clip denoised, clip original)
{
s = denoised.minblur(1,1) # Damp down remaining spots of the denoised clip.
allD = mt_makediff(original,denoised) # The difference achieved by the denoising.
ssD = mt_makediff(s,s.removegrain(11,-1)) # The difference of a simple kernel blur.
ssDD = ssD.repair(allD,1) # Limit the difference to the max of what the denoising removed locally.
ssDD = SSDD.mt_lutxy(ssD,"x 128 - abs y 128 - abs < x y ?") # abs(diff) after limiting may not be bigger than before.

denoised.mt_adddiff(ssDD,U=2,V=2) # Apply the limited difference. (Sharpening is just inverse blurring.)

return( last )
}
# Nifty Gauss/Median combination
function MinBlur(clip clp, int r, int "uv")
{
uv = default(uv,3)
uv2 = (uv==2) ? 1 : uv
rg4 = (uv==3) ? 4 : -1
rg11 = (uv==3) ? 11 : -1
rg20 = (uv==3) ? 20 : -1
medf = (uv==3) ? 1 : -200


RG11D = (r==1) ? mt_makediff(clp,clp.removegrain(11,rg11),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20).removegrain(20,rg20),U=uv2,V=uv2)
RG4D = (r==1) ? mt_makediff(clp,clp.removegrain(4,rg4),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.medianblur(2,2*medf,2*medf),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.medianblur(3,3*medf,3*medf),U=uv2,V=uv2)
DD = mt_lutxy(RG11D,RG4D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",U=uv2,V=uv2)
clp.mt_makediff(DD,U=uv,V=uv)
return(last)
}





It blurred the video terribly and made alot of the black lines very thin making it look terrible in comparison.

The ending result makes the video look like shit to where its far from acceptable in quality but it did get rid of the dots.

[Reel.Deel]

Hello DarkDream, this maybe overkill but you may want to try this. It's slow but is does seem to get rid of *almost* all artifacts. The orginal post of this script can be found here. (http://forum.doom9.org/showthread.php?p=1584186#post1584186)

# Required plugins:
# -----------------
# dgdecode.dll
# TIVTC.dll
# mt_masktools.dll
# removegrain.dll
# repair.dll
# frfun7.dll
# checkmate.dll

mpeg2source("VTS_01_1.demuxed.d2v")

checkmate()

borders = bicubicresize(width()/2,height()/2).bicubicresize(width()+8,height())
stackhorizontal(borders.crop(0,0,4,0),last,borders.crop(width()+4,0,0,0))

o=last ox=o.width() oy=o.height()

drr = o.separatefields().vinverse2().weave()
both = stackhorizontal(drr,o)
IVTC = both.tfm(pp=0,clip2=stackhorizontal(drr,drr)).tdecimate(mode=1)
raw = IVTC.crop(ox,0,ox,oy)
clean1 = IVTC.crop(0,0,ox,oy)

D1 = mt_makediff(raw,clean1)
D8 = D1.bicubicresize(ox/2-72,oy).blur(1,0).bicubicresize(ox,oy,1,0)
D9 = mt_lutxy(D1,D8,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")
clean1a = clean1.mt_adddiff(D9,U=2,V=2)
clean1b = clean1a.frfun7(1.01,8,1)

allD = mt_makediff(raw,clean1b).greyscale()
shrpD = mt_makediff(clean1b,clean1b.removegrain(20))
DD = shrpD.repair(allD,13,0).mt_lutxy(shrpD,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")
clean1c = clean1b.mt_lutxy(clean1a,"x 3 + y < x 2 + x y < x 1 + x 3 - y > x 2 - x y > x 1 - x ? ? ? ?",U=2,V=2)
\.mt_adddiff(DD.sbr(),U=2,V=2)

raw = raw.crop(4,0,-4,0,true)
clean1c = clean1c.crop(4,0,-4,0,true)

#interleave(raw,clean1c) # interleave "only-IVTC" with result of full filterchain, for frame comparison
# clean1c # just result
return(clean1c)

#--- end of script ---#

#######################################################

#--- Helper functions below ---#

function Vinverse2(clip clp, float "sstr", int "amnt", int "uv")
{
uv = default(uv,3)
sstr = default(sstr,2.7)
amnt = default(amnt,255)
uv2 = (uv==2) ? 1 : uv
STR = string(sstr)
AMN = string(amnt)
vblur = clp.sbrV()
vblurD = mt_MakeDiff(clp,vblur,U=uv2,V=uv2)
Vshrp = mt_LutXY(vblur,vblur.mt_convolution("1","1 2 1"),expr="x x y - "+STR+" * +",U=uv2,V=uv2)
VshrpD = mt_MakeDiff(Vshrp,vblur,U=uv2,V=uv2)
VlimD = mt_LutXY(VshrpD,VblurD,expr="x 128 - y 128 - * 0 < x 128 - abs y 128 - abs < x y ? 128 - 0.25 * 128 + x 128 - abs y 128 - abs < x y ? ?",U=uv2,V=uv2)
mt_AddDiff(Vblur,VlimD,U=uv,V=uv)
(amnt>254) ? last : (amnt==0) ? clp : mt_LutXY(clp,last,expr="x "+AMN+" + y < x "+AMN+" + x "+AMN+" - y > x "+AMN+" - y ? ?",U=uv,V=uv)
return(last)
}

function sbr(clip o) {
rg11=o.removegrain(11)
rg11D=mt_makediff(o,rg11)
rg11DD=mt_makediff(rg11D,rg11D.removegrain(11)).mt_lutxy(rg11D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")
o.mt_makediff(rg11DD,U=2,V=2)
}

function sbrV(clip o) {
rg11=o.mt_convolution("1","1 2 1")
rg11D=mt_makediff(o,rg11)
rg11DD=mt_makediff(rg11D,rg11D.mt_convolution("1","1 2 1")).mt_lutxy(rg11D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")
o.mt_makediff(rg11DD,U=2,V=2)
}

[lansing]

well the basic concept of removing the dots is, mask out all the lines(edges) in the image, and then apply blur to the lines to a point where the dots blend into a line, and then merge the mask back to the image.
Unfortunately, the edge masking filter is not smart enough to filter only the line with dots, so you'll end up with a blurrier image overall, with the "good" areas also blurred (smoothed) out.

[sirt]

Do you think you can make it "smarter" in this case ?

[DarkDream]

None of the above scripts worked well enough. They got rid of the dots but it made the video look like shit so I played around with my own combination of things and didnt even bother with the above submissions.

Actually, I finally got it quite well and I didnt even have to ruin much if anything that I can see. I dont notice anything going bad about it really. I didnt need any masking that I know of and I didnt really blur the image.

Heres some comparison screenshots.

(This will be much easier for you to see changes if you save all the pictures and then flip back and forth between them on your computer one right after another. Then you can see the changes easily because the picture wont be moving and will be switching back and forth between them)


TOP = My original encode that I posted shots of with dots
BOTTOM = The one I just did last night that I was going to see how it turned out


Ok in this first example, 99% of the dots on the shoulders are gone and I didnt really notice anything ruined about it. The new one even looks a bit sharper I think. If you look around the eyes, it looks like the new one might have darkened/thickened black lines in some areas of the video a tiny bit but no one will really notice during playback unless they had the original footage to compare to it. I could barely tell in the screenshots.


http://img191.imageshack.us/img191/2928/noqtgmc3.png
http://img145.imageshack.us/img145/5349/qtgmc3.png


Thats much better than the original footage as you can see

http://img17.imageshack.us/img17/3065/vts011vobsnapshot062020.png



In this next example again 99% of the dots are gone and I dont really see anything gone bad yet. Again I think I see black lines barely getting thicker/darker in some spots but without footage to compare and being picky I dont really think it means too much or is a bad thing. The only real change I notice between the two besides the dots gone is the bottom left corner of that opened door when I flip back and forth between the two quickly.


http://img534.imageshack.us/img534/2161/noqtgmc2.png
http://img217.imageshack.us/img217/6168/qtgmc2ver2.png

[Guest]

So what was your script that produced these results?

[DarkDream]

So what was your script that produced these results?

I added in

checkmate(tthr2=0) ------- [[[did (tthr2=0) so artifacts wouldnt show up during rapid movement]]]

QTGMC(preset="Slow", InputType=1) ------- [[[This is what I added in last night that took away the rest of the dots with ease]]]

and I added an antialiaser of my choice, a denoiser of my choice, and a sharpener of my choice at the end to get rid of the rest of the dots and to sharpen the image.

The antialiaser and QTGMC are what got rid of them completely. In my first encode I only had the antialiaser and checkmate without QTGMC. They reduced them alot but didnt fully get rid of them alone.

[Guest]

Can you please post the whole script for us?

[DarkDream]

Can you please post the whole script for us?


LoadPlugin("C:\Program Files (x86)\megui\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("C:\Documents and Settings\Administrator\Desktop\Zoids New Century Zero\Disc 1\Episode 1\VTS_01_1.d2v", info=3)
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, interlaced=true, threads=0)
checkmate(tthr2=0)
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\TIVTC.dll")
tfm(order=-1).tdecimate(mode=1,hybrid=1)
QTGMC(preset="Slow", InputType=1)
crop( 8, 0, -8, 0)
LanczosResize(640,480) # Lanczos (Sharp)
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\UnDot.dll")
Undot() # Minimal Noise
[[[denoiser of my choice]]]
[[[antialiaser of my choice]]]
[[[sharpener of my choice]]]


thats the entire thing

[DarkDream]

Can you please post the whole script for us?



DGDecode_mpeg2source("C:\Documents and Settings\Administrator\Desktop\Zoids New Century Zero\Disc 1\Episode 1\VTS_01_1.d2v", info=3)
ColorMatrix(hints=true, interlaced=true, threads=0)
checkmate(tthr2=0)
tfm(order=-1).tdecimate(mode=1,hybrid=1)
QTGMC(preset="Slow", InputType=1)
crop( 8, 0, -8, 0)
LanczosResize(640,480) # Lanczos (Sharp)
Undot() # Minimal Noise
[[[denoiser of my choice]]]
[[[antialiaser of my choice]]]
[[[sharpener of my choice]]]


thats the entire thing

[sirt]

It looks like he has gotten satisfied with the result in spite of the fact combining tfm and QTGMC is certainly a bad idea and he doesn't want to share with us ! Too bad !

[DarkDream]

Can you please post the whole script for us?


DGDecode_mpeg2source("C:\Documents and Settings\Administrator\Desktop\Zoids New Century Zero\Disc 1\Episode 1\VTS_01_1.d2v", info=3)
ColorMatrix(hints=true, interlaced=true, threads=0)
checkmate(tthr2=0)
tfm(order=-1).tdecimate(mode=1,hybrid=1)
QTGMC(preset="Slow", InputType=1)
crop( 8, 0, -8, 0)
LanczosResize(640,480) # Lanczos (Sharp)
Undot() # Minimal Noise
[[[Denoiser of my choice]]]
[[[Antialiaser of my choice]]]
[[[Sharpener of my choice]]]


Thats the whole script I used



It looks like he has gotten satisfied with the result in spite of the fact combining tfm and QTGMC is certainly a bad idea and he doesn't want to share with us ! Too bad !

If you already know what I did, and I already said what I used then I dont see why its "too bad" if I supposedly wasnt going to share. It sounds like you already know what I did.

[lansing]

he's using the deshimmering side of QTGMC from the QTGMC document. However the downside of it is that he will get blended frame(ghosting) because of the temporal smoothing like this:

[DarkDream]

he's using the deshimmering side of QTGMC from the QTGMC document. However the downside of it is that he will get blended frame(ghosting) because of the temporal smoothing like this:

I didnt see any ghosting during playback and I watched it a few times to look for it. Only when slowly viewing one frame at a time in an editor or something that can play one frame at a time can I even see anything that looks like ghosting once in a rare while. Also, most of its in the unaltered source as well it looked like.

[DarkDream]

Can you please post the whole script for us?


DGDecode_mpeg2source("C:\Documents and Settings\Administrator\Desktop\Zoids New Century Zero\Disc 1\Episode 1\VTS_01_1.d2v", info=3)
ColorMatrix(hints=true, interlaced=true, threads=0)
checkmate(tthr2=0)
tfm(order=-1).tdecimate(mode=1,hybrid=1)
QTGMC(preset="Slow", InputType=1)
crop( 8, 0, -8, 0)
LanczosResize(640,480) # Lanczos (Sharp)
Undot() # Minimal Noise
[[[Denoiser of my choice]]]
[[[Antialiaser of my choice]]]
[[[Sharpener of my choice]]]


Thats the whole script I used



It looks like he has gotten satisfied with the result in spite of the fact combining tfm and QTGMC is certainly a bad idea and he doesn't want to share with us ! Too bad !

If you already know what I did, and I already said what I used then I dont see why its "too bad" if I supposedly wasnt going to share. It sounds like you already know what I did.

[Guest]

[[[Denoiser of my choice]]]
[[[Antialiaser of my choice]]]
[[[Sharpener of my choice]]]
Why are you hiding the full script from us? Please provide the entire unobfuscated script. Thank you.

All your links are dead now too. Have you removed the samples?

I have to say, this kind of behavior is not appreciated here and you risk your reputation here with this kind of thing. We are all about open sharing and pursuit of video truth.

[DarkDream]

Why are you hiding the full script from us? Please provide the entire unobfuscated script. Thank you.

All your links are dead now too. Have you removed the samples?

I have to say, this kind of behavior is not appreciated here and you risk your reputation here with this kind of thing. We are all about open sharing and pursuit of video truth.

I dont see whats wrong with what I put. I was honest about it, just go ahead and pick whatever sharpener antialiaser and denoiser you want to its that simple it doesnt really matter. I hid nothing because that part is up to the user what they want to pick.

I didnt remove any samples, perhaps they got taken down for whatever reason.

[Guest]

Why won't you tell us what *you* used?

I don't know about these things and I need your help to make a functioning script. Thank you.

Also please re-upload the source sample.

[DarkDream]

Why won't you tell us what *you* used?

I don't know about these things and I need your help to make a functioning script. Thank you.

Also please re-upload the source sample.

Ill tell, I dont have to hide anything. I thought what I told was enough because its what I used.

You mean to tell me you have almost 20,000 posts helping people and talking in threads, you hang out in the avisynth usage section, from the looks of it you have administrative abilities, and you dont know how to make a functioning script? :confused: somethin smells fishy >_> lol

sure, Ill gladly reupload when I wake up tomorrow and get back home. I dont know why it went down, Ill see if I can find out. if it keeps happening I will just use another file host I guess so it wont go back down :)

[Guest]

Banned for multiple accounts. See #14 here:

http://forum.doom9.org/forum-rules.htm