Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 Doom9's Forum Dot Crawl issue
 Register FAQ Calendar Search Today's Posts Mark Forums Read

 23rd July 2012, 16:16 #61  |  Link jmac698 Registered User   Join Date: Jan 2006 Posts: 1,865 Source Filter means, a way to open your video file. For example, va=DirectShowSource("music video.mpg") You don't actually need all of his script. For example, I believe the purpose of the SmoothFilter line was just to make the video look better. The main line to remove dot crawl could just be a quiver command. I don't know what he meant by EmuFFT. Perhaps you can work on another project until you get a reply in some days.
 23rd July 2012, 20:35 #62  |  Link Didée Registered User   Join Date: Apr 2002 Location: Germany Posts: 5,394 How do you make a fishing net? -- Take ninety-thousand holes, and knot them together. Code: # Required plugins: # ----------------- # dgdecode.dll # TIVTC.dll # mt_masktools.dll # removegrain.dll # repair.dll # frfun7.dll mpeg2source("G:\Video\_Work\_dotcrawl\dot_crawled.d2v") borders = bicubicresize(width()/2,height()/2).bicubicresize(width()+8,height()) stackhorizontal(borders.crop(0,0,4,0),last,borders.crop(width()+4,0,0,0)) o=last ox=o.width() oy=o.height() drr = o.separatefields().vinverse2().weave() both = stackhorizontal(drr,o) IVTC = both.tfm(pp=0,clip2=stackhorizontal(drr,drr)).tdecimate() raw = IVTC.crop(ox,0,ox,oy) clean1 = IVTC.crop(0,0,ox,oy) D1 = mt_makediff(raw,clean1) D8 = D1.bicubicresize(ox/2-72,oy).blur(1,0).bicubicresize(ox,oy,1,0) D9 = mt_lutxy(D1,D8,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?") clean1a = clean1.mt_adddiff(D9,U=2,V=2) clean1b = clean1a.frfun7(1.01,8,1) allD = mt_makediff(raw,clean1b).greyscale() shrpD = mt_makediff(clean1b,clean1b.removegrain(20)) DD = shrpD.repair(allD,13,0).mt_lutxy(shrpD,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?") clean1c = clean1b.mt_lutxy(clean1a,"x 3 + y < x 2 + x y < x 1 + x 3 - y > x 2 - x y > x 1 - x ? ? ? ?",U=2,V=2) \.mt_adddiff(DD.sbr(),U=2,V=2) raw = raw.crop(4,0,-4,0,true) clean1c = clean1c.crop(4,0,-4,0,true) interleave(raw,clean1c) # interleave "only-IVTC" with result of full filterchain, for frame comparison # clean1c # just result return(last) #--- end of script ---# ####################################################### #--- Helper functions below ---# function Vinverse2(clip clp, float "sstr", int "amnt", int "uv") { uv = default(uv,3) sstr = default(sstr,2.7) amnt = default(amnt,255) uv2 = (uv==2) ? 1 : uv STR = string(sstr) AMN = string(amnt) vblur = clp.sbrV() vblurD = mt_MakeDiff(clp,vblur,U=uv2,V=uv2) Vshrp = mt_LutXY(vblur,vblur.mt_convolution("1","1 2 1"),expr="x x y - "+STR+" * +",U=uv2,V=uv2) VshrpD = mt_MakeDiff(Vshrp,vblur,U=uv2,V=uv2) VlimD = mt_LutXY(VshrpD,VblurD,expr="x 128 - y 128 - * 0 < x 128 - abs y 128 - abs < x y ? 128 - 0.25 * 128 + x 128 - abs y 128 - abs < x y ? ?",U=uv2,V=uv2) mt_AddDiff(Vblur,VlimD,U=uv,V=uv) (amnt>254) ? last : (amnt==0) ? clp : mt_LutXY(clp,last,expr="x "+AMN+" + y < x "+AMN+" + x "+AMN+" - y > x "+AMN+" - y ? ?",U=uv,V=uv) return(last) } function sbr(clip o) { rg11=o.removegrain(11) rg11D=mt_makediff(o,rg11) rg11DD=mt_makediff(rg11D,rg11D.removegrain(11)).mt_lutxy(rg11D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?") o.mt_makediff(rg11DD,U=2,V=2) } function sbrV(clip o) { rg11=o.mt_convolution("1","1 2 1") rg11D=mt_makediff(o,rg11) rg11DD=mt_makediff(rg11D,rg11D.mt_convolution("1","1 2 1")).mt_lutxy(rg11D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?") o.mt_makediff(rg11DD,U=2,V=2) } Apart from IVTC, all filtering in fact is purely spatial. Not any temporal filter is involved. <--> <--> <--> Sorry, for the screenshots I forgot to crop the 4 pixels left/right that are added in the script. __________________ - We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
 23rd July 2012, 20:46 #63  |  Link sirt x264 fan     Join Date: Feb 2011 Location: In the trap Posts: 458 Oops Didée I pmed you beofre seing this. I thank you, I can't wait to test this tomorrow !
 24th July 2012, 01:37 #64  |  Link jmac698 Registered User   Join Date: Jan 2006 Posts: 1,865 Good job Didée! Can I add it to the composite artefacts package?
 24th July 2012, 04:24 #65  |  Link Reel.Deel Registered User   Join Date: Mar 2012 Location: Texas Posts: 1,173 sirt: Now all you need is some derainbowing and you'll be good to go! Didée: The results look great. I tried it with a few different samples (anime and film) and this script seems to do a excellent job on all of them. I'm truly amazed at what Avisynth can accomplish under the hands of a skilled operator. Thanks Didée! I have a few questions, if you don't mind? What are some of the key parameters that dictate the strength of the dot crawl detection and or removal? Also, what does one have to do in order to be able to process progressive/interlaced NTSC and PAL? Removing TFM and TDecimate seems too easy. Last edited by Reel.Deel; 24th July 2012 at 04:27.
 24th July 2012, 08:54 #66  |  Link sirt x264 fan     Join Date: Feb 2011 Location: In the trap Posts: 458 Reel.Deel Yes, In fact I've filtered that separetly in an adaptative way according to the scenes of the video. Didée Fantastic ! It's quite good. Thanks for your time and efforts spent on this. I would like to thank everybody that helped me on this subject. One question more remains pending...here Didée iVTCed my source and based his work on raw and clean1. But let's say I would like to apply the code on an AVI source as Input that has been ivtced, which means I would also use the script after iVTC like there, how could you do ? I've reflected about this but I'm lost, it looks like I will have to modify too many things. Do you have a suggestion ?
24th July 2012, 09:05   #67  |  Link
Ghitulescu
Registered User

Join Date: Mar 2009
Location: Germany
Posts: 5,717
Quote:
 Originally Posted by Didée
Is it only me the one that doesn't see the full image?
__________________
Born in the USB (not USA)

 24th July 2012, 09:57 #68  |  Link jmac698 Registered User   Join Date: Jan 2006 Posts: 1,865 It was a link a image host http://www.imagebam.com/image/303a44202846639
24th July 2012, 10:30   #69  |  Link
Didée
Registered User

Join Date: Apr 2002
Location: Germany
Posts: 5,394
@Ghitulescu
Don't know how it looks on your side, but for me at least, ImageBam surely shows the full images.
Just in case - those pics zipped.

@sirt
If a video already has been IVTC'ed, then everything is different. Remember
Quote:
 Originally Posted by Didée [from PM] dotcrawl is not always the same, and very small differences in the characteristics can make the difference between "filter works" and "filter fails".
The posted script relies on a very certain symmetry being present in the dotcrawl's pixel-pattern. If IVTC already has been done beforehand, that symmetry has been destroyed. Those things that "naturally cancel-out each other" prior to IVTC, will not cancel-out anymore after IVTC.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)

24th July 2012, 10:51   #70  |  Link
sirt
x264 fan

Join Date: Feb 2011
Location: In the trap
Posts: 458
Quote:
 Originally Posted by Didée @sirt If a video already has been IVTC'ed, then everything is different. Remember The posted script relies on a very certain symmetry being present in the dotcrawl's pixel-pattern. If IVTC already has been done beforehand, that symmetry has been destroyed. Those things that "naturally cancel-out each other" prior to IVTC, will not cancel-out anymore after IVTC.
Didée then you won't like what I will ask for...obviously I will use your script if I plan on using tfm. But, in the case I have an AVI source which is progressive (that has been ivtced upstream) or whatever the source, consider it had previously been iVTCEd. In this situation, do you think you can write a spatial script similar to the last one where you used my sample MPG source or is it totally irrelevant and requires too much time ? I know it is different, but yesterday you made some temporal script that applies on the iVTCed video, that's why I'm asking for a spatial script now.

I know it is somewhat twisted but I am also corecting patterns manually and I don't want to use an automatic tfm as you did.

 24th July 2012, 12:18 #71  |  Link Didée Registered User   Join Date: Apr 2002 Location: Germany Posts: 5,394 Whatever. If IVTC has been done already, then you left the field of certainty, and are in the swamp of vague possibilities. Try FRFun7 on its own, this alone often snaps surprisingly nice onto dotcrawl. But, it always will also soften the picture ... propably it is indicated to shoot back with "shift-values-two-notches-back", or contrasharpen, or both together. You can teach how to swing the racket, but not how to get the feeling for the ball. __________________ - We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
 24th July 2012, 13:27 #72  |  Link sirt x264 fan     Join Date: Feb 2011 Location: In the trap Posts: 458 Thanks Didée, I'm giving it a try. It is really not bad when taking your time to adjust it !
 24th July 2012, 13:32 #73  |  Link jmac698 Registered User   Join Date: Jan 2006 Posts: 1,865 In your case, maybe you don't need automated IVTC at all. There could be one constant pattern throughout the whole video, so you can use that pattern and there will never be a mistake. If the pattern changes, it's likely at scene changes when it was edited. A check could be made at those points.
24th July 2012, 13:39   #74  |  Link
sirt
x264 fan

Join Date: Feb 2011
Location: In the trap
Posts: 458
Quote:
 Originally Posted by jmac698 In your case, maybe you don't need automated IVTC at all. There could be one constant pattern throughout the whole video, so you can use that pattern and there will never be a mistake. If the pattern changes, it's likely at scene changes when it was edited. A check could be made at those points.
jmac698, thanks for your advise. As part of my daily task, I am not neccessarly required to "manually ivtc" the input. The idea is to iVTC the video then test such and such filters. In this case, it turns out the pattern is indeed constant but even if it would not be I would not be compelled to solve it perfectly. I was just asking in case in fact, but using tfm is ample !

24th July 2012, 14:06   #75  |  Link
Ghitulescu
Registered User

Join Date: Mar 2009
Location: Germany
Posts: 5,717
Quote:
 Originally Posted by jmac698 It was a link a image host http://www.imagebam.com/image/303a44202846639

It happens to all images posted on imagebam.com.

Quote:
 Originally Posted by Didée @Ghitulescu Don't know how it looks on your side, but for me at least, ImageBam surely shows the full images. Just in case - those pics zipped.
Thanks.
__________________
Born in the USB (not USA)

24th July 2012, 14:37   #76  |  Link
Groucho2004

Join Date: Mar 2006
Location: Barcelona
Posts: 5,040
Quote:
 Originally Posted by Ghitulescu It happens to all images posted on imagebam.com.
The problem must be on your side. It seems everyone else can view the images.

 24th July 2012, 16:29 #77  |  Link Reel.Deel Registered User   Join Date: Mar 2012 Location: Texas Posts: 1,173 Hello, I have some examples of dot crawl in NTSC interlaced footage. QTGMC does a good job at removing most of it, but it still leaves subtle hints of the existing dot crawl. Using Didée's solution before QTGMC helps almost completely eliminate the remaining dot crawl. Here is the change I made to dot crawl script to make it work (at least I think) with the interlaced footage. Code: ........ drr = o.separatefields().vinverse2().weave() both = stackhorizontal(drr,o) IVTC = both # Removed TFM and TDecimate. raw = IVTC.crop(ox,0,ox,oy) clean1 = IVTC.crop(0,0,ox,oy) ........ The rest of the script is the same and its followed by QTGMC(preset="slower",EdiMode="NNEDI3", EdiThreads=2) The results look fine to me but I was wondering if there's a better way to go about this?
 24th July 2012, 17:14 #78  |  Link sirt x264 fan     Join Date: Feb 2011 Location: In the trap Posts: 458 Great. Would you like showing some frames comparison then ?
 25th July 2012, 06:48 #79  |  Link Reel.Deel Registered User   Join Date: Mar 2012 Location: Texas Posts: 1,173 So doing a little bit more testing I found out that simply just removing TFM and TDecimate does not work. It seems as Vinverse2() does a partial vertical pixel shift. Processing the interlaced content as individual frames produced better results. Here's the comparison. Code: ........ o=last.separatefields() ox=o.width() oy=o.height() drr = o.vinverse2() # Removed separatefields and weave both = stackhorizontal(drr,o) IVTC = both # Removed TFM and TDecimate. ........ The rest of the script is the same and it's followed by QTGMC(preset="slower",EdiMode="NNEDI3", EdiThreads=2)
 25th July 2012, 08:03 #80  |  Link sirt x264 fan     Join Date: Feb 2011 Location: In the trap Posts: 458 Even though Dot crawl + QTGMC solve much more things such as the baloons, I don't think using QTGMC is a good idea given that it is a deinterlacer. It echoes badly on man's shirt for example and on the bricks of the wall behind.