Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 17th July 2012, 08:47   #1  |  Link
sirt
x264 fan
 
sirt's Avatar
 
Join Date: Feb 2011
Location: In the trap
Posts: 458
Dot Crawl issue

Hello,

I know this subject has been talked over many times...howhever, in spite of the numerous threads dealing with Dot Crawl, I still can't manage to get rid of it.

So let's put the record straight : I am currently working at a video compression firm and I am trying to correct artefacts like that dotcrawl on many frames. Here is the problematic one :



First of all, do you agree it is indeed dot crawl ?

Then my problem is I can't correct it without giving rise to harmful secondary effects typically blending on the hair or edges. Of course I still haven't been able to completely eliminate dotcrawl from that frame above. I've tried this :

DeDot(luma2d=255, lumaT=25, chromaT1=0, chromaT2=255)

LUTDeCrawl(ythresh=10, cthresh=10, maxdiff=50, scnchg=25, usemaxdiff=true, mask=false)

It is not worth I show you the bad results I get, either I won't show you the Checkmate() one in so far it blurs too much. I've tried to ajust them with no sucess.

Then how the hell could I correct that frame without reverberations where not intended ? Only the face and the borders should be processed !
sirt is offline   Reply With Quote
Old 17th July 2012, 10:43   #2  |  Link
Ghitulescu
Registered User
 
Ghitulescu's Avatar
 
Join Date: Mar 2009
Location: Germany
Posts: 5,765
Quote:
Originally Posted by sirt View Post
First of all, do you agree it is indeed dot crawl ?
Dot crawl can 100% sure be diagnosticised only on moving images

Dot crawl is caused by chroma interference with luma. So it should appear on the neck too as the same conditions apply there too.
__________________
Born in the USB (not USA)
Ghitulescu is offline   Reply With Quote
Old 17th July 2012, 10:54   #3  |  Link
sirt
x264 fan
 
sirt's Avatar
 
Join Date: Feb 2011
Location: In the trap
Posts: 458
Thanks, here is another examples :







(For the second one, I ignore rainbow)

Assuming it is dot crawl, what would you to correct it in accordance with my previous expectations ?

Last edited by sirt; 17th July 2012 at 11:12.
sirt is offline   Reply With Quote
Old 17th July 2012, 11:09   #4  |  Link
Emulgator
Big Bit Savings Now !
 
Emulgator's Avatar
 
Join Date: Feb 2007
Location: close to the wall
Posts: 1,228
Yes, that looks like dot crawl. The saturation borders tell.
Rainbowing is in there as well. The left/right railing shows it.
I had good success with FFT in cases of dot crawl from SVHS recordings.
First you want to check which planes Y or (UV) are affected.
For Y F2Quiver with pairs of elliptical filters (nef=2,...)may help.
For UV F2QuiverUV with gp4 or bp4.
First test=true, then find the frequency spots and apply.
Both FFT filters eat up their domain, so need refixing.
F2Quiver eats up Y, F2QuiverUV eats up saturation and shifts hue.
You got to find out.
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain)
"Data reduction ? Yep, Sir. We're working on that issue. Synce invntoin uf lingöage..."

Last edited by Emulgator; 17th July 2012 at 11:11.
Emulgator is offline   Reply With Quote
Old 17th July 2012, 11:13   #5  |  Link
jmac698
Registered User
 
Join Date: Jan 2006
Posts: 1,862
I have some ideas for that that have never been used before. It is ntsc or pal? Can we have a short clip?
jmac698 is offline   Reply With Quote
Old 17th July 2012, 11:21   #6  |  Link
sirt
x264 fan
 
sirt's Avatar
 
Join Date: Feb 2011
Location: In the trap
Posts: 458
Emulgator

Do you refer to FFT3Dfilter ?

I have never went in for which planes are affected, so that's new to me ! Assuming I find the good ones, do you mean FFT3Dfilter can do the trick ?

jmac698

Thanks for your interest. I would be in rapture you to give me some ideas, unfortunately I can't share a short clip in so far - as I said - I am currently doing postprocessing at a video compression firm and the source is a YUV Uncompressed one that belongs to them.
sirt is offline   Reply With Quote
Old 17th July 2012, 11:23   #7  |  Link
Emulgator
Big Bit Savings Now !
 
Emulgator's Avatar
 
Join Date: Feb 2007
Location: close to the wall
Posts: 1,228
Not FFT3D, but VCMohan's F2Quiver.
http://avisynth.org/vcmohan/

In my case the F2Quiver/UV combo left all details untouched and only removed what I intended to.
But setting the filters and their appropriate compensation may well take hours to adjust.

I would have tried Fizick's Defreq as well, but I haven't gotten it to work.
The frequency pattern showed, but I saw no effect when applied.
Maybe driver error on my side...
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain)
"Data reduction ? Yep, Sir. We're working on that issue. Synce invntoin uf lingöage..."

Last edited by Emulgator; 17th July 2012 at 11:31.
Emulgator is offline   Reply With Quote
Old 17th July 2012, 11:28   #8  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,393
prefilter with removegrain(4).removegrain(11)
get vectors on prefilter
mdegrain3 the original, with high thSAD (800 or so)
get difference from mdegrain3
sieve difference by, again, removegrain(4).removegrain(11)
apply sieved difference


BTW, it seems pretty unclear what your original source actually is. The first screenshot is 800x533, the later ones are 1094x729. Varying resolution within one source is unlikely. None of those resolution is a standard res.

In any case, process dotcrawl *without* any resizing beforehand.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)

Last edited by Didée; 17th July 2012 at 11:34.
Didée is offline   Reply With Quote
Old 17th July 2012, 11:36   #9  |  Link
sirt
x264 fan
 
sirt's Avatar
 
Join Date: Feb 2011
Location: In the trap
Posts: 458
Thanks Emulgator and Didée. It looks like your ideas are really interesting, but unfortunately I really don't understand what to do. Would you give me an or several example(s) of your methods ? I don't ask for the "adjusted parameters" for my case of course but for one or several scripts to exemplify what you tell me to do.

EDIT : about my frames, it is just my mistakes, saving them unproperly, sorry.

Last edited by sirt; 17th July 2012 at 11:41.
sirt is offline   Reply With Quote
Old 17th July 2012, 11:43   #10  |  Link
Emulgator
Big Bit Savings Now !
 
Emulgator's Avatar
 
Join Date: Feb 2007
Location: close to the wall
Posts: 1,228
Keep in mind: This might (and most probably will) not work for your source !
Code:
#[*#%%%%%%%%%%%%%%%%%%%%%%%%%%%%Start of DotCrawl Removal%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
[*#-------------------------------------------------------------Start of Defreq-FFT---------------------------------------------------------
Defreq(fx=64.0, dx=2.0, fy=16.0, dy=6.0, sharp=8.0, fx2=64.0, dx2=2.0, fy2=-48.0, dy2=6.0, sharp2=8.0, fx3=44.0, dx3=2.0, fy3=-0.0, dy3=6.0, sharp3=8.0, fx4=44.0, dx4=2.0, fy4=32.0, dy4=6.0, sharp4=8.0, show=0, info=true, plane=0)
Defreq(fx=44.0, dx=2.0, fy=-32.0, dy=6.0, sharp=8.0, fx2=44.0, dx2=2.0, fy2=-64.0, dy2=6.0, sharp2=8.0, fx3=44.0, dx3=2.0, fy3=64.0, dy3=6.0, sharp3=8.0, fx4=64.0, dx4=2.0, fy4=-16.0, dy4=6.0, sharp4=8.0, show=0, info=true, plane=0)
Defreq(fx=0.0, dx=3.0, fy=63.9, dy=4.0, sharp=8.0, fx2=0.0, dx2=8.0, fy2=0.0, dy2=3.0, sharp2=8.0, show=0, info=true, plane=1)
Defreq(fx=0.0, dx=3.0, fy=63.9, dy=4.0, sharp=8.0, fx2=0.0, dx2=8.0, fy2=0.0, dy2=3.0, sharp2=8.0, show=0, info=true, plane=2)
#Defreq nje rabotajet !
*]#-------------------------------------------------------------End of Defreq-FFT---------------------------------------------------------
#[*#---------------------------------------------------------------Start of Y-FFT-----------------------------------------------------------
#Das folgende Y-Muster ist ein PAL-Beispiel, durch Dotcrawl verursacht, somit inhaltsabhänging
#und gilt nur für Sektionen mit hoher Rot-Sättigung wie 113687, 119200-119300 bzw. Blausättigung wie Frame 92800-92900
SmoothCurve(interp=100, limiter=false, Ycurve="0-16;18-25;50-70;110-150;180-220;255-255", Ucurve="0-0;127-127;255-255", Vcurve="0-0;127-127;255-255", show=false)#Pre für 5x Y-F2Quiver)
F2Quiver(nef=2, f1x1=176, f1y1=+60, f1x2=176, f1y2=+85, mf1=1.01, sh1=1, f2x1=176, f2y1=-50, f2x2=176, f2y2=-87, mf2=1.01, sh2=1)
F2Quiver(nef=2, f1x1=176, f1y1=+165, f1x2=176, f1y2=+210, mf1=1.005, sh1=1, f2x1=176, f2y1=-165, f2x2=176, f2y2=-210, mf2=1.005, sh2=1)
F2Quiver(nef=2, f1x1=232, f1y1=120, f1x2=232, f1y2=135, mf1=1.02, sh1=1, f2x1=232, f2y1=-120, f2x2=232, f2y2=-135, mf2=1.01, sh2=1)
F2Quiver(nef=2, f1x1=232, f1y1=+8, f1x2=232, f1y2=-8, mf1=1.03, sh1=1, f2x1=232, f2y1=+245, f2x2=232, f2y2=+255, mf2=1.1, sh2=1)
F2Quiver(nef=2, f1x1=90, f1y1=192, f1x2=110, f1y2=192, mf1=1.1, sh1=1, f2x1=90, f2y1=-192, f2x2=110, f2y2=-192, mf2=1.1, sh2=1)
#SmoothCurve(interp=100, limiter=false, Ycurve="0-0;30-30;110-120;160-175;255-255", Ucurve="0-0;127-127;255-255", Vcurve="0-0;127-127;255-255", show=false)#für 4x Y-F2Quiver)
SmoothCurve(show=false, interp=100, limiter=false, Ycurve="0-0;16-22;60-80;90-128;150-200;255-255", Ucurve="0-0;127-127;255-255", Vcurve="0-0;127-127;255-255")#Post für 5x Y-F2Quiver)
#F2Quiver(test=true)
#*]#---------------------------------------------------------------End of Y-FFT-----------------------------------------------------------
#SmoothCurve(show=true, interp=100, limiter=false, Ycurve="0-0;63-63;127-127;150-150;191-191;255-255", Ucurve="0-0;255-255", Vcurve="0-0;255-255")#zur Kontrolle
#[*#-------------------------------------------------------------Start of UV-FFT---------------------------------------------------------
SmoothTweak(hue1=+6, hue2=+6, saturation=1.05, show=false).F2QuiverUV("gp4", 0,255, degree=2)#gp4, degree=2 kompensiert die Ortsfrequenzen noch besser
#SmoothTweak(hue1=+13, hue2=+13, saturation=1.1, show=false).F2QuiverUV("bp4", 0, 165, degree=1)#bp=4, degree=1 war fast genau so erfolgreich.
#F2QuiverUV(test=true)
#*]#---------------------------------------------------------------End of UV-FFT---------------------------------------------------------
#unbt()
#*]#%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%End of DotCrawl Removal%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The Defreq() and unbt() sections I left in for testing reasons and to switch them on/off.

Yes, and as Didée just said: No resizing beforehand !
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain)
"Data reduction ? Yep, Sir. We're working on that issue. Synce invntoin uf lingöage..."

Last edited by Emulgator; 17th July 2012 at 11:50.
Emulgator is offline   Reply With Quote
Old 17th July 2012, 12:06   #11  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,393
Here's the sieved MDegrain:

(Setup is made very slow on purpose.)
(Also, without having the egg in my hand, I can't tell how hard you can press without breaking the egg.)

Code:
YourSourceFilter("source.ext")
ConvertToYV12()
o=last

pre = o.removegrain(4).merge(o,0.5).removegrain(4).removegrain(11)
sup1 = pre.msuper()
sup2 = o.msuper(levels=1)

bvec3 = sup1.manalyse(isb=true, delta=3,blksize=16,overlap=8,search=5,searchparam=6,DCT=5)
bvec2 = sup1.manalyse(isb=true, delta=2,blksize=16,overlap=8,search=5,searchparam=6,DCT=5)
bvec1 = sup1.manalyse(isb=true, delta=1,blksize=16,overlap=8,search=5,searchparam=6,DCT=5)
fvec1 = sup1.manalyse(isb=false,delta=1,blksize=16,overlap=8,search=5,searchparam=6,DCT=5)
fvec2 = sup1.manalyse(isb=false,delta=2,blksize=16,overlap=8,search=5,searchparam=6,DCT=5)
fvec3 = sup1.manalyse(isb=false,delta=3,blksize=16,overlap=8,search=5,searchparam=6,DCT=5)

mdg = o.mdegrain3(sup2,bvec1,fvec1,bvec2,fvec2,bvec3,fvec3,thSAD=800) # drop to 600 or 400 ifyoucan ... raise to 1200 ifyoumust ...

diff = mt_makediff(o,mdg,U=3,V=3)
diff2 = mt_lutxy(diff,diff.removegrain(4).merge(diff,0.5).removegrain(4).removegrain(11),
 \                "x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")

mdg.mt_adddiff(diff2,U=2,V=2)

stackhorizontal(o,last)
return(last)
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 17th July 2012, 13:36   #12  |  Link
sirt
x264 fan
 
sirt's Avatar
 
Join Date: Feb 2011
Location: In the trap
Posts: 458
Thank you both for the time spent to write this. Unfortunately, you are not lucky with me because it is still hard for to get what you have exactly done.

Emulgator :

Let's say I want to use your script on a frame (with trim function for example). How can I ? Must I save it as an .avs file and try to apply it with :

a=Emulgator()
trim(0,1) + a+trim(2,3) + trim(3,0)

Didee :

My question is the same even though your script directly works on AVSP.
sirt is offline   Reply With Quote
Old 17th July 2012, 13:44   #13  |  Link
Emulgator
Big Bit Savings Now !
 
Emulgator's Avatar
 
Join Date: Feb 2007
Location: close to the wall
Posts: 1,228
va=(Sourcefilter)
dedot=(EmuFFT....)
firstframetodedot=100
lastframetodedot=200
Trim(va,0,frametoDeDot-1)++Trim(dedot,firstframetodedot,lastframetodedot)++Trim(va,lastframetodedot+1,0)
#with the usual exception of the first and last frame...
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain)
"Data reduction ? Yep, Sir. We're working on that issue. Synce invntoin uf lingöage..."
Emulgator is offline   Reply With Quote
Old 17th July 2012, 14:46   #14  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,393
BTW ... Is the source footage progressive, at least? All of a sudden, I got the fear that it might be hard telecined in fact, and you just manually selected some of the progressive frames for posting ...
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 17th July 2012, 15:32   #15  |  Link
sirt
x264 fan
 
sirt's Avatar
 
Join Date: Feb 2011
Location: In the trap
Posts: 458
Didée : indeed, the source is an interlaced NTSC one ! I firstly iVTCed it without cropping either resizing it (I do have to do encoding tests with my compression firms's own h264 encoder on PROGRESSIVE stuff at that time) ; I guess that has an importance...
sirt is offline   Reply With Quote
Old 20th July 2012, 20:39   #16  |  Link
sirt
x264 fan
 
sirt's Avatar
 
Join Date: Feb 2011
Location: In the trap
Posts: 458
Then Didée have you vanished ?
sirt is offline   Reply With Quote
Old 20th July 2012, 20:51   #17  |  Link
Bloax
The speed of stupid
 
Bloax's Avatar
 
Join Date: Sep 2011
Posts: 317
He had a heart attack after your latest post.
Bloax is offline   Reply With Quote
Old 20th July 2012, 21:02   #18  |  Link
sirt
x264 fan
 
sirt's Avatar
 
Join Date: Feb 2011
Location: In the trap
Posts: 458
Well I would rather prefer him helping me to ajust an existing filter or write a new one from zero ! Nothing works there whatever I tried:

- Dedot : zero

- Tcomb : not enough for my taste

- Lutdecrawl : zero

- Checkmate : it deletes some dot crawl at least but blends unexpected areas

Of course I maybe don't manage to set them correctly. So I suggest you jsaving one of the frames below and trying to remove this awful dot crawl ! It is terribly hard in my opinion..
sirt is offline   Reply With Quote
Old 20th July 2012, 21:08   #19  |  Link
Reel.Deel
Registered User
 
Join Date: Mar 2012
Location: Texas
Posts: 1,508
Is always best is you post a small sample.
Reel.Deel is offline   Reply With Quote
Old 20th July 2012, 21:22   #20  |  Link
sirt
x264 fan
 
sirt's Avatar
 
Join Date: Feb 2011
Location: In the trap
Posts: 458
Unfortunately it is NOT possible. Not only I would violate the rules but also the source is not my own property as I said ; I just have the right to test them, that's all. I think you can try to solve one of the frames below (check my second post), it would be the same.
sirt is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 15:03.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.