an idea for fixing haloed, chroma bleeding edges

[zee944]

I have a source with ugly, chroma bleeding, haloed, and blurry edges I'm unable to fix for quite a long time now. Here's a sample picture from it (fragments of frames actually, no resizing).

It may not look so bad from the pictures, but guaranteed to look really bad after adding any sharpening or heavier color correction.

MergeChroma(aWarpSharp(depth=32,blurlevel=1,thresh=0.5,cm=1)) does a help a little; YAHR() also helps a bit, but all in all, the root problem is still there, the edge problems are more apparent then ever after further tweaking:



I've been playing around with chroma shifters and dehaloers, with little to no results. The only thing that really fixed the edges was aWarpSharp(depth=32,blurlevel=2,thresh=0.75). I just loved those edges but the result was obviously unusable due to the heavy deformations and the lack of details.

I'm desperate by now. I thought of something to have these great edges with the original geometry and details:

1. take a frame, use aWarpSharp to create a deformed frame with perfect edges

2. motion compensate the deformed frame to the original frame

3. copy the perfect edges onto the original frame


Here's my script so far (sample m2v):

LoadPlugin("mvtools2[2.5.11.1].dll")
LoadPlugin("aWarpSharp[b1].dll")			# chroma-probléma javításához kell
LoadPlugin("mt_masktools[2.0a30].dll")

DirectShowSource("TD_sample.m2v")
ConvertToYV12()

AddBorders(16,0,16,0)
MergeChroma(aWarpSharp(depth=32,blurlevel=1,thresh=0.5,cm=1))		# pre-fixing a chroma noise
Crop(16,0,-16,0)

original=last

# Instead of chroma bleeding, haloed and blurry edges we're creating good edges, but cartoonish and deformed look:
awarpsharped=original.aWarpSharp(depth=32,blurlevel=2,thresh=0.99)

# Getting back to the original shapes with motion compensation to have good and undeformed edges, but still cartoonish look:
Interleave(original,awarpsharped)
super=MSuper(last)
forward_vectors = MAnalyse(super, isb=true)
MFlow(super,forward_vectors)
SelectEvery(2,0)
fixedawarpsharped=last		

edgemask=original.MergeLuma(mt_edge("sobel",7,7,5,5))	
# Extending and blurring the mask to have some transition at overlaying:
edgemask=edgemask.mt_inflate().Blur(1.58).Blur(1.58)

# Bad edges replaced with good edges on the original image:
endresult=Overlay(original,fixedawarpsharped,mask=edgemask)

StackVertical(original.Crop(170,70,-170,-70).subtitle("original"),endresult.Crop(170,70,-170,-70).subtitle("end result"))

Script result:




The script shows only the general idea, the result doesn't look good due to the imperfect motion compensating and poorly selected edges. It's cartoonish and the sharpness difference between the tweaked and non-tweaked edges are not handled either (that'll be a much later step I guess).

Questions:

1. Why the motion compensated frames ("fixedawarpsharped") seem not fully compensated? Like if the motion on it was halfway between the "original" and the "awarpsharped" motion. Am I miss something here?

2. How could I create a better edgemask to select only the "important" (problematic) edges?

3. The compensated edges look somewhat pixelated and unstable, can MVTools2 produce better result with different settings?

4. How to "cartoonize" the original clip to look very similar to the awarpsharped one (for better motion estimation)?

Could you help me getting the best out of this method? It really looks promising to me, I hope it'll fix my video eventually, and could be used for other sources too. Any tweaks, ideas and answers are appreciated

[Didée]

Regarding the motion-compensation part: Reduce blocksize to 4, reduce hierarchical levels to 2 or even 1, and make a small-range exhaustive search. (The only thing you want to compensate is the warpsharp deformations. Hence, the wanted compensation target should be very close to the starting pixel, usually not more than 1~2 pixels away.)

It might pay off to apply some blurring to the motionsearch clip - after all, the warpsharpen'ed edges are sharper than the original edges. Would be better if, in the searchclip, both clips have a similar level of sharpness (or blurryness) .... else, Manalyse would need to match a "sharp" edge onto a "blurry" edge, which obviously isn't ideal.


edit: Regarding detail loss: make a comparison where the original input has "more edge" than the result. Those are (probably) the areas where merging-back the original is appropriate.

[zee944]

All of your suggestions helped, thanks for them. I've ended up with this script:

Function mcaWarpSharp(clip source, int "supsamp")
{# by z. ndmn.	dependencies: MVTools2, masktools, aWarpSharp[20090619]
supsamp = default(supsamp, 4)	# supsamp=2 recommended for testing, supsamp=4 for encoding.
source
# for no supersampling (supsamp=1): (thresh=255,blur=3,type=1,depth=8,chroma=4)
# for 4x supersampling (supsamp=4): (thresh=192,blur=36,type=1,depth=96,chroma=4)
WarpedEdges=last.Lanczos4Resize(width*supsamp,height*supsamp).\
	aWarpSharp2(thresh=int(255*pow(supsamp,-0.207)),blur=3*int(pow(supsamp,1.7925)),type=1,depth=8*int(pow(supsamp,1.7925)),chroma=4)

# Getting back to the original shapes with motion compensation to have good and undeformed edges, but still cartoonish look:
Interleave(last.Lanczos4Resize(width*supsamp,height*supsamp),WarpedEdges)
blurredclip=GaussResize(width*2,height*2,p=32*pow(supsamp,-2)).GaussResize(width,height,p=32*pow(supsamp,-2))
super=MSuper(blurredclip, levels=1)
super2=MSuper(last, levels=1)
forward_vectors = MAnalyse(super, blksize=4, levels=1, search=3, searchparam=5, isb=true)
MFlow(super2,forward_vectors)
SelectEvery(2,0)
Lanczos4Resize(width/supsamp,height/supsamp)

# Replacing 'bad' edges with 'good' edges on the original image:
edgemask=source.mt_edge("min/max").GreyScale()
Overlay(source,last,mask=edgemask)
#Overlay(last,                      last.Blur(0.5),mask=edgemask)	# a little blurring on the sharp new edges...
#Overlay(last.sharpen(0.5),last,               mask=edgemask)	# ...or sharpening everywhere else to make the image consistent.
Return(last)
}

The motion estimation is nearly perfect now, the only thing in the current workflow that could be better is the edge masking. But all in all, the end result seems quite good.

[Dogway]

Can somebody help me to speed up this script? I tweaked mcawarpsharp a bit for speed and control but this run at around 0.15fps on a 720x480 clip with 96000 frames. I think the bottleneck is at mflow in the 4x supsampled clip but I'd like to know if I can speed up that part or others as well without using a smaller supsampling.

setmtmode(5,2)
ffvideoSource("FFV1intermediary.avi")
a=last
setmtmode(2,2)


white=70
mt_edge("min/max",thY1=0,thY2=255).removegrain(11).mt_expand.mt_expand.mt_lut("x "+string(white)+" > 255 x "+string(white)+" / 255 * ?")

BicubicResize (180,120, 1, 0)
TemporalSoften(10,30,0,15,2)
BicubicResize (720,480, 1, 0)
global edgemask=last

deconv_v =  "-3 10 4 -4 8 -6 100 -6 8 -4 4 10 -3"
mt_merge(a,a.mt_convolution(horizontal="1",vertical=deconv_v, Y=3, U=2, V=2),edgemask,u=4,v=4)

MCaWarpSharp4(4,2,0.6,true,640,360)

smdegrain(tr=3,thSAD=400,hpad=0,vpad=0,chroma=false,plane=0)



function MCaWarpSharp4(clip source, int "supsamp",int "post", float "PPstr", bool "fast", int "w", int "h"){

supsamp = default(supsamp, 4)	# supsamp=2 recommended for testing, supsamp=4 for encoding.
post    = default(post, 4)      # mix the warpsharped edges with original footage thru blur and sharpen post processing
str     = default(PPstr,1.0)
fast    = default(fast,true)
w       = default(w,source.width())
h       = default(h,source.height())


var=32*pow(supsamp,-2)
w1=source.width()*supsamp
h1=source.height()*supsamp

source
sourceSup=fast ? bicubicResize(w1,h1,b=0.0,c=0.5) : nnedi3_rpow2(rfactor=supsamp,cshift="spline64resize",nns=4,qual=2)

WarpedEdges=sourceSup.aWarpSharp2(
\ thresh=int(255*pow(supsamp,-0.207)),
\ blur=3*int(pow(supsamp,1.7925)),
\ type=1,depth=8*int(pow(supsamp,1.7925)),chroma=4)

Interleave(sourceSup,WarpedEdges)

blurredclip=GaussResize(w1*2,h1*2,p=var).GaussResize(w1,h1,p=var)

super  =          MSuper(blurredclip, levels=1, pel=fast?1:2, sharp=fast?0:2)
super2 =          MSuper(levels=1, pel=fast?1:2, sharp=fast?0:2)
forward_vectors = MAnalyse(super, blksize=4, levels=1, search=3, searchparam=5, isb=true)
                  MFlow(super2,forward_vectors)

SelectEvery(2,0)
fast ? bicubicResize(w,h,b=0.0,c=0.5) : Spline36Resize(w,h)

# Replacing 'bad' edges with 'good' edges on the original image
source    = post==1 && (Defined(w) || Defined(h)) ? (fast ? source.bicubicResize(w1,h1,b=0.0,c=0.5) : source.Spline36Resize(w1,h1)) : source
#~ edgemask  = source.mt_edge("min/max").blur(1.0)

Assert  ((post >= 1 && post <= 4 )  ? true : false, chr(10) + "'post' have not a correct value! [1,2,3,4]" + chr(10))
post =   (post==1) ? mt_merge(source,last,edgemask.bilinearresize(w,h),luma=true)
    \  : (post==2) ? mt_merge(Blur(min(str*0.5,1.58)),edgemask.bilinearresize(w,h),luma=true)
    \  : (post==3) ? mt_merge(LSFmod(defaults=fast?"fast":"slow",strength=int(str*20),edgemode=0,soothe=true,ss_x=1.0,ss_y=1.0),last,edgemask.bilinearresize(w,h),luma=true)
    \  : (post==4) ? mt_merge(LSFmod(defaults=fast?"fast":"slow",strength=int(str*20),edgemode=0,soothe=true,ss_x=1.0,ss_y=1.0),Blur(min(str*0.5,1.58)),edgemask.bilinearresize(w,h),luma=true)
    \  :  blankclip(source, width=width, height=120).SubTitle("post = " + string(post),text_color=$FFFFFF,font="COURIER NEW",size=12,x=320,y=90)
return post}

edit: I tweaked the manalyse portion of the mcawarpsharp4 function with the new mscalevectors from cretindesalpes, but results are no good, I can't go sub 4 for blksize so I don't know if that's the reason, also mode=2 doesn't work, so I couldn't check.

function MCaWarpSharp4(clip source, int "supsamp",int "post", float "PPstr", bool "fast", int "w", int "h"){

supsamp = default(supsamp, 4)	# supsamp=2 recommended for testing, supsamp=4 for encoding.
post    = default(post, 4)      # mix the warpsharped edges with original footage thru blur and sharpen post processing
str     = default(PPstr,1.0)
fast    = default(fast,true)
w       = default(w,source.width())
h       = default(h,source.height())


var=32*pow(supsamp,-2)
w1=source.width()*supsamp
h1=source.height()*supsamp

source
sourceSup=fast ? bicubicResize(w1,h1,b=0.0,c=0.5) : nnedi3_rpow2(rfactor=supsamp,cshift="spline64resize",nns=4,qual=2)

WarpedEdges=sourceSup.aWarpSharp2(
\ thresh=int(255*pow(supsamp,-0.207)),
\ blur=3*int(pow(supsamp,1.7925)),
\ type=1,depth=8*int(pow(supsamp,1.7925)),chroma=4)

Interleave(sourceSup,WarpedEdges)

blurredclip=GaussResize(w1*2,h1*2,p=var).GaussResize(720,480,p=32*pow(1,-2))

super  =          MSuper(blurredclip, levels=1, pel=fast?1:2, sharp=fast?0:2,hpad=0,vpad=0)
super2 =          MSuper(levels=1, pel=fast?1:2, sharp=fast?0:2,hpad=0,vpad=0)
forward_vectors = MAnalyse(super, blksize=4, levels=1, search=3, searchparam=5, isb=true).MScaleVect(4)
                  MFlow(super2,forward_vectors)

SelectEvery(2,0)
fast ? bicubicResize(w,h,b=0.0,c=0.5) : Spline36Resize(w,h)

# Replacing 'bad' edges with 'good' edges on the original image
source    = post==1 && (Defined(w) || Defined(h)) ? (fast ? source.bicubicResize(w1,h1,b=0.0,c=0.5) : source.Spline36Resize(w1,h1)) : source
#~ edgemask  = source.mt_edge("min/max").blur(1.0)

Assert  ((post >= 1 && post <= 4 )  ? true : false, chr(10) + "'post' have not a correct value! [1,2,3,4]" + chr(10))
post =   (post==1) ? mt_merge(source,last,edgemask.bilinearresize(w,h),luma=true)
    \  : (post==2) ? mt_merge(Blur(min(str*0.5,1.58)),edgemask.bilinearresize(w,h),luma=true)
    \  : (post==3) ? mt_merge(LSFmod(defaults=fast?"fast":"slow",strength=int(str*20),edgemode=0,soothe=true,ss_x=1.0,ss_y=1.0),last,edgemask.bilinearresize(w,h),luma=true)
    \  : (post==4) ? mt_merge(LSFmod(defaults=fast?"fast":"slow",strength=int(str*20),edgemode=0,soothe=true,ss_x=1.0,ss_y=1.0),Blur(min(str*0.5,1.58)),edgemask.bilinearresize(w,h),luma=true)
    \  :  blankclip(source, width=width, height=120).SubTitle("post = " + string(post),text_color=$FFFFFF,font="COURIER NEW",size=12,x=320,y=90)
return post}

[*.mp4 guy]

... While not strictly an optimization of the posted script, this probably fits the bill. I've been using it for a while, never thought to post it.

Soft thresholded warpsharp:

Function Twarp(Clip c, int "thresh", float "depth", int "blurlevel", float "wthresh")
{

	Thresh = Default(Thresh, 8)
	depth = Default(depth, 5)
	blurlevel = Default(blurlevel, 1)
	wthresh = Default(wthresh, 0.99)
		
		B1 = C.awarpsharp(depth=depth, blurlevel=blurlevel, thresh=wthresh)
			Mt_LutXY(C, b1, " X Y - abs X Y - abs * X Y -  * X Y - abs X Y - abs * "+string(thresh)+" + / 128 +", u=1, v=1)
	
Mt_AddDiff(last, b1, u=1, v=1)
}

Thresh is the strength of the limiting, all other parameter map to awarpsharp equivalents.

Defaults are very conservative, intended to reduce haloing without any other effects. For stronger settings, try something such as: Twarp(64, 12, 2, 0.7)

Re-writing for new warpsharp should be very simple if it bothers anyone, but I can't be bothered atm.

[Dogway]

Probably something around this?

Function Twarp(Clip c, int "thresh", int "depth", int "blurlevel", int "wthresh")
{

	Thresh = Default(Thresh, 8)
	depth = Default(depth, 3)
	blurlevel = Default(blurlevel, 8)
	wthresh = Default(wthresh, 253)
		
		B1 = C.awarpsharp2(depth=depth, blur=blurlevel, type=1,chroma=3,thresh=wthresh)
			Mt_LutXY(C, b1, " X Y - abs X Y - abs * X Y -  * X Y - abs X Y - abs * "+string(thresh)+" + / 128 +", u=1, v=1)
	
Mt_AddDiff(last, b1, u=1, v=1)}

Twarp(64, 6, 16, 179)


It didn't perform too nice, it deformed too much (I'm using live footage), or that is my impression, but thanks for sharing I think I will make use of it eventually when I test more on other sources.