an idea for fixing haloed, chroma bleeding edges

[zee944]

I have a source with ugly, chroma bleeding, haloed, and blurry edges I'm unable to fix for quite a long time now. Here's a sample picture from it (fragments of frames actually, no resizing).

It may not look so bad from the pictures, but guaranteed to look really bad after adding any sharpening or heavier color correction.

MergeChroma(aWarpSharp(depth=32,blurlevel=1,thresh=0.5,cm=1)) does a help a little; YAHR() also helps a bit, but all in all, the root problem is still there, the edge problems are more apparent then ever after further tweaking:



I've been playing around with chroma shifters and dehaloers, with little to no results. The only thing that really fixed the edges was aWarpSharp(depth=32,blurlevel=2,thresh=0.75). I just loved those edges but the result was obviously unusable due to the heavy deformations and the lack of details.

I'm desperate by now. I thought of something to have these great edges with the original geometry and details:

1. take a frame, use aWarpSharp to create a deformed frame with perfect edges

2. motion compensate the deformed frame to the original frame

3. copy the perfect edges onto the original frame


Here's my script so far (sample m2v):

LoadPlugin("mvtools2[2.5.11.1].dll")
LoadPlugin("aWarpSharp[b1].dll")			# chroma-probléma javításához kell
LoadPlugin("mt_masktools[2.0a30].dll")

DirectShowSource("TD_sample.m2v")
ConvertToYV12()

AddBorders(16,0,16,0)
MergeChroma(aWarpSharp(depth=32,blurlevel=1,thresh=0.5,cm=1))		# pre-fixing a chroma noise
Crop(16,0,-16,0)

original=last

# Instead of chroma bleeding, haloed and blurry edges we're creating good edges, but cartoonish and deformed look:
awarpsharped=original.aWarpSharp(depth=32,blurlevel=2,thresh=0.99)

# Getting back to the original shapes with motion compensation to have good and undeformed edges, but still cartoonish look:
Interleave(original,awarpsharped)
super=MSuper(last)
forward_vectors = MAnalyse(super, isb=true)
MFlow(super,forward_vectors)
SelectEvery(2,0)
fixedawarpsharped=last		

edgemask=original.MergeLuma(mt_edge("sobel",7,7,5,5))	
# Extending and blurring the mask to have some transition at overlaying:
edgemask=edgemask.mt_inflate().Blur(1.58).Blur(1.58)

# Bad edges replaced with good edges on the original image:
endresult=Overlay(original,fixedawarpsharped,mask=edgemask)

StackVertical(original.Crop(170,70,-170,-70).subtitle("original"),endresult.Crop(170,70,-170,-70).subtitle("end result"))

Script result:




The script shows only the general idea, the result doesn't look good due to the imperfect motion compensating and poorly selected edges. It's cartoonish and the sharpness difference between the tweaked and non-tweaked edges are not handled either (that'll be a much later step I guess).

Questions:

1. Why the motion compensated frames ("fixedawarpsharped") seem not fully compensated? Like if the motion on it was halfway between the "original" and the "awarpsharped" motion. Am I miss something here?

2. How could I create a better edgemask to select only the "important" (problematic) edges?

3. The compensated edges look somewhat pixelated and unstable, can MVTools2 produce better result with different settings?

4. How to "cartoonize" the original clip to look very similar to the awarpsharped one (for better motion estimation)?

Could you help me getting the best out of this method? It really looks promising to me, I hope it'll fix my video eventually, and could be used for other sources too. Any tweaks, ideas and answers are appreciated