[foxyshadis]

Originally Posted by lansing

I kind of get it working now. I created an average clip with convolution matrix=[1, 1, 1, 1, 1, 1, 1, 1, 1]), and then compare the color-corrected clip to the average clip. If the value of the pixel in the cc is less than the average, change it to the value of the average. The line edge will darkened. Then I have copy the linemask code from fastlinedarkenmod and do a maskmerge between cc and the darken clip.

lut_file = r'test.cube' 

clip = core.ffms2.Source(r"abc") 

rgb_clip = core.resize.Bicubic(clip, matrix_in_s="709", format=vs.RGBS) 

match_clip = core.timecube.Cube(rgb_clip, cube=lut_file) 

yuv_clip = core.resize.Bicubic(match_clip, matrix_s="709", format=vs.YUV420P8)

avg_clip = core.std.Convolution(yuv_clip, matrix=[1, 1, 1, 1, 1, 1, 1, 1, 1])

# compares clip to average, change value when below average
def darken_edge(x, y):         
	if x < y:
		return x
	else:
		if abs(x-y) < 30:
			return y 
		else:
			return x

darken_clip = core.std.Lut2(clipa=yuv_clip, clipb=avg_clip, function=darken_edge)

# linemask codes from fastlinedarkenmod
def scale(value, peak):
    return value * peak // 255

strength=48
protection=5
luma_cap=191
threshold=4
thinning=54


peak = (1 << yuv_clip.format.bits_per_sample) - 1 #255
Str = strength / 128
lum = scale(luma_cap, peak)
thr = scale(threshold, peak)
thn = thinning / 16

tmp = scale(127, peak)
     
exin = core.std.Maximum(yuv_clip, threshold=peak // (protection + 1)).std.Minimum()   

diff = core.std.Expr([yuv_clip, exin], ['y {lum} < y {lum} ? x {thr} + > x y {lum} < y {lum} ? - 0 ? {i} +'.format(lum=lum, thr=thr, i=tmp)])
    

expr = 'x {i} - {thn} * {peak} +'.format(i=tmp, thn=thn, peak=peak)
linemask = core.std.Expr([core.std.Minimum(diff)], [expr]).std.Convolution(matrix=[1, 1, 1, 1, 1, 1, 1, 1, 1])

# merge
result_clip = core.std.MaskedMerge(darken_clip, yuv_clip, linemask)

result_clip.set_output()

The problem I have now is that with the linemask, the edge darken effect is so subtle that it's unnoticeable. Without the mask it looks good.

Perhaps add .std.Maximum or .std.Inflate to the linemask? That'll widen its effective area.