Protecting lines from thinning when applying a LUT in anime?

[lansing]

I've got the Sailor Moon blu-ray, it was upscale from SD source and it has a red color cast. After I color corrected it with a LUT, some of the lines appeared to be thinned. You can see it here with this before and after, escpecially in the white areas:

source (https://i.imgur.com/1CsklQT.png)
color correct (https://i.imgur.com/RDx2oRk.png)

The reason for that is in the source, because of the color cast, the outer part (softer part) of the lines appeared to be color casted as well (partially red). So when the LUT was applied with a mapping like "red->white", those outer parts were changed to white as well, resulting in lines to look thinned.

My question is how do I protect those outer parts?

This is my script:

lut_file = r'test.cube'

clip = core.ffms2.Source(r"abc")

rgb_clip = core.resize.Bicubic(clip, matrix_in_s="709", format=vs.RGBS)

match_clip = core.timecube.Cube(rgb_clip, cube=lut_file)

match_clip.set_output()


lut file (http://www.mediafire.com/file/2z5l60c2gpp4qih/test.cube)

[poisondeathray]

Did you try a line mask ? Maybe masktools, tcanny or some other method ?

Or merge back darker lines, or even line darkener ?

Or try adjusting the cast with other method ?

[lansing]

Did you try a line mask ? Maybe masktools, tcanny or some other method ?

Or merge back darker lines, or even line darkener ?

Or try adjusting the cast with other method ?

I'm thinking of line darkening only the edge of the lines after the LUT? But I don't see any edge masking filter that can select those.

[fAy01]

I'm thinking of line darkening only the edge of the lines after the LUT? But I don't see any edge masking filter that can select those.

https://kageru.moe/blog/article/edgemasks/

:sly:

[lansing]

So I was trying to create a darker clip with levels for testing with maskmerge, but the Levels filter doesn't seem to work properly.


lut_file = r'test.cube'

clip = core.ffms2.Source(r"abc")

rgb_clip = core.resize.Bicubic(clip, matrix_in_s="709", format=vs.RGBS)

match_clip = core.timecube.Cube(rgb_clip, cube=lut_file)

# this works properly as it brighten the highs
darken_clip = core.std.Levels(match_clip, min_in=0, max_in=225, min_out=0, max_out=255)

# this gives me a complete dark frame
darken_clip = core.std.Levels(match_clip, min_in=50, max_in=255, min_out=0, max_out=255)

darken_clip.set_output()

[poisondeathray]

0 to 255 is for 8 bit code values . You probably need to change the range for RGBS

It should be 0.0 to 1.0 white to black for float

[lansing]

0 to 255 is for 8 bit code values . You probably need to change the range for RGBS

It should be 0.0 to 1.0 white to black for float
You are right, I change all values to 0-1 and it works.

https://kageru.moe/blog/article/edgemasks/

:sly:

I don't see anything about selecting lines in that article.

[poisondeathray]

0 to 255 is for 8 bit code values . You probably need to change the range for RGBS

It should be 0.0 to 1.0 white to black for float

oops I meant black to white





You might be able modify a line mask for parts of edges by thinning the selected lines (maybe awarpsharp1/2) if that' s what you meant . But it might be difficult to thin say, outer edge or inner edge selectively

[lansing]

I kind of get it working now. I created an average clip with convolution matrix=[1, 1, 1, 1, 1, 1, 1, 1, 1]), and then compare the color-corrected clip to the average clip. If the value of the pixel in the cc is less than the average, change it to the value of the average. The line edge will darkened. Then I have copy the linemask code from fastlinedarkenmod and do a maskmerge between cc and the darken clip.


lut_file = r'test.cube'

clip = core.ffms2.Source(r"abc")

rgb_clip = core.resize.Bicubic(clip, matrix_in_s="709", format=vs.RGBS)

match_clip = core.timecube.Cube(rgb_clip, cube=lut_file)

yuv_clip = core.resize.Bicubic(match_clip, matrix_s="709", format=vs.YUV420P8)

avg_clip = core.std.Convolution(yuv_clip, matrix=[1, 1, 1, 1, 1, 1, 1, 1, 1])

# compares clip to average, change value when below average
def darken_edge(x, y):
if x < y:
return x
else:
if abs(x-y) < 30:
return y
else:
return x

darken_clip = core.std.Lut2(clipa=yuv_clip, clipb=avg_clip, function=darken_edge)

# linemask codes from fastlinedarkenmod
def scale(value, peak):
return value * peak // 255

strength=48
protection=5
luma_cap=191
threshold=4
thinning=54


peak = (1 << yuv_clip.format.bits_per_sample) - 1 #255
Str = strength / 128
lum = scale(luma_cap, peak)
thr = scale(threshold, peak)
thn = thinning / 16

tmp = scale(127, peak)

exin = core.std.Maximum(yuv_clip, threshold=peak // (protection + 1)).std.Minimum()

diff = core.std.Expr([yuv_clip, exin], ['y {lum} < y {lum} ? x {thr} + > x y {lum} < y {lum} ? - 0 ? {i} +'.format(lum=lum, thr=thr, i=tmp)])


expr = 'x {i} - {thn} * {peak} +'.format(i=tmp, thn=thn, peak=peak)
linemask = core.std.Expr([core.std.Minimum(diff)], [expr]).std.Convolution(matrix=[1, 1, 1, 1, 1, 1, 1, 1, 1])

# merge
result_clip = core.std.MaskedMerge(darken_clip, yuv_clip, linemask)

result_clip.set_output()


The problem I have now is that with the linemask, the edge darken effect is so subtle that it's unnoticeable. Without the mask it looks good.

[foxyshadis]

I kind of get it working now. I created an average clip with convolution matrix=[1, 1, 1, 1, 1, 1, 1, 1, 1]), and then compare the color-corrected clip to the average clip. If the value of the pixel in the cc is less than the average, change it to the value of the average. The line edge will darkened. Then I have copy the linemask code from fastlinedarkenmod and do a maskmerge between cc and the darken clip.


lut_file = r'test.cube'

clip = core.ffms2.Source(r"abc")

rgb_clip = core.resize.Bicubic(clip, matrix_in_s="709", format=vs.RGBS)

match_clip = core.timecube.Cube(rgb_clip, cube=lut_file)

yuv_clip = core.resize.Bicubic(match_clip, matrix_s="709", format=vs.YUV420P8)

avg_clip = core.std.Convolution(yuv_clip, matrix=[1, 1, 1, 1, 1, 1, 1, 1, 1])

# compares clip to average, change value when below average
def darken_edge(x, y):
if x < y:
return x
else:
if abs(x-y) < 30:
return y
else:
return x

darken_clip = core.std.Lut2(clipa=yuv_clip, clipb=avg_clip, function=darken_edge)

# linemask codes from fastlinedarkenmod
def scale(value, peak):
return value * peak // 255

strength=48
protection=5
luma_cap=191
threshold=4
thinning=54


peak = (1 << yuv_clip.format.bits_per_sample) - 1 #255
Str = strength / 128
lum = scale(luma_cap, peak)
thr = scale(threshold, peak)
thn = thinning / 16

tmp = scale(127, peak)

exin = core.std.Maximum(yuv_clip, threshold=peak // (protection + 1)).std.Minimum()

diff = core.std.Expr([yuv_clip, exin], ['y {lum} < y {lum} ? x {thr} + > x y {lum} < y {lum} ? - 0 ? {i} +'.format(lum=lum, thr=thr, i=tmp)])


expr = 'x {i} - {thn} * {peak} +'.format(i=tmp, thn=thn, peak=peak)
linemask = core.std.Expr([core.std.Minimum(diff)], [expr]).std.Convolution(matrix=[1, 1, 1, 1, 1, 1, 1, 1, 1])

# merge
result_clip = core.std.MaskedMerge(darken_clip, yuv_clip, linemask)

result_clip.set_output()


The problem I have now is that with the linemask, the edge darken effect is so subtle that it's unnoticeable. Without the mask it looks good.

Perhaps add .std.Maximum or .std.Inflate to the linemask? That'll widen its effective area.