Region-based structure line detection for cartoons (Cartoons mask)

[Dogway]

The results looks good!
I didn't know it was all for Gibbs noise, in that case you don't want to touch edges but just right outside them. Normally edge masks will have their center at the gradient change, that means half the edge mask will occupy the edge itself. FlatMask() is very robust so what you can do is subtract the edge mask with FlatMask's ridge mask.

Example:

a=ex_edge("qprewitt")#~ .ex_expand()
b=flatmask()
ex_makediff(a,b,dif=false)

I updated the filter with new args so now you can tune the mask gain (to convert from ridge to a proper flat mask) and low cutoff. It's also faster since I changed the derivatives to custom ones and changed defaults.

[Blankmedia]

Originally Posted by Dogway

I didn't know it was all for Gibbs noise, in that case you don't want to touch edges but just right outside them.

Nice, I'll try that snippet and your new flatmask!

It wasn't all about Gibbs Noise, but the mask at first picked up a bit to much of it, so it was protecting it. (if I understand correctly, i.e the thin parallel line).


Here's a sample, it encoded at 1.2fps with that command :

avs4x26x64 --x26x-binary x265 --preset veryslow --profile main10 --merange 44 --no-rect --no-am --no-sao --aq-mode 1 --aq-strength 1.0 --rd 4 --psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1 --qcomp 0.75 --rdpenalty 1 --tu-inter-depth 2 --tu-intra-depth 2 --ctu 32 --max-tu-size 16 --input-depth 16 --dither --crf 22 --pools 4 --colormatrix "bt709" --colorprim "bt709" --transfer "bt709" --videoformat 2 --sar 1:1 -o "output.265" "input.avs"

https://mega.nz/file/hl4mlLDS#hPe987...qHUZYZD44mqtDI

Originally Posted by Blankmedia

Thanks Dogway I used your code and it's amazing!

Also I added a bit of a luma mask (thanks again) and a debanding call after your cleanup. I made 2 Hqderingmod call to reduce some parallel lines, but I'm not sure yet if I' gonna keep it.

It seems to give consistent result :

src = FFMS2() 
src.HQderingmod(mrad=2,minp = 2 , nrmode =0, sharp = 0, show=False).HQderingmod(mrad=2,minp = 2 , nrmode =3, sharp = 0, show=False)
a = last
ex_boxblur(0,1,mode="mean")
turnright()
ex_median("verticalS")
turnleft()
ex_median("IQM")
neo_f3kdb(5,preset="very high",grainy=0, grainc=0, blur_first=true, sample_mode=4)

# Add lumamask to get dark haris and shoes and stuff, also limit texture degradation in dark area
lumen = lumamask(b=00,w=50, invert=True,uv=128).Santiag().ex_inflate(1,uv=128).Converttoy()
lumen = lumen.Levels(40,0.7,240,0,255)
# Valley Mask
dana = FlatMask(1)
# Add regular edge mask
a=a.ex_edge("qprewitt").Converttoy()
emask = dana.ex_lutxy(a,"x dup * 0.1 * y + 0.5 *")

#~ emask = ex_logic(emask, lumen, "or") ## create artefact
emask = emask.ex_lutxy(lumen,"y x > y x ?")
# Clean up
emask.ex_smooth(2,sharp=true)

This is my whole script so far, but it needs tuning a bit. I went way over board with a lot of things.

FFMS2(chemin)
RoboCrop(LeftAdd=2, TopAdd=2, RightAdd=2,  BotAdd =2, align=True)
fmtc_bitdepth(bits=32)
fmtc_resample(css="444", fulls=False, fulld=True)
fmtc_matrix(mats="601", matd="709", fulls=True, fulld=True)

fmtc_matrix(mat="709", fulls=True, fulld=False, col_fam="YUV")
fmtc_resample (css="420")
src16 = fmtc_bitdepth(bits=16, dmode=8)
fmtc_bitdepth(bits=8, dmode=8)
src = last
HQderingmod(mrad=2,minp = 2 , nrmode =0, sharp = 0, show=False).HQderingmod(mrad=2,minp = 2 , nrmode =3, sharp = 0, show=False)
ex_boxblur(0,1,mode="mean")
turnright()
ex_median("verticalS")
turnleft()
ex_median("IQM")
neo_f3kdb(5,preset="very high",grainy=0, grainc=0, blur_first=true, sample_mode=4)

# Add lumamask to get dark haris and shoes and stuff, also limit texture degradation in dark area
lumen = lumamask(b=00,w=50, invert=True,uv=128).Santiag().ex_inflate(1,uv=128).Converttoy()
lumen = lumen.Levels(40,0.7,240,0,255)

# Valley Mask
dana = FlatMask(1)
# Add regular edge mask
a=src.ex_edge("qprewitt").Converttoy()
emask = dana.ex_lutxy(a,"x dup * 0.1 * y + 0.5 *")



emask = emask.ex_lutxy(lumen,"y x > y x ?")
# Clean up
emask = emask.ex_smooth(2,sharp=true)

emask16 = emask.fmtc_bitdepth(bits=16)

src16= src16.GibbsNoiseBlock(mode_RG=26)

sharp0 = src16.NonlinUSM(z=3, pow=1.1, str=0.25, rad=9).mergechroma(src16)
sup1   = src16.MSuper(pel=2,sharp=2)
sup2   = sharp0.MSuper(pel=2,sharp=2,levels=1)
bv1    = sup1.MAnalyse(isb=true, delta=1)
fv1    = sup1.MAnalyse(isb=false,delta=1)
bv2    = sup1.MAnalyse(isb=true, delta=2)
fv2    = sup1.MAnalyse(isb=false,delta=2)

sharpy = src16.MDegrain2(sup2,bv1,fv1,bv2,fv2)


denoised = src16.GibbsNoiseBlock(mode_RG=26).minideen(radius=1).Halobuster(h=12)
denoised  = denoised.TemporalDegrain2(degrainTR=1, grainLevel=1, postFFT=0, postTR=1, postSigma=1.0, postDither=-1, postMix=0, extraSharp=True, fftThreads=1, debug=false)

ex_merge(denoised, sharpy, emask16,true).maa2()
DFMDeRainbow(maskthresh=12)
EDI_RPow2(4,CShift="Spline64Resize",FWidth=960,FHeight=720, nsize=0, nns=3, qual=2, etype=1, taps=4)
emask16 = emask16.spline64resize(960,720)


source = last
sharp0 = source.mt_adddiff(mt_makediff(source,source.removegrain(4)),U=2,V=2) # "median sharpen" (won't create halos on its own, IF the source is halo-free)
sup1   = source.MSuper(pel=2,sharp=2)
sup2   = sharp0.MSuper(pel=2,sharp=2,levels=1)
bv1    = sup1.MAnalyse(isb=true, delta=1)
fv1    = sup1.MAnalyse(isb=false,delta=1)
bc1    = source.MCompensate(sup2,bv1,thSAD=500)  # using the super clip from the median-sharpener, to provide
fc1    = source.MCompensate(sup2,fv1,thSAD=500)  # more headroom for the limiting process
max    = mt_logic(bc1,fc1,"max").mt_logic(source,"max")
min    = mt_logic(bc1,fc1,"min").mt_logic(source,"min")
sharp1 = source.LSFmod(preset="slow", strength=200, smode=5, edgemode=1).CAS(0.5).mergechroma(last) # chroma sharpen usually is unadvantageous

sharpy2 = sharp1.mt_clamp(max,min,0,0,U=2,V=2).FastLineDarkenMOD4(strength=75, prot=60, luma_cap=190, threshold=4, thinning=1).FastLineDarkenMOD4(strength=75, prot=60, luma_cap=190, threshold=4, thinning=1)
#~ shapy2

ex_merge(source, sharpy2, emask16,true).maa2()

emask16 = emask16.ex_inpand(1,"plus",thres=128)

deh = source.FineDehalo(rx=2.5, ry=2.5, thmi=80, thma=128, thlimi=50, thlima=100, darkstr=0.3, brightstr=1.0, showmask=0, contra=0.3, excl=true)
ex_merge(deh, last, emask16,true)
w=656
h=480
fr=44
a1 = src16.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Source frame" + String(fr))
a2 = emask16.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Mask frame" + String(fr)).ConvertToYUV420()
a3 = last.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Result frame" + String(fr))
ligne1 = Stackhorizontal(a1, a3, a2)

fr=77
a1 = src16.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Source frame" + String(fr))
a2 = emask16.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Mask frame" + String(fr)).ConvertToYUV420()
a3 = last.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Result frame" + String(fr))
ligne2 = Stackhorizontal(a1, a3, a2)

fr=5000
a1 = src16.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Source frame" + String(fr))
a2 = emask16.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Mask frame" + String(fr)).ConvertToYUV420()
a3 = last.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Result frame" + String(fr))
ligne3 = Stackhorizontal(a1, a3, a2)

fr=10000
a1 = src16.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Source frame" + String(fr))
a2 = emask16.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Mask frame" + String(fr)).ConvertToYUV420()
a3 = last.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Result frame" + String(fr))
ligne4 = Stackhorizontal(a1, a3, a2)

fr=15000
a1 = src16.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Source frame" + String(fr))
a2 = emask16.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Mask frame" + String(fr)).ConvertToYUV420()
a3 = last.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Result frame" + String(fr))
ligne5 = Stackhorizontal(a1, a3, a2)

fr=20000
a1 = src16.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Source frame" + String(fr))
a2 = emask16.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Mask frame" + String(fr)).ConvertToYUV420()
a3 = last.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Result frame" + String(fr))
ligne6 = Stackhorizontal(a1, a3, a2)

fr=25000
a1 = src16.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Source frame" + String(fr))
a2 = emask16.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Mask frame" + String(fr)).ConvertToYUV420()
a3 = last.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Result frame" + String(fr))
ligne7 = Stackhorizontal(a1, a3, a2)

fr=30000
a1 = src16.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Source frame" + String(fr))
a2 = emask16.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Mask frame" + String(fr)).ConvertToYUV420()
a3 = last.Trim(fr, fr).Spline64Resize(w,h).Subtitle("Result frame" + String(fr))
ligne8 = Stackhorizontal(a1, a3, a2)

stack = Stackvertical(ligne1, ligne2, ligne3, ligne4, ligne5, ligne6, ligne7, ligne8)

return stack

#~ ConvertToStacked().Dither_out()
Prefetch(threads=1)

Zoom in


Thanks to everyone again.

Hello @Blankmedia

it looks very impressive.
Could you advice how to use your script in StaxRip app?

[Blankmedia]

Originally Posted by madey83

Hello @Blankmedia

it looks very impressive.
Could you advice how to use your script in StaxRip app?

I wish I could help you but I've nerver used StaxRip.

But can't you just use avisynth and x264 or x265?