Working on an encode - dirt removal + sharpening + dithering + etc - help!

[Blue_MiSfit]

Hey folks,

So, I'm working on a script. It's loosely based around one I found floating around, and has been tweaked for my purposes. However, when I run it, the results of the script are very screwed up. Everything is green :)

Source is a studio master - PAL 4:2:2 8mbit MPEG-2 - 4x3. It's a B+W film.

Here's the current code:

MPEG2Source("E:\Rework\122113\VIDEO_TS\122113.d2v", cpu=0)
greyscale
ConvertToYV12
a=last
RemoveDirt()

DeGrainMedian(limitY=2,limitUV=3,mode=3)
DeGrainMedian(limitY=2,limitUV=3,mode=3)

gradfun2db(1.7)
mt_average(AddGrainC(2),AddGrainC(2,seed=10))

grainy=dfttest(smode=0,sbsize=5,tbsize=5,sigma=.8)
clean=dfttest(smode=0,sbsize=5,tbsize=5,sigma=3)

dull=grainoptimizer(grainy,clean,tdist=4000)

sharp=dull.limitedsharpenfaster(2,2,strength=150)
soothe(sharp,dull).subtitle("sharp as fuck")

interleave(last,a)


With the interleave, I can clearly see that the original frames are fine (but very dirty), and the processed frames are very goofed up. Everything was fine until I add the limitedsharpenfaster call. I know this has something to do with Masktools :)

here's my limitedsharpenfaster.avsi:

# LimitedSharpen() ( a modded version, 29 Oct 2005 )
#
# A multi-purpose sharpener by Didée
#
#
# Changes in this mod:
#
# - RemoveGrain >= v0.9 IS REQUIRED!!
# ==================================
#
# - Smode=4 / sometimes does the magic ;-)
# - a separate "undershoot" parameter, to allow for some line darkening in comic or Anime
# - Lmode=3 / on edges, limited sharpening with zero OS & US. On not-edges, limited sharpening with specified OS + LS
# - "soft" acts different now: no more boolean true/false, but instead integer 0 - 100 (or -1 -> automatic)
# instead of blurring before finding minima/maxima, it now softens the "effect-of-sharpening"
# - edgemode=-1 now shows the edgemask. (scaling still not implemented :p )
#
## - MODIFIED version using MaskTools 2.0

function LimitedSharpenFaster( clip clp,
\ float "ss_x", float "ss_y",
\ int "dest_x", int "dest_y",
\ int "Smode" , int "strength", int "radius",
\ int "Lmode", bool "wide", int "overshoot", int "undershoot",
\ int "soft", int "edgemode", bool "special",
\ int "exborder" )
{
ox = clp.width
oy = clp.height

Smode = default( Smode, 3 )
ss_x = (Smode==4)
\ ? default( ss_x, 1.25)
\ : default( ss_x, 1.5 )
ss_y = (Smode==4)
\ ? default( ss_y, 1.25)
\ : default( ss_y, 1.5 )
dest_x = default( dest_x, ox )
dest_y = default( dest_y, oy )
strength = (Smode==1)
\ ? default( strength, 160 )
\ : default( strength, 100 )
strength = (Smode==2&&strength>100) ? 100 : strength
radius = default( radius, 2 )
Lmode = default( Lmode, 1 )
wide = default( wide, false )
overshoot = default( overshoot, 1)
undershoot= default( undershoot, overshoot)
softdec = default( soft, 0 )
soft = softdec!=-1 ? softdec : sqrt( (((ss_x+ss_y)/2.0-1.0)*100.0) ) * 10
soft = soft>100 ? 100 : soft
edgemode = default( edgemode, 0 )
special = default( special, false )
exborder = default( exborder, 0)
#radius = round( radius*(ss_x+ss_y)/2) # If it's you, Mug Funky - feel free to activate it again
xxs=round(ox*ss_x/8)*8
yys=round(oy*ss_y/8)*8
smx=exborder==0?dest_x:round(dest_x/Exborder/4)*4
smy=exborder==0?dest_y:round(dest_y/Exborder/4)*4

clp.isYV12() ? clp : clp.converttoyv12()

ss_x != 1.0 || ss_y != 1.0 ? last.lanczosresize(xxs,yys) : last
tmp = last

edge = mt_logic( tmp.mt_edge(thY1=0,thY2=255,"8 16 8 0 0 0 -8 -16 -8 4")
\ ,tmp.mt_edge(thY1=0,thY2=255,"8 0 -8 16 0 -16 8 0 -8 4")
\ ,"max") .mt_lut("x 128 / 0.86 ^ 255 *") #.levels(0,0.86,128,0,255,false)

tmpsoft = tmp.removegrain(11,-1)
dark_limit1 = tmp.mt_inpand()
bright_limit1 = tmp.mt_expand()
dark_limit = (wide==false) ? dark_limit1 : dark_limit1 .removegrain(20,-1).mt_inpand()
bright_limit = (wide==false) ? bright_limit1 : bright_limit1.removegrain(20,-1).mt_expand()
minmaxavg = special==false
\ ? mt_average(dark_limit1, bright_limit1)
\ : mt_merge(dark_limit,bright_limit,tmp.removegrain(11,-1),Y=3,U=-128,V=-128)

Str=string(float(strength)/100.0)
normsharp = Smode==1 ? unsharpmask(strength,radius,0)
\ : Smode==2 ? sharpen(float(strength)/100.0)
\ : Smode==3 ? mt_lutxy(tmp,minmaxavg,yexpr="x x y - "+Str+" * +")
\ : mt_lutxy(tmp,tmpsoft,"x y == x x x y - abs 16 / 1 2 / ^ 16 * "+Str+
\ " * x y - 2 ^ x y - 2 ^ "+Str+" 100 * 25 / + / * x y - x y - abs / * + ?")

OS = string(overshoot)
US = string(undershoot)
mt_lutxy( bright_limit, normsharp, yexpr="y x "+OS+" + < y x y x - "+OS+" - 1 2 / ^ + "+OS+" + ?")
mt_lutxy( dark_limit, last, yexpr="y x "+US+" - > y x x y - "+US+" - 1 2 / ^ - "+US+" - ?")

Lmode==1 ? mt_clamp(normsharp, bright_limit, dark_limit, overshoot, undershoot) : last

normal = last
zero = mt_clamp(normsharp, bright_limit, dark_limit, 0,0)

Lmode==3 ? mt_merge(normal,zero,edge.mt_inflate()) : normal

edgemode==0 ? last
\ : edgemode==1 ? mt_merge(tmp,last,edge.mt_inflate().mt_inflate().removegrain(11,-1),Y=3,U=1,V=1)
\ : mt_merge(last,tmp,edge.mt_inflate().mt_inflate().removegrain(11,-1),Y=3,U=1,V=1)

AMNT = string(soft)
AMNT2 = string(100-soft)
sharpdiff=mt_makediff(tmp,last)
sharpdiff2=mt_lutxy(sharpdiff,sharpdiff.removegrain(19,-1),
\ "x 128 - abs y 128 - abs > y "+AMNT+" * x "+AMNT2+" * + 100 / x ?")

soft==0 ? last : mt_makediff(tmp,sharpdiff2)

(ss_x != 1.0 || ss_y != 1.0)
\ || (dest_x != ox || dest_y != oy) ? lanczosresize(dest_x,dest_y) : last

ex=blankclip(last,width=smx,height=smy,color=$FFFFFF).addborders(2,2,2,2).coloryuv(levels="TV->PC")
\.blur(1.3).mt_inpand().blur(1.3).bicubicresize(dest_x,dest_y,1.0,.0)
tmp = clp.lanczosresize(dest_x,dest_y)

clp.isYV12() ? ( exborder==0 ? tmp.mergeluma(last)
\ : mt_merge(tmp,last,ex,Y=3,U=1,V=1) )
\ : ( exborder==0 ? tmp.mergeluma(last.converttoyuy2())
\ : tmp.mergeluma( mt_merge(tmp.converttoyv12(),last,ex,Y=3,U=1,V=1)
\ .converttoyuy2()) )

(edgemode!= -1) ? last : edge.lanczosresize(dest_x,dest_y).greyscale

return last
}


I'm using the two versions (according to Windows:
mt-MaskTools-25.dll - 644kb - 2.0.34.0 version
MaskTools.dll - 188kb - 1.5.1.0 version

Which ones *should* I be using? :)

Thanks guys!!! I just got Removedirt working (it was a little confusing, but WOW that filter is amazing for cleaning up old films!!

~MiSfit

[Didée]

Try to return interleave(sharp,a) instead. If that one is not "green", then you've fallen for the soothe-of-SeeSaw pitfall. ;)

[Blue_MiSfit]

Interesting... The green problem wasn't really consistent. I played with it a bit and realized that RemoveDirt as was doing all the work, and all the fancy dithering, grain optimization, and sharpening did almost nothing to the original. It seems like RemoveDirt is doing some strange things! It seems to kill a lot of details unfortunately. I will have to study the (horribly confusing and poorly formatted) documentation!

Thanks for the quick replay though Didée! Maybe you can advise me on how to configure RemoveDirt without so much spatial annhilation? :) Maybe the two go hand in hand...

Im intrigued by what you said about soothe of seesaw. I have seesaw.avsi autoloading - is that the problem? If so, where is "another" soothe function?

At any rate, I'm still hugely impressed by RemoveDirt's ability to surgically extract bits of dirt.


MPEG2Source("E:\Rework\122113\VIDEO_TS\122113.d2v", cpu=0)
greyscale
ConvertToYV12
a=last

RemoveDirt()
interleave(a,last)

#cleaned=last
#
#gradfun2db(1.7)
#mt_average(AddGrainC(2),AddGrainC(2,seed=10))
#grainy=last
#clean=fft3dgpu(sigma=4)
#dull=grainoptimizer(grainy,clean,tdist=4000)
#dithered=last
#sharp=dull.limitedsharpenfaster(ss_x=2, ss_y=2, SMode=4, strength=150)
#sharpened=soothe(sharp,dull)

#interleave(a,sharpened.subtitle("cleaned"), dithered.subtitle("dithered"), sharpened.subtitle("sharpened"))

[Blue_MiSfit]

changing repmode to 2 or 5 made things worse is there any way to repair more than the default of 16?

[Didée]

Im intrigued by what you said about soothe of seesaw. I have seesaw.avsi autoloading - is that the problem? If so, where is "another" soothe function? That's a problem, yes. The "original" SeeSaw internally uses a Soothe function that doesn't care for chroma. If you import SeeSaw.avs after Soothe.avs, then the standalone Soothe function gets overloaded. And if you import SeeSaw before Soothe, then SeeSaw isn't guaranteed to work correctly anymore.
Foxyshadis once had made the SeeSaw_MT2 version with a renamed internal Soothe function, which eleminiates the problem.
(BTW, it's a pity that Avisynth doesn't provide "private" functions-inside-functions, the same way as variables are handled. Would avoid lots of confusion!)


Maybe you can advise me on how to configure RemoveDirt without so much spatial annhilation?
There's not really much to do. RemoveDirt basically is just a temporal median with block-based postprocessing. The median part of course acts strong, killing much crud, but also detail. If the blocks after killing noise+detail still fit together well, then it seems okay from the filter's point of view. There's also the other protection mode that counts the number of changed pixels ... with that mode you can reduce detail loss quite nicely. But no surprise: if you reduce detail loss, then you also reduce dirt removal. It's always the same story. :)

[Blue_MiSfit]

There's not really much to do. RemoveDirt basically is just a temporal median with block-based postprocessing. The median part of course acts strong, killing much crud, but also detail. If the blocks after killing noise+detail still fit together well, then it seems okay from the filter's point of view. There's also the other protection mode that counts the number of changed pixels ... with that mode you can reduce detail loss quite nicely. But no surprise: if you reduce detail loss, then you also reduce dirt removal. It's always the same story.


What you said makes perfect sense :) I'm curious about the other protection mode? I'm ok with finding a balance between dirt removal and detail loss. In RemoveDirt.avs, I only see two options, bool _grey and int repmode. I played with repmode, but haven't messed with _grey. Surely you must be referring to an integer or a float somewhere else! Must be in RestoreMotionBlocks...

~miSfit

[thetoof]

and all the fancy dithering, grain optimization, and sharpening did almost nothing to the original.

Of course! This is almost a direct copy of what I suggested for another source... The first dfttest's purpose is to remove all the noise/grain you don't want. The second is to remove ALL the grain to give a clean clip do grainopt. Also, with a source that is already grainy (i.e. with no or almost no banding), you don't need gradfun + addgrain; the only thing you need is to ajust the first dfttest to remove what you want, but keep the good grain that prevents banding.

You can also use another denoiser that could be more adapted to your source, but the most important thing is that the denoining step must be tweaked depending on your source. It's not an "all purpose dithering filter chain".

[Blue_MiSfit]

I think I stole your script and vivisected it! I'm so sorry thetoof :)

It was a pretty interesting script, but I definitely modified it to fit what I thought would be necessary.

Now that I understand how to do what you suggest, I will go back and play some more - but first I want to keep RemoveDirt from throwing away so much detail / changing things spatially too much!

One step at a time :)

~MiSfit

[thetoof]

I'm so sorry thetoof :)
No need to be :p

You can also find an interesting use of fft3dfilter instead of the first dfttest call in the same thread where you found the original script.

[Blue_MiSfit]

Interesting!

Now, if only I could figure out how to tweak the "other protection mode" as Didee called it.

Help! :)

~MiSfit

[thetoof]

I've just encountered a source where I need to use removedirt().
The only way I found to remove some dirt & prevent major detail loss was to disable removegrain.

Then, I realized that the only thing the rest of the script removes is very small dirt that can be removed with any good denoiser. Also, RemoveGrain is capable of removing mid-sized dirt, but can't do a thing with big spots (that are still too small to be removed by despot).

So, I thought "TemporalDegrain kicks ass when you want to remove grain... let's see what it can do with dirt!" and it worked!

Original (http://img215.imageshack.us/img215/2998/origeu0.jpg)//RemoveDirt (http://img143.imageshack.us/img143/6987/rdll9.jpg)//RemoveDirtwithout RemoveGrain (http://img143.imageshack.us/img143/565/norgrc8.jpg)//dfttest (http://img259.imageshack.us/img259/2223/dfthc5.jpg)//TemporalDegrain (http://img299.imageshack.us/img299/9505/tdxu0.jpg)
Nothing is optimized (default settings).

But, as we all know, it's damn slow... so I'm still eager to learn another method (like removedirt, but without the spatial scew up it does)
Didée, what is that other protection mode you're talking about? I too only have two options...

[Didée]

Well, I was only loosely remembering v0.9 RemoveDirt's principle of operation when talking about that "other protection mode", and was referring not to the RemoveDirt function as a whole, but to RestoreMotionBlocks (which is the main filter responsible for noise+dirt removal, when fed with a temporal median). After revisiting the docs, it's the three modes "SAD", "NSAD" and "NPC" that can be used to decide if the pixel changes within an 8x8 block are to be considered correct, or are probably incorrect.
noise=0 & noisy=0 --> SAD
noise>0 & noisy=0 --> NSAD ("noise adjusted" SAD)
noise>0 & noisy>0 --> NPC (noisy pixel counting)

For more insight, look into the documentation. When doing so, you are reading the same sentences as I do. :headache: :D

NPC is what I had in mind with "the other" mode, however now I see it's the default mode hardcoded in RemoveDirt v0.9.

[g-force]

GTDegrain

-G

[Blue_MiSfit]

Thanks Didée!

Wading through kassando's documentation can be... daunting. I will look into those parameters. The way RemoveDirt works is a little confusing to me :)

~MiSfit

[foxyshadis]

If it wasn't soothe, it could also be mt_average(AddGrainC(2),AddGrainC(2,seed=10)) -> you need a chroma="process" in there. All of the mt_ (and old mt) functions just give you garbage for chroma, which occasionally happens to be correct but usually happens to be all zeroed (which shows up as green).