Grain appears after sharpening, need suggestions on removal (anime)

[Comatose]

Hi! I'm pretty much new when it comes to avisynth and encoding, but I've been reading other people's scripts and learning from them.

Anyway, like the title says, I'm training on an anime called Polyphonica. My script is pretty much done, but I'm stuck on how to get rid of the grain (did I use the right term?) that appears after using LimitedSharpenFaster.

LSF makes the image look a lot more defined and gives it the HD-ish look, but then that annoying grain appears.

See the before (http://img247.imageshack.us/img247/4913/lsfbqi2.png) and after (http://img174.imageshack.us/img174/3711/lsfaqe7.png) images.

Pay special attention to the front and back of the bike, where it is most noticeable (but it is still a bit subtle).

The script used to produce it (the "after LSF" image):

directshowsource("e11_raw.mkv")

Trim(0,2156)

lp("RemapFrames")
ReplaceFramesSimple(Tweak(sat=1.00, cont=0.93, bright=-11), mappings="[400 551] [552 624] [1391 1458]")
ReplaceFramesSimple(Tweak(sat=1.00, cont=0.93, bright=-7), mappings="[308 399]")
ReplaceFramesSimple(Tweak(sat=1.00, cont=0.93, bright=-5), mappings="[95 306] [625 705] [1459 1580]")
ReplaceFramesSimple(Tweak(sat=1.00, cont=0.93, bright=-3), mappings="[711 1078] [1581 1660]")
Levels(16, 1, 235, 0, 255, coring=false)
lp("RemoveGrainsse2")
lp("vaguedenoiser")
lp("awarpsharp")
lp("toon")
lp("warpsharp")
lp("mt_masktools")
ls("limitedsharpenfaster")
lp("sangnom")
lp("masktools")
ls("aa")
lp("degrainmedian")
lp("convolution3dyv12")

RemoveGrain(mode=1)
VagueDenoiser(threshold=0.8, method=3, nsteps=6, chromaT=2)
Convolution3d(preset="animeHQ")

Toon(strength=0.3)
WarpSharp(96, 10, 128, -0.6)


LimitedSharpenFaster(smode=4, strength=70)
ReplaceFramesSimple(DeGrainMedian(limitY=2,limitUV=3,mode=1).DeGrainMedian(limitY=2,limitUV=3,mode=1).DeGrainMedian(limitY=2,limitUV=3,mode=1), mappings="[1581 1660]")
aa()
LimitedSharpenFaster(smode=4, strength=120)

Note that VagueDenoiser with threshold=1.4 hurts details. Using DeGrainMedian 3 times removed 99% of the grain, but hurts details like lines in hair badly, so I ended up using it just for one scene where no details are lost.

Thanks in advance! :)

[Terranigma]

That grain is very subtle. Maybe you could try using RemoveGrain modes 1, 5 or 2 (I listed them in the order from the weakest to greatest compression) after Limitedsharpen.

[Comatose]

Thanks. The only one that seems to remove most of it is mode 2, twice, but it's getting hard to compare when the difference is that small. Is there some filter that will highlight (something like Tweak(sat=0)) the difference between 2 frames for me?

[Terranigma]

Thanks. The only one that seems to remove most of it is mode 2, twice, but it's getting hard to compare when the difference is that small. Is there some filter that will highlight (something like Tweak(sat=0)) the difference between 2 frames for me?
well, you could try using subtract and stackhorizontal.
e.g.'s

left=RemoveGrain(mode=5)
right=RemoveGrain(mode=2)
stackhorizontal(left,right)
(the width's size will be doubled)

and for subtract:
input=RemoveGrain(mode=2)
subtract(input)

[Dark Shikari]

ttempsmooth?

hqdn3d?

[kumi]

Is there some filter that will highlight (something like Tweak(sat=0)) the difference between 2 frames for me?

################################################################################################
# BeforeAfterDiff by Corran (Eric Parsons) Based off of BeforeAfter by AbsoluteDestiny #
# and Corran #
# #
#order = Specifies if the filtered footage will be on the top #
# or bottom of screen (Default=true | bottom) #
#showdiff = enables/disable difference layer (Default=false | Disabled) #
#amp = Enable/disable amplify noise (Default=true | Enabled) #
#xdisplace = slides difference layer horizontally by x number #
# of pixels (Default=2*width()/3 | far right hand side of screen) #
#filters = filters used to see differences before and after their application #
# #
#Example: #
#beforeafterdiff(order=true,showdiff=true,amp=false,xdisplace=300,filters="""deen("a3d",4)""") #
################################################################################################

function beforeafterdiff(clip a, bool "order", bool "showdiff", bool "amp",int "xdisplace", string "filters"){

Assert((Float(Width(a))/16)==Round(Width(a)/16), "Source image width must be a multiple of 16")

amp = Default(amp, True)
order = Default(order, True)
showdiff = Default(showdiff, False)
placement = Default(xdisplace, 2*width(a)/3)

placement = (placement > 2*Width(a)/3) ? 2*Width(a)/3 : placement
placement = (placement < 0) ? 0 : placement
xval = (placement > Width(a)/3) ? 5 : placement+Width(a)/3+5
line = BlankClip(a,color=$FFFFFF).Crop(0,0,(Width(a)),2)
line2 = BlankClip(a,color=$FFFFFF).Crop(0,0,2,(Height(a)))
b = Eval("a."+filters)

c = (amp==true) ? Subtract(a,b).Levels(120, 1, 140, 0, 255).Tweak(bright=-5).ConvertToRGB24().\
Crop(placement,0,Round(-Width(a)+placement+Width(a)/3),0).Subtitle("Difference").\
Subtitle("(Amplified)",y=35).Overlay(line2,x=0) : Subtract(a,b).ConvertToRGB24().\
Crop(placement,0,Round(-Width(a)+placement+Width(a)/3),0).Subtitle("Difference").\
Overlay(line2,x=0)

c = Overlay(c,line2,x=Width(c)-2)

d = (order==true) ? StackVertical(a.Crop(0,0,0,-Height(a)/2).Subtitle("Before",x=xval),b.\
Crop(0,Height(a)/2,0,0).Subtitle("After",x=xval)).Overlay(line,y=Height(a)/2-1) : StackVertical(b.\
Crop(0,0,0,-height(a)/2).Subtitle("After",x=xval),a.Crop(0,Height(a)/2,0,0).\
Subtitle("Before",x=xval)).Overlay(line,y=Height(a)/2-1)

e = (showdiff==false) ? d : Overlay(d,c,x=placement)

Return e
}

[Comatose]

Thanks everybody. I ended up using HQDN3D.

[Comatose]

So, um, I eventually dropped HQDN3D in favor of TTempSmooth, but still need suggestions. I'd really like to improve the image, if at all possible. Encoding with X264 (@ 1500 kbps) makes it even worse :(
I used a friend's megui settings, and while it improved the quality,

I'm still unhappy with the result. I've also come to dislike what I got before encoding, as well ><

I want to make it look better ;-; The noise bothers me a lot, especially after encoding.

Screenshots:
Reds raw (http://img169.imageshack.us/img169/2655/rawce8.png) | Lagarith (http://img174.imageshack.us/img174/7573/lagsvi8.png) | h264 1500kbps (http://img146.imageshack.us/img146/1943/h264jk9.png)

Bike raw (http://img208.imageshack.us/img208/2723/rawbikeed5.png) | Lagarith (http://img146.imageshack.us/img146/5345/lagsbikeou1.png) | h264 1500kbps (http://img228.imageshack.us/img228/206/h264bikefo0.png)

x264 params (from megui):
--pass 2 --bitrate 1500 --stats ".stats" --keyint 240 --min-keyint 24 --ref 10 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --subme 7 --trellis 2 --analyse all --8x8dct --pbratio 1.4 --ratetol 3.0 --qcomp 0.5 --me umh --merange 25 --threads auto --thread-input --progress --no-psnr --no-ssim --output "output" "input"

Avisynth script used to encode Lagarith (almost the same as before):
directshowsource("e11_raw.mkv")
assumefps(24000,1001)

Trim(0,2156)
lp("RemapFrames")
ReplaceFramesSimple(Tweak(sat=1.00, cont=0.93, bright=-11), mappings="[400 551] [552 624] [1391 1458]")
ReplaceFramesSimple(Tweak(sat=1.00, cont=0.93, bright=-7), mappings="[308 399]")
ReplaceFramesSimple(Tweak(sat=1.00, cont=0.93, bright=-5), mappings="[95 306] [625 705] [1459 1580]")
ReplaceFramesSimple(Tweak(sat=1.00, cont=0.93, bright=-3), mappings="[711 1078] [1581 1660]")
Levels(16, 1, 235, 0, 255, coring=false)
lp("RemoveGrainsse2")
lp("vaguedenoiser")
lp("awarpsharp")
lp("toon")
lp("warpsharp")
lp("mt_masktools")
ls("limitedsharpenfaster")
lp("sangnom")
lp("masktools")
ls("aa")
lp("degrainmedian")
lp("convolution3dyv12")

RemoveGrain(mode=1)
VagueDenoiser(threshold=0.8, method=3, nsteps=6, chromaT=2)
Convolution3d(preset="animeHQ")

Toon(strength=0.3)
WarpSharp(96, 10, 128, -0.6)


LimitedSharpenFaster(smode=4, strength=70)
ReplaceFramesSimple(DeGrainMedian(limitY=2,limitUV=3,mode=1).DeGrainMedian(limitY=2,limitUV=3,mode=1).DeGrainMedian(limitY=2,limitUV=3,mode=1), mappings="[1581 1660]")
aa()
LimitedSharpenFaster(smode=4, strength=120)

lp("ttempsmooth")
TTempSmooth(maxr=7,lthresh=10,cthresh=10,lmdiff=2,cmdiff=3,strength=2,scthresh=12,fp=true)
RemoveGrain(mode=1)

Thanks for any suggestions ;-; PS: It is a must that further filtering doesn't hurt details. If it does, then I'll consider this to be as good as it's going to get.

[kumi]

IMO, you're bringing out an excessive amount of haloing from all those sharpening steps.

[Zep]

So, um, I eventually dropped HQDN3D in favor of TTempSmooth, but still need suggestions. I'd really like to improve the image, if at all possible. Encoding with X264 (@ 1500 kbps) makes it even worse :(
I used a friend's megui settings, and while it improved the quality,

I'm still unhappy with the result. I've also come to dislike what I got before encoding, as well ><

I want to make it look better ;-; The noise bothers me a lot, especially after encoding.




try Didee's script. it is really good at getting rid of all noise and keeping original detail and sharpness. I encoded the 300 at 1280 x720p using it and it is better looking and way more watchable than the original source. (I hate grain and dancing grain big time lol)


http://forum.doom9.org/showthread.php?p=1076491#post1076491

[Zep]

IMO, you're bringing out an excessive amount of haloing from all those sharpening steps.

yeah I agree way over doing it. it is better to just clean it up then call 1 good sharpen function. if you can mask it well enough you may not even need to sharpen. (unless the source is not sharp)