Multithreading LSF and other questions.

[bilditup1]

The past two days I've been diving into avs in order to clean up a DVD I just bought (DBZ Rock the Dragon Edition). It's been fun tweaking this and that, and I'm pretty much amazed at how powerful avs filters can be, but I seem to have hit a snag.
In order to use InterFrame (which I didn't actually use just yet) I switched to AviSynth MT. All seems to be well in my filter-chain, until the very end, when I try to apply sharpening with Didée's LimitedSharpenFaster. Alone, this seems to work, but if I upscale using nnedi3 just before it, I'll get some weird warping and missing pieces of frames (like blips). The same thing happens if I try to apply LSF twice in a row, or add an anti-aliasing script call right after (just to try, doesn't appear to be necessary).
I managed to avoid this problem when using nnedi3_rpow2 and one instance of LSF by changing MTMode to (3,8) just before LSF. But adding another LSF call right after it brings the issue back - not as pronounced as before, but still clearly unacceptable.
So it's clear that whatever LSF is doing, it is not thread-safe, at least not in this script. Manau's MaskTools2 and Kassandro's RemoveGrain are previously used by AnimeIVTC. But that doesn't seem to be the issue (at least I don't think so?) since switching to a single function from TIVTC.dll doesn't seem to help matters. In sum, I don't know what's going on here, or if it's fixable. If it is - great! If not - what should I use for sharpening instead?

My other question (for now) - I get some weird pixelation at the beginning, on the outermost parts of the Eternal Dragon's eyes. Doesn't appear to be in the source. Artifact doesn't happen anywhere else, but I have no idea what to make of it. Can post screens if that would be helpful.
Other notes - I am using Set's 32-bit AviSynthMT 2.6 from this past September. My CPU is a 4770K @ 4.6Ghz, and I've got 32GB of RAM. Most probably I'm denoising a bit too much, so I'll likely pull back on that, but I don't think that has anything to do with my issues. I'd be grateful for any other comments or suggestions though.

For those who can help, here's the latest version of my script:


##
## DBZ_d1_mk7s
##
##
## Set memory size and multithreading mode
##
SetMemoryMax(4096)
SetMTMode(3,8)
##
## Load plugin dependencies for AnimeIVTC2
##
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\dup230\Dup.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\fizick\mvtools2.dll")
LoadCPlugin("C:\blu\MeGUI\tools\avisynth_plugin\fizick\yadif.dll") # CPlugin
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\kassandro\RemoveGrainSSE3.dll") # Also for LSF
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\kassandro\ReduceFlickerSSE3.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\kassandro\RepairSSE3.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\manau\MaskTools.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\manau\MaskTools2.dll") # Also for LSF
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\MarcFD\SangNom.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\mg262\Average.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\set\aWarpSharp.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\set\Toon-v1.1.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tritical\eedi2.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tritical\nnedi2.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tritical\nnedi3.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tritical\TDeint.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tritical\TIVTC.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tritical\TMM.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tritical\yadifmod.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tsp\medianblur.dll")
##
## Load source plugins
##
LoadPlugin("C:\blu\MeGUI\tools\dgindex\DGDecode.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tritical\ColorMatrix.dll")
##
## Load denoising/degraining plugins
##
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\DeGrainMedian.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\FFT3DFilter.dll")
LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\VagueDenoiser.dll")
##
## Load Anti-aliasing plugins
##
#LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tp7\SangNom2.dll")
#LoadPlugin("C:\blu\MeGUI\tools\avisynth_plugin\tp7\fturn.dll")
##
## Load script dependencies for AnimeIVTC2
##
Import("C:\blu\MeGUI\tools\avisynth_plugin\Lato\LSFmod.v1.9.avsi")
Import("C:\blu\MeGUI\tools\avisynth_plugin\MOmonster\srestore.avs")
Import("C:\blu\MeGUI\tools\avisynth_plugin\tritical\MCBob.avsi")
Import("C:\blu\MeGUI\tools\avisynth_plugin\tritical\Vinverse.avsi")
Import("C:\blu\MeGUI\tools\avisynth_plugin\tritical\VinverseD.avsi")
Import("C:\blu\MeGUI\tools\avisynth_plugin\tritical\VinverseD.avsi")
Import("C:\blu\MeGUI\tools\avisynth_plugin\thetoof\TGMCmod.avsi")
##
## Load scripts
##
Import("C:\blu\MeGUI\tools\avisynth_plugin\thetoof\AnimeIVTC2.avsi")
Import("C:\blu\MeGUI\tools\avisynth_plugin\LimitedSharpenFaster.avsi")
#Import("C:\blu\MeGUI\tools\avisynth_plugin\line0\maa2_mod.avsi")
##
## Load video
##
SetMTMode(5)
DGDecode_mpeg2source("I:\rip\DBZ\D1\video.d2v", cpu=4, info=3) # Load video
ColorMatrix(hints=true, interlaced=true, threads=8) # Fix colors
##
## IVTC
##
AnimeIVTC(mode=4, omode=1)
##
## Denoising/degraining
##
SetMTMode(2,8)
DeGrainMedian(limitY=2,limitUV=3,mode=1)
DeGrainMedian(limitY=2,limitUV=3,mode=1)
FFT3DFilter(sigma=3.0, bt=5, plane=4, bw=64, bh=64, ow=32, oh=32, wintype=1, ncpu=8)
VagueDenoiser(threshold=0,method=3, nsteps=5, chromaT=0)
##
## Resizing
##
nnedi3_rpow2(rfactor=2,cshift="spline64resize",fwidth=1280,fheight=936,nsize=0,nns=3,qual=2)
##
## Sharpening
##
SetMTMode(3,8) # Or else awful things happen
LimitedSharpenFaster(strength=255, ss_x=2, ss_y=2)
LimitedSharpenFaster() # awful things happen anyway with this line
##
## Anti-aliasing
##
#maa2(threads=8)
##
## END OF FILE
##


Thanks a lot!

[thecoreyburton]

Hey bilditup1!

I know it's two years later, but I remember seeing your thread back when it was posted because I bought this box-set too. I was pretty disappointed with the colors and overall presentation at first, but the release itself was definitely a good thing to experiment with. I wrote a script based off the one you posted here as some of my first AVISynth work and now two years later I'm finally completely satisfied with the results of my script. As a thank you, I thought I'd return the unintentional favor you did for me by sharing my script with you. I think looks fantastic, especially considering what the source is like - but problem is that it's too slow for real-time use (I get about 3fps tops). That's not an issue for me however, as I always encode the files. If interframe is still your goal, you could do this and then process them a second time with interframe (or even use the SVP for playback). It's not ideal, but it'll work.

Here (http://www.mediafire.com/file/t61p565y535cbrp/dbz_rockthedragon.zip) is a zip file containing four mkv files encoded with my script at CRF19 - so you can see the result before sitting through a slow-rendering script.

#Import, process the source and correct the colors as necessary.
DGDecode_MPEG2Source("dbz.d2v", cpu=6, info=3)
ColorMatrix(hints=true, interlaced=true, threads=8)
#IVTC and decimation with parameters ensuring no residual combing is left.
TFM(pp=2,clip2=last.qtgmc(preset="very slow",fpsdivisor=2),mi=30)
TDecimate()
#Correct the levels.
ColorYUV(Levels="TV->PC")
#Resize the source to 4:3 without losing any width
Spline36Resize(720,540)
#Heavy motion-compensated denoising with additional sharpening and deringing to combat artifacts introduced in certain scenes.
MCTemporalDenoise(Settings="Very High",Enhance=True)
LimitedSharpenFaster(Strength=10)
HQDeRing()
#Line thinning and darkening with additional anti-aliasing
AWarpSharp(depth=20.0, blurlevel=1)
FastLineDarkenMod(thinning=0)
FastLineDarkenMod()
Santiag()
#Chroma bleed removal
MergeChroma(AWarpSharp2(Depth=25))
#Deringing and Debanding for final output
HQDeRing()
GradFun2DBMod()

As you can see, it's a bit all over the place. I commented it on the off-chance you wanted to remove a certain filter or collection of filters to make the script more manageable. Most of the lag comes from the denoiser but in this case it's incredibly effective. I'm considering adding a second levels adjustment to follow the ColorYUV line too for some of the Namek episodes and also for the Tree of Might movie, as there still seems to be some discrepancy. I haven't had a chance to check or test it thoroughly though and wouldn't want to accidentally filter out some of the darker colors. The current script looks good enough anyway.

Anyway, whilst it didn't answer your question at all, I hope this script is of some use to you or fellow box-set owners in the same way your script helped me two years ago.

Thanks again, bilditup1!

[FranceBB]

@thecoreyburton... you know what's sad? That after 2 years (actually, more than 2 considering when the filter has been written) we still don't have a multithreading version of LSFmod, which still is one of the best sharpening filters available for avisynth.

[bxyhxyh]

I see he uses old RemoveGrain and other older versions of filters.

Correct if I'm wrong, Aren't new masktools and rgtools both multithreaded?
Only filter LSF requires that isn't internally multithreaded is warpsharp.

[Groucho2004]

Correct if I'm wrong, Aren't new masktools and rgtools both multithreaded?No. Anyway, multithreading this with AVS+ is easy and quite stable.

[bilditup1]

I never came back to this project, as it happens, but decided to rerip the DVDs yesterday and try again. I still have all my old scripts but will probably start from scratch. (...there's too much going on there.) Thanks everyone for chiming in.