Fine anime antialiasing

[Mystery Keeper]

Fast motion scenes are handled fine, but look a little jerky due to unusual sharpness. Actually, nothing special needs to be done for fast motion. Motion compensation is done for slow motion to compensate the flicker.

[Adub]

Hmm...I seem to see a couple of artifacts produced from this script, such as here:
http://i34.photobucket.com/albums/d125/Merlin7777/snapshot20080807082354.jpg


I assume there is a setting I can change to protect against this?

[Mystery Keeper]

If you are sure that it is caused by that function, just disable motion compensation. I'll make it adjustable in the function a bit later.

@Didée, how can we make it apply MC only to if scene changes a little? 'Twould be good if it took a scene change threshold as parameter.

[Adub]

Yep, this was the only function in the filter chain. It's funny, I have seen pictures of artifacts like this before, and there is a way to guard against it in MVTools, but I can't for the life of me recall what it is.

[Didée]

That artefact looks like if the sharpen-difference is merged in without motion compensation from a neighbor frame. From the way how MVAnalyse & MVDegrain work and how the script is layed out, I don't see how such a thing can sneak in at all.

Try truemotion=false,search=3,searchparam=256 :p

[Mystery Keeper]

0_0 WEIRD. I suddenly started getting an error "RemoveGrain: invalid mode 19 (LimitedSharpenFaster.avsi, line 106)"

Fixed

[Mystery Keeper]

Sorry, Merlin. I don't get such artifacts anywhere, so I can't help. Here's the function in which you can disable/enable MC easily.

function SharpAAMC(clip orig, int "aastr", float "ds", int "ShPre", int "ShPost", int "SmPost", bool "MC")
{
aastr=default(aastr,255) #antialiasing strength
ds=default(ds,0.2) #strokes darkening strength
ShPre=default(ShPre,80) #Presharpening
ShPost=default(ShPost,300) #Postsharpening
SmPost=default(SmPost,100) #Postsmoothing
MC=default(MC, true) #Motion Compensation

a=orig
w=width(a)
h=height(a)
m=logic( a.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=3,V=3)
\ ,a.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=3,V=3)
\ ,"max").greyscale.levels(0,0.8,128,0,255,false)
b=a.Toon(ds).Warpsharp(depth=ShPre)
b=b.spline64resize(w*2,h*2).TurnLeft().SangNom(aa=aastr).TurnRight().SangNom(aa=aastr)
b=b.spline64resize(w,h).LimitedSharpenFaster(edgemode=1,strength=ShPost,overshoot=1,soft=SmPost)

b=mt_merge(a,b,m,Y=3,U=3,V=3)
sD=mt_makediff(a,b)
fv2 = a.MVAnalyse(isb=false,delta=2,idx=13)
fv1 = a.MVAnalyse(isb=false,delta=1,idx=13)
bv1 = a.MVAnalyse(isb=true, delta=1,idx=13)
bv2 = a.MVAnalyse(isb=true, delta=2,idx=13)

sDD = sD.MVDegrain2(bv1,fv1,bv2,fv2,idx=14)

reduc = 0.4
sDD = mt_lutxy(sD,sDD,"x 128 - abs y 128 - abs < x y ?").mergeluma(sDD,1.0-reduc)

b=MC ? a.mt_makediff(sDD,U=2,V=2) : b

return b
}

[Sagekilla]

IIRC, I saw somewhere that you could do a pseudo scene-change based on change in luminance, and depending on that it could ramp up a filter to process more if there was fast motion or less if low motion.

[martino]

http://avisynth.org/mediawiki/ConditionalFilter#Advanced_conditional_filtering:_part_III
It's not thought through well. I've got a badly modified version with some fixes, so if you want to use it as a reference or something just ask. Also masktools has some scenechange stuff, as well as mvtools IIRC.

[Adub]

Blast you Didee!!! Blast you!!

It takes nearly a minute to render 1 frame! That is insane!! Hell, I can't see the artifacts when the clip is in motion. It was pure chance that I managed to spot it when stepping through frames.

[Seraphic-]

Hi,

What function is related to "spline64resize"?
I'm getting a no function named error for "spline64resize".
Not sure why as the script was working the other day.

[Adub]

Did you recently uninstall and reinstall avisynth?

It is an internal function, but I don't think it comes in avisynth 2.57. I believe it was implemented in 2.58.

[Seraphic-]

Did you recently uninstall and reinstall avisynth?

It is an internal function, but I don't think it comes in avisynth 2.57. I believe it was implemented in 2.58.

Ah, yes I did. Must have installed the wrong version there. Thanks

Also, is there anyway to something like "threads=4" to take advantage of four CPU cores?

[Adub]

For what? There are filters that can do this, yes, but many don't.

Normally, the way to run filters multithreaded is to use MT. Just run a search and you will find it.

[thetoof]

As requested by PM, here my modded version with a new "MT" parameter. It will be used as aa=4 in the next version of animeivtc and will be included in animeivtc.avsi
I may add other parameters (such as overlap, pel, search...) later.
True = use josey_wells' multithreaded MVTools with the new syntax (default)
False = Old syntax, "official" development is up-to-date, but no MT
Also added tradius parameter if you want to use more frames to degrain. Default is 2 since SharpAAMC() uses MVDegrain2.

function SharpAAMC(clip orig, int "aastr", float "ds", int "ShPre", int "ShPost", int "SmPost", bool "MC", bool "MT", int "tradius")
{
aastr=default(aastr,255) #antialiasing strength
ds=default(ds,0.2) #strokes darkening strength
ShPre=default(ShPre,80) #Presharpening
ShPost=default(ShPost,300) #Postsharpening
SmPost=default(SmPost,100) #Postsmoothing
MC=default(MC, true) #Motion Compensation
MT=default(MT,true) # Use josey_wells' branch of MVTools
Tradius=default(Tradius,2) # 2 = MVDegrain2 / 3 = MVDegrain3 /// 28 = MVDegrain 28

a=orig
w=width(a)
h=height(a)
m=logic( a.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=3,V=3)
\ ,a.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=3,V=3)
\ ,"max").greyscale().levels(0,0.8,128,0,255,false)
b=a.Toon(ds).Warpsharp(depth=ShPre)
b=b.spline64resize(w*2,h*2).TurnLeft().SangNom(aa=aastr).TurnRight().SangNom(aa=aastr)
b=b.spline64resize(w,h).LimitedSharpenFaster(edgemode=1,strength=ShPost,overshoot=1,soft=SmPost)

MaskedMerge(a,b,m,Y=3,U=3,V=3)
sD=mt_makediff(a,last)

fv2 = a.MVAnalyse(isb=false,delta=2,idx=13)
fv1 = a.MVAnalyse(isb=false,delta=1,idx=13)
bv1 = a.MVAnalyse(isb=true, delta=1,idx=13)
bv2 = a.MVAnalyse(isb=true, delta=2,idx=13)
allv = a.MVAnalyseMulti(refframes=tradius,idx=13)

sDD = (MT) ? sD.MVDegrainMulti(allv,idx=14) : sD.MVDegrain2(bv1,fv1,bv2,fv2,idx=14)

reduc = 0.4
sDD = mt_lutxy(sD,sDD,"x 128 - abs y 128 - abs < x y ?").mergeluma(sDD,1.0-reduc)

return MC ? a.mt_makediff(sDD,U=2,V=2) : b
}

[egrimisu]

what aa is aa4 ?

[Mystery Keeper]

sDD = (MT) ? sD.MVDegrainMulti(allv,idx=14) : (MT==false) ? sD.MVDegrain2(bv1,fv1,bv2,fv2,idx=14) : NOP()


o_0 ??? With what condition would that NOP() work?

[thetoof]

haha, you're right... dunno what I was thinking when I wrote that :p *edited the first post*

[Mystery Keeper]

By the way, thetoof. There's NNEDI antialiasing in your script which you say is my suggestion, modified by you. I suggested using EEDI2. What made you change it to NNEDI? NNEDI is more ACCURATE interpolation. 'Tis not as good for antialiasing as EEDI2.

[thetoof]

Woah, that's some bad mistake I made back then... I simply thought "nnedi is more accurate, so it must be better", changed it without testing and took it for granted afterwards.
I just tested it on the image we had in the "Denoise of tv-anime" thread and it's uber obvious that nnedi is ineffective where eedi2 works like wonders.
It will be changed in the next version animeivtc.

[thetoof]

@ MysteryKeeper: you've had a 1/2-pel registration error between the original (a) and antialiased (b) clip, due to EEDI2. Corrected by the blue parts.
May I ask you what is the purpose/theory/explanation behind 2*w+.001,2*h+.001?

edit: I also compared a random original and aaed clip by using the following script to find which values were appropriate (the image did not move in any direction after ediaa)
function ediaa(clip a) {return a.EEDI2(field=1).TurnRight().EEDI2(field=1).TurnLeft().spline36resize(a.width,a.height,-0.5,-0.5,2*a.width+.001,2*a.height+.001)}
interleave(source, source.ediaa())
Note that I used -0.5,-0.5 to make it work and that you recommended 0.5,-0.5Am I missing something?

[Gavino]

function ediaa(clip a) {return a.EEDI2(field=1).TurnRight().EEDI2(field=1).TurnLeft().spline36resize(a.width,a.height,-0.5,-0.5,2*a.width+.001,2*a.height+.001)}
interleave(source, source.ediaa())
Note that I used -0.5,-0.5 to make it work and that you recommended 0.5,-0.5. Am I missing something?
Didée's version did the eedi upsizing in a different order:
b=a.Warpsharp(depth=100).TurnLeft().EEDI2().TurnRight().EEDI2()
Since his horizontal resizing was done on a left-turned version, and yours on a right-turned one, the horizontal shift correction is reversed.

[thetoof]

Oh, you're right. Thanks!

Now about the 2*w+.001,2*h+.001 thingy...

[Gavino]

Now about the 2*w+.001,2*h+.001 thingy...
No, I don't get that bit either.

It may have been intended to force resampling to be done (avoiding the no-op optimisation for same-size resizing), but using a non-zero crop position does that already.

[Didée]

It's for backward compatibility with Avisynth 2.5.6. That version would have the resizer going NOP without the +.001 thingy, indeed.