Blocky artifacts.

[EroKaos]

What kind of mistake/error am I making to get the artifacts (if you can call it that) that appear on the hair?

Sample: http://www.mediafire.com/?ipb1lnhnzbd

It's not the first time I see it, had it before plenty of time, which is the reason I have never completed anything I have tried to finish. But since I have plenty of time now I decided to finish unfinished stuff now.

Thanks

[Ranguvar]

Please post your script.

Also, can you provide PNG screens of just it filtered? The Xvid compression may cause mosquito noise or blocks.

[EroKaos]

I tried two different ways to do it, this one is megui:


# Set DAR in encoder to 2253 : 1250. The following line is for automatic signalling
global MeGUI_darx = 2253
global MeGUI_dary = 1250
LoadPlugin("G:\E Drive\Apps\Encoding Tools\YATTA 6.118\Plugins\UnFilter.dll")
LoadPlugin("G:\E Drive\Apps\Encoding Tools\YATTA 6.118\Plugins\SangNom.dll")
import("G:\E Drive\Apps\Encoding Tools\YATTA 6.118\Plugins\aaa.avs")
DGDecode_mpeg2source("G:\Trinity Blood\megui\VTS_02_1.d2v",cpu=4,info=3)
ColorMatrix(hints=true,interlaced=true)
tfm(order=-1).tdecimate(hybrid=1)
crop( 4, 0, -4, 0)

Lanczos4Resize(720,480) # Lanczos4 (Sharp)
mergechroma(blur(1.3)) # Little Noise
aaa()


Images from huffy file:
http://img364.imageshack.us/my.php?image=30140487jo6.png
http://img364.imageshack.us/my.php?image=40787143ia4.png
http://img355.imageshack.us/my.php?image=64158179js7.png

Images from avs:
http://img355.imageshack.us/my.php?image=20409032zw6.png
http://img364.imageshack.us/my.php?image=40983771fh4.png
http://img364.imageshack.us/my.php?image=25052430mc5.png

(They are not the same frames.)

[TheRyuu]

It appears as though it's xvid causing the macroblocking and since the source is probably a dvd, I bet it doesn't have that same problem.

I'd recommend to just up the bitrate or something if it's giving you that much trouble. If you're going for a set size, cap the quants at 1 to 5 for all of them, don't use AQ, don't use trellis.

[EroKaos]

It appears as though it's xvid causing the macroblocking and since the source is probably a dvd, I bet it doesn't have that same problem.

I'd recommend to just up the bitrate or something if it's giving you that much trouble. If you're going for a set size, cap the quants at 1 to 5 for all of them, don't use AQ, don't use trellis.

The thing is that it also appears in the huffy files and when I preview it in megui/vdub.

Edit: Found out the reason for the blocking stuff. It's aaa() causing it. So I can just remove it now, but what can I use to replace aaa()? I need to run some sort of anti-alaising.

[themostestultimategenius]

Add gradfun2db() after aaa()

[Ranguvar]

Also, try swapping out mergechroma... UnDot is good for light mosquito noise, LRemoveDust_YV12(17,1) is good for anime (change the 1 to 2 for heavier denoising), DeHalo_alpha may remove some around edges, and FFT3DFilter/FFT3DGPU usually do a good job...

[MadRat]

You've got a little blocking in Able Nightroad's clothing but the girl's hair isn't blocking, that's banding. Normally you see that kind of banding as a denoising artifact but I'm not sure if mergechroma(blur(1.3)) is strong enough to do it. It might be that's just a weak spot to begin with and even a little denoising created the banding.

I agree with themostestultimategenius and Ranguvar and I'd also suggest a "better" anti-aliasing. You'll see a big slow down with these filters but you'll get better quality. Try something like this replace mergechroma and aaa with...

fft3dfilter(sigma=1.0,bt=4,plane=4,bw=32,bh=32,ow=16,oh=16) # denoise
gradfun2db() # deband
LanczosResize(width*2,height*2).TurnLeft().SangNom().TurnRight().SangNom().LanczosResize(width,height) #anti-alias

To make the denoising on FFT3DFilter stronger or weaker adjust the sigma, it's a floating point number so you can change it from 0.1 to 10.0. You could also add some sharpening to make your picture clearer. So for example, fft3dfilter(sigma=2.0,bt=4,plane=4,sharpen=0.5,bw=32,bh=32,ow=16,oh=16) would increase the denoising and add a sharpen of 0.5 out of 0.1 to 1.0.

You can also put a floating point number between the () in gradfun2db to make it stronger or weaker, I think the default is 1.2.

Hopefully if I told you the wrong thing or over simplified too badly someone will come along, shake their finger and correct me. :)

[Ranguvar]

Just did some testing, LRemoveDust_YV12 is actually beating out the slower FFT3DFilter on most of my anime :o

Make sure you try them both.

And MadRat, how is your little function better for anti-aliasing than AAA? (Not challenging you, just wondering)

[TheRyuu]

The thing is that it also appears in the huffy files and when I preview it in megui/vdub.

Edit: Found out the reason for the blocking stuff. It's aaa() causing it. So I can just remove it now, but what can I use to replace aaa()? I need to run some sort of anti-alaising.

Most anime shouldn't even need anti-aliasing, especially if it's a dvd source.

Also, if you're going to use gradfun, make sure to try out gradfunkmirror because it does the same thing just doesn't destroy edges. But one again, if the source is dvd, you shouldn't need that either. Keep it simple. Gradfun would likely cause destruction of detail in dark areas (will "blend" it together sometimes, depends on the raw).

There is a better function then AAA for anti-aliasing. It's slower but it is higher quality. That being said, it still should not be needed for a dvd source.

Better AA function (IMO):
function antialiasing( clip orig, int "th_luma", int "th_croma", string "type", int "order", int "aath", bool "HQedge")
{

# "th_luma" and "th_croma" affect directly the edge detection: higher values = more edges filtered
# set "order = 0" for Top Field First; order = 1 --> Bottom Field First
# aath = anti-aliasing strenght (default should be fine)


th_luma = Default(th_luma, 20)
th_croma = Default(th_croma, 20)
type = Default(type, "sobel")
order = Default(order, 1)
aath = Default(aath, 48)
HQedge = Default(HQedge, false)
ox = orig.width
oy = orig.height
dx = orig.width * 2
dy = orig.height * 2

orig.convertToYV12()
a=last
b=lanczos4Resize(dx, dy).TurnLeft().SangNom(order,aath).TurnRight().SangNom(order,aath)

# native chroma edges:
#c=b.lanczosresize(ox,oy).EdgeMask(0, th_luma, 0, th_croma, type,Y=3,U=3,V=3)

# use luma edges for chroma:
c=b.lanczosresize(ox,oy).EdgeMask(0, th_luma, 0, th_croma, type,Y=3,U=1,V=1).FitY2UV()

d= logic( b.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=4,Y=3,U=1,V=1)
\ ,b.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=4,Y=3,U=1,V=1)
\ ,"max").greyscale.levels(0,0.8,128,0,255,false).FitY2UV().lanczosresize(ox,oy)
b=b.lanczosresize(ox,oy)

HQedge ? MaskedMerge(a,b,d,Y=3,U=3,V=3) : MaskedMerge(a,b,c,Y=3,U=3,V=3)
}

GradFunkMirror:
Function GradFunkMirror( clip c, float "strength" )
{
#counters edge effects for gradfun2db()
#gradfun2db() requires width mod8 and height mod2
strength = default( strength, 1.2 )
w = c.width()
h = c.height()
vflip = c.FlipVertical()
hflip = c.FlipHorizontal()

stackhorizontal( hflip.crop( w-16, 0, 16, h ).addborders( 0, 16, 0, 16 ),
\ stackvertical( vflip.crop( 0, h-16, w, 16 ), c, vflip.crop(0, 0, w, 16 ) ),
\ hflip.crop( 0, 0, 16, h ).addborders( 0, 16, 0, 16 ) )
gradfun2db( strength )
crop( 16, 16, -16, -16 )

Return last
}

[EroKaos]

This is the main reason I use some antialaising:

http://img530.imageshack.us/my.php?image=66980105nj2.png

It's not just this frame, there are a bunch of other frames that have tiny jagged borders all over the video (nothing serious but annoying to see over and over.) If it weren't for this I would'nt use antialaising.

[TheRyuu]

This is the main reason I use some antialaising:

http://img530.imageshack.us/my.php?image=66980105nj2.png

It's not just this frame, there are a bunch of other frames that have tiny jagged borders all over the video (nothing serious but annoying to see over and over.) If it weren't for this I would'nt use antialaising.

Bad field detection maybe with telecine (or tfm) so it's setting it to post process.

[EroKaos]

So as long as I get the field detection correct I shouldn't get that kind of jagged border? I'm using the megui automated avisynth script creator to try it (never used it before), like the *press Analyze, wait 2-3 mins* and poof, script is done, wheter it's completely perfect or flawed, I don't know. Just using it while I remember stuff.

[MadRat]

And MadRat, how is your little function better for anti-aliasing than AAA? (Not challenging you, just wondering)

I spent 4 or 5 hours one morning testing every anti-aliasing function I could find and that one subjectively looked the best and was reasonably fast. There might have been anti-aliasing functions and plug-ins that I missed.

I'm really interested to try the functions wizboy11 mentioned and see how they compare to what I've been using.

[Ranguvar]

At least in one of my tests so far, MadRat's anti-aliasing had the best appearance (may not be best in quality, but it looked like it) over antialiasing() and AAA(). And thanks, wizboy, for intdocuing me to gradfunkmirror, works well, especially when using (foxyshadis' I believe) little AddGrain() mod.