Is there a filter that helps with this?

[LordTrace]

I am trying to encode my TMNT S09 DVD into 140MB mp4s (x264/80Kbps AAC).

I don't know if there is a name for this problem or not. I think it is aliasing.. but it's not happening on every frame. One frame will be really badly aliasing and the next 2 or 3 frames will be fine. Even if the image didn't change. In motion it looks like "shimmering?" lol.

This is what i have so far. It doesn't fix the problem. Just thins the lines out. Its probably good enough. I am just wondering if there is anything else i could try. (I tried a few AA filters but they just seemed to wash everything out to much for me)

http://screenshotcomparison.com/comparison/120519


# Set DAR in encoder to 4 : 3. The following line is for automatic signalling
global MeGUI_darx = 4
global MeGUI_dary = 3
LoadPlugin("D:\Programs\MeGUI\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("E:\Ripping\VTS_01_1.d2v", info=3)
LoadPlugin("D:\Programs\MeGUI\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, threads=0)
# Dot Crawl
LoadPlugin("D:\Programs\MeGUI\tools\avisynth_plugin\checkmate.dll")
Checkmate()
# IVTC
LoadPlugin("D:\Programs\MeGUI\tools\avisynth_plugin\TIVTC.dll")
tfm(order=1).tdecimate(hybrid=1)
# Denoise1
LoadPlugin("D:\Programs\MeGUI\tools\avisynth_plugin\UnDot.dll")
Undot() # Minimal Noise
# Denoise2
LoadPlugin("D:\Programs\MeGUI\tools\avisynth_plugin\VagueDenoiser.dll")
VagueDenoiser(threshold=2, method=3, nsteps=6, chromaT=2.0)
# Sharpen
LoadPlugin("D:\Programs\MeGUI\tools\avisynth_plugin\aWarpSharp.dll")
aWarpSharp2()


Sample of source
http://www.sendspace.com/file/7yy8bv 8MB

EDIT: Screens showing what i mean by only some frames..

http://screenshotcomparison.com/comparison/120575

1st is an aliasing frame. 2nd is the very next frame (not aliasing)

[StainlessS]

By no means any kind of expert on this, but you seem impatient (lurking for an answer),
can I suggest you look at nnedi3_rpow2, in nnedi3.

EDIT: Even if I'm wrong (& I often am, I'm guessing that there will be a multitude of
guys just dying to correct me, so you should get a solution). EDIT: Only viewed the pic.

[TheProfileth]

Ugh warpsharp, don't use warpsharp
but also you could try using daa or daa3 after your ivtc that may help

[LordTrace]

Whats wrong with aWarpSharp2 in your opinion? I think it looks ok here. All the other sharpeners i tried on these episodes didn't. It's also the only one that helped the problem instead of making it worse. I am also trying to only use filters that help compression (not hurt it). As stated above im going for a pretty low bit rate. You might be right though, If i can fix the problem before sharpening. Care to suggest a better sharpener?

Ill check out nnedi3 and DAA if i get some time tonight

[lansing]

Whats wrong with aWarpSharp2 in your opinion?

because it thin lines, thus modified the image too much. From my experience, it can be very useful as to create a mask, but not preferred to use as a standalone sharpener

[LordTrace]

because it thin lines, thus modified the image too much.

Yea that's why i didn't want to use it. I tied a few others and they made the problem at least twice as bad. Then i thought thining the lines would help the weird aliasing. and it does make it a lot better *shrugs*

Is it normal for American toons to have this? It seems weird that a line would be perfectly fine in one frame but in the next frame its aliasing like crazy.

[lansing]

you probably may want to upload another sample clip, the one you posted doesn't have the aliasing you talked about

[LordTrace]

It is there. just hard to notice (looks worse in motion and in other parts of the episodes). It was the only clip i had at the time i posted (didn't have time to make another)

http://i.imgur.com/dPYM6.jpg

*hint" bald doods head

I will make another clip when i get home.

EDIT: My wife was able to upload these pics i had on my hdd. I was pretty amazed! haha. She doesn't want to try cutting a vob.

http://screenshotcomparison.com/comparison/120575

1st is an aliasing frame. 2nd is the very next frame (not aliasing)

[LordTrace]

Here's another sample. It's most noticeable in Leonardo's headband in the last couple seconds of the clip.

http://www.sendspace.com/file/3ax967 20MB

[LordTrace]

I tried a few of the simple AA scripts/filters (i am just learning Avisynth) based on SangNom. Also tried TIsophote. They all wash the crap out of the picture while barely doing anything to help with the problem. :( heh

[lansing]

well it's not my post in here (http://forum.doom9.org/showthread.php?p=1570547#post1570547) was intended for, but it works on your problem also.


checkmate() # remove dotcrawl

o=last

pref = o.FFT3dgpu(sigma=5)

emask = o.mt_edge("min/max").RemoveGrain(19)
mt_merge(o, pref, emask, chroma="copy second")

lsfmod() #sharpen


change the sigma value to control the strength, i tried 8, but it took out some lines in darker area, so i would stick to something less than that.
A down side is that you might see some ghosting, i saw one on frame 255.

[LordTrace]

id try it if i knew how to load it lol. I tried adding it to my script and importing it as an .avs. I get..

no function named FFT3dgpu ..

Even after finding FFT3dgpu and adding it (Along with FFTW3.dll) to my avisynth plugin folder.

hehe

[lansing]

id try it if i knew how to load it lol. I tried adding it to my script and importing it as an .avs. I get..

no function named FFT3dgpu ..

Even after finding FFT3dgpu and adding it (Along with FFTW3.dll) to my avisynth plugin folder.

hehe

this one requires you to have a nvida or ati video card, you can use the original fft3dfilter if you don't have those gpu.

what error message are you getting btw?

[StainlessS]

FFTW3.dll goes in System32 or whatever the equivalent is on W7 64 bit.
EDIT: Maybe SysWow64 or something like that (the one for 32 bit on 64, think it seems back to front from the obvious).
EDIT: Think the Wow64 stands for "Windows on Windows 64" ie Win32 on Win64.
Also note, some other plugs require the same dll in same place but named as libfftw3f-3.dll.

[mandarinka]

By no means any kind of expert on this, but you seem impatient (lurking for an answer),
can I suggest you look at nnedi3_rpow2, in nnedi3.

EDIT: Even if I'm wrong (& I often am, I'm guessing that there will be a multitude of
guys just dying to correct me, so you should get a solution). EDIT: Only viewed the pic.

I don't think this is a case for nnedi3rpow2 - the problem is that because it interpolates from the whole frame, its anti-aliasing effect won't be strong enough.

In this case, it will be better to use daa3 (like the regular daa, but using nnedi3). The reason is that daa instead interpolates both fields separately and merges the result, achieving much more powerful antialiasing (if the aliasing artifacts were introduced by interlacing, like in this case).

If you still want to warpsharp, do it afterwards, but yeah, warpsharping is an evil thing.

As for denoising (instead of undot), you could get some success by running mdegrain2 (os smdegrain for ease of use) on edges (use for example tedgemask() for masking), after daa3 is run. That could temporally stabilise the image against shimmering and further suppress the aliasing.

(Imho, nnedi3rpow2 is usable as AA in the case of pixelization, but not in interlacing-caused snafu like this.)

[LordTrace]

this one requires you to have a nvida or ati video card, you can use the original fft3dfilter if you don't have those gpu.

what error message are you getting btw?

I have an ATI HD. Error msg is no function named "FFT3dgpu"

FFTW3.dll goes in System32 or whatever the equivalent is on W7 64 bit.
EDIT: Maybe SysWow64 or something like that (the one for 32 bit on 64, think it seems back to front from the obvious).
EDIT: Think the Wow64 stands for "Windows on Windows 64" ie Win32 on Win64.
Also note, some other plugs require the same dll in same place but named as libfftw3f-3.dll.

Didn't work it any of those places :(

I don't think this is a case for nnedi3rpow2 - the problem is that because it interpolates from the whole frame, its anti-aliasing effect won't be strong enough.

In this case, it will be better to use daa3 (like the regular daa, but using nnedi3). The reason is that daa instead interpolates both fields separately and merges the result, achieving much more powerful antialiasing (if the aliasing artifacts were introduced by interlacing, like in this case).

If you still want to warpsharp, do it afterwards, but yeah, warpsharping is an evil thing.

As for denoising (instead of undot), you could get some success by running mdegrain2 (os smdegrain for ease of use) on edges (use for example tedgemask() for masking), after daa3 is run. That could temporally stabilise the image against shimmering and further suppress the aliasing.

(Imho, nnedi3rpow2 is usable as AA in the case of pixelization, but not in interlacing-caused snafu like this.)

I can't seem to get DAA to work. no function named "repair". Repair.dll is there.

Are we sure these are interlacing artifacts? When i go frame by frame in vdub (d2v) you can clearly see 2 interlaced frames and 3 progressive frames. It's happening in the progressive frames

Edit: NM i got it to work. Used DAA() (lol) Cleans it up quite a bit (not perfect but WAY better then anything else i have tried). Thank you!
http://screenshotcomparison.com/comparison/120880

[StainlessS]

I dont use the gpu version but, think you need "fft3dgpu.hlsl" in same dir as the plugin. (EDIT: Avisynth/plugins)
EDIT: The hlsl file looks like a source file, so the dll must have some kind of compiler/interpreter in it.

EDIT: Repair comes from the RemoveGrain plug suite.

[Didée]

In a nutshell, the source is a poor mess.

During lunchbreak I played the "how to kick this" game, and surprise: the script is quite similar to the guidelines mandarinka layed out. Basically IVTC, then a slightly modified daa(3), then MDegrain2 with MinBlur as prefilter.
(And, in imitation of one StartTrek TNG quote: "We never use WarpSharp when we have sand in our boots.") :D

Result:
(Sorry for the splitting, but the company's firewall allows only very small web uploads)

<part1> (http://www.fastshare.org/download/Turtles.part1.rar) <part2> (http://www.fastshare.org/download/Turtles.part2.rar) <part3> (http://www.fastshare.org/download/Turtles.part3.rar) <part4> (http://www.fastshare.org/download/Turtles.part4.rar) <part5> (http://www.fastshare.org/download/Turtles.part5.rar) (5MB each)
_____

edit - quickly, two shots before leaving the building:

http://thumbnails19.imagebam.com/18648/8b991c186479662.jpg (http://www.imagebam.com/image/8b991c186479662) http://thumbnails61.imagebam.com/18648/6819e8186479680.jpg (http://www.imagebam.com/image/6819e8186479680)

[LordTrace]

In a nutshell, the source is a poor mess.

During lunchbreak I played the "how to kick this" game, and surprise: the script is quite similar to the guidelines mandarinka layed out. Basically IVTC, then a slightly modified daa(3), then MDegrain2 with MinBlur as prefilter.
(And, in imitation of one StartTrek TNG quote: "We never use WarpSharp when we have sand in our boots.") :D

Result:
(Sorry for the splitting, but the company's firewall allows only very small web uploads)

<part1> (http://www.fastshare.org/download/Turtles.part1.rar) <part2> (http://www.fastshare.org/download/Turtles.part2.rar) <part3> (http://www.fastshare.org/download/Turtles.part3.rar) <part4> (http://www.fastshare.org/download/Turtles.part4.rar) <part5> (http://www.fastshare.org/download/Turtles.part5.rar) (5MB each)

That looks great. Can i see your script? :P

[Didée]

When it's been revisited it @ home. I didn't "really" look at the result yet ...

[LordTrace]

Cool. Can't wait to have a look.

One thing i have noticed after adding DAA and DeGrainMedian to my script. If i load it into AvsP with one tab no filters (just IVTC) and another tab with filters. The frames aren't always the same when i go back and forth to see the difference its making. If i take DAA and DeGrainMedian out the match up again. Kinda weird.

EDIT: Wow.. I used this...

import("C:\Program Files (x86)\AviSynth 2.5\plugins\DAA3.avsi")
daa3()

super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)

.. and nothing else. CRF 21 = 93 MB (Video) for episode 1 !!!

EDIT: hmm, seems those settings for MDegrain2 may have been a bit to powerfull .. some places with really thin lines now have almost no lines there anymore hehe

[Didée]

Yep, that's basically it.

The script for my blind shot looked like this:

mpeg2source("D:\__DL\VTS_05_1 (3).d2v") +\
mpeg2source("D:\__DL\vobby_12.d2v")

tfm().tdecimate()
o=last ox=o.width() oy=o.height()

o.nnedi3(field=-2,nsize=6,nns=2)

AA = merge(selecteven(),selectodd())
D1 = mt_makediff(o,AA)
D2 = mt_makediff(AA,AA.mt_edge("06 25 06 12 30 12 06 25 06 128",thy1=0,thy2=255,U=2,V=2).removegrain(4)).mt_lut("x 128 - 1.49 * x 128 - 2 ^ x 128 - 2 ^ 9 + / * 128 +")
AA.mt_adddiff(D2.repair(D1,13*1,-1).mt_lutxy(D2,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",Y=3),U=2,V=2)
spat=last

pre=AA.minblur(0).minblur(2)
sup1=pre.msuper()
sup2=spat.msuper(levels=1)

bv2=sup1.manalyse(isb=true, delta=2,truemotion=false,global=true,blksize=16,overlap=8,search=5,searchparam=6,dct=5)
bv1=sup1.manalyse(isb=true, delta=1,truemotion=false,global=true,blksize=16,overlap=8,search=5,searchparam=6,dct=5)
fv1=sup1.manalyse(isb=false,delta=1,truemotion=false,global=true,blksize=16,overlap=8,search=5,searchparam=6,dct=5)
fv2=sup1.manalyse(isb=false,delta=2,truemotion=false,global=true,blksize=16,overlap=8,search=5,searchparam=6,dct=5)

spat.mdegrain2(sup2,bv1,fv1,bv2,fv2)

up2=spline36resize(ox*2,oy*2)
wd=mt_makediff(up2,up2.awarpsharp2(depth=4,thresh=200,blur=2).awarpsharp2(depth=3,thresh=200,blur=2)).bicubicresize(ox,oy,-.4,.2)
last.mt_makediff(wd,U=2,V=2)
sbr().merge(last,0.51)
mt_lutxy(removegrain(4),"x x y - abs 1.49 / 1 1.25 / ^ x y - x y - abs 0.001 + / * +",U=2,V=2)

stackvertical(o.subtitle("TFM().TDecimate() ...",size=14),last.subtitle("... postprocessed.",size=14))
return(last)



#=====# Helper Functions #=====#

function sbr(clip o)
{
rg11=o.removegrain(11)
rg11D=mt_makediff(o,rg11)
rg11DD=mt_makediff(rg11D,rg11D.removegrain(11)).mt_lutxy(rg11D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?")
o.mt_makediff(rg11DD,U=2,V=2)
}

function MinBlur(clip clp, int r, int "uv")
{
uv = default(uv,3)
uv2 = (uv==2) ? 1 : uv
rg4 = (uv==3) ? 4 : -1
rg11 = (uv==3) ? 11 : -1
rg20 = (uv==3) ? 20 : -1
medf = (uv==3) ? 1 : -200

RG11D = (r==0) ? mt_makediff(clp,clp.sbr(),U=uv2,V=uv2)
\ : (r==1) ? mt_makediff(clp,clp.removegrain(11,rg11),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20).removegrain(20,rg20),U=uv2,V=uv2)
RG4D = (r<=1) ? mt_makediff(clp,clp.removegrain(4,rg4),U=uv2,V=uv2)
\ : (r==2) ? mt_makediff(clp,clp.repair(clp.medianblur(2,2*medf,2*medf),1),U=uv2,V=uv2)
\ : mt_makediff(clp,clp.repair(clp.repair(clp.medianblur(3,3*medf,3*medf),1,0),1,0),U=uv2,V=uv2)
DD = mt_lutxy(RG11D,RG4D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",U=uv2,V=uv2)
clp.mt_makediff(DD,U=uv,V=uv)
return(last)
}

... almost identical. ;)

[LordTrace]

LOL! Mine doesn't look anything like that! I only understand about %7 of whats going on in yours hehe :\

[mandarinka]

Aw damn, I wish I could tweak daa() like that ;_;

[LordTrace]

is there any way to make MDegrain2 less "aggressive" and preserve thin lines better?

[mandarinka]

Reduce thSAD a lot (I'd roughly guess 150-250 as a start).

[Didée]

Sue, but it's a double-edged sword. When you reduce thSAD sufficiently so that thin lines aren't affected, then most probably you also lose the shimmer-reduction along with. For MVTools it doesn't make much of a difference if SAD is created because motion search "missed" a line, or if SAD is created because the same line "looks very different" in adjacent frames. SAD is SAD, difference is difference.

[LordTrace]

I took a look at the encode vs the source, looking for thin lines that where made to thin, and didn't find many. I then looked at those parts in motion. Pretty hard to notice the difference. So i decided to just keep the default settings i posted above.

Insane how compressible it makes this source. CRF20 is around 105 MB (Video only) per episode. Which is well under my goal.