Anime/Cartoon Noise Removal & Resolution Boost

[mikeytown2]

This Script Pro's

Remove Noise
Sharpen Image
Increase Resolution
Simple


This Script Cons's

Loss Of Minor Details
Slow
Only Tested On Anime/Cartoon



deen("a3d",4,7,7)
FFT3dGPU(sigma=4, precision=1, sharpen=1.3)

lanczos4resize(720,480)
#lanczos4resize(1280,720)
#lanczos4resize(1920,1080)

aWarpSharp(depth=32,blurlevel=4,thresh=0.3,cm=2)
LimitedSharpenFaster(edgemode=1,strength=400)
deen("a2d",3,6,6)


EDIT
Direct Link to Last Post With Code
http://forum.doom9.org/showthread.php?p=1072022#post1072022
/Edit

List of Filters and what they do.
Deen is a set of assembly-optimised denoisers, like various 3d and 2d convolutions.
http://ziquash.chez-alice.fr/

FFT3dGPU is a GPU version of Fizick's FFT3DFilter. The algorithm (Fast Fourier Transform, denoising)
http://bag.hotmail.ru/fft3dfilter/fft3dfilter.dhtml
http://forum.doom9.org/showthread.php?t=89941

lanczos4resize - built in resizer. Used because source material is 432x324
http://avisynth.org/mediawiki/Lanczos4Resize

aWarpSharp - It detects unevenness from luma. It blurs unevenness. It computes the warp value from unevenness, interpolates luma to warp's destination by cubic interpolation.
http://ziquash.chez-alice.fr/

LimitedSharpenFaster - use it like a traditional sharpener, but it produces less artifacts
http://avisynth.org/mediawiki/LimitedSharpen
http://forum.doom9.org/showthread.php?s=&threadid=84196

I am wondering if i can somehow speed this up and retain more detail in the output, while keeping the clarity of the video.

Source -------------- After Filter
http://img263.imageshack.us/img263/903/normaltn8.th.jpg (http://img263.imageshack.us/my.php?image=normaltn8.jpg)http://img263.imageshack.us/img263/6479/filtereddj9.th.jpg (http://img263.imageshack.us/my.php?image=filtereddj9.jpg)

http://img263.imageshack.us/img263/4175/orginal0091tb3.th.jpg (http://img263.imageshack.us/my.php?image=orginal0091tb3.jpg)http://img263.imageshack.us/img263/3012/flitered0091ow7.th.jpg (http://img263.imageshack.us/my.php?image=flitered0091ow7.jpg)

Thanks!

[DarkT]

Oh you did NOT just list X-MEN as ANIME...

>.<

As for speeding things up... I gave up on fft3d entirely due to the speed stuff, some other filters put together usually do the trick... But anyway...

Oh, you also over-sharpen it... And it could use some more filtering...

[Chainmax]

Oh you did NOT just list X-MEN as ANIME...

>.<


An X-Men anime would be AWESOME, have you seen the two anime intros that were made for this series when aired in Japan (they can be found on YouTube)? It's OMFG, funny-feeling-in-the-pants material :).



mikeytown2: is the source's resolution 432x324, 720x480 or ~1300x730?

[Dark Shikari]

You're definitely oversharpening it. It also looks like it could use a real deblocker, like SmoothDeblock or Deblock_QED or similar.

[mikeytown2]

lol, i think just about everyone knows that x-men is not anime (unless you look at this http://members.aol.com/drg4/xanime.html ). i used that title because cartoons and anime are similar in that you can't go out and film it, and it is usually hand drawn.

You are right about me over-sharpening it, but if i kill the sharpeners it looks too blurry imho.

Removing this CodeaWarpSharp(depth=32,blurlevel=4,thresh=0.3,cm=2)
LimitedSharpenFaster(edgemode=1,strength=400)

http://img84.imageshack.us/img84/8196/xfliterednotsharpfm4.th.jpg (http://img84.imageshack.us/my.php?image=xfliterednotsharpfm4.jpg)http://img525.imageshack.us/img525/5246/xfilteredtz8.th.jpg (http://img525.imageshack.us/my.php?image=xfilteredtz8.jpg)

Here is an Anime Screen Capture, to show that this works well for Anime too.

I grabed frame 5544 from this vid
http://www.youtube.com/watch?v=1kkveNsEXqk

To get the video i used
firefox with Video DownloadHelper
https://addons.mozilla.org/en-US/firefox/addon/3006

Then read this thread
http://forum.doom9.org/showthread.php?t=112883

and downloaded the FLV Splitter
http://sourceforge.net/project/showfiles.php?group_id=82303&package_id=183318&release_id=403108

http://img523.imageshack.us/img523/715/orginalfy2.th.jpg (http://img523.imageshack.us/my.php?image=orginalfy2.jpg)http://img153.imageshack.us/img153/5928/filterjm8.th.jpg (http://img153.imageshack.us/my.php?image=filterjm8.jpg)

One thing i am noticing is that the lanczos4resize is giving me edge artifacts... any ideas?

And in terms of adding more filters, you have any recommendations on this? I found this old thread
http://forum.doom9.org/showthread.php?t=31009

Thanks!

EDIT
SmoothDeblock
http://forum.doom9.org/showthread.php?t=111526

Deblock QED
http://avisynth.org/mediawiki/Deblock_QED

Both look interesting, how would they be better then fft3d? (i'm kinda sold on this filter, i love it)

[mikeytown2]

It appears that lanczos4resize could be causing edge problems. I also backed off the sharpeners

here is the new code
deen("a3d",4,7,7)
FFT3dGPU(sigma=4, precision=1, sharpen=0.7)

Spline64Resize(1280,720)

aWarpSharp(depth=12,blurlevel=4,thresh=0.3,cm=2)
LimitedSharpenFaster(edgemode=1,strength=200)
deen("a2d",3,6,6)
It's not as sharp as the orginal one but it still looks fairly good. Decided to switch to a different resizer after reading
http://forum.doom9.org/showthread.php?t=119283
http://www.all-in-one.ee/~dersch/interpolator/interpolator.html
Orginal ----------- Filters w/o sharpening ----- code from first post ----- code from this post
http://img522.imageshack.us/img522/2183/21572118cp6.th.jpg (http://img522.imageshack.us/my.php?image=21572118cp6.jpg)http://img523.imageshack.us/img523/4584/42983115ry4.th.jpg (http://img523.imageshack.us/my.php?image=42983115ry4.jpg)http://img503.imageshack.us/img503/9256/94075181pg3.th.jpg (http://img503.imageshack.us/my.php?image=94075181pg3.jpg)http://img503.imageshack.us/img503/4580/25298002wp4.th.jpg (http://img503.imageshack.us/my.php?image=25298002wp4.jpg)

[Chainmax]

I concur with Dark Shikari in that DeBlock_QED_MT2 would be the best choice for deblocking. FFT3DFilter is a denoiser, which is a completely different thing. One thing though: I've seen FFT3DFilter create some artifacting on large framesizes when denoising luma (don't know about chroma), so be on the lookout for that.

As for denoising, Deen should be ditched for MVDeGrain2 or at least DeGrainMedian.

Regarding resizing, If I recall correctly some posts made in this forum a long time ago, Spline36Resize actually gives better results than Spline64Resize. Also, this 432x324 --> 1280x720 is insane. The farthest I'd go if I were you is to use nnediresize2x to obtain 864x648 and then use Spline36Resize(960,540) if you absolutely need 16:9 AR.

Sharpening should be done after resizing. X-Men plays extremely well with aWarpSharp, so I'd suggest Spline36Resize(960,540).aWarpSharp(depth=16,cm=1).LimitedSharpenFaster(SMode=4,Strength=200) which should be more than enough.

[mikeytown2]

Thanks Chainmax! I'll be playing around with this.

[Chainmax]

Anytime man, it's always good to meet fellow X-Men 90s fans :).

[mikeytown2]

function MVxx(clip s)
{
backward_vec2 = MVAnalyse(s,isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
backward_vec1 = MVAnalyse(s,isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec1 = MVAnalyse(s,isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec2 = MVAnalyse(s,isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
s=s.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=1)
Return s
}

MVxx()
FFT3dGPU(sigma=2, precision=1, sharpen=0.6)

Spline36Resize(1280,720)
aWarpSharp(depth=16,blurlevel=4,thresh=0.2,cm=1)

deen("a3d",4,8,9)
aWarpSharp(depth=4,blurlevel=4,thresh=0.7,cm=1)

Code needs this plugin to work
http://avisynth.org.ru/mvtools/mvtools.html
file is at the bottom of the page
Orginal ------------- New Filter
http://img521.imageshack.us/img521/959/orgrg6.th.jpg (http://img521.imageshack.us/my.php?image=orgrg6.jpg)http://img264.imageshack.us/img264/5397/flicg9.th.jpg (http://img264.imageshack.us/my.php?image=flicg9.jpg)

Speed increase by getting rid of LimitedSharpenFaster; used another aWarpSharp.


Got lazy didn't want to test this, but here is a link for a deblocker
http://avisynth.org/mediawiki/Deblock_QED

NNEDI
http://forum.doom9.org/showthread.php?t=129953
http://web.missouri.edu/~kes25c/
Can't get this working right or its super slow? Either way i could not see a difference.

X-Men 90s: Best of the bunch imho
http://en.wikipedia.org/wiki/Category:X-Men_television_series

[mikeytown2]

function MVxx(clip s)
{
backward_vec2 = MVAnalyse(s,isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
backward_vec1 = MVAnalyse(s,isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec1 = MVAnalyse(s,isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec2 = MVAnalyse(s,isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
s=s.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=1)
Return s
}

MVxx()
FFT3dGPU(sigma=2, precision=1, sharpen=0.2)

Spline36Resize(720,480)
aWarpSharp(depth=16,blurlevel=4,thresh=0.2,cm=1)

deen("a3d",4,8,9)
aWarpSharp(depth=4,blurlevel=4,thresh=0.7,cm=1)
LimitedSharpenFaster(SMode=4,edgemode=1,strength=400)
This one outputs a little sharper then the other one... looks way better then the very first filter.

First Post ------------- This Post
http://img168.imageshack.us/img168/9795/firstpostxj2.th.jpg (http://img168.imageshack.us/my.php?image=firstpostxj2.jpg)http://img522.imageshack.us/img522/7021/newfilterxe0.th.jpg (http://img522.imageshack.us/my.php?image=newfilterxe0.jpg)

EDIT
1280,720 is too big... i just like it for comparing output.
Also I get green on the left/right edges... any idea? MVDegrain2?

EDIT
green on the left/right edges is caused by aWarpSharp(cm=1), change it to cm=2 and problem goes away.

[*.mp4 guy]

Could you post an unprocessed and unresized image aswell, in the future, it would make comparisons more simple by providing a "low anchor". I'm not asking you to go back and add one now, but generally people can give better advice if you provide a completely unprocessed example.

[mikeytown2]

Found 2 Gems!

So my original idea of a "simple" filter is starting to get thrown out the window... I found

FastLineDarken
http://forum.doom9.org/showthread.php?p=616362#post616362
http://manao4.free.fr/MaskTools.htm

DeHalo_alpha
http://forum.doom9.org/showthread.php?p=777956#post777956

Orginal -------- Orginal Spline36 --------- Filter from last post----This Filter
http://img85.imageshack.us/img85/9836/sourcesx6.th.jpg (http://img85.imageshack.us/my.php?image=sourcesx6.jpg)http://img90.imageshack.us/img90/5158/sourcelargezx6.th.jpg (http://img90.imageshack.us/my.php?image=sourcelargezx6.jpg)http://img212.imageshack.us/img212/9601/filterlastiterza5.th.jpg (http://img212.imageshack.us/my.php?image=filterlastiterza5.jpg)http://img524.imageshack.us/img524/4833/filterthisitereq6.th.jpg (http://img524.imageshack.us/my.php?image=filterthisitereq6.jpg)


function MVxx(clip s)
{
backward_vec2 = MVAnalyse(s,isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
backward_vec1 = MVAnalyse(s,isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec1 = MVAnalyse(s,isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec2 = MVAnalyse(s,isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
s=s.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=1)
Return s
}


MVxx()
FFT3dGPU(sigma=2, precision=1, sharpen=0.2)

DeHalo_Alpha()
FastLineDarken(strength=100)

Spline36Resize(720,480)
#aWarpSharp(depth=16,blurlevel=4,thresh=0.2,cm=2)

deen("a3d",4,8,9)
aWarpSharp(depth=4,blurlevel=4,thresh=0.7,cm=2)
LimitedSharpenFaster(SMode=4,edgemode=1,strength=400)


FastLineDarken()
DeHalo_Alpha()

Thanks *.mp4 guy for the Suggestion!

I haven't optimized these functions like i did for the other ones... and i could probably remove one of the sharpeners. I just wanted to post this since it gave me such an improvement over the last filter.

[*.mp4 guy]

You probably won't be able to get much more sharpness, atleast not with a script that will run at more then 1fps.

This is the sharpest (http://img110.imageshack.us/img110/4425/tsetuk2.png) I could manage, using unreasonably slow filters.

[Sagekilla]

@mikeytown2: I tried upscaling to a much higher resolution first where you have your first resize, then downsize at the end of the script and got much better results than simply doing one resize. It was about twice as slow but I did get a noticeable quality boost.

Here's my sample script I used for upscaling a clip of Futurama from 640x352 to 1024x576:

toon(0.15)
deen("a3d",4,7,7)
DeGrainMedian(1,4,1)

AAA(Chroma=true)
Spline36Resize(1536,864)

aWarpSharp(depth=12,blurlevel=4,thresh=0.3,cm=1)
LimitedSharpenFaster(edgemode=1,strength=200)
deen("a2d",3,6,6)
Dehalo_alpha()

Spline36Resize(1024,576)

A bit slow, extremely slow even, but I was able to reduce bitrates by 30% and increase quality on my particular clip from first converting to a higher resolution. That's 30% less relative to doing a simple resize first instead of upscaling to such a large resolution.

[Adub]

Dude, get rid of Deen already. See chainmax's post.

[mikeytown2]

@Merlin7777
I have tried to... i got rid of one of them by using MVDegrain2. But the second one after the resize i can't find a suitable filter to replace that Deen.

MVDegrain2 seems to crash if i use it more then once and Deen gives better results then DeGrainMedian.

DeGrainMedian ---------- Deen ------- original source
http://img337.imageshack.us/img337/5941/degrainmedianai0.th.jpg (http://img337.imageshack.us/my.php?image=degrainmedianai0.jpg)http://img404.imageshack.us/img404/7918/deentk0.th.jpg (http://img404.imageshack.us/my.php?image=deentk0.jpg)http://img146.imageshack.us/img146/4218/orginalsourcetz2.th.jpg (http://img146.imageshack.us/my.php?image=orginalsourcetz2.jpg)

DeGrainMedian:
function MVxx(clip s)
{
backward_vec2 = MVAnalyse(s,isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
backward_vec1 = MVAnalyse(s,isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec1 = MVAnalyse(s,isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec2 = MVAnalyse(s,isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
s=s.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=1)
Return s
}
AddBorders(0,0,0,4)
Deblock()
Crop(0,0,0,-4)

MVxx()
FFT3dGPU(sigma=2, precision=1, sharpen=0.2)
DeHalo_Alpha()
FastLineDarken()

Crop(0,0,-2,-2)
Spline64Resize(720,480)

AddBorders(4, 0, 4, 0)
aWarpSharp(depth=16,blurlevel=4,thresh=0.2,cm=1)
DeGrainMedian(limitY=8,limitUV=15,mode=0)

FastLineDarken(strength=150)
aWarpSharp(depth=8,blurlevel=4,thresh=0.7,cm=1)
Crop(4,0,-4,0)
FastLineDarken()


Deen:
function MVxx(clip s)
{
backward_vec2 = MVAnalyse(s,isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
backward_vec1 = MVAnalyse(s,isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec1 = MVAnalyse(s,isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec2 = MVAnalyse(s,isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
s=s.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=1)
Return s
}
AddBorders(0,0,0,4)
Deblock()
Crop(0,0,0,-4)

MVxx()
FFT3dGPU(sigma=2, precision=1, sharpen=0.2)
DeHalo_Alpha()
FastLineDarken()

Crop(0,0,-2,-2)
Spline64Resize(720,480)

AddBorders(4, 0, 4, 0)
aWarpSharp(depth=16,blurlevel=4,thresh=0.2,cm=1)
deen("a3d",4,8,9)

FastLineDarken(strength=150)
aWarpSharp(depth=8,blurlevel=4,thresh=0.7,cm=1)
Crop(4,0,-4,0)
FastLineDarken()

DeGrainMedian
http://avisynth.org.ru/degrain/degrainmedian.html

Deblock
http://neuron2.net/dgmpgdec/dgmpgdec.html

[Sagekilla]

I seem to get better results from Deen as well, I can't seem to sufficiently get rid of discolorations and noise when I use only Degrainmedian or MVDegrain2. Speed really isn't a concern for me at this point either, it might be slow as hell running at 1 fps (Woohoo, 16 hours to go!) right now but considering the quality boost and relatively small size increase I'm fine with it.

Edit: DeGrainMedian alone is doing a fairly poor job at ridding the artifacts, I'll go try MVDegrain 2 again.

[Chainmax]

If DeGrainMedian isn't cutting it, try dfttest. I dabbled a bit with it and it seems very, very good :). Also, it might be a good idea to add gradfun2db at the very end of the script in order to get rid of the banding that will likely be in a source that needs deblocking.

[mikeytown2]

I Looked into the various denoisers that Chainmax suggested. Here is the new code i got... Its a little bit faster, not as sharp but i think it can retain more detail sometimes. Source is 432x324

AddBorders(0,0,0,4)
Deblock()
Crop(0,0,0,-4)

dfttest()
FastLineDarken()
deen("a3d",4,8,9)

Crop(0,0,-2,-2)
Spline36Resize(720,480)

AddBorders(4, 0, 4, 0)
aWarpSharp(depth=12,blurlevel=4,thresh=0.2,cm=1)
FastLineDarken()
aWarpSharp(depth=6,blurlevel=4,thresh=0.7,cm=1)
Crop(4,0,-4,0)

FastLineDarken()
DeHalo_Alpha()
gradfun2db()

dfttest()
http://forum.doom9.org/showthread.php?p=1071662
Put libfftw3f-3.dll in windows\system32 dir

gradfun2db()
http://forum.doom9.org/showthread.php?t=108681

Source ------- Source Spline36 ---------- Old Filter Last post ------This Filter
http://img472.imageshack.us/img472/2840/sourcepp9.th.jpg (http://img472.imageshack.us/my.php?image=sourcepp9.jpg)http://img451.imageshack.us/img451/7308/sourcespline36ob6.th.jpg (http://img451.imageshack.us/my.php?image=sourcespline36ob6.jpg)http://img465.imageshack.us/img465/1959/oldfiltercp6.th.jpg (http://img465.imageshack.us/my.php?image=oldfiltercp6.jpg)http://img460.imageshack.us/img460/2958/newfilterch7.th.jpg (http://img460.imageshack.us/my.php?image=newfilterch7.jpg)
http://img472.imageshack.us/img472/3959/sourcemo1.th.jpg (http://img472.imageshack.us/my.php?image=sourcemo1.jpg)http://img451.imageshack.us/img451/6406/sourcespline36xo2.th.jpg (http://img451.imageshack.us/my.php?image=sourcespline36xo2.jpg)http://img465.imageshack.us/img465/363/oldfilterse5.th.jpg (http://img465.imageshack.us/my.php?image=oldfilterse5.jpg)http://img460.imageshack.us/img460/3765/newfilteraz5.th.jpg (http://img460.imageshack.us/my.php?image=newfilteraz5.jpg)

[hkazemi]

Readers looking for the latest developments in this area may want to check out these posts in the thread 'Restoring low-quality Youtube/similar videos':

Dark Shikari made a derivation of mikeytown2's script
http://forum.doom9.org/showthread.php?t=134078

SageKilla made a function called Animation Restoration:
http://forum.doom9.org/showthread.php?t=134078#post1091251

[Adub]

You should have your own thread hkazemi. What with all the summarizing you have been doing lately.

[hkazemi]

Thanks, and actually I'm working on a denoiser summary thread, but haven't posted it yet. It takes a lot of time to find existing comparisons and put them all together.

Personally, I'm seeking the video processing holy grail, which means even if I don't find it, I'm going to get as close as reasonably possible with the existing tools. (And even though I know I don't currently have the CPU power to run the 'best' settings (like mcbob deinterlacing), I want to know what they are so that I can ratchet up my settings as my processing capabilities also increase.)

[Adub]

Have you considered editing the Avisynth wiki instead/also? It tends to be a little more finite and more organized.

When you say video processing holy grail, what exactly do you mean? The perfect Source? Or the Filter the makes the imperfect source Perfect?

[hkazemi]

Have you considered editing the Avisynth wiki instead/also? It tends to be a little more finite and more organized.

I hadn't considered that yet because most of the activity is over here on the forums, the search results are over here, and discussion can more easily take place here than on the wiki in case people have more links to comparisons to offer. That doesn't mean I can't/won't add similar content to the wiki...there's so much information to distill that I have to start somewhere because it hasn't been done yet, for so many areas of video processing.

When you say video processing holy grail, what exactly do you mean? The perfect Source? Or the Filter the makes the imperfect source Perfect?

To me, the video processing holy grail is a chain of applications and filters that can automatically handle basically any source and turn it into a clean, decently compressible, in-sync output. My primary use would be for the automated handling of ATSC transport streams recorded over the air, which may have packet glitches/data corruption, AC3 streams with variable bitrate headers, interlacing, telecine, and field order transitions. This chain of apps and filters should be robust to the many kinds of problems seen in ATSC content, some of which I think are more complex than DVD content, because of the nature of the transport mechanism.

To this goal, I'm looking for scriptable tools that can automatically handle the above (AC3 tools, interlace detection tools, etc.).

[Adub]

Well then, good look with your search and let me know if you ever find it! :D

But seriously, if you are going to distill all of the information on this forum (huge undertaking), I think one centralized location to store it all would be a good idea. Doesn't have to be the wiki, but a designated thread/(maybe sticky?).

You have already made a good start with your denoiser list.

[Wilbert]

interlacing, telecine, and field order transitions.
everything about automatic interlace detection:

http://avisynth.org/mediawiki/Interlace_detection

I found a backup, and will restore the links this weekend.

[hkazemi]

everything about automatic interlace detection:

http://avisynth.org/mediawiki/Interlace_detection

I found a backup, and will restore the links this weekend.

I just used the wiki's history feature to revert that page to the March 30, 2006 version which was the last version before the content and formatting got messed up. All the links are restored as of that edition.