Anime Cleaning, Novice style

[Pyrii]

Eh, first time poster. Yes, in the encoding world I'm a n00b, I dove straight in at the deep end and crashed myself in the ways of film (mainly anime) encoding.

So getting to the point, I've been being something over the past few days, after having telecide drive me barmy, I've yatta'd the whole thing and gone through it frame by frame, with that all done it's just touching up and cleaning, but I'm a perfectionist c***. I personally feel I could make the source cleaner. Anyways, I'll give a before and after of what I've done so far. Please be kind ^.^

Unfiltered:
http://pyrii.pfak.org/crap/unfiltered.png

Avisynth script:
dgdecode_Mpeg2Source("C:\Documents and Settings\Administrator\Desktop\azu1\azu1.d2v")

FieldHint(ovr="C:\Documents and Settings\Administrator\Desktop\azu1\azu1.d2v.tel.txt")
KernelDeint(order=1,sharp=false,twoway=true,threshold=7)




Filtered:
http://pyrii.pfak.org/crap/filtered.png

Avisynth script:
LoadPlugin("C:\Documents and Settings\Administrator\Desktop\plugins\kerneldeint.dll")
LoadPlugin("C:\Documents and Settings\Administrator\Desktop\plugins\fieldhint.dll")
LoadPlugin("C:\Documents and Settings\Administrator\Desktop\plugins\dgdecode.dll")
#LoadPlugin("C:\Documents and Settings\Administrator\Desktop\plugins\decomb521.dll")
LoadPlugin("C:\Documents and Settings\Administrator\Desktop\plugins\Msmooth.dll")
LoadPlugin("C:\Documents and Settings\Administrator\Desktop\plugins\MSharpen.dll")
LoadPlugin("C:\Documents and Settings\Administrator\Desktop\plugins\RemoveGrainSSE2.dll")

dgdecode_Mpeg2Source("C:\Documents and Settings\Administrator\Desktop\azu1\azu1.d2v")

FieldHint(ovr="C:\Documents and Settings\Administrator\Desktop\azu1\azu1.d2v.tel.txt")
KernelDeint(order=1,sharp=false,twoway=true,threshold=7)

#Decimate(cycle=5,quality=3,ovr="C:\Documents and settings\Administrator\Desktop\azu1\azu1.d2v.dec.txt")

RemoveGrain(mode=1)

MSmooth(show=false,mask=false,highq=true,threshold=4,strength=1)
Msharpen(threshold=4,strength=15,highq=true)




Eh, if you can improve, please do so, and tell me why you're changing what you're changing, helps me to learn. Well an encouraging point is, the screenshot sizes (I used PNG, using a lossy format would be pointless >:O), is this:

http://pyrii.pfak.org/crap/filterscreen.jpg

Eh, I dunno if this may be the wrong forum or not, but I felt this may be the best place to ask.

[Pyrii]

Eh, also, msmooth seems to make edges have a weird yellow glow, it's not that bad on strength 1, but it gets worse the higher the strength is.

Unless I'm using it wrong that is.

EDIT: Just comparing the 2 screens, everything seems to have a yellow tinge, edges being slightly worse, maybe I should stop using msmooth o.o

[madoka]

Just throwing in my 2 cents.

The screenshot is from Azumanga Daioh, right? Most of the scenes are static, with flat shaded backgrounds and characters. It's a recent series, which means the productions is most likely done on computers, so you won't be seeing jitters, dirt, color / brightness shifts, etc. Barring a bad DVD transfer (I think ADV did a good job, except for the mini-movie on the last disc), what you start with is already very clean.

IMHO there's no need to do any filtering on this kind of material. Sure, there are some MPEG2 artifacts, and the edges may not be as sharp as one wants. However, most sharpen / smoother filters are very slow, so you're spending a lot of time for only marginal improvements.

(As a side note, MSharpen / MSmooth compute a mask to determine which part of the image to sharpen/smooth. Essentially the same mask can be reused, but as it is right now, it's being computed twice. I've asked neuron2 if he would release a filter that combines the functions of MSharpen / MSmooth together, but I guess his priorities lies elsewhere...:()

Finally, you are eventually going to recompress to DivX / XviD / DVD, right? The compressed output is going to have varying degrees of artifacts. My personal, unscientific, totally subjective experience is that it almost always wipes out whatever pre-processing improvements you have, because they're so small to begin with. Moreover, it takes 3x+ as long to recompress.

But you mentioned that you're a perfectionist. Well, I'm not. :) So I guess it's best if you try compressing a small clip with and without the filters, and see if you see enough of an improvement to expend the processing time required.

For older works (stuff from the 80's to mid 90's), some heavy filtering may be necessary, but for many recent series it's just overkill.

[Guest]

MSmooth in YV12 has that problem with the color casting as the strength increases. I keep meaning to fix that.

[Mug Funky]

azumanga is extremely clean. no noise at all (unless it's intentional)... good sharp lines too.

no point filtering. IVTC shouldn't be a big problem either (beware: a lot of the animation, esp. in the intro is in 30p rather than telecine. you might be better off encoding at 30p rather than decimating).

IMHO, even if you encode full D1, interlaced Xvid and don't touch the video (just have the mpeg2source line), you'll get a good encode on 1 CD.

the PAL version is another matter :(. ADV films did their own conversion, which looks to have too low a threshold, so static details wobble up and down slightly all through it.

For older works (stuff from the 80's to mid 90's), some heavy filtering may be necessary, but for many recent series it's just overkill.

sadly, i've seen analog noise on a lot of new stuff. bog knows where it came from. my guess is a transfer through s-video. component wouldn't cause what i've been seeing - it's chroma crosstalk, but it doesn't affect the luma. Saiyuki is a good example of this (ADV do this one too).

[Pyrii]

Eh, it's a shame people don't have my eyes, so I thought, to get my point across, I'd oversaturate the pictures and then you could see what I mean, obviously, as neuron said, msmooth has a colouring problem, which I think is what is causing my main headache... blocks. But first:

Unfiltered (Oversaturated) (http://pyrii.pfak.org/crap/unfiltered2.png)

Filtered (Oversaturated) (http://pyrii.pfak.org/crap/filtered2.png)

I did them using the PNGs in an image editor.

But where I noticed it first was the scene where yukari steps into the wrong class and turns white from shock, the back of the boys blazer is supposed to be nice and flat, but as you can see from these pics, not the case.

Unfiltered (Oversaturated) (http://pyrii.pfak.org/crap/newunfiltered.png)

Filtered (But no Removegrain() (Oversaturated) (http://pyrii.pfak.org/crap/newfilterednongrain.png)
z0mg, the blocks, probably caused by the msmooth problem. as you can see, everything looks alot more... yellow. This setup is what first made the alarm bells ring.

Filtered (Oversaturated) (http://pyrii.pfak.org/crap/newfiltered.png)
Much less blocky, but still...


---------
Originally posted by Mug Funky
sadly, i've seen analog noise on a lot of new stuff. bog knows where it came from. my guess is a transfer through s-video. component wouldn't cause what i've been seeing - it's chroma crosstalk, but it doesn't affect the luma. Saiyuki is a good example of this (ADV do this one too).

I've seen some rips by A-T where this problem occurs (if you're talking about what I think you're talking about) and you're right, it looks like a single channel svideo passthrough. I had the same problem with my Xbox, I thought it was a TV problem because I never had the problem on other TVs, but an upgrade to a full composite cable and all was fixed. Though fixing this effect on DVD, I haven't had suh a pleasure yet.

Originally posted by Chiiiiiiiiii
the PAL version is another matter . ADV films did their own conversion, which looks to have too low a threshold, so static details wobble up and down slightly all through it.

After the appaulling copy of OMG I got in PAL, I decided to get azumanga in NTSC form. And boy am I glad I did.

Originally posted by Hideki?
no point filtering. IVTC shouldn't be a big problem either (beware: a lot of the animation, esp. in the intro is in 30p rather than telecine. you might be better off encoding at 30p rather than decimating).

I am keeping it 30fps, which is annoying as that 20% extra room would be nice. The IVTC was fine, bar scenes where you have small mouths moving, I had to swap fields for those frames, and bad scene changes. And one part of the title sequence had interlacing through it. So that had to be completely postprocessed.

[Guest]

The MSmooth issue does not cause the blocking; that is there in the source VOB (it's MPEG2!). MSmooth is just smoothing smaller source blocks into larger ones. You can avoid the color cast until I fix it by running it in YUY2 space.

[Pyrii]

I got an error message saying msmooth only supports rgb32 and yv12, so I converted to yv12 and I still get the "colour cast"

[Guest]

Oops, I meant use it in RGB.

[Pyrii]

Done, but there's a massive performance hit in doing so. Problem is, the output seems blocky, it's not grainy, but it's not flat either. And correct me if I'm wrong, the encoder doesn't care if the change is subtle, it will still take it into account, x.x

[Guest]

Where are you seeing this blockiness? After filtering, or after re-encoding? What is your current script and how are you encoding? Your screenshots don't look blocky, so either you are amplifying the low-intensity blocks with your filtering, or you are encoding with too low of a bitrate, so that blocks appear in your final product.

[Mug Funky]

i think the blocking would be better handled with deblocking in the decoder... if it's only evident on large flat areas, and only when you amp up the saturation, it's not exactly a big problem. it'll probably be ignored completely by the encoder you use.

perhaps try doing only chroma deblocking (using cpu2="oxoxoo" i think) so that the deblocking process doesn't mess with sharp details (title text is particularly sensitive to this, and you'll know it when you see it).

hehe... what's with the Chii/Hideki stuff? Mug will do (perhaps i should change my avatar to a Necoconeco?)

[Guest]

Yeah, but what if it's only appearing after he re-encodes?

[Pyrii]

I did a couple of test encodes, target filesize for each was 5120Kb (30 seconds) as that would be target filesize (25mins at 250Mb)

I didn't enable Qpel in XviD for the first one, I wanted to see the difference, and I think it shows.

http://pyrii.pfak.org/video/test1.mkv

http://pyrii.pfak.org/video/test2.mkv

Rendering times for each was about 5 minutes, for each pass.

I'm quite pleased witht he second one.

tester.avs
LoadPlugin("C:\Documents and Settings\Administrator\Desktop\plugins\kerneldeint.dll")
LoadPlugin("C:\Documents and Settings\Administrator\Desktop\plugins\fieldhint.dll")
LoadPlugin("C:\Documents and Settings\Administrator\Desktop\plugins\dgdecode.dll")
#LoadPlugin("C:\Documents and Settings\Administrator\Desktop\plugins\decomb521.dll")
LoadPlugin("C:\Documents and Settings\Administrator\Desktop\plugins\Msmooth.dll")
LoadPlugin("C:\Documents and Settings\Administrator\Desktop\plugins\MSharpen.dll")
LoadPlugin("C:\Documents and Settings\Administrator\Desktop\plugins\RemoveGrainSSE2.dll")

dgdecode_Mpeg2Source("C:\Documents and Settings\Administrator\Desktop\azu1.d2v")

FieldHint(ovr="C:\Documents and Settings\Administrator\Desktop\azu1\azu1.d2v.tel.txt")
KernelDeint(order=1,sharp=false,twoway=true,threshold=7)

#Decimate(cycle=5,quality=3,ovr="C:\Documents and settings\Administrator\Desktop\azu1\azu1.d2v.dec.txt")

RemoveGrain(mode=2)

ConvertToRGB()
MSmooth(show=false,mask=false,highq=true,threshold=5,strength=3)
Msharpen(threshold=4,strength=20,highq=true)

[Wilbert]

Could you tell me about fieldhint.dll? I can't find any info about it here.

[Pyrii]

It's used by yatta for teleciding. it parses the output from telecide() and stores the values in a file (C:\Documents and Settings\Administrator\Desktop\azu1\azu1.d2v.tel.txt), then you can use the yatta interface to go through frame by frame and change frame fields at will. Here's part of the .tel.txt file:

3259,3258,+
3260,3259,-
3260,3260,+
3261,3261,-
3262,3262,+
3264,3263,+
3264,3263,-
3265,3265,-
3266,3266,-
3267,3267,+

Official description:
FieldHint combines arbitrary fields from the input clip, and optionally adds Telecide-compatible postprocessing hints.

usage:
FieldHint(clip, string ovr)

ovr: required.
Filename from which to read field pairs.
Format:
0,0,-
3,42,+
1,2
Each line consist of a pair of numbers: the frames from which to take
the top and bottom fields, respectively.
The optional 3rd parameter sets postprocessing hints.
("+" => mark as interlaced. "-" => mark as progressive.)

[madoka]

Originally posted by Pyrii
I did a couple of test encodes, target filesize for each was 5120Kb (30 seconds) as that would be target filesize (25mins at 250Mb)

I didn't enable Qpel in XviD for the first one, I wanted to see the difference, and I think it shows.

http://pyrii.pfak.org/video/test1.mkv

http://pyrii.pfak.org/video/test2.mkv

Rendering times for each was about 5 minutes, for each pass.

I'm quite pleased witht he second one.
These are excerpted from my own backup. IIRC they were compressed at ~1000 kbps, with all of XviD's bells and whistles enabled. They were not pre-processed in anyway, other than de-interlacing--done without any manual intervention. (Sorry about breaking the first clip in half; GeoCities caps upload size to 5 MB.)

http://www.geocities.com/ayanami_rei_ii/clip/mytest1a.avi
http://www.geocities.com/ayanami_rei_ii/clip/mytest1b.avi
http://www.geocities.com/ayanami_rei_ii/clip/mytest2.avi

I've been wearing glasses since 5th grade, so my eyesight isn't exactly the best. However, I compared your test clips to my backup, and I honestly can't tell much of a difference.

I don't mean to pass judgement here. Just because I don't see a difference doesn't mean you don't either. However, to me all of your efforts--you even went through the trouble of manually sifting the interlaced frames--would not be worth the time it took.

But seeing your results made me really question my own judgement. Are my standards so low that I couldn't appreciate the improvements you've made? Do any other see significant differences that I don't? :confused:

[SirCanealot]

Originally posted by Pyrii
http://pyrii.pfak.org/video/test1.mkv

http://pyrii.pfak.org/video/test2.mkv



Yeah, I can see the swiming blocks in the solid blocks of colours ^^
Are you sure the sources are that clean? I've seen the type of artifacting those .mkv samples have on my DVD of DN Angel Vol1 - basically, grain-like mass of swiming blocks(not to mention dotcrawl and rainbowing, haha).

Look at Yukari's table in clip 2 - it's absolutely swiming with artifacting.

I can see how removegrain is helping via your screenshots - perhaps you could try Degrainmedian? In my own experience, Degrainmedian can be set to a much higher level without destroying the image's line, etc (you should see what it did for a 1970s anime where they forgot to restore the op/ed - degrainmedian was very helpful). However, degrainmedian DOES blur somewhat...
Perhaps Removedirt too?

Perhaps you could also try using FFT3D instead of removegrain? In my own personal experience, FFT can work miracles or, err, "un-miarcles" for anime (as in: be careful, since it likes to randomly ADD grain to anime sources).
And as Mug Funky suggested, you could also try the decoder's deblocking (though, for some reason, I've never been a fan of this).

You might also want to look into "Colormatrix". I'm too lazy to read up on what this is ACTUALLY meant to do, but isn't part of the reason we easily see blocking like this on DVD-Rips is because we use different colour standards on a monitor(or whatever)? Colormatrix is meant to restore these, so you might want to check it out...

You could also look into fansub-style industrial-acid filtering, which would be Deen or 2DCleaner (along with C3D and some other random stuff) :P
However, keep in mind, fansub ethic requires that you remove ALL detail from the source to make sure you can aim for tehr 100 meh0r l33t0r encr0de (encoded at quant 8).

I also noticed you had your B-Frame quantiser ratio set to 1.5: you should try setting this to 1.0 for anime (or encoding in general; I don't know).
You've also got quant 1 frames enabled. You may want to turn these off: the odd quant 1 frame is only going to increase file size hugely for only a tiny quality increase. Quant 1 frames are only useful when there's a large number of them (which is impossible for "normal" filesizes: I only ever use them for short dark/hard to compress scenes which I run off at quant 1 or 1.5).

Also may want to use Kagura's 2-pass settings, which you can read about here: http://forum.doom9.org/showthread.php?s=&threadid=92366
These have helped me a lot (make quant 3/4 look almost as good as quant 2 in low-motion scenes :P). Also consider capping the quants as Kagura suggests.

azumanga is extremely clean. no noise at all (unless it's intentional)... good sharp lines too.

Even Spee-DVD authored(/god authored) Last Exile still has SOME blocking. Encoding this with NO filtering would just be silly (though I'd recomend some sort of trimmed script, since most scenes and insanely low on blocking and even Undot, for once, kills a decent amount of the noise in them).

[Kagura]

For DVDs, go back to basics. Strong denoising tools (more likely cleaners and smoothers) such as deen, 2dclean, and fft3dfilter aren't needed. Instead, I find filters such as fluxsmooth, temporalsoften, removegrain, and dust (certainly not all of them at the same time) effective at combating dvd noise.

[DigitalDeviant]

I don't know if this is a good method or not, but 80% of the "clean" anime sources I encode looks pretty good to me after using MipSmooth(preset="animeHQ") followed by a medium strength LimitedSharpen().

I fully agree with SirCanealot, and especialy with anime, almost all mpeg-2 can stand use a little filtering. Azumanga is fairly flat looking, without a lot of texture, it should be easy to clean this up a little without killing much detail. The only times I wouldn't filter something would be instances where there are more complex textures, like Wonderful Days with the combination of 2d/3d and all the fog and rain.

As for IVTC, as long as it's mostly 3:2 pulldown I would go ahead and IVTC or make it VFR. I usualy encode it first, then watch it and look hard for any artifacts before I start manualy overiding. If I see anything then it's usualy it's pretty quick and easy to find the offending frame in vdub and make the appropriate override.

[mg262]

Two quick things that occur about deblocking... the first is that in my (limited) experience, blocks seem to be much more visible on a flat screen monitor than on an old-style one or a TV. The second is that whether it is worth deblocking, how much effort to put into deblocking, etc seem to me to depend heavily on the target bit rate and codec... so it may not make sense to suggest pushing this one way or the other unless these things are clearly specified.

[I tend to encode at ~7000 to 9000 kbps, PAL DVD, and so I find I have almost no common ground to discuss these things with anyone else!]