Req: anime clean-up advice

[FredThompson]

(edit: updated on 2005-11-05 to show current filter/results)

current script:

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MaskTools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\PeachSmoother.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrainS.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RepairS.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\SSEToolsS.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Undot.dll")
Import("C:\Program Files\AviSynth 2.5\plugins\FastLineDarken.avs")
Import("C:\Program Files\AviSynth 2.5\plugins\LimitedSharpen.avs")
Import("C:\Program Files\AviSynth 2.5\plugins\LRemoveDust.avs")
Import("C:\Program Files\AviSynth 2.5\plugins\Soothe.avs")
ConvertToYUY2()
PeachSmoother(NoiseReduction=50)
ConvertToYV12()
Undot()
LRemoveDust_YV12(17,2)
Soothe(LimitedSharpen(ss_x=1.0,ss_y=1.0,Smode=4),last,24)
FastLineDarken(strength=4, luma_cap=191, threshold=4, thinning=0)
ConvertToYV12()

Raw Frame (http://home.mindspring.com/~utils/Raw%20Frame.png)
Filtered Frame (http://home.mindspring.com/~utils/Filter%20Test.png)

Loss of detail with PeachSmoother is very, very small but the chroma smoothing is nice.

[Didée]

In that source frame, I don't see any halos to start with. So instead of crude sharpening and repairing the damage done afterwards, I'd rather use smarter sharpening :)

Compare here (http://forum.doom9.org/showthread.php?p=728986#post728986). (Although not even used there,) LS' edge detection method is more likely to seperate detail from grain. The "undershoot" parameter is available at Didée's ;

[Chainmax]

I don't see any haloing in that frame either, and I wouldn't recommend BlindDeHalo (any of them) for dehaloing as I feel they blur the picture way too much and completely destroy small details. I recommend the following:

LoadPlugin(TIVTC or TDeint)

LoadPlugin("C:\MPEG-4\Avisynth Plugins\SSETools.dll")
LoadPlugin("C:\MPEG-4\Avisynth Plugins\RemoveGrain.dll")
LoadPlugin("C:\MPEG-4\Avisynth Plugins\Repair.dll")
Import("C:\MPEG-4\Avisynth Plugins\LRemoveDust.avs")

LoadPlugin("C:\MPEG-4\Avisynth Plugins\MaskTools.dll")
Import("C:\MPEG-4\Avisynth Plugins\LimitedSharpen.avs")

IVTC or Deinterlacing
LRemoveDust_YV12(17,2)
Crop(your_settings,align=true)
Lanczos4Resize(your_size)
LimitedSharpen()

The latest version of LimitedSharpen can be found here (http://forum.doom9.org/showthread.php?t=84196&highlight=LimitedSharpen) and I'm using the following version of LRemoveDust:

function LRemoveDust_YUY2(clip input, int clmode, int "limit")
{
clmode=default(clmode,17)
limit=default(limit,2)
repmode = 2
clensed = Clense(input, grey=true)
rep = Repair(clensed, input, mode=repmode, modeU=-1)
rg = RemoveGrain(rep, mode=_mode, modeU=-1)
return LimitChange(rg, input, limit, limitU=255)
}

function LRemoveDust_YV12(clip input, int clmode, int "limit")
{
clmode=default(clmode,17)
limit=default(limit,2)
repmode = 2
clensed = Clense(input)
rep = Repair(clensed, input, mode=repmode)
rg = RemoveGrain(rep, mode=clmode)
return LimitChange(rg, input, limit)
}

[FredThompson]

Thanks, guys. Yes, "smarter" sharpening which won't enhacne grain is far preferable to oversharpening then trying to correct it. Thanks. If BlindDeHalos blur too much, how to remove oversharp artifacts?

[Chainmax]

I don't see any halos, but if there are you could try FixVHSOversharp or HQDering. They both can cause slight artifacting, so try them both and see which one you prefer.

By the way, which cartoon is that?

[FredThompson]

This is the NTSC anniversary DVD of Heavy Metal. The PAL one is horribly blurred. You'd think they'd have cleaned the film a little better. Things which are pure white in the original, like the sun, are marred with grey.

I toyed with making a comment about how I'm not an anime fan but maybe I'd get some help with filters because this one has large-breasted promiscuous women, even if they aren't Japanese schoolgirls :P

[Didée]

maybe I'd get some help with filters because this one has large-breasted permiscuous women,
Ssssure!! - Bring 'em here, we'll pimp 'em up to perfection!

The samples, I mean.

[FredThompson]

I'm very, very happy with the script which has been developed. There are 2 parts for which I'm requesting additional help.

I'd like a way to make the black outlines have a little more punch. There is a slight softness to them after filtering. Maybe that's just the way it has to be, maybe it's an optical trick of human vision and apparent complexity from the noise, I don't know. MSharpen won't do the trick. This seems like something that would need to be done inside LimitedSharpen. I've got the pseudio-updated version from a few days ago and played with the parameters but couldn't see how to make the blacks more prominent without increasing the noise in the rest of the image. This is kind of like the old DarkSmoother issue that bugged me for a long time. IOW, the effect needs to be applied just to edges between dark and light pixels to increase the contrast but not create whities. This is all a minor issue, though. Compared to the original, the filtered stream is a joy. I only bring this up because it seems a natural thing for animation.

The second issue is removing posterization (contours) which are in the original. Cleaning the frame actually makes them more visible. Deblockign routines made them more prominent because the edges were longer. The "best" option, other than hand-editing the frames, would be a paint-like fill routine which is frame, co-ordinate and color specific. Since that's not an option, I'm hoping the animation fans might have some insight. The challenge is only smoohting certain aspects of certain frames. This is only a problem in a handful of very short segments. It is quite visible and, I'm guessing, is something from which other people can benefit if there is a good technique.

Here are frame grabs showing the result with and without the LRemoveDust call. They are JPEGs but there is more than enough quality to illustrate the posterization problem. The original areas are solid colors but the transfer was poor. The drop in complexity of the background (look at the purple rock face) is not a big deal. There's enough motion between frames that it looks OK.

http://home.mindspring.com/~utils/Den%20posterization%20with%20LRemoveDust.jpg
http://home.mindspring.com/~utils/Den%20posterization.jpg

Yes, I know about the dark rectangle. Funny thing, it's not in the original....sheesh :rolleyes:

Does anyone have a way to "cure" this posterization mess?

Edit: Is there a way to automate from an AviSynth script export of frames, mod with GiMP, then re-incorporate the frames? That would be really, really cool.

[Tripx]

You can try FastLineDarken (http://forum.doom9.org/showthread.php?t=82125) to darken your lines.

I've used a plugin called ReplaceFrame (http://zeratul.cellosoft.com/replaceframe/) to export frames out of and back into a video. It's really quite simple, this example would export frames 62 and 478 to 480.

ReplaceFrameSetup(write=true,path="d:\encoding\")
ReplaceFrame(62)
ReplaceFrame(478,480)

I can fix them in photoshop then just save over the originals and change write to false.

[FredThompson]

Yeah, I've come to the same conclusion, export the trouble frames and touch them up then run the regular filter with the "cleaned up" frames replacing the puked ones. Not as elegant as a script but it does let me "git 'er done."

[FredThompson]

FastLineDarken is nice. In this case, only a little touchup was needed. Zooming in shows the darkening changes the charater of the lines a little bit but the overall effect removes the smearing of the noise reduction and the differences between frames should even out as it plays. Thanks a lot.

I wonder if intentionally oversmoothing with something MSmooth then using FastLineDarken could create a paint-like effect, similar to the PhotoShop filters that turn a photo into a "painting" or "drawing."

[FredThompson]

Well, that's not going to work. There are just far too many frames to do this by hand.

The closeups with this posterization have relatively simple backgrounds most of the time. There are some medium shots with posterization in the characters but it's not that evident.

Maybe a solution is ApplyRange and the "regular" script for all but the closeups. Those would use VagueDenoiser or FFT3DFilter. I wonder if they could be applied to only those pixels which fall within a specified chroma range. That would be nice. The skin color of the characters could be subject to smoothing but the backgrounds would be left alone. In the screenshots up a couple of posts from here just the reds would be smoothed.

[hartford]

You might try a bit of temporal smoothing at the
end of the script. Something like:

TemporalSoften(4,4,8,10,2) # temp_radius, luma_thresh, chroma_thresh, scenechange, mode


I don't use this as I prefer VirtualDub's internal TemporalSmoother, usually
set to "4."

Often after a scenechange, the first frame is noisy. It can be fixed using
a deleteframe() duplicateframe() operation, if there is no motion.

Well, my 2-cents. :)

[FredThompson]

I've added Grape just before Peach which helps a little for color swim. Peach is already doing some adaptive temporal smoothing. I really think it's near the limit of what smoothing can do without killing the detail. Wilbert suggested using ColorKeyMask which I'll do if I can't find a paint-like fill function.

[freshNewB]

this is my config
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\fft3dfilter.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MaskTools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\PeachSmoother.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\despot.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RepairS.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\despot.dll")

DirectShowSource("03.DAT",fps=29.970, audio=false)
FFT3DFilter(sigma=2, plane=0)
FFT3DFilter(sigma=3, plane=1)
FFT3DFilter(sigma=3, plane=2)
PeachSmoother(NoiseReduction=50)
ConvertToYV12()
DeSpot()

Undot()

i do a little change from FredThompson 's script
the video source is a poor quality vcd
i compress it to 390kbps of avc stream by x.264
using auto 3-pass
Next job job2-3 is a video job. encoder commandline:
"C:\x264\x264.exe" --pass 2 --bitrate 390 --stats "C:\temp\03.stats" --ref 8 --bframes 3 --b-pyramid --filter -1,-1 --subme 7 --b-rdo --weightb --analyse all --8x8dct --ratetol 5.0 --qcomp 1.0 --b-bias 1 --me umh --merange 24 --threads 2 --progress --no-psnr --output "C:\temp\03.264" "C:\temp\03.avs"
http://img454.imageshack.us/img454/1212/snapshot200511062136310bc.jpg
i find it helps a lot at removing the noise
but it also makes up new noise . it would be called ghost or something that appears in the image above.
how to avoid these ghost?