Need advice on denoising/derainbowing anime

[YnK]

I'm encoding an anime that has some... interesting effects going on. Flickering rainbows and dots, usually near edges. My attempts at denoising it don't give much result, since I'm still a bit clueless about why these effects happen at all and how to deal with them properly (I only have basic knowledge of Avisynth - as in, "left everything up to MeGUI until some problems were encountered"). Tried to search the forum for info, but honestly most of this stuff is still beyond my understanding. Most of functions I download tend not to work at all because I don't know what they need to work. So, if possible, I'd like an advice from somebody more experienced than me. I sincerely apologize for my stupid question.

Video sample (http://www.mediafire.com/?txngj4kjlyv).

[osgZach]

Holy dot-crawl Batman ! :p

Have you tried DeGrainMedian ?

Sometimes its more beneficial to hit it with multiple passes of the same filter at different strengths, too.

[osgZach]

edit: how the hell did this reply end up here? ugh.. sorry

[Nightshiver]

It's really not that difficult. A simple

Chubbyrain2()
Checkmate()

was all that was needed to get rid of the rainbowing and dotcrawl. Are you sure you have de-interlaced it properly? And by this I don't mean by leaving it up for the "automatic" analyzing tool in MeGUI.

[YnK]

It's really not that difficult. A simple

Chubbyrain2()
Checkmate()

was all that was needed to get rid of the rainbowing and dotcrawl.

Oh... well, thank you - it looks better now. I just never had to deal with this kind of video before, so I weren't sure if I can treat it properly. In most cases, whenever I'm trying to use something for the first time, it fails to work. At least the filters I tried before did so.

Are you sure you have de-interlaced it properly? And by this I don't mean by leaving it up for the "automatic" analyzing tool in MeGUI.

Eh? If you're referring to my sample, that anime is fully 29.97 progressive (save for one scene that had combing and needed to be fixed by a field-matcher, which was the exact point where I stopped using MeGUI's source detector altogether - there's no way it could have spotted it or done anything about it, so I only determine the video type by actually watching it :)). Also, I'm under impression that it being progressive is the reason it got all these artifacts in the first place - there's a second episode that is normal 3:2 telecine save for the opening, and it doesn't have any color/noise issues. Funny...

[Nightshiver]

Just wanted to be sure. You're lucky to have a progressive source for an anime, that usually doesn't happen too often.

[YnK]

Well, progressive anime is tricky - it seems to be easy to encode, because you don't have to bother with IVTC or deinterlacing, so if you aren't paying attention, you'll miss a lot of interesting stuff in it (noise, rainbows, and even some job for TFM (http://www.mediafire.com/?nww1mndtifm).).

Hmm. Apparently, checkmate produces some artifacts on its own - currently looking at a high-motion scene, and it has dots cleaned up, but there are some lines that end up in frames they shouldn't be in... so I'm doing something wrong again. Does that filter have a manual or something? I can't seem to find anything like that...

[osgZach]

http://www.animemusicvideos.org/guides/avtech/avspostqual.html
(I think this link is OK?)
Try reading over the various sections of this guide (its linked to the post processing page)

Information is kind of old.. but some well known filters are listed there with some general instructions..

As for dot crawl and rainbowing, I always thought that had more to do with poor-quality hardware during the transfer, signal leakage, etc.. Like dumping something over SVideo or Composite.
I can't say I've ever seen deinterlacing actually cause those artifacts on the stuff I've worked on, they always seemed to be there regardless.

edit: actually this is the page I was thinking of.. (that guide is weird, different version or something)

http://www.animemusicvideos.org/guides/avtech3/post-qual.html

Here's another I found a long time ago.. may still have some useful information
http://www.aquilinestudios.org/avsfilters/

Dot crawl in specific - http://www.aquilinestudios.org/avsfilters/dotcrawl.html

[Nightshiver]

Don't follow those guides, as they are old and unreliable, honestly. It's good for a general idea, but it's horrible as a modern day guide. Ynk, can you post a sample with motion in it?

[osgZach]

Well like I said, its probably outdated, but if he wants lots of examples to get on idea of settings to tweak, there's load of them

[YnK]

Here (http://www.mediafire.com/?n3zdmej4nom) is the kind of scene where checkmate does weird things, maybe because frames change a lot. The background is moving, the guy is moving... so some of this movement shows up on frames where it wasn't originally.

[Nightshiver]

Took a look and I didn't see anything wrong. Are you using this (http://www.mediafire.com/?mmwhztnlgnz) version?
Here's (http://www.mediafire.com/?g3y2tyzjhnc) a link to Chubbyrain2 in case you needed it.

[YnK]

You mean you aren't getting this effect (http://www.mediafire.com/i/?1mqdmmftdbg) (look at the bottom half of the image for a few stray lines)? I updated the filter from your link, and it's still the same.

[osgZach]

You sure that's not film scratches or some other kind of damage?

Do you get the exact same pattern (or close variations in the same spots) on every single frame, or does it appear and go away randomly in different shapes, sizes and places?

[Stephen R. Savage]

That's one of the classic problems with Checkmate, and there is not a lot you can do about it. You can try adjusting the parameters (there are three, but they are not documented). In particular, I think lowering tthr (?) may reduce this, but your results will probably vary, and you will not be able to completely eliminate both the dotcrawl and the artifacts. The artifacts do have some properties in common with film scratches and spots, so you can try to smooth them out with temporal-spatial denoising. I find that dfttest (MC) is somewhat effective at attacking these artifacts.

In reality, the problem is that Avisynth is seriously lacking for a highly effective dotcrawl filter. Checkmate is the only filter that can remove dotcrawl on moving parts of the frame, but it suffers from that ghosting issue you noticed. The only alternative I can think of is to use TComb and settle for partial dotcrawl elimination.

[Vitaliy Gorbatenko]

To avoid artifact's with checkmate() use: checkmate(tthr2=0)

[*.mp4 guy]

In reality, the problem is that Avisynth is seriously lacking for a highly effective dotcrawl filter. Checkmate is the only filter that can remove dotcrawl on moving parts of the frame, but it suffers from that ghosting issue you noticed. The only alternative I can think of is to use TComb and settle for partial dotcrawl elimination.

This is slow (http://forum.doom9.org/showthread.php?p=1379930#post1379930), but its effective and shouldn't ever cause artifacts. But to date it has been tested insufficiently. Also, its slow. Not fast. At all.

[edit] I say it doesn't cause artifacts, and apparently unbt does on this source, for some reason I don't feel like exploring right now.

unbs(strong=1) works fine though, even though its basically a spatial only filter.

[edit2] at a guess I would say that the subpixel scroll is making the spatio-temporal median go mental. Anime, meh.

[YnK]

My guess is that nobody actually had to deal with this problem on this particular type of source (progressive animation) before. Or something. Yes, progressive anime is relatively rare, but I wonder if these effects can happen in other similar sources (such as natively PAL cartoons that are normally 25fps progressive by default).

Dotcrawl is annoying. I realize I probably won't be able to remove it completely, but I'm trying to at least making it less noticeable (flickering = headache). Will try some combination of filters, though I'm not sure my PC can handle several of them at once. It's slow...

[osgZach]

To be honest.. that picture looked pretty ok.. Granted I don't know what this looks like in motion, but this is one of those cases where you should watch the processed clip in full motion, and see if you really notice those artifacts...

and at the end of the day, would you want to look at them? or a highly visible checkerboard all over your picture ?

Sometimes it comes down to taking what you can get unfortunately.. You have the original source, so you can always try new things in the future when they are developed.

[YnK]

It's just that I'm an unfortunate kind of a person who automatically notices everything that's wrong with video without really trying to (this doesn't apply to just video, of course, but it's how it works). I already encoded this thing without filtering once -- it's not that bad compared to some other sources, maybe, but I keep noticing the flickering because it's all over the place in this episode. So yes, it's really more of a personal issue, sorry. However, now I at least know some theory behind dealing with it, so thank you for help anyway - I'll probably continue some research on the problem on my own.

(Incidentally, I probably would like to get rid of my video watching quirks, because I actually own a nice collection of various pathological anime DVDs, including: NTSC field-blended anime, NTSC anime with that awesome half-frame movement thing, and PAL-converted anime with full-frame blends. I have to make backups of this stuff, because my DVDs tend to only live for a year or so due to my tendency of scratching them - long story and not really my fault. So...)

[osgZach]

I completely get where you're coming from.. I notice nearly every little thing wrong with my sources too.. But in the grand scheme I know I personally would settle for a few weird-but-not-very-visible-unless-your-looking-for-them-artifacts, versus a flickering checkerboard across the whole image.

As a side note.. I've always wondered what the word "pathological" refers to when discussing videos.. I've seen it mentioned several times on the forum but never understood what was meant.

As for your other DVD's.. There are some good filters that can help with field/frame blending. Not always to perfect success but at least reducing the anomalies.. It might also be worth your time to look into YATTA if you haven't already. For various problems (IVTC being its main purpose) it allows very fine control over each frame, for those who have the time & iron will to work at a snails pace.. (I don't, but luckily I was able to write a tool to knock IVTC in Yatta down from a few hours to a couple of minutes).

One last question. What is the "awesome half-frame movement thing" ?

[YnK]

I completely get where you're coming from.. I notice nearly every little thing wrong with my sources too.. But in the grand scheme I know I personally would settle for a few weird-but-not-very-visible-unless-your-looking-for-them-artifacts, versus a flickering checkerboard across the whole image.

Maybe you're right... And so far I only have one case of it anyway, so I won't have to spend too much time re-encoding it in case I find a better way to deal with the problem.

I do have some DVDs that only require "Force film" option in DGIndex to be deinterlaced properly, so if I start to compare them with those I'm currently encoding...

IIRC, I had YATTA at one point, but that was some ancient version, apparently, because I didn't really find it useful that time and gradually switched back to writing scripts entirely by hand.

One last question. What is the "awesome half-frame movement thing" ?

Partial field blending. You know, when the bottom half of the image goes into the next frame, while the top half stays in the current frame? I have DVDs like this, the best part being that they're a local release, which in theory means they could've been converted to PAL for further botching...