SmartSSIQ and Bifrost are awesome rainbow removers

[Chainmax]

I recently took on the task of backing up the whole Trigun series at 640x480 to a single DVD-R with the original japanese audio. I already read that this source had some nasty rainbowing, but I never expected it to be that bad. Anyway, after trying a few alternatives, I settled on SmartSSIQ (http://forum.doom9.org/showthread.php?t=98267&highlight=SmartSSIQ) and Bifrost (http://yatta.mellbin.org/misc/), and the results were stunning. Here's a comparison:


Before:
http://img243.imageshack.us/img243/3005/before6ye.png (http://imageshack.us)

After:
http://img243.imageshack.us/img243/8448/after8fp.png (http://imageshack.us)

If you're interested, this is the script that was used (minus the dehaloing since I haven't found a suitable one):

LoadPlugin("X:\wherever\DGDecode.dll")

LoadPlugin("X:\wherever\TIVTC.dll")

Import("X:\wherever\VDub_Filters.avs")
LoadPlugin("X:\wherever\WarpSharp.dll")
Import("X:\wherever\SmartSSIQ.avs")
LoadPlugin("X:\wherever\Bifrost.dll")

LoadPlugin("X:\wherever\SSETools.dll")
LoadPlugin("X:\wherever\RemoveGrain.dll")
LoadPlugin("X:\wherever\Repair.dll")
Import("X:\wherever\LRemoveDust.avs")

LoadPlugin("X:\wherever\aWarpSharp.dll")

LoadPlugin("X:\wherever\MaskTools.dll")
Import("X:\wherever\LimitedSharpen.avs")

MPEG2Source("X:\wherever\Trigun.d2v")
TFM(d2v="X:\wherever\Trigun.d2v",mode=6,PP=7,slow=2,mChroma=false)
TDecimate(mode=1)
SmartSSIQ()
ConvertToYV12()
BiFrost(interlaced=false)
Deblock(quant=35)
LRemoveDust_YV12(17,2)
Crop(10,0,704,480,align=true)
aWarpSharp(depth=16,cm=1)
Lanczos4Resize(640,480)
LimitedSharpen()

[mg262]

That was pretty effective on the rainbows... but you've killed a lot of detail. Compare the top left mountain before and after...

[Fizick]

mg262,
What was interesting in this mountain? :)

[Revgen]

That was pretty effective on the rainbows... but you've killed a lot of detail. Compare the top left mountain before and after...

Who in their right mind would want to stare at a mountain? :p

The focal point of the scene is the two characters. The characters look better without rainbow snot hanging on their noses and in their ice cream. LOL!

[Leak]

mg262,
What was interesting in this mountain? :)
Well, in the before-image the mountains (and the buildings) didn't look as if they were ripped straight out of Super Mario World on the SNES... :p

Also, I doubt that that effect comes from the de-rainbowing itself - looks more like general oversmoothing to me.

np: Otto Von Schirach - Tre Duece Ave. Smash & Grab Version (Oddjob Discrimination)

[AVIL]

Hi,

I've found Fizick 's fft3dfilter over U and V planes very effective against VHS rainbows. And I also say "Save our mountains" (I love details). ;)

[Chainmax]

mg262: if you look at the mountains, you'll see that each shape from the mountain in the "before" picture is also on the "after" one. For me, not much detail has been lost, it's just that all the noise has been completely wiped out. Then again, I'm not really sure how to define "detail". What exactly on the mountains (or elsewhere) do you think was integral to the picture and is missing in the "after" frame?

Leak: I didn't check filter by filter, but this is probably caused by Deblock + LRemoveDust_YV12. The source has some awful blockyness in many places.


To me, the result is much better: there's no ranbowing and the characters look much sharper. I can provide more comparison frames if you guys want to discuss this further.

[Chainmax]

AVIL: I used FFT3DFilter at sigma=6 and bt=3 (alone and with TemporalSoften) but it didn't do much to the rainbows.

[foxyshadis]

Everyone kinda has their own preferences about animated detail... I used to really like the abstract feel of heavy smart smoother/smart blur, now I'm more inclined to prefer saving more detail and grain (although I still can't stand screening and will remove it at any cost).

[Chainmax]

But do you like the results? By the way, what do you mean by "screening"?

[foxyshadis]

The amount taken off the mountains is probably a little more than I'd like, but it's acceptable, I'd just watch it with a very low grain filter so smoothness (and encode artifacts) doesn't stand out. The important question is, do you like it? ;)

Screening is the pattern of dots to give printouts the illusion of solid color and a wide gamut with 1 or 4 pigments; in high res scans they stand out and bloat the filesize, when shrunk they cause ugly moire patterns, and reprinting is even worse. It's hard to find an example that isn't enormous, but here's one: kenshin (http://foxyshadis.slightlydark.com/random/manga300.jpg)

[FredThompson]

The loss of detail removes depth. In the second image, the mountains looks more like clouds. It looks like a lot of the heavvy-handed oversmoothing I used to do with some camcorder footage in an attempt to remove moire and rainbows from moving metal roller conveyor or venetian blinds during a pan.

The fastest rainbow filter I've ever seen is called GreyScale()...

Seriously, I like the way the main characters look but the loss of texture makes the image more flat, less rich, more...children cartoony. Sorry, I can't think of an appropriate phrase to explain.

[mg262]

Fred sums up my feelings better than I could. Very subtle loss of detail is always difficult to describe verbally -- but you may be able to see a slight, bright, horizontal line on the top left mountain that is destroyed. It's not just detail lost -- the warpsharp that makes the foreground look so good makes the colour transitions in background objects sharper than they should be.

I do really like the end result on the foreground... but not at the cost of what it does to the background. Would that we could separate them with a mask and filter them separately. (Even that wouldn't be perfect -- compare the cup at the right-hand side of the table before and after; to me the pattern is a lot more two-dimensional and less distinct after filtering.) But at the end of the day, this is a matter of personal preference -- if you're happy with it, Chainmax, go with it.

Edit: foxyshadis, I would guess that that kind of periodic noise might be dealt with pretty effectively by frequency domain filtering... but it would have to be a filter that targeted a particular frequency. I've never used one of these, but I know they're out there...

Edit: http://www.avisynth.org/fizick/defreq/defreq.html

[Mug Funky]

hehe... composite video is a pain. this sort of thing really shouldn't get on a DVD - after all, if a DVD looks as good as an analog capture, why buy the disc? (unless you're in australia, in which case buy every anime disc you can find except those released by Siren or Magna Pacific ;)).

alas, i suspect a lot of this stuff had rainbows before it was even sent to america and dubbed.

btw, i'm in the camp that prefer to keep a little noise and all the detail then keep a little detail and none of the noise. but i use higher bitrates, so that's probably not a representative opinion.

[Chainmax]

From inspecting filter by filter, I discovered that the main culprits for the oversmoothing are Deblock and LRemoveDust_YV12. I'll see if I can replace them by something less destructive, although it's going to be hard: even with these two filters, there are a few scenes that don't come out perfectly, so screwed up they are. I'll post a comparison of this filterchain to the source in one such frame and one with a new filterchain. Thanks for the feedback :).


P.S: it seems BiFrost was enough after all.

[FredThompson]

Deblock might be more of a problem than LRemoveDust_YV12. MDeblock is worth trying IF you need to deblock. http://www.mdeblock.de.tf/

Why are you deblocking, anyhow? That's going to make some ugly smoothing if you aren't careful.

Look at the script I'm using now. It's a slightly modified version from suggestions in this area. http://home.mindspring.com/~utils/

[mg262]

I would also recommend trying to restrict the area on which sharpening/warpsharpening is applied if you can pull it off. Some of mf's scripts try this kind of thing by using masks, as does neuron2's MSharpen. (In theory, warp sharpening only affects edges anyway... but in practice I find it picks up a lot of detail.)

[Chainmax]

FredThompson: Deblock is from DGDecode and uses an AVC-like deblocking technique. Are you sure it's that destructive compared to other deblockers? And yeah, I do need to deblock. You'll see why when I post the comparison frames I talked about in my previous post.

mg262, that sounds like a good idea, but it's probably more effort than it's worth for me. I only know basic Avisynth filtering and scripting is beyond my reach, at least for now.

[Chainmax]

Here are two comparisons that show how not even this filterchain can completely remove all issues on them:

Before:
http://img139.imageshack.us/img139/6826/fuframe329source1uh.png (http://imageshack.us)

After:
http://img139.imageshack.us/img139/4848/fuframe329result9hr.png (http://imageshack.us)



Before:
http://img495.imageshack.us/img495/4673/fuframe2904source7ma.png (http://imageshack.us)

After:
http://img495.imageshack.us/img495/9495/fuframe2904result7dk.png (http://imageshack.us)

[Chainmax]

Here are the results of a new filterchain:

http://img463.imageshack.us/img463/9152/nwfltrchn2572result5xi.png (http://imageshack.us)


And here is the filterchain:

TFM(d2v="X:\wherever\Trigun.d2v",mode=6,PP=7,slow=2,mChroma=false)
TDecimate(mode=1)
BiFrost(interlaced=false)
Deblock()
FFT3DFilter(sigma=1,bt=3,bw=32,bh=32,ow=16,oh=16)
Crop(10,0,704,480,align=true)
aWarpSharp(depth=16,cm=1)
Lanczos4Resize(640,480)
LimitedSharpen()

This filterchain does look slightly worse than the original one on the two frames in my previous post (I can upload the results on those if you guys want), but the difference in image detail on the rainbow picture is amazing. You guys were right all along, thanks for the feedback :).

[neuron2]

Moderator note: The discussion from post 21 and on was split to a new thread called "New halo removers discussion". Please use this thread for the rainbow removal discussion and the new thread for halo removal. Thank you.

[Chainmax]

Thanks :).