Pesky rainbowed source

[Eretria-chan]

I'm looking to see if anyone here has maybe some advice on removing some pesky rainbows from a rainbowy-ish source. Actually, there's more than one, but that's not the point.
Now, I have tried both DeRainbow and SmartSSIQ. With SmartSSIQ, I haven't had much luck. It makes the picture jerk. As if it can't keep up. CPU utailization is very high, not not maxing out. Not sure why it isn't working. As for DeRainbow, I've indeed had some luck, but... it just won't remove some of the rainbows.

The current scripts I've gathered are as follows:

DeRainbow:
LoadPlugin("MSharpen.dll")
LoadPlugin("MaskTools.dll")
LoadPlugin("MipSmooth.dll")

input = DirectShowSource("...")
return DeRainbow(input, 0)

function DeRainbow(clip org, int "thresh")
{
assert(org.isYV12(),"DeRainbow() requires YV12 input!")
thresh = default(thresh, 10)

org_u = utoy(org)
org_v = vtoy(org)

msharpen(org, threshold = thresh, mask=true)
reduceby2()
greyscale()
uv = blur(1.5).levels(0,2.0,255,0,255, coring=false).blur(1.5).blur(1.5).levels(50,2.0,255,0,255, coring=false)

filtered_u = org_u.mipsmooth(spatial=255, temporal=255, scenechange=3, show=false, method="strong", scalefactor=0.5)
filtered_v = org_v.mipsmooth(spatial=255, temporal=255, scenechange=3, show=false, method="strong", scalefactor=0.5)

u_final = MaskedMerge(org_u, filtered_u, uv)
v_final = MaskedMerge(org_v, filtered_v, uv)

return ytouv(u_final, v_final, org)
}

SmartSSIQ:
LoadPlugin("mt_masktools.dll")
LoadPlugin("SSIQ.dll")

input = DirectShowSource("...")
SmartSSIQ(input, 1)

function SmartSSIQ(clip input, int "strength")
{
strength = default(strength, 1) #processing strength, 0-255

# apply rainbow remover to the entire video
derainbow = input.SSIQ(11, 300, false)

# create an edge mask from the rainbow video
lutexpr = "x .2 ^ 110 *"
# uncomment to use masktools 1
#edgemask = input.edgemask(thY1=1, thY2=255, type="sobel").inflate().yv12lut(yexpr=lutexpr).GreyScale()
edgemask = input.mt_edge(thY1=1, thY2=255, mode="sobel").mt_inflate().mt_lut(yexpr=lutexpr).GreyScale()

# apply ssiq only to masked area, using "strength"
result = overlay(input, derainbow, mode="blend", mask=edgemask, opacity=strength/255.0).Limiter()

return result
}

DeRainbow seems to work better with lower values, but anything below 0 doesn't really seem to do anything. I'm not familiar with avisynth and I did not write these scripts. I merely copied them from examples.
From what I know (or found?), there isn't a 100% effective derainbow remover? If someone could help me to get rid of these rainbows, it would help greatly.

:thanks:

[BigDid]

Hi,

I use Smoothuv for analog/composite captures if (and only if) rainbows are light to medium. It is a short/reduce version of SSHiQ, details here: http://www.avisynth.org.ru/docs/english/externalfilters/smoothuv.htm

Light
radius=1
threshold=20 to 99
field=true

Medium
radius=2
threshold=20 to 99
field=true

For strong the radius is 3 or more but the field have to become false and may have side effects like hue change...

Did

[Eretria-chan]

Thanks!
It looks like a good filter, and I'll have to test it further. However, unfortunaly, it was unable to remove the most pesky of the rainbows. Even at maximum strength:

LoadPlugin("SmoothUV.dll")

input = DirectShowSource("...")
return SmoothUV(input, 7, 99, false)

I will test this further to see if it effective to remove rainbows at a decent cpu usage. Still looking for something that might help remove the peskiest of pesky rainbows, however.

[BigDid]

...Even at maximum strength...
I suppose the maximum strength is related to the max threshold wich should be 255: "SmoothUV( 7, 255, false)"

I just don't use it that way and, like you, am looking for one or many pesky removers :)

Did

[Revgen]

Sometimes rainbows are there and they are part of the original signal. I've captured CATV shows that still had rainbow even though I capped them from an S-VHS VCR with a TBC.

[Eretria-chan]

I suppose the maximum strength is related to the max threshold wich should be 255: "SmoothUV( 7, 255, false)"

I just don't use it that way and, like you, am looking for one or many pesky removers :)

Did
Actually... at maximum strength, it removes even the most pesky rainbow but I can still see the flickering where the rainbow was initially.
But even if they are there originally, they can still be removed, right?
Oh, and what are the side effects of having these rainbow removers operating at maximum strength?

[Chainmax]

For rainbows, the best options I can think of are BiFrost (http://yatta.mellbin.org/misc/) (read the readme carefully in order to know where to put in on the script), DeCross (http://rapidshare.de/files/22711944/DeCross_v0.0.02___English_readme.rar.html) or FFT3DFilter(sigma=x,plane=3,bw=32,bh=32,bt=3,ow=16,oh=16) with x = 3 to 6.

[Eretria-chan]

Question! I read somewhere that there is a GPU vrsion of FFT3DFilter?
Also, considering I'm a n00b at avisynth, if someone recommends something, can you also give an example on how to use it? Because I haven't the slightest clue on how to use them and don't understand all the technical details.
I'll definetly get around to testing all those suggestions, however.

[Chainmax]

Well, BiFrost's readme says exactly how to tell where to use it. As for DeCross, you only need to use Decross() right after loading your source, before IVTCing/Deinterlacing. FFT3DFilter can be used before or after deinterlacing, but I recommend you to use it before like this: FFT3DFilter(sigma=x,plane=3,bw=32,bh=32,bt=3,ow=16,oh=16,interlaced=true) right after loading your source. So, the script would be something like this:

SetMemoryMax(yourmem/2)
LoadPlugin("X:\wherever\DeCross.dll")

LoadPlugin("X:\wherever\BiFrost.dll")

LoadPlugin("X:\wherever\FFT3DFilter.dll")


[ ]Source("X:\wherever\yoursource.[ ]")

DeCross()

BiFrost() <--- if Type1 rainbows

FFT3DFilter(sigma=x,plane=3,bw=32,bh=32,bt=3,ow=16,oh=16,interlaced=true)

IVTCing/Deinterlacing

BiFrost(interlaced=false) <-- if Type2 rainbows

, where you would choose which of the three derainbowers to use.
In order to know what is interlacing, what is telecine and how to remove them, I suggest you to read Decomb's (http://www.neuron2.net/decomb/decombnew.html) readmes and tutorials, the IVTC tutorial (http://www.doom9.org/ivtc-tut.htm) and the capture guide (http://forum.doom9.org/showthread.php?t=78797).Those are a great source of starter knowledge and will explain telecining and interlacing with detail.

Also, take your time to read the readme of each filter you use, as they usually contain descriptions of functionality and lots of extremely useful knowledge. Finally, use the search, as most novice questions have already been answered many times.


P.S: there is a GPU version of FFT3DFilter, it's called FFT3DGPU. Be mindful that some of its switches are named differently.

[Eretria-chan]

Thanks. I know what interlacing is, but the sources where the rainbows are, are progressive so no deinterlacing should be necessary. You've given me plenty of information to look about. There is much work to be done, so thanks, I will look into it and see if I can get rid of those pesky rainbows.

[Chainmax]

Don't forget to tell us how it goes :).

[Eretria-chan]

So far, I've used SmoothUV at presumably maximum strength. I haven't been able to notice any rainbows. However, I did encounter a side effect of it. Colors are extended to where they shouldn't be? Don't know if it means anything or if it has a special name.

Perhaps I should try a different derainbowing method next and see how it fares, and if I can reduce the strength and if I see any other side effects.

UPDATE:
On bifrost, as usual, I have no idea what the arguments mean. They're too technical:

Bifrost(clip, clip altclip (Returns the first clip if none is specified, has to be yv12), float scenelumathresh (Default 1.5), int variation (Default 5), bool conservativemask (Default false), bool interlaced (Default true))
Altclip is the clip to return frames from when scenelumathresh is passed. This allows rainbows to be removed even in high motion scenes with other processing.
Scenelumathresh is how much every pixel in the frame/field may change on average and still be processed.
Variation determines how much the chroma must change for it to be considered a rainbow. Increasing it decreases false detection.
Conservativemask determines if the area around pixels that are considered rainbows also are processed. Can reduce miscoloring a little when there's both movement and rainbows. Interlaced changes between field based and frame based decisions for scenelumathresh.

I shall try as in your example and see what it does.
More on bifrost. Following script does not work:

import("bifrost.avs")

input = DirectShowSource("...")
Bifrost(interlaced=false)
return input

Invalid arguments to function bifrost.

Investigating DeCross. I have been able to get it to work. Now to see if I can notice any rainbows or anything out of the regular. Any "artefacts..."
Initial tests... FAILED. At default settings, it was unable to remove a rainbow.
After experimenting with DeCross, I'm willing to call it a failiure, as it does not seem to remove the rainbows properly. Even settings ThresholdY and Noise to 255 did not help.

Using FFT3DFilter: FFT3DFilter(sigma=x,plane=3,bw=32,bh=32,bt=3,ow=16,oh=16,interlaced=false) - "I don't know what x is." is the error I get.
Btw, how do you disable/comment out code in avisynth? Is it possible? Like // in C++.

[Chainmax]

I explained how to use DeCross and BiFrost, if they don't work it could be due to many different factors. As for FFT3DFilter, like I said you have to substitute x for a value between 3 and 6.

[Eretria-chan]

As for FFT3DFilter, like I said you have to substitute x for a value between 3 and 6.
Huh? I didn't notice that. DeCross did indeed work, but it was not strong enough to get rid of the rainbows. And Bifrost... Well, the imported avisynth script only loads the plugin, bifrost.dll.

UPDATE: I changed 'x' to 3 for fft3dfilter and I got invalid arguments :/

[foxyshadis]

Update fft3dfilter then.

Bifrost is kind of tricky to know when to use, but it's basically a post-processor for ssiq, applying its effects to non-edges. If you follow the type 2 example it should help you out, since your video is progressive, although I'm happier with fft3d and smartssiq.

[Eretria-chan]

I believe I have version 1.6, which I got from http://www.avisynth.org/warpenterprises/
I'll do a search on google and on the forum to see if I can find anything newer. In the meantime, if anyone knows where to get a newer version...

[Wilbert]

http://avisynth.org.ru/fizick.html

[Eretria-chan]

Thank you.
Not it says it's missing FFTW3.dll. Ah, all these dlls. No such file in the package. Although the source does exist. Dll hunting again, then. Let's see what I can dig out.
UPDATE: Okay, found it. Now let's see if it works.
UPDATE2: FFT3DFilter gives massive lag, akin to SmartSSIQ. No max cpu usage. I'll try to get the GPU version working!
UPDATE3: WOW, the GPU version runs without any speed hits!
UPDATE4: I was able to remove rainbows with a sigma of 100. Is this a good idea?
UPDATE5: It seems it isn't. It degrades the quality of the movie overall. FFT3DFilter just isn't a good filter for removing such strong rainbows, huh? Or is it? What are the side effects? All I can see is that it seems a little darker than without using it.

[tsp]

Eretria-chan: could you post a screenshot (using http://www.imageshack.us/ or something else) before filtering.

[Eretria-chan]

All screenshots are in 1920x1200, the native resolution I'm using. ImageShack will automatically scale them down to fit your screen size, though:

No filtering at all:
http://img63.imageshack.us/img63/8233/1920nofiltering2ut.th.jpg (http://img63.imageshack.us/my.php?image=1920nofiltering2ut.jpg)
Filtering with FFT3DGPU:
http://img516.imageshack.us/img516/7161/1920withfft3d6bq.th.jpg (http://img516.imageshack.us/my.php?image=1920withfft3d6bq.jpg)
Filtering with SmoothUV:
http://img516.imageshack.us/img516/8638/1920withsmoothuv1lt.th.jpg (http://img516.imageshack.us/my.php?image=1920withsmoothuv1lt.jpg)

They do look very noisy, yes? Looking for a good denoiser. Actually, I think I found one, but I haven't gotten it to work yet.
These screenshots only use rainbow removers.
The script is...

#LoadPlugin("fft3dgpu.dll")
#import("RemoveNoiseMC.avs")

#DirectShowSource("...")
#FFT3DGPU(sigma=90,plane=3,bw=32,bh=32,bt=3,ow=16,oh=16,interlaced=false)

import("SmoothUV.avs")
input = DirectShowSource("...")
return SmoothUV(input, 5, 255, false)


...And I think that's it. The rainbow removers have high enough settings to remove all the pesky rainbows in the source.

[tsp]

try
FFT3DGPU(sigma2=90,sigma3=90,sigma=8,sigma4=0,plane=3,bw=64,bh=64,bt=3,ow=16,oh=16)
and see if it is better. It should preserve the color intensity better. You can try lowering bt to 1 also

[Eretria-chan]

Gives me invalid arguments.

[tsp]

get the latest version from http://www.avisynth.org/tsp/

[Eretria-chan]

I have fft3dgpu 0.7, which is also the latest shown at that page. Perhaps those arguments only work with the non-gpu version. But it is too slow to use when watching the videos.

[tsp]

do you use

DirectShowSource("...")
FFT3DGPU(sigma2=90,sigma3=90,sigma=8,sigma4=0,plane=3,bw=64,bh=64,bt=3,ow=16,oh=16)
return last

?

[Eretria-chan]

Even this...

LoadPlugin("fft3dgpu.dll")
#import("RemoveNoiseMC.avs")

import("SmoothUV.avs")
input = DirectShowSource("...")
#FFT3DGPU(sigma=90,plane=3,bw=32,bh=32,bt=3,ow=16,oh=16,interlaced=false)
#SmoothUV(input, 5, 255, false)
FFT3DGPU(sigma2=90,sigma3=90,sigma=8,sigma4=0,plane=3,bw=64,bh=64,bt=1,ow=16,oh=16)
return last

...does not work. I get invalid arguments to FFT3DGPU at line 8.

EDIT:
Got it to work by modifying the script a little:

LoadPlugin("fft3dgpu.dll")
#import("RemoveNoiseMC.avs")

#import("SmoothUV.avs")
DirectShowSource("...")
#FFT3DGPU(sigma=90,plane=3,bw=32,bh=32,bt=3,ow=16,oh=16,interlaced=false)
#SmoothUV(input, 5, 255, false)
FFT3DGPU(sigma2=90,sigma3=90,sigma=8,sigma4=0,plane=3,bw=64,bh=64,bt=1,ow=16,oh=16)

New result:
http://img76.imageshack.us/img76/4384/newfft3dgpu5xo.th.jpg (http://img76.imageshack.us/my.php?image=newfft3dgpu5xo.jpg)

Old result:
No filtering at all:
http://img63.imageshack.us/img63/8233/1920nofiltering2ut.th.jpg (http://img63.imageshack.us/my.php?image=1920nofiltering2ut.jpg)
Filtering with FFT3DGPU:
http://img516.imageshack.us/img516/7161/1920withfft3d6bq.th.jpg (http://img516.imageshack.us/my.php?image=1920withfft3d6bq.jpg)
Filtering with SmoothUV:
http://img516.imageshack.us/img516/8638/1920withsmoothuv1lt.th.jpg (http://img516.imageshack.us/my.php?image=1920withsmoothuv1lt.jpg)

So in conclusion, I do not think it was much better. It still does a bad job. It's smearing color where it shouldn't be.

[tsp]

strange not exactly what I get using you screenshot
with this script:

s=ImageSource("unfiltered.jpg").converttoyv12()
s.FFT3DGPU(sigma2=40,sigma3=40,sigma=2,sigma4=0,plane=3,bw=64,bh=64,bt=1,ow=16,oh=16)

I get this
http://img503.imageshack.us/img503/4898/fft3dgpu6hm.th.jpg (http://img503.imageshack.us/my.php?image=fft3dgpu6hm.jpg)
the unfiltered result looks like this
http://img54.imageshack.us/img54/764/without3qo.th.jpg (http://img54.imageshack.us/my.php?image=without3qo.jpg)
the blocking artifacts comes from the colorspace conversion but anyway the rainbow is gone while the colorsmear is less than on your screenshot. I use bt=1 instead of bt=3. Anyway what graphics card do you use?

[Eretria-chan]

It doesn't matter whether I use bt 1 or 3, the result seems the same. A little colorsmear.
I have a GeForce 7800 GT, store overclocked, core: 470, memory: 1100 I believe.

[tsp]

did you try using the values in my script above? Else could you try uploading a 10 frame clip to ftp://tempclips%40avisynth%2Eorg:QfJY(86m@avisynth.org/
so I can try it on the actual clip instead of just a screenshot.

[Eretria-chan]

Oh. You've changed parameters. I'll try it out right away and get back to you with the results.

[Eretria-chan]

Still a little smeaing, but better than before:
http://img220.imageshack.us/img220/7585/colorsmear5lu.th.jpg (http://img220.imageshack.us/my.php?image=colorsmear5lu.jpg)
Will try to upload a clip.

UPDATE: I uploaded a 6 sec clip named Clip.avi. It's ca 800 kb.

Another update. I've found another side effect of the filter. Look at the people in the background at the picture here. They're losing color. This is the filter's fault as it isn't present in the original.
http://img155.imageshack.us/img155/2876/temp0hj.th.jpg (http://img155.imageshack.us/my.php?image=temp0hj.jpg)

[foxyshadis]

A realtime rainbow remover on HD+ material? Or are you just resizing to see the effect? (btw, why don't you mention the realtime need if it's important? If you're not watching it, but encoding, why does speed matter?) The only way you'll get that working is heavy smoothing without a lot of fallbacks. I'd just use fft3dgpu, lowering the sigma and possibly the raising the others until it works in tradeoff.

Another option is degrainmedian, but I don't think that'll be much better than DeCross.

Hey, you have such heavy dot crawl, why not try TComb? That works on dot crawl & rainbowing, rather well, but I have no idea what the speed is.

[Eretria-chan]

This isn't HD material. The actual resolution is 576 x 432, but my monitor's native resolution is 1920 x 1200, which is why the material is resized to that resolution when watching in full screen.

I'm not encoding this material, at least not yet. I'm watching it realtime. My processor is an Athlon64 X2 3800+ and my gpu I've already mentioned. The fps of this movie is 23 (or so it says in explorer's property pages), so any filter should be able to attain 23 fps or more, preferbly.

The bad thing about such a high resolution is that you see all these artefacts very well, as you can see. But the biggest problem was the rainbows which I was unsure how to remove. I'm looking at RemoveNoiseMC (http://forum.doom9.org/showthread.php?t=110078&page=3) for denoising the movie. Of course, I'm open to any other suggestions for making the source look better.

However, I need to know usage and good settings if I'm to use them as I simply do not know enough to make use of them without guidelines.

[tsp]

eretria-chan: You could have said that before. What program do you use to view it? If you captured it yourself you may want to increase the bitrate next time. It is not pretty when resized to 1920x1200. I experimented with you clip and this is the best I can come up with until now:
FFT3DGPU(sigma2=40,sigma3=2,sigma=90,sigma4=2,plane=3,bw=8,bh=8,bt=1,ow=4,oh=4)

[foxyshadis]

(Looking at the avi) The problem is that your source isn't lossless rainbowed. Dot Crawl and Rainbowing are very specific artifacts, while your source has trashed it by smearing the colors and dot crawl together. Drat. TComb is useless thus, and others aren't terribly useful, compared to a real composite capture.

I bet the sigma2=90 was just for testing, since it's pretty guaranteed to give you a greyed out pic. I got good results with

FFT3DGPU(sigma=2,sigma2=5,sigma3=3,sigma4=1,plane=3,bw=64,bh=64,bt=3,ow=16,oh=16)

You still have a bit of chroma bleeding and a bit of rainbow but both are very much reduced from each extreme.

You'll still need to use something for the luma, plane=4 (apply to all planes) is decent but kind of blurry, so a second real luma filter will probably add some more slowdown.

Edit: To be entirely honest, this is overkill for watching it in realtime; you'll never get rid of the big green and red blotches because of the way the codec smeared them and you'd barely be able to see real rainbowing. If you intend to restore this, asking for realtime is silly when the only way to get something decent out of the video would likely be .5 fps with quite a few heavy filters. If not then why re-encode such a terrible thing in the first place?

[Eretria-chan]

Well... I think I get a pretty good picture with SmoothUV. Nevertheless, I shall try those parameters. Although, are there any filter for removing this "bleeding" or smearing effect?

[Chainmax]

scharfis_brain once recommended me to use TemporalSoften(2,0,50,mode=2) on a DV capture from a 3rd generation VHS copy. Try it out if you can.

[Eretria-chan]

What does it do? Should I use it to replace the rainbow removers? Or use both?

[Chainmax]

It should filter the chroma quite aggressively but still look ok. Try using it instead of the rainbow removers and if it isn't enough then add one of them.

[Eretria-chan]

It seems lovely. But often as the scene changes, you notice a small color loss. Not that big a deal, but is it possible to counter that effect? Color loss sometimes happens.

[foxyshadis]

http://www.avisynth.org/mediawiki/wiki/Soften
"Added scenechange=n parameter: Using this parameter will avoid blending across scene changes. 'n' defines the maximum average pixel change between frames. Good values for 'n' are between 5 and 30. Requires ISSE."
Play with that value.

[Eretria-chan]

TemporalSoften seems to work wonders. I've seen no rainbows as of yet. The scene problem still prevails even with the extra parameter, but it is a small price to pay if it isn't fixable.
No color bleed :D

[Chainmax]

Have you tried several different scenechange values? In any case, remember that TemporalSoften is a last resort solution. For regular scenarios, DeCross, BiFrost and FFT3DFilter should be used instead.

[Eretria-chan]

Everything from 5, 30 to 255 and 999. It doesn't eleminate, at least, all of the scene change problems.
I feel like I've learned a good deal about avisynth filters now. A good start.