GuavaComb for rainbows or...

[NightMare]

i'm encoding escaflowne right now and i noteced some rainbows in the source and i'm thinkin of using GuavaComb to get rid of them should i use it or is there somthin better

thanx

[qwerpoi]

For an up-to-date discussion on removing rainbows, check out this thread here:

http://forum.doom9.org/showthread.php?threadid=62873

Somewhere on the second page of the thread you will find a script sh0dan came up with to remove rainbows, reprinted here:



org = last
org_u = utoy(org)
org_v = vtoy(org)

msharpen(threshold = 10, mask=true)
reduceby2()
greyscale()
uv = blur(1.5).levels(0,2.0,255,0,255).blur(1.5)

filtered_u = org_u.blur(1.5).blur(1.5).blur(1.5).temporalsoften(2,255,0,3,2)
filtered_v = org_v.blur(1.5).blur(1.5).blur(1.5).temporalsoften(2,255,0,3,2)

u_final = MaskedMerge(org_u, filtered_u, uv)
v_final = MaskedMerge(org_v, filtered_v, uv)

ytouv(u_final, v_final, org)

I found it to be very effective in removing rainbows, read the thread to see how it works. You could also try the suggestions on the first page of the thread if you want to experiment, there's a lot there to play around with.

[NightMare]

GuavaComb failed misribly even on the highest setings it couldent remove the rainbow using cnr2 right now its much better

thanx for the script but i'm lookin for a dll filter but thanx anyway it really helps :D

[qwerpoi]

Glad you found something that you like. As far as using dlls go, you could also try using MipSmooth (see my post on the first page of the thread I mentioned for the settings). Also, scripts are just as convinient as dll filters if you make them into a function. For example, you can convert sh0dan's script into a function as follows:

EDIT: See sh0dan's post below for a working script.

Simply save the above code into a new text file and rename it to "function.avs". Now you can load it as if it were a dll filter using the Import() command, ie,

Import("C:\function.avs")
Avisource("C:\something.avi")
DeRainbow()

If you want to use the function like an autoloading filter, name the text file to "function.avsi" instead and put it in the same directory as your autoloading plugins- now you don't need to use the Import() command. Keep in mind that sh0dan's script requires you to have msharpen as well as an updated version of Masktools in order to work.

Functions are one of the more fun things you can do with avisynth, although one might argue it requires a bit of programming background to do some of the more complicated things. If you don't know how to program, I think that learning AviSynth functions is a simple and great way to get started, since it does not have a lot of the difficulties involved in learning a language like c++ from scratch. Check out the shared functions (http://www.avisynth.org/index.php?page=ShareFunctions) on the AviSynth homepage to see what some other people are doing, and try some functions out for yourself in your free time, it will really help you get the most out of AviSynth.

[NightMare]

Originally posted by qwerpoi
Glad you found something that you like. As far as using dlls go, you could also try using MipSmooth (see my post on the first page of the thread I mentioned for the settings). Also, scripts are just as convinient as dll filters if you make them into a function. For example, you can convert sh0dan's script into a function as follows:

function DeRainbow(clip org)
{
org_u = utoy(org)
org_v = vtoy(org)

msharpen(threshold = 10, mask=true)
reduceby2()
greyscale()
uv = blur(1.5).levels(0,2.0,255,0,255).blur(1.5)

filtered_u = org_u.blur(1.5).blur(1.5).blur(1.5).temporalsoften(2,255,0,3,2)
filtered_v = org_v.blur(1.5).blur(1.5).blur(1.5).temporalsoften(2,255,0,3,2)

u_final = MaskedMerge(org_u, filtered_u, uv)
v_final = MaskedMerge(org_v, filtered_v, uv)

return ytouv(u_final, v_final, org)
}

Simply save the above code into a new text file and rename it to "function.avs". Now you can load it as if it were a dll filter using the Import() command, ie,

Import("C:\function.avs")
Avisource("C:\something.avi")
DeRainbow()

If you want to use the function like an autoloading filter, name the text file to "function.avsi" instead and put it in the same directory as your autoloading plugins- now you don't need to use the Import() command. Keep in mind that sh0dan's script requires you to have msharpen as well as an updated version of Masktools in order to work.

Functions are one of the more fun things you can do with avisynth, although one might argue it requires a bit of programming background to do some of the more complicated things. If you don't know how to program, I think that learning AviSynth functions is a simple and great way to get started, since it does not have a lot of the difficulties involved in learning a language like c++ from scratch. Check out the shared functions (http://www.avisynth.org/index.php?page=ShareFunctions) on the AviSynth homepage to see what some other people are doing, and try some functions out for yourself in your free time, it will really help you get the most out of AviSynth.

allready did that named it antirainbow

i really dont know much about advanced scriptin i wish i could learn it an maybe some day i will write somthin like this

thanx alot u helped me alot :D

[Wilbert]

GuavaComb failed misribly even on the highest setings it couldent remove the rainbow using cnr2 right now its much better
If that's the case you are not really experiencing rainbows (in the sense of GuavaComb). Have a look at the analog capture guide (http://www.doom9.org/capture/rainbows.html) for the difference between the two. Maybe you can post a screenshot?

[NightMare]

Originally posted by Wilbert
If that's the case you are not really experiencing rainbows (in the sense of GuavaComb). Have a look at the analog capture guide (http://www.doom9.org/capture/rainbows.html) for the difference between the two. Maybe you can post a screenshot?

i used this
GuavaComb(Mode="NTSC",Recall=85,MaxVariation=100,Activation=70)

[Wilbert]

Could you upload a screen shot of the original clip somewhere (or the clip itself)?

[NightMare]

Originally posted by Wilbert
Could you upload a screen shot of the original clip somewhere (or the clip itself)?

http://www.geocities.com/nightmare30002003/01.jpg

[mf]

Originally posted by NightMare
http://www.geocities.com/nightmare30002003/01.jpg
Those certainly look like rainbows. Are you sure you're using guavacomb BEFORE any IVTC or deinterlacing? If the source is already deinterlaced, use progressive rainbow cleaners like mfRainbow (http://www.avisynth.org/index.php?page=mfRainbow).

(No, I wasn't referring to my own creation to advertise, but because it was the one I could find easily and of which I know it works :D.)

[Wilbert]

Btw, GuavaComb only removes Rainbows in static scenes. Although I don't know how your clip goes, I guess the person moves?

[MasterYoshidino]

you can also try this setting for rainbows in static scenes (works fine for me, not strong enough for temporal errors from minor patterns, but works fairly well with static rainbow, line noise, and that hideous dot crawl)

GuavaComb(mode="NTSC",recall=85,maxvariation=35,activation=30)
watch out for patterns...

and did you give "smoothuv" a try? that works spatially. does not work as well as smoothiq but its way faster.

I looked at your "problem.avi" and my guavacomb setting should work pretty well to get rid of that line noise. tweak maxvariation if too much line noise comes through.

[NightMare]

Originally posted by mf
Those certainly look like rainbows. Are you sure you're using guavacomb BEFORE any IVTC or deinterlacing? If the source is already deinterlaced, use progressive rainbow cleaners like mfRainbow (http://www.avisynth.org/index.php?page=mfRainbow).

(No, I wasn't referring to my own creation to advertise, but because it was the one I could find easily and of which I know it works :D.)

i used GC befor ivtc
and i used the script at the start of the forum it removed it completly

[NightMare]

Originally posted by MasterYoshidino
and did you give "smoothuv" a try? that works spatially. does not work as well as smoothiq but its way faster.

I looked at your "problem.avi" and my guavacomb setting should work pretty well to get rid of that line noise. tweak maxvariation if too much line noise comes through.

smoothuv where can i find it i will give it a try
and if u can help me with my "problem" i will be very greatful

[MasterYoshidino]

i remember there was a thread one time... where is it now...
a/w here is the dll for avisynth2.5.

http://members.cox.net/hentai/SmoothUV.rar

syntax is (radius,strength,interlaced boolean)
example SmoothUV(5,16,false)
accepts YV12 data only.

hits subtitle and credit rainbows pretty easily, some rainbows it only hits at too high a strength and ends up cross coloring / color bleeding. if that happens, stuck with slow as can be smoothiq.

[High Speed Dubb]

About Shodan’s chroma smoothing script — It will work substantially better if it only smooths vertically within a field. What you really want for this purpose with NTSC video is a 1:2:1 FIR filter. (In other words, weight 1 for the line above, 2 for the current line, and 1 for the line below.) For PAL you would want (this gets icky) 1:0:2:0:1.

For Guava stuff, I wouldn’t raise MaxVariation all the way to 100 — That’ll show a lot of artifacts. For a more permissive setting, I’d reduce Recall and possibly Activation, with something like
GuavaComb(Mode="NTSC", Recall=70, Maxvariation=50, Activation=40)

But the best MaxVariation does depend on your input card. And like Wilbert mentioned, regardless of its settings, Guava will only work on completely static parts of a picture.

[jarthel]

I followed what was said below. I have the following lines in my .avs file

-----
LoadPlugin("D:\downloads\windows\utilities\divx\avisynth\plugins25\MPEG2Dec3.dll")
LoadPlugin("D:\downloads\windows\utilities\divx\avisynth\plugins25\undot.dll")
LoadPlugin("D:\downloads\windows\utilities\divx\avisynth\plugins25\Decomb.dll")
LoadPlugin("D:\downloads\windows\utilities\divx\avisynth\plugins25\STMedianFilter.dll")
LoadPlugin("D:\downloads\windows\utilities\divx\avisynth\plugins25\awarpsharp.dll")
LoadPlugin("D:\downloads\windows\utilities\divx\avisynth\plugins25\masktools.dll")
LoadPlugin("D:\downloads\windows\utilities\divx\avisynth\plugins25\msharpen.dll")
Import("D:\downloads\windows\utilities\divx\avisynth\plugins25\rainbow.avsi")

mpeg2source("c:\dvdrips\escaflowne18\escaflowne18.d2v",cpu=6,iPP=true)
crop(12,0,696,480)
DeRainbow()
Telecide(Post=true,blend=true)
Decimate(cycle=5)
awarpsharp(20,2,0.5,2)
lanczosresize(640, 480)
-----------------

Every time I load the avs file, it says msharpen has invalid arguments. I am using the latest one from Donald Graft's website.

thanks for the reply.

Originally posted by qwerpoi
Glad you found something that you like. As far as using dlls go, you could also try using MipSmooth (see my post on the first page of the thread I mentioned for the settings). Also, scripts are just as convinient as dll filters if you make them into a function. For example, you can convert sh0dan's script into a function as follows:

function DeRainbow(clip org)
{
org_u = utoy(org)
org_v = vtoy(org)

msharpen(threshold = 10, mask=true)
reduceby2()
greyscale()
uv = blur(1.5).levels(0,2.0,255,0,255).blur(1.5)

filtered_u = org_u.blur(1.5).blur(1.5).blur(1.5).temporalsoften(2,255,0,3,2)
filtered_v = org_v.blur(1.5).blur(1.5).blur(1.5).temporalsoften(2,255,0,3,2)

u_final = MaskedMerge(org_u, filtered_u, uv)
v_final = MaskedMerge(org_v, filtered_v, uv)

return ytouv(u_final, v_final, org)
}

Simply save the above code into a new text file and rename it to "function.avs". Now you can load it as if it were a dll filter using the Import() command, ie,

Import("C:\function.avs")
Avisource("C:\something.avi")
DeRainbow()

If you want to use the function like an autoloading filter, name the text file to "function.avsi" instead and put it in the same directory as your autoloading plugins- now you don't need to use the Import() command. Keep in mind that sh0dan's script requires you to have msharpen as well as an updated version of Masktools in order to work.

Functions are one of the more fun things you can do with avisynth, although one might argue it requires a bit of programming background to do some of the more complicated things. If you don't know how to program, I think that learning AviSynth functions is a simple and great way to get started, since it does not have a lot of the difficulties involved in learning a language like c++ from scratch. Check out the shared functions (http://www.avisynth.org/index.php?page=ShareFunctions) on the AviSynth homepage to see what some other people are doing, and try some functions out for yourself in your free time, it will really help you get the most out of AviSynth.

[sh0dan]

The function above did not work. I fixed it and added a YUY2 version (much slower though), but with no chroma downsampling. I tweaked it a bit for best results on my current material. I also changed the blurs to MipSmooth.


function DeRainbow(clip org, int "thresh")
{
assert(org.isYV12(),"DeRainbow() requires YV12 input!")
thresh = default(thresh, 10)

org_u = utoy(org)
org_v = vtoy(org)

msharpen(org, threshold = thresh, mask=true)
reduceby2()
greyscale()
uv = blur(1.5).levels(0,2.0,255,0,255, coring=false).blur(1.5).blur(1.5).levels(50,2.0,255,0,255, coring=false)

filtered_u = org_u.mipsmooth(spatial=255, temporal=255, scenechange=3, show=false, method="strong", scalefactor=0.5)
filtered_v = org_v.mipsmooth(spatial=255, temporal=255, scenechange=3, show=false, method="strong", scalefactor=0.5)

u_final = MaskedMerge(org_u, filtered_u, uv)
v_final = MaskedMerge(org_v, filtered_v, uv)

return ytouv(u_final, v_final, org)
}


function DeRainbowYUY2(clip org, int "thresh")
{
assert(org.isyuy2(),"DeRainbowYUY2() requires YUY2 input!")
thresh = default(thresh, 10)

org_yv12 = org.converttoyv12()
org_u = utoy(org).converttoyv12()
org_v = vtoy(org).converttoyv12()

msharpen(org_yv12, threshold = thresh, mask=true)
bilinearresize(last.width/2, last.height)
greyscale()
uv = blur(1.5).levels(0,2.0,255,0,255, coring=false).blur(1.5).blur(1.5).levels(50,2.0,255,0,255, coring=false)

filtered_u = org_u.mipsmooth(spatial=255, temporal=255, scenechange=3, show=false, method="superstrong", scalefactor=0.5)
filtered_v = org_v.mipsmooth(spatial=255, temporal=255, scenechange=3, show=false, method="superstrong", scalefactor=0.5)

u_final = MaskedMerge(org_u, filtered_u, uv).converttoyuy2()
v_final = MaskedMerge(org_v, filtered_v, uv).converttoyuy2()

return ytouv(u_final, v_final, org)
}


Usage:

YV12: DeRainbow(int thresh)
YUY2: DeRainbowYUY2(int thresh)

Threshold is optional, and defaults to 10 (which is fairly strong).

Required plugins:
MipSmooth (http://cultact-server.novi.dk/kpo/avisynth/mipsmooth.html).
Masktools (http://cultact-server.novi.dk/kpo/avisynth/masktools-1.4.1.zip).
mSharpen (http://neuron2.net/msharpen/msharpen.html).

[NightMare]

thanx for the new script will try it soon

[Suikun]

mf hates me for sometimes using this, but you could try

BlindPP(cpu2="ooxxox", quant=31)

which will do some heavy chroma filtering and usually kills about all chroma noise including rainbows, but might lead to "glowing edges".
BlindPP is part of MarcFD's MPEG2Dec3.
Oh, and don't try this with non-anime stuff...

[jarthel]

/me highly suggest smoothuv for rainbows. I've tried it the other day and it does work.

/me still has to try shodan's solution

[qwerpoi]

The function above did not work.

Yeah, sorry about posting that function, I wrote that reply when I was at work without access to Avisynth to double-check it, my bad. Thanks to sh0dan for writing a working script (as well as for a new build of Avisynth :) ).

[Didée]

Somehow I never managed to get some usable improvement out of GuavaComb on rainbowing PAL material. (If someone could tell me how -- PAL!)

Just for fun, yesterday I took a scene with "maximum rainbowing" (showing *saturated* shades of red and green, wandering), and put it through "TemporalSoften(2,0,255)".
Ooops! - just perfect on the static scenes!
Now this would need a little script, with some nifty conditionals and static detection ... I'll put it on my list.

- Didée

[qwerpoi]

Be sure to try out MipSmooth for a similar effect, since it has the conditional stuff built in. These are the settings I use (might be out of date, I haven't tried the more recent versions):

MipSmooth(downsizer="lanczos",upsizer="lanczos",temporal_chroma=225,spatial=0,temporal=0,spatial_chroma=50,scenechange=1)

I'm guessing it will be slower than temporalsoften, but should provide similar results.

[jarthel]

smoothuv with default parameters. It works. No need for tweaking!

[Didée]

Originally posted by jarthel
smoothuv with default parameters. It works. No need for tweaking! Yes, this worked for me, too, for quite some time now ;)
BUT:

1. Only in YV12. Applying on YUY2 material seems to have no effect.(?)
2. It works spatial only :( - see below.


Originally posted by jarthel
Be sure to try out MipSmooth for a similar effect, since it has the conditional stuff built in.
Frankly, I didn't even try MipSmooth for de-rainbowing yet ... but consider the following:

The TemporalSoften-experiment only worked with a temporal radius of "2" - perfect result. With a temp. radius of "1", the effect was pretty weak, hardly worth anything.
Now - since MipSmooth only considers the frames n-1 & n+1, you see my concern. It certainly will do the job quite good though, because of the combination with spatial smoothing.

My problem with spatial chroma smoothing is, I deal with some sources that, if de-rainbowed by strong spatial chroma smoothing, are hurt *very noticeable* in overall impression. For that reason, spatial handling is out of question in this case, and I started looking into that temporal-only approach.
But, all of that is actually low priority for me - other problems are calling ...

- Didée

[jarthel]

Originally posted by Didée
Yes, this worked for me, too, for quite some time now ;)
BUT:

1. Only in YV12. Applying on YUY2 material seems to have no effect.(?)
2. It works spatial only :( - see below.
- Didée

my suggestion was for the thread starter. :)

[Chainmax]

Originally posted by sh0dan:
The function above did not work. I fixed it and added a YUY2 version (much slower though), but with no chroma downsampling. I tweaked it a bit for best results on my current material. I also changed the blurs to MipSmooth.


Should this version work better than the previous one on YV12 material? And on YUY2 material, should this work better than using converttoYV12 and then the previous version?

[sh0dan]

I've tested it on some Cowboy Bebop anime that had a great amount of rainbowing, and I adjusted it to remove it, while still retaining the biggest amount of chroma detail. Hard job I admit, but for this material it worked nicely.


PS. Extremely great anime btw. get at yout local dealer ASAP. :)

[NightMare]

Originally posted by sh0dan
I've tested it on some Cowboy Bebop anime that had a great amount of rainbowing, and I adjusted it to remove it, while still retaining the biggest amount of chroma detail. Hard job I admit, but for this material it worked nicely.


PS. Extremely great anime btw. get at yout local dealer ASAP. :)

great news

i finished encoding escaflowne and i'm going to encode cowboy bebop next

thanx alot

[ThunderGatti]

Originally posted by sh0dan
The function above did not work. I fixed it and added a YUY2 version (much slower though), but with no chroma downsampling. I tweaked it a bit for best results on my current material. I also changed the blurs to MipSmooth.


function DeRainbow(clip org, int "thresh")
{
assert(org.isYV12(),"DeRainbow() requires YV12 input!")
thresh = default(thresh, 10)

org_u = utoy(org)
org_v = vtoy(org)

msharpen(org, threshold = thresh, mask=true)
reduceby2()
greyscale()
uv = blur(1.5).levels(0,2.0,255,0,255, coring=false).blur(1.5).blur(1.5).levels(50,2.0,255,0,255, coring=false)

filtered_u = org_u.mipsmooth(spatial=255, temporal=255, scenechange=3, show=false, method="strong", scalefactor=0.5)
filtered_v = org_v.mipsmooth(spatial=255, temporal=255, scenechange=3, show=false, method="strong", scalefactor=0.5)

u_final = MaskedMerge(org_u, filtered_u, uv)
v_final = MaskedMerge(org_v, filtered_v, uv)

return ytouv(u_final, v_final, org)
}


function DeRainbowYUY2(clip org, int "thresh")
{
assert(org.isyuy2(),"DeRainbowYUY2() requires YUY2 input!")
thresh = default(thresh, 10)

org_yv12 = org.converttoyv12()
org_u = utoy(org).converttoyv12()
org_v = vtoy(org).converttoyv12()

msharpen(org_yv12, threshold = thresh, mask=true)
bilinearresize(last.width/2, last.height)
greyscale()
uv = blur(1.5).levels(0,2.0,255,0,255, coring=false).blur(1.5).blur(1.5).levels(50,2.0,255,0,255, coring=false)

filtered_u = org_u.mipsmooth(spatial=255, temporal=255, scenechange=3, show=false, method="superstrong", scalefactor=0.5)
filtered_v = org_v.mipsmooth(spatial=255, temporal=255, scenechange=3, show=false, method="superstrong", scalefactor=0.5)

u_final = MaskedMerge(org_u, filtered_u, uv).converttoyuy2()
v_final = MaskedMerge(org_v, filtered_v, uv).converttoyuy2()

return ytouv(u_final, v_final, org)
}


Usage:

YV12: DeRainbow(int thresh)
YUY2: DeRainbowYUY2(int thresh)

Threshold is optional, and defaults to 10 (which is fairly strong).

Required plugins:
MipSmooth (http://cultact-server.novi.dk/kpo/avisynth/mipsmooth.html).
Masktools (http://cultact-server.novi.dk/kpo/avisynth/masktools-1.4.1.zip).
mSharpen (http://neuron2.net/msharpen/msharpen.html).

Great script, and a big improvement over the one you originally introduced in the "80's anime LD capture" thread. Even at lower thresholds there seems to be alot less heavy desaturation of the colors around the afflicted areas of the image. The older version also created alot of ghosting, even with more conservative settings.

One thing I did notice though was that it has a tendency to desaturate the image a bit as a whole, with reds taking the brunt of the damage, even during sequences in the target video that aren't showing any of the rainbow artifacts the function is trying to eliminate. This seems to happen with both the YV12 and YUY2 flavors. Known issue that I missed or something that can be improved upon?

[Avisynth_challenged]

The function above did not work. I fixed it and added a YUY2 version (much slower though), but with no chroma downsampling. I tweaked it a bit for best results on my current material. I also changed the blurs to MipSmooth.


function DeRainbow(clip org, int "thresh")
{
assert(org.isYV12(),"DeRainbow() requires YV12 input!")
thresh = default(thresh, 10)

org_u = utoy(org)
org_v = vtoy(org)

msharpen(org, threshold = thresh, mask=true)
reduceby2()
greyscale()
uv = blur(1.5).levels(0,2.0,255,0,255, coring=false).blur(1.5).blur(1.5).levels(50,2.0,255,0,255, coring=false)

filtered_u = org_u.mipsmooth(spatial=255, temporal=255, scenechange=3, show=false, method="strong", scalefactor=0.5)
filtered_v = org_v.mipsmooth(spatial=255, temporal=255, scenechange=3, show=false, method="strong", scalefactor=0.5)

u_final = MaskedMerge(org_u, filtered_u, uv)
v_final = MaskedMerge(org_v, filtered_v, uv)

return ytouv(u_final, v_final, org)
}


function DeRainbowYUY2(clip org, int "thresh")
{
assert(org.isyuy2(),"DeRainbowYUY2() requires YUY2 input!")
thresh = default(thresh, 10)

org_yv12 = org.converttoyv12()
org_u = utoy(org).converttoyv12()
org_v = vtoy(org).converttoyv12()

msharpen(org_yv12, threshold = thresh, mask=true)
bilinearresize(last.width/2, last.height)
greyscale()
uv = blur(1.5).levels(0,2.0,255,0,255, coring=false).blur(1.5).blur(1.5).levels(50,2.0,255,0,255, coring=false)

filtered_u = org_u.mipsmooth(spatial=255, temporal=255, scenechange=3, show=false, method="superstrong", scalefactor=0.5)
filtered_v = org_v.mipsmooth(spatial=255, temporal=255, scenechange=3, show=false, method="superstrong", scalefactor=0.5)

u_final = MaskedMerge(org_u, filtered_u, uv).converttoyuy2()
v_final = MaskedMerge(org_v, filtered_v, uv).converttoyuy2()

return ytouv(u_final, v_final, org)
}


Usage:

YV12: DeRainbow(int thresh)
YUY2: DeRainbowYUY2(int thresh)

Threshold is optional, and defaults to 10 (which is fairly strong).

Required plugins:
MipSmooth (http://cultact-server.novi.dk/kpo/avisynth/mipsmooth.html).
Masktools (http://cultact-server.novi.dk/kpo/avisynth/masktools-1.4.1.zip).
mSharpen (http://neuron2.net/msharpen/msharpen.html).


Sorry to resurrect an old post, but I have a question on the above script.

Did some searching on the issue of whether or not this function can be used on progressive (non-interlaced) sources, and I couldn't find an answer. Most derainbowing filters require that the source be interlaced. Can anyone shed some light on this? Thanks :)