Derainbow Problem

[egrimisu]

How to fix that rainbowing , tryed derainbow() but creates some artifacts.

http://img687.imageshack.us/i/44288936.png/

thanks in advance

[neuron2]

Please fix your thread title ASAP to avoid a rule 9 strike.

[rkalwaitis]

http://www.doom9.org/index.html?/capture/chroma_artefacts.html Look here for some alternate ways to get rid of rainbows,

and here too. http://avisynth.org/mediawiki/External_filters

[MadRat]

If your video is NTSC and interlaced you can use the tcomb plugin (it requires a little adjustment but it's easy to do), otherwise use the ChubbyRain2 script. I've read using fft3dfilter on the chroma layer works the best but I haven't see specific instructions on how to remove rainbows that way.

[egrimisu]

Can't deal with it, tried all the filters, all of them none deald with it, and tcomb derainbow chubbyrain2 produced some ugly color bleeding (i believe this is the name, but please see the screenshots). Bifrost haven't produces any artifacts but haven't deald with the rainbows or the dowcrawl. i've postet a sample please take a look at it.
Thnaks again.

http://www.megaupload.com/?d=9OVUV973

artifact screenshots:
http://img511.imageshack.us/i/derainbow.png/
http://img156.imageshack.us/i/derainbow2.png/

[tengo6dedos]

fft3d line may be something like this

strength = 10
FFT3DFilter(plane=3, bt=3, sigma=3, sigma2=3, sigma3=strength, sigma4=3, bw=32, bh=32,ow=16,oh=16)

and at the end maybe a mergechroma(AddBorders(4, 0, 4, 0).awarpsharp(depth=20.0, thresh=0.75, blurlevel=2, cm=1).Crop(4,0,-4,0)), or a awarpsharp2 line in mergechroma to avoid addborders and crop, or something else for the extra bleeding fft3d may add

[Nightshiver]

What in the world? Your first screenshot is from Ducktails I presume, and the sample is from something totally different. Which source is your true one? As for the sample, there were very few that I saw and nothing that

DFMDerainbow() didn't fix. Make sure you remove the telecine first. You may even want to use SharpAAMCMod(), LSF and some saturation to help it.

animeivtc(mode=1,aa=4)
dfmderainbow()
LimitedSharpenFaster(ss_x=1.8, ss_y=1.8, smode=4, strength=180, overshoot=1, undershoot=1)
Tweak(cont=1.0,sat=1.2,bright=0, hue=0)
crop( 8, 2, -8, 0)
Spline36Resize(640,480)

As a side note, SharpAAMCMod is taken from inside AnimeIVTC, so you need it to make it function. aa=4 is SharpAAMCMod.

[egrimisu]

ducktales and anne are bowth my source, problem is the same but in ducktales i managed to remove some of it but creates the bleeding, but anne is totally killing me, nothing works.



What in the world? Your first screenshot is from Ducktails I presume, and the sample is from something totally different. Which source is your true one? As for the sample, there were very few that I saw and nothing that

DFMDerainbow() didn't fix. Make sure you remove the telecine first. You may even want to use SharpAAMCMod(), LSF and some saturation to help it.

animeivtc(mode=1,aa=4)
dfmderainbow()
LimitedSharpenFaster(ss_x=1.8, ss_y=1.8, smode=4, strength=180, overshoot=1, undershoot=1)
Tweak(cont=1.0,sat=1.2,bright=0, hue=0)
crop( 8, 2, -8, 0)
Spline36Resize(640,480)


I'm trying all the filters now and geting back with a feed.

[rkalwaitis]

Why don't you try to use a different denoiser, I like dfttest. It is a bit slow and see how your sample comes out.

[thewebchat]

SSIQ completely removes the rainbows on your ANNE clip. Applying aWarpSharp to the chroma channel kills some of that chroma bleed. You can also kill that interesting aliasing artifact with EEDI2 or nnedi2 (for the love of God, don't use "Sharp" "AA" "MC").

However, what I find curious is how the video keeps jumping from perfectly sharp to blurry and covered in rainbows.

[Nightshiver]

You know theweb, just what is it that you have so much against AnimeIVTC? Everything works just fine, and you complaining about it constantly doesn't help anything at all.

for the love of God, don't use "Sharp" "AA" "MC"

Not exactly the kindest thing to say, as it works just fine.

[rkalwaitis]

Afterlooking at your script, would it be possible animeivtc has the incorrect mode setting? How is your movie interlaced? Telecide? Hybrid, is it progressive?

I would check the mode you are using against your script animeivtc mode setting to ensure they are correct. How did you test your clip to see how it is interlaced and the fps.

I think this is the problem, maybe.

[thewebchat]

I'm pretty sure that if you took the time to look at the clip, you would see that it is standard 3/2 telecine. The rainbows and aliasing are totally unrelated to the telecine, and even AnimuIVTC can't possibly create rainbows.

[rkalwaitis]

Thanks thewebchat, did not see the download, just the png files, my mistake. Who said animeivtc caused rainbows? Not I.
K

[MadRat]

egrimisu, it looks like the Anne of Green Gables example isn't rainbowing but a deinterlacing problem, I'm not sure if anyone has said so yet. It looks to me like what's happened is there's been a scene change and one field (let's say the even lines of resolution) are from the previous scene and the other field (the odd lines of resolution) are from the next; this is called scene bleed. The best way to get rid of it is to use freezeframe() to copy the previous frame on top of it or the next frame on top of it.

[rkalwaitis]

MadRat,

I think you are right. I download the file and ran it through DGINDEX the results were Progressive and interlaced and Film 68% . Im not smart enough to make DGINDEX tell me just how many frames are progressive and then interlaced, but I will say watching it run, there was no real pattern. Sometimes it ran just progressive and a bit of interlaced and other times it bounced back and forth like it was cool. Mode 1 used above:

For all these cases, you must load a d2v created with "honor pulldown flags" (DGIndex) in avisynth and check frame by frame with vdub, vdubmod or avsp.
The identification of the type of your source if of the utmost importance, because a bad evaluation of the problem will lead to other ones, such as blends and jerkiness.
1 - DGIndex tells you "100% Film" and you see a 3:2 pattern.
2 - DGIndex tells you "100% Film" and you see progressive and interlaced frames with no specific pattern.
3 - DGIndex tells you "something else than 100% Film" and you see a 3:2 pattern.
4 - DGIndex tells you "100% Video (NTSC or PAL)" and you see progressive and interlaced frames with no specific pattern.
5 - DGIndex tells you "100% Video (NTSC)" and you see a 3:2 pattern + the credits are truly interlaced and/or progressive.
6 - DGIndex tells you "100% Video (NTSC or PAL)" and you see progressive and interlaced frames with no specific pattern + the credits are truly interlaced and/or progressive.
7 - DGIndex gives you a percentage of Film or Video from 50 to 95% and you see a 3:2 pattern + progressive sequences (not only the credits, but the whole frame is). DGIndex never gave a reading of 100% film so perhaps this is the mode to try first!!!
8 - DGIndex gives you a percentage of Film or Video from 50 to 100% and you see a 3:2 pattern + progressive sequences (not only the credits, but the whole frame is) + the credits are truly interlaced and/or progressive.

# Source type:

1 - Soft telecined, so your stream can easily be extracted at the film rate
2 - Double hard telecined encoded as progressive
3 - Hard telecined
4 - Double hard telecined
5 - Hard telecined + interlaced or progressive credits on top of it
6 - Double hard telecined + interlaced or progressive credits on top of it
7 - Hybrid
8 - Hybrid + interlaced or progressive credits on top of it

Maybe mode another mode instead of 1. Its worth playing with it.

I would test it quickly with the script they are using, but I get a cblend error.

DGDecode_mpeg2source("C:\Users\Kalwaitis\Desktop\Movies Done\Vts_01_1_cut.d2v", info=3)
ColorMatrix(hints=true, interlaced=true, threads=0)
tfm(order=1).tdecimate(mode=1,hybrid=1) **I think this may work. Looked fine on my side. Give it a whorl and tell me if you like it.

The documentation for animeivtc say it would be mode 5 or 6.

[egrimisu]

egrimisu, it looks like the Anne of Green Gables example isn't rainbowing but a deinterlacing problem, I'm not sure if anyone has said so yet. It looks to me like what's happened is there's been a scene change and one field (let's say the even lines of resolution) are from the previous scene and the other field (the odd lines of resolution) are from the next; this is called scene bleed. The best way to get rid of it is to use freezeframe() to copy the previous frame on top of it or the next frame on top of it.

MadRat,

I think you are right. I download the file and ran it through DGINDEX the results were Progressive and interlaced and Film 68% . Im not smart enough to make DGINDEX tell me just how many frames are progressive and then interlaced, but I will say watching it run, there was no real pattern. Sometimes it ran just progressive and a bit of interlaced and other times it bounced back and forth like it was cool. Mode 1 used above:

For all these cases, you must load a d2v created with "honor pulldown flags" (DGIndex) in avisynth and check frame by frame with vdub, vdubmod or avsp.
The identification of the type of your source if of the utmost importance, because a bad evaluation of the problem will lead to other ones, such as blends and jerkiness.
1 - DGIndex tells you "100% Film" and you see a 3:2 pattern.
2 - DGIndex tells you "100% Film" and you see progressive and interlaced frames with no specific pattern.
3 - DGIndex tells you "something else than 100% Film" and you see a 3:2 pattern.
4 - DGIndex tells you "100% Video (NTSC or PAL)" and you see progressive and interlaced frames with no specific pattern.
5 - DGIndex tells you "100% Video (NTSC)" and you see a 3:2 pattern + the credits are truly interlaced and/or progressive.
6 - DGIndex tells you "100% Video (NTSC or PAL)" and you see progressive and interlaced frames with no specific pattern + the credits are truly interlaced and/or progressive.
7 - DGIndex gives you a percentage of Film or Video from 50 to 95% and you see a 3:2 pattern + progressive sequences (not only the credits, but the whole frame is). DGIndex never gave a reading of 100% film so perhaps this is the mode to try first!!!
8 - DGIndex gives you a percentage of Film or Video from 50 to 100% and you see a 3:2 pattern + progressive sequences (not only the credits, but the whole frame is) + the credits are truly interlaced and/or progressive.

# Source type:

1 - Soft telecined, so your stream can easily be extracted at the film rate
2 - Double hard telecined encoded as progressive
3 - Hard telecined
4 - Double hard telecined
5 - Hard telecined + interlaced or progressive credits on top of it
6 - Double hard telecined + interlaced or progressive credits on top of it
7 - Hybrid
8 - Hybrid + interlaced or progressive credits on top of it

Maybe mode another mode instead of 1. Its worth playing with it.

I would test it quickly with the script they are using, but I get a cblend error.

DGDecode_mpeg2source("C:\Users\Kalwaitis\Desktop\Movies Done\Vts_01_1_cut.d2v", info=3)
ColorMatrix(hints=true, interlaced=true, threads=0)
tfm(order=1).tdecimate(mode=1,hybrid=1) **I think this may work. Looked fine on my side. Give it a whorl and tell me if you like it.

The documentation for animeivtc say it would be mode 5 or 6.


Thanks guys for the advice but i have runned anne with mode 5 and get it didn't find any progressive frames:
# timecode format v1
Assume 29.970030
# TDecimate v1.0.3 by tritical
# Mode 5 - Auto-generated mkv timecodes file
0,5591,23.976024
5592,5594,17.982018
5595,28826,23.976024
# vfr stats: 100.00% film 00.00% video
# vfr stats: 36035 - film 0 - video 36035 - total
# vfr stats: longest vid section - 0 frames
# vfr stats: # of detected vid sections - 0

this means that mode=1 is enough

For ducktales it seems that the best results i get with the next script
strength = 20
FFT3DFilter(plane=3, bt=3, sigma=3, sigma2=3, sigma3=strength, sigma4=3, bw=32, bh=32,ow=16,oh=16)

i little rainbow still remains but at least the image collors are unchanged and no bleeding seems to apear. SSIQ seems to be quite good but total ruins the colors and in ducktales produce bleeding, in anne the result is quite good but i think i'l stick to fft3dfilter there to.
I believe freezeframe need manual adjustment and i don't have the time for it.

For anne ssiq does a better job but is extremely slow.fft3dfilter does a good job.

Testing a full episode of anne and duck and getting back with feed.

[Blue_MiSfit]

Tweak the different sigma parameters in fft3dgpu. Sometimes chroma issues are more in the low frequency areas (the first sigma value)

~MiSfit

[Didée]

(the last sigma value)

hi-freq -------------------> low-freq
sigma -> sigma2 -> sigma3 -> sigma4

[thewebchat]

I'm p. sure that SSIQ is not slower than FFT3D, but I may be wrong. You can (try to) prevent general hue shifts and chroma bleeding by using an edge mask for SSIQ, since that's where the rainbows are most likely to occur.

[egrimisu]

I'm p. sure that SSIQ is not slower than FFT3D, but I may be wrong. You can (try to) prevent general hue shifts and chroma bleeding by using an edge mask for SSIQ, since that's where the rainbows are most likely to occur.

This is what i'm looking for, how do i do this?

[Nightshiver]

############################################################################
# This is a script which works with SSIQ. The basic problem with SSIQ is
# that it can alter the color on the entire picture, which is a very very
# bad thing.
# All we want to do is remove rainbows, while leaving everything else alone.
# This filter first applies SSIQ to the entire picture. Then it locates the
# edges.
# Finally, it layers ONLY the de-rainbowed edges onto the orig video.
#
#
# Filter Requirements: SSIQ()
############################################################################
#LoadPlugin("mt_masktools.dll")
#LoadPlugin("SSIQ.dll")

#input = DirectShowSource("...")
#SmartSSIQ(input, 1)

function SmartSSIQ(clip input, int "strength")
{
strength = default(strength, 1) #processing strength, 0-255

# apply rainbow remover to the entire video
derainbow = input.SSIQ(11, 300, false)

# create an edge mask from the rainbow video
lutexpr = "x .2 ^ 110 *"
# uncomment to use masktools 1
edgemask = input.edgemask(thY1=1, thY2=255, type="sobel").inflate().yv12lut(yexpr=lutexpr).GreyScale()
#edgemask = input.mt_edge(thY1=1, thY2=255, mode="sobel").mt_inflate().mt_lut(yexpr=lutexpr).GreyScale()

# apply ssiq only to masked area, using "strength"
result = overlay(input, derainbow, mode="blend", mask=edgemask, opacity=strength/255.0).Limiter()

return result
}

Taken from search. It may be what you are looking for.

[egrimisu]

Seems that smartssiq is the way to go for anne but needs some more tweaking that i'm not capable of making ssiq creates some bleeding, the bleeding color is always red and goes over the black lines, is there a way to configure it proper? This bleadin issue is not visible on my crt monitor but on my lcd is visible.

For ducktales, smartssiq doesn't derainbow at all so i try to be smart, probably got dumb and editet a litte the smartssiq and replaced the ssiq function with fft3d to derainbow only the edges but it seems that this lowers very the strength of fft3d. What shall i try to tweak? tried sigma=80 sigma2=80 sigma3=80, sigma4=80, this derainbows quite well but the bright colors are very affected lose the contrast :(

the modified scrfipt that i called smartfft3d, probably shall call it dumbfft3d cause 90% i dind't know what i realy did because i don't realy understand what i was doing

function SmartFFT(clip input, int "strength") {
strength = default(strength, 10) #processing strength, 0-255

# apply rainbow remover to the entire video
derainbow = input.FFT3DFilter(plane=3, bt=1, sigma=3, sigma2=3, sigma3=strength, sigma4=3, bw=32, bh=32,ow=16,oh=16)

# create an edge mask from the rainbow video
lutexpr = "x .2 ^ 110 *"
edgemask = input.mt_edge(thY1=1, thY2=255, mode="sobel").mt_inflate().mt_lut(yexpr=lutexpr).GreyScale()

# apply ssiq only to masked area, using "strength"
result = overlay(input, derainbow, mode="blend", mask=edgemask, opacity=strength/255.0).Limiter()

return result
}

if using .FFT3DFilter(plane=3, bt=1, sigma=3, sigma2=3, sigma3=strength, sigma4=3, bw=32, bh=32,ow=16,oh=16) result are great but bleeading apears in a lot of frames if used befor ivtc, the bleading disapears if used after ivtc but the does not remove the rainbows in 70% of the frames.

So, if you fot more tips i will like to hear them, thanks in advance.

[egrimisu]

In short, why any derainbow filter produce bleeding in some scenes? is there a way to get rid of it?

[Didée]

In short, why does applying a blur filter make the picture soft? :D

if using .FFT3DFilter(plane=3, bt=1, sigma=3, sigma2=3, sigma3=strength, sigma4=3, bw=32, bh=32,ow=16,oh=16) result are great but bleeading apears in a lot of frames if used befor ivtc, the bleading disapears if used after ivtc but the does not remove the rainbows in 70% of the frames.
First off, note that FFT3DFilter is a filter that can work pure spatially (with bt=1), or spatio-temporally (with bt>1). (Simplified. The frequency-domain filtering is more convoluted, but lets keep it simple here.)

With bt=1, fft3dfilter is [acting like] a spatial filter. Before IVTC, the source technically is interlaced (simplified, again). FFT3DFilter has that "interlaced=true/false" parameter - guess what its purpose is? Would you consider that using FFT3DFilter without interlaced=true on an "interlaced" source could be problematic? (For a blataneous example, try using "Blur(1)" before IVTC ... ah, obviously it's not a good idea.)

Then, pure spatial filtering is a rather weak attempt to attack rainbows, since the problem is of spatio-temporal nature.

The reason you found that FFT3D seems more effective when used before IVTC, and less effective when used after, is simply this: in the former case, it was acting spatio-temporal, albeit "falsely", because of the missing interlaced=true. In the latter case, it was working pure spatially, correctly, and less efficient for abovementioned reason.

For using FFT3D against rainbows, try this way: first do IVTC, then call FFT3DFilter like you have it now, but with additional temporal filtering: set bt=2/3/4/5. (five is slowest, but also strongest.)

There might be better ways, I'm hardly ever faced with rainbowing problems and have little experience with that. However, when seeing an obvious technical error like it was here, the error should be pointed out. Which is hereby done.