BIG Rainbowing...

[Frank62]

Dear people,
is there anything one could do against this macro-structured-rainbowing? We tried a lot but no luck so far. There is no better master than this one worldwide.
The bad deinterlacing someone caused long ago we could tame with good old placebo-QTGMC, but how to fix the rainbows?

https://we.tl/t-oYhW0QzXIs

Unfortunately the boy wears this shirt all the film long.

[mastrboy]

Most derainbow filters for avisynth is designed to work with telecined content pre IVTC unfortunately, I've only found 2 other decent filters that works outside of that scope.
1. ASTDRmc: http://avisynth.nl/index.php/ASTDR
2. vs-basicvsrpp with model 3,4 or 5 and just merge in the chroma plane, though this is a Vapoursynth machine learning based filter, so very slow and needs a decent GPU, RTX 3080 or faster: https://github.com/HolyWu/vs-basicvsrpp/blob/master/vsbasicvsrpp/__init__.py#L34

[Frank62]

Thanks for your answer!
For the second proposal: This will take some time to make it usable, which we don't have (as always). You wrote "just merge in the chroma plane". But: Even if this (am I wrong?) upscaler will eliminate most or all of the rainbowing this won't be enough - also luma is much affected by this monstrous rainbowing.
But I will do some experiments, thanks anyway.

Will also try ASTDR.

We tried the rather old DFMDerainbow ( http://avisynth.nl/index.php/DFMDeRainbow ) which we used often when there was really heavy rainbows. But in this case it fails.

[Selur]

I used Vapoursynth and applied RainbowSmooth a few times and it helps a lot:
https://i.ibb.co/WWpGznC/temp-Preview-Vapoursynth-File19-53-05-490-vpy-53.png (https://imgbb.com/)
# Imports
import vapoursynth as vs
import os
import sys
# getting Vapoursynth core
core = vs.core
# Import scripts folder
scriptPath = 'i:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# Loading Plugins
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/Support/TCanny.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/ColorFilter/Retinex/Retinex.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/DerainbowFilter/SmoothUV/libsmoothuv.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# Import scripts
import masked
import RainbowSmooth
# source: 'C:\Users\Selur\Desktop\Btest.mp4'
# current color space: YUV420P8, bit depth: 8, resolution: 230x408, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading C:\Users\Selur\Desktop\Btest.mp4 using LibavSMASHSource
clip = core.lsmas.LibavSMASHSource(source="C:/Users/Selur/Desktop/Btest.mp4")
# Setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=5)
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
clip = core.std.SetFrameProp(clip=clip, prop="_FieldBased", intval=0)
# rainbow removal using RainbowSmooth
clip = RainbowSmooth.RainbowSmooth(clip=clip, radius=7, mask="retinex_edgemask")
# rainbow removal using RainbowSmooth
clip = RainbowSmooth.RainbowSmooth(clip=clip, radius=7, mask="retinex_edgemask")
# rainbow removal using RainbowSmooth
clip = RainbowSmooth.RainbowSmooth(clip=clip, radius=7, mask="retinex_edgemask")
# rainbow removal using RainbowSmooth
clip = RainbowSmooth.RainbowSmooth(clip=clip, radius=7, mask="retinex_edgemask")
# rainbow removal using RainbowSmooth
clip = RainbowSmooth.RainbowSmooth(clip=clip, radius=7, mask="retinex_edgemask")
# rainbow removal using RainbowSmooth
clip = RainbowSmooth.RainbowSmooth(clip=clip, radius=7, mask="retinex_edgemask")
# rainbow removal using RainbowSmooth
clip = RainbowSmooth.RainbowSmooth(clip=clip, radius=7, mask="retinex_edgemask")
# rainbow removal using RainbowSmooth
clip = RainbowSmooth.RainbowSmooth(clip=clip, radius=7, mask="retinex_edgemask")
# adjusting output color from: YUV420P8 to YUV420P10 for x265Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, range_s="limited")
# set output frame rate to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()
quick x265 encode (https://drive.google.com/file/d/11h2X6kXp1ZMGmpbIgy2kxZgFK7f5TzDC/view?usp=share_link)

I guess using the Avisynth http://avisynth.nl/index.php/Rainbow_Smooth filter script should do the same. :)
The main point is to apply it a few times with retinex masking.

Cu Selur

[mastrboy]

You can use the luma from basicvsrpp filter of course, but be aware that even if it 100% fixes the rainbowing issue, it might introduce new issues, I found it's very scene dependent if the filter improves or worsens the image.

A workaround I usually use for that to achieve some kind of middle ground is to use "Merge" and control how much of the filter you want with the weight parameter: http://www.vapoursynth.com/doc/functions/video/merge.html

Edit: That is pretty impressive Selur, can't believe I haven't heard about RainbowSmooth before...

[Selur]

@mastrboy: I used to use the Avisynth filter ages and did use it a while a go through Vapoursynth when some Hybrid used asked how to tackle some private videos that looked similar to the ones from Frank62. :)
Avisynth: http://avisynth.nl/index.php/Rainbow_Smooth
Vapoursynth: https://github.com/dubhater/vapoursynth-smoothuv

[kedautinh12]

You can try ChubbyRain2_
https://github.com/Asd-g/AviSynthPlus-Scripts/blob/master/ChubbyRain2_.avsi

[Frank62]

Thanks for every post here! Great help!

@Selur Fantastic! Almost unbelievable, but this seems to be the solution. I will try to use this only for parts with this heavy rainbowing tomorrow.

[Frank62]

Works quite nice, even with the avisynth script. 8 times in a row - hard to believe, but gives an acceptable result that can be used to do some other tricks...

:thanks:

[StainlessS]

Does it change, frame to frame, if so then maybe try a Spotless(RadT=1 or 2) between calls to the de-rainbow-er.

EDIT: Maybe NOT between every call to de-rainbower, that would be maybe too much,
maybe only once in middle of all attempts to get rid of the 'pot of gold'.
(anyways, suck it and see if it does any good)

[anton_foy]

Maybe try: this? (https://forum.doom9.org/showpost.php?p=1976644&postcount=60)

[johnmeyer]

StainlessS suggested using his Spotless filter. I did just that and it worked great. Here is my post in the Videohelp forum where I showed the before/after and posted the script:

https://forum.videohelp.com/threads/398864-Need-help-on-Avisynth?highlight=rainbow#post2595644

[edit]You obviously don't need SRestore, shown at the beginning of the script. That was used to correct a different problem, unrelated to the rainbow.

[takla]

Maybe try: this? (https://forum.doom9.org/showpost.php?p=1976644&postcount=60)

Surprisingly, EZdenoise has almost no effect. I guess it is too good, and by that I mean it preserves too much detail :P

[Frank62]

Thanks for these new hints!
We/I used Mr. StainlessS' fabulous SpotlessS several times, and very carefully, maybe here it's worth a try. I'll see.
But meanwhile we have other problems with this source: Even placebo-QTGMC did not do the complete trick: Each time you see diagonal windows or car lines they still look awful. I'll load up a sequence tomorrow.
I tried a lot, incl. De-Bilinear/-cubic combinations before and after Separatefields with different parameters, a. s. o., found nothing to make it look acceptable.
This is the only master available - they sent it as "new scanned HD-source" - good joke - and unfortunately cannot find the original before they did the worst deinterlacing I've ever seen and even A.I.-upscaled the ugly result. Hard to believe, what you sometimes get from professionals.
Hope someone will find a better solution than I did so far tomorrow.

[takla]

If you have a Nvidia GPU, look into training your own ESRGAN model. You can get rid of any type of artefact with that, with very good results. Takes a lot of time to train and process though.

[StainlessS]

Does it change, frame to frame, if so then maybe try a Spotless(RadT=1 or 2) between calls to the de-rainbow-er.
Maybe with ThSAD/ThSAD2 about 400 [6*8*8=384, luma diff average of 6 for the 8x8 <64> SAD block], or a bit higher.
Also Chroma=FALSE a MUST.
With that low a ThSAD/ThSAD2, and Chroma = false, could probably make RadT=2 or a bit more without too much self created artefacts.
(Maybe ThSAD2 = about ThSAD - 64 : EDIT: Or maybe ThSAD2 = ThSAD - (RadT*64))

[kedautinh12]

But meanwhile we have other problems with this source: Even placebo-QTGMC did not do the complete trick: Each time you see diagonal windows or car lines they still look awful. I'll load up a sequence tomorrow.


you can try:
ex_bob(nnedi3=true).SelectEven()
https://github.com/Dogway/Avisynth-Scripts/blob/b4ad8bc47aaef8052f370864f984116217881850/MIX%20mods/QTGMC%2B.avsi#L1486

[Frank62]

So here is the real problem:
https://we.tl/t-87bthYSVl5

Thanks again, but I fear exbob won't be enough for that. The clue should be to somehow revert the bad-deinterlacing-plus-upscaling effect.

The worst of all is that there never had been true interlacing, but only simple field-delay in the original PAL source, so that if the original had been preserved we wouldn't have any problem.

@StainlessS: Thanks! Will try all of those after having solved the other (off topic bad deinterlacing) problem.

@takla: Unfortunately I do not know the secrets of A. I... I can use some of the pre-trained things with a lot of success sometimes (thanks to @selur I used this for scratches a lot), but don't know at all how to train myself. So this would simply cost too much time which we don't have. Thanks anyway.

(I will be away until tuesday, so it's not meant unpolite that I won't answer before if there are answers.)

[StainlessS]

Afterthought,
EDIT: Or maybe ThSAD2 = ThSAD - (RadT*64))
Maybe better. [above not quite right]
ThSAD2 = ThSAD - ((RadT-1)*64))

[Reel.Deel]

So here is the real problem:
https://we.tl/t-87bthYSVl5


When working with badly interlaced material that has been upscale, it's almost always best to downscale to the "original" resolution. Anti-aliasing scrips and QTGMC's progressive input processing are almost always ineffective unless you scale down.

This gets rid of the heavy aliasing without loosing detail but some of the moire is still there for the most part.

LSMASHVideoSource("test2.mp4")
z_ConvertFormat(720, 540, pixel_type="YV24")
QTGMC()
SelectEven()

I added pixel_type="YV24" just in case you upscale back to the original resolution, you wont loose half of the color info (again). Probably does not make a huge difference.

By the way, there are other scripts you can try:

This one was intended for badly deinterlaced animation: https://forum.doom9.org/showthread.php?p=1968153#post1968153

This script nukes aliasing and some of the moire but with lots of detail loss: https://forum.doom9.org/showthread.php?p=1598922#post1598922

NoMoreMoire script, output is really calm and stable but with strong detail loss and other artifacts: https://forum.doom9.org/showthread.php?p=1373580#post1373580

Choose your poison :p

[Frank62]

Thanks for all these interesting things!
I did a lot of testing, but unfortunately nothing really works, so that it would look somehow better than what I already tried: simple placebo-QTGMC after downsizing in its current state (no separatefields with later weave-back a. s. o. was better) with a friendly not-too-sharpening spline36resize.
If you want to get rid of the massive moiree-blinking you loose EVERY detail there is, no way.
Also my approaches with pre-de-bilinear did lead to nothing, the A. I. "upscaling" did too much harm to reconstruct this way.

So I will see to get somehow rid of the rainbowing first and post-process this with fabulous SpotLess, using the parameters @StainlessS pointed out. Maybe this will have a positive effect also on the blinking.

[kedautinh12]

DFMDeRainbow latest ver:
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/DFMDeRainbow.avsi

[Frank62]

Thanks, but this is always our first try. Meanwhile this project is done, with nearly no rainbowing, a few (rainbow caused) artefacts, and unfortunately with some residual artefacts caused by deinterlacing. But it's watchable, thanks to some help here.