Searching plugin for my issue

[Hat3L0v3]

Hello, I'm a bit newbie in avisynth yet and I trying to find plugin that can fix my issue.
After filtering DVD and Upscale - I get this problem:

https://i.imgur.com/0rEhYe7.png

Guess it's because upscale deal with sharp filters on it. So I try to find something that can filter-out that grey on source. And the only thing I find it's - aWarp. But oblivious if I increase the depth on it - it will screw all proportions and image became less attractive. So I prefer not to use it.
Maybe there is any other plugins that can filter it out? Will be very appreciated for help.

[kedautinh12]

Try another resize like deep_resize():
https://github.com/Dogway/Avisynth-Scripts/blob/57513ac1f55548a42ffc280fc4e28b8d0d8e19f8/ResizersPack.avsi#L37
Check dependencies before ask if had error. You can try upload image to other severs or you will wating for approvaled by mod forever :D

[Dogway]

Don't use deep_resize on non-natural images (https://github.com/Dogway/Avisynth-Scripts/blob/57513ac1f55548a42ffc280fc4e28b8d0d8e19f8/ResizersPack.avsi#L39).
For this case you can use awarpsharp on chroma only, something like this:
mergechroma(aWarpSharp2(depth=20, thresh=190, blur=3, chroma=4))

You can also use chroma_reconstruction filter, but it's slow.

[Hat3L0v3]

Don't use deep_resize on non-natural images (https://github.com/Dogway/Avisynth-Scripts/blob/57513ac1f55548a42ffc280fc4e28b8d0d8e19f8/ResizersPack.avsi#L39).
For this case you can use awarpsharp on chroma only, something like this:
mergechroma(aWarpSharp2(depth=20, thresh=190, blur=3, chroma=4))

You can also use chroma_reconstruction filter, but it's slow.

thankyou for help. I've tested 4 variation where I can put this filter on:
1. after deinterlace
2. after dehalo, before resize sd->hd and color convert
3. after resize, before aa, degrain, deband
4. after all filters
Seems 2 and 3 give the best result, 1- the worst.

https://i.imgur.com/jjEC7K4.png

Result maybe not the best but that grey line became less intense. Need to figure what settings I can tweak without screwing image.
About chroma_reconstruction I would like to know more. If I gives better result - then why not? I don't chase the speed atm.

about deep_resize() - I dont get where and why I should use it. for sd-hd resize? dont realy think it will change anything.

[kedautinh12]

Cause deep_resize is wrap scripts so i think deep_resize's result will less error than other resizer filters. Example:
https://forum.doom9.org/showthread.php?p=1979592#post1979592

[Hat3L0v3]

Cause deep_resize is wrap scripts so i think deep_resize's result will less error than other resizer filters. Example:
https://forum.doom9.org/showthread.php?p=1979592#post1979592

So I need to change my LanczosResize() to deep_resize() when I going from SD to HD and then convert colors?
I will try it later when figure how to use it.

I was testing different parameters for mergechroma(aWarpSharp2(depth=20, thresh=190, blur=3, chroma=4))
Guess depth=40 is max that I can go, increasing it cause significant eating edges and details when grey doesn't decrease at all.
Changing thresh to lower or higher value doesn't give me better result for some reason.
Blur feels the same way.
And chroma=4 I didn't touch. I dont really know if I cant change it to 6 or not.
So I'll stay at depth=40, thresh=190, blur=3, chroma=4 for now, which is atleast something compare to what I had before. Maybe I will lower it because I didn't test on other scenes.
Comparison (https://i.imgur.com/AZOQ9cT.png)

Will go and check deep_resize.

[kedautinh12]

I suggested deep_resize cause it's less error than resizer filters alone. If your LanczosResize very fine result, don't need change to deep_resize. If deep_resize add very much grain and increase very much filesize, you can try grain=0 and add grain by yourself

[Reel.Deel]

So I need to change my LanczosResize() to deep_resize() when I going from SD to HD and then convert colors?
I will try it later when figure how to use it.

I was testing different parameters for mergechroma(aWarpSharp2(depth=20, thresh=190, blur=3, chroma=4))... And chroma=4 I didn't touch. I dont really know if I cant change it to 6 or not.
....

aWarpSharp2(depth=20, thresh=190, blur=3, chroma=6) is the same as mergechroma(aWarpSharp2(depth=20, thresh=190, blur=3, chroma=4)) but faster since it does not process the luma plane. It only uses the luma mask to guide the chroma. You might get better results using aWarp4, see the wiki page for usage examples: http://avisynth.nl/index.php/AWarpsharp2/aWarp4

[Hat3L0v3]

I suggested deep_resize cause it's less error than resizer filters alone. If your LanczosResize very fine result, don't need change to deep_resize. If deep_resize add very much grain and increase very much filesize, you can try grain=0 and add grain by yourself

I guess for me to make that plugin work - is already a challenge.
Stuck on this error (https://i.imgur.com/pojweA3.png) atm.

aWarpSharp2(depth=20, thresh=190, blur=3, chroma=6) is the same as mergechroma(aWarpSharp2(depth=20, thresh=190, blur=3, chroma=4)) but faster since it does not process the luma plane. It only uses the luma mask to guide the chroma. You might get better results using aWarp4, see the wiki page for usage examples: http://avisynth.nl/index.php/AWarpsharp2/aWarp4

something like mergechroma(aWarp4(depth=20, chroma=4)) ? I will test

[kedautinh12]

I said you read Dependencies before to easy to make that script work
https://github.com/Dogway/Avisynth-Scripts/blob/57513ac1f55548a42ffc280fc4e28b8d0d8e19f8/ResizersPack.avsi#L75

[kedautinh12]

You need get avs+ and vc++ latest ver
https://gitlab.com/uvz/AviSynthPlus-Builds
https://github.com/abbodi1406/vcredist

And you need add propSet("_FieldBased",0) after deinterlaced, before deep_resize

[Reel.Deel]

something like mergechroma(aWarp4(depth=20, chroma=4)) ? I will test

No. Something like this:

input = YourSourceFilter("video.mkv")
input4x = input.nnedi3(dh=true, u=false, v=false).Spline36Resize(width*4, height*4, 0.25, 0.25)
edge_mask = input.aSobel(thresh=128, chroma=1).aBlur(blur=3, type=1, chroma=1)

aWarp4(input4x, edge_mask, depth=x, chroma=6)

There's no need to use MergeChroma after since the luma is copied when chroma=6.

[Hat3L0v3]

No. Something like this:

input = YourSourceFilter("video.mkv")
input4x = input.nnedi3(dh=true, u=false, v=false).Spline36Resize(width*4, height*4, 0.25, 0.25)
edge_mask = input.aSobel(thresh=128, chroma=1).aBlur(blur=3, type=1, chroma=1)

aWarp4(input4x, edge_mask, depth=x, chroma=6)

There's no need to use MergeChroma after since the luma is copied when chroma=6.

Is chroma mode must be =1 for aSobel and aBlur ? It give me green image (https://i.imgur.com/0DhY4DQ.png). Tried to change for 4 - doesn't give me anything better.
Seems problem in nnedi3(dh=true, u=false, v=false) . If I deinterlace video with animeIVTC and wanna use it inside same file, should I write as:
input=last
input4x=input.nnedi3(dh=true, u=false, v=false).Spline36Resize(width*4, height*4, 0.25, 0.25)
edge_mask=input.aSobel(thresh=190, chroma=1).aBlur(blur=3, type=1, chroma=1)
aWarp4(input4x, edge_mask, depth=40, chroma=6)
You need get avs+ and vc++ latest ver
https://gitlab.com/uvz/AviSynthPlus-Builds
https://github.com/abbodi1406/vcredist

And you need add propSet("_FieldBased",0) after deinterlaced, before deep_resize

I thought avs+ 3.7.2 is the latest version.
Tested with deep_resize(946,720,flat="nnedi3",grain=0,qual=2) against my LanczosResize(946,720) . It gives more sharp edges but doesn't feel it change something in that gray area. screen (https://i.imgur.com/toSUjUX.png).
propSet("_FieldBased",0) doesn give any difference if I put it or not. I'll upscale later to compare, maybe will use as replace of mine.

[kedautinh12]

avs+ 3.7.2 is stable ver (March 18, 2022). My link is up-to-date just 2 month ago (r3825)
https://github.com/AviSynth/AviSynthPlus/blob/master/distrib/Readme/readme_history.txt

[Reel.Deel]

Is chroma mode must be =1 for aSobel and aBlur ? It give me green image (https://i.imgur.com/0DhY4DQ.png). Tried to change for 4 - doesn't give me anything better.
Seems problem in nnedi3(dh=true, u=false, v=false) . If I deinterlace video with animeIVTC and wanna use it inside same file, should I write as:
input=last
input4x=input.nnedi3(dh=true, u=false, v=false).Spline36Resize(width*4, height*4, 0.25, 0.25)
edge_mask=input.aSobel(thresh=190, chroma=1).aBlur(blur=3, type=1, chroma=1)
aWarp4(input4x, edge_mask, depth=40, chroma=6)


Oops, had a brain fart. Use this:

input=last
input4x=input.nnedi3_rpow2(rfactor=2).Spline36Resize(width*4, height*4, 0.25, 0.25)
edge_mask=input.aSobel(thresh=190, chroma=1).aBlur(blur=3, type=1, chroma=1)
aWarp4(input4x, edge_mask, depth=40, chroma=6)

[kedautinh12]

Sometime, deep_resize will error when use with TIVTC or script use TIVTC (AnimeaIVTC). That why i suggested use propSet("_FieldBased",0) to avoid error
https://forum.doom9.org/showthread.php?p=1979223#post1979223

[Hat3L0v3]

Oops, had a brain fart. Use this:

input=last
input4x=input.nnedi3_rpow2(rfactor=2).Spline36Resize(width*4, height*4, 0.25, 0.25)
edge_mask=input.aSobel(thresh=190, chroma=1).aBlur(blur=3, type=1, chroma=1)
aWarp4(input4x, edge_mask, depth=40, chroma=6)

and this one gives me another colors (https://i.imgur.com/L6CKyyc.png)

[kedautinh12]

try mergechroma(aWarp4(input4x, edge_mask, depth=40, chroma=6)) in last line

[Reel.Deel]

and this one gives me another colors (https://i.imgur.com/L6CKyyc.png)

It seems a while back the behavior of the updated aWarpSharp2 was changed for aWarp4. Not sure why but:

v1.0.2 - 2017/12/06 - aWarp4 don't copy planes on some chroma modes (doc updated).

Docs:
Copy on aWarp4 make no sense, as source size is 4*output size. It's up to the user to use
whatever resampling mode he wants to create the "not cared" missing planes.


I did not know the behavior was changed from the original aWarp4 :confused: ... so yeah, use MergeChroma afterwards.

[Hat3L0v3]

Tested with mergechroma(aWarp4(input4x, edge_mask, depth=40, chroma=6)) , doesn't feel better at all. compare (https://i.imgur.com/S28h89G.png).
Face became pale, alot of color artifacts everywhere, and my grey issue doesn't feel better. Maybe it needs some another settings. I'll try with deph 15 (40 seems too big).
Also nnedi3_rpow2 was complaining that my input width must be mod 4. (I had 946) so I change to 948 and stretched image a bit. Is there possible ways to pass this issue without cropping/streching image?

[Reel.Deel]

I think your settings for aWarp4 might be too high. aWarp4 was meant to be used with lower settings.

Unlike aWarp (aWarpSharp2), this uses a 4x larger source clip than the edge mask. This is useful for improving the quality of the subpixel interpolation quality in the final step. Supersampling allows to produce a sharper result with lower depth, which equals to less deformations.

If you don't want to crop or resize your image, you can use Padding() (https://forum.doom9.org/showpost.php?p=1596804&postcount=5) to make the width mod4 and after just crop back the added pixels.

[Hat3L0v3]

Tested with deph=15, now it's look proper. IMG (https://i.imgur.com/mUiGPIT.png)
It gives better result on the chin, but worst on forehead. shoulder seems a little better too. I guess I need to tweak edge_mask parameter (aSobel or aBlur) ?


If you don't want to crop or resize your image, you can use Padding() (https://forum.doom9.org/showpost.php?p=1596804&postcount=5) to make the width mod4 and after just crop back the added pixels.

Oh thankyou for advise, definitely will try it!

[Reel.Deel]

Tested with deph=15, now it's look proper. IMG (https://i.imgur.com/mUiGPIT.png)
It gives better result on the chin, but worst on forehead. shoulder seems a little better too. I guess I need to tweak edge_mask parameter (aSobel or aBlur) ?


Yes, try to adjust those settings and see what gives you the best result. I think the most important ones are depth and thresh. If you start getting deformations, they are likely too high.

This may also help, try applying a somewhat strong unsharp mask on the clip before the warpsharpening. This helps in the edge mask detection, specially with soft footage, like the one you have. Something like http://avisynth.nl/index.php/VsMSharpen

...
input4x=input.YourPreferedSharpener().nnedi3_rpow2(rfactor=2).Spline36Resize(width*4, height*4, 0.25, 0.25)
...

[kedautinh12]

Try ex_unsharp
ex_unsharp(0.2,0.4,float(width())/5, 1)
Or
ex_unsharp(-0.5,0,float(width())/4,0, safe=false)
https://github.com/Dogway/Avisynth-Scripts/blob/57513ac1f55548a42ffc280fc4e28b8d0d8e19f8/MIX%20mods/SharpenersPack.avsi#L484

[Hat3L0v3]

I feeling like i'm stuck by tweaking depth and thresh parameters beeing blindfolded. It changes small detail, when grey lines doesn't going to disapear with any settings. Like I missing something.
Adding unsharp mask give the same result as without it, atleast I didn't find settings that make it different.
:(

[StainlessS]

EDIT: Oops, I was answering last post #20 on previous page. (but the MergeChroma thingy does not seem to have been addressed in intervening posts)
Adding unsharp mask give the same result as without it, atleast I didn't find settings that make it different.
try mergechroma(aWarp4(input4x, edge_mask, depth=40, chroma=6)) in last line
I did not know the behavior was changed from the original aWarp4 ... so yeah, use MergeChroma afterwards.

input=last
input4x=input.nnedi3_rpow2(rfactor=2).Spline36Resize(width*4, height*4, 0.25, 0.25)
edge_mask=input.aSobel(thresh=190, chroma=1).aBlur(blur=3, type=1, chroma=1)
aWarp4(input4x, edge_mask, depth=40, chroma=6)
mergechroma(aWarp4(input4x, edge_mask, depth=40, chroma=6) /* Missing chroma source arg, so is is using Last as input luma, aWarp4x as chroma */ )

Not sure what above is supposed to do, but below seems more appropriate to me

input=last
input4x=input.nnedi3_rpow2(rfactor=2).Spline36Resize(width*4, height*4, 0.25, 0.25)
edge_mask=input.aSobel(thresh=190, chroma=1).aBlur(blur=3, type=1, chroma=1)
mergechroma(aWarp4(input4x, edge_mask, depth=40, chroma=6),input4x)
# or if that dont work (I'm not sure what aWarp4 does)
# mergechroma(aWarp4(input4x, edge_mask, depth=40, chroma=6),input)


EDIT: Maybe the wrong-ish MergeLuma() thingy sent the thread off on a 'wild goose chase' :)
EDIT: Or, maybe I've totally misunderstood what you are trying to do.

EDIT: Additional,
Also nnedi3_rpow2 was complaining that my input width must be mod 4.


ConvertToYV24()
input=last
input4x=input.nnedi3_rpow2(rfactor=2).Spline36Resize(width*4, height*4, 0.25, 0.25)
edge_mask=input.aSobel(thresh=190, chroma=1).aBlur(blur=3, type=1, chroma=1)
mergechroma(aWarp4(input4x, edge_mask, depth=40, chroma=6),input4x)
# or if that dont work (I'm not sure what aWarp4 does)
# mergechroma(aWarp4(input4x, edge_mask, depth=40, chroma=6),input)
ConvertToYV12()

[Hat3L0v3]

Not sure what above is supposed to do, but below seems more appropriate to me

input=last
input4x=input.nnedi3_rpow2(rfactor=2).Spline36Resize(width*4, height*4, 0.25, 0.25)
edge_mask=input.aSobel(thresh=190, chroma=1).aBlur(blur=3, type=1, chroma=1)
mergechroma(aWarp4(input4x, edge_mask, depth=40, chroma=6),input4x)
# or if that dont work (I'm not sure what aWarp4 does)
# mergechroma(aWarp4(input4x, edge_mask, depth=40, chroma=6),input)



Adding input4x cause error about image must be the same width and height.
Adding input gives me the same color image (https://i.imgur.com/FInk5z5.png).

[StainlessS]

See also the stuff in blue added later. [ConvertToYV24 and ConvertToYV12] think that should work

EDIT: try, [NO EMPTY LINES, what line number does it error on ? And What is the error message]

ConvertToYV24()
input=last
input4x=input.nnedi3_rpow2(rfactor=2).Spline36Resize(width*4, height*4, 0.25, 0.25)
edge_mask=input.aSobel(thresh=190, chroma=1).aBlur(blur=3, type=1, chroma=1)
WP = aWarp4(input4x, edge_mask, depth=40, chroma=6)
mergechroma(WP, input4x)
# or if that dont work, COMMENT OUT ABOVE LINE and UNCOMMENT BELOW (I'm not sure what aWarp4 does)
#mergechroma(WP, input)
ConvertToYV12()


EDIT: Also, can try below if both above MergeChroma() etc fails.
After WP=aWarp( Etc), insert line

Return WP.Info()

Note the dimensions, and then again change to
Return Input4X.Info()
Note the dimensions, and then again change to
Return Input.Info()

Report findings, error lines, error messages, dimensions of all 3 clips, WP, Input4X, and Input.

[Hat3L0v3]

EDIT: Also, can try below if both above MergeChroma() etc fails.
After WP=aWarp( Etc), insert line

Return WP.Info()

Note the dimensions, and then again change to
Return Input4X.Info()
Note the dimensions, and then again change to
Return Input.Info()

Report findings, error lines, error messages, dimensions of all 3 clips, WP, Input4X, and Input.

Adding ConvertToYV24() doesn't change anything.
I dont really understand what you saying, made all 3 screenshots here (https://i.imgur.com/v4LWdHF.png).

[Reel.Deel]

ssS, this script is fine:

input=last
input4x=input.nnedi3_rpow2(rfactor=2).Spline36Resize(width*4, height*4, 0.25, 0.25)
edge_mask=input.aSobel(thresh=190, chroma=1).aBlur(blur=3, type=1, chroma=1)
aWarp4(input4x, edge_mask, depth=40, chroma=6)
MergeChroma(input, last)

aWarp4 takes a 4x upscaled clip as input but returns a 4x downscaled clip. So trying to MergeChroma with input4x won't work.

[kedautinh12]

I think ssS now not him/her. This is ChatGPT, and need training to work like ssS :D

[StainlessS]

Phew, glad we got there, thank you.

@Ked12, You keep quiet, chatterbox :)

[Hat3L0v3]

Is there possibility to view chroma (and maybe luma) separately? or maybe view mask for visual tweaking?

[StainlessS]

Maybe something like this

input=last
input4x = input.nnedi3_rpow2(rfactor=2).Spline36Resize(width*4, height*4, 0.25, 0.25)

edge_mask = input.aSobel(thresh=190, chroma=1).aBlur(blur=3, type=1, chroma=1)
chroma = aWarp4(input4x, edge_mask, depth=40, chroma=6)
Result = MergeChroma(input,chroma)

# UnComment 1
#Return input.GreyScale # Luma [set chroma U & V 128]
#Return chroma.MergeLuma(chroma.BlankClip(Color_yuv=$808080)) # Chroma with luma set mid gray [or $807E7E for Tv.Levels mid grey]
#Return edge_mask # mask
#Last=Result Return StackVertical(Subtitle("YV12 Result", align=2),StackHorizontal(UtoY.Subtitle("UtoY", align=2),VtoY.Subtitle("VtoY", align=2))) # Result with separate chroma UV
#Last=input Return StackVertical(Subtitle("YV12 Input", align=2),StackHorizontal(UtoY.Subtitle("UtoY", align=2),VtoY.Subtitle("VtoY", align=2))) # Input with separate chroma UV
#Last=input Return GreyScale.StackVertical(Subtitle("Y Input", align=2),StackHorizontal(UtoY.Subtitle("UtoY", align=2),VtoY.Subtitle("VtoY", align=2))) # Input with separate YUV
#Last=Result Return GreyScale.StackVertical(Subtitle("Y Result", align=2),StackHorizontal(UtoY.Subtitle("UtoY", align=2),VtoY.Subtitle("VtoY", align=2))) # Result with separate YUV

Return Result # final answer

from Wiki:- http://avisynth.nl/index.php/Swap
Like so
https://i.postimg.cc/W3yc0F2c/Untitled-00.jpg (https://postimg.cc/KktW2Yk0)

Added a few more, untested.

[Hat3L0v3]

Maybe something like this

input=last
input4x = input.nnedi3_rpow2(rfactor=2).Spline36Resize(width*4, height*4, 0.25, 0.25)

edge_mask = input.aSobel(thresh=190, chroma=1).aBlur(blur=3, type=1, chroma=1)
chroma = aWarp4(input4x, edge_mask, depth=40, chroma=6)
Result = MergeChroma(input,chroma)

# UnComment 1
#Return input.GreyScale # Luma [set chroma U & V 128]
#Return chroma.MergeLuma(chroma.BlankClip(Color_yuv=$808080)) # Chroma with luma set mid gray [or $807E7E for Tv.Levels mid grey]
#Return edge_mask # mask
#Last=Result Return StackVertical(Subtitle("YV12 Result", align=2),StackHorizontal(UtoY.Subtitle("UtoY", align=2),VtoY.Subtitle("VtoY", align=2))) # Result with separate chroma UV

Return Result # final answer

Untested
Updated. from Wiki:- http://avisynth.nl/index.php/Swap

Thankyou.
Is there something that can increase color intensity for choma? Very hard to see difference, I almost can't see the colors.

[StainlessS]

Have a look at the others I've just added.
Chroma does not usually wander too far away from center 128.

Time for bed, 05:09 AM.

[Hat3L0v3]

I just find out that almost whole difference was caused by Stab filter, which I wasn't use at old version. Don't thought that he can made that.
So I tested again with and without stab. (screens comparison (https://imgsli.com/MTQ1OTgz/))
Result - mergechroma definitely fix some chroma glowing around face and hair. Thats good.
But to my issue (gray line along egde on forehead, chin and shoulder) it give almost nothing. So I start thinking that maybe its not chroma problem. Maybe without luma its not fixing? or maybe its upscaler problem. I tried other upscale models and they gives me only worst results, that's why I create this thread.
:(

[kedautinh12]

What stab() you use??

[Hat3L0v3]

What stab() you use??

Stab3, but why are you asking? stab not an issue. specially when it make even better.

[kedautinh12]

Stab3, but why are you asking? stab not an issue. specially when it make even better.

oh, i know this stab3. My hobby is collection avs filters :D

[StainlessS]

My hobby is collection avs filters
Get a Life :)

EDIT To below: My hobby is sharpening my axe, for little critters like you.

[kedautinh12]

Get a Life :)

Ok, android;)

[Hat3L0v3]

I don't know what to do. The only thing that helps it's aWarpSharp (comparison (https://imgsli.com/MTQ2MTg4/0/2)). mergechroma doesn't give any good results in my case. Is this grey issue need luma correction?
don't wanna use aWarpSharp, proportions changes a lot. :(