HAvsFunc

[Myrsloik]

And if it helps. The problem is produced somewhere in contrahsrapening and not smdegrain: https://github.com/HomeOfVapourSynthEvolution/havsfunc/blob/master/havsfunc.py#L5406
It's a short script and the pink border still happens if you remove miniblur.

When I remove the only Expr line, the stripes disappear but the pink border is still there. Border could be Repair or MakeDiff. Will test later more.


EDIT
should this produce a pink image?
clip = core.std.BlankClip(width=200,height=100, format=vs.YUV420P8)
matrix1 = [1, 2, 1, 2, 4, 2, 1, 2, 1]
con = core.std.Convolution(clip, matrix=matrix1)

clip=core.std.MakeDiff(clip, con)

I've narrowed down the green lines. They happen with expr and 16 bit formats in the sse2 code path.

The script you posted definitely shouldn't be pink. For some odd reason it's not pink on my computer which should use exactly the same code path (threadripper 1950x).

[Are_]

AMD Ryzen 7 1700 here and no pink image, only the green strips bug.

[ChaosKing]

I tested now with the portable version and it only shows the green stripes. It seems I had still 48-test2 installed,.. I was sure it was the RC edition, sry :o
Installing the RC removed the pink border

At least my incompetence has found another bug :p

[ChaosKing]

Is it possible to port also FineDehalo_contrasharp? It returns a nice sharp image.



avs+ version
# level == 1.0 : normal contrasharp
Function FineDehalo_contrasharp (clip dehaloed, clip src, float level)
{
bb = dehaloed.RemoveGrain (11, -1)
bb2 = bb.Repair (bb.Repair (bb.Medianblur (2, -256, -256), 1), 1)
xd = mt_makediff (bb, bb2)
xd = VersionNumber() < 2.6 ? xd.mt_lut ("x 128 - 2.49 * "+String(level)+" * 128 +")
\ : xd.mt_lut ("x range_half - 2.49 * "+String(level)+" * range_half +")
xdd = mt_lutxy (
\ xd,
\ mt_makediff (src, dehaloed),
\ VersionNumber() < 2.6 ? "x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?"
\ : "x range_half - y range_half - * 0 < range_half x range_half - abs y range_half - abs < x y ? ?"
\ )

dehaloed.mt_adddiff (xdd, y=3, u=2, v=2)
}

Function FineDehalo_contrasharp (clip dehaloed, clip src, float level)
{
bb = dehaloed.RemoveGrain (11, -1)
bb2 = bb.Repair (bb.Repair (bb.Medianblur (2, -256, -256), 1), 1)
xd = mt_makediff (bb, bb2)
xd = xd.mt_lut ("x 128 - 2.49 * "+String(level)+" * 128 +")
xdd = mt_lutxy (
\ xd,
\ mt_makediff (src, dehaloed),
\ "x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?"
\ )

dehaloed.mt_adddiff (xdd, y=3, u=2, v=2)
}

[HolyWu]

Is it possible to port also FineDehalo_contrasharp? It returns a nice sharp image.

Doesn't FineDehalo already have a contra parameter for contrasharpening?

[ChaosKing]

lol I never noticed xD
But an external "dehalo contrasharp" is still usefull. It can be used with other dehalo functions.

[HolyWu]

But an external "dehalo contrasharp" is still usefull. It can be used with other dehalo functions.

I'd like to know how is the result compared to Didée's ContraSharpening function. Is it only suitable for denoising but not the others?

[ChaosKing]

ContraSharpening adds halos back. It seems it is only usefull for denoised clips.

[poisondeathray]

Does the overlay function work for RGBS ? It seems values are incorrect, but correct with maskedmerge


a = core.std.BlankClip(format=vs.RGBS, color=[1, 1, 1])
b = core.std.BlankClip(format=vs.RGBS, color=[2, 2, 2])
c = core.std.BlankClip(format=vs.RGBS, color=[0.5, 0.5, 0.5])

opacity = 0.5

msk = core.std.BlankClip(a, color=[opacity, opacity, opacity])

ov = haf.Overlay(a,b,opacity=0.5)

mm = core.std.MaskedMerge(a,b,mask=msk)

ov.set_output()
#mm.set_output()



For example, a,b with opacity 0.5 correctly gives RGB [1.5,1.5,1.5] with maskedmerge, but some error value for overlay

[HolyWu]

Does the overlay function work for RGBS ? It seems values are incorrect, but correct with maskedmerge

Fixed.

[poisondeathray]

Fixed.

Thanks!

Would it be possible to add overlay blend modes? Such as "add", "multiply", "screen", etc..

[HolyWu]

Would it be possible to add overlay blend modes? Such as "add", "multiply", "screen", etc..

Added. But I don't see "screen" as one of the modes in http://avisynth.nl/index.php/Overlay :confused:

[poisondeathray]

Added. But I don't see "screen" as one of the modes in http://avisynth.nl/index.php/Overlay :confused:

Thanks!

"screen" is not in the overlay avs version, but it's one of the more common compositing/photoshop blend modes used

rr42 translated some here for uu_mt_blend based on maskttools, and there are some links to the math behind them in the 1st post. I don't think that version supports different bit depths

https://forum.doom9.org/showthread.php?t=170490
https://photoblogstop.com/photoshop/photoshop-blend-modes-explained


Screen
1−(1−A)×(1−B)
A inverted multiplied by B inverted, and the product is inverted

[poisondeathray]

YUV overlay mask transparency issue. Partial transparency issue on 100% white areas

-affects YUV overlay operations in 10bit, different subsamplings.

-RGB overlay operations seem ok

-does not matter if you mark YUV versions as "limited" range or "full"

-Works ok if you use Gray8 for the mask, and base clip / overlay clip are matching 8bit YUV ; or Gray16 and base/overlay are matching 16bit YUV . But there is no Gray10 format, so 10bit YUV overlay with mask is out of luck, or 12bit YUV

-works as expected if you go back to older havsfunc version a few months ago, with matching pixel format YUVxxxPx mask instead of using GRAYx mask. Not sure where the exact commit altered it, possibly the recent ones

eg.


import vapoursynth as vs
import havsfunc as haf
core = vs.get_core()

orig = core.imwri.Read(r'testchart.png')
logo = core.imwri.Read(r'logo.png', alpha=True)

orig2 = core.resize.Bicubic(orig, format=vs.YUV420P10, matrix_s="709")

logobase2 = core.resize.Bicubic(logo[0], format=vs.YUV420P10, matrix_s="709", range_s="full")

logoalpha2 = core.resize.Bicubic(logo[1], format=vs.YUV420P10, matrix_s="709", range_s="full")
#logoalpha2 = core.std.SetFrameProp(logoalpha2, prop="_ColorRange", intval=1) # Mark video as limited range.
#logoalpha2 = core.resize.Bicubic(logo[1], format=vs.GRAY8, matrix_s="709", range_s="full")

#RGB24, RGB30, RGB48 work ok
#orig2 = core.resize.Bicubic(orig, format=vs.RGB30)
#logobase2 = core.resize.Bicubic(logo[0], format=vs.RGB30)
#logoalpha2 = core.resize.Bicubic(logo[1], format=vs.RGB30)


overl = haf.Overlay(orig2, logobase2, mask=logoalpha2)

overl.set_output()




Test images
https://www.mediafire.com/file/vy6k8c6eon4widc/overlay_test_images.zip/file

[HolyWu]

YUV overlay mask transparency issue. Partial transparency issue on 100% white areas

Thanks. Should be fixed now (rebased into the last commit).


But there is no Gray10 format, so 10bit YUV overlay with mask is out of luck, or 12bit YUV

You can use core.register_format (http://www.vapoursynth.com/doc/pythonreference.html#Core.register_format) for formats without constants predefined.

logoalpha2 = core.resize.Bicubic(logo[1], format=core.register_format(vs.GRAY, vs.INTEGER, 10, 0, 0).id, matrix_s="709", range_s="full")

[poisondeathray]

Thanks for the fix and the info HolyWu . Didn't know about core.register_format

[poisondeathray]

Sorry, it doesn't seem "fixed" . Same script as above

Or are you saying you should always use GRAY, and with core.register_format if it's not a predefined format ?

[HolyWu]

Sorry, it doesn't seem "fixed" . Same script as above

These are my results.



orig = core.imwri.Read(r'testchart.png')
logo = core.imwri.Read(r'logo.png', alpha=True)

orig2 = core.resize.Bicubic(orig, format=vs.YUV420P10, matrix_s="709")
logobase2 = core.resize.Bicubic(logo[0], format=vs.YUV420P10, matrix_s="709")
logoalpha2 = core.resize.Bicubic(logo[1], format=vs.YUV420P10, matrix_s="709", range_s="full")

overl = haf.Overlay(orig2, logobase2, mask=logoalpha2)
overl.set_output()


https://i.postimg.cc/bs7FrwVK/YUV420-P10.png (https://postimg.cc/bs7FrwVK)



orig = core.imwri.Read(r'testchart.png')
logo = core.imwri.Read(r'logo.png', alpha=True)

orig2 = core.resize.Bicubic(orig, format=vs.RGB30)
logobase2 = core.resize.Bicubic(logo[0], format=vs.RGB30)
logoalpha2 = core.resize.Bicubic(logo[1], format=vs.RGB30)

overl = haf.Overlay(orig2, logobase2, mask=logoalpha2)
overl.set_output()


https://i.postimg.cc/zVCTwDGm/RGB30.png (https://postimg.cc/zVCTwDGm)

[poisondeathray]

It's correct. Sorry for false alarm

It turns out wrong havsfunc.py version was loading

Thanks again

[sl1pkn07]

offtopic

dat logo. where?

greetings

EDIT: done! http://forum.doom9.net/showthread.php?p=1847472#post1847472

[tormento]

I am thinking about giving a try to VapourSynth.

I mainly use SMDegrain from real.finder and it basically needs MVTools, Masktools and KNLMeansCL.

AFAIK Masktools is included into VS core, KNLMeans is compatible but I have doubts about which versione of VS MVTools I have to use: single precision ones or the other one? What are the differences?

I'd like also to know if there is an implementation of CompTest.avsi:
function Comptest(clip c, float percent)
{
frange = floor(14./(percent/100.))
return SelectRangeEvery(c, frange, 14)
}

[Myrsloik]

I am thinking about giving a try to VapourSynth.

I mainly use SMDegrain from real.finder and it basically needs MVTools, Masktools and KNLMeansCL.

AFAIK Masktools is included into VS core, KNLMeans is compatible but I have doubts about which versione of VS MVTools I have to use: single precision ones or the other one? What are the differences?

I'd like also to know if there is an implementation of CompTest.avsi:
function Comptest(clip c, float percent)
{
frange = floor(14./(percent/100.))
return SelectRangeEvery(c, frange, 14)
}

1. ALWAYS use vsrepo. It's your friend and will give you the right versions of everything. Especially for things like HAvsFunc it will save you hours of looking for things.

2. MVTools differences; the single precision one is just what the name implies, a special version that only supports single precision float.

3. Untested but something like this:import math

def comptest(clip, percent)
frange = math.floor(1400/percent)
return clip.std.SelectEvery(cycle=frange, offsets=list(range(0, 14)))

[tormento]

MVTools differences; the single precision one is just what the name implies, a special version that only supports single precision float.
Thanks for your reply.

But what version do I need to run HAVSFunc? :)

[Myrsloik]

Thanks for your reply.

But what version do I need to run HAVSFunc? :)

Normal

[tormento]

Normal
As you are so kind, how would you translate this AVS script into VS script?
SetMemoryMax(8000)
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\eseguibili\media\DGDecNV_x64\DGDecodeNV.dll")
DGSource("E:\in\something.dgi",ct=0,cb=0,cl=0,cr=0)
CompTest(1)
SMDegrain (tr=4, thSAD=500, refinemotion=false, n16=true, mode=0, contrasharp=false, PreFilter=4, truemotion=false, plane=4, chroma=true)
Prefetch(6)

[ChaosKing]

import vapoursynth as vs
import math
import havsfunc as haf
core = vs.core
core.max_cache_size = 8000

def comptest(clip, percent):
frange = math.floor(1400/percent)
return clip.std.SelectEvery(cycle=frange, offsets=list(range(0, 14)))

clip = core.dgdecodenv.DGSource(r"E:\in\something.dgi", ct=0,cb=0,cl=0,cr=0)
clip = comptest(clip, 1)
clip = haf.SMDegrain(clip, tr=3, thSAD=500, RefineMotion=False, truemotion=False, plane=4, chroma=True, prefilter=4)


clip.set_output()

mvtools only supports tr=3 or less.
You don't need prefetch or MTModes in VS.

[tormento]

mvtools only supports tr=3 or less.
:(

That's a big NO for me as I recover old videos with lot of noise/grain.

Is it a limit of the SMDegrain script implementation or of MVTools standard? I read that MVTools goes up to MDegrain24, at least for simple precision version.

tr [int: 1, "2", 3, 4, 5, 6, ...]
Temporal radius. Select between MDegrain 1, 2, 3, 4, 5, 6 or higher. Higher is generally better, but also much slower and improvements get less drastic.
This can be considered the strength of the denoising. tr 4, 5 and 6 (limit is tr=128, 64 for interlaced) require Dither's MVTools2 mod.

EDIT10:
test11
1. Binary Part: Extended Degrain to Degrain24 (24, it's my lucky number!)
2. Resurrected vmulti features from MVTools 2.6.0.5, implemented via a python module, "tr" works up to 24, guess no one will ever use a time radius > 24.... maybe?

You don't need prefetch or MTModes in VS.
How I set or limit the number of threads? DGSource and KNLMeansCL are very Video RAM demanding on multiple sessions with hi resolution video.

[ChaosKing]

You can use a different smdegrain version instead, it supports tr > 3: https://github.com/Selur/VapoursynthScriptsInHybrid/blob/master/G41Fun.py

It uses the single precision MVTools version from here
https://forum.doom9.org/showthread.php?t=172525

that means you must filter it in 32 float precision.

Add also this then:
import mvsfunc as mvf
import math, mvmulti
clip = mvf.Depth(clip, 32) # this before smdegrain

How to limit thread count: http://www.vapoursynth.com/doc/pythonreference.html#Core.num_threads

[tormento]

that means you must filter it in 32 float precision.
Thanks, man!

And how will I dither back to 8 bit?

[ChaosKing]

Thanks, man!

And how will I dither back to 8 bit?
clip = mvf.Depth(clip, 8)

[tormento]

clip = mvf.Depth(clip, 8)
I ended to do a try with your first script before using higher TR and I get lower fps on VS than AVS+, i.e. 10,58 vs 14,43.

So I watched at the created mkv: the VS is a slideshow of the same lenght of the original movie, made of still pictures, the AVS+ is the correct sampled video.

There is something wrong with the comptest. When you have time, would you please debug it for me? :p

P.S: I tried to use the other function with tr=4 and VS crashed miserably. Is there anything wrong?

import vapoursynth as vs
import math
import G41Fun as G41
import mvsfunc as mvf
import math, mvmulti
core = vs.get_core()
core.max_cache_size = 8000

core.std.LoadPlugin("d:/eseguibili/media/DGDecNV_x64/DGDecodeNV.dll")
clip = core.dgdecodenv.DGSource("E:/in/2_06trinita/chiamarlo.dgi",ct=136,cb=136,cl=0,cr=0)

clip = mvf.Depth(clip, 32) # this before smdegrain

clip = G41.SMDegrain(clip, tr=4, thSAD=500, prefilter=4)

clip = mvf.Depth(clip, 8)

clip.set_output()

[ChaosKing]

Script looks correct to me. Have you looked at the fps of your created video?
You can also see some infos in vsedit if you press F6, like this

Script was successfully evaluated. Output video info:
Frames: 28621 | Time: 0:19:53.734 | Size: 1920x1080 | FPS: 24000/1001 = 23.976 | Format: YUV420P8

[tormento]

Script looks correct to me. Have you looked at the fps of your created video?
No video created, as it crashed.

Could you please apply to a real clip and tell me?

[ChaosKing]

The Mvtools dll by feisty2 expects avx2 instruction set. Does you cpu support avx2?

You could try another compiled version from here https://forum.doom9.org/showthread.php?p=1823382#post1823382

[tormento]

The Mvtools dll by feisty2 expects avx2 instruction set. Does you cpu support avx2?
Nope. :D

Will give it a try.

Could you please find some spare time to write a working comptest?

Thanks!

EDIT: not crashing anymore but painfully slow, such as 1.0 fps instead of the 10-12 I was getting with standard MVTools.

[ChaosKing]

Nope. :D

Will give it a try.

Could you please find some spare time to write a working comptest?

Thanks!

EDIT: not crashing anymore but painfully slow, such as 1.0 fps instead of the 10-12 I was getting with standard MVTools.

The comptest output is the same except for the fps
This should use the original video fps
def comptest(clip, percent):
frange = math.floor(1400/percent)
return clip.std.SelectEvery(cycle=frange, offsets=list(range(0, 14))).std.AssumeFPS(clip)

[tormento]

The comptest output is the same except for the fps
This should use the original video fps
def comptest(clip, percent):
frange = math.floor(1400/percent)
return clip.std.SelectEvery(cycle=frange, offsets=list(range(0, 14))).std.AssumeFPS(clip)
Thanks, I will try when this movie will finish to encode.

No hint anyway for MVTools single precision. Too slow to have acceptable performance. I asked for another no AVX2 binary and the author replied me that the script is a whole mess.

How I wish standard MVTools would support highter TR...

[Boulder]

Do you absolutely need SMDegrain or would some other denoiser do? Off the top of my head, I'd say that any details will be long gone with radius 3 at strong settings anyway so you could do with something faster.

[tormento]

Do you absolutely need SMDegrain or would some other denoiser do?
I am open to alternatives with same level of detail retaining/performance. Not found any, until now.

[Sharc]

I am open to alternatives with same level of detail retaining/performance. Not found any, until now.
Difficult to advise when we don't know your source.....

[tormento]

Difficult to advise when we don't know your source.....



Mostly old reels with lot of grain, the type of movies that drive you crazy to denoise.

[Sharc]

Mostly old reels with lot of grain, the type of movies that drive you crazy to denoise.
KNLMeansCL? dfttest? MDegrain? TemporalDegrain2(degrainTR=8)? Any of these possibly followed by a pure temporal denoiser ... Have you tried all these?
Much depends on the particular noise/grain of the source. There is no one fits all filter.

[Boulder]

KNLMeansCL could be one to test. Maybe add some motion compensation with MVTools.

[tormento]

KNLMeansCL could be one to test. Maybe add some motion compensation with MVTools.



I.e. SMDegrain that I correctly use.

On some reels I have to use tr>3 and it’s where VS stops being of any use.

[tormento]

I'd say that any details will be long gone with radius 3 at strong settings anyway so you could do with something faster.
The higher tr is, the better the results, mostly on fast speed scenes. Strong settings is all another matter. On really old BW movies I have to use tr=9 with lot less detail loss than tr=3.

[Sharc]

I.e. SMDegrain that I correctly use.

On some reels I have to use tr>3 and it’s where VS stops being of any use.
Try Avisynth TemporalDegrain2(degrainTR=xx). xx=16 for example. It's slow though.

[Boulder]

I.e. SMDegrain that I correctly use.

On some reels I have to use tr>3 and it’s where VS stops being of any use.

SMDegrain uses MDegrain to denoise as far as I know, a very different method compared to KNLMeans.

It would be interesting to have sample clips of before and after filtering of one severe case where a high temporal radius is needed. I personally never go above 1 when denoising, but then again, I prefer keeping things as close to the original as it's possible.

[tormento]

SMDegrain uses MDegrain to denoise as far as I know, a very different method compared to KNLMeans.
KNLMeans is applied as prefilter.

Again: higher tr means more precise results as motion vectors.

High thSAD is where details start be lost.

[Sharc]

Maybe tormento's original has very low details?
For VHS tapes a higher TR can be beneficial.

[tormento]

Maybe tormento's original has very low details?
For VHS tapes a higher TR can be beneficial.
Old BW movies have way lot more details than anything produced now.

On very high quality scans, i.e. 4k or more, you can easily see the single silver particle grain.

Anyway we are going OT. All i need is a VS MVTools with support for tr>3 :)